vinishjail97 commented on issue #855:
URL:
https://github.com/apache/incubator-xtable/issues/855#issuecomment-5223005730
# Release Notes for release-0.4.0-incubating-rc2
Apache XTable™ (Incubating) is a cross-table converter for table formats
that facilitates omni-directional interoperability across data processing
systems and query engines. Currently, Apache XTable™ supports widely adopted
open-source table formats such as Apache Hudi, Apache Iceberg, Delta Lake, and
(as a source) Apache Paimon.
## 🔁 Changes since rc1
rc1 was cancelled. This candidate contains the following additional changes.
- **Make the source release build with `mvn package`** (#864, #868).
`quarkus-maven-plugin` bootstraps the application to generate code, so it
resolves `xtable-service`'s dependencies with Quarkus' own bootstrap resolver
instead of through the reactor. That resolver discovered the workspace from the
raw poms without interpolating properties, so the module declaring
`xtable-core_${scala.binary.version}` was never matched against the
`xtable-core_2.12` dependency and resolution fell back to Maven Central, where
the artifact does not exist until the release is published. `.mvn/maven.config`
now sets `-Dquarkus.bootstrap.effective-model-builder=true` so workspace
discovery uses the effective model. A new **Source Build Check** workflow
builds `xtable-service` with the local repository cleared of XTable artifacts
and no caching, so this cannot regress silently.
- **Cover 2026 in the NOTICE copyright range** (#872, #874), and check the
range at release time as part of `validate_source_copyright.sh`.
- Exclude the `xtable-service/examples` screenshots from the source release
tarball, so the source release contains no binary files.
### Known issues
The license documentation issues raised during the rc1 review remain open
and are targeted for the next release: incomplete license information for
bundled jars (#700), multiple LICENSE files in some jars (#701), and orphaned
license files in bundled jars (#865).
## ✨ What's Changed
### 🔄 Core Functionality Enhancements
- **New formats**:
- Added a **Delta Kernel** conversion target (#801, #729).
- Added **Apache Paimon source** support, with incremental sync and column
stats (#742, #780, #767).
- Added **Apache Parquet source** support, with partition extraction and
incremental sync (#728, #669, #768, #805).
- **Sync improvements**:
- Delta source: extract **nested field statistics** (#769).
- **Nested-column partition** support in `HudiPartitionValuesExtractor`
with hive-style partitioning (#738).
- Fall back to **parquet footers** when metadata-table column stats are
missing (#818).
- Enable **Delta checkpoint CRC** exception throwing by default (#817).
- Add additional properties when converting for **Hudi** (#714).
- **Bug fixes**:
- Delta: null partition value for composite generated-column partitions
(#828), map-key path handling (#826), NPE for binary in map/array schemas
(#795), column-mapping-enabled source tables (#766).
- Hudi: batch `INSERT_OVERWRITE` replacecommits dropping adds (#816),
empty `EarliestCommitToRetain` (#732).
- Iceberg: nested comments with qualified name (#797), schema sync (#749),
empty source table (#756).
- Parquet: snapshot sync on multiple commits with partitions (#806),
schema-conversion bugs (#805).
### 📦 Packaging & Distribution
- Added **xtable-spark-runtime**, a thin drop-in Spark bundle for Spark 3.4
and 3.5 (#843).
- Made the bundled **xtable-utilities** jar runnable again (#841, #736).
- Fixed the scope of spark-catalyst in the utilities bundle (#790).
### 🧪 Testing & Quality
- Added an initial integration test for the **XTable REST service** (#716).
- Improved `TestHudiInstantUtils` with parameterization and more cases
(#703).
- Removed wildcard imports and enforced with **Spotless** (#809).
- Removed redundant `getSnapshotAt` calls per commit (#791).
### 📘 Documentation & Community
- Added a **0.3.0 release page** (#725) and updated the **community sync**
page (#734).
### 🧰 Infrastructure & Maintenance
- Upgraded **Apache Iceberg** to 1.9.2 and **Avro** to 1.12.0 (#784).
- Added a **Maven wrapper** for consistent builds (#788).
- Automated **license file generation and validation** (#822); removed a
duplicate `javassist` LICENSE entry (#745).
- Restored the **delombok** execution so release builds generate javadoc
jars (#853).
- Added a **Source Build Check** workflow that builds the source release
without any installed XTable artifacts (#868).
- Added `.sdkmanrc` to `.gitignore` (#793); bumped GitHub Actions
(`checkout`, `setup-node`, `setup-java`).
## Detailed Change Log
* [553] Parquet Schema and Column Stats Converters by @sapienza88 in
https://github.com/apache/incubator-xtable/pull/669
* Add additional properties when doing conversion for hudi by @rahil-c in
https://github.com/apache/incubator-xtable/pull/714
* Improved all tests in TestHudiInstantUtils by parameterising and adding
more test cases by @Monilnarang in
https://github.com/apache/incubator-xtable/pull/703
* Fix typo in InternalTable by @vinishjail97 in
https://github.com/apache/incubator-xtable/pull/720
* [DOCS] Add a new 0.3.0 release page by @sagarlakshmipathy in
https://github.com/apache/incubator-xtable/pull/725
* Add intial IT for XTable REST Service by @rahil-c in
https://github.com/apache/incubator-xtable/pull/716
* Update community sync page by @pratyakshsharma in
https://github.com/apache/incubator-xtable/pull/734
* Bump actions/setup-java from 4 to 5 by @dependabot[bot] in
https://github.com/apache/incubator-xtable/pull/740
* [739] Nested column partition support in HudiPartitionValuesExtractor when
hive style partitioning is enabled by @kroushan-nit in
https://github.com/apache/incubator-xtable/pull/738
* Bump actions/setup-node from 4 to 5 by @dependabot[bot] in
https://github.com/apache/incubator-xtable/pull/741
* Bump actions/checkout from 4 to 5 by @dependabot[bot] in
https://github.com/apache/incubator-xtable/pull/737
* [553] parquet source with partition extraction by @sapienza88 in
https://github.com/apache/incubator-xtable/pull/728
* [LICENSE] Remove duplicate entry for org.javassist:javassist:3.25.0-G… by
@piyushdubey in https://github.com/apache/incubator-xtable/pull/745
* [750] Fix schema sync for Iceberg tables by @the-other-tim-brown in
https://github.com/apache/incubator-xtable/pull/749
* Paimon Source Support by @mikedias in
https://github.com/apache/incubator-xtable/pull/742
* Handle empty Iceberg table source in conversion by @brishi19791 in
https://github.com/apache/incubator-xtable/pull/756
* Bump actions/checkout from 5 to 6 by @dependabot[bot] in
https://github.com/apache/incubator-xtable/pull/761
* Delta Kernel Draft PR by @vaibhavk1992 in
https://github.com/apache/incubator-xtable/pull/729
* Delta source: extract nested field's stats by @emilie-wang in
https://github.com/apache/incubator-xtable/pull/769
* Fix Delta to Iceberg not working on column mapping enabled Delta source
table by @xr-chen in https://github.com/apache/incubator-xtable/pull/766
* Implement Paimon Source Incremental Sync by @mikedias in
https://github.com/apache/incubator-xtable/pull/780
* [787] Add maven wrapper for consistent builds by @the-other-tim-brown in
https://github.com/apache/incubator-xtable/pull/788
* [755] Support column stats for paimon by @mao-liu in
https://github.com/apache/incubator-xtable/pull/767
* [789] Fix scope of spark catalyst in utilities bundle by
@the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/790
* Upgrade Iceberg to 1.9.2 and Avro to 1.12.0 by @kevinjqliu in
https://github.com/apache/incubator-xtable/pull/784
* fix(delta): avoid NPE for binary in map/array schemas by @parisni in
https://github.com/apache/incubator-xtable/pull/795
* Remove redundant getSnapshotAt calls per commit by @brishi19791 in
https://github.com/apache/incubator-xtable/pull/791
* fix(iceberg): nested comments with qualified name by @parisni in
https://github.com/apache/incubator-xtable/pull/797
* add .sdkmanrc to .gitignore by @kevinjqliu in
https://github.com/apache/incubator-xtable/pull/793
* Remove wildcard imports and enforce with spotless by @the-other-tim-brown
in https://github.com/apache/incubator-xtable/pull/809
* Parquet source: Expand column stats support, fix bugs in schema conversion
by @the-other-tim-brown in https://github.com/apache/incubator-xtable/pull/805
* [bug] Parquet Source: snapshot sync fails on multiple commits with
partitions by @the-other-tim-brown in
https://github.com/apache/incubator-xtable/pull/806
* Enable Delta checkpoint CRC exception throwing by default by @vinishjail97
in https://github.com/apache/incubator-xtable/pull/817
* Bump actions/setup-node from 5 to 6 by @dependabot[bot] in
https://github.com/apache/incubator-xtable/pull/753
* Handle empty EarliestCommitToRetain by @alexr17 in
https://github.com/apache/incubator-xtable/pull/732
* [bug] Fix batch INSERT_OVERWRITE replacecommits dropping adds in
HudiDataFileExtractor by @vinishjail97 in
https://github.com/apache/incubator-xtable/pull/816
* Fallback to parquet footers when metadata table col stats are missing by
@vinishjail97 in https://github.com/apache/incubator-xtable/pull/818
* Automate license file generation and validation by @the-other-tim-brown in
https://github.com/apache/incubator-xtable/pull/822
* Delta kernel conversion target by @vaibhavk1992 in
https://github.com/apache/incubator-xtable/pull/801
* Fix map key path handling in Delta schema extractors by @brishi19791 in
https://github.com/apache/incubator-xtable/pull/826
* Parquet Incremental Sync by @sapienza88 in
https://github.com/apache/incubator-xtable/pull/768
* [MINOR] Preserve original cause in ReadException on close failure by
@alealandreev in https://github.com/apache/incubator-xtable/pull/827
* fix(delta): null partition value for composite generated-column partitions
by @alealandreev in https://github.com/apache/incubator-xtable/pull/828
* [736] Make the bundled xtable-utilities jar runnable again (Hudi 0.x) + CI
on branch-0.4 by @vinishjail97 in
https://github.com/apache/incubator-xtable/pull/841
* [836] xtable-spark-runtime: thin drop-in Spark bundle for 0.4.x (Spark 3.4
+ 3.5) by @vinishjail97 in https://github.com/apache/incubator-xtable/pull/843
* fix(build): restore delombok execution so release builds generate javadoc
jars by @vinishjail97 in https://github.com/apache/incubator-xtable/pull/853
* [864] Make the source release build with mvn package by @vinishjail97 in
https://github.com/apache/incubator-xtable/pull/868
* [872] Cover 2026 in the NOTICE copyright range and check it at release
time by @vinishjail97 in https://github.com/apache/incubator-xtable/pull/874
## New Contributors
* @sapienza88 made their first contribution in
https://github.com/apache/incubator-xtable/pull/669
* @Monilnarang made their first contribution in
https://github.com/apache/incubator-xtable/pull/703
* @pratyakshsharma made their first contribution in
https://github.com/apache/incubator-xtable/pull/734
* @mikedias made their first contribution in
https://github.com/apache/incubator-xtable/pull/742
* @xr-chen made their first contribution in
https://github.com/apache/incubator-xtable/pull/766
* @mao-liu made their first contribution in
https://github.com/apache/incubator-xtable/pull/767
* @kevinjqliu made their first contribution in
https://github.com/apache/incubator-xtable/pull/784
* @parisni made their first contribution in
https://github.com/apache/incubator-xtable/pull/795
* @alexr17 made their first contribution in
https://github.com/apache/incubator-xtable/pull/732
* @alealandreev made their first contribution in
https://github.com/apache/incubator-xtable/pull/827
**Full Changelog**:
https://github.com/apache/incubator-xtable/compare/0.3.0-incubating...0.4.0-incubating-rc2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
