Thanks Vinish! I've merged the license check and will do an initial round of validation before cutting the first release candidate.
-Tim On Mon, Nov 2, 2026 at 10:26 AM Vinish Reddy via dev <[email protected]> wrote: > Hi all, > > Status update on the 0.5.0 release, the PRs we were holding the release on > have now been merged on main, so we can start with pending license > validating changes and cutting rc1 after that. > > Merged since the last update: > - Hudi table version 9 support in the Hudi target and source (#835), and > the column-stats index for existing Hudi targets (#961) > - Iceberg versions of TableFormat and HoodieTableMetadata for the pluggable > table format (#894) > - Hudi-backed secondary index for XTable source tables (#935) and the Spark > SQL extensions for it (#992) > - xtable-spark-runtime on main (#964) with Parquet source partitioning > (#931) > - Delta Kernel source and target coverage in ITConversionController (#903) > - Correct bundled-jar license metadata checked in CI (#895), plus shade > include validation (#971) and slimmer xtable-hive-metastore dependencies > (#974, #979) > - Fixes for CLEAN handling in incremental Hudi sync (#824), degenerate > metadata table stats (#951), GlueCatalog and S3FileIO in xtable-utilities > (#952), Guava compatibility (#967), Delta partition values under column > mapping (#973) and parquet group nullability (#989) > > Thanks to everyone who contributed, reviewed and tested these. > > Tim, over to you as release manager for cutting rc1. #997 (thin jars keep > third-party license texts out, #701) is approved and would be good to merge > before the cut. > > PENDING PR REVIEWS > > Below is the current backlog of open PRs waiting for review. None of these > are planned as rc1 blockers. If you think one should be, please reply on > this thread. > > Delta Kernel Fixes (moved to 0.6.0 as the kernel is not the default path in > 0.5.0) > - #947 Preserve local paths for Delta Kernel (#943) > - #942 Delta Kernel commit tag mapping (#819) > - #938 Fail-safe handling for Delta deletion vectors (#657) > > Bug fixes > - #993 Make Hudi snapshot sync idempotent for an already-synced source > commit (#962) > - #988 Preserve Hudi schema-on-read field IDs on column rename (#978) > - #986 Decode Delta log paths when reading and encode them when writing > (#970) > - #985 Support VALUE partitions in the Parquet source (#927) > - #984 Omit unconfigured column-mapping IDs (#936) > - #976 Fix typed partition values in the Paimon source (#975) > - #814 Filter metadata directories out of Hudi partitions (#813) > > Docs, tests and CI > - #995 Running the Spark runtime jar on AWS Glue (#941) > - #922 Review comments on the 0.4.0 Spark runtime docs (#920) > - #998 Match Delta engine jars to the Spark version in the spark-runtime IT > (#969) > > Features > - #829 xtable-converter Claude Code skill > - #830 Detect source table format > - #823 Per-table timeout for RunSync > - #621 Custom Iceberg metadata cleaner > - #794, #796, #798, #799, #802, #811, #815 from Nicolas (fail on sync > error, Avro comments, float/double stats, omit metadata columns, Unity > Catalog sync, optional column stats, concurrent Hudi S3 footer collection) > - #760 Column stats with Parquet footer fallback for Delta and Iceberg > - #731 ORC schema converter > - #712 Column rename in IcebergSchemaSync > > The full list is here: > https://github.com/apache/incubator-xtable/pulls > < > https://www.google.com/url?q=https://github.com/apache/incubator-xtable/pulls&source=gmail&ust=1793719382341000&sa=E > > > > I will share the roadmap for 0.6.0 on a separate thread this week, and the > PRs above that miss 0.5.0 will be part of it. > > Thanks, > Vinish > > On Fri, Oct 09, 2026 06:09 PM, Vinish <[email protected]> > wrote: > > > Hi Sylwester, > > > > Thanks for going through the queue in so much detail. > > > > Field ids for map and list children > > Thanks for flagging this. It stays on the 0.5.0 label, and if a fix > > doesn't land in time we'll call out the remaining gap as a known issue in > > the release notes. > > > > Spark/Delta and the col-stats PR > > Would you be able to reopen your Spark 3.5.9 and Delta 3.3.3 PR, with the > > scope limited to the Spark 3.5 and Delta Standalone version upgrade? As > Tim > > mentioned, having the version bump in its own PR would be valuable, and > it > > would give the Spark 4 work a clean base. The col-stats default change > can > > then rebase on top of it. > > > > Release notes and labels > > Tim will start the 0.5.0 release notes after the rc1 branch is cut. > > > > Stale PRs and reviews > > Agreed on both. As Tim mentioned, many of the older PRs were waiting on > > Hudi 1.x fixes, and main is now on Hudi 1.2.1. I'll comment on the PRs > > opened before 2026 and ask the authors to rebase or close them, and I'll > > bring up a regular review cadence on the 0.6.0 kick off thread. > > > > Thanks, > > Vinish > > > > On Fri, Oct 09, 2026 06:05 PM, Vinish <[email protected]> > > wrote: > > > >> Hi Tim, > >> > >> Thanks for driving this. Taking your questions in order. > >> > >> License checks > >> I’ve added some feedback on your PR ( > >> https://github.com/apache/incubator-xtable/pull/895) about using > >> check_rc.py, one of the incubator skills scripts for license > verification. > >> IPMC members have recently started relying on it when approving > releases, > >> so running it ourselves should help catch issues before the vote > >> > >> Spark and Delta upgrade > >> As agreed on the August scope thread, let’s keep this for the next > >> release. Sylwester closed the Spark 3.5.9 and Delta 3.3.3 PR in favour > of > >> the Spark 4 upgrade. I agree a standalone Spark and Delta bump would > still > >> be valuable, so it could be the first step of the Spark 4 work early in > >> 0.6.0. I will be starting a 0.6.0 kick off thread early next week. > >> > >> Timeline > >> Let's target Tuesday October 13 for reviewing and merging all the 0.5.0 > >> tagged PR's? I will be reviewing PR's from Ethan, Akash, Vaibhav, Zhang > and > >> few others tagged with 0.5.0 > >> > >> > >> > https://github.com/apache/incubator-xtable/pulls?q=is%3Apr+state%3Aopen+label%3A0.5.0 > >> > >> > >> Thanks, > >> Vinish > >> > >> On Tue, Oct 06, 2026 06:28 PM, Tim Brown <[email protected]> > wrote: > >> > >>> Hi folks, > >>> The biggest hurdle I see right now is the handling of the license files > >>> and > >>> automation around it. There doesn't seem to be much alignment there > and I > >>> don't personally have much experience with it. Is there anyone with a > >>> clear > >>> idea of the path we should take? I know there have been a few threads > >>> that > >>> have died out on the subject. > >>> > >>> To add some context on some of the stale branches, they were dependent > on > >>> some features in the Hudi 1.X releases. Some of those releases > introduced > >>> regressions in behavior that XTable relied on so we had to wait for > >>> another > >>> release cycle. > >>> > >>> I still think it may be valuable to have the Delta and Spark versions > >>> upgraded as part of their own PR as they are in #908 but it looks like > >>> that > >>> is now closed. Are we targeting a Spark and Delta upgrade in this > >>> release? > >>> > >>> -Tim > >>> > >>> On Mon, Oct 5, 2026 at 6:51 AM Vinoth Chandar <[email protected]> wrote: > >>> > >>> > Thanks Tim for driving this release. > >>> > > >>> > On Fri, Oct 2, 2026 at 15:58 Vinish Reddy <[email protected]> wrote: > >>> > > >>> > > Hi all, > >>> > > > >>> > > Here is a status update on the 0.5.0-incubating release, following > >>> up on > >>> > > the scope and timeline thread from August [1]. Tim is the release > >>> manager > >>> > > for 0.5.0. > >>> > > > >>> > > The full list of tracked work is under the 0.5.0 label. > >>> > > > >>> > > >>> > https://github.com/apache/incubator-xtable/issues?q=is%3Aopen+label%3A0.5.0 > >>> > > > >>> > > MOMENTUM SINCE 0.4.0 > >>> > > > >>> > > Since August 1, 31 PRs have merged on main from 9 contributors. > Some > >>> > > highlights: > >>> > > > >>> > > - The xtable-utilities bundled jar shrank from 1.1 GB to 423 MB > >>> (#897), > >>> > and > >>> > > the duplicate AWS SDK v2 modules were removed (#923). > >>> > > - Delta nested field ids now carry into the converted schema > (#912), > >>> and > >>> > > Delta log listing handles absolute paths (#932). > >>> > > - Incremental sync is skipped when a Delta source has no new > commits > >>> > > (#930), and the Delta snapshot is reused across an incremental > >>> backlog > >>> > > (#861). > >>> > > - Hudi partitions are batch-loaded before the parallel file listing > >>> > (#950). > >>> > > - Glue Catalog Federation works with Delta tables (#940). > >>> > > - The parquet source now populates record_count (#929), and its > >>> > limitations > >>> > > are documented (#937). > >>> > > - The build moved to ASF parent POM 39 and compiles on JDK 21+ > (#881, > >>> > > #900), with the Maven build cache persisted in CI (#882). > >>> > > - The 0.4.0 release notes, blog post and Spark runtime quickstart > are > >>> > live > >>> > > (#913, #915). > >>> > > > >>> > > Thanks to Sylwester Lachiewicz, Ranga Reddy, Arvin, Abdul Megoon, > >>> Akash > >>> > > Reddy Jammula, Tim Brown, Vamsi Karnika and AadhiKat for their > >>> > > contributions, and to everyone who reviewed and tested. > >>> > > > >>> > > STATUS BY THEME > >>> > > > >>> > > 1. Hudi table version 9 (Hudi 1.x) support (#834) > >>> > > - #835 adds v9 to the Hudi target and source. It now builds on the > >>> > released > >>> > > Hudi 1.2.1, CI is green, and it is ready for review. > >>> > > > >>> > > 2. Iceberg pluggable table format managed by Hudi (#722), led by > >>> Ethan > >>> > > - #894 implements the Iceberg versions of TableFormat and > >>> > > HoodieTableMetadata. CI is green and review is in progress. > >>> > > - #954 (Iceberg V3 deletion vectors and table services for MOR > >>> tables) > >>> > and > >>> > > #955 (a Spark plugin that routes Iceberg writes through Hudi) are > >>> early > >>> > > drafts that build on #894. > >>> > > > >>> > > 3. Cross-format indexing (#887) > >>> > > - #933 is RFC-4 for index support. > >>> > > - #935 adds a Hudi-backed secondary index for Iceberg tables. It > has > >>> been > >>> > > validated on large tables on S3 and is close to ready. > >>> > > - The xtable-spark-extensions work (Spark SQL to build, drop and > >>> check > >>> > the > >>> > > staleness of indexes) is pushed and stacks on #935. A PR will > follow. > >>> > > > >>> > > 4. xtable-spark-runtime for Hudi 1.x (#836) > >>> > > - The runtime shipped for 0.4.x on branch-0.4 (#843). > >>> > > - #838 (thin drop-in bundle and resilient target discovery) and > #839 > >>> > > (XTableSyncListener for in-job sync) bring it to main. > >>> > > - #931 adds Parquet source partitioning to the spark runtime > (#927). > >>> > > - #908 from Sylwester moves the build to Spark 3.5.9 and Delta > >>> 3.3.3, as > >>> > > the first step toward the Spark 4 upgrade (#902). Reviews are > welcome > >>> > there > >>> > > too. > >>> > > > >>> > > 5. Delta Kernel as the default Delta path (#886, #713) > >>> > > - #903 adds Delta Kernel source and target coverage to > >>> > > ITConversionController and is approved. > >>> > > - #947 (local paths, #943) and #942 (commit tag mapping, #819) fix > >>> Kernel > >>> > > gaps found along the way. > >>> > > > >>> > > 6. Correctness and robustness fixes. > >>> > > - #951 falls back to parquet-footer stats when the Hudi metadata > >>> table > >>> > > returns degenerate values. It is approved and ready to merge. > >>> > > - #621 adds a custom Iceberg metadata cleaner. > >>> > > - #956 resolves the parquet source file system from the base path, > >>> and > >>> > #957 > >>> > > keeps absolute paths for data files outside the table base path in > >>> Delta > >>> > > targets. > >>> > > - #938 adds fail-safe handling for Delta deletion vectors (#657). > >>> > > - #952 bundles Iceberg's GlueCatalog and S3FileIO in > xtable-utilities > >>> > > (#944). > >>> > > - #814 filters metadata directories out of Hudi partitions (#813). > >>> > > - Open bugs without a PR yet are #958, #959, #941, #936, #901 and > >>> #736. > >>> > > Help on any of these is welcome. > >>> > > > >>> > > 7. License and packaging (#701, #717, #865) > >>> > > - #876 keeps exactly one LICENSE and NOTICE in every published jar, > >>> and > >>> > > #895 makes the bundled-jar license metadata correct and checks it > in > >>> CI. > >>> > > - It would be good to land these in 0.5.0, since they address the > >>> > footnotes > >>> > > from the 0.4.0 vote. > >>> > > > >>> > > 8. AI agent tooling (#888) > >>> > > - The issues for an agent-legible CLI, a datasetConfig JSON Schema > >>> and an > >>> > > MCP surface are filed (#889, #890, #891), and #829 adds an > >>> > xtable-converter > >>> > > skill. These are open for contributors. > >>> > > > >>> > > 9. Docs > >>> > > - #922 addresses the open review comments on the 0.4.0 Spark > runtime > >>> docs > >>> > > (#920), and #921 covers when Delta Kernel is selected. > >>> > > > >>> > > PROPOSED TIMELINE > >>> > > > >>> > > I propose Thursday, October 8 as the code cutoff for 0.5.0, with > >>> anything > >>> > > not merged by then moving to 0.6.0. Tim, does that work for you? > >>> > > > >>> > > Reviews this week would make the biggest difference. In particular, > >>> it > >>> > > would be good to get eyes on #835, #894, #933, #935, #838, #839, > >>> #876 and > >>> > > #895. > >>> > > If there is an item you think must be in 0.5.0 that is not on the > >>> label, > >>> > or > >>> > > if the date does not work for your area, please reply on this > thread. > >>> > > > >>> > > [1] [DISCUSS] Scope and timeline for Apache XTable (incubating) > >>> 0.5.0 and > >>> > > future releases > >>> > > > >>> > > Thanks, > >>> > > Vinish > >>> > > > >>> > > >>> > >> >
