We found a few more issues and small items that were not resolved since the last release. Those have been fixed and I plan on cutting our release candidate later today.
-Tim On Mon, Nov 2, 2026 at 9:21 PM Tim Brown <[email protected]> wrote: > Thanks Vinish! > > I've merged the license check and will do an initial round of validation > before cutting the first release candidate. > > -Tim > > On Mon, Nov 2, 2026 at 10:26 AM Vinish Reddy via dev < > [email protected]> wrote: > >> Hi all, >> >> Status update on the 0.5.0 release, the PRs we were holding the release on >> have now been merged on main, so we can start with pending license >> validating changes and cutting rc1 after that. >> >> Merged since the last update: >> - Hudi table version 9 support in the Hudi target and source (#835), and >> the column-stats index for existing Hudi targets (#961) >> - Iceberg versions of TableFormat and HoodieTableMetadata for the >> pluggable >> table format (#894) >> - Hudi-backed secondary index for XTable source tables (#935) and the >> Spark >> SQL extensions for it (#992) >> - xtable-spark-runtime on main (#964) with Parquet source partitioning >> (#931) >> - Delta Kernel source and target coverage in ITConversionController (#903) >> - Correct bundled-jar license metadata checked in CI (#895), plus shade >> include validation (#971) and slimmer xtable-hive-metastore dependencies >> (#974, #979) >> - Fixes for CLEAN handling in incremental Hudi sync (#824), degenerate >> metadata table stats (#951), GlueCatalog and S3FileIO in xtable-utilities >> (#952), Guava compatibility (#967), Delta partition values under column >> mapping (#973) and parquet group nullability (#989) >> >> Thanks to everyone who contributed, reviewed and tested these. >> >> Tim, over to you as release manager for cutting rc1. #997 (thin jars keep >> third-party license texts out, #701) is approved and would be good to >> merge >> before the cut. >> >> PENDING PR REVIEWS >> >> Below is the current backlog of open PRs waiting for review. None of these >> are planned as rc1 blockers. If you think one should be, please reply on >> this thread. >> >> Delta Kernel Fixes (moved to 0.6.0 as the kernel is not the default path >> in >> 0.5.0) >> - #947 Preserve local paths for Delta Kernel (#943) >> - #942 Delta Kernel commit tag mapping (#819) >> - #938 Fail-safe handling for Delta deletion vectors (#657) >> >> Bug fixes >> - #993 Make Hudi snapshot sync idempotent for an already-synced source >> commit (#962) >> - #988 Preserve Hudi schema-on-read field IDs on column rename (#978) >> - #986 Decode Delta log paths when reading and encode them when writing >> (#970) >> - #985 Support VALUE partitions in the Parquet source (#927) >> - #984 Omit unconfigured column-mapping IDs (#936) >> - #976 Fix typed partition values in the Paimon source (#975) >> - #814 Filter metadata directories out of Hudi partitions (#813) >> >> Docs, tests and CI >> - #995 Running the Spark runtime jar on AWS Glue (#941) >> - #922 Review comments on the 0.4.0 Spark runtime docs (#920) >> - #998 Match Delta engine jars to the Spark version in the spark-runtime >> IT >> (#969) >> >> Features >> - #829 xtable-converter Claude Code skill >> - #830 Detect source table format >> - #823 Per-table timeout for RunSync >> - #621 Custom Iceberg metadata cleaner >> - #794, #796, #798, #799, #802, #811, #815 from Nicolas (fail on sync >> error, Avro comments, float/double stats, omit metadata columns, Unity >> Catalog sync, optional column stats, concurrent Hudi S3 footer collection) >> - #760 Column stats with Parquet footer fallback for Delta and Iceberg >> - #731 ORC schema converter >> - #712 Column rename in IcebergSchemaSync >> >> The full list is here: >> https://github.com/apache/incubator-xtable/pulls >> < >> https://www.google.com/url?q=https://github.com/apache/incubator-xtable/pulls&source=gmail&ust=1793719382341000&sa=E >> > >> >> I will share the roadmap for 0.6.0 on a separate thread this week, and the >> PRs above that miss 0.5.0 will be part of it. >> >> Thanks, >> Vinish >> >> On Fri, Oct 09, 2026 06:09 PM, Vinish <[email protected]> >> wrote: >> >> > Hi Sylwester, >> > >> > Thanks for going through the queue in so much detail. >> > >> > Field ids for map and list children >> > Thanks for flagging this. It stays on the 0.5.0 label, and if a fix >> > doesn't land in time we'll call out the remaining gap as a known issue >> in >> > the release notes. >> > >> > Spark/Delta and the col-stats PR >> > Would you be able to reopen your Spark 3.5.9 and Delta 3.3.3 PR, with >> the >> > scope limited to the Spark 3.5 and Delta Standalone version upgrade? As >> Tim >> > mentioned, having the version bump in its own PR would be valuable, and >> it >> > would give the Spark 4 work a clean base. The col-stats default change >> can >> > then rebase on top of it. >> > >> > Release notes and labels >> > Tim will start the 0.5.0 release notes after the rc1 branch is cut. >> > >> > Stale PRs and reviews >> > Agreed on both. As Tim mentioned, many of the older PRs were waiting on >> > Hudi 1.x fixes, and main is now on Hudi 1.2.1. I'll comment on the PRs >> > opened before 2026 and ask the authors to rebase or close them, and I'll >> > bring up a regular review cadence on the 0.6.0 kick off thread. >> > >> > Thanks, >> > Vinish >> > >> > On Fri, Oct 09, 2026 06:05 PM, Vinish <[email protected]> >> > wrote: >> > >> >> Hi Tim, >> >> >> >> Thanks for driving this. Taking your questions in order. >> >> >> >> License checks >> >> I’ve added some feedback on your PR ( >> >> https://github.com/apache/incubator-xtable/pull/895) about using >> >> check_rc.py, one of the incubator skills scripts for license >> verification. >> >> IPMC members have recently started relying on it when approving >> releases, >> >> so running it ourselves should help catch issues before the vote >> >> >> >> Spark and Delta upgrade >> >> As agreed on the August scope thread, let’s keep this for the next >> >> release. Sylwester closed the Spark 3.5.9 and Delta 3.3.3 PR in favour >> of >> >> the Spark 4 upgrade. I agree a standalone Spark and Delta bump would >> still >> >> be valuable, so it could be the first step of the Spark 4 work early in >> >> 0.6.0. I will be starting a 0.6.0 kick off thread early next week. >> >> >> >> Timeline >> >> Let's target Tuesday October 13 for reviewing and merging all the 0.5.0 >> >> tagged PR's? I will be reviewing PR's from Ethan, Akash, Vaibhav, >> Zhang and >> >> few others tagged with 0.5.0 >> >> >> >> >> >> >> https://github.com/apache/incubator-xtable/pulls?q=is%3Apr+state%3Aopen+label%3A0.5.0 >> >> >> >> >> >> Thanks, >> >> Vinish >> >> >> >> On Tue, Oct 06, 2026 06:28 PM, Tim Brown <[email protected]> >> wrote: >> >> >> >>> Hi folks, >> >>> The biggest hurdle I see right now is the handling of the license >> files >> >>> and >> >>> automation around it. There doesn't seem to be much alignment there >> and I >> >>> don't personally have much experience with it. Is there anyone with a >> >>> clear >> >>> idea of the path we should take? I know there have been a few threads >> >>> that >> >>> have died out on the subject. >> >>> >> >>> To add some context on some of the stale branches, they were >> dependent on >> >>> some features in the Hudi 1.X releases. Some of those releases >> introduced >> >>> regressions in behavior that XTable relied on so we had to wait for >> >>> another >> >>> release cycle. >> >>> >> >>> I still think it may be valuable to have the Delta and Spark versions >> >>> upgraded as part of their own PR as they are in #908 but it looks like >> >>> that >> >>> is now closed. Are we targeting a Spark and Delta upgrade in this >> >>> release? >> >>> >> >>> -Tim >> >>> >> >>> On Mon, Oct 5, 2026 at 6:51 AM Vinoth Chandar <[email protected]> wrote: >> >>> >> >>> > Thanks Tim for driving this release. >> >>> > >> >>> > On Fri, Oct 2, 2026 at 15:58 Vinish Reddy <[email protected]> >> wrote: >> >>> > >> >>> > > Hi all, >> >>> > > >> >>> > > Here is a status update on the 0.5.0-incubating release, following >> >>> up on >> >>> > > the scope and timeline thread from August [1]. Tim is the release >> >>> manager >> >>> > > for 0.5.0. >> >>> > > >> >>> > > The full list of tracked work is under the 0.5.0 label. >> >>> > > >> >>> > >> >>> >> https://github.com/apache/incubator-xtable/issues?q=is%3Aopen+label%3A0.5.0 >> >>> > > >> >>> > > MOMENTUM SINCE 0.4.0 >> >>> > > >> >>> > > Since August 1, 31 PRs have merged on main from 9 contributors. >> Some >> >>> > > highlights: >> >>> > > >> >>> > > - The xtable-utilities bundled jar shrank from 1.1 GB to 423 MB >> >>> (#897), >> >>> > and >> >>> > > the duplicate AWS SDK v2 modules were removed (#923). >> >>> > > - Delta nested field ids now carry into the converted schema >> (#912), >> >>> and >> >>> > > Delta log listing handles absolute paths (#932). >> >>> > > - Incremental sync is skipped when a Delta source has no new >> commits >> >>> > > (#930), and the Delta snapshot is reused across an incremental >> >>> backlog >> >>> > > (#861). >> >>> > > - Hudi partitions are batch-loaded before the parallel file >> listing >> >>> > (#950). >> >>> > > - Glue Catalog Federation works with Delta tables (#940). >> >>> > > - The parquet source now populates record_count (#929), and its >> >>> > limitations >> >>> > > are documented (#937). >> >>> > > - The build moved to ASF parent POM 39 and compiles on JDK 21+ >> (#881, >> >>> > > #900), with the Maven build cache persisted in CI (#882). >> >>> > > - The 0.4.0 release notes, blog post and Spark runtime quickstart >> are >> >>> > live >> >>> > > (#913, #915). >> >>> > > >> >>> > > Thanks to Sylwester Lachiewicz, Ranga Reddy, Arvin, Abdul Megoon, >> >>> Akash >> >>> > > Reddy Jammula, Tim Brown, Vamsi Karnika and AadhiKat for their >> >>> > > contributions, and to everyone who reviewed and tested. >> >>> > > >> >>> > > STATUS BY THEME >> >>> > > >> >>> > > 1. Hudi table version 9 (Hudi 1.x) support (#834) >> >>> > > - #835 adds v9 to the Hudi target and source. It now builds on the >> >>> > released >> >>> > > Hudi 1.2.1, CI is green, and it is ready for review. >> >>> > > >> >>> > > 2. Iceberg pluggable table format managed by Hudi (#722), led by >> >>> Ethan >> >>> > > - #894 implements the Iceberg versions of TableFormat and >> >>> > > HoodieTableMetadata. CI is green and review is in progress. >> >>> > > - #954 (Iceberg V3 deletion vectors and table services for MOR >> >>> tables) >> >>> > and >> >>> > > #955 (a Spark plugin that routes Iceberg writes through Hudi) are >> >>> early >> >>> > > drafts that build on #894. >> >>> > > >> >>> > > 3. Cross-format indexing (#887) >> >>> > > - #933 is RFC-4 for index support. >> >>> > > - #935 adds a Hudi-backed secondary index for Iceberg tables. It >> has >> >>> been >> >>> > > validated on large tables on S3 and is close to ready. >> >>> > > - The xtable-spark-extensions work (Spark SQL to build, drop and >> >>> check >> >>> > the >> >>> > > staleness of indexes) is pushed and stacks on #935. A PR will >> follow. >> >>> > > >> >>> > > 4. xtable-spark-runtime for Hudi 1.x (#836) >> >>> > > - The runtime shipped for 0.4.x on branch-0.4 (#843). >> >>> > > - #838 (thin drop-in bundle and resilient target discovery) and >> #839 >> >>> > > (XTableSyncListener for in-job sync) bring it to main. >> >>> > > - #931 adds Parquet source partitioning to the spark runtime >> (#927). >> >>> > > - #908 from Sylwester moves the build to Spark 3.5.9 and Delta >> >>> 3.3.3, as >> >>> > > the first step toward the Spark 4 upgrade (#902). Reviews are >> welcome >> >>> > there >> >>> > > too. >> >>> > > >> >>> > > 5. Delta Kernel as the default Delta path (#886, #713) >> >>> > > - #903 adds Delta Kernel source and target coverage to >> >>> > > ITConversionController and is approved. >> >>> > > - #947 (local paths, #943) and #942 (commit tag mapping, #819) fix >> >>> Kernel >> >>> > > gaps found along the way. >> >>> > > >> >>> > > 6. Correctness and robustness fixes. >> >>> > > - #951 falls back to parquet-footer stats when the Hudi metadata >> >>> table >> >>> > > returns degenerate values. It is approved and ready to merge. >> >>> > > - #621 adds a custom Iceberg metadata cleaner. >> >>> > > - #956 resolves the parquet source file system from the base path, >> >>> and >> >>> > #957 >> >>> > > keeps absolute paths for data files outside the table base path in >> >>> Delta >> >>> > > targets. >> >>> > > - #938 adds fail-safe handling for Delta deletion vectors (#657). >> >>> > > - #952 bundles Iceberg's GlueCatalog and S3FileIO in >> xtable-utilities >> >>> > > (#944). >> >>> > > - #814 filters metadata directories out of Hudi partitions (#813). >> >>> > > - Open bugs without a PR yet are #958, #959, #941, #936, #901 and >> >>> #736. >> >>> > > Help on any of these is welcome. >> >>> > > >> >>> > > 7. License and packaging (#701, #717, #865) >> >>> > > - #876 keeps exactly one LICENSE and NOTICE in every published >> jar, >> >>> and >> >>> > > #895 makes the bundled-jar license metadata correct and checks it >> in >> >>> CI. >> >>> > > - It would be good to land these in 0.5.0, since they address the >> >>> > footnotes >> >>> > > from the 0.4.0 vote. >> >>> > > >> >>> > > 8. AI agent tooling (#888) >> >>> > > - The issues for an agent-legible CLI, a datasetConfig JSON Schema >> >>> and an >> >>> > > MCP surface are filed (#889, #890, #891), and #829 adds an >> >>> > xtable-converter >> >>> > > skill. These are open for contributors. >> >>> > > >> >>> > > 9. Docs >> >>> > > - #922 addresses the open review comments on the 0.4.0 Spark >> runtime >> >>> docs >> >>> > > (#920), and #921 covers when Delta Kernel is selected. >> >>> > > >> >>> > > PROPOSED TIMELINE >> >>> > > >> >>> > > I propose Thursday, October 8 as the code cutoff for 0.5.0, with >> >>> anything >> >>> > > not merged by then moving to 0.6.0. Tim, does that work for you? >> >>> > > >> >>> > > Reviews this week would make the biggest difference. In >> particular, >> >>> it >> >>> > > would be good to get eyes on #835, #894, #933, #935, #838, #839, >> >>> #876 and >> >>> > > #895. >> >>> > > If there is an item you think must be in 0.5.0 that is not on the >> >>> label, >> >>> > or >> >>> > > if the date does not work for your area, please reply on this >> thread. >> >>> > > >> >>> > > [1] [DISCUSS] Scope and timeline for Apache XTable (incubating) >> >>> 0.5.0 and >> >>> > > future releases >> >>> > > >> >>> > > Thanks, >> >>> > > Vinish >> >>> > > >> >>> > >> >>> >> >> >> >
