Hi Sylwester, Thanks for going through the queue in so much detail.
Field ids for map and list children Thanks for flagging this. It stays on the 0.5.0 label, and if a fix doesn't land in time we'll call out the remaining gap as a known issue in the release notes. Spark/Delta and the col-stats PR Would you be able to reopen your Spark 3.5.9 and Delta 3.3.3 PR, with the scope limited to the Spark 3.5 and Delta Standalone version upgrade? As Tim mentioned, having the version bump in its own PR would be valuable, and it would give the Spark 4 work a clean base. The col-stats default change can then rebase on top of it. Release notes and labels Tim will start the 0.5.0 release notes after the rc1 branch is cut. Stale PRs and reviews Agreed on both. As Tim mentioned, many of the older PRs were waiting on Hudi 1.x fixes, and main is now on Hudi 1.2.1. I'll comment on the PRs opened before 2026 and ask the authors to rebase or close them, and I'll bring up a regular review cadence on the 0.6.0 kick off thread. Thanks, Vinish On Fri, Oct 09, 2026 06:05 PM, Vinish <[email protected]> wrote: > Hi Tim, > > Thanks for driving this. Taking your questions in order. > > License checks > I’ve added some feedback on your PR ( > https://github.com/apache/incubator-xtable/pull/895) about using > check_rc.py, one of the incubator skills scripts for license verification. > IPMC members have recently started relying on it when approving releases, > so running it ourselves should help catch issues before the vote > > Spark and Delta upgrade > As agreed on the August scope thread, let’s keep this for the next > release. Sylwester closed the Spark 3.5.9 and Delta 3.3.3 PR in favour of > the Spark 4 upgrade. I agree a standalone Spark and Delta bump would still > be valuable, so it could be the first step of the Spark 4 work early in > 0.6.0. I will be starting a 0.6.0 kick off thread early next week. > > Timeline > Let's target Tuesday October 13 for reviewing and merging all the 0.5.0 > tagged PR's? I will be reviewing PR's from Ethan, Akash, Vaibhav, Zhang and > few others tagged with 0.5.0 > > > https://github.com/apache/incubator-xtable/pulls?q=is%3Apr+state%3Aopen+label%3A0.5.0 > > > Thanks, > Vinish > > On Tue, Oct 06, 2026 06:28 PM, Tim Brown <[email protected]> wrote: > >> Hi folks, >> The biggest hurdle I see right now is the handling of the license files >> and >> automation around it. There doesn't seem to be much alignment there and I >> don't personally have much experience with it. Is there anyone with a >> clear >> idea of the path we should take? I know there have been a few threads that >> have died out on the subject. >> >> To add some context on some of the stale branches, they were dependent on >> some features in the Hudi 1.X releases. Some of those releases introduced >> regressions in behavior that XTable relied on so we had to wait for >> another >> release cycle. >> >> I still think it may be valuable to have the Delta and Spark versions >> upgraded as part of their own PR as they are in #908 but it looks like >> that >> is now closed. Are we targeting a Spark and Delta upgrade in this release? >> >> -Tim >> >> On Mon, Oct 5, 2026 at 6:51 AM Vinoth Chandar <[email protected]> wrote: >> >> > Thanks Tim for driving this release. >> > >> > On Fri, Oct 2, 2026 at 15:58 Vinish Reddy <[email protected]> wrote: >> > >> > > Hi all, >> > > >> > > Here is a status update on the 0.5.0-incubating release, following up >> on >> > > the scope and timeline thread from August [1]. Tim is the release >> manager >> > > for 0.5.0. >> > > >> > > The full list of tracked work is under the 0.5.0 label. >> > > >> > >> https://github.com/apache/incubator-xtable/issues?q=is%3Aopen+label%3A0.5.0 >> > > >> > > MOMENTUM SINCE 0.4.0 >> > > >> > > Since August 1, 31 PRs have merged on main from 9 contributors. Some >> > > highlights: >> > > >> > > - The xtable-utilities bundled jar shrank from 1.1 GB to 423 MB >> (#897), >> > and >> > > the duplicate AWS SDK v2 modules were removed (#923). >> > > - Delta nested field ids now carry into the converted schema (#912), >> and >> > > Delta log listing handles absolute paths (#932). >> > > - Incremental sync is skipped when a Delta source has no new commits >> > > (#930), and the Delta snapshot is reused across an incremental backlog >> > > (#861). >> > > - Hudi partitions are batch-loaded before the parallel file listing >> > (#950). >> > > - Glue Catalog Federation works with Delta tables (#940). >> > > - The parquet source now populates record_count (#929), and its >> > limitations >> > > are documented (#937). >> > > - The build moved to ASF parent POM 39 and compiles on JDK 21+ (#881, >> > > #900), with the Maven build cache persisted in CI (#882). >> > > - The 0.4.0 release notes, blog post and Spark runtime quickstart are >> > live >> > > (#913, #915). >> > > >> > > Thanks to Sylwester Lachiewicz, Ranga Reddy, Arvin, Abdul Megoon, >> Akash >> > > Reddy Jammula, Tim Brown, Vamsi Karnika and AadhiKat for their >> > > contributions, and to everyone who reviewed and tested. >> > > >> > > STATUS BY THEME >> > > >> > > 1. Hudi table version 9 (Hudi 1.x) support (#834) >> > > - #835 adds v9 to the Hudi target and source. It now builds on the >> > released >> > > Hudi 1.2.1, CI is green, and it is ready for review. >> > > >> > > 2. Iceberg pluggable table format managed by Hudi (#722), led by Ethan >> > > - #894 implements the Iceberg versions of TableFormat and >> > > HoodieTableMetadata. CI is green and review is in progress. >> > > - #954 (Iceberg V3 deletion vectors and table services for MOR tables) >> > and >> > > #955 (a Spark plugin that routes Iceberg writes through Hudi) are >> early >> > > drafts that build on #894. >> > > >> > > 3. Cross-format indexing (#887) >> > > - #933 is RFC-4 for index support. >> > > - #935 adds a Hudi-backed secondary index for Iceberg tables. It has >> been >> > > validated on large tables on S3 and is close to ready. >> > > - The xtable-spark-extensions work (Spark SQL to build, drop and check >> > the >> > > staleness of indexes) is pushed and stacks on #935. A PR will follow. >> > > >> > > 4. xtable-spark-runtime for Hudi 1.x (#836) >> > > - The runtime shipped for 0.4.x on branch-0.4 (#843). >> > > - #838 (thin drop-in bundle and resilient target discovery) and #839 >> > > (XTableSyncListener for in-job sync) bring it to main. >> > > - #931 adds Parquet source partitioning to the spark runtime (#927). >> > > - #908 from Sylwester moves the build to Spark 3.5.9 and Delta 3.3.3, >> as >> > > the first step toward the Spark 4 upgrade (#902). Reviews are welcome >> > there >> > > too. >> > > >> > > 5. Delta Kernel as the default Delta path (#886, #713) >> > > - #903 adds Delta Kernel source and target coverage to >> > > ITConversionController and is approved. >> > > - #947 (local paths, #943) and #942 (commit tag mapping, #819) fix >> Kernel >> > > gaps found along the way. >> > > >> > > 6. Correctness and robustness fixes. >> > > - #951 falls back to parquet-footer stats when the Hudi metadata table >> > > returns degenerate values. It is approved and ready to merge. >> > > - #621 adds a custom Iceberg metadata cleaner. >> > > - #956 resolves the parquet source file system from the base path, and >> > #957 >> > > keeps absolute paths for data files outside the table base path in >> Delta >> > > targets. >> > > - #938 adds fail-safe handling for Delta deletion vectors (#657). >> > > - #952 bundles Iceberg's GlueCatalog and S3FileIO in xtable-utilities >> > > (#944). >> > > - #814 filters metadata directories out of Hudi partitions (#813). >> > > - Open bugs without a PR yet are #958, #959, #941, #936, #901 and >> #736. >> > > Help on any of these is welcome. >> > > >> > > 7. License and packaging (#701, #717, #865) >> > > - #876 keeps exactly one LICENSE and NOTICE in every published jar, >> and >> > > #895 makes the bundled-jar license metadata correct and checks it in >> CI. >> > > - It would be good to land these in 0.5.0, since they address the >> > footnotes >> > > from the 0.4.0 vote. >> > > >> > > 8. AI agent tooling (#888) >> > > - The issues for an agent-legible CLI, a datasetConfig JSON Schema >> and an >> > > MCP surface are filed (#889, #890, #891), and #829 adds an >> > xtable-converter >> > > skill. These are open for contributors. >> > > >> > > 9. Docs >> > > - #922 addresses the open review comments on the 0.4.0 Spark runtime >> docs >> > > (#920), and #921 covers when Delta Kernel is selected. >> > > >> > > PROPOSED TIMELINE >> > > >> > > I propose Thursday, October 8 as the code cutoff for 0.5.0, with >> anything >> > > not merged by then moving to 0.6.0. Tim, does that work for you? >> > > >> > > Reviews this week would make the biggest difference. In particular, it >> > > would be good to get eyes on #835, #894, #933, #935, #838, #839, #876 >> and >> > > #895. >> > > If there is an item you think must be in 0.5.0 that is not on the >> label, >> > or >> > > if the date does not work for your area, please reply on this thread. >> > > >> > > [1] [DISCUSS] Scope and timeline for Apache XTable (incubating) 0.5.0 >> and >> > > future releases >> > > >> > > Thanks, >> > > Vinish >> > > >> > >> >
