I agree with keeping Spark 4 out of 0.5.0. I don't think there is bandwidth to get it all in the next release.
-Tim On Mon, Aug 17, 2026 at 7:10 PM Vinish <[email protected]> wrote: > Hi all, > > One more item I would like the list to weigh in on: the Spark 4 upgrade. Do > we want it in 0.5.0, or should it move to 0.6.0? > > Where we are today. main builds against Spark 3.4.2 and Scala 2.12, with a > Java 8 compiler target. Delta is pinned at 2.4.0 and Iceberg at 1.9.2. We > never moved the default to Spark 3.5 either. PR #671 was the last attempt > at that and I closed it in April as stale. > > What the upgrade actually involves. Spark 4.0 only ships for Scala 2.13 and > needs Java 17, so this is not a one-line version bump. It means: > - flipping the default build to Scala 2.13. Our scala-2.13 profile is still > marked "once Hudi supports Scala 2.13" and still passes -target:jvm-1.8. > - raising the compiler target off Java 8. > - moving Delta from 2.4.0 to 4.0.0 and Iceberg from 1.9.2 to 1.10.0 or > later, since those are the first releases with Spark 4.0 artifacts. > > The upstream artifacts all exist already: hudi-spark4.0-bundle_2.13 1.2.0, > iceberg-spark-runtime-4.0_2.13 1.10.0+, delta-spark_2.13 4.0.0, and > paimon-spark-4.0 1.3.1. Nothing external blocks us. > > My own read is that it does not fit 0.5.0. We already have six items > against a 30 September cutoff, and the Delta 2.4 to 4.0 jump lands directly > on top of "Delta Kernel as the default Delta path". Doing both in one > release makes it hard to tell which change broke what. I would rather land > Kernel-as-default in 0.5.0, and then make Spark 4 a headline item for 0.6.0 > with the 3.5 step folded > into it. But I am happy to be argued out of this if someone wants to pick > it up early. > > So the question for the community is just this one: Spark 4 in 0.5.0, or > 0.6.0? If there is agreement on the direction I will file a tracking issue > for it. > > Thanks, > Vinish > > On Mon, Aug 17, 2026 03:58 PM, Vinish <[email protected]> > wrote: > > > Thanks Tim, and thanks for taking it on. I have noted you as the release > > manager for 0.5.0-incubating. > > > > Thanks, > > Vinish > > > > On Thu, Aug 13, 2026 04:58 AM, Tim Brown <[email protected]> wrote: > > > >> Thanks for kicking off the discussion Vinish. I can be the release > manager > >> for 0.5.0. > >> > >> -Tim > >> > >> On Mon, Aug 10, 2026 at 9:01 PM Vinish <[email protected]> > >> wrote: > >> > >> > Hi all, > >> > > >> > I have gone through the project's open issues and the earlier > [DISCUSS] > >> > threads on this list. Based on those, the following six seem like the > >> > natural scope for 0.5.0-incubating, each with a tracking issue > >> > 1. Hudi table version 9 - > >> > https://github.com/apache/incubator-xtable/issues/834 > >> > 2. Iceberg pluggable table format - > >> > https://github.com/apache/incubator-xtable/issues/722 > >> > 3. Delta Kernel as the default Delta path - > >> > https://github.com/apache/incubator-xtable/issues/886 > >> > 4. xtable-spark-runtime for Hudi 1.x - > >> > https://github.com/apache/incubator-xtable/issues/836 > >> > 5. Cross-format indexing support - > >> > https://github.com/apache/incubator-xtable/issues/887 > >> > 6. AI skills and tooling - > >> > https://github.com/apache/incubator-xtable/issues/888 > >> > > >> > What else should we pick up? If you are working on something you would > >> like > >> > in 0.5.0-incubating, please reply here. > >> > > >> > Timeline. I think we should time-box these releases and fit the scope > to > >> > the date. If the community agrees, the cutoffs would be: > >> > - 0.5.0: 30 September 2026 > >> > - 0.6.0: 15 November 2026 > >> > - 0.7.0: 5 January 2027 > >> > > >> > Realistically, some of these may not be ready by the first cutoff. If > >> that > >> > happens, my preference would be to hold the date and let the scope > >> move, so > >> > anything not quite ready is picked up in the next release a few weeks > >> > later. That said, these dates are only a starting point, and we can > >> adjust > >> > them if they do not work for the community. > >> > > >> > Release manager. Stamatis started a thread on rotating the roll, and > >> Vinoth > >> > kindly offered to take 0.6.0 or 0.7.0, which leaves 0.5.0 open. Any > >> > volunteers for 0.5.0 release manager? > >> > > >> > Thanks, > >> > Vinish > >> > > >> > > >
