vinishjail97 opened a new issue, #902:
URL: https://github.com/apache/incubator-xtable/issues/902
## Background
Discussed on the dev list in [DISCUSS] Scope and timeline for Apache XTable
(incubating) 0.5.0 and future releases. The list agreed to keep this **out of
0.5.0** and make it a headline item for **0.6.0**, with the skipped Spark 3.5
step folded in. Two reasons were given: 0.5.0 already carries six items against
a 30 September 2026 cutoff, and the Delta 2.4 to 4.0 jump lands directly on top
of "Delta Kernel as the default Delta path" (#886), which would make it hard to
attribute a regression to either change.
Filing this to track the work.
## Where we are today
`main` is still on the Spark 3.4 / Scala 2.12 / Java 8 combination:
| Property | Value | Location |
|---|---|---|
| `spark.version` | 3.4.2 | `pom.xml:88` |
| `spark.version.prefix` | 3.4 | `pom.xml:89` |
| `scala.version` | `${scala12.version}` = 2.12.20 | `pom.xml:86` |
| `scala.binary.version` | 2.12 | `pom.xml:87` |
| `javaVersion` | 8, fed to `<release>` | `pom.xml:64`, `pom.xml:712` |
| `delta.version` | 2.4.0 | `pom.xml:92` |
| `iceberg.version` | 1.9.2 | `pom.xml:90` |
| `hudi.version` | 1.2.0 | `pom.xml:73` |
| `paimon.version` | 1.3.1 | `pom.xml:93` |
| CI `JDK_VERSION` | 11 | `.github/workflows/mvn-ci-build.yml:36` |
The default was never moved to Spark 3.5 either. #671 was the last attempt
and it was closed as stale in April 2026.
The `scala-2.13` profile at `pom.xml:1017` exists but is not usable as
written:
- It is still gated by the comment `<!-- Once hudi supports scala 2.13 then
enable following profile -->` (`pom.xml:1016`), which is now stale.
`hudi-utilities_2.13` and `hudi-spark4.0-bundle_2.13` both publish at 1.2.0.
- Its only body is a `pluginManagement` block configuring
`scala-maven-plugin` with `-target:jvm-1.8` (`pom.xml:1038`). That
configuration is inert: the repository contains **zero `.scala` sources**, and
no module declares `scala-maven-plugin` under `<build><plugins>`, so the plugin
never executes. The `-target:jvm-1.8` argument is therefore not a real blocker,
only dead configuration to remove.
The practical effect is that the profile only swaps `scala.binary.version`,
which is all that is actually needed, since every Scala-suffixed dependency is
templated on that property.
## Upstream artifacts
All required Spark 4.0 artifacts are published. Nothing external blocks this
work. Verified on Maven Central:
| Artifact | Versions available |
|---|---|
| `org.apache.hudi:hudi-spark4.0-bundle_2.13` | 1.1.0, 1.1.1, 1.2.0 |
| `org.apache.hudi:hudi-utilities_2.13` | up to 1.2.0 |
| `org.apache.iceberg:iceberg-spark-runtime-4.0_2.13` | 1.10.0, 1.10.1,
1.10.2, 1.11.0 |
| `io.delta:delta-spark_2.13` | 4.0.0 |
| `org.apache.paimon:paimon-spark-4.0` | 1.3.0, 1.3.1, 1.3.2 |
| `io.delta:delta-standalone_2.13` | 3.3.0 and later |
| `io.delta:delta-hive_2.13` | 3.0.0 and later |
The last two matter because `xtable-hive-metastore` pins
`delta.standalone.version` 3.3.0 and `delta.hive.version` 3.0.0
(`pom.xml:98-99`). Both cross-build for 2.13, so that module does not block the
Scala flip.
One convenient detail: Hudi, Iceberg and Paimon all use `4.0` as their Spark
qualifier, matching the existing artifactId templates
`hudi-spark${spark.version.prefix}-bundle_${scala.binary.version}`,
`iceberg-spark-runtime-${spark.version.prefix}_${scala.binary.version}` and
`paimon-spark-${spark.version.prefix}`. Setting `spark.version.prefix` to `4.0`
covers all three with no artifactId edits.
## Build work
1. **Flip the default build to Scala 2.13.** Spark 4.0 only publishes for
2.13. Make `scala-2.13` the `activeByDefault` profile, drop the stale Hudi
comment, and delete the inert `scala-maven-plugin` block.
2. **Raise the compiler target off Java 8.** Spark 4.0 requires Java 17. Set
`javaVersion` to 17, and raise `JDK_VERSION` in `mvn-ci-build.yml` and
`mvn-license-check.yml`, plus the hardcoded `java-version: '11'` in
`package-deploy.yml:31` and `source-build-check.yml:50`. The release profile
also hardcodes `<source>1.8</source>` for the javadoc plugin at `pom.xml:1120`.
The `quarkus.platform.version` pin at 3.2.12.Final carries the comment `<!--
compatible with Java 11 -->` and should be re-evaluated for `xtable-service`.
3. **Rename the Delta Spark artifact.** Delta renamed `delta-core` to
`delta-spark` in 3.0, so `io.delta:delta-core_${scala.binary.version}`
(`pom.xml:331`) becomes `delta-spark_${scala.binary.version}` at 4.0.0. Four
consumers reference the old coordinate: `xtable-core`, `xtable-service`,
`xtable-hudi-support-extensions`, and `xtable-utilities`, where it also appears
as a shade `<include>`.
4. **Bump Iceberg** from 1.9.2 to 1.10.0 or later, the first release with
Spark 4.0 artifacts.
5. **Decide the Spark 3.5 step.** The list preference was to fold 3.5 into
this upgrade rather than land it separately. A single intermediate commit
setting the prefix to `3.5` may still be useful to isolate Delta 2.4 to 3.x
breakage from the Scala and Java flip.
## Code work
This is not a version-bump-only change. Two areas need real edits.
**Delta internal Scala APIs.** The Delta reader and writer call Delta's
internal Scala classes directly from Java, not the public `io.delta.tables`
API. Across 18 files (9 in `src/main`, 9 in tests) the imports include
`org.apache.spark.sql.delta.DeltaLog`, `Snapshot`, `OptimisticTransaction`,
`DeltaHistoryManager`, `DeltaConfigs`, `DeltaOperations`, `GeneratedColumn`,
and `actions.{AddFile, RemoveFile, Metadata, Action, CommitInfo, Format,
DeletionVectorDescriptor}`. These carry no compatibility guarantee and change
between Delta 2.4 and 4.0. Expect this to be the bulk of the work. Main-source
files affected:
- `DeltaActionsConverter`, `DeltaConversionSource`, `DeltaConversionTarget`,
`DeltaConversionUtils`, `DeltaDataFileExtractor`,
`DeltaDataFileUpdatesExtractor`, `DeltaIncrementalChangesState`,
`DeltaStatsExtractor`, `DeltaTableExtractor`
This is a second, code-level argument for landing #886 first. The Kernel
path uses the stable `io.delta.kernel` API instead, and `delta.kernel.version`
is already at 4.0.0 (`pom.xml:60`). The more of the Delta path that runs
through Kernel before this upgrade starts, the smaller the internal-API surface
left to port.
**Scala 2.12 to 2.13 interop from Java.** 15 Java files import `scala.*`
directly, centered on `ScalaUtils.java` and the Delta and Kernel packages. The
two risk points are `scala.collection.JavaConverters` (11 usages), deprecated
in 2.13 in favour of `scala.jdk.javaapi.CollectionConverters`, and
`scala.collection.Seq` (6 usages), whose relationship to `scala.Seq` changed in
2.13.
## Scope
- [ ] Make `scala-2.13` the default profile; remove the stale Hudi comment
and the inert `scala-maven-plugin` configuration.
- [ ] Set `javaVersion` to 17; update all four workflows and the javadoc
`<source>`; re-evaluate the Quarkus pin.
- [ ] Set `spark.version` to 4.0.x and `spark.version.prefix` to `4.0`.
- [ ] Move `delta-core_` to `delta-spark_` and `delta.version` to 4.0.0, in
all four consuming modules including the shade includes.
- [ ] Bump `iceberg.version` to 1.10.0 or later.
- [ ] Port the Delta internal-API call sites across the 18 affected files.
- [ ] Fix the Scala 2.13 interop in the 15 files importing `scala.*`.
- [ ] Confirm `xtable-hive-metastore` builds on 2.13 with the current
`delta-standalone` and `delta-hive` pins.
- [ ] Run the full integration suite, `ITConversionController` in
particular, on Spark 4.0.
- [ ] Decide whether to keep an intermediate Spark 3.5 commit.
## Open questions
- Do we keep a Spark 3.4 or 3.5 profile for users who cannot move to Java
17, or is the default the only supported combination? A cross-build matrix in
CI would roughly double build time.
- Does the `xtable-spark-runtime` work in #836 need to land first, so the
bundle is built once against the new baseline rather than twice?
## Related
- #886, Delta Kernel as the default Delta path. Strongly preferred to land
before this.
- #836, `xtable-spark-runtime`.
- #671, the closed Spark 3.5 attempt.
- #896, shrink the `xtable-utilities` bundled jar, which touches the same
shade configuration.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
