vinishjail97 opened a new pull request, #843: URL: https://github.com/apache/incubator-xtable/pull/843
Draft for review — ports `xtable-spark-runtime` onto the **Hudi 0.x** line (`main-hudi-0x`, Hudi 0.14 / Spark 3.4). A thin, relocated Spark bundle that runs an incremental XTable sync in-job; engines are provided by the cluster, never bundled. ### Why (context) Per the [[DISCUSS] rc1 for 0.4.0 thread](https://lists.apache.org/thread/dd020pzf1lkdjpko83rz0hx231m6yrzk), **0.4.0 is cut from the Hudi 0.x line** and `xtable-spark-runtime` is a headline on-ramp (a better path than the standalone utilities jar). This is the Hudi-0.x counterpart of the packaging PR #838 (which targets `main` / Hudi 1.x). ### Module - `xtable-spark-runtime_${scala.binary.version}` — `xtable-core` compile; Spark/Hadoop + engines `provided`. Curated shade allowlist (xtable modules + guava/protobuf/commons-cli relocated); avro/parquet/jackson **not** relocated (exchanged with the engines). Thin **~3.7 MB** bundle. - `XTableSparkSync` — standalone `spark-submit` entry point (Apache Commons CLI). - `XTableSyncService` / `TableSyncSpec` — build an `INCREMENTAL` `ConversionConfig` and run `ConversionController.sync`; the target metadata path is the source **data** path (required by Hudi; Iceberg data lives under `<basePath>/data`). - `ConversionTargetFactory` — resilient ServiceLoader discovery so a subset of engines works when others are absent (warn + skip on `LinkageError`/`ServiceConfigurationError`; name-based Delta-Kernel check). ### Hudi 0.14 specifics (vs the Hudi 1.x variant on `main`) - No `hudi-hadoop-common` (that split is 1.x); `hudi-common` still provides `FSUtils`. - The engine classpath uses the **`hudi-spark` bundle** — its regenerated Avro model classes (e.g. `DecimalWrapper`) link on **Avro 1.12** (required by Iceberg 1.9.2), whereas raw `hudi-common`'s are Avro-1.8/1.9-generated and reference `TimeConversions.TimeConversion`, removed in Avro ≥1.10. Plus `hudi-java-client` for the Hudi target's Java write client, with raw `hudi-common` excluded so the bundle's clean class wins. ### Testing `ITXTableSparkRuntimeBundle` `spark-submit`s the shaded jar for one case per direction across **Hudi, Iceberg and Delta** (source and target), engines on a flat `spark.driver.extraClassPath`, asserting data-equivalence over comparable scalar columns (local-timestamp columns excluded — HUDI-7088). Verified locally green (4/4) on a Spark 3.4.3 `SPARK_HOME`. The IT is `SPARK_HOME`-gated, so CI skips it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
