This is an automated email from the ASF dual-hosted git repository. vinishjail97 pushed a commit to branch spark-runtime-docs in repository https://gitbox.apache.org/repos/asf/incubator-xtable.git
commit 06e1175c318eb30885fcd949c79d5c668beb2329 Author: Vinish Reddy <[email protected]> AuthorDate: Tue Jul 21 12:08:29 2026 -0700 [docs] xtable-spark-runtime: call it a runtime jar, tighten prose Rename "bundle jar" to "runtime jar" throughout to match the module name and Iceberg's convention; keep "(shaded, relocated)" once in the developer README. Drop em-dashes from prose. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --- website/docs/spark-runtime.md | 30 +++++++++++++++--------------- xtable-spark-runtime/README.md | 20 ++++++++++---------- 2 files changed, 25 insertions(+), 25 deletions(-) diff --git a/website/docs/spark-runtime.md b/website/docs/spark-runtime.md index 759504a4..a6cbff5d 100644 --- a/website/docs/spark-runtime.md +++ b/website/docs/spark-runtime.md @@ -4,7 +4,7 @@ title: "Run XTable on Apache Spark" # Running Apache XTable™ (Incubating) on Apache Spark -The `xtable-spark-runtime` module publishes a self-contained (shaded) bundle jar that runs an +The `xtable-spark-runtime` module publishes a self-contained runtime jar that runs an XTable metadata sync with `spark-submit` on an existing Apache Spark cluster. It is the `spark-submit` equivalent of the `RunSync` utility: no data is rewritten, only the target table format metadata is generated alongside the existing data files. @@ -12,7 +12,7 @@ format metadata is generated alongside the existing data files. Use this when you already run Spark (EMR, Dataproc, HDInsight, Databricks, or a local install) and want to add interoperability without standing up a separate process. -## Build the bundle +## Build the runtime jar From the project root: @@ -20,7 +20,7 @@ From the project root: ./mvnw clean package -pl xtable-spark-runtime -am -DskipTests ``` -This produces the shaded bundle at +This produces the runtime jar at `xtable-spark-runtime/target/xtable-spark-runtime_2.12-<version>.jar`. ## Quick start @@ -60,17 +60,17 @@ $SPARK_HOME/bin/spark-submit \ --partitionspec level:VALUE ``` -The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided` — the bundle expects them -on the Spark runtime classpath, which is the case on a standard Spark install with the relevant +The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided`. The runtime jar expects +them on the Spark runtime classpath, which is the case on a standard Spark install with the relevant format support. ## Adding to an existing Spark job -A common setup is a Spark job that already writes a table in one format — for example a job that -writes Hudi — where the same table should also be readable as Iceberg or Delta. Because the bundle's -engine libraries are `provided`, it reuses the Hudi, Iceberg, and Delta libraries already present on -the Spark runtime that job uses; the only additional artifact is this jar. Add it to that Spark -runtime's classpath and run the sync — no separate installation is required. +A common setup is a Spark job that already writes a table in one format (for example a job that +writes Hudi) where the same table should also be readable as Iceberg or Delta. Because the runtime +jar's engine libraries are `provided`, it reuses the Hudi, Iceberg, and Delta libraries already +present on the Spark runtime that job uses; the only additional artifact is this jar. Add it to that +Spark runtime's classpath and run the sync, with no separate installation required. Run `XTableSparkSync` on the same Spark cluster, pointed at the table the job writes, as a follow-on step after the write completes. Set `--sourceformat` to the format the job writes and list the @@ -150,7 +150,7 @@ Paimon and Parquet are read-only sources (there is no corresponding write target ## Spark version compatibility Hudi and Iceberg conversion use only Spark-free core classes, so they run on any of the Spark lines -below. Delta is the only Spark-version-sensitive engine, and the bundle picks the right +below. Delta is the only Spark-version-sensitive engine, and the runtime jar picks the right implementation automatically: | Spark version | Hudi / Iceberg | Delta implementation | @@ -158,10 +158,10 @@ implementation automatically: | 3.4.x | ✅ | Delta Standalone (`delta-core`) | | 3.5.x and newer | ✅ | Delta Kernel (Spark-free), selected automatically | -On Spark 3.5+, the bundled `delta-core` does not run, so a Delta source or target is routed through -the Spark-free [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html) implementation -automatically — no flag needed. To force Kernel on any Spark version (e.g. Spark 3.4), pass -`--usedeltakernel`. +On Spark 3.5+, the `delta-core` in the runtime jar does not run, so a Delta source or target is +routed through the Spark-free [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html) +implementation automatically, with no flag needed. To force Kernel on any Spark version (e.g. Spark +3.4), pass `--usedeltakernel`. ## Next steps diff --git a/xtable-spark-runtime/README.md b/xtable-spark-runtime/README.md index d8c9a21e..267d5bda 100644 --- a/xtable-spark-runtime/README.md +++ b/xtable-spark-runtime/README.md @@ -19,13 +19,13 @@ # XTable Spark Runtime -`xtable-spark-runtime` publishes a self-contained (shaded, relocated) bundle jar that runs an +`xtable-spark-runtime` publishes a self-contained (shaded, relocated) runtime jar that runs an Apache XTable™ metadata sync with `spark-submit` on an existing Apache Spark cluster. It is the -`spark-submit` equivalent of the `RunSync` utility — no data is rewritten, only the target table +`spark-submit` equivalent of the `RunSync` utility: no data is rewritten, only the target table format metadata is generated alongside the existing data files. The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided`: the user brings their -own engine versions from the Spark runtime, so the thin bundle stays compatible across versions. +own engine versions from the Spark runtime, so the thin runtime jar stays compatible across versions. ## Build @@ -35,7 +35,7 @@ From the project root: ./mvnw clean package -pl xtable-spark-runtime -am -DskipTests ``` -The shaded bundle is written to +The runtime jar is written to `xtable-spark-runtime/target/xtable-spark-runtime_2.12-<version>.jar`. ## Usage @@ -68,8 +68,8 @@ Paimon and Parquet are read-only sources; targets are Hudi, Iceberg, and Delta. ## Adding to an existing Spark job If a Spark job already writes a table in one format (for example Hudi) and the same table should -also be readable as Iceberg or Delta, the bundle's `provided` engine libraries let it reuse the -Hudi, Iceberg, and Delta libraries already on that job's Spark runtime — the only additional +also be readable as Iceberg or Delta, the runtime jar's `provided` engine libraries let it reuse the +Hudi, Iceberg, and Delta libraries already on that job's Spark runtime; the only additional artifact is this jar. Add it to that Spark runtime's classpath and run `XTableSparkSync` against the table as a follow-on step after the write completes, with `--sourceformat` set to the format the job writes and the formats to add listed in `--targets`: @@ -116,19 +116,19 @@ A complete example covering the 24 tables of the TPC-DS schema is at ## Spark version compatibility Hudi and Iceberg conversion use only Spark-free core classes and run on any Spark line below. Delta -is the only Spark-version-sensitive engine, and the bundle selects the implementation automatically: +is the only Spark-version-sensitive engine, and the runtime jar selects the implementation automatically: | Spark version | Hudi / Iceberg | Delta | | --- | :---: | --- | | 3.4.x | ✅ | Delta Standalone (`delta-core`) | | 3.5.x and newer | ✅ | Delta Kernel (Spark-free), selected automatically | -On Spark 3.5+ the bundled `delta-core` does not run, so a Delta source/target is routed through the +On Spark 3.5+ the `delta-core` in the runtime jar does not run, so a Delta source/target is routed through the Spark-free Delta Kernel automatically. Pass `--usedeltakernel` to force it on any Spark version. -## Bundle validation IT +## Runtime jar validation IT -`ITXTableSparkRuntimeBundle` `spark-submit`s the shaded jar against a real Spark distribution, one +`ITXTableSparkRuntimeBundle` `spark-submit`s the runtime jar against a real Spark distribution, one case per direction across Hudi, Iceberg, and Delta, and asserts the target is data-equivalent to the source. It self-skips unless `SPARK_HOME` is set, so it does not run in the normal build:
