This is an automated email from the ASF dual-hosted git repository.
vinishjail97 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-xtable.git
The following commit(s) were added to refs/heads/main by this push:
new c2055989 [855] Add the 0.4.0 release notes and a Spark runtime
quickstart with a runnable demo (#913)
c2055989 is described below
commit c2055989950308c62dbde27d2d40ae068555bceb
Author: Vinish Reddy <[email protected]>
AuthorDate: Tue Aug 25 20:50:09 2026 +0530
[855] Add the 0.4.0 release notes and a Spark runtime quickstart with a
runnable demo (#913)
* [docs] Add the 0.4.0-incubating release notes and downloads entry
Adds website/releases/release-0.4.0-incubating.mdx and links it from the
Downloads page, alongside the source release, its signature and checksum on
dlcdn.
The page leads with xtable-spark-runtime, the one new artifact the release
publishes to Maven Central, and groups the rest into new source formats
(Paimon, Parquet), the Delta Kernel target, conversion fixes, and build
work.
It also lists the three open license documentation issues (#700, #701, #865)
that were footnoted during the release vote.
* [docs] Add a "Run a sync on Apache Spark" quickstart
xtable-spark-runtime_2.12 0.4.0-incubating is on Maven Central, so the guide
points at the released jar instead of a source build. The module itself is
not
on main, so nothing here depends on it being in the tree.
The page leads with the case the jar exists for: a Spark job that already
writes a table adds the jar with --jars and calls XTableSyncService after
its
write. The standalone spark-submit entry point and the multi-table
--datasetconfig follow, then the CLI options, the format matrix and the
Spark
version table.
It also explains why --jars is the recommendation. Every dependency in the
published pom is provided or test scoped, so --packages resolves the one jar
and no transitives; --jars simply avoids a Maven Central lookup on each
submit. The real hazard is adding the engines with --packages, which loads a
second Avro or Parquet into a child class loader and breaks casts across the
boundary, so engine jars belong on extraClassPath.
Linked from the docs sidebar under Quick Start and from the Installation
page.
* [docs] Add a runnable in-job sync demo for xtable-spark-runtime
demo/spark-runtime runs the case the guide describes, and checks it. A Spark
job writes a table, calls XTableSyncService from inside the same job, and
then
reads the table back in the other formats and compares row counts. Both
directions run: Hudi to Iceberg/Delta, and Iceberg to Hudi/Delta.
The demo needs no Docker and no Scala compiler. spark-shell -i compiles the
script, so a reader runs one command. The jar comes from Maven Central with
curl and goes on --jars, exactly as it would on a job they already run.
A local Spark distribution ships no engines, so the README resolves them
from
the downloaded pom with dependency:build-classpath and puts them on
extraClassPath, matching how a cluster supplies them and how
ITXTableSparkRuntimeBundle runs. Spark 3.4.x is required because Hudi 0.14.0
publishes no Spark 3.5 bundle for the datasource that direction 1 writes
with;
the runtime jar itself supports 3.4 and 3.5.
Verified end to end against Spark 3.4.2 with the published 0.4.0-incubating
jar. All four checks pass.
* [docs] Address review feedback on the 0.4.0 docs
Renames the guide to how-to-spark-runtime so it matches how-to and
how-to-catalog-sync, and retitles it "Run an XTable sync on Apache Spark".
Drops two sections that documented Spark itself rather than XTable: the
--jars vs --packages comparison, and "Read a Hudi target". The Hudi metadata
read option is already covered on the Apache Spark page, which Next steps
now
links to.
Replaces the invented dataset YAML with the config from the project README,
and says outright that it is the same file RunSync takes. Adds a short note
that the runtime jar addresses tables by path and has no equivalent of the
RunSync Iceberg catalog config, which is why an Iceberg source needs
dataPath.
Release notes: the Delta Kernel entry now covers the source as well as the
target, since DeltaKernelConversionSource landed in #729; the checkpoint CRC
bullet says fail fast; the Hudi additional-properties bullet is gone.
Neither
page mentions delta-core any more, and the Spark table names Delta
Standalone
and Delta Kernel instead.
Prose across all files follows the Hudi and Iceberg docs voice rather than
clipped one-clause sentences, and consistently says a table is interoperable
rather than readable or queryable.
The demo banners changed with that wording, so the demo was re-run end to
end
against Spark 3.4.2 with the published 0.4.0-incubating jar. All four checks
still pass, and the expected output in the README is from that run.
* [docs] Don't claim Iceberg data files live under <basePath>/data
<basePath>/data is what the Hadoop catalog happens to produce, not something
Iceberg guarantees; write.data.path and object-storage layouts can place
data
files anywhere. Both the guide and the demo comment stated it as a rule.
The guide now frames dataPath by what it is for: set it when a table's data
files do not sit directly under basePath, because that is where each target
writes its metadata. The demo comment scopes its <basePath>/data claim to
the
catalog the demo configures.
No behaviour change; dataPath was always a free-form path.
---
demo/README.md | 3 +
demo/spark-runtime/README.md | 160 +++++++++++++++++++
demo/spark-runtime/XTableInJobSyncDemo.scala | 163 +++++++++++++++++++
website/docs/how-to-spark-runtime.md | 220 ++++++++++++++++++++++++++
website/docs/setup.md | 5 +-
website/releases/downloads.mdx | 5 +
website/releases/release-0.4.0-incubating.mdx | 79 +++++++++
website/sidebars.js | 3 +-
8 files changed, 636 insertions(+), 2 deletions(-)
diff --git a/demo/README.md b/demo/README.md
index 777aa381..e018203d 100644
--- a/demo/README.md
+++ b/demo/README.md
@@ -26,6 +26,9 @@ When you are done, use `./stop_demo.sh` to stop and remove
the containers. Pass
## Notebooks
- `work/demo.ipynb` — the main demo: two existing datasets (Hudi and Delta)
are synced across Hudi, Delta and Iceberg, updated from Trino, joined, and
validated from Trino.
+## Standalone Spark demo
+- `spark-runtime/` — keeps a table interoperable by adding the
`xtable-spark-runtime` jar with `--jars`. No Docker required; see
`spark-runtime/README.md`.
+
## Accessing Services
### Jupyter Notebook
To access the notebook, look for a log line during startup that contains `To
access the server, open this file in a browser: ... Or copy and paste one of
these URLs: ...` and use the `http://127.0.0.1:8888/...` url to open the
notebook in your browser. The demo is located at `work/demo.ipynb`.
diff --git a/demo/spark-runtime/README.md b/demo/spark-runtime/README.md
new file mode 100644
index 00000000..883223c3
--- /dev/null
+++ b/demo/spark-runtime/README.md
@@ -0,0 +1,160 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one
+ - or more contributor license agreements. See the NOTICE file
+ - distributed with this work for additional information
+ - regarding copyright ownership. The ASF licenses this file
+ - to you under the Apache License, Version 2.0 (the
+ - "License"); you may not use this file except in compliance
+ - with the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+-->
+
+# In-job sync demo for `xtable-spark-runtime`
+
+This demo shows what the `xtable-spark-runtime` jar is for: a Spark job that
already writes a table
+adds the jar with `--jars` and calls `XTableSyncService` right after its
write, making the table
+interoperable without rewriting any data.
+
+It runs the sync in both directions and verifies the row counts, so it also
serves as a smoke test
+for a release.
+
+| Direction | The job writes | XTable adds |
+| --- | --- | --- |
+| 1 | `HUDI` | `ICEBERG`, `DELTA` |
+| 2 | `ICEBERG` | `HUDI`, `DELTA` |
+
+No Docker and no Scala compiler are needed — `spark-shell` compiles the script
for you. This is
+separate from the notebook demo in the parent directory.
+
+## Prerequisites
+
+- A Spark **3.4.x** distribution. Hudi 0.14.0 publishes no Spark 3.5 bundle,
so the Hudi Spark
+ datasource that direction 1 writes with needs a 3.4 runtime. The runtime jar
itself supports both
+ Spark 3.4 and 3.5.
+- JDK 11.
+- Maven, to resolve the engine jars in step 2.
+
+Set `SPARK_HOME` and work in a scratch directory:
+
+```shell
+export SPARK_HOME=/path/to/spark-3.4.x-bin-hadoop3
+mkdir -p /tmp/xtable-spark-demo && cd /tmp/xtable-spark-demo
+```
+
+## 1. Get the runtime jar from Maven Central
+
+```shell
+BASE=https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating
+curl -O $BASE/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+curl -O $BASE/xtable-spark-runtime_2.12-0.4.0-incubating.pom
+```
+
+The jar is about 4 MB. It holds XTable code only, because it declares Spark,
Hadoop, Hudi, Iceberg
+and Delta as `provided`.
+
+## 2. Build the engine classpath
+
+A real cluster already supplies the engines. A local Spark distribution does
not, so resolve them
+from the POM you just downloaded:
+
+```shell
+mvn -q -f xtable-spark-runtime_2.12-0.4.0-incubating.pom
dependency:build-classpath \
+ -DincludeScope=provided -Dmdep.outputFile=engine-classpath-raw.txt
+```
+
+Keep the engine jars only. Spark and Hadoop come from the distribution:
+
+```shell
+tr ':' '\n' < engine-classpath-raw.txt \
+ | grep -E '/(hudi-|iceberg-|delta-|avro-|parquet-|jol-core)' \
+ | grep -v 'avro-mapred\|avro-ipc' \
+ | paste -sd: - > engine-classpath.txt
+```
+
+Keep every `parquet-*` jar, `parquet-format-structures` included. If you drop
it, the older copy in
+the distribution wins and the read fails with `NoSuchFieldError:
size_statistics`.
+
+Direction 2 writes an Iceberg table from Spark, so add the Iceberg Spark
runtime as well:
+
+```shell
+curl -O
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.4_2.12/1.9.2/iceberg-spark-runtime-3.4_2.12-1.9.2.jar
+echo "$PWD/iceberg-spark-runtime-3.4_2.12-1.9.2.jar:$(cat
engine-classpath.txt)" > engine-classpath.txt
+```
+
+## 3. Run
+
+The engines go on the flat classpath with `extraClassPath`, the way a cluster
supplies them. The
+runtime jar goes on `--jars`, the way you add it to a job you already run.
+
+```shell
+export SPARK_LOCAL_IP=127.0.0.1
+export XT_DEMO_DIR=$PWD/out
+
+$SPARK_HOME/bin/spark-shell \
+ --master 'local[4]' \
+ --driver-memory 4g \
+ --conf spark.driver.extraClassPath="$(cat engine-classpath.txt)" \
+ --conf spark.executor.extraClassPath="$(cat engine-classpath.txt)" \
+ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf
spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
\
+ --conf spark.sql.catalog.xt=org.apache.iceberg.spark.SparkCatalog \
+ --conf spark.sql.catalog.xt.type=hadoop \
+ --conf spark.sql.catalog.xt.warehouse=$XT_DEMO_DIR/iceberg_warehouse \
+ --jars $PWD/xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+ -i /path/to/incubator-xtable/demo/spark-runtime/XTableInJobSyncDemo.scala
+```
+
+Set `SPARK_HOME` explicitly. `spark-submit` and `spark-shell` read it from the
environment, so
+calling the script by path is not enough when another Spark is already
exported. On a Spark 3.5
+runtime, the Hudi write in direction 1 fails with
+`ClassNotFoundException: org.apache.spark.sql.adapter.Spark2Adapter`.
+
+## Expected output
+
+```
+==============================================================================
+ DIRECTION 1: the job writes HUDI -> interoperable with ICEBERG and DELTA
+==============================================================================
+ wrote the Hudi table at /tmp/xtable-spark-demo/out/hudi_orders
+ XTableSyncService returned [ICEBERG, DELTA]
+ [PASS] hudi -> iceberg 1000 rows
+ [PASS] hudi -> delta 1000 rows
+
+==============================================================================
+ DIRECTION 2: the job writes ICEBERG -> interoperable with HUDI and DELTA
+==============================================================================
+ wrote the Iceberg table at
/tmp/xtable-spark-demo/out/iceberg_warehouse/db/orders
+ XTableSyncService returned [HUDI, DELTA]
+ [PASS] iceberg -> hudi 1000 rows
+ [PASS] iceberg -> delta 1000 rows
+
+==============================================================================
+ RESULT
+==============================================================================
+ ALL CHECKS PASSED
+```
+
+Set `XT_DEMO_ROWS` to change the row count. The default is 1000.
+
+## Notes
+
+- **A Hudi target needs `hoodie.metadata.enable=true` on read.** XTable
records a Hudi target's
+ file listing in the Hudi metadata table, so without this option the sync
reports success and the
+ read comes back empty. See [Apache
Spark](https://xtable.apache.org/docs/spark) for the read
+ options each format needs.
+- For paths on S3, add S3A and credentials:
+
+ ```shell
+ --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
+ --conf
spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.EnvironmentVariableCredentialsProvider
+ ```
+
+See [Run an XTable sync on Apache
Spark](https://xtable.apache.org/docs/how-to-spark-runtime) for the full guide.
diff --git a/demo/spark-runtime/XTableInJobSyncDemo.scala
b/demo/spark-runtime/XTableInJobSyncDemo.scala
new file mode 100644
index 00000000..525dcf79
--- /dev/null
+++ b/demo/spark-runtime/XTableInJobSyncDemo.scala
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Keeps a table interoperable from inside the Spark job that writes it.
+ * The job adds the xtable-spark-runtime jar with --jars and calls
XTableSyncService after
+ * the write. No data is rewritten.
+ *
+ * The script runs both directions and checks the row counts:
+ * 1. the job writes HUDI -> XTable adds ICEBERG and DELTA
+ * 2. the job writes ICEBERG -> XTable adds HUDI and DELTA
+ *
+ * See demo/spark-runtime/README.md for how to run it, and
+ * https://xtable.apache.org/docs/how-to-spark-runtime for the guide.
+ *
+ * Environment:
+ * XT_DEMO_DIR output directory (default /tmp/xtable-demo)
+ * XT_DEMO_ROWS number of sample rows (default 1000)
+ */
+
+import java.util.{Arrays => JArrays}
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.spark.sql.{DataFrame, SaveMode}
+import org.apache.spark.sql.functions.{col, concat, lit}
+
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val outPrefix = sys.env.getOrElse("XT_DEMO_DIR",
"/tmp/xtable-demo").stripSuffix("/")
+val rows = sys.env.getOrElse("XT_DEMO_ROWS", "1000").toInt
+val hadoopConf: Configuration = spark.sparkContext.hadoopConfiguration
+var failures = List.empty[String]
+
+def banner(s: String): Unit = {
+ println()
+ println("=" * 78)
+ println(s" $s")
+ println("=" * 78)
+}
+
+/**
+ * Read options a cross-engine target needs. XTable records the file listing
of a Hudi target
+ * in the Hudi metadata table, so the reader must enable that table or it sees
no rows. The
+ * vectorized Arrow reader of Iceberg mis-casts timestamps on a table another
engine wrote.
+ */
+def readOptions(format: String): Map[String, String] = format.toUpperCase
match {
+ case "HUDI" =>
+ Map(
+ "hoodie.metadata.enable" -> "true",
+ "hoodie.datasource.read.extract.partition.values.from.path" -> "true")
+ case "ICEBERG" => Map("vectorization-enabled" -> "false")
+ case _ => Map.empty
+}
+
+def check(label: String, expected: Long, format: String, path: String): Unit =
{
+ val actual =
+ try
Right(spark.read.options(readOptions(format)).format(format).load(path).count())
+ catch { case t: Throwable => Left(s"${t.getClass.getSimpleName}:
${t.getMessage}") }
+ actual match {
+ case Right(n) if n == expected =>
+ println(f" [PASS] $label%-22s $n rows")
+ case Right(n) =>
+ val m = s"$label: expected $expected rows, read $n"
+ println(s" [FAIL] $m")
+ failures ::= m
+ case Left(e) =>
+ val m = s"$label: read failed: $e"
+ println(s" [FAIL] $m")
+ failures ::= m
+ }
+}
+
+banner("Sample data")
+val orders: DataFrame = spark
+ .range(rows)
+ .toDF("order_id")
+ .withColumn("customer", concat(lit("cust-"), col("order_id") % 50))
+ .withColumn("amount", col("order_id") * 3 + 7)
+ .withColumn("ts", lit(System.currentTimeMillis()))
+println(s" $rows rows, ${orders.schema.fields.length} columns")
+
+banner("DIRECTION 1: the job writes HUDI -> interoperable with ICEBERG and
DELTA")
+val hudiPath = s"$outPrefix/hudi_orders"
+
+// ---- the write the job already does
------------------------------------------------------
+orders.write
+ .format("hudi")
+ .option("hoodie.table.name", "orders")
+ .option("hoodie.datasource.write.recordkey.field", "order_id")
+ .option("hoodie.datasource.write.precombine.field", "ts")
+ .option("hoodie.datasource.write.operation", "bulk_insert")
+ .option("hoodie.datasource.write.table.type", "COPY_ON_WRITE")
+ .option("hoodie.metadata.enable", "false")
+ .mode(SaveMode.Overwrite)
+ .save(hudiPath)
+println(s" wrote the Hudi table at $hudiPath")
+
+// ---- the one call the job adds
-----------------------------------------------------------
+val hudiResults = new XTableSyncService().sync(
+ TableSyncSpec
+ .builder()
+ .key("orders")
+ .basePath(hudiPath)
+ .sourceFormat("HUDI")
+ .targets(JArrays.asList("ICEBERG", "DELTA"))
+ .build(),
+ hadoopConf)
+println(s" XTableSyncService returned ${hudiResults.keySet()}")
+
+check("hudi -> iceberg", rows, "iceberg", hudiPath)
+check("hudi -> delta", rows, "delta", hudiPath)
+
+banner("DIRECTION 2: the job writes ICEBERG -> interoperable with HUDI and
DELTA")
+val icebergBase = s"$outPrefix/iceberg_warehouse/db/orders"
+
+// ---- the write the job already does
------------------------------------------------------
+spark.sql("CREATE NAMESPACE IF NOT EXISTS xt.db")
+spark.sql("DROP TABLE IF EXISTS xt.db.orders")
+orders.writeTo("xt.db.orders").create()
+println(s" wrote the Iceberg table at $icebergBase")
+
+// ---- the one call the job adds
-----------------------------------------------------------
+// This demo's Iceberg table lands its data under <basePath>/data, the layout
the Hadoop catalog
+// uses here. Each target writes its metadata next to the data files, so
dataPath is set to match.
+// Iceberg does not mandate that layout, so check write.data.path for your own
tables.
+val icebergResults = new XTableSyncService().sync(
+ TableSyncSpec
+ .builder()
+ .key("orders")
+ .basePath(icebergBase)
+ .dataPath(s"$icebergBase/data")
+ .sourceFormat("ICEBERG")
+ .targets(JArrays.asList("HUDI", "DELTA"))
+ .build(),
+ hadoopConf)
+println(s" XTableSyncService returned ${icebergResults.keySet()}")
+
+check("iceberg -> hudi", rows, "hudi", s"$icebergBase/data")
+check("iceberg -> delta", rows, "delta", s"$icebergBase/data")
+
+banner("RESULT")
+if (failures.isEmpty) {
+ println(" ALL CHECKS PASSED")
+} else {
+ println(s" ${failures.size} CHECK(S) FAILED")
+ failures.reverse.foreach(f => println(s" - $f"))
+}
+System.exit(if (failures.isEmpty) 0 else 1)
diff --git a/website/docs/how-to-spark-runtime.md
b/website/docs/how-to-spark-runtime.md
new file mode 100644
index 00000000..bafa2caf
--- /dev/null
+++ b/website/docs/how-to-spark-runtime.md
@@ -0,0 +1,220 @@
+---
+sidebar_position: 3
+title: "Run an XTable sync on Apache Spark"
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Run an XTable sync on Apache Spark
+
+`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™
(Incubating) sync on an Apache
+Spark cluster. As with any XTable sync, no data files are rewritten — the sync
reads the source
+table's metadata and writes the target format's metadata alongside the data
that is already there.
+
+The jar is meant to be dropped into a Spark job you already run, so you don't
need a separate
+process or a separate cluster to keep a table interoperable across formats.
+
+## Getting the jar
+
+The jar is published to Maven Central as
+`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it
once:
+
+```shell md title="shell"
+curl -O
https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar
+```
+
+Every engine dependency is `provided`, so the jar is about 4 MB and reuses the
Hudi, Iceberg and
+Delta libraries your cluster already has. See the
[Downloads](/releases/downloads) page for the
+full list of releases.
+
+## Adding the sync to a Spark job
+
+This is what the jar is for. Your job already writes a table in one format,
and you want that same
+table to be interoperable with the others as soon as the write finishes.
+
+Add the jar to the `spark-submit` you already use, alongside your application
jar:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+ --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+ --class com.example.OrdersJob \
+ orders-job.jar
+```
+
+Then call `XTableSyncService` after your write. You describe the table with a
`TableSyncSpec` and
+hand it the session's Hadoop configuration:
+
+<Tabs
+groupId="language"
+defaultValue="scala"
+values={[
+{ label: 'Scala', value: 'scala', },
+{ label: 'Java', value: 'java', },
+]}
+>
+<TabItem value="scala">
+
+```scala md title="OrdersJob.scala"
+import java.util.{Arrays => JArrays}
+import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}
+
+val basePath = "s3://example-warehouse/db/orders"
+
+// the write your job already does
+df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)
+
+// the one call you add
+new XTableSyncService().sync(
+ TableSyncSpec.builder()
+ .key("orders")
+ .basePath(basePath)
+ .sourceFormat("HUDI")
+ .targets(JArrays.asList("ICEBERG", "DELTA"))
+ .build(),
+ spark.sparkContext.hadoopConfiguration)
+```
+
+</TabItem>
+<TabItem value="java">
+
+```java md title="OrdersJob.java"
+import java.util.Arrays;
+import org.apache.xtable.spark.TableSyncSpec;
+import org.apache.xtable.spark.XTableSyncService;
+
+String basePath = "s3://example-warehouse/db/orders";
+
+// the write your job already does
+df.write().format("hudi").options(hudiOptions).mode("append").save(basePath);
+
+// the one call you add
+new XTableSyncService()
+ .sync(
+ TableSyncSpec.builder()
+ .key("orders")
+ .basePath(basePath)
+ .sourceFormat("HUDI")
+ .targets(Arrays.asList("ICEBERG", "DELTA"))
+ .build(),
+ spark.sparkContext().hadoopConfiguration());
+```
+
+</TabItem>
+</Tabs>
+
+To compile against these classes, add the same Maven coordinates to your build
with `provided`
+scope.
+
+The sync runs incrementally and tracks its own watermark in the target's sync
metadata, falling
+back to a full snapshot whenever an incremental sync isn't safe — the first
run, for example. That
+makes it safe to call after every write.
+
+:::note Tables are addressed by path
+The runtime jar identifies tables by path rather than through a catalog, so
there is no equivalent
+of the `RunSync` Iceberg catalog config (`-i`) yet. When a table's data files
don't sit directly
+under `basePath`, set `dataPath` to wherever they do, because that's the
location each target
+writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but
nothing in Iceberg
+requires it — `write.data.path` and object-storage layouts can put data files
anywhere — so check
+where your table actually writes.
+:::
+
+:::tip Runnable example
+[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime)
is
+a complete job that syncs both directions and verifies the row counts.
+:::
+
+## Running a sync as its own job
+
+The jar also ships a `spark-submit` entry point, which is the equivalent of
`RunSync` for this
+bundle. Pass the jar as the application jar rather than with `--jars`:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+ --class org.apache.xtable.spark.XTableSparkSync \
+ --master 'local[*]' \
+ xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+ --basepath /path/to/hudi_table \
+ --sourceformat HUDI \
+ --targets ICEBERG,DELTA
+```
+
+To sync several tables in one submit, use `--datasetconfig`. It takes the same
YAML that `RunSync`
+uses, so an existing config works unchanged, and unlike `RunSync` the config
itself may live on
+cloud storage:
+
+```yaml md title="my_config.yaml"
+sourceFormat: HUDI
+targetFormats:
+ - DELTA
+ - ICEBERG
+datasets:
+ -
+ tableBasePath: s3://tpc-ds-datasets/1GB/hudi/call_center
+ tableDataPath: s3://tpc-ds-datasets/1GB/hudi/call_center/data
+ tableName: call_center
+ namespace: my.db
+ -
+ tableBasePath: s3://tpc-ds-datasets/1GB/hudi/catalog_sales
+ tableName: catalog_sales
+ partitionSpec: cs_sold_date_sk:VALUE
+ -
+ tableBasePath: s3://hudi/multi-partition-dataset
+ tableName: multi_partition_dataset
+ partitionSpec: time_millis:DAY:yyyy-MM-dd,type:VALUE
+```
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+ --class org.apache.xtable.spark.XTableSparkSync \
+ xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+ --datasetconfig my_config.yaml
+```
+
+`--datasetconfig` and `--basepath` are mutually exclusive — pass one or the
other.
+
+| Option | Description |
+| --- | --- |
+| `--basepath` | Base path of the source table. |
+| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or
`PARQUET`. |
+| `--targets` | Comma-separated target formats, for example `ICEBERG,DELTA`. |
+| `--datasetconfig` | Path to a YAML config listing several tables. May be
local or on cloud storage. |
+| `--datapath` | Path to the data files, when it differs from the base path. |
+| `--tablename` | Table name. Defaults to the last segment of the base path. |
+| `--namespace` | Dot-separated table namespace. |
+| `--partitionspec` | Hudi source partition field spec, for example
`level:VALUE`. |
+| `--usedeltakernel` | Force Delta Kernel for a Delta source or target. |
+| `--help` | Print the usage text. |
+
+## Supported formats
+
+Paimon and Parquet are read-only sources; XTable does not write either format
as a target.
+
+| Source ↓ / Target → | Hudi | Iceberg | Delta |
+| --- | :---: | :---: | :---: |
+| **Hudi** | – | ✅ | ✅ |
+| **Iceberg** | ✅ | – | ✅ |
+| **Delta** | ✅ | ✅ | – |
+| **Paimon** | ✅ | ✅ | ✅ |
+| **Parquet** | ✅ | ✅ | ✅ |
+
+## Spark version support
+
+Converting to and from Hudi and Iceberg doesn't require Spark at all, so those
run on any of the
+Spark versions below. Delta is the only engine whose implementation depends on
the Spark version,
+and the jar chooses the right one automatically:
+
+| Spark version | Hudi and Iceberg | Delta implementation |
+| --- | :---: | --- |
+| 3.4.x | ✅ | Delta Standalone |
+| 3.5.x and newer | ✅ | [Delta
Kernel](https://docs.delta.io/latest/delta-kernel.html), selected automatically
|
+
+Delta Standalone doesn't run on Spark 3.5, so on 3.5 and newer a Delta source
or target is routed
+through Delta Kernel with no flag needed. If you want Kernel on Spark 3.4 as
well, pass
+`--usedeltakernel`, or set `.useDeltaKernel(true)` on the `TableSyncSpec`.
+
+## Next steps
+
+- See the [Quickstart](/docs/how-to) for an end-to-end interoperability
walkthrough.
+- See [Apache Spark](/docs/spark) for the options each format needs when you
query a synced table.
+- See [Installation](/docs/setup) if you'd rather build the project from
source.
diff --git a/website/docs/setup.md b/website/docs/setup.md
index d7b38da6..b0e2b763 100644
--- a/website/docs/setup.md
+++ b/website/docs/setup.md
@@ -23,4 +23,7 @@ For more information on the steps, follow the project's
GitHub [README.md](https
## Next Steps
See the [Quickstart](/docs/how-to) guide to learn to use Apache XTable™
(Incubating) to add interoperability between
-different table formats.
\ No newline at end of file
+different table formats.
+
+To run a sync on a Spark cluster you already have, see
+[Run an XTable sync on Apache Spark](/docs/how-to-spark-runtime).
\ No newline at end of file
diff --git a/website/releases/downloads.mdx b/website/releases/downloads.mdx
index f8179789..3d2bf16f 100644
--- a/website/releases/downloads.mdx
+++ b/website/releases/downloads.mdx
@@ -7,6 +7,11 @@ Apache XTable™ (incubating) releases are available under the
Apache License, V
To ensure that you have downloaded the true release, you should
[verify](https://www.apache.org/info/verification.html) the integrity of the
files using the signatures and checksums available in this page. All releases
are signed using the [Apache XTable KEYS
file](https://downloads.apache.org/incubator/xtable/KEYS).
+### Release 0.4.0-incubating
+* Source Release: [Apache XTable 0.4.0-incubating Source
Release](https://dlcdn.apache.org/incubator/xtable/0.4.0-incubating/apache-xtable-0.4.0-incubating.src.tgz)
([asc](https://downloads.apache.org/incubator/xtable/0.4.0-incubating/apache-xtable-0.4.0-incubating.src.tgz.asc),
[sha512](https://downloads.apache.org/incubator/xtable/0.4.0-incubating/apache-xtable-0.4.0-incubating.src.tgz.sha512))
+* Release Notes: [Release Notes 0.4.0-incubating](release-0.4.0-incubating.mdx)
+* Maven Search: [Apache XTable
0.4.0-incubating](https://central.sonatype.com/search?q=g:org.apache.xtable%20v:0.4.0-incubating&smo=true)
+
### Release 0.3.0-incubating
* Source Release: [Apache XTable 0.3.0-incubating Source
Release](https://dlcdn.apache.org/incubator/xtable/0.3.0-incubating/apache-xtable-0.3.0-incubating.src.tgz)
([asc](https://downloads.apache.org/incubator/xtable/0.3.0-incubating/apache-xtable-0.3.0-incubating.src.tgz.asc),
[sha512](https://downloads.apache.org/incubator/xtable/0.3.0-incubating/apache-xtable-0.3.0-incubating.src.tgz.sha512))
* Release Notes: [Release Notes 0.3.0-incubating](release-0.3.0-incubating.mdx)
diff --git a/website/releases/release-0.4.0-incubating.mdx
b/website/releases/release-0.4.0-incubating.mdx
new file mode 100644
index 00000000..7e3d36ef
--- /dev/null
+++ b/website/releases/release-0.4.0-incubating.mdx
@@ -0,0 +1,79 @@
+---
+title: "Release 0.4.0-incubating"
+sidebar_position: -4
+---
+
+## [Release
0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating)
([docs](https://xtable.apache.org/docs/how-to))
+
+This is the fourth official Apache release for Apache XTable (Incubating), an
incubating project under the Apache Software Foundation.
+Apache XTable™ (Incubating) facilitates omni-directional interoperability
across data processing systems and query engines by allowing users to convert
between open table formats without the need to rewrite any data files.
+Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and
Delta Lake, and with this release it can also read Apache Paimon and Apache
Parquet tables as sources.
+
+## ✨ What's Changed
+
+### 📦 Run an XTable sync on Apache Spark
+
+This release introduces `xtable-spark-runtime`, a runtime jar that runs a sync
on a Spark cluster
+you already have. Add it to an existing job with `--jars` and call the sync
right after your write,
+or submit it on its own with `spark-submit`. Every engine dependency is
`provided`, so the jar
+reuses the Hudi, Iceberg and Delta libraries already on your cluster.
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+ --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+ --class com.example.OrdersJob \
+ orders-job.jar
+```
+
+It is published to Maven Central as
`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`
+and supports Spark 3.4 and 3.5. See [Run an XTable sync on Apache
Spark](/docs/how-to-spark-runtime).
+
+### 🔄 New source formats
+
+* **Apache Paimon source**, with incremental sync and column statistics.
+* **Apache Parquet source**, with schema conversion, column statistics,
partition extraction and incremental sync.
+
+### 🆕 Delta Kernel
+
+* Added a **Delta Kernel** conversion source and target, which no longer
depend on Spark for the conversion itself.
+
+### 🔧 Conversion improvements
+
+* Delta source: extract statistics for nested fields.
+* Hudi source: support nested partition columns when hive-style partitioning
is enabled.
+* Fall back to Parquet footers when the metadata table has no column
statistics.
+* Fail fast on a corrupt Delta checkpoint CRC, instead of continuing with bad
statistics.
+
+### 🐞 Bug fixes
+
+* **Delta**: null partition value for composite generated-column partitions,
map-key path handling, a null pointer on binary fields inside maps and arrays,
and conversion from a source with column mapping enabled.
+* **Hudi**: batch `INSERT_OVERWRITE` replace commits that dropped adds, and an
empty `EarliestCommitToRetain`.
+* **Iceberg**: nested comments with a qualified name, schema sync, and an
empty source table.
+* **Parquet**: snapshot sync across several commits on a partitioned table,
and schema conversion.
+
+### 🧰 Build and infrastructure
+
+* Upgraded **Apache Iceberg** to 1.9.2 and **Apache Avro** to 1.12.0.
+* Added a **Maven wrapper** so every build uses the same Maven version.
+* Bundled license files are now generated and verified as part of the build.
+* Added a **source build check** that builds the source release with no XTable
artifact installed.
+* Restored the delombok step, so release builds produce javadoc jars again.
+
+### 🧪 Testing
+
+* Added a first integration test for the XTable REST service.
+* Added an integration test that runs the Spark runtime jar through a real
`spark-submit`.
+* Parameterised `TestHudiInstantUtils` and added more cases.
+* Removed wildcard imports and enforced the rule with Spotless.
+
+### Known issues
+
+Three license documentation issues remain open and are targeted for the next
release:
+
+* Incomplete license information for the bundled jars
([#700](https://github.com/apache/incubator-xtable/issues/700)).
+* More than one LICENSE file in some jars
([#701](https://github.com/apache/incubator-xtable/issues/701)).
+* Orphaned license files in the bundled jars
([#865](https://github.com/apache/incubator-xtable/issues/865)).
+
+## 📥 Download
+
+See the [Downloads](/releases/downloads) page for the source release, its
signature and its checksum.
diff --git a/website/sidebars.js b/website/sidebars.js
index 9f0b5594..79cb498d 100644
--- a/website/sidebars.js
+++ b/website/sidebars.js
@@ -19,7 +19,8 @@ module.exports = {
collapsed: false,
items: [
'how-to',
- 'how-to-catalog-sync'
+ 'how-to-catalog-sync',
+ 'how-to-spark-runtime'
],
},
{
