vinishjail97 commented on code in PR #913: URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3848765966
########## website/docs/setup.md: ########## @@ -23,4 +23,7 @@ For more information on the steps, follow the project's GitHub [README.md](https ## Next Steps See the [Quickstart](/docs/how-to) guide to learn to use Apache XTable™ (Incubating) to add interoperability between -different table formats. \ No newline at end of file +different table formats. + +To run a sync on an Apache Spark cluster you already have, see +[Run a sync on Apache Spark](/docs/spark-runtime). Review Comment: Renamed to `how-to-spark-runtime.md` in 8248b3f6, so it lines up with `how-to` and `how-to-catalog-sync`. Updated the sidebar, this link, the release notes link and the demo README. Also retitled the page to "Run an XTable sync on Apache Spark". ########## website/releases/release-0.4.0-incubating.mdx: ########## @@ -0,0 +1,81 @@ +--- +title: "Release 0.4.0-incubating" +sidebar_position: -4 +--- + +## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to)) + +This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation. +Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files. +Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats. + +## ✨ What's Changed + +### 📦 Run a sync on Apache Spark + +This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster +you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit +it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi, +Iceberg and Delta libraries your cluster already supplies. + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It +supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime). + +### 🔄 New source formats + +* **Apache Paimon source**, with incremental sync and column statistics. +* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync. + +### 🆕 Delta Kernel + +* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs Review Comment: You're right — `DeltaKernelConversionSource` landed in #729, so the release ships both. Fixed in 8248b3f6: the bullet now reads "Added a **Delta Kernel** conversion source and target". ########## website/releases/release-0.4.0-incubating.mdx: ########## @@ -0,0 +1,81 @@ +--- +title: "Release 0.4.0-incubating" +sidebar_position: -4 +--- + +## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to)) + +This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation. +Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files. +Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats. + +## ✨ What's Changed + +### 📦 Run a sync on Apache Spark + +This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster +you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit +it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi, +Iceberg and Delta libraries your cluster already supplies. + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It +supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime). + +### 🔄 New source formats + +* **Apache Paimon source**, with incremental sync and column statistics. +* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync. + +### 🆕 Delta Kernel + +* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs Review Comment: Simplified in 8248b3f6. It's now a single bullet with no `delta-core`: > Added a **Delta Kernel** conversion source and target, which no longer depend on Spark for the conversion itself. Dropped `delta-core` from the Spark version table on the docs page too — it now says Delta Standalone and Delta Kernel. ########## website/releases/release-0.4.0-incubating.mdx: ########## @@ -0,0 +1,81 @@ +--- +title: "Release 0.4.0-incubating" +sidebar_position: -4 +--- + +## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to)) + +This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation. +Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files. +Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats. + +## ✨ What's Changed + +### 📦 Run a sync on Apache Spark + +This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster +you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit +it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi, +Iceberg and Delta libraries your cluster already supplies. + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It +supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime). + +### 🔄 New source formats + +* **Apache Paimon source**, with incremental sync and column statistics. +* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync. + +### 🆕 Delta Kernel + +* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs + on Spark 3.5 and newer, where `delta-core` does not. + +### 🔧 Conversion improvements + +* Delta source: extract statistics for nested fields. +* Hudi source: support nested partition columns when hive-style partitioning is on. +* Fall back to Parquet footers when the metadata table has no column statistics. +* Throw on a bad Delta checkpoint CRC by default. Review Comment: Reworded in 8248b3f6 to "Fail fast on a corrupt Delta checkpoint CRC, instead of continuing with bad statistics." ########## website/releases/release-0.4.0-incubating.mdx: ########## @@ -0,0 +1,81 @@ +--- +title: "Release 0.4.0-incubating" +sidebar_position: -4 +--- + +## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to)) + +This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation. +Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files. +Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats. + +## ✨ What's Changed + +### 📦 Run a sync on Apache Spark + +This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster +you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit +it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi, +Iceberg and Delta libraries your cluster already supplies. + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It +supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime). + +### 🔄 New source formats + +* **Apache Paimon source**, with incremental sync and column statistics. +* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync. + +### 🆕 Delta Kernel + +* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs + on Spark 3.5 and newer, where `delta-core` does not. + +### 🔧 Conversion improvements + +* Delta source: extract statistics for nested fields. +* Hudi source: support nested partition columns when hive-style partitioning is on. +* Fall back to Parquet footers when the metadata table has no column statistics. +* Throw on a bad Delta checkpoint CRC by default. +* Pass additional properties through to a Hudi target. Review Comment: Agreed, removed in 8248b3f6. ########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files +there, and each target writes its metadata next to those files. + +:::note Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) +holds a complete job that runs both directions and checks the row counts. +::: + +## Why `--jars` and not `--packages` + +The runtime jar carries only XTable code plus a few relocated helper libraries. It declares +Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your +cluster already supplies. + +Use `--jars`. It takes the file you downloaded and needs no network access at submit time. + +`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped, +so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on +each submit, which is the only reason to prefer `--jars`. + +:::danger Do not add the engines with `--packages` +Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply +them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or +Parquet there breaks casts across the class loader boundary. Put engine jars on the flat +classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead. +::: + +## Run a sync without a job + +The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate +step. Pass the jar as the application jar, not with `--jars`: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and +`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`. +The config path can be local or on cloud storage. + +```yaml md title="dataset.yaml" +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://example-warehouse/db/store_sales + - tableBasePath: s3://example-warehouse/db/store_returns + - tableBasePath: s3://example-warehouse/db/item +``` + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --datasetconfig dataset.yaml +``` + +`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two. + +| Option | Description | +| --- | --- | +| `--basepath` | Base path of the source table. | +| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. | +| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. | +| `--datasetconfig` | Path to a YAML config that lists many tables. | +| `--datapath` | Path to the data files when it differs from the base path. | +| `--tablename` | Table name. Defaults to the last segment of the base path. | +| `--namespace` | Dot-separated table namespace. | +| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. | +| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. | +| `--help` | Print the usage text. | + +## Supported formats + +Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target. + +| Source ↓ / Target → | Hudi | Iceberg | Delta | +| --- | :---: | :---: | :---: | +| **Hudi** | – | ✅ | ✅ | +| **Iceberg** | ✅ | – | ✅ | +| **Delta** | ✅ | ✅ | – | +| **Paimon** | ✅ | ✅ | ✅ | +| **Parquet** | ✅ | ✅ | ✅ | + +## Spark versions + +Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line +below. Delta is the one engine that depends on the Spark version, and the jar picks the right +implementation for you. + +| Spark version | Hudi and Iceberg | Delta implementation | +| --- | :---: | --- | +| 3.4.x | ✅ | Delta Standalone (`delta-core`) | +| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected for you | + +On Spark 3.5 and newer, `delta-core` does not run. A Delta source or target goes through the +Spark-free Delta Kernel instead, with no flag. To force Kernel on Spark 3.4, pass +`--usedeltakernel`, or set `.useDeltaKernel(true)` on the `TableSyncSpec`. + +## Read a Hudi target + +XTable records the file listing of a Hudi target in the Hudi metadata table. Enable that table +in the reader. Without the option, the sync reports success and the read returns zero rows. + +```scala md title="scala" +spark.read + .option("hoodie.metadata.enable", "true") + .option("hoodie.datasource.read.extract.partition.values.from.path", "true") + .format("hudi") + .load(basePath) +``` + +Iceberg and Delta targets need no extra read option. One exception: the vectorized Arrow reader +of Iceberg mis-casts timestamp columns on a table that another engine wrote. Pass +`vectorization-enabled=false` when you see that. Review Comment: Removed the whole section in 8248b3f6. Next steps now links to [Apache Spark](/docs/spark), which already documents the Hudi read options, so the version-specific detail lives in one place instead of two. Kept the note in `demo/spark-runtime/README.md`, since that demo returns zero rows without it — it now links to the same page. ########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files +there, and each target writes its metadata next to those files. + +:::note Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) +holds a complete job that runs both directions and checks the row counts. +::: + +## Why `--jars` and not `--packages` + +The runtime jar carries only XTable code plus a few relocated helper libraries. It declares +Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your +cluster already supplies. + +Use `--jars`. It takes the file you downloaded and needs no network access at submit time. + +`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped, +so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on +each submit, which is the only reason to prefer `--jars`. + +:::danger Do not add the engines with `--packages` +Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply +them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or +Parquet there breaks casts across the class loader boundary. Put engine jars on the flat +classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead. +::: + +## Run a sync without a job + +The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate +step. Pass the jar as the application jar, not with `--jars`: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and +`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`. +The config path can be local or on cloud storage. + +```yaml md title="dataset.yaml" +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://example-warehouse/db/store_sales + - tableBasePath: s3://example-warehouse/db/store_returns + - tableBasePath: s3://example-warehouse/db/item +``` + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --datasetconfig dataset.yaml +``` + +`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two. + +| Option | Description | +| --- | --- | +| `--basepath` | Base path of the source table. | +| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. | +| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. | +| `--datasetconfig` | Path to a YAML config that lists many tables. | +| `--datapath` | Path to the data files when it differs from the base path. | +| `--tablename` | Table name. Defaults to the last segment of the base path. | +| `--namespace` | Dot-separated table namespace. | +| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. | +| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. | +| `--help` | Print the usage text. | + +## Supported formats + +Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target. + +| Source ↓ / Target → | Hudi | Iceberg | Delta | +| --- | :---: | :---: | :---: | +| **Hudi** | – | ✅ | ✅ | +| **Iceberg** | ✅ | – | ✅ | +| **Delta** | ✅ | ✅ | – | +| **Paimon** | ✅ | ✅ | ✅ | +| **Parquet** | ✅ | ✅ | ✅ | + +## Spark versions + +Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line +below. Delta is the one engine that depends on the Spark version, and the jar picks the right +implementation for you. + +| Spark version | Hudi and Iceberg | Delta implementation | +| --- | :---: | --- | +| 3.4.x | ✅ | Delta Standalone (`delta-core`) | +| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected for you | + +On Spark 3.5 and newer, `delta-core` does not run. A Delta source or target goes through the Review Comment: Fixed in 8248b3f6. The Spark version table now reads Delta Standalone / Delta Kernel, and `delta-core` no longer appears on either page. ########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files +there, and each target writes its metadata next to those files. + +:::note Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) +holds a complete job that runs both directions and checks the row counts. +::: + +## Why `--jars` and not `--packages` + +The runtime jar carries only XTable code plus a few relocated helper libraries. It declares +Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your +cluster already supplies. + +Use `--jars`. It takes the file you downloaded and needs no network access at submit time. + +`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped, +so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on +each submit, which is the only reason to prefer `--jars`. + +:::danger Do not add the engines with `--packages` +Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply +them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or +Parquet there breaks casts across the class loader boundary. Put engine jars on the flat +classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead. +::: + +## Run a sync without a job + +The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate +step. Pass the jar as the application jar, not with `--jars`: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and +`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`. +The config path can be local or on cloud storage. + +```yaml md title="dataset.yaml" +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://example-warehouse/db/store_sales + - tableBasePath: s3://example-warehouse/db/store_returns + - tableBasePath: s3://example-warehouse/db/item +``` + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --datasetconfig dataset.yaml +``` + +`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two. + +| Option | Description | +| --- | --- | +| `--basepath` | Base path of the source table. | +| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. | +| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. | +| `--datasetconfig` | Path to a YAML config that lists many tables. | +| `--datapath` | Path to the data files when it differs from the base path. | +| `--tablename` | Table name. Defaults to the last segment of the base path. | +| `--namespace` | Dot-separated table namespace. | +| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. | +| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. | +| `--help` | Print the usage text. | + +## Supported formats + +Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target. + +| Source ↓ / Target → | Hudi | Iceberg | Delta | +| --- | :---: | :---: | :---: | +| **Hudi** | – | ✅ | ✅ | +| **Iceberg** | ✅ | – | ✅ | +| **Delta** | ✅ | ✅ | – | +| **Paimon** | ✅ | ✅ | ✅ | +| **Parquet** | ✅ | ✅ | ✅ | + +## Spark versions + +Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line +below. Delta is the one engine that depends on the Spark version, and the jar picks the right +implementation for you. Review Comment: Reworded in 8248b3f6 to "Converting to and from Hudi and Iceberg doesn't require Spark at all, so those run on any of the Spark versions below." ########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files +there, and each target writes its metadata next to those files. + +:::note Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) +holds a complete job that runs both directions and checks the row counts. +::: + +## Why `--jars` and not `--packages` Review Comment: Removed in 8248b3f6. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
