vinishjail97 commented on code in PR #913: URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3848657318
########## website/releases/release-0.4.0-incubating.mdx: ########## @@ -0,0 +1,81 @@ +--- +title: "Release 0.4.0-incubating" +sidebar_position: -4 +--- + +## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to)) + +This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation. +Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files. +Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats. + +## ✨ What's Changed + +### 📦 Run a sync on Apache Spark + +This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster +you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit +it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi, +Iceberg and Delta libraries your cluster already supplies. + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It +supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime). + +### 🔄 New source formats + +* **Apache Paimon source**, with incremental sync and column statistics. +* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync. + +### 🆕 Delta Kernel + +* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs Review Comment: DeltaKernelConversionSource is also present? ########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files +there, and each target writes its metadata next to those files. + +:::note Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) +holds a complete job that runs both directions and checks the row counts. +::: + +## Why `--jars` and not `--packages` + +The runtime jar carries only XTable code plus a few relocated helper libraries. It declares +Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your +cluster already supplies. + +Use `--jars`. It takes the file you downloaded and needs no network access at submit time. + +`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped, +so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on +each submit, which is the only reason to prefer `--jars`. + +:::danger Do not add the engines with `--packages` +Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply +them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or +Parquet there breaks casts across the class loader boundary. Put engine jars on the flat +classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead. +::: + +## Run a sync without a job + +The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate +step. Pass the jar as the application jar, not with `--jars`: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and +`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`. +The config path can be local or on cloud storage. + +```yaml md title="dataset.yaml" +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://example-warehouse/db/store_sales + - tableBasePath: s3://example-warehouse/db/store_returns + - tableBasePath: s3://example-warehouse/db/item +``` + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --datasetconfig dataset.yaml +``` + +`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two. + +| Option | Description | +| --- | --- | +| `--basepath` | Base path of the source table. | +| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. | +| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. | +| `--datasetconfig` | Path to a YAML config that lists many tables. | +| `--datapath` | Path to the data files when it differs from the base path. | +| `--tablename` | Table name. Defaults to the last segment of the base path. | +| `--namespace` | Dot-separated table namespace. | +| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. | +| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. | +| `--help` | Print the usage text. | + +## Supported formats + +Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target. + +| Source ↓ / Target → | Hudi | Iceberg | Delta | +| --- | :---: | :---: | :---: | +| **Hudi** | – | ✅ | ✅ | +| **Iceberg** | ✅ | – | ✅ | +| **Delta** | ✅ | ✅ | – | +| **Paimon** | ✅ | ✅ | ✅ | +| **Parquet** | ✅ | ✅ | ✅ | + +## Spark versions + +Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line +below. Delta is the one engine that depends on the Spark version, and the jar picks the right +implementation for you. + +| Spark version | Hudi and Iceberg | Delta implementation | +| --- | :---: | --- | +| 3.4.x | ✅ | Delta Standalone (`delta-core`) | +| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected for you | + +On Spark 3.5 and newer, `delta-core` does not run. A Delta source or target goes through the Review Comment: Avoid using delta-core use delta standalone or delta-kernel. ########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files +there, and each target writes its metadata next to those files. + +:::note Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) +holds a complete job that runs both directions and checks the row counts. +::: + +## Why `--jars` and not `--packages` + +The runtime jar carries only XTable code plus a few relocated helper libraries. It declares +Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your +cluster already supplies. + +Use `--jars`. It takes the file you downloaded and needs no network access at submit time. + +`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped, +so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on +each submit, which is the only reason to prefer `--jars`. + +:::danger Do not add the engines with `--packages` +Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply +them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or +Parquet there breaks casts across the class loader boundary. Put engine jars on the flat +classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead. +::: + +## Run a sync without a job + +The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate +step. Pass the jar as the application jar, not with `--jars`: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and +`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`. +The config path can be local or on cloud storage. + +```yaml md title="dataset.yaml" +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://example-warehouse/db/store_sales + - tableBasePath: s3://example-warehouse/db/store_returns + - tableBasePath: s3://example-warehouse/db/item +``` + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --datasetconfig dataset.yaml +``` + +`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two. + +| Option | Description | +| --- | --- | +| `--basepath` | Base path of the source table. | +| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. | +| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. | +| `--datasetconfig` | Path to a YAML config that lists many tables. | +| `--datapath` | Path to the data files when it differs from the base path. | +| `--tablename` | Table name. Defaults to the last segment of the base path. | +| `--namespace` | Dot-separated table namespace. | +| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. | +| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. | +| `--help` | Print the usage text. | + +## Supported formats + +Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target. + +| Source ↓ / Target → | Hudi | Iceberg | Delta | +| --- | :---: | :---: | :---: | +| **Hudi** | – | ✅ | ✅ | +| **Iceberg** | ✅ | – | ✅ | +| **Delta** | ✅ | ✅ | – | +| **Paimon** | ✅ | ✅ | ✅ | +| **Parquet** | ✅ | ✅ | ✅ | + +## Spark versions + +Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line +below. Delta is the one engine that depends on the Spark version, and the jar picks the right +implementation for you. Review Comment: Spark free core classes is very vauge and meaningless say don't need spark for the conversion? ########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files +there, and each target writes its metadata next to those files. + +:::note Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) +holds a complete job that runs both directions and checks the row counts. +::: + +## Why `--jars` and not `--packages` + +The runtime jar carries only XTable code plus a few relocated helper libraries. It declares +Spark, Hadoop, Hudi, Iceberg, Delta, Avro and Parquet as `provided`, so it uses the copies your +cluster already supplies. + +Use `--jars`. It takes the file you downloaded and needs no network access at submit time. + +`--packages` also works. Every dependency in the published POM is `provided` or `test` scoped, +so Ivy downloads this one jar and no transitive jars. It costs you a Maven Central lookup on +each submit, which is the only reason to prefer `--jars`. + +:::danger Do not add the engines with `--packages` +Add Hudi, Iceberg or Delta with `--packages` only when your cluster does not already supply +them. `--packages` and `--jars` load classes in a child class loader. A second copy of Avro or +Parquet there breaks casts across the class loader boundary. Put engine jars on the flat +classpath with `spark.driver.extraClassPath` and `spark.executor.extraClassPath` instead. +::: + +## Run a sync without a job + +The jar also ships a `spark-submit` entry point. Use it when you want the sync as a separate +step. Pass the jar as the application jar, not with `--jars`: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +To sync many tables in one submit, pass `--datasetconfig` with a YAML file. `sourceFormat` and +`targetFormats` apply to every table. Each entry under `datasets` needs only a `tableBasePath`. +The config path can be local or on cloud storage. + +```yaml md title="dataset.yaml" +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://example-warehouse/db/store_sales + - tableBasePath: s3://example-warehouse/db/store_returns + - tableBasePath: s3://example-warehouse/db/item +``` + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --datasetconfig dataset.yaml +``` + +`--datasetconfig` and `--basepath` are mutually exclusive. Pass one of the two. + +| Option | Description | +| --- | --- | +| `--basepath` | Base path of the source table. | +| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. | +| `--targets` | Comma-separated target formats, such as `ICEBERG,DELTA`. | +| `--datasetconfig` | Path to a YAML config that lists many tables. | +| `--datapath` | Path to the data files when it differs from the base path. | +| `--tablename` | Table name. Defaults to the last segment of the base path. | +| `--namespace` | Dot-separated table namespace. | +| `--partitionspec` | Hudi source partition field spec, such as `level:VALUE`. | +| `--usedeltakernel` | Force the Spark-free Delta Kernel for a Delta source or target. | +| `--help` | Print the usage text. | + +## Supported formats + +Paimon and Parquet are read-only sources. XTable writes no Paimon or Parquet target. + +| Source ↓ / Target → | Hudi | Iceberg | Delta | +| --- | :---: | :---: | :---: | +| **Hudi** | – | ✅ | ✅ | +| **Iceberg** | ✅ | – | ✅ | +| **Delta** | ✅ | ✅ | – | +| **Paimon** | ✅ | ✅ | ✅ | +| **Parquet** | ✅ | ✅ | ✅ | + +## Spark versions + +Hudi and Iceberg conversion use Spark-free core classes only, so they run on every Spark line +below. Delta is the one engine that depends on the Spark version, and the jar picks the right +implementation for you. + +| Spark version | Hudi and Iceberg | Delta implementation | +| --- | :---: | --- | +| 3.4.x | ✅ | Delta Standalone (`delta-core`) | +| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected for you | + +On Spark 3.5 and newer, `delta-core` does not run. A Delta source or target goes through the +Spark-free Delta Kernel instead, with no flag. To force Kernel on Spark 3.4, pass +`--usedeltakernel`, or set `.useDeltaKernel(true)` on the `TableSyncSpec`. + +## Read a Hudi target + +XTable records the file listing of a Hudi target in the Hudi metadata table. Enable that table +in the reader. Without the option, the sync reports success and the read returns zero rows. + +```scala md title="scala" +spark.read + .option("hoodie.metadata.enable", "true") + .option("hoodie.datasource.read.extract.partition.values.from.path", "true") + .format("hudi") + .load(basePath) +``` + +Iceberg and Delta targets need no extra read option. One exception: the vectorized Arrow reader +of Iceberg mis-casts timestamp columns on a table that another engine wrote. Pass +`vectorization-enabled=false` when you see that. Review Comment: Remove this `Read a Hudi target` section and internal details on read completely..these are open gaps depending on the hudi/iceberg versions used not something that needs to be documented in xtable docs. ########## website/docs/setup.md: ########## @@ -23,4 +23,7 @@ For more information on the steps, follow the project's GitHub [README.md](https ## Next Steps See the [Quickstart](/docs/how-to) guide to learn to use Apache XTable™ (Incubating) to add interoperability between -different table formats. \ No newline at end of file +different table formats. + +To run a sync on an Apache Spark cluster you already have, see +[Run a sync on Apache Spark](/docs/spark-runtime). Review Comment: Should this have the same how-to prefix? ########## website/releases/release-0.4.0-incubating.mdx: ########## @@ -0,0 +1,81 @@ +--- +title: "Release 0.4.0-incubating" +sidebar_position: -4 +--- + +## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to)) + +This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation. +Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files. +Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats. + +## ✨ What's Changed + +### 📦 Run a sync on Apache Spark + +This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster +you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit +it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi, +Iceberg and Delta libraries your cluster already supplies. + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It +supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime). + +### 🔄 New source formats + +* **Apache Paimon source**, with incremental sync and column statistics. +* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync. + +### 🆕 Delta Kernel + +* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs + on Spark 3.5 and newer, where `delta-core` does not. + +### 🔧 Conversion improvements + +* Delta source: extract statistics for nested fields. +* Hudi source: support nested partition columns when hive-style partitioning is on. +* Fall back to Parquet footers when the metadata table has no column statistics. +* Throw on a bad Delta checkpoint CRC by default. Review Comment: `Throw on a bad Delta checkpoint CRC by default` Re-word this say fail fast instead? ########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files +there, and each target writes its metadata next to those files. + +:::note Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) +holds a complete job that runs both directions and checks the row counts. +::: + +## Why `--jars` and not `--packages` Review Comment: Remove this section..the user knows how spark works and not something that needs to be added in xtable docs. ########## website/releases/release-0.4.0-incubating.mdx: ########## @@ -0,0 +1,81 @@ +--- +title: "Release 0.4.0-incubating" +sidebar_position: -4 +--- + +## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to)) + +This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation. +Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files. +Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats. + +## ✨ What's Changed + +### 📦 Run a sync on Apache Spark + +This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster +you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit +it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi, +Iceberg and Delta libraries your cluster already supplies. + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It +supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime). + +### 🔄 New source formats + +* **Apache Paimon source**, with incremental sync and column statistics. +* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync. + +### 🆕 Delta Kernel + +* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs + on Spark 3.5 and newer, where `delta-core` does not. + +### 🔧 Conversion improvements + +* Delta source: extract statistics for nested fields. +* Hudi source: support nested partition columns when hive-style partitioning is on. +* Fall back to Parquet footers when the metadata table has no column statistics. +* Throw on a bad Delta checkpoint CRC by default. +* Pass additional properties through to a Hudi target. Review Comment: `Pass additional properties through to a Hudi target` This is not worth something that needs to be documented on the site. ########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files Review Comment: For iceberg document the catalog path instead? No one uses the absolute paths for iceberg writes. ########## website/releases/release-0.4.0-incubating.mdx: ########## @@ -0,0 +1,81 @@ +--- +title: "Release 0.4.0-incubating" +sidebar_position: -4 +--- + +## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to)) + +This is the fourth official Apache release for Apache XTable™ (Incubating), an incubating project under the Apache Software Foundation. +Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files. +Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake. This release adds Apache Paimon and Apache Parquet as source formats. + +## ✨ What's Changed + +### 📦 Run a sync on Apache Spark + +This release adds `xtable-spark-runtime`, a runtime jar that runs a sync on an Apache Spark cluster +you already have. Add it to a Spark job with `--jars` and call the sync after the write, or submit +it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar uses the Hudi, +Iceberg and Delta libraries your cluster already supplies. + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +The jar is on Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. It +supports Spark 3.4 and Spark 3.5. See [Run a sync on Apache Spark](/docs/spark-runtime). + +### 🔄 New source formats + +* **Apache Paimon source**, with incremental sync and column statistics. +* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync. + +### 🆕 Delta Kernel + +* Added a **Delta Kernel** conversion target. Delta Kernel needs no Spark, so a Delta sync now runs Review Comment: `Delta Kernel needs no Spark, so a Delta sync now runs on Spark 3.5 and newer, where `delta-core` does not` Simplify this it's repetitive, and delta-core the user isn't aware of the internal details like delta-core is the standalone library..? ########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files Review Comment: Using --catalogConfig? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
