rangareddy commented on code in PR #913: URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3849571398
########## website/docs/how-to-spark-runtime.md: ########## @@ -0,0 +1,220 @@ +--- +sidebar_position: 3 +title: "Run an XTable sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run an XTable sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache +Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source +table's metadata and writes the target format's metadata alongside the data that is already there. + +The jar is meant to be dropped into a Spark job you already run, so you don't need a separate +process or a separate cluster to keep a table interoperable across formats. + +## Getting the jar + +The jar is published to Maven Central as +`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and +Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the +full list of releases. + +## Adding the sync to a Spark job + +This is what the jar is for. Your job already writes a table in one format, and you want that same +table to be interoperable with the others as soon as the write finishes. + +Add the jar to the `spark-submit` you already use, alongside your application jar: Review Comment: Please update the structure so it first covers building the orders-job, and then details how to run it using spark-submit. ########## website/docs/how-to-spark-runtime.md: ########## @@ -0,0 +1,220 @@ +--- +sidebar_position: 3 +title: "Run an XTable sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run an XTable sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache +Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source +table's metadata and writes the target format's metadata alongside the data that is already there. + +The jar is meant to be dropped into a Spark job you already run, so you don't need a separate +process or a separate cluster to keep a table interoperable across formats. + +## Getting the jar + +The jar is published to Maven Central as +`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and +Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the +full list of releases. + +## Adding the sync to a Spark job + +This is what the jar is for. Your job already writes a table in one format, and you want that same +table to be interoperable with the others as soon as the write finishes. + +Add the jar to the `spark-submit` you already use, alongside your application jar: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and +hand it the session's Hadoop configuration: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against these classes, add the same Maven coordinates to your build with `provided` +scope. + +The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling +back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That +makes it safe to call after every write. + +:::note Tables are addressed by path +The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent +of the `RunSync` Iceberg catalog config (`-i`) yet. When a table's data files don't sit directly +under `basePath`, set `dataPath` to wherever they do, because that's the location each target +writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg +requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check +where your table actually writes. +::: + +:::tip Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is +a complete job that syncs both directions and verifies the row counts. +::: + +## Running a sync as its own job + +The jar also ships a `spark-submit` entry point, which is the equivalent of `RunSync` for this +bundle. Pass the jar as the application jar rather than with `--jars`: Review Comment: Could we mention that users need to create the Hudi table using the Quickstart guide before running spark-submit? ########## website/docs/how-to-spark-runtime.md: ########## @@ -0,0 +1,220 @@ +--- +sidebar_position: 3 +title: "Run an XTable sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run an XTable sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache +Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source +table's metadata and writes the target format's metadata alongside the data that is already there. + +The jar is meant to be dropped into a Spark job you already run, so you don't need a separate +process or a separate cluster to keep a table interoperable across formats. + +## Getting the jar + +The jar is published to Maven Central as +`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and +Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the +full list of releases. + +## Adding the sync to a Spark job + +This is what the jar is for. Your job already writes a table in one format, and you want that same +table to be interoperable with the others as soon as the write finishes. + +Add the jar to the `spark-submit` you already use, alongside your application jar: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and +hand it the session's Hadoop configuration: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against these classes, add the same Maven coordinates to your build with `provided` +scope. + +The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling +back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That +makes it safe to call after every write. + +:::note Tables are addressed by path +The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent +of the `RunSync` Iceberg catalog config (`-i`) yet. When a table's data files don't sit directly +under `basePath`, set `dataPath` to wherever they do, because that's the location each target +writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg +requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check +where your table actually writes. +::: + +:::tip Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is +a complete job that syncs both directions and verifies the row counts. +::: + +## Running a sync as its own job + +The jar also ships a `spark-submit` entry point, which is the equivalent of `RunSync` for this +bundle. Pass the jar as the application jar rather than with `--jars`: + +```shell md title="shell" Review Comment: Can we add a section on running XTableSparkSync with and without a configuration file or single-table vs. multi-table syncs? ########## website/releases/downloads.mdx: ########## Review Comment: Can we move 'Downloads' to the top, right after 'Latest Releases'? ########## website/docs/how-to-spark-runtime.md: ########## @@ -0,0 +1,220 @@ +--- +sidebar_position: 3 +title: "Run an XTable sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run an XTable sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache +Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source +table's metadata and writes the target format's metadata alongside the data that is already there. + +The jar is meant to be dropped into a Spark job you already run, so you don't need a separate +process or a separate cluster to keep a table interoperable across formats. + +## Getting the jar + +The jar is published to Maven Central as +`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and +Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the +full list of releases. + +## Adding the sync to a Spark job Review Comment: Move 'Adding the sync to a Spark job' after 'Running a sync as its own job'. Reason: The current order causes confusion for new users because references like orders-job.jar are introduced without context. ########## website/releases/release-0.4.0-incubating.mdx: ########## @@ -0,0 +1,79 @@ +--- +title: "Release 0.4.0-incubating" +sidebar_position: -4 +--- + +## [Release 0.4.0-incubating](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) ([docs](https://xtable.apache.org/docs/how-to)) + +This is the fourth official Apache release for Apache XTable (Incubating), an incubating project under the Apache Software Foundation. +Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files. +Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake, and with this release it can also read Apache Paimon and Apache Parquet tables as sources. + +## ✨ What's Changed + +### 📦 Run an XTable sync on Apache Spark + +This release introduces `xtable-spark-runtime`, a runtime jar that runs a sync on a Spark cluster +you already have. Add it to an existing job with `--jars` and call the sync right after your write, +or submit it on its own with `spark-submit`. Every engine dependency is `provided`, so the jar +reuses the Hudi, Iceberg and Delta libraries already on your cluster. + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +It is published to Maven Central as `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating` +and supports Spark 3.4 and 3.5. See [Run an XTable sync on Apache Spark](/docs/how-to-spark-runtime). + +### 🔄 New source formats + +* **Apache Paimon source**, with incremental sync and column statistics. +* **Apache Parquet source**, with schema conversion, column statistics, partition extraction and incremental sync. + +### 🆕 Delta Kernel Review Comment: Could we include a callout/note here to explicitly state Spark 3.5 support? ########## website/docs/how-to-spark-runtime.md: ########## @@ -0,0 +1,220 @@ +--- +sidebar_position: 3 +title: "Run an XTable sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run an XTable sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache +Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source +table's metadata and writes the target format's metadata alongside the data that is already there. + +The jar is meant to be dropped into a Spark job you already run, so you don't need a separate +process or a separate cluster to keep a table interoperable across formats. + +## Getting the jar + +The jar is published to Maven Central as +`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and +Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the +full list of releases. + +## Adding the sync to a Spark job + +This is what the jar is for. Your job already writes a table in one format, and you want that same +table to be interoperable with the others as soon as the write finishes. + +Add the jar to the `spark-submit` you already use, alongside your application jar: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and +hand it the session's Hadoop configuration: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against these classes, add the same Maven coordinates to your build with `provided` +scope. + +The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling +back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That +makes it safe to call after every write. + +:::note Tables are addressed by path +The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent +of the `RunSync` Iceberg catalog config (`-i`) yet. When a table's data files don't sit directly +under `basePath`, set `dataPath` to wherever they do, because that's the location each target +writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg +requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check +where your table actually writes. +::: + +:::tip Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is +a complete job that syncs both directions and verifies the row counts. +::: + +## Running a sync as its own job + +The jar also ships a `spark-submit` entry point, which is the equivalent of `RunSync` for this +bundle. Pass the jar as the application jar rather than with `--jars`: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +To sync several tables in one submit, use `--datasetconfig`. It takes the same YAML that `RunSync` +uses, so an existing config works unchanged, and unlike `RunSync` the config itself may live on +cloud storage: + +```yaml md title="my_config.yaml" +sourceFormat: HUDI +targetFormats: + - DELTA + - ICEBERG +datasets: + - + tableBasePath: s3://tpc-ds-datasets/1GB/hudi/call_center + tableDataPath: s3://tpc-ds-datasets/1GB/hudi/call_center/data + tableName: call_center + namespace: my.db + - + tableBasePath: s3://tpc-ds-datasets/1GB/hudi/catalog_sales + tableName: catalog_sales + partitionSpec: cs_sold_date_sk:VALUE + - + tableBasePath: s3://hudi/multi-partition-dataset + tableName: multi_partition_dataset + partitionSpec: time_millis:DAY:yyyy-MM-dd,type:VALUE +``` + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --datasetconfig my_config.yaml +``` + +`--datasetconfig` and `--basepath` are mutually exclusive — pass one or the other. + +| Option | Description | +| --- | --- | +| `--basepath` | Base path of the source table. | +| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. | +| `--targets` | Comma-separated target formats, for example `ICEBERG,DELTA`. | +| `--datasetconfig` | Path to a YAML config listing several tables. May be local or on cloud storage. | +| `--datapath` | Path to the data files, when it differs from the base path. | +| `--tablename` | Table name. Defaults to the last segment of the base path. | +| `--namespace` | Dot-separated table namespace. | +| `--partitionspec` | Hudi source partition field spec, for example `level:VALUE`. | +| `--usedeltakernel` | Force Delta Kernel for a Delta source or target. | +| `--help` | Print the usage text. | + +## Supported formats + +Paimon and Parquet are read-only sources; XTable does not write either format as a target. + +| Source ↓ / Target → | Hudi | Iceberg | Delta | +| --- | :---: | :---: | :---: | +| **Hudi** | – | ✅ | ✅ | +| **Iceberg** | ✅ | – | ✅ | +| **Delta** | ✅ | ✅ | – | +| **Paimon** | ✅ | ✅ | ✅ | +| **Parquet** | ✅ | ✅ | ✅ | + +## Spark version support + +Converting to and from Hudi and Iceberg doesn't require Spark at all, so those run on any of the +Spark versions below. Delta is the only engine whose implementation depends on the Spark version, +and the jar chooses the right one automatically: + +| Spark version | Hudi and Iceberg | Delta implementation | +| --- | :---: | --- | +| 3.4.x | ✅ | Delta Standalone | +| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected automatically | Review Comment: When we say 'newer', it implies compatibility with Spark 4.x. It's better to explicitly specify the supported version or remove the 'newer' word. ########## website/releases/downloads.mdx: ########## @@ -7,6 +7,11 @@ Apache XTable™ (incubating) releases are available under the Apache License, V To ensure that you have downloaded the true release, you should [verify](https://www.apache.org/info/verification.html) the integrity of the files using the signatures and checksums available in this page. All releases are signed using the [Apache XTable KEYS file](https://downloads.apache.org/incubator/xtable/KEYS). +### Release 0.4.0-incubating +* Source Release: [Apache XTable 0.4.0-incubating Source Release](https://dlcdn.apache.org/incubator/xtable/0.4.0-incubating/apache-xtable-0.4.0-incubating.src.tgz) ([asc](https://downloads.apache.org/incubator/xtable/0.4.0-incubating/apache-xtable-0.4.0-incubating.src.tgz.asc), [sha512](https://downloads.apache.org/incubator/xtable/0.4.0-incubating/apache-xtable-0.4.0-incubating.src.tgz.sha512)) +* Release Notes: [Release Notes 0.4.0-incubating](release-0.4.0-incubating.mdx) +* Maven Search: [Apache XTable 0.4.0-incubating](https://central.sonatype.com/search?q=g:org.apache.xtable%20v:0.4.0-incubating&smo=true) + ### Release 0.3.0-incubating Review Comment: We need to add the links for each release hyper link. ########## website/docs/how-to-spark-runtime.md: ########## @@ -0,0 +1,220 @@ +--- +sidebar_position: 3 +title: "Run an XTable sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run an XTable sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache +Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source +table's metadata and writes the target format's metadata alongside the data that is already there. + +The jar is meant to be dropped into a Spark job you already run, so you don't need a separate +process or a separate cluster to keep a table interoperable across formats. + +## Getting the jar + +The jar is published to Maven Central as +`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and +Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the +full list of releases. + +## Adding the sync to a Spark job + +This is what the jar is for. Your job already writes a table in one format, and you want that same +table to be interoperable with the others as soon as the write finishes. + +Add the jar to the `spark-submit` you already use, alongside your application jar: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and +hand it the session's Hadoop configuration: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against these classes, add the same Maven coordinates to your build with `provided` +scope. + +The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling +back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That +makes it safe to call after every write. + +:::note Tables are addressed by path +The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent +of the `RunSync` Iceberg catalog config (`-i`) yet. When a table's data files don't sit directly +under `basePath`, set `dataPath` to wherever they do, because that's the location each target +writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg +requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check +where your table actually writes. +::: + +:::tip Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is +a complete job that syncs both directions and verifies the row counts. +::: + +## Running a sync as its own job + +The jar also ships a `spark-submit` entry point, which is the equivalent of `RunSync` for this +bundle. Pass the jar as the application jar rather than with `--jars`: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +To sync several tables in one submit, use `--datasetconfig`. It takes the same YAML that `RunSync` +uses, so an existing config works unchanged, and unlike `RunSync` the config itself may live on +cloud storage: + +```yaml md title="my_config.yaml" +sourceFormat: HUDI +targetFormats: + - DELTA + - ICEBERG +datasets: + - + tableBasePath: s3://tpc-ds-datasets/1GB/hudi/call_center + tableDataPath: s3://tpc-ds-datasets/1GB/hudi/call_center/data + tableName: call_center + namespace: my.db + - + tableBasePath: s3://tpc-ds-datasets/1GB/hudi/catalog_sales + tableName: catalog_sales + partitionSpec: cs_sold_date_sk:VALUE + - + tableBasePath: s3://hudi/multi-partition-dataset + tableName: multi_partition_dataset + partitionSpec: time_millis:DAY:yyyy-MM-dd,type:VALUE +``` + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --datasetconfig my_config.yaml +``` + +`--datasetconfig` and `--basepath` are mutually exclusive — pass one or the other. + +| Option | Description | +| --- | --- | +| `--basepath` | Base path of the source table. | +| `--sourceformat` | Source format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON` or `PARQUET`. | +| `--targets` | Comma-separated target formats, for example `ICEBERG,DELTA`. | +| `--datasetconfig` | Path to a YAML config listing several tables. May be local or on cloud storage. | +| `--datapath` | Path to the data files, when it differs from the base path. | +| `--tablename` | Table name. Defaults to the last segment of the base path. | +| `--namespace` | Dot-separated table namespace. | +| `--partitionspec` | Hudi source partition field spec, for example `level:VALUE`. | +| `--usedeltakernel` | Force Delta Kernel for a Delta source or target. | +| `--help` | Print the usage text. | + +## Supported formats + +Paimon and Parquet are read-only sources; XTable does not write either format as a target. + +| Source ↓ / Target → | Hudi | Iceberg | Delta | +| --- | :---: | :---: | :---: | +| **Hudi** | – | ✅ | ✅ | +| **Iceberg** | ✅ | – | ✅ | +| **Delta** | ✅ | ✅ | – | +| **Paimon** | ✅ | ✅ | ✅ | +| **Parquet** | ✅ | ✅ | ✅ | + +## Spark version support + +Converting to and from Hudi and Iceberg doesn't require Spark at all, so those run on any of the +Spark versions below. Delta is the only engine whose implementation depends on the Spark version, +and the jar chooses the right one automatically: + +| Spark version | Hudi and Iceberg | Delta implementation | +| --- | :---: | --- | +| 3.4.x | ✅ | Delta Standalone | +| 3.5.x and newer | ✅ | [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html), selected automatically | + +Delta Standalone doesn't run on Spark 3.5, so on 3.5 and newer a Delta source or target is routed Review Comment: Can we move/add this section under the Note section for better visibility? ########## website/docs/how-to-spark-runtime.md: ########## @@ -0,0 +1,220 @@ +--- +sidebar_position: 3 +title: "Run an XTable sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run an XTable sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache +Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source +table's metadata and writes the target format's metadata alongside the data that is already there. + +The jar is meant to be dropped into a Spark job you already run, so you don't need a separate +process or a separate cluster to keep a table interoperable across formats. + +## Getting the jar + +The jar is published to Maven Central as +`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and +Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the +full list of releases. + +## Adding the sync to a Spark job + +This is what the jar is for. Your job already writes a table in one format, and you want that same +table to be interoperable with the others as soon as the write finishes. + +Add the jar to the `spark-submit` you already use, alongside your application jar: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and +hand it the session's Hadoop configuration: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against these classes, add the same Maven coordinates to your build with `provided` +scope. + +The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling +back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That +makes it safe to call after every write. + +:::note Tables are addressed by path +The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent +of the `RunSync` Iceberg catalog config (`-i`) yet. When a table's data files don't sit directly +under `basePath`, set `dataPath` to wherever they do, because that's the location each target +writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg +requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check +where your table actually writes. +::: + +:::tip Runnable example +[`demo/spark-runtime`](https://github.com/apache/incubator-xtable/tree/main/demo/spark-runtime) is Review Comment: when i click on link, '404 - page not found' error is dispalyed. ########## website/releases/downloads.mdx: ########## @@ -7,6 +7,11 @@ Apache XTable™ (incubating) releases are available under the Apache License, V To ensure that you have downloaded the true release, you should [verify](https://www.apache.org/info/verification.html) the integrity of the files using the signatures and checksums available in this page. All releases are signed using the [Apache XTable KEYS file](https://downloads.apache.org/incubator/xtable/KEYS). +### Release 0.4.0-incubating Review Comment: We need to add the links for each release hyper link. ########## website/docs/how-to-spark-runtime.md: ########## @@ -0,0 +1,220 @@ +--- +sidebar_position: 3 +title: "Run an XTable sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run an XTable sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache +Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source +table's metadata and writes the target format's metadata alongside the data that is already there. + +The jar is meant to be dropped into a Spark job you already run, so you don't need a separate +process or a separate cluster to keep a table interoperable across formats. + +## Getting the jar + +The jar is published to Maven Central as +`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and +Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the +full list of releases. + +## Adding the sync to a Spark job + +This is what the jar is for. Your job already writes a table in one format, and you want that same +table to be interoperable with the others as soon as the write finishes. + +Add the jar to the `spark-submit` you already use, alongside your application jar: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and +hand it the session's Hadoop configuration: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; Review Comment: In the other quick-start examples, we used local paths, but here an S3 path is used instead. Could we update this to keep it consistent? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
