vinishjail97 commented on code in PR #913: URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3848767930
########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files Review Comment: Checked `XTableSparkSync` on the 0.4.0 tag and there's no catalog support at all — the options are `--basepath`, `--datapath`, `--sourceformat`, `--targets`, `--tablename`, `--namespace`, `--partitionspec`, `--datasetconfig`, `--usedeltakernel`. So the runtime jar is path-based only today and I can't document catalog usage that doesn't exist yet. Added a short note in 8248b3f6 instead of burying it: > **Tables are addressed by path** — The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent of the `RunSync` Iceberg catalog config (`-i`) yet. For an Iceberg source, point `basePath` at the table root and set `dataPath` to `<basePath>/data`. Happy to open a follow-up issue for catalog support in the runtime jar if you want it tracked. ########## website/docs/spark-runtime.md: ########## @@ -0,0 +1,237 @@ +--- +sidebar_position: 3 +title: "Run a sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run a sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an +Apache Spark cluster. The sync reads the source table metadata and writes the target format +metadata next to the data files that already exist. It rewrites no data. + +Add the jar to a Spark job you already run. You need no separate service and no separate cluster. + +## Get the jar + +The jar is on Maven Central under `org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. +Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +The [Downloads](/releases/downloads) page lists every release. + +## Add the sync to a Spark job + +This is the main use case. Your job already writes a table in one format. You want the same +table to be readable in the other formats when the write completes. + +Pass the jar to your existing `spark-submit` with `--jars`, next to the application jar you +already submit: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Your job then calls `XTableSyncService` after the write. Build one `TableSyncSpec` for the +table and pass the Hadoop configuration of the Spark session: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against the classes, add the same coordinates to your build with `provided` scope. + +`sync` runs in incremental mode and keeps its watermark in the target sync metadata. It falls +back to a full snapshot when an incremental sync is not safe, such as the first run. A call +after every write is therefore safe to repeat. + +For an Iceberg source, also set `dataPath` to `<basePath>/data`. Iceberg keeps the data files Review Comment: There's no `--catalogConfig` on `XTableSparkSync` — only `RunSync` has the Iceberg catalog config, as `-i`, and it's a separate file from the dataset YAML. Worth noting the dataset config itself *is* shared. I diffed `RunSync.DatasetConfig` against `XTableSparkSync.DatasetConfig` and the field sets are identical (`sourceFormat`, `targetFormats`, and `datasets[]` with `tableBasePath`, `tableDataPath`, `tableName`, `partitionSpec`, `namespace`), which matches the javadoc: *"mirrors the RunSync dataset YAML so configs are shareable."* So in 8248b3f6 I dropped my invented example and used the config from the project README verbatim, and the page now says it's the same file `RunSync` takes. One bonus: `XTableSparkSync` reads that path through the Spark Hadoop config, so the config can live on S3/GCS/ABFS, whereas `RunSync` uses `Files.newInputStream` and needs it local. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
