vinishjail97 commented on code in PR #913: URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3848815264
########## website/docs/how-to-spark-runtime.md: ########## @@ -0,0 +1,218 @@ +--- +sidebar_position: 3 +title: "Run an XTable sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run an XTable sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache +Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source +table's metadata and writes the target format's metadata alongside the data that is already there. + +The jar is meant to be dropped into a Spark job you already run, so you don't need a separate +process or a separate cluster to keep a table interoperable across formats. + +## Getting the jar + +The jar is published to Maven Central as +`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and +Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the +full list of releases. + +## Adding the sync to a Spark job + +This is what the jar is for. Your job already writes a table in one format, and you want that same +table to be interoperable with the others as soon as the write finishes. + +Add the jar to the `spark-submit` you already use, alongside your application jar: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and +hand it the session's Hadoop configuration: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against these classes, add the same Maven coordinates to your build with `provided` +scope. + +The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling +back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That +makes it safe to call after every write. + +:::note Tables are addressed by path +The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent +of the `RunSync` Iceberg catalog config (`-i`) yet. For an Iceberg source, point `basePath` at the Review Comment: Good catch — that was wrong. `<basePath>/data` is just what the Hadoop catalog happens to produce; `write.data.path` and object-storage layouts can put data files anywhere, and Iceberg guarantees nothing about the layout. Fixed in db557cdf. The note now frames `dataPath` by what it's actually for rather than by an Iceberg convention: > When a table's data files don't sit directly under `basePath`, set `dataPath` to wherever they do, because that's the location each target writes its metadata to. Iceberg tables often end up at `<basePath>/data`, but nothing in Iceberg requires it — `write.data.path` and object-storage layouts can put data files anywhere — so check where your table actually writes. The demo had the same overstatement in a code comment, so I scoped that one to the catalog the demo configures. No behaviour change — `dataPath` was always a free-form path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
