vinishjail97 commented on code in PR #913: URL: https://github.com/apache/incubator-xtable/pull/913#discussion_r3848795619
########## website/docs/how-to-spark-runtime.md: ########## @@ -0,0 +1,218 @@ +--- +sidebar_position: 3 +title: "Run an XTable sync on Apache Spark" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run an XTable sync on Apache Spark + +`xtable-spark-runtime` is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache +Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source +table's metadata and writes the target format's metadata alongside the data that is already there. + +The jar is meant to be dropped into a Spark job you already run, so you don't need a separate +process or a separate cluster to keep a table interoperable across formats. + +## Getting the jar + +The jar is published to Maven Central as +`org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating`. Download it once: + +```shell md title="shell" +curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar +``` + +Every engine dependency is `provided`, so the jar is about 4 MB and reuses the Hudi, Iceberg and +Delta libraries your cluster already has. See the [Downloads](/releases/downloads) page for the +full list of releases. + +## Adding the sync to a Spark job + +This is what the jar is for. Your job already writes a table in one format, and you want that same +table to be interoperable with the others as soon as the write finishes. + +Add the jar to the `spark-submit` you already use, alongside your application jar: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \ + --class com.example.OrdersJob \ + orders-job.jar +``` + +Then call `XTableSyncService` after your write. You describe the table with a `TableSyncSpec` and +hand it the session's Hadoop configuration: + +<Tabs +groupId="language" +defaultValue="scala" +values={[ +{ label: 'Scala', value: 'scala', }, +{ label: 'Java', value: 'java', }, +]} +> +<TabItem value="scala"> + +```scala md title="OrdersJob.scala" +import java.util.{Arrays => JArrays} +import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService} + +val basePath = "s3://example-warehouse/db/orders" + +// the write your job already does +df.write.format("hudi").options(hudiOptions).mode("append").save(basePath) + +// the one call you add +new XTableSyncService().sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(JArrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext.hadoopConfiguration) +``` + +</TabItem> +<TabItem value="java"> + +```java md title="OrdersJob.java" +import java.util.Arrays; +import org.apache.xtable.spark.TableSyncSpec; +import org.apache.xtable.spark.XTableSyncService; + +String basePath = "s3://example-warehouse/db/orders"; + +// the write your job already does +df.write().format("hudi").options(hudiOptions).mode("append").save(basePath); + +// the one call you add +new XTableSyncService() + .sync( + TableSyncSpec.builder() + .key("orders") + .basePath(basePath) + .sourceFormat("HUDI") + .targets(Arrays.asList("ICEBERG", "DELTA")) + .build(), + spark.sparkContext().hadoopConfiguration()); +``` + +</TabItem> +</Tabs> + +To compile against these classes, add the same Maven coordinates to your build with `provided` +scope. + +The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling +back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That +makes it safe to call after every write. + +:::note Tables are addressed by path +The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent +of the `RunSync` Iceberg catalog config (`-i`) yet. For an Iceberg source, point `basePath` at the Review Comment: ``` For an Iceberg source, point `basePath` at the table root and set `dataPath` to `<basePath>/data`, since that is where Iceberg keeps its data files and where each target writes its metadata ``` This is not really true..the data files can be present anywhere for iceberg. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
