vinishjail97 commented on code in PR #922:
URL: https://github.com/apache/incubator-xtable/pull/922#discussion_r3962285712
##########
website/docs/how-to-spark-runtime.md:
##########
@@ -28,22 +32,101 @@ Every engine dependency is `provided`, so the jar is about
4 MB and reuses the H
Delta libraries your cluster already has. See the
[Downloads](/releases/downloads) page for the
full list of releases.
-## Adding the sync to a Spark job
+## Prerequisites
+
+XTable converts the metadata of a table that already exists, so you need a
source table before you
+can sync anything. If you don't have one, follow
+[Creating your first interoperable table](/docs/how-to) to create the Hudi
table `people` under
+`file:///tmp/hudi-dataset`. Every example on this page syncs that table.
+
+You also need a Spark 3.4.x or 3.5.x installation with `$SPARK_HOME` set. See
Review Comment:
The spark runtime POM marks the Hudi, Iceberg, and Delta dependencies as
`provided`, and `ITXTableSparkRuntimeBundle` adds both
`spark.driver.extraClassPath` and `spark.executor.extraClassPath` before
invoking this entry point.
The linked Quickstart adds Hudi only to its earlier PySpark process, not
this separate submit, while this example also needs Iceberg and Delta. Would it
be reasonable to document the required engine coordinates or classpath here and
include them in the runnable command? I think this blocks the quickstart from
working as written.
##########
website/docs/how-to-spark-runtime.md:
##########
@@ -28,22 +32,101 @@ Every engine dependency is `provided`, so the jar is about
4 MB and reuses the H
Delta libraries your cluster already has. See the
[Downloads](/releases/downloads) page for the
full list of releases.
-## Adding the sync to a Spark job
+## Prerequisites
+
+XTable converts the metadata of a table that already exists, so you need a
source table before you
+can sync anything. If you don't have one, follow
+[Creating your first interoperable table](/docs/how-to) to create the Hudi
table `people` under
+`file:///tmp/hudi-dataset`. Every example on this page syncs that table.
+
+You also need a Spark 3.4.x or 3.5.x installation with `$SPARK_HOME` set. See
+[Spark version support](#spark-version-support) for what each line provides.
-This is what the jar is for. Your job already writes a table in one format,
and you want that same
-table to be interoperable with the others as soon as the write finishes.
+## Running a sync as its own job
+
+The jar ships a `spark-submit` entry point,
`org.apache.xtable.spark.XTableSparkSync`. Pass the jar
+as the application jar rather than with `--jars`. It runs in one of two modes.
-Add the jar to the `spark-submit` you already use, alongside your application
jar:
+### Syncing a single table
+
+Describe the table with command line options. This is the shortest path from a
source table to an
+interoperable one:
```shell md title="shell"
$SPARK_HOME/bin/spark-submit \
- --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
- --class com.example.OrdersJob \
- orders-job.jar
+ --class org.apache.xtable.spark.XTableSparkSync \
+ --master 'local[*]' \
+ xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+ --basepath file:///tmp/hudi-dataset/people \
+ --sourceformat HUDI \
+ --targets ICEBERG,DELTA
```
-Then call `XTableSyncService` after your write. You describe the table with a
`TableSyncSpec` and
-hand it the session's Hadoop configuration:
+After this finishes, `/tmp/hudi-dataset/people` carries Iceberg and Delta
metadata next to the Hudi
+data files, and all three formats read the same rows.
+
+### Syncing several tables from a config file
+
+To sync more than one table in a single submit, list them in a YAML file and
pass
+`--datasetconfig`. It takes the same config that `RunSync` uses, so an
existing file works
+unchanged:
+
+```yaml md title="my_config.yaml"
+sourceFormat: HUDI
+targetFormats:
+ - DELTA
+ - ICEBERG
+datasets:
+ -
+ tableBasePath: file:///tmp/hudi-dataset/people
+ tableName: people
+ -
+ tableBasePath: file:///tmp/hudi-dataset/orders
Review Comment:
I am not sure this example can succeed after following the prerequisite
above. The Quickstart creates only `people`, while this configuration also
syncs `orders` and `events`. `XTableSparkSync` records each missing source as a
failure and exits nonzero after processing the list. Would it work to use
source paths that the page creates, or label these two entries as placeholders
and explain that readers must create them first? Because the page says every
example uses the created `people` table, I think the current example is
blocking for a runnable quickstart.
##########
website/docs/how-to-spark-runtime.md:
##########
@@ -28,22 +32,101 @@ Every engine dependency is `provided`, so the jar is about
4 MB and reuses the H
Delta libraries your cluster already has. See the
[Downloads](/releases/downloads) page for the
full list of releases.
-## Adding the sync to a Spark job
+## Prerequisites
+
+XTable converts the metadata of a table that already exists, so you need a
source table before you
+can sync anything. If you don't have one, follow
+[Creating your first interoperable table](/docs/how-to) to create the Hudi
table `people` under
+`file:///tmp/hudi-dataset`. Every example on this page syncs that table.
+
+You also need a Spark 3.4.x or 3.5.x installation with `$SPARK_HOME` set. See
+[Spark version support](#spark-version-support) for what each line provides.
-This is what the jar is for. Your job already writes a table in one format,
and you want that same
-table to be interoperable with the others as soon as the write finishes.
+## Running a sync as its own job
+
+The jar ships a `spark-submit` entry point,
`org.apache.xtable.spark.XTableSparkSync`. Pass the jar
+as the application jar rather than with `--jars`. It runs in one of two modes.
-Add the jar to the `spark-submit` you already use, alongside your application
jar:
+### Syncing a single table
+
+Describe the table with command line options. This is the shortest path from a
source table to an
+interoperable one:
```shell md title="shell"
$SPARK_HOME/bin/spark-submit \
- --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
- --class com.example.OrdersJob \
- orders-job.jar
+ --class org.apache.xtable.spark.XTableSparkSync \
+ --master 'local[*]' \
+ xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+ --basepath file:///tmp/hudi-dataset/people \
+ --sourceformat HUDI \
+ --targets ICEBERG,DELTA
Review Comment:
The linked Quickstart writes `people` partitioned by `city`, and its own
RunSync configuration passes `partitionSpec: city:VALUE`. This command and both
`TableSyncSpec` examples omit that setting. In 0.4.0, `HudiSourceConfig`
converts a missing setting to an empty partition-field list, so the target
metadata is created as unpartitioned. Would it be reasonable to add
`--partitionspec city:VALUE` here and `.partitionSpec("city:VALUE")` in both
embedded examples? I think preserving the source partition specification is
part of the documented interoperability result, so this looks blocking to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
