rangareddy commented on code in PR #922:
URL: https://github.com/apache/incubator-xtable/pull/922#discussion_r3986538089
##########
website/docs/how-to-spark-runtime.md:
##########
@@ -28,22 +32,101 @@ Every engine dependency is `provided`, so the jar is about
4 MB and reuses the H
Delta libraries your cluster already has. See the
[Downloads](/releases/downloads) page for the
full list of releases.
-## Adding the sync to a Spark job
+## Prerequisites
+
+XTable converts the metadata of a table that already exists, so you need a
source table before you
+can sync anything. If you don't have one, follow
+[Creating your first interoperable table](/docs/how-to) to create the Hudi
table `people` under
+`file:///tmp/hudi-dataset`. Every example on this page syncs that table.
+
+You also need a Spark 3.4.x or 3.5.x installation with `$SPARK_HOME` set. See
Review Comment:
You were right, and it was worse than the classpath. I set up Spark 3.4.2
(matching
`spark.version` at the tag) and JDK 11, and ran the page end to end against
the released jar.
Added a **Getting the engine libraries** section that resolves the
`provided` engines from the
published POM, the recipe `demo/spark-runtime/README.md` already uses:
```
mvn -q -f xtable-spark-runtime_2.12-0.4.0-incubating.pom
dependency:build-classpath \
-DincludeScope=provided -Dmdep.outputFile=engine-classpath-raw.txt
```
filtered to 21 engine jars (`hudi-spark3.4-bundle 0.14.0`,
`hudi-java-client`,
`hudi-client-common`, `hudi-timeline-service`,
`iceberg-core/-api/-common/-bundled-guava 1.9.2`,
`delta-core 2.4.0`, `delta-storage`, `delta-kernel-api/-defaults 4.0.0`,
`avro 1.12.0`, the seven
`parquet-* 1.15.2`, `jol-core 0.16`), then passed to every `spark-submit` on
the page through
`spark.driver.extraClassPath` and `spark.executor.extraClassPath`, as
`ITXTableSparkRuntimeBundle` does. A note records why it has to be a flat
classpath rather than
`--packages` or `--jars`.
Running it that way, the documented submit now works:
```
INFO XTableSparkSync: Starting XTable sync for people
(file:///tmp/hudi-dataset/people)
INFO XTableSparkSync: Completed XTable sync for people
EXIT=0
```
and Hudi, Delta and Iceberg all read back the same six rows.
One more thing the run exposed, which the classpath alone does not fix: the
Quickstart's shell now
uses `hudi-spark3.4-bundle_2.12:1.2.0`, which writes Hudi table version 9.
`0.4.0-incubating` is
built against Hudi `0.14.0`, so the sync fails before reading anything:
```
java.lang.IllegalArgumentException
at
org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:33)
at
org.apache.hudi.common.table.timeline.versioning.TimelineLayoutVersion.<init>(TimelineLayoutVersion.java:40)
at
org.apache.xtable.hudi.HudiConversionSourceProvider.getConversionSourceInstance(HudiConversionSourceProvider.java:41)
```
Creating the table with `0.14.0` instead makes it pass. Added a caution to
Prerequisites. The
underlying cause is #904 bumping the Quickstart to match `main`
(`hudi.version` `1.2.0`) while the
site documents the released `0.4.0-incubating` (`0.14.0`); I can file that
separately.
##########
website/docs/how-to-spark-runtime.md:
##########
@@ -28,22 +32,101 @@ Every engine dependency is `provided`, so the jar is about
4 MB and reuses the H
Delta libraries your cluster already has. See the
[Downloads](/releases/downloads) page for the
full list of releases.
-## Adding the sync to a Spark job
+## Prerequisites
+
+XTable converts the metadata of a table that already exists, so you need a
source table before you
+can sync anything. If you don't have one, follow
+[Creating your first interoperable table](/docs/how-to) to create the Hudi
table `people` under
+`file:///tmp/hudi-dataset`. Every example on this page syncs that table.
+
+You also need a Spark 3.4.x or 3.5.x installation with `$SPARK_HOME` set. See
+[Spark version support](#spark-version-support) for what each line provides.
-This is what the jar is for. Your job already writes a table in one format,
and you want that same
-table to be interoperable with the others as soon as the write finishes.
+## Running a sync as its own job
+
+The jar ships a `spark-submit` entry point,
`org.apache.xtable.spark.XTableSparkSync`. Pass the jar
+as the application jar rather than with `--jars`. It runs in one of two modes.
-Add the jar to the `spark-submit` you already use, alongside your application
jar:
+### Syncing a single table
+
+Describe the table with command line options. This is the shortest path from a
source table to an
+interoperable one:
```shell md title="shell"
$SPARK_HOME/bin/spark-submit \
- --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
- --class com.example.OrdersJob \
- orders-job.jar
+ --class org.apache.xtable.spark.XTableSparkSync \
+ --master 'local[*]' \
+ xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+ --basepath file:///tmp/hudi-dataset/people \
+ --sourceformat HUDI \
+ --targets ICEBERG,DELTA
Review Comment:
Confirmed and fixed, and I measured it both ways rather than reasoning about
it.
`HudiSourceConfig.parsePartitionFieldSpecs` returns
`Collections.emptyList()` for a null or empty
spec, and on the Quickstart's `people` table that shows up as:
| | Iceberg `default-spec-id` | Delta `partitionColumns` |
| --- | --- | --- |
| without the flag | `0` (fields `[]`) | `[]` |
| with `city:VALUE` | `1` (identity transform on `city`) | `['city']` |
So the targets really were being created unpartitioned. The single-table
command now passes
`--partitionspec city:VALUE`, both `TableSyncSpec` examples pass
`.partitionSpec("city:VALUE")`,
and the `--datasetconfig` entry carries `partitionSpec: city:VALUE`. I also
changed the
option-table example from `level:VALUE` to `city:VALUE` so the page is
consistent throughout.
##########
website/docs/how-to-spark-runtime.md:
##########
@@ -28,22 +32,101 @@ Every engine dependency is `provided`, so the jar is about
4 MB and reuses the H
Delta libraries your cluster already has. See the
[Downloads](/releases/downloads) page for the
full list of releases.
-## Adding the sync to a Spark job
+## Prerequisites
+
+XTable converts the metadata of a table that already exists, so you need a
source table before you
+can sync anything. If you don't have one, follow
+[Creating your first interoperable table](/docs/how-to) to create the Hudi
table `people` under
+`file:///tmp/hudi-dataset`. Every example on this page syncs that table.
+
+You also need a Spark 3.4.x or 3.5.x installation with `$SPARK_HOME` set. See
+[Spark version support](#spark-version-support) for what each line provides.
-This is what the jar is for. Your job already writes a table in one format,
and you want that same
-table to be interoperable with the others as soon as the write finishes.
+## Running a sync as its own job
+
+The jar ships a `spark-submit` entry point,
`org.apache.xtable.spark.XTableSparkSync`. Pass the jar
+as the application jar rather than with `--jars`. It runs in one of two modes.
-Add the jar to the `spark-submit` you already use, alongside your application
jar:
+### Syncing a single table
+
+Describe the table with command line options. This is the shortest path from a
source table to an
+interoperable one:
```shell md title="shell"
$SPARK_HOME/bin/spark-submit \
- --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
- --class com.example.OrdersJob \
- orders-job.jar
+ --class org.apache.xtable.spark.XTableSparkSync \
+ --master 'local[*]' \
+ xtable-spark-runtime_2.12-0.4.0-incubating.jar \
+ --basepath file:///tmp/hudi-dataset/people \
+ --sourceformat HUDI \
+ --targets ICEBERG,DELTA
```
-Then call `XTableSyncService` after your write. You describe the table with a
`TableSyncSpec` and
-hand it the session's Hadoop configuration:
+After this finishes, `/tmp/hudi-dataset/people` carries Iceberg and Delta
metadata next to the Hudi
+data files, and all three formats read the same rows.
+
+### Syncing several tables from a config file
+
+To sync more than one table in a single submit, list them in a YAML file and
pass
+`--datasetconfig`. It takes the same config that `RunSync` uses, so an
existing file works
+unchanged:
+
+```yaml md title="my_config.yaml"
+sourceFormat: HUDI
+targetFormats:
+ - DELTA
+ - ICEBERG
+datasets:
+ -
+ tableBasePath: file:///tmp/hudi-dataset/people
+ tableName: people
+ -
+ tableBasePath: file:///tmp/hudi-dataset/orders
Review Comment:
Confirmed by running it. With `people` (which the prerequisite creates) plus
`orders` (which it
does not):
```
INFO XTableSparkSync: Completed XTable sync for people
ERROR XTableSparkSync: XTable sync failed for orders
(file:///tmp/hudi-dataset/orders)
Exception in thread "main" java.lang.IllegalStateException: 1 of 2 table
sync(s) failed
EXIT=1
```
so the example as written could not succeed. The runnable config now lists
only `people` and exits
0. The multi-table shape is still shown, but as a separate snippet clearly
labelled as a second
`datasets` entry rather than as part of the file you run, and the text now
states the behaviour:
> Add one entry per table. Every entry has to point at a table that already
exists:
> `XTableSparkSync` logs each table that fails, carries on with the rest,
then exits nonzero, so
> one missing path fails the whole submit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
