vinishjail97 opened a new pull request, #913:
URL: https://github.com/apache/incubator-xtable/pull/913
## What
Website documentation plus one runnable demo. No module sources are touched.
- **0.4.0-incubating release page**
(`website/releases/release-0.4.0-incubating.mdx`) and a matching entry on the
**Downloads** page.
- **New docs page** `Run a sync on Apache Spark`
(`website/docs/spark-runtime.md`), under **Quick Start** in the sidebar and
linked from **Installation**.
- **New demo** `demo/spark-runtime/` — a Spark job that keeps a table
readable in the other formats from inside the job that writes it.
This replaces #847, which was closed so the docs could be rewritten and get
a fresh review.
## The Spark quickstart
`xtable-spark-runtime_2.12` `0.4.0-incubating` is on Maven Central, so the
page points at the released jar rather than a source build. The module is not
on `main`, so nothing here depends on it being in the tree.
The page leads with the case the jar exists for. A job already writes a
table, adds the jar with `--jars`, and calls `XTableSyncService` after its
write:
```shell
$SPARK_HOME/bin/spark-submit \
--jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
--class com.example.OrdersJob \
orders-job.jar
```
The standalone `spark-submit` entry point and the multi-table
`--datasetconfig` follow, then the CLI options, the format matrix and the Spark
version table.
### On `--jars` vs `--packages`
Every dependency in the published pom is `provided` or `test` scoped, so
`--packages` on this jar resolves one jar and no transitives. Both flags work.
`--jars` is the recommendation only because it needs no Maven Central lookup on
each submit.
The real hazard is adding the **engines** with `--packages`. `--packages`
and `--jars` both load into a child class loader, and a second Avro or Parquet
there breaks casts across the boundary. Engine jars belong on
`spark.driver.extraClassPath`, which is what `ITXTableSparkRuntimeBundle`
already does. The page and the demo README both say this.
## The demo
`demo/spark-runtime/XTableInJobSyncDemo.scala` runs both directions and
checks the row counts, so it doubles as a smoke test of a release:
| Direction | The job writes | XTable adds |
| --- | --- | --- |
| 1 | `HUDI` | `ICEBERG`, `DELTA` |
| 2 | `ICEBERG` | `HUDI`, `DELTA` |
It needs no Docker and no Scala compiler — `spark-shell -i` compiles the
script, so a reader runs one command. The jar comes from Maven Central with
`curl` and goes on `--jars`.
A local Spark distribution ships no engines, so the README resolves them
from the downloaded pom with `dependency:build-classpath` and puts them on
`extraClassPath`. Spark 3.4.x is required, because Hudi 0.14.0 publishes no
Spark 3.5 bundle for the datasource direction 1 writes with; the runtime jar
itself supports 3.4 and 3.5.
## Verification
- Ran the demo README verbatim from a clean directory against **Spark
3.4.2** with the published `0.4.0-incubating` jar:
```
[PASS] hudi -> iceberg 1000 rows
[PASS] hudi -> delta 1000 rows
[PASS] iceberg -> hudi 1000 rows
[PASS] iceberg -> delta 1000 rows
ALL CHECKS PASSED
```
- `npm run build` in `website/` succeeds and both new pages generate.
- `apache-rat:check` passes on the new files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
