vinishjail97 opened a new pull request, #964:
URL: https://github.com/apache/incubator-xtable/pull/964
## What is the purpose of the pull request
Adds the `xtable-spark-runtime` module to `main`: a thin, drop-in Spark
bundle that runs XTable metadata sync, validated end to end with `spark-submit`.
The same module was reviewed and merged on `branch-0.4` in #843. This PR
brings that version to `main` on Hudi 1.2.1 and replaces #838, which had
drifted too far from `main`. The RFC lives in its own PR, #960.
## Brief change log
- New module `xtable-spark-runtime_${scala.binary.version}`, a thin shaded
bundle (~4.5 MB).
- `XTableSparkSync`, a `spark-submit` entry point, the equivalent of
`RunSync` for this bundle:
- single-table mode (`--basepath`, `--sourceformat`, `--targets`, ...) or
`--datasetconfig <yaml>` for several tables, on a local or cloud path;
- `--usedeltakernel` for the Spark-free Delta Kernel, enabled
automatically on Spark 3.5+;
- Hudi, Iceberg, Delta, Paimon and Parquet sources; Hudi, Iceberg and
Delta targets;
- formats are validated before Spark starts, and one failing table does
not stop the rest.
- `TableSyncSpec` / `XTableSyncService`, the sync building blocks for use
inside a Spark job.
- `ConversionTargetFactory` skips registered targets whose engine is not on
the classpath, so a runtime with only some engines still works. `hasNext()` is
inside the guard too, because it loads providers lazily.
- `LICENSE-bundled` / `NOTICE-bundled` for the bundled guava, commons-cli
and snakeyaml.
- `.github/workflows/spark-runtime-validation.yml` runs the bundle IT on
Spark 3.4.3 and 3.5.9.
The Java sources are the same as on `branch-0.4`. The differences from #843
are:
- The Hudi 1.2.1 version and the `HoodieCleanStat` builder already landed on
`main` with #835.
- The module's shade `<filters>` use `combine.self="override"`. Without it,
Maven merges the parent's `software.amazon.awssdk:bundle` filter into the
snakeyaml filter by position and drops every snakeyaml class.
- The shade plugin writes `dependency-reduced-pom.xml` next to the module
pom, so it is now in `.gitignore`, and its spotless exclude is in the POM
format block.
- Shorter comments.
### Packaging model
Engines are not bundled. Spark, Hadoop and Hudi/Iceberg/Delta are
`provided`, so users bring their own engine versions.
- The shaded jar is the main artifact with a dependency-reduced POM, the
same shape as `iceberg-spark-runtime` and `hudi-spark-bundle`.
- The shade allowlist is XTable plus guava, commons-cli and snakeyaml,
relocated under `org.apache.xtable.shaded.*`.
- Libraries XTable exchanges with the engines (`avro`, `parquet`, `jackson`)
are not bundled or relocated, because a relocated Avro breaks the boundary with
Hudi.
## Verify this pull request
- `XTableSparkSyncTest`: 45 unit tests for option parsing, format validation
and the dataset YAML.
- `ITXTableSparkRuntimeBundle` (skipped unless `SPARK_HOME` is set)
`spark-submit`s a real sync for one case per direction across Hudi, Iceberg and
Delta, and checks the target data against the source. It passes 4/4 locally on
Spark 3.4.2 and 3.5.2:
```
SPARK_LOCAL_IP=127.0.0.1 SPARK_HOME=<spark> ./mvnw -pl xtable-spark-runtime
clean verify
```
- `install -DskipITs` for `xtable-core` and this module, `spotless:check`
and `validate_shaded_license_coverage.sh` for this module pass.
- Also ran the bundle as Spark jobs on Kubernetes against S3, on Spark 3.4.2
and on Spark 3.5.2 with the Hudi 3.5 bundle and `delta-spark` 3.2.1. Hudi to
Iceberg/Delta covered three TPC-DS-style tables up to 6.87M rows in 2,180
partitioned files, and Iceberg to Hudi/Delta covered one table. For every
table, the target metadata lists the same files, sizes and record counts as the
source, and reading each format back gives the same row count and content hash.
Part of #836.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
