vinishjail97 opened a new pull request, #838:
URL: https://github.com/apache/incubator-xtable/pull/838
## What is the purpose of the pull request
First of two PRs splitting #837 (per review feedback) into **packaging** and
**listener**. This is the **packaging** PR (the listener stacks on top): it
introduces the `xtable-spark-runtime` module producing a thin, drop-in Spark
bundle, plus RFC-3, and validates the jar end-to-end via `spark-submit`.
## Brief change log
- **RFC-3** (`rfc/rfc-3/rfc-3.md`).
- New module `xtable-spark-runtime_${scala.binary.version}` (thin shaded
bundle, **~4 MB**).
- `XTableSparkSync` — standalone `spark-submit` entry point
(RunSync-equivalent for this bundle).
- `TableSyncSpec` / `XTableSyncService` — shared sync building blocks
(reused by the listener PR).
- `ConversionTargetFactory`: make `ServiceLoader` target discovery
**resilient** — skip targets whose engine lib isn't on the classpath (separable
core change; see commit 1).
- `ITXTableSparkRuntimeBundle` — spark-submit jar-validation test.
### Packaging model (the important part)
Per review feedback, engines are **not bundled**. Spark, Hadoop, **and
Hudi/Iceberg/Delta are all `provided`** — the user brings their own engine
versions (Hudi 1.1 / 1.2 / …) and the thin XTable jar stays compatible across
them.
- Curated shade allowlist: only XTable modules + `guava`/`protobuf`
(relocated under `org.apache.xtable.shaded.*`).
- Libraries XTable **exchanges with the engines are deliberately not
relocated/bundled** — relocating `avro` breaks the boundary (Hudi returns a
real `org.apache.avro.Schema` → `NoSuchMethodError`).
`avro`/`parquet`/`jackson`/`commons` come from the provided runtime.
- Because target discovery uses `ServiceLoader`, it's made resilient so a
user can run with only the engines they use (e.g. Hudi + Iceberg, no Delta).
## Verify this pull request
`ITXTableSparkRuntimeBundle` (skipped unless `SPARK_HOME` is set) builds the
shaded jar and runs a real sync:
```
SPARK_LOCAL_IP=127.0.0.1 mvn -pl xtable-spark-runtime -am verify
```
It creates a local Hudi table, `spark-submit`s the bundle with Hudi +
Iceberg supplied via `--packages` (versions from the root pom: Hudi 1.2.0,
Iceberg 1.9.2), and asserts the Iceberg metadata is produced. Validated locally
on Spark 3.5.2 (engines built for 3.4 → also a cross-version check).
Part of #836. Listener PR follows, stacked on this branch.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
