vinishjail97 opened a new issue, #836:
URL: https://github.com/apache/incubator-xtable/issues/836
### Motivation
XTable's modules are published to Maven, so a user can depend on
`xtable-core` and assemble their own runtime today. But there is no
maintained, thin, drop-in artifact — everyone re-solves the same
shading/classpath problem, or uses `xtable-utilities`, an unshaded ~1GB fat
jar that isn't practical to add to a Spark job and requires running a
separate sync process with multiple config files.
XTable conversion is metadata-only and lightweight, so the common case —
"I already write this table with Spark, keep it in sync in other formats" —
should be a one-dependency, config-only addition to an existing pipeline.
### Proposal
A new module `xtable-spark-runtime` producing a thin, relocated bundle that:
- is added via `--jars` / `--packages`,
- is activated through config only (no user code change),
- runs incremental `ConversionController.sync(...)` on the driver after each
successful write to a source table.
**Usage**
```
spark-submit --packages org.apache.xtable:xtable-spark-runtime_2.12:<ver> \
--conf
spark.sql.queryExecutionListeners=org.apache.xtable.spark.XTableSyncListener \
--conf spark.xtable.tables=/warehouse/db/orders \
--conf spark.xtable.orders.sourceFormat=HUDI \
--conf spark.xtable.orders.targets=ICEBERG,DELTA
```
### Packaging (Hudi/Iceberg model)
- Spark / Hadoop → `provided` (on the cluster, not bundled).
- Curated dependency allowlist (no bundling of the full transitive closure).
- Relocate `guava` / `jackson` / `protobuf` / `avro` / `commons` under
`org.apache.xtable.shaded.*` to avoid classpath clashes with the cluster.
- Target size: tens of MB.
### Design
- `XTableSyncListener implements QueryExecutionListener` (batch) and a
`StreamingQueryListener` variant.
- On write-success: resolve target table(s) from `spark.xtable.*`, submit an
incremental sync to a debounced, single-flight-per-table driver executor.
- Sync is idempotent + incremental, so a missed trigger self-heals on the
next commit.
### Scope / non-goals
- No data rewriting — metadata translation only.
- Does not replace the standalone CLI (`RunSync`); complements it.
### Open questions
- [ ] Table selection: config-declared (proposed) vs. plan-inspection
auto-detect.
- [ ] Batch jobs: async (needs shutdown-hook drain) vs. blocking.
- [ ] Also ship a `CALL xtable.sync(...)` SQL procedure in the same jar?
### Tasks
- [ ] Scaffold `xtable-spark-runtime` module (pom: provided engines,
allowlist, relocations).
- [ ] Implement `XTableSyncListener` + `spark.xtable.*` config parsing.
- [ ] Debounced driver executor + shutdown drain.
- [ ] `ITXTableSyncListener` (embedded `local[*]`, Hudi→Delta/Iceberg).
- [ ] spark-shell smoke test + README section.
- [ ] (optional) `CALL xtable.sync(...)` procedure.
### Related
- Bundle/licensing history: dev@ thread "[DISCUSS] Excluding
xtable-utilities from the release process", #536.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
