vinishjail97 opened a new pull request, #837:
URL: https://github.com/apache/incubator-xtable/pull/837
## What is the purpose of the pull request
Implements the first slice of #836 (RFC-3): a new `xtable-spark-runtime`
module that runs incremental XTable metadata sync **inside an existing Spark
job**, activated by config only — no separate `RunSync` process, no user code
change.
A driver-side `QueryExecutionListener` runs `ConversionController.sync(...)`
after a successful write to a configured source table:
```
spark-submit --packages org.apache.xtable:xtable-spark-runtime_2.12:<ver> \
--conf
spark.sql.queryExecutionListeners=org.apache.xtable.spark.XTableSyncListener \
--conf spark.xtable.tables=orders \
--conf spark.xtable.orders.basePath=/warehouse/db/orders \
--conf spark.xtable.orders.sourceFormat=HUDI \
--conf spark.xtable.orders.targets=ICEBERG,DELTA
```
Design details are in `rfc/rfc-3/rfc-3.md`.
## Brief change log
- New module `xtable-spark-runtime_${scala.binary.version}` (Spark/Hadoop
`provided`), registered in the parent `pom.xml`.
- `XTableSparkConfig` — parse `spark.xtable.*` into `TableSyncSpec` (path-
and name-based table selection).
- `XTableSyncService` — build `ConversionConfig(INCREMENTAL)` and run
`ConversionController.sync(...)`.
- `PlanTargetResolver` — best-effort extraction of the written path from the
analyzed plan (isolated Spark-internal dependence).
- `XTableSyncListener` — **stateless, synchronous**: on a recognized write
matching a configured table, run the sync inline on the callback. Single-flight
falls out for free (single-threaded `LiveListenerBus`); per-table failures are
caught as `Throwable` so a listener never destabilizes the bus.
- RFC-3.
### Design notes
- **Stateless**: the sync watermark already lives in the target's
`TableSyncMetadata`, and `INCREMENTAL` auto-falls back to a full snapshot when
unsafe, so there is nothing to track client-side.
- **Synchronous**: `QueryExecutionListener` callbacks are delivered
asynchronously by Spark off `SparkListenerSQLExecutionEnd`, so a listener can't
block `write()`; running inline is the simplest correct choice and completes
before JVM exit via the bus drain on `SparkContext.stop()`. Async / "when to
trigger" is intentionally left to the user's DAG as an opt-in follow-up.
### Non-goals (follow-ups)
- Thin shaded/relocated bundle (curated allowlist,
`org.apache.xtable.shaded.*`).
- `StreamingQueryListener` variant, `CALL xtable.sync(...)` procedure, async
execution opt-in.
## Verify this pull request
This change added tests and can be verified as follows:
- Unit: `TestXTableSparkConfig`, `TestPlanTargetResolver`,
`TestXTableSyncListener` (path matching).
- Integration: `ITXTableSyncListener` — embedded `local[*]`; a real Hudi
datasource write triggers the listener and syncs to **Delta + Iceberg** in the
same JVM, asserting the `id` sets match across all three formats.
```
SPARK_LOCAL_IP=127.0.0.1 mvn -pl xtable-spark-runtime -am verify
```
Closes #836 (first slice).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
