Hi all, Thanks everyone for the thoughtful comments on this thread and on the GitHub issue. The questions on Spark version support, Delta Kernel, catalog/name-based tables, and the sync execution model were all really helpful in shaping the design.
I've written up the proposal as RFC-3 and opened a draft PR with a first working implementation: - Issue: https://github.com/apache/incubator-xtable/issues/836 - Draft PR (RFC + code): https://github.com/apache/incubator-xtable/pull/837 In short: a new xtable-spark-runtime module with a driver-side QueryExecutionListener that runs an incremental ConversionController.sync(...) in-job after a successful write, activated through spark.xtable.* config only — no separate process, no user code change. It's stateless and synchronous by design (the sync watermark already lives in the target metadata, and the async/"when to trigger" decision is left to the user's DAG). The PR includes unit tests and an embedded local[*] Hudi->Delta/Iceberg integration test. It's still in draft state though, I want to validate the runtime jar in a real Spark + K8s job and run the license checks before marking it ready for review. To keep everything in one place, let's continue detailed design/implementation discussion on the GitHub issue (#836) and the PR. Feedback is very welcome. Thanks, Vinish
