+1 to this package. I think even without the listeners this will provide a more clear way to use code in your current spark pipelines.
-Tim On Wed, Jul 1, 2026 at 1:30 AM Vinoth Chandar <[email protected]> wrote: > Really look forward to this. > > On 2026/07/01 01:37:25 Vinish Reddy Pannala wrote: > > Hi all, > > > > I'd like to propose a new artifact, xtable-spark-runtime, to make XTable > > dramatically easier to use inside existing Spark pipelines. > > > > Problem > > ------- > > XTable's modules are published to Maven, so today a user can pull in > > xtable-core and assemble their own runtime, but everyone has to solve the > > same shading/classpath problem themselves, or fall back to the > > xtable-utilities bundle, which ships every dependency unshaded (~1GB) and > > is impractical to add to a Spark job. Running it also means building the > > fat jar and invoking a separate sync process with several config files. > > There's no maintained, thin, drop-in artifact for the common case: "I > > already write this table with Spark, keep it in sync in other formats." > > > > Proposal > > -------- > > A thin, relocated runtime jar that a user adds to an existing Spark job: > > spark-submit --packages > > org.apache.xtable:xtable-spark-runtime_2.12:<version>...and activates > > purely through config, no code change: > > > > spark.sql.queryExecutionListeners = > > org.apache.xtable.spark.XTableSyncListener > > spark.xtable.tables = /warehouse/db/orders > > spark.xtable.orders.sourceFormat = HUDI > > spark.xtable.orders.targets = ICEBERG,DELTA > > > > After each successful write to the source table, XTable incrementally > > translates its metadata to the target formats on the driver, so the table > > becomes readable by other engines with no extra job or orchestration. > > > > This follows the bundle model Hudi and Iceberg already use: engine deps > > marked provided, a curated dependency allowlist, and relocations so the > jar > > coexists with the cluster's own guava/jackson/avro. The result should be > a > > bundle in the tens of MB, not ~1GB. > > > > Why it helps the community > > -------------------------- > > - Lowest-friction on-ramp: add one dependency + a few configs to a > pipeline > > you already run, instead of hand-rolling a shaded runtime. > > - One maintained, tested runtime instead of everyone reinventing shading. > > - No standalone job to schedule or babysit. > > > > A couple of things worth settling here: > > 1. Config-declared tables (explicit) vs. auto-detecting the written table > > from the query plan (more magic, less predictable). > > 2. Async vs Blocking sync for batch jobs? (a batch driver can exit before > > an async sync finishes). > > 3. Whether to also ship an Iceberg-style CALL xtable.sync(...) procedure > in > > the same jar for on-demand/backfill use. > > > > I'll open a GitHub issue with the detailed design. Feedback welcome on > the > > overall direction and the three questions above. > > > > Thanks, > > Vinish > > >
