I think that we need to figure out what the goal is before proposing the structure. If the goal is to allow a standalone java jar to run conversions, I would suggest we simply revisit the utilities jar or the web service for this.
The main issue with the utilities jar was that it was bundling all sorts of things, including spark. We have a similar issue with the layout of the project in general where it is hard to pick and choose the lean set of dependencies that you want for your particular use case. That was the reason for the RFC for module restructuring [1]. Before starting any work on this there are a few questions that need to be answered. What will the xtable-java-runtime jar look like? Will it be a bundled jar? Will it have relocated dependencies or will it require users to provide the implementation of the table formats? Before declaring the Delta Kernel ready for use, I would like to see it integrated into our integration test suite in ITConversionController [2]. It is also worthwhile to look back at the recent issues reported for Delta conversion and ensure they are also fixed in the Delta Kernel path. 1. https://github.com/apache/incubator-xtable/pull/612 2. https://github.com/apache/incubator-xtable/blob/main/xtable-core/src/test/java/org/apache/xtable/ITConversionController.java On Mon, Jul 13, 2026 at 10:31 PM Vaibhav Kumar <[email protected]> wrote: > Hey Vinish > > +1 for the idea, I can take this up. > > On Tue, 14 Jul 2026 at 6:39 AM, Vinish Reddy Pannala < > [email protected]> wrote: > > > Hi all, > > > > I'd like to open a discussion on publishing an xtable-java-runtime - a > > lightweight, Spark-free runtime for running XTable metadata sync - now > that > > Delta Kernel support has landed on main. > > > > Motivation > > > > Until now, any XTable sync involving Delta effectively required a > > SparkSession, since the Delta source/target went through delta-spark's > > DeltaLog APIs. The Hudi and Iceberg paths are already pure Java, so Delta > > was the one format anchoring us to Spark. Today the runtime options we > ship > > (the xtable-utilities CLI and xtable-spark-runtime, #843) both carry > Spark. > > > > With Delta Kernel PR's now merged, Delta can be read and written without > > Spark. That removes the last hard Spark dependency in the sync path, > which > > means we can publish a genuinely Spark-free, pure-Java runtime jar > covering > > Hudi, Iceberg, and Delta (via Kernel). > > > > Why it matters - non-Spark engines > > > > A Java-only runtime makes XTable embeddable anywhere on the JVM without > > dragging in Spark: > > > > - Flink jobs syncing metadata in-line > > - Trino / Presto plugins and connectors > > - Plain Java services, functions, or orchestration steps > > > > This drastically shrinks the footprint and the dependency-conflict > surface > > versus a Spark bundle, and opens XTable to the large set of users who > > aren't on Spark. > > > > Context > > > > We discussed this during the community sync this morning and agreed it's > > worth pursuing - the notes capture that with Delta Kernel merged, an > > xtable-java-runtime jar can be published with Delta Kernel as > > source/target. Full notes: > > > > > https://docs.google.com/document/d/1mSthtQBVDDzi9bLn9sWDsPaJLJHCDoK_MxDsSphhlos/edit?usp=sharing > > > > Questions for the list > > > > - Do we agree an xtable-java-runtime is worth publishing in the next > > release? > > - Any concerns about Delta Kernel feature parity for a Kernel-only > runtime > > (e.g. deletion vectors, issue #713)? > > > > Looking forward to your thoughts. > > > > Thanks, > > Vinish > > >
