Hi Tim,
Thanks for the response and sharing the RFC. You're right that we need to be clear on the goal. Let me be precise as per our discussion in the last sync call : the goal is not to replace or patch the utilities jar. It is to allow XTable to be used as a drop-in library in JVM applications that do not run Spark at all — Flink jobs, Trino/Presto plugins, plain Java services. The utility jar cannot serve that goal even if we slimmed it down, because it is designed as a CLI tool and bundles Spark by design. This actually lines up directly with the pain points your RFC-2 [1] identifies: ▎ "When running the prebuilt utilities bundle jar, the user will bring in all the required ▎ dependencies which currently include three table formats, spark, and hadoop. This results in a ▎ very large jar containing more than the user really needs." ▎ "When building your own jar, you can run into issues with conflicts in versions for the ▎ specific table formats you want to use since the user will often already have an implementation ▎ of at least one format on their classpath." An xtable-java-runtime addresses both pain points directly for the non-Spark user segment. It would be a shaded jar with relocated dependencies — the same pattern used in xtable-spark-runtime(#843) — bundling Delta Kernel, Hudi Java client, and Iceberg core with no Spark. Because dependencies are relocated inside the jar, version conflicts on the user's classpath are avoided. This is exactly the per-format shaded jar approach your RFC-2 proposes. To your point on jar structure: it would be a shaded fat jar with relocated dependencies. Users would consume it via --jars or as a Maven dependency, and the dependency-reduced POM would expose no transitive engine dependencies. On Delta Kernel readiness — you are right that it is not yet integrated into ITConversionController as a first-class path alongside the Spark-based Delta tests. The dedicated test suite (ITDeltaKernelConversionSource, TestDeltaKernelReadWriteIntegration, TestDeltaKernelSync) covers the Kernel path in depth, but I agree the shared harness is the right bar. Would it make sense to track the ITConversionController integration as a prerequisite for the java-runtime publish, so we can keep the discussion moving forward while that work gets done in parallel? To summarise — the xtable-java-runtime is not a workaround for the module restructuring RFC. It is one concrete deliverable that RFC-2 leads to, scoped narrowly to a packaging module so it does not depend on the full restructure being complete first. Happy to open a tracking issue for the ITConversionController gap and put together a more detailed design doc if the community agrees with the direction. Thanks, Vaibhav On Tue, Jul 14, 2026 at 5:53 PM Tim Brown <[email protected]> wrote: > I think that we need to figure out what the goal is before proposing the > structure. If the goal is to allow a standalone java jar to run > conversions, I would suggest we simply revisit the utilities jar or the web > service for this. > > The main issue with the utilities jar was that it was bundling all sorts of > things, including spark. We have a similar issue with the layout of the > project in general where it is hard to pick and choose the lean set of > dependencies that you want for your particular use case. That was the > reason for the RFC for module restructuring [1]. > > Before starting any work on this there are a few questions that need to be > answered. What will the xtable-java-runtime jar look like? Will it be a > bundled jar? Will it have relocated dependencies or will it require users > to provide the implementation of the table formats? > > Before declaring the Delta Kernel ready for use, I would like to see it > integrated into our integration test suite in ITConversionController [2]. > It is also worthwhile to look back at the recent issues reported for Delta > conversion and ensure they are also fixed in the Delta Kernel path. > > 1. https://github.com/apache/incubator-xtable/pull/612 > 2. > > https://github.com/apache/incubator-xtable/blob/main/xtable-core/src/test/java/org/apache/xtable/ITConversionController.java > > On Mon, Jul 13, 2026 at 10:31 PM Vaibhav Kumar <[email protected]> > wrote: > > > Hey Vinish > > > > +1 for the idea, I can take this up. > > > > On Tue, 14 Jul 2026 at 6:39 AM, Vinish Reddy Pannala < > > [email protected]> wrote: > > > > > Hi all, > > > > > > I'd like to open a discussion on publishing an xtable-java-runtime - a > > > lightweight, Spark-free runtime for running XTable metadata sync - now > > that > > > Delta Kernel support has landed on main. > > > > > > Motivation > > > > > > Until now, any XTable sync involving Delta effectively required a > > > SparkSession, since the Delta source/target went through delta-spark's > > > DeltaLog APIs. The Hudi and Iceberg paths are already pure Java, so > Delta > > > was the one format anchoring us to Spark. Today the runtime options we > > ship > > > (the xtable-utilities CLI and xtable-spark-runtime, #843) both carry > > Spark. > > > > > > With Delta Kernel PR's now merged, Delta can be read and written > without > > > Spark. That removes the last hard Spark dependency in the sync path, > > which > > > means we can publish a genuinely Spark-free, pure-Java runtime jar > > covering > > > Hudi, Iceberg, and Delta (via Kernel). > > > > > > Why it matters - non-Spark engines > > > > > > A Java-only runtime makes XTable embeddable anywhere on the JVM without > > > dragging in Spark: > > > > > > - Flink jobs syncing metadata in-line > > > - Trino / Presto plugins and connectors > > > - Plain Java services, functions, or orchestration steps > > > > > > This drastically shrinks the footprint and the dependency-conflict > > surface > > > versus a Spark bundle, and opens XTable to the large set of users who > > > aren't on Spark. > > > > > > Context > > > > > > We discussed this during the community sync this morning and agreed > it's > > > worth pursuing - the notes capture that with Delta Kernel merged, an > > > xtable-java-runtime jar can be published with Delta Kernel as > > > source/target. Full notes: > > > > > > > > > https://docs.google.com/document/d/1mSthtQBVDDzi9bLn9sWDsPaJLJHCDoK_MxDsSphhlos/edit?usp=sharing > > > > > > Questions for the list > > > > > > - Do we agree an xtable-java-runtime is worth publishing in the next > > > release? > > > - Any concerns about Delta Kernel feature parity for a Kernel-only > > runtime > > > (e.g. deletion vectors, issue #713)? > > > > > > Looking forward to your thoughts. > > > > > > Thanks, > > > Vinish > > > > > >
