Hi Xin, Thanks for the feedback and for sharing your experience with Delta; this is very useful input, and I agree with all the points you raised.
To address a few of your suggestions based on the proof of concept (POC): - Scoping shims to genuine API adaptation: The POC already reflects this. Of the 67 files that diverged between the 4.1 and 4.2 versions I tested, only one was pure drift (which I moved to the shared folder). The remaining 66 were due to actual Spark API changes. - Stable Iceberg-owned contract: I agree that this will be the most challenging part. I am happy to dig into what a concrete implementation would look like if the community is interested. - Shared business logic and test suites: The POC is already structured this way. The full test suite of around 17,800 tests is shared, and only version-specific expectations are shimmed. - CI duplication reporting: I am thinking we could introduce a CI job that periodically diffs adjacent-version shim directories to flag file pairs above a certain similarity threshold as promotion candidates. It could also flag any shim file that exceeds a specific size threshold for additional review scrutiny. Thanks, Anurag On Fri, Jul 31, 2026 at 4:03 PM Xin Huang via dev <[email protected]> wrote: > Hi Anurag, > > Thank you and Sebastian for proposing this idea, and thank you for > building and testing the proof of concept. I wanted to share a few thoughts > based on my personal experience and feelings. > > I think a shared source tree is a promising direction. It should make > Spark version upgrades smaller and easier to review. However, Delta’s > experience suggests that shim governance should be considered from the > beginning. Delta initially accumulated mirrored per-version shims and later > needed #7201 <https://github.com/delta-io/delta/pull/7201> and #7202 > <https://github.com/delta-io/delta/pull/7202> to remove roughly 2.8K > lines of duplication and introduce additional cross-version shim > directories. > > A few ideas on top of my mind that might help keep the shim layer > manageable over time: > > - It may be useful to keep shims focused on Spark API adaptation. For > example, the Spark 4.1 GeometryVal/GeographyVal APIs changed to > BinaryView in Spark 4.2. A small geospatial adapter could handle that > difference while allowing the surrounding Iceberg reader and writer logic > to remain shared. > - Where practical, shared code could depend on a stable, > version-neutral contract. For example, shared view-handling logic could > operate on an Iceberg-owned view metadata representation, while small > version-specific adapters translate between it and Spark 4.1’s ViewInfo > API or Spark 4.2’s new View API. This might be a main challenging point > - Keeping business logic and complete test suites shared where > possible may help avoid recreating some of the same maintenance burden > inside the shim directories. > - It may also help to derive source roots from one central definition > and add CI reporting for duplicated or growing shim code. > > These are just some initial thoughts. Look forward to hearing what you > think and would be happy to discuss further. > Thanks > Xin > >
