Hi Anurag, Thank you and Sebastian for proposing this idea, and thank you for building and testing the proof of concept. I wanted to share a few thoughts based on my personal experience and feelings.
I think a shared source tree is a promising direction. It should make Spark version upgrades smaller and easier to review. However, Delta’s experience suggests that shim governance should be considered from the beginning. Delta initially accumulated mirrored per-version shims and later needed #7201 <https://github.com/delta-io/delta/pull/7201> and #7202 <https://github.com/delta-io/delta/pull/7202> to remove roughly 2.8K lines of duplication and introduce additional cross-version shim directories. A few ideas on top of my mind that might help keep the shim layer manageable over time: - It may be useful to keep shims focused on Spark API adaptation. For example, the Spark 4.1 GeometryVal/GeographyVal APIs changed to BinaryView in Spark 4.2. A small geospatial adapter could handle that difference while allowing the surrounding Iceberg reader and writer logic to remain shared. - Where practical, shared code could depend on a stable, version-neutral contract. For example, shared view-handling logic could operate on an Iceberg-owned view metadata representation, while small version-specific adapters translate between it and Spark 4.1’s ViewInfo API or Spark 4.2’s new View API. This might be a main challenging point - Keeping business logic and complete test suites shared where possible may help avoid recreating some of the same maintenance burden inside the shim directories. - It may also help to derive source roots from one central definition and add CI reporting for duplicated or growing shim code. These are just some initial thoughts. Look forward to hearing what you think and would be happy to discuss further. Thanks Xin
