pan3793 commented on PR #18749: URL: https://github.com/apache/iceberg/pull/18749#issuecomment-6594646828
@anuragmantri @szehon-ho, for your single jar for spark-4.x release, which version are you supposed to use to compile? I'm afraid this will cause potential linkage issues at runtime. A concrete example is Paimon, which has a spark-common module and builds against a higher version of Spark, and may throw `NoClassDefFoundError` in certain cases with a lower version of the Spark runtime. https://github.com/apache/paimon/issues/5037 (this is not the only case, I hit many similar cases, but I don't think the Paimon community wants to handle it, so I didn't report); Another common issue is that some Spark API changes are source-compatible but not binary-compatible, for example, `case class Foo(id: Int)` => `case class Foo(id: Int, name: String = "")`, `new Foo(1)` can compile with both version, but will cause linkage issue when run againt another version. That means, if we want to use one shared code for multiple Spark feature versions, we need either: 1. one shared code build against each Spark feature version and publish the runtime jar, or 2. one shared code build against the latest Spark feature version and publish one unified runtime jar, and build a large smoke test set to ensure the unified runtime jar works well on all supported Spark versions, this likely increases the CI budget and test maintenance effort. TBH, I feel the current approach - copying a source tree for each version of Spark runs well, with AI assistance, it's easy to backport and sync the code. No reflection, little dependency conflict trouble. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
