rdblue edited a comment on pull request #3256: URL: https://github.com/apache/iceberg/pull/3256#issuecomment-941094431
> My original thought in #3237 was that we can define the rule that the shared code can continue to live, and any code that cannot build againt all versions should then be refactored to different version modules. I think that compiling against different versions of Spark is beginning to be a significant risk due to changes across versions. That's only going to get worse when we start adding flags for Scala version as well and continue to add more extensions that touch Spark internals. I don't think it's a good idea to try to refactor common classes out unless they don't have a compile dependency on Spark. And in that case, we can more them to core like the `Writer` classes. > . . . The intention is again to minimize duplicated code by minimizing the number of version folders. I think we're actually better off the other way. What we have now does some pretty crazy things to deal with multiple Spark versions and I think we'll end up with cleaner code if we just duplicate it. This is one of the reasons why I think we want these build changes. > In #3237 I had another scalaVersion system property to allow setting a flexible Scala version. Do you think that is something worth adding? Yes, but as a follow-up. This is just a PR to put a basic structure in place and we will continue to improve it in future commits. > What would be the naming scheme for modules after v3.0? We don't need to decide this now, since this doesn't introduce new modules yet. I'd like to focus on getting this in with minimal changes since it's already so large! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
