openinx commented on issue #3115: URL: https://github.com/apache/iceberg/issues/3115#issuecomment-938280504
Linked to the [mail list discussion](https://lists.apache.org/x/thread.html/r6adba306a593c6c036ed8cce9af9fb4cde831e2b11e744375ca341ea@%3Cdev.iceberg.apache.org%3E): > Thanks for the context on the Flink side! I think it sounds reasonable to keep up to date with the latest supported Flink version. If we want, we could later go with something similar to what we do for Spark but we’ll see how it goes and what the Flink community needs. We should probably add a section to our Flink docs that explains and links to Flink’s support policy and has a table of Iceberg versions that work with Flink versions. (We should probably have the same table for Spark, too!) > > For Spark, I’m also leaning toward the modified option 3 where we keep all of the code in the main repository but only build with one module at a time by default. It makes sense to switch based on modules — rather than selecting src paths within a module — so that it is easy to run a build with all modules if you choose to — for example, when building release binaries. > > The reason I think we should go with option 3 is for testing. If we have a single repo with api, core, etc. and spark then changes to the common modules can be tested by CI actions. Updates to individual Spark modules would be completely independent. There is a slight inconvenience that when an API used by Spark changes, the author would still need to fix multiple Spark versions. But the trade-off is that with a separate repository like option 2, changes that break Spark versions are not caught and then the Spark repository’s CI ends up failing on completely unrelated changes. That would be a major pain, felt by everyone contributing to the Spark integration, so I think option 3 is the best path forward. > > It sounds like we probably have some agreement now, but please speak up if you think another option would be better. > > The next step is to prototype the build changes to test out option 3. Or if you prefer option 2, then prototype those changes as well. I think that Anton is planning to do this, but if you have time and the desire to do it please reach out and coordinate with us! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
