Wing, sorry, my earlier message probably misled you. I was speaking my personal opinion on Flink version support.
On Tue, Sep 28, 2021 at 8:03 PM Wing Yew Poon <wyp...@cloudera.com.invalid> wrote: > Hi OpenInx, > I'm sorry I misunderstood the thinking of the Flink community. Thanks for > the clarification. > - Wing Yew > > > On Tue, Sep 28, 2021 at 7:15 PM OpenInx <open...@gmail.com> wrote: > >> Hi Wing >> >> As we discussed above, we community prefer to choose option.2 or >> option.3. So in fact, when we planned to upgrade the flink version from >> 1.12 to 1.13, we are doing our best to guarantee the master iceberg repo >> could work fine for both flink1.12 & flink1.13. More context please see >> [1], [2], [3] >> >> [1] https://github.com/apache/iceberg/pull/3116 >> [2] https://github.com/apache/iceberg/issues/3183 >> [3] >> https://lists.apache.org/x/thread.html/ra438e89eeec2d4623a32822e21739c8f2229505522d73d1034e34198@%3Cdev.flink.apache.org%3E >> >> >> On Wed, Sep 29, 2021 at 5:27 AM Wing Yew Poon <wyp...@cloudera.com.invalid> >> wrote: >> >>> In the last community sync, we spent a little time on this topic. For >>> Spark support, there are currently two options under consideration: >>> >>> Option 2: Separate repo for the Spark support. Use branches for >>> supporting different Spark versions. Main branch for the latest Spark >>> version (3.2 to begin with). >>> Tooling needs to be built for producing regular snapshots of core >>> Iceberg in a consumable way for this repo. Unclear if commits to core >>> Iceberg will be tested pre-commit against Spark support; my impression is >>> that they will not be, and the Spark support build can be broken by changes >>> to core. >>> >>> A variant of option 3 (which we will simply call Option 3 going >>> forward): Single repo, separate module (subdirectory) for each Spark >>> version to be supported. Code duplication in each Spark module (no attempt >>> to refactor out common code). Each module built against the specific >>> version of Spark to be supported, producing a runtime jar built against >>> that version. CI will test all modules. Support can be provided for only >>> building the modules a developer cares about. >>> >>> More input was sought and people are encouraged to voice their >>> preference. >>> I lean towards Option 3. >>> >>> - Wing Yew >>> >>> ps. In the sync, as Steven Wu wrote, the question was raised if the same >>> multi-version support strategy can be adopted across engines. Based on what >>> Steven wrote, currently the Flink developer community's bandwidth makes >>> supporting only a single Flink version (and focusing resources on >>> developing new features on that version) the preferred choice. If so, then >>> no multi-version support strategy for Flink is needed at this time. >>> >>> >>> On Thu, Sep 23, 2021 at 5:26 PM Steven Wu <stevenz...@gmail.com> wrote: >>> >>>> During the sync meeting, people talked about if and how we can have the >>>> same version support model across engines like Flink and Spark. I can >>>> provide some input from the Flink side. >>>> >>>> Flink only supports two minor versions. E.g., right now Flink 1.13 is >>>> the latest released version. That means only Flink 1.12 and 1.13 are >>>> supported. Feature changes or bug fixes will only be backported to 1.12 and >>>> 1.13, unless it is a serious bug (like security). With that context, >>>> personally I like option 1 (with one actively supported Flink version in >>>> master branch) for the iceberg-flink module. >>>> >>>> We discussed the idea of supporting multiple Flink versions via shm >>>> layer and multiple modules. While it may be a little better to support >>>> multiple Flink versions, I don't know if there is enough support and >>>> resources from the community to pull it off. Also the ongoing maintenance >>>> burden for each minor version release from Flink, which happens roughly >>>> every 4 months. >>>> >>>> >>>> On Thu, Sep 16, 2021 at 10:25 PM Peter Vary <pv...@cloudera.com.invalid> >>>> wrote: >>>> >>>>> Since you mentioned Hive, I chime in with what we do there. You might >>>>> find it useful: >>>>> - metastore module - only small differences - DynConstructor solves >>>>> for us >>>>> - mr module - some bigger differences, but still manageable for Hive >>>>> 2-3. Need some new classes, but most of the code is reused - extra module >>>>> for Hive 3. For Hive 4 we use a different repo as we moved to the Hive >>>>> codebase. >>>>> >>>>> My thoughts based on the above experience: >>>>> - Keeping Hive 4 and Hive 2-3 code in sync is a pain. We constantly >>>>> have problems with backporting changes between repos and we are slacking >>>>> behind which hurts both projects >>>>> - Hive 2-3 model is working better by forcing us to keep the things in >>>>> sync, but with serious differences in the Hive project it still doesn't >>>>> seem like a viable option. >>>>> >>>>> So I think the question is: How stable is the Spark code we are >>>>> integrating to. If I is fairly stable then we are better off with a "one >>>>> repo multiple modules" approach and we should consider the multirepo only >>>>> if the differences become prohibitive. >>>>> >>>>> Thanks, Peter >>>>> >>>>> On Fri, 17 Sep 2021, 02:21 Anton Okolnychyi, >>>>> <aokolnyc...@apple.com.invalid> wrote: >>>>> >>>>>> Okay, looks like there is consensus around supporting multiple Spark >>>>>> versions at the same time. There are folks who mentioned this on this >>>>>> thread and there were folks who brought this up during the sync. >>>>>> >>>>>> Let’s think through Option 2 and 3 in more detail then. >>>>>> >>>>>> Option 2 >>>>>> >>>>>> In Option 2, there will be a separate repo. I believe the master >>>>>> branch will soon point to Spark 3.2 (the most recent supported version). >>>>>> The main development will happen there and the artifact version will be >>>>>> 0.1.0. I also suppose there will be 0.1.x-spark-2 and 0.1.x-spark-3.1 >>>>>> branches where we will cherry-pick applicable changes. Once we are ready >>>>>> to >>>>>> release 0.1.0 Spark integration, we will create 0.1.x-spark-3.2 and cut 3 >>>>>> releases: Spark 2.4, Spark 3.1, Spark 3.2. After that, we will bump the >>>>>> version in master to 0.2.0 and create new 0.2.x-spark-2 and >>>>>> 0.2.x-spark-3.1 >>>>>> branches for cherry-picks. >>>>>> >>>>>> I guess we will continue to shade everything in the new repo and will >>>>>> have to release every time the core is released. We will do a maintenance >>>>>> release for each supported Spark version whenever we cut a new >>>>>> maintenance Iceberg >>>>>> release or need to fix any bugs in the Spark integration. >>>>>> Under this model, we will probably need nightly snapshots (or on each >>>>>> commit) for the core format and the Spark integration will depend on >>>>>> snapshots until we are ready to release. >>>>>> >>>>>> Overall, I think this option gives us very simple builds and provides >>>>>> best separation. It will keep the main repo clean. The main downside is >>>>>> that we will have to split a Spark feature into two PRs: one against the >>>>>> core and one against the Spark integration. Certain changes in core can >>>>>> also break the Spark integration too and will require adaptations. >>>>>> >>>>>> Ryan, I am not sure I fully understood the testing part. How will we >>>>>> be able to test the Spark integration in the main repo if certain changes >>>>>> in core may break the Spark integration and require changes there? Will >>>>>> we >>>>>> try to prohibit such changes? >>>>>> >>>>>> Option 3 (modified) >>>>>> >>>>>> If I get correctly, the modified Option 3 sounds very close to >>>>>> the initially suggested approach by Imran but with code duplication >>>>>> instead >>>>>> of extra refactoring and introducing new common modules. >>>>>> >>>>>> Jack, are you suggesting we test only a single Spark version at a >>>>>> time? Or do we expect to test all versions? Will there be any difference >>>>>> compared to just having a module per version? I did not fully >>>>>> understand. >>>>>> >>>>>> My worry with this approach is that our build will be very >>>>>> complicated and we will still have a lot of Spark-related modules in the >>>>>> main repo. Once people start using Flink and Hive more, will we have to >>>>>> do >>>>>> the same? >>>>>> >>>>>> - Anton >>>>>> >>>>>> >>>>>> >>>>>> On 16 Sep 2021, at 08:11, Ryan Blue <b...@tabular.io> wrote: >>>>>> >>>>>> I'd support the option that Jack suggests if we can set a few >>>>>> expectations for keeping it clean. >>>>>> >>>>>> First, I'd like to avoid refactoring code to share it across Spark >>>>>> versions -- that introduces risk because we're relying on compiling >>>>>> against >>>>>> one version and running in another and both Spark and Scala change >>>>>> rapidly. >>>>>> A big benefit of options 1 and 2 is that we mostly focus on only one >>>>>> Spark >>>>>> version. I think we should duplicate code rather than spend time >>>>>> refactoring to rely on binary compatibility. I propose we start each new >>>>>> Spark version by copying the last one and updating it. And we should >>>>>> build >>>>>> just the latest supported version by default. >>>>>> >>>>>> The drawback to having everything in a single repo is that we >>>>>> wouldn't be able to cherry-pick changes across Spark versions/branches, >>>>>> but >>>>>> I think Jack is right that having a single build is better. >>>>>> >>>>>> Second, we should make CI faster by running the Spark builds in >>>>>> parallel. It sounds like this is what would happen anyway, with a >>>>>> property >>>>>> that selects the Spark version that you want to build against. >>>>>> >>>>>> Overall, this new suggestion sounds like a promising way forward. >>>>>> >>>>>> Ryan >>>>>> >>>>>> On Wed, Sep 15, 2021 at 11:46 PM Jack Ye <yezhao...@gmail.com> wrote: >>>>>> >>>>>>> I think in Ryan's proposal we will create a ton of modules anyway, >>>>>>> as Wing listed we are just using git branch as an additional dimension, >>>>>>> but >>>>>>> my understanding is that you will still have 1 core, 1 extension, 1 >>>>>>> runtime >>>>>>> artifact published for each Spark version in either approach. >>>>>>> >>>>>>> In that case, this is just brainstorming, I wonder if we can explore >>>>>>> a modified option 3 that flattens all the versions in each Spark branch >>>>>>> in >>>>>>> option 2 into master. The repository structure would look something >>>>>>> like: >>>>>>> >>>>>>> iceberg/api/... >>>>>>> /bundled-guava/... >>>>>>> /core/... >>>>>>> ... >>>>>>> /spark/2.4/core/... >>>>>>> /extension/... >>>>>>> /runtime/... >>>>>>> /3.1/core/... >>>>>>> /extension/... >>>>>>> /runtime/... >>>>>>> >>>>>>> The gradle build script in the root is configured to build against >>>>>>> the latest version of Spark by default, unless otherwise specified by >>>>>>> the >>>>>>> user. >>>>>>> >>>>>>> Intellij can also be configured to only index files of specific >>>>>>> versions based on the same config used in build. >>>>>>> >>>>>>> In this way, I imagine the CI setup to be much easier to do things >>>>>>> like testing version compatibility for a feature or running only a >>>>>>> specific subset of Spark version builds based on the Spark version >>>>>>> directories touched. >>>>>>> >>>>>>> And the biggest benefit is that we don't have the same difficulty as >>>>>>> option 2 of developing a feature when it's both in core and Spark. >>>>>>> >>>>>>> We can then develop a mechanism to vote to stop support of certain >>>>>>> versions, and archive the corresponding directory to avoid accumulating >>>>>>> too >>>>>>> many versions in the long term. >>>>>>> >>>>>>> -Jack Ye >>>>>>> >>>>>>> >>>>>>> On Wed, Sep 15, 2021 at 4:17 PM Ryan Blue <b...@tabular.io> wrote: >>>>>>> >>>>>>>> Sorry, I was thinking about CI integration between Iceberg Java and >>>>>>>> Iceberg Spark, I just didn't mention it and I see how that's a big >>>>>>>> thing to >>>>>>>> leave out! >>>>>>>> >>>>>>>> I would definitely want to test the projects together. One thing we >>>>>>>> could do is have a nightly build like Russell suggests. I'm also >>>>>>>> wondering >>>>>>>> if we could have some tighter integration where the Iceberg Spark >>>>>>>> build can >>>>>>>> be included in the Iceberg Java build using properties. Maybe the >>>>>>>> github >>>>>>>> action could checkout Iceberg, then checkout the Spark integration's >>>>>>>> latest >>>>>>>> branch, and then run the gradle build with a property that makes Spark >>>>>>>> a >>>>>>>> subproject in the build. That way we can continue to have Spark CI run >>>>>>>> regularly. >>>>>>>> >>>>>>>> On Wed, Sep 15, 2021 at 3:08 PM Russell Spitzer < >>>>>>>> russell.spit...@gmail.com> wrote: >>>>>>>> >>>>>>>>> I agree that Option 2 is considerably more difficult for >>>>>>>>> development when core API changes need to be picked up by the external >>>>>>>>> Spark module. I also think a monthly release would probably still be >>>>>>>>> prohibitive to actually implementing new features that appear in the >>>>>>>>> API, I >>>>>>>>> would hope we have a much faster process or maybe just have snapshot >>>>>>>>> artifacts published nightly? >>>>>>>>> >>>>>>>>> On Sep 15, 2021, at 4:46 PM, Wing Yew Poon < >>>>>>>>> wyp...@cloudera.com.INVALID> wrote: >>>>>>>>> >>>>>>>>> IIUC, Option 2 is to move the Spark support for Iceberg into a >>>>>>>>> separate repo (subproject of Iceberg). Would we have branches such as >>>>>>>>> 0.13-2.4, 0.13-3.0, 0.13-3.1, and 0.13-3.2? For features that can be >>>>>>>>> supported in all versions or all Spark 3 versions, then we would need >>>>>>>>> to >>>>>>>>> commit the changes to all applicable branches. Basically we are >>>>>>>>> trading >>>>>>>>> more work to commit to multiple branches for simplified build and CI >>>>>>>>> time per branch, which might be an acceptable trade-off. However, the >>>>>>>>> biggest downside is that changes may need to be made in core Iceberg >>>>>>>>> as >>>>>>>>> well as in the engine (in this case Spark) support, and we need to >>>>>>>>> wait for >>>>>>>>> a release of core Iceberg to consume the changes in the subproject. >>>>>>>>> In this >>>>>>>>> case, maybe we should have a monthly release of core Iceberg (no >>>>>>>>> matter how >>>>>>>>> many changes go in, as long as it is non-zero) so that the subproject >>>>>>>>> can >>>>>>>>> consume changes fairly quickly? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Sep 15, 2021 at 2:09 PM Ryan Blue <b...@tabular.io> wrote: >>>>>>>>> >>>>>>>>>> Thanks for bringing this up, Anton. I’m glad that we have the set >>>>>>>>>> of potential solutions well defined. >>>>>>>>>> >>>>>>>>>> Looks like the next step is to decide whether we want to require >>>>>>>>>> people to update Spark versions to pick up newer versions of >>>>>>>>>> Iceberg. If we >>>>>>>>>> choose to make people upgrade, then option 1 is clearly the best >>>>>>>>>> choice. >>>>>>>>>> >>>>>>>>>> I don’t think that we should make updating Spark a requirement. >>>>>>>>>> Many of the things that we’re working on are orthogonal to Spark >>>>>>>>>> versions, >>>>>>>>>> like table maintenance actions, secondary indexes, the 1.0 API, >>>>>>>>>> views, ORC >>>>>>>>>> delete files, new storage implementations, etc. Upgrading Spark is >>>>>>>>>> time >>>>>>>>>> consuming and untrusted in my experience, so I think we would be >>>>>>>>>> setting up >>>>>>>>>> an unnecessary trade-off between spending lots of time to upgrade >>>>>>>>>> Spark and >>>>>>>>>> picking up new Iceberg features. >>>>>>>>>> >>>>>>>>>> Another way of thinking about this is that if we went with option >>>>>>>>>> 1, then we could port bug fixes into 0.12.x. But there are many >>>>>>>>>> things that >>>>>>>>>> wouldn’t fit this model, like adding a FileIO implementation for >>>>>>>>>> ADLS. So >>>>>>>>>> some people in the community would have to maintain branches of newer >>>>>>>>>> Iceberg versions with older versions of Spark outside of the main >>>>>>>>>> Iceberg >>>>>>>>>> project — that defeats the purpose of simplifying things with option >>>>>>>>>> 1 >>>>>>>>>> because we would then have more people maintaining the same 0.13.x >>>>>>>>>> with >>>>>>>>>> Spark 3.1 branch. (This reminds me of the Spark community, where we >>>>>>>>>> wanted >>>>>>>>>> to release a 2.5 line with DSv2 backported, but the community >>>>>>>>>> decided not >>>>>>>>>> to so we built similar 2.4+DSv2 branches at Netflix, Tencent, Apple, >>>>>>>>>> etc.) >>>>>>>>>> >>>>>>>>>> If the community is going to do the work anyway — and I think >>>>>>>>>> some of us would — we should make it possible to share that work. >>>>>>>>>> That’s >>>>>>>>>> why I don’t think that we should go with option 1. >>>>>>>>>> >>>>>>>>>> If we don’t go with option 1, then the choice is how to maintain >>>>>>>>>> multiple Spark versions. I think that the way we’re doing it right >>>>>>>>>> now is >>>>>>>>>> not something we want to continue. >>>>>>>>>> >>>>>>>>>> Using multiple modules (option 3) is concerning to me because of >>>>>>>>>> the changes in Spark. We currently structure the library to share as >>>>>>>>>> much >>>>>>>>>> code as possible. But that means compiling against different Spark >>>>>>>>>> versions >>>>>>>>>> and relying on binary compatibility and reflection in some cases. To >>>>>>>>>> me, >>>>>>>>>> this seems unmaintainable in the long run because it requires >>>>>>>>>> refactoring >>>>>>>>>> common classes and spending a lot of time deduplicating code. It also >>>>>>>>>> creates a ton of modules, at least one common module, then a module >>>>>>>>>> per >>>>>>>>>> version, then an extensions module per version, and finally a runtime >>>>>>>>>> module per version. That’s 3 modules per Spark version, plus any new >>>>>>>>>> common >>>>>>>>>> modules. And each module needs to be tested, which is making our CI >>>>>>>>>> take a >>>>>>>>>> really long time. We also don’t support multiple Scala versions, >>>>>>>>>> which is >>>>>>>>>> another gap that will require even more modules and tests. >>>>>>>>>> >>>>>>>>>> I like option 2 because it would allow us to compile against a >>>>>>>>>> single version of Spark (which will be much more reliable). It would >>>>>>>>>> give >>>>>>>>>> us an opportunity to support different Scala versions. It avoids the >>>>>>>>>> need >>>>>>>>>> to refactor to share code and allows people to focus on a single >>>>>>>>>> version of >>>>>>>>>> Spark, while also creating a way for people to maintain and update >>>>>>>>>> the >>>>>>>>>> older versions with newer Iceberg releases. I don’t think that this >>>>>>>>>> would >>>>>>>>>> slow down development. I think it would actually speed it up because >>>>>>>>>> we’d >>>>>>>>>> be spending less time trying to make multiple versions work in the >>>>>>>>>> same >>>>>>>>>> build. And anyone in favor of option 1 would basically get option 1: >>>>>>>>>> you >>>>>>>>>> don’t have to care about branches for older Spark versions. >>>>>>>>>> >>>>>>>>>> Jack makes a good point about wanting to keep code in a single >>>>>>>>>> repository, but I think that the need to manage more version >>>>>>>>>> combinations >>>>>>>>>> overrides this concern. It’s easier to make this decision in python >>>>>>>>>> because >>>>>>>>>> we’re not trying to depend on two projects that change relatively >>>>>>>>>> quickly. >>>>>>>>>> We’re just trying to build a library. >>>>>>>>>> >>>>>>>>>> Ryan >>>>>>>>>> >>>>>>>>>> On Wed, Sep 15, 2021 at 2:58 AM OpenInx <open...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks for bringing this up, Anton. >>>>>>>>>>> >>>>>>>>>>> Everyone has great pros/cons to support their preferences. >>>>>>>>>>> Before giving my preference, let me raise one question: what's >>>>>>>>>>> the top >>>>>>>>>>> priority thing for apache iceberg project at this point in time ? >>>>>>>>>>> This >>>>>>>>>>> question will help us to answer the following question: Should we >>>>>>>>>>> support >>>>>>>>>>> more engine versions more robustly or be a bit more aggressive and >>>>>>>>>>> concentrate on getting the new features that users need most in >>>>>>>>>>> order to >>>>>>>>>>> keep the project more competitive ? >>>>>>>>>>> >>>>>>>>>>> If people watch the apache iceberg project and check the issues >>>>>>>>>>> & PR frequently, I guess more than 90% people will answer the >>>>>>>>>>> priority >>>>>>>>>>> question: There is no doubt for making the whole v2 story to be >>>>>>>>>>> production-ready. The current roadmap discussion also proofs the >>>>>>>>>>> thing : >>>>>>>>>>> https://lists.apache.org/x/thread.html/r84e80216c259c81f824c6971504c321cd8c785774c489d52d4fc123f@%3Cdev.iceberg.apache.org%3E >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>>> In order to ensure the highest priority at this point in time, I >>>>>>>>>>> will prefer option-1 to reduce the cost of engine maintenance, so >>>>>>>>>>> as to >>>>>>>>>>> free up resources to make v2 production-ready. >>>>>>>>>>> >>>>>>>>>>> On Wed, Sep 15, 2021 at 3:00 PM Saisai Shao < >>>>>>>>>>> sai.sai.s...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> From Dev's point, it has less burden to always support the >>>>>>>>>>>> latest version of Spark (for example). But from user's point, >>>>>>>>>>>> especially for us who maintain Spark internally, it is not easy to >>>>>>>>>>>> upgrade >>>>>>>>>>>> the Spark version for the first time (since we have many >>>>>>>>>>>> customizations >>>>>>>>>>>> internally), and we're still promoting to upgrade to 3.1.2. If the >>>>>>>>>>>> community ditches the support of old version of Spark3, users have >>>>>>>>>>>> to >>>>>>>>>>>> maintain it themselves unavoidably. >>>>>>>>>>>> >>>>>>>>>>>> So I'm inclined to make this support in community, not by users >>>>>>>>>>>> themselves, as for Option 2 or 3, I'm fine with either. And to >>>>>>>>>>>> relieve the >>>>>>>>>>>> burden, we could support limited versions of Spark (for example 2 >>>>>>>>>>>> versions). >>>>>>>>>>>> >>>>>>>>>>>> Just my two cents. >>>>>>>>>>>> >>>>>>>>>>>> -Saisai >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Jack Ye <yezhao...@gmail.com> 于2021年9月15日周三 下午1:35写道： >>>>>>>>>>>> >>>>>>>>>>>>> Hi Wing Yew, >>>>>>>>>>>>> >>>>>>>>>>>>> I think 2.4 is a different story, we will continue to support >>>>>>>>>>>>> Spark 2.4, but as you can see it will continue to have very >>>>>>>>>>>>> limited >>>>>>>>>>>>> functionalities comparing to Spark 3. I believe we discussed >>>>>>>>>>>>> about option 3 >>>>>>>>>>>>> when we were doing Spark 3.0 to 3.1 upgrade. Recently we are >>>>>>>>>>>>> seeing the >>>>>>>>>>>>> same issue for Flink 1.11, 1.12 and 1.13 as well. I feel we need a >>>>>>>>>>>>> consistent strategy around this, let's take this chance to make a >>>>>>>>>>>>> good >>>>>>>>>>>>> community guideline for all future engine versions, especially >>>>>>>>>>>>> for Spark, >>>>>>>>>>>>> Flink and Hive that are in the same repository. >>>>>>>>>>>>> >>>>>>>>>>>>> I can totally understand your point of view Wing, in fact, >>>>>>>>>>>>> speaking from the perspective of AWS EMR, we have to support over >>>>>>>>>>>>> 40 >>>>>>>>>>>>> versions of the software because there are people who are still >>>>>>>>>>>>> using Spark >>>>>>>>>>>>> 1.4, believe it or not. After all, keep backporting changes will >>>>>>>>>>>>> become a >>>>>>>>>>>>> liability not only on the user side, but also on the service >>>>>>>>>>>>> provider side, >>>>>>>>>>>>> so I believe it's not a bad practice to push for user upgrade, as >>>>>>>>>>>>> it will >>>>>>>>>>>>> make the life of both parties easier in the end. New feature is >>>>>>>>>>>>> definitely >>>>>>>>>>>>> one of the best incentives to promote an upgrade on user side. >>>>>>>>>>>>> >>>>>>>>>>>>> I think the biggest issue of option 3 is about its >>>>>>>>>>>>> scalability, because we will have an unbounded list of packages >>>>>>>>>>>>> to add and >>>>>>>>>>>>> compile in the future, and we probably cannot drop support of >>>>>>>>>>>>> that package >>>>>>>>>>>>> once created. If we go with option 1, I think we can still >>>>>>>>>>>>> publish a few >>>>>>>>>>>>> patch versions for old Iceberg releases, and committers can >>>>>>>>>>>>> control the >>>>>>>>>>>>> amount of patch versions to guard people from abusing the power of >>>>>>>>>>>>> patching. I see this as a consistent strategy also for Flink and >>>>>>>>>>>>> Hive. With >>>>>>>>>>>>> this strategy, we can truly have a compatibility matrix for >>>>>>>>>>>>> engine versions >>>>>>>>>>>>> against Iceberg versions. >>>>>>>>>>>>> >>>>>>>>>>>>> -Jack >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Sep 14, 2021 at 10:00 PM Wing Yew Poon < >>>>>>>>>>>>> wyp...@cloudera.com.invalid> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I understand and sympathize with the desire to use new DSv2 >>>>>>>>>>>>>> features in Spark 3.2. I agree that Option 1 is the easiest for >>>>>>>>>>>>>> developers, >>>>>>>>>>>>>> but I don't think it considers the interests of users. I do not >>>>>>>>>>>>>> think that >>>>>>>>>>>>>> most users will upgrade to Spark 3.2 as soon as it is released. >>>>>>>>>>>>>> It is a >>>>>>>>>>>>>> "minor version" upgrade in name from 3.1 (or from 3.0), but I >>>>>>>>>>>>>> think we all >>>>>>>>>>>>>> know that it is not a minor upgrade. There are a lot of changes >>>>>>>>>>>>>> from 3.0 to >>>>>>>>>>>>>> 3.1 and from 3.1 to 3.2. I think there are even a lot of users >>>>>>>>>>>>>> running >>>>>>>>>>>>>> Spark 2.4 and not even on Spark 3 yet. Do we also plan to stop >>>>>>>>>>>>>> supporting >>>>>>>>>>>>>> Spark 2.4? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please correct me if I'm mistaken, but the folks who have >>>>>>>>>>>>>> spoken out in favor of Option 1 all work for the same >>>>>>>>>>>>>> organization, don't >>>>>>>>>>>>>> they? And they don't have a problem with making their users, all >>>>>>>>>>>>>> internal, >>>>>>>>>>>>>> simply upgrade to Spark 3.2, do they? (Or they are already >>>>>>>>>>>>>> running an >>>>>>>>>>>>>> internal fork that is close to 3.2.) >>>>>>>>>>>>>> >>>>>>>>>>>>>> I work for an organization with customers running different >>>>>>>>>>>>>> versions of Spark. It is true that we can backport new features >>>>>>>>>>>>>> to older >>>>>>>>>>>>>> versions if we wanted to. I suppose the people contributing to >>>>>>>>>>>>>> Iceberg work >>>>>>>>>>>>>> for some organization or other that either use Iceberg in-house, >>>>>>>>>>>>>> or provide >>>>>>>>>>>>>> software (possibly in the form of a service) to customers, and >>>>>>>>>>>>>> either way, >>>>>>>>>>>>>> the organizations have the ability to backport features and >>>>>>>>>>>>>> fixes to >>>>>>>>>>>>>> internal versions. Are there any users out there who simply use >>>>>>>>>>>>>> Apache >>>>>>>>>>>>>> Iceberg and depend on the community version? >>>>>>>>>>>>>> >>>>>>>>>>>>>> There may be features that are broadly useful that do not >>>>>>>>>>>>>> depend on Spark 3.2. Is it worth supporting them on Spark >>>>>>>>>>>>>> 3.0/3.1 (and even >>>>>>>>>>>>>> 2.4)? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am not in favor of Option 2. I do not oppose Option 1, but >>>>>>>>>>>>>> I would consider Option 3 too. Anton, you said 5 modules are >>>>>>>>>>>>>> required; what >>>>>>>>>>>>>> are the modules you're thinking of? >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Wing Yew >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Sep 14, 2021 at 5:38 PM Yufei Gu < >>>>>>>>>>>>>> flyrain...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Option 1 sounds good to me. Here are my reasons: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. Both 2 and 3 will slow down the development. Considering >>>>>>>>>>>>>>> the limited resources in the open source community, the upsides >>>>>>>>>>>>>>> of option 2 >>>>>>>>>>>>>>> and 3 are probably not worthy. >>>>>>>>>>>>>>> 2. Both 2 and 3 assume the use cases may not exist. It's >>>>>>>>>>>>>>> hard to predict anything, but even if these use cases are >>>>>>>>>>>>>>> legit, users can >>>>>>>>>>>>>>> still get the new feature by backporting it to an older version >>>>>>>>>>>>>>> in case of >>>>>>>>>>>>>>> upgrading to a newer version isn't an option. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yufei >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> `This is not a contribution` >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Sep 14, 2021 at 4:54 PM Anton Okolnychyi < >>>>>>>>>>>>>>> aokolnyc...@apple.com.invalid> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> To sum up what we have so far: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Option 1 (support just the most recent minor Spark 3 >>>>>>>>>>>>>>>> version)* >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The easiest option for us devs, forces the user to upgrade >>>>>>>>>>>>>>>> to the most recent minor Spark version to consume any new >>>>>>>>>>>>>>>> Iceberg features. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Option 2 (a separate project under Iceberg)* >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can support as many Spark versions as needed and the >>>>>>>>>>>>>>>> codebase is still separate as we can use separate branches. >>>>>>>>>>>>>>>> Impossible to consume any unreleased changes in core, may >>>>>>>>>>>>>>>> slow down the development. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Option 3 (separate modules for Spark 3.1/3.2)* >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Introduce more modules in the same project. >>>>>>>>>>>>>>>> Can consume unreleased changes but it will required at >>>>>>>>>>>>>>>> least 5 modules to support 2.4, 3.1 and 3.2, making the build >>>>>>>>>>>>>>>> and testing >>>>>>>>>>>>>>>> complicated. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Are there any users for whom upgrading the minor Spark >>>>>>>>>>>>>>>> version (e3.1 to 3.2) to consume new features is a blocker? >>>>>>>>>>>>>>>> We follow Option 1 internally at the moment but I would >>>>>>>>>>>>>>>> like to hear what other people think/need. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Anton >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 14 Sep 2021, at 09:44, Russell Spitzer < >>>>>>>>>>>>>>>> russell.spit...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think we should go for option 1. I already am not a big >>>>>>>>>>>>>>>> fan of having runtime errors for unsupported things based on >>>>>>>>>>>>>>>> versions and I >>>>>>>>>>>>>>>> don't think minor version upgrades are a large issue for >>>>>>>>>>>>>>>> users. I'm >>>>>>>>>>>>>>>> especially not looking forward to supporting interfaces that >>>>>>>>>>>>>>>> only exist in >>>>>>>>>>>>>>>> Spark 3.2 in a multiple Spark version support future. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sep 14, 2021, at 11:32 AM, Anton Okolnychyi < >>>>>>>>>>>>>>>> aokolnyc...@apple.com.INVALID> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> First of all, is option 2 a viable option? We discussed >>>>>>>>>>>>>>>> separating the python module outside of the project a few >>>>>>>>>>>>>>>> weeks ago, and >>>>>>>>>>>>>>>> decided to not do that because it's beneficial for code cross >>>>>>>>>>>>>>>> reference and >>>>>>>>>>>>>>>> more intuitive for new developers to see everything in the >>>>>>>>>>>>>>>> same repository. >>>>>>>>>>>>>>>> I would expect the same argument to also hold here. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> That’s exactly the concern I have about Option 2 at this >>>>>>>>>>>>>>>> moment. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Overall I would personally prefer us to not support all the >>>>>>>>>>>>>>>> minor versions, but instead support maybe just 2-3 latest >>>>>>>>>>>>>>>> versions in a >>>>>>>>>>>>>>>> major version. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This is when it gets a bit complicated. If we want to >>>>>>>>>>>>>>>> support both Spark 3.1 and Spark 3.2 with a single module, it >>>>>>>>>>>>>>>> means we have >>>>>>>>>>>>>>>> to compile against 3.1. The problem is that we rely on DSv2 >>>>>>>>>>>>>>>> that is being >>>>>>>>>>>>>>>> actively developed. 3.2 and 3.1 have substantial differences. >>>>>>>>>>>>>>>> On top of >>>>>>>>>>>>>>>> that, we have our extensions that are extremely low-level and >>>>>>>>>>>>>>>> may break not >>>>>>>>>>>>>>>> only between minor versions but also between patch releases. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> f there are some features requiring a newer version, it >>>>>>>>>>>>>>>> makes sense to move that newer version in master. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Internally, we don’t deliver new features to older Spark >>>>>>>>>>>>>>>> versions as it requires a lot of effort to port things. >>>>>>>>>>>>>>>> Personally, I don’t >>>>>>>>>>>>>>>> think it is too bad to require users to upgrade if they want >>>>>>>>>>>>>>>> new features. >>>>>>>>>>>>>>>> At the same time, there are valid concerns with this approach >>>>>>>>>>>>>>>> too that we >>>>>>>>>>>>>>>> mentioned during the sync. For example, certain new features >>>>>>>>>>>>>>>> would also >>>>>>>>>>>>>>>> work fine with older Spark versions. I generally agree with >>>>>>>>>>>>>>>> that and that >>>>>>>>>>>>>>>> not supporting recent versions is not ideal. However, I want >>>>>>>>>>>>>>>> to find a >>>>>>>>>>>>>>>> balance between the complexity on our side and ease of use for >>>>>>>>>>>>>>>> the users. >>>>>>>>>>>>>>>> Ideally, supporting a few recent versions would be sufficient >>>>>>>>>>>>>>>> but our Spark >>>>>>>>>>>>>>>> integration is too low-level to do that with a single module. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 13 Sep 2021, at 20:53, Jack Ye <yezhao...@gmail.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> First of all, is option 2 a viable option? We discussed >>>>>>>>>>>>>>>> separating the python module outside of the project a few >>>>>>>>>>>>>>>> weeks ago, and >>>>>>>>>>>>>>>> decided to not do that because it's beneficial for code cross >>>>>>>>>>>>>>>> reference and >>>>>>>>>>>>>>>> more intuitive for new developers to see everything in the >>>>>>>>>>>>>>>> same repository. >>>>>>>>>>>>>>>> I would expect the same argument to also hold here. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Overall I would personally prefer us to not support all the >>>>>>>>>>>>>>>> minor versions, but instead support maybe just 2-3 latest >>>>>>>>>>>>>>>> versions in a >>>>>>>>>>>>>>>> major version. This avoids the problem that some users are >>>>>>>>>>>>>>>> unwilling to >>>>>>>>>>>>>>>> move to a newer version and keep patching old Spark version >>>>>>>>>>>>>>>> branches. If >>>>>>>>>>>>>>>> there are some features requiring a newer version, it makes >>>>>>>>>>>>>>>> sense to move >>>>>>>>>>>>>>>> that newer version in master. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In addition, because currently Spark is considered the most >>>>>>>>>>>>>>>> feature-complete reference implementation compared to all >>>>>>>>>>>>>>>> other engines, I >>>>>>>>>>>>>>>> think we should not add artificial barriers that would slow >>>>>>>>>>>>>>>> down its >>>>>>>>>>>>>>>> development speed. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So my thinking is closer to option 1. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>> Jack Ye >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Sep 13, 2021 at 7:39 PM Anton Okolnychyi < >>>>>>>>>>>>>>>> aokolnyc...@apple.com.invalid> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hey folks, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I want to discuss our Spark version support strategy. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> So far, we have tried to support both 3.0 and 3.1. It is >>>>>>>>>>>>>>>>> great to support older versions but because we compile >>>>>>>>>>>>>>>>> against 3.0, we >>>>>>>>>>>>>>>>> cannot use any Spark features that are offered in newer >>>>>>>>>>>>>>>>> versions. >>>>>>>>>>>>>>>>> Spark 3.2 is just around the corner and it brings a lot of >>>>>>>>>>>>>>>>> important features such dynamic filtering for v2 tables, >>>>>>>>>>>>>>>>> required >>>>>>>>>>>>>>>>> distribution and ordering for writes, etc. These features are >>>>>>>>>>>>>>>>> too important >>>>>>>>>>>>>>>>> to ignore them. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Apart from that, I have an end-to-end prototype for >>>>>>>>>>>>>>>>> merge-on-read with Spark that actually leverages some of the >>>>>>>>>>>>>>>>> 3.2 features. >>>>>>>>>>>>>>>>> I’ll be implementing all new Spark DSv2 APIs for us >>>>>>>>>>>>>>>>> internally and would >>>>>>>>>>>>>>>>> love to share that with the rest of the community. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I see two options to move forward: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Option 1 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Migrate to Spark 3.2 in master, maintain 0.12 for a while >>>>>>>>>>>>>>>>> by releasing minor versions with bug fixes. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Pros: almost no changes to the build configuration, no >>>>>>>>>>>>>>>>> extra work on our side as just a single Spark version is >>>>>>>>>>>>>>>>> actively >>>>>>>>>>>>>>>>> maintained. >>>>>>>>>>>>>>>>> Cons: some new features that we will be adding to master >>>>>>>>>>>>>>>>> could also work with older Spark versions but all 0.12 >>>>>>>>>>>>>>>>> releases will only >>>>>>>>>>>>>>>>> contain bug fixes. Therefore, users will be forced to migrate >>>>>>>>>>>>>>>>> to Spark 3.2 >>>>>>>>>>>>>>>>> to consume any new Spark or format features. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Option 2 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Move our Spark integration into a separate project and >>>>>>>>>>>>>>>>> introduce branches for 3.0, 3.1 and 3.2. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Pros: decouples the format version from Spark, we can >>>>>>>>>>>>>>>>> support as many Spark versions as needed. >>>>>>>>>>>>>>>>> Cons: more work initially to set everything up, more work >>>>>>>>>>>>>>>>> to release, will need a new release of the core format to >>>>>>>>>>>>>>>>> consume any >>>>>>>>>>>>>>>>> changes in the Spark integration. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Overall, I think option 2 seems better for the user but my >>>>>>>>>>>>>>>>> main worry is that we will have to release the format more >>>>>>>>>>>>>>>>> frequently >>>>>>>>>>>>>>>>> (which is a good thing but requires more work and time) and >>>>>>>>>>>>>>>>> the overall >>>>>>>>>>>>>>>>> Spark development may be slower. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I’d love to hear what everybody thinks about this matter. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Anton >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Ryan Blue >>>>>>>>>> Tabular >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Ryan Blue >>>>>>>> Tabular >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Tabular >>>>>> >>>>>> >>>>>>
