I think it’s generally dangerous and error-prone to try to support two versions of the same library in the same build, in the same published artifacts. This is the stance that Baseline<https://github.com/palantir/gradle-baseline> + Gradle Consistent Versions<https://github.com/palantir/gradle-consistent-versions> takes. Gradle Consistent Versions is specifically opinionated towards building against one version of a library across all modules in the build.
I would think that branching would be the best way to build and publish against multiple versions of a dependency. -Matt Cheah From: Saisai Shao <sai.sai.s...@gmail.com> Reply-To: "dev@iceberg.apache.org" <dev@iceberg.apache.org> Date: Tuesday, March 3, 2020 at 5:45 PM To: Iceberg Dev List <dev@iceberg.apache.org> Cc: Ryan Blue <rb...@netflix.com> Subject: Re: [Discuss] Merge spark-3 branch into master I didn't realized that Gradle cannot support two different versions in one build. I think I did such things for Livy to build scala 2.10 and 2.11 jars simultaneously with Maven. I'm not so familiar with Gradle thing, I can take a shot to see if there's some hacky ways to make it work. Besides, are we saying that we will move to spark-3 support after 0.8 release in the master branch to replace Spark-2, or we maintain two branches for both spark-2 and spark-3 and make two releases? From my understanding, the adoption of spark-3 may not be so fast, and there still has lots users who stick on spark-2. Ideally, it might be better to support two versions in a near future. Thanks Saisai Mass Dosage <massdos...@gmail.com<mailto:massdos...@gmail.com>> 于2020年3月4日周三 上午1:33写道： +1 for a 0.8.0 release with Spark 2.4 and then move on for Spark 3.0 when it's ready. On Tue, 3 Mar 2020 at 16:32, Ryan Blue <rb...@netflix.com.invalid> wrote: Thanks for bringing this up, Saisai. I tried to do this a couple of months ago, but ran into a problem with dependency locks. I couldn't get two different versions of Spark packages in the build with baseline, but maybe I was missing something. If you can get it working, I think it's a great idea to get this into master. Otherwise, I was thinking about proposing an 0.8.0 release in the next month or so based on Spark 2.4. Then we could merge the branch into master and do another release for Spark 3.0 when it's ready. rb On Tue, Mar 3, 2020 at 6:07 AM Saisai Shao <sai.sai.s...@gmail.com<mailto:sai.sai.s...@gmail.com>> wrote: Hi team, I was thinking of merging spark-3 branch into master, also per the discussion before we could make spark-2 and spark-3 coexisted into 2 different sub-modules. With this, one build could generate both spark-2 and spark-3 runtime jars, user could pick either at preference. One concern is that they share lots of common code in read/write path, this will increase the maintenance overhead to keep consistency of two copies. So I'd like to hear your thoughts, any suggestions on it? Thanks Saisai -- Ryan Blue Software Engineer Netflix
