I also like option 2 (allowing differing dependencies for runners) better. With the current situation this would mean splitting PreCommit_Java_MavenInstall (and possibly also PreCommit_Python_MavenInstall and PreCommit_Go_MavenInstall) into separate jobs. For my goals splitting into one job for "direct-runner,dataflow-runner,spark-runner,apex-runner" and one for "flank-runner" would be enough so we should probably go with that until we have the "custom make" solution.
What do you think? @Jason For pulling this off I would copy the groovy files in .test-infra and change the --activate-profiles line, right? Are there still manual steps required for "re-seeding" the jobs? > On 9. Oct 2017, at 18:06, Kenneth Knowles <[email protected]> wrote: > > +1 to the goal, and replying inline on details. > > On Mon, Oct 9, 2017 at 8:06 AM, Aljoscha Krettek <[email protected]> > wrote: > >> >> - We want to update the Flink dependencies to _2.11 dependencies because >> 2.10 is quite outdated > > - This doesn't work well because some modules (examples, for example) >> depend on all Runners and at least the Spark Runner has _2.10 dependencies >> > > Let's expedite making this possible, and circle back to getting the build > to an ideal state after unblocking this very reasonable change. > > It is not reasonable for any runner dependencies in Beam to be coupled, > certainly not Scala version. We've been monolithic so far because it is > easier to manage, but it was never a long term solution. It will mean that > the examples cannot have -P flink-runner and -P spark-runner at the same > time. But my position is that we should never expect to be able to use two > such profiles at the same time. > > Of course, if it is technically feasible to transition both runners (I > don't really know about Spark's coupling with Scala versions) that is event > easier and defers the larger issue for a bit. > > I see two solutions for this: >> - Introducing a project-wide Scala version property >> - Allowing differing Scala versions for different runners, ensure that we >> never have a situation where we have several Runners as dependency. For the >> the "Maven Install" pre-commit hook this could mean splitting it up per >> runner. >> > > I support the latter regardless. I want separate configurations for > separate runners, fully embracing the fact that they can diverge. > > We already intend to split up the build to be much more fine grained. The > only reason we have a monolithic precommit is Maven's extraordinary lack of > support for any other way of building. We have essentially started to build > something akin to a particular invocation of "make" in the form of > interrelated Jenkins jobs to work around Maven's limitations [1]. Until we > can get that approach working at 100% I am fine with splitting builds > aggressively. You will find that it is quite hard not to duplicate a lot of > work when splitting them, unless we abandon the Jenkins plugin and use a > sequence of carefully crafted shell commands. > > Kenn > > [1] > https://github.com/apache/beam/blob/master/.test-infra/jenkins/PreCommit_Pipeline.groovy
