Hey, James, In order to merge your PR without breaking the jdk8 older modules, we will need the changes proposed here. Can you try to add those build script changes in the same PR? We will definitely help review and merge it.
Best! -Yi On Wed, Sep 14, 2022 at 8:00 AM James DeMichele <james.demich...@redfin.com.invalid> wrote: > Also, do you have a timeline for when this could be completed? Thanks. > > On Wed, Sep 14, 2022 at 7:16 AM James DeMichele < > james.demich...@redfin.com> > wrote: > > > That sounds like a great solution to me if that works for y'all! > > > > Note too, the Java 11 and yarn 3 module need to only use the Scala 2.12 > > version of the build. > > > > Jamie > > > > > > On Wed, Sep 14, 2022, 2:38 AM Yi Pan <nickpa...@gmail.com> wrote: > > > >> Hi, James, > >> > >> Sorry to reply late. I just came back from a trip and had a discussion > >> with > >> our internal team as well. So, there is one proposal other than > creating a > >> branch. Let me elaborate it below: > >> a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and be > the > >> hosting module for most of the jdk11 related changes. > >> b) modify the build script s.t. samza-yarn3 will only compile and build > >> with jdk11 and samza-yarn only compile and build with jdk8. > >> Thus, we can have two builds: jdk8 build that builds with samza-yarn w/ > >> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN 3.3.0. > >> We > >> can manage to publish both jdk8 and jdk11 artifacts if needed. > >> The benefit of this approach is that we can still maintain the trunk > >> release while opening up the jdk11 support. > >> > >> Let me know if that works for you and we can work together to get the > code > >> in. > >> > >> Best! > >> > >> -Yi > >> > >> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele > >> <james.demich...@redfin.com.invalid> wrote: > >> > >> > Hey Yi, I wanted to follow up here and figure what a path forward is > >> here. > >> > We need to move to Java 11, and Samza currently is our only blocking > >> issue. > >> > In order to move to Java 11, the Yarn Cluster would need to run on > Java > >> 11 > >> > correct? If that's the case, then it would need to be 3.3+. I don't > know > >> > what it entails on your end to have a new Major version, but that > seems > >> > like a good option here right? Version 2 could be where we can move > this > >> > project forward to Java 11, while Version 1 can still remain, and > would > >> not > >> > break people that can't/won't upgrade to Java 11. > >> > > >> > -Jamie > >> > > >> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele < > >> james.demich...@redfin.com > >> > > > >> > wrote: > >> > > >> > > Yeah I mean if Samza works fine with the hadoop-yarn library running > >> > > against a 3.3.x YARN cluster, then I don't mind keeping that library > >> of > >> > > 2.10.x in Samza's code. But it is still a moot point in terms of > >> > upgrading > >> > > your YARN cluster, since it must be upgraded to 3.3.x+ in order to > be > >> > able > >> > > to run the Cluster with Java 11. > >> > > > >> > > @Yi, I think that moving to a new major version might be the > solution > >> > > here. That way Linkedin can still have a pathway of upgrading code > for > >> > the > >> > > old legacy 1.x version of Samza. While a new major version of 2.x of > >> > Samza > >> > > could then make it a requirement that it runs with a YARN cluster of > >> > 3.3.x > >> > > if you want to use Java 11. > >> > > > >> > > The only issue there is that you'll probably need to backport > changes > >> > > between the 2 versions. But in all honestly, this project does not > >> look > >> > > extremely active with commits so it might not be that big of a > >> problem. > >> > > > >> > > -Jamie > >> > > > >> > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland < > >> mmcfarl...@cavulus.com > >> > > > >> > > wrote: > >> > > > >> > >> Hi all, > >> > >> > >> > >> I've been doing a little bit of testing with Samza and Hadoop > 3.3.4; > >> > >> afaict, in light testing, Samza seems to work fine using the 2.10.x > >> > >> hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie > >> > pointed > >> > >> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 ( > >> > >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any > >> unit > >> > >> tests > >> > >> in Samza that verify compatibility against a YARN cluster? If so, > >> that > >> > >> could be a place to validate YARN v2.10/v3.3 cross-compatibility. > >> > >> > >> > >> Just throwing my 2 cents out there, > >> > >> Malcolm McFarland > >> > >> Cavulus > >> > >> > >> > >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele > >> > >> <james.demich...@redfin.com.invalid> wrote: > >> > >> > >> > >> > Hey Yi, > >> > >> > > >> > >> > Thanks for getting back to me. I have not tried the older yarn > >> cluster > >> > >> > version yet in the Samza app running against 3.3.4 but I am wary > it > >> > >> would > >> > >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you > >> > would > >> > >> > have to update yarn even if the Java library here wasn't updated. > >> > >> > > >> > >> > Could we move this version I'm proposing to a 2.x version of > >> Samza? So > >> > >> > people that wanted to move forward with yarn upgrade and Samza > and > >> > Java > >> > >> 11 > >> > >> > (like us) could do so? Then 1.x could only be java 8 compatible > and > >> > 2.x > >> > >> > could be java 11. > >> > >> > > >> > >> > Jamie > >> > >> > > >> > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <nickpa...@gmail.com> wrote: > >> > >> > > >> > >> > > Hey, James, > >> > >> > > > >> > >> > > Thanks for the ping. @prateek, can we have someone to review > this > >> > >> change? > >> > >> > > > >> > >> > > One question: have you tested the change w/ the older YARN > >> cluster > >> > >> > version > >> > >> > > (running 2.10.1)? If this change requires YARN cluster upgrade > to > >> > >> 3.3.4 > >> > >> > as > >> > >> > > well, that may be a breaking change to existing Samza users > (i.e. > >> > >> > LinkedIn > >> > >> > > is still running a YARN cluster with version 2.10.1). > >> > >> > > > >> > >> > > Best and apologize for the delay. > >> > >> > > > >> > >> > > -Yi > >> > >> > > > >> > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele > >> > >> > > <james.demich...@redfin.com.invalid> wrote: > >> > >> > > > >> > >> > > > Hey y'all. I just am not sure how to get some traction on > these > >> > >> Java 11 > >> > >> > > > PRs. > >> > >> > > > > >> > >> > > > https://github.com/apache/samza/pull/1628 > >> > >> > > > https://github.com/apache/samza-hello-samza/pull/87 > >> > >> > > > > >> > >> > > > Would someone that is a maintainer for Samza just let us know > >> that > >> > >> > y'all > >> > >> > > > are looking at them? I can stop pestering you :) > >> > >> > > > > >> > >> > > > I ran all tests in both PRs, all pass. I also confirmed that > >> using > >> > >> my > >> > >> > > Samza > >> > >> > > > PR in the Hello World app all works with Java 11. > >> > >> > > > > >> > >> > > > Thanks! > >> > >> > > > > >> > >> > > > -Jamie > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > > > >> > > >> > > >
