Yeah I mean if Samza works fine with the hadoop-yarn library running against a 3.3.x YARN cluster, then I don't mind keeping that library of 2.10.x in Samza's code. But it is still a moot point in terms of upgrading your YARN cluster, since it must be upgraded to 3.3.x+ in order to be able to run the Cluster with Java 11.
@Yi, I think that moving to a new major version might be the solution here. That way Linkedin can still have a pathway of upgrading code for the old legacy 1.x version of Samza. While a new major version of 2.x of Samza could then make it a requirement that it runs with a YARN cluster of 3.3.x if you want to use Java 11. The only issue there is that you'll probably need to backport changes between the 2 versions. But in all honestly, this project does not look extremely active with commits so it might not be that big of a problem. -Jamie On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <mmcfarl...@cavulus.com> wrote: > Hi all, > > I've been doing a little bit of testing with Samza and Hadoop 3.3.4; > afaict, in light testing, Samza seems to work fine using the 2.10.x > hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie pointed > out, YARN didn't incorporate Java 11 compatibility until v3.3.0 ( > https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any unit > tests > in Samza that verify compatibility against a YARN cluster? If so, that > could be a place to validate YARN v2.10/v3.3 cross-compatibility. > > Just throwing my 2 cents out there, > Malcolm McFarland > Cavulus > > On Fri, Sep 2, 2022 at 6:27 PM James DeMichele > <james.demich...@redfin.com.invalid> wrote: > > > Hey Yi, > > > > Thanks for getting back to me. I have not tried the older yarn cluster > > version yet in the Samza app running against 3.3.4 but I am wary it would > > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you would > > have to update yarn even if the Java library here wasn't updated. > > > > Could we move this version I'm proposing to a 2.x version of Samza? So > > people that wanted to move forward with yarn upgrade and Samza and Java > 11 > > (like us) could do so? Then 1.x could only be java 8 compatible and 2.x > > could be java 11. > > > > Jamie > > > > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <nickpa...@gmail.com> wrote: > > > > > Hey, James, > > > > > > Thanks for the ping. @prateek, can we have someone to review this > change? > > > > > > One question: have you tested the change w/ the older YARN cluster > > version > > > (running 2.10.1)? If this change requires YARN cluster upgrade to 3.3.4 > > as > > > well, that may be a breaking change to existing Samza users (i.e. > > LinkedIn > > > is still running a YARN cluster with version 2.10.1). > > > > > > Best and apologize for the delay. > > > > > > -Yi > > > > > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele > > > <james.demich...@redfin.com.invalid> wrote: > > > > > > > Hey y'all. I just am not sure how to get some traction on these Java > 11 > > > > PRs. > > > > > > > > https://github.com/apache/samza/pull/1628 > > > > https://github.com/apache/samza-hello-samza/pull/87 > > > > > > > > Would someone that is a maintainer for Samza just let us know that > > y'all > > > > are looking at them? I can stop pestering you :) > > > > > > > > I ran all tests in both PRs, all pass. I also confirmed that using my > > > Samza > > > > PR in the Hello World app all works with Java 11. > > > > > > > > Thanks! > > > > > > > > -Jamie > > > > > > > > > >
