Hey Yi, I wanted to follow up here and figure what a path forward is here. We need to move to Java 11, and Samza currently is our only blocking issue. In order to move to Java 11, the Yarn Cluster would need to run on Java 11 correct? If that's the case, then it would need to be 3.3+. I don't know what it entails on your end to have a new Major version, but that seems like a good option here right? Version 2 could be where we can move this project forward to Java 11, while Version 1 can still remain, and would not break people that can't/won't upgrade to Java 11.
-Jamie On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <james.demich...@redfin.com> wrote: > Yeah I mean if Samza works fine with the hadoop-yarn library running > against a 3.3.x YARN cluster, then I don't mind keeping that library of > 2.10.x in Samza's code. But it is still a moot point in terms of upgrading > your YARN cluster, since it must be upgraded to 3.3.x+ in order to be able > to run the Cluster with Java 11. > > @Yi, I think that moving to a new major version might be the solution > here. That way Linkedin can still have a pathway of upgrading code for the > old legacy 1.x version of Samza. While a new major version of 2.x of Samza > could then make it a requirement that it runs with a YARN cluster of 3.3.x > if you want to use Java 11. > > The only issue there is that you'll probably need to backport changes > between the 2 versions. But in all honestly, this project does not look > extremely active with commits so it might not be that big of a problem. > > -Jamie > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <mmcfarl...@cavulus.com> > wrote: > >> Hi all, >> >> I've been doing a little bit of testing with Samza and Hadoop 3.3.4; >> afaict, in light testing, Samza seems to work fine using the 2.10.x >> hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie pointed >> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 ( >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any unit >> tests >> in Samza that verify compatibility against a YARN cluster? If so, that >> could be a place to validate YARN v2.10/v3.3 cross-compatibility. >> >> Just throwing my 2 cents out there, >> Malcolm McFarland >> Cavulus >> >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele >> <james.demich...@redfin.com.invalid> wrote: >> >> > Hey Yi, >> > >> > Thanks for getting back to me. I have not tried the older yarn cluster >> > version yet in the Samza app running against 3.3.4 but I am wary it >> would >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you would >> > have to update yarn even if the Java library here wasn't updated. >> > >> > Could we move this version I'm proposing to a 2.x version of Samza? So >> > people that wanted to move forward with yarn upgrade and Samza and Java >> 11 >> > (like us) could do so? Then 1.x could only be java 8 compatible and 2.x >> > could be java 11. >> > >> > Jamie >> > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <nickpa...@gmail.com> wrote: >> > >> > > Hey, James, >> > > >> > > Thanks for the ping. @prateek, can we have someone to review this >> change? >> > > >> > > One question: have you tested the change w/ the older YARN cluster >> > version >> > > (running 2.10.1)? If this change requires YARN cluster upgrade to >> 3.3.4 >> > as >> > > well, that may be a breaking change to existing Samza users (i.e. >> > LinkedIn >> > > is still running a YARN cluster with version 2.10.1). >> > > >> > > Best and apologize for the delay. >> > > >> > > -Yi >> > > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele >> > > <james.demich...@redfin.com.invalid> wrote: >> > > >> > > > Hey y'all. I just am not sure how to get some traction on these >> Java 11 >> > > > PRs. >> > > > >> > > > https://github.com/apache/samza/pull/1628 >> > > > https://github.com/apache/samza-hello-samza/pull/87 >> > > > >> > > > Would someone that is a maintainer for Samza just let us know that >> > y'all >> > > > are looking at them? I can stop pestering you :) >> > > > >> > > > I ran all tests in both PRs, all pass. I also confirmed that using >> my >> > > Samza >> > > > PR in the Hello World app all works with Java 11. >> > > > >> > > > Thanks! >> > > > >> > > > -Jamie >> > > > >> > > >> > >> >