Also, do you have a timeline for when this could be completed? Thanks.

On Wed, Sep 14, 2022 at 7:16 AM James DeMichele <james.demich...@redfin.com>
wrote:

> That sounds like a great solution to me if that works for y'all!
>
> Note too, the Java 11 and yarn 3 module need to only use the Scala 2.12
> version of the build.
>
> Jamie
>
>
> On Wed, Sep 14, 2022, 2:38 AM Yi Pan <nickpa...@gmail.com> wrote:
>
>> Hi, James,
>>
>> Sorry to reply late. I just came back from a trip and had a discussion
>> with
>> our internal team as well. So, there is one proposal other than creating a
>> branch. Let me elaborate it below:
>> a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and be the
>> hosting module for most of the jdk11 related changes.
>> b) modify the build script s.t. samza-yarn3 will only compile and build
>> with jdk11 and samza-yarn only compile and build with jdk8.
>> Thus, we can have two builds: jdk8 build that builds with samza-yarn w/
>> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN 3.3.0.
>> We
>> can manage to publish both jdk8 and jdk11 artifacts if needed.
>> The benefit of this approach is that we can still maintain the trunk
>> release while opening up the jdk11 support.
>>
>> Let me know if that works for you and we can work together to get the code
>> in.
>>
>> Best!
>>
>> -Yi
>>
>> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele
>> <james.demich...@redfin.com.invalid> wrote:
>>
>> > Hey Yi, I wanted to follow up here and figure what a path forward is
>> here.
>> > We need to move to Java 11, and Samza currently is our only blocking
>> issue.
>> > In order to move to Java 11, the Yarn Cluster would need to run on Java
>> 11
>> > correct? If that's the case, then it would need to be 3.3+. I don't know
>> > what it entails on your end to have a new Major version, but that seems
>> > like a good option here right? Version 2 could be where we can move this
>> > project forward to Java 11, while Version 1 can still remain, and would
>> not
>> > break people that can't/won't upgrade to Java 11.
>> >
>> > -Jamie
>> >
>> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <
>> james.demich...@redfin.com
>> > >
>> > wrote:
>> >
>> > > Yeah I mean if Samza works fine with the hadoop-yarn library running
>> > > against a 3.3.x YARN cluster, then I don't mind keeping that library
>> of
>> > > 2.10.x in Samza's code. But it is still a moot point in terms of
>> > upgrading
>> > > your YARN cluster, since it must be upgraded to 3.3.x+ in order to be
>> > able
>> > > to run the Cluster with Java 11.
>> > >
>> > > @Yi, I think that moving to a new major version might be the solution
>> > > here. That way Linkedin can still have a pathway of upgrading code for
>> > the
>> > > old legacy 1.x version of Samza. While a new major version of 2.x of
>> > Samza
>> > > could then make it a requirement that it runs with a YARN cluster of
>> > 3.3.x
>> > > if you want to use Java 11.
>> > >
>> > > The only issue there is that you'll probably need to backport changes
>> > > between the 2 versions. But in all honestly, this project does not
>> look
>> > > extremely active with commits so it might not be that big of a
>> problem.
>> > >
>> > > -Jamie
>> > >
>> > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <
>> mmcfarl...@cavulus.com
>> > >
>> > > wrote:
>> > >
>> > >> Hi all,
>> > >>
>> > >> I've been doing a little bit of testing with Samza and Hadoop 3.3.4;
>> > >> afaict, in light testing, Samza seems to work fine using the 2.10.x
>> > >> hadoop-yarn library against a YARN cluster running 3.3.x. As Jamie
>> > pointed
>> > >> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 (
>> > >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any
>> unit
>> > >> tests
>> > >> in Samza that verify compatibility against a YARN cluster? If so,
>> that
>> > >> could be a place to validate YARN v2.10/v3.3 cross-compatibility.
>> > >>
>> > >> Just throwing my 2 cents out there,
>> > >> Malcolm McFarland
>> > >> Cavulus
>> > >>
>> > >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
>> > >> <james.demich...@redfin.com.invalid> wrote:
>> > >>
>> > >> > Hey Yi,
>> > >> >
>> > >> > Thanks for getting back to me. I have not tried the older yarn
>> cluster
>> > >> > version yet in the Samza app running against 3.3.4 but I am wary it
>> > >> would
>> > >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so you
>> > would
>> > >> > have to update yarn even if the Java library here wasn't updated.
>> > >> >
>> > >> > Could we move this version I'm proposing to a 2.x version of
>> Samza? So
>> > >> > people that wanted to move forward with yarn upgrade and Samza and
>> > Java
>> > >> 11
>> > >> > (like us) could do so? Then 1.x could only be java 8 compatible and
>> > 2.x
>> > >> > could be java 11.
>> > >> >
>> > >> > Jamie
>> > >> >
>> > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <nickpa...@gmail.com> wrote:
>> > >> >
>> > >> > > Hey, James,
>> > >> > >
>> > >> > > Thanks for the ping. @prateek, can we have someone to review this
>> > >> change?
>> > >> > >
>> > >> > > One question: have you tested the change w/ the older YARN
>> cluster
>> > >> > version
>> > >> > > (running 2.10.1)? If this change requires YARN cluster upgrade to
>> > >> 3.3.4
>> > >> > as
>> > >> > > well, that may be a breaking change to existing Samza users (i.e.
>> > >> > LinkedIn
>> > >> > > is still running a YARN cluster with version 2.10.1).
>> > >> > >
>> > >> > > Best and apologize for the delay.
>> > >> > >
>> > >> > > -Yi
>> > >> > >
>> > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
>> > >> > > <james.demich...@redfin.com.invalid> wrote:
>> > >> > >
>> > >> > > > Hey y'all. I just am not sure how to get some traction on these
>> > >> Java 11
>> > >> > > > PRs.
>> > >> > > >
>> > >> > > > https://github.com/apache/samza/pull/1628
>> > >> > > > https://github.com/apache/samza-hello-samza/pull/87
>> > >> > > >
>> > >> > > > Would someone that is a maintainer for Samza just let us know
>> that
>> > >> > y'all
>> > >> > > > are looking at them? I can stop pestering you :)
>> > >> > > >
>> > >> > > > I ran all tests in both PRs, all pass. I also confirmed that
>> using
>> > >> my
>> > >> > > Samza
>> > >> > > > PR in the Hello World app all works with Java 11.
>> > >> > > >
>> > >> > > > Thanks!
>> > >> > > >
>> > >> > > > -Jamie
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>

Reply via email to