Hey Yi. I can take a look at this. I do want to point out that your current
"master" branch is actually broken for using Scala 2.11.

You can repro by just going into the master branch using Java 8 and
compiling like this

$ java -version
openjdk version "1.8.0_332"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (Temurin)(build 25.332-b09, mixed mode)

./gradlew build -PscalaSuffix=2.11

The build fails with this command using that version of Java 8 ^.

Anyway, just wanted to point that out since I hit this in my branch
trying to utilize the "bin/check-all.sh" script. That doesn't block
me/us, but just wanted to call it out.

-Jamie



On Wed, Sep 14, 2022 at 5:52 PM Yi Pan <nickpa...@gmail.com> wrote:

> Hey, James,
>
> In order to merge your PR without breaking the jdk8 older modules, we will
> need the changes proposed here. Can you try to add those build script
> changes in the same PR? We will definitely help review and merge it.
>
> Best!
>
> -Yi
>
> On Wed, Sep 14, 2022 at 8:00 AM James DeMichele
> <james.demich...@redfin.com.invalid> wrote:
>
> > Also, do you have a timeline for when this could be completed? Thanks.
> >
> > On Wed, Sep 14, 2022 at 7:16 AM James DeMichele <
> > james.demich...@redfin.com>
> > wrote:
> >
> > > That sounds like a great solution to me if that works for y'all!
> > >
> > > Note too, the Java 11 and yarn 3 module need to only use the Scala 2.12
> > > version of the build.
> > >
> > > Jamie
> > >
> > >
> > > On Wed, Sep 14, 2022, 2:38 AM Yi Pan <nickpa...@gmail.com> wrote:
> > >
> > >> Hi, James,
> > >>
> > >> Sorry to reply late. I just came back from a trip and had a discussion
> > >> with
> > >> our internal team as well. So, there is one proposal other than
> > creating a
> > >> branch. Let me elaborate it below:
> > >> a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and be
> > the
> > >> hosting module for most of the jdk11 related changes.
> > >> b) modify the build script s.t. samza-yarn3 will only compile and
> build
> > >> with jdk11 and samza-yarn only compile and build with jdk8.
> > >> Thus, we can have two builds: jdk8 build that builds with samza-yarn
> w/
> > >> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN
> 3.3.0.
> > >> We
> > >> can manage to publish both jdk8 and jdk11 artifacts if needed.
> > >> The benefit of this approach is that we can still maintain the trunk
> > >> release while opening up the jdk11 support.
> > >>
> > >> Let me know if that works for you and we can work together to get the
> > code
> > >> in.
> > >>
> > >> Best!
> > >>
> > >> -Yi
> > >>
> > >> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele
> > >> <james.demich...@redfin.com.invalid> wrote:
> > >>
> > >> > Hey Yi, I wanted to follow up here and figure what a path forward is
> > >> here.
> > >> > We need to move to Java 11, and Samza currently is our only blocking
> > >> issue.
> > >> > In order to move to Java 11, the Yarn Cluster would need to run on
> > Java
> > >> 11
> > >> > correct? If that's the case, then it would need to be 3.3+. I don't
> > know
> > >> > what it entails on your end to have a new Major version, but that
> > seems
> > >> > like a good option here right? Version 2 could be where we can move
> > this
> > >> > project forward to Java 11, while Version 1 can still remain, and
> > would
> > >> not
> > >> > break people that can't/won't upgrade to Java 11.
> > >> >
> > >> > -Jamie
> > >> >
> > >> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele <
> > >> james.demich...@redfin.com
> > >> > >
> > >> > wrote:
> > >> >
> > >> > > Yeah I mean if Samza works fine with the hadoop-yarn library
> running
> > >> > > against a 3.3.x YARN cluster, then I don't mind keeping that
> library
> > >> of
> > >> > > 2.10.x in Samza's code. But it is still a moot point in terms of
> > >> > upgrading
> > >> > > your YARN cluster, since it must be upgraded to 3.3.x+ in order to
> > be
> > >> > able
> > >> > > to run the Cluster with Java 11.
> > >> > >
> > >> > > @Yi, I think that moving to a new major version might be the
> > solution
> > >> > > here. That way Linkedin can still have a pathway of upgrading code
> > for
> > >> > the
> > >> > > old legacy 1.x version of Samza. While a new major version of 2.x
> of
> > >> > Samza
> > >> > > could then make it a requirement that it runs with a YARN cluster
> of
> > >> > 3.3.x
> > >> > > if you want to use Java 11.
> > >> > >
> > >> > > The only issue there is that you'll probably need to backport
> > changes
> > >> > > between the 2 versions. But in all honestly, this project does not
> > >> look
> > >> > > extremely active with commits so it might not be that big of a
> > >> problem.
> > >> > >
> > >> > > -Jamie
> > >> > >
> > >> > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland <
> > >> mmcfarl...@cavulus.com
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > >> Hi all,
> > >> > >>
> > >> > >> I've been doing a little bit of testing with Samza and Hadoop
> > 3.3.4;
> > >> > >> afaict, in light testing, Samza seems to work fine using the
> 2.10.x
> > >> > >> hadoop-yarn library against a YARN cluster running 3.3.x. As
> Jamie
> > >> > pointed
> > >> > >> out, YARN didn't incorporate Java 11 compatibility until v3.3.0 (
> > >> > >> https://hadoop.apache.org/docs/r3.3.0/index.html). Are there any
> > >> unit
> > >> > >> tests
> > >> > >> in Samza that verify compatibility against a YARN cluster? If so,
> > >> that
> > >> > >> could be a place to validate YARN v2.10/v3.3 cross-compatibility.
> > >> > >>
> > >> > >> Just throwing my 2 cents out there,
> > >> > >> Malcolm McFarland
> > >> > >> Cavulus
> > >> > >>
> > >> > >> On Fri, Sep 2, 2022 at 6:27 PM James DeMichele
> > >> > >> <james.demich...@redfin.com.invalid> wrote:
> > >> > >>
> > >> > >> > Hey Yi,
> > >> > >> >
> > >> > >> > Thanks for getting back to me. I have not tried the older yarn
> > >> cluster
> > >> > >> > version yet in the Samza app running against 3.3.4 but I am
> wary
> > it
> > >> > >> would
> > >> > >> > work. Yarn itself is not compatible at 2.10.1 with Java 11 so
> you
> > >> > would
> > >> > >> > have to update yarn even if the Java library here wasn't
> updated.
> > >> > >> >
> > >> > >> > Could we move this version I'm proposing to a 2.x version of
> > >> Samza? So
> > >> > >> > people that wanted to move forward with yarn upgrade and Samza
> > and
> > >> > Java
> > >> > >> 11
> > >> > >> > (like us) could do so? Then 1.x could only be java 8 compatible
> > and
> > >> > 2.x
> > >> > >> > could be java 11.
> > >> > >> >
> > >> > >> > Jamie
> > >> > >> >
> > >> > >> > On Fri, Sep 2, 2022, 6:44 PM Yi Pan <nickpa...@gmail.com>
> wrote:
> > >> > >> >
> > >> > >> > > Hey, James,
> > >> > >> > >
> > >> > >> > > Thanks for the ping. @prateek, can we have someone to review
> > this
> > >> > >> change?
> > >> > >> > >
> > >> > >> > > One question: have you tested the change w/ the older YARN
> > >> cluster
> > >> > >> > version
> > >> > >> > > (running 2.10.1)? If this change requires YARN cluster
> upgrade
> > to
> > >> > >> 3.3.4
> > >> > >> > as
> > >> > >> > > well, that may be a breaking change to existing Samza users
> > (i.e.
> > >> > >> > LinkedIn
> > >> > >> > > is still running a YARN cluster with version 2.10.1).
> > >> > >> > >
> > >> > >> > > Best and apologize for the delay.
> > >> > >> > >
> > >> > >> > > -Yi
> > >> > >> > >
> > >> > >> > > On Fri, Sep 2, 2022 at 8:56 AM James DeMichele
> > >> > >> > > <james.demich...@redfin.com.invalid> wrote:
> > >> > >> > >
> > >> > >> > > > Hey y'all. I just am not sure how to get some traction on
> > these
> > >> > >> Java 11
> > >> > >> > > > PRs.
> > >> > >> > > >
> > >> > >> > > > https://github.com/apache/samza/pull/1628
> > >> > >> > > > https://github.com/apache/samza-hello-samza/pull/87
> > >> > >> > > >
> > >> > >> > > > Would someone that is a maintainer for Samza just let us
> know
> > >> that
> > >> > >> > y'all
> > >> > >> > > > are looking at them? I can stop pestering you :)
> > >> > >> > > >
> > >> > >> > > > I ran all tests in both PRs, all pass. I also confirmed
> that
> > >> using
> > >> > >> my
> > >> > >> > > Samza
> > >> > >> > > > PR in the Hello World app all works with Java 11.
> > >> > >> > > >
> > >> > >> > > > Thanks!
> > >> > >> > > >
> > >> > >> > > > -Jamie
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> > >
> > >> >
> > >>
> > >
> >
>

Reply via email to