Re: Java 11 Checkin again
Hey, Yi! Thanks. I started a slack channel that maybe would make it easier to communicate if I have questions. I do have one issue that I am hitting between the 2 different Yarn versions I think and that I am not entirely sure what to do about. I made a change to a Test class that was needed for a compilation fix: https://github.com/apache/samza/pull/1628/files#diff-34db8b18730bda1058014e87ec3ad88dfc03f79854b00a339407761d224f66e9 and the class is TestSamzaYarnAppMasterLifecycle.scala. I'm not sure how to go about toggling this class between a compatible one for yarn 2.10.1 and 3.3.4. Thanks! -Jamie On Mon, Sep 19, 2022 at 11:00 AM Yi Pan wrote: > Hi, James, > > Thanks a lot for reporting this. I will take a look this week. > > Best! > > -Yi > > On Mon, Sep 19, 2022 at 8:20 AM James DeMichele > wrote: > > > Hey Yi. I can take a look at this. I do want to point out that your > current > > "master" branch is actually broken for using Scala 2.11. > > > > You can repro by just going into the master branch using Java 8 and > > compiling like this > > > > $ java -version > > openjdk version "1.8.0_332" > > OpenJDK Runtime Environment (Temurin)(build 1.8.0_332-b09) > > OpenJDK 64-Bit Server VM (Temurin)(build 25.332-b09, mixed mode) > > > > ./gradlew build -PscalaSuffix=2.11 > > > > The build fails with this command using that version of Java 8 ^. > > > > Anyway, just wanted to point that out since I hit this in my branch > > trying to utilize the "bin/check-all.sh" script. That doesn't block > > me/us, but just wanted to call it out. > > > > -Jamie > > > > > > > > On Wed, Sep 14, 2022 at 5:52 PM Yi Pan wrote: > > > > > Hey, James, > > > > > > In order to merge your PR without breaking the jdk8 older modules, we > > will > > > need the changes proposed here. Can you try to add those build script > > > changes in the same PR? We will definitely help review and merge it. > > > > > > Best! > > > > > > -Yi > > > > > > On Wed, Sep 14, 2022 at 8:00 AM James DeMichele > > > wrote: > > > > > > > Also, do you have a timeline for when this could be completed? > Thanks. > > > > > > > > On Wed, Sep 14, 2022 at 7:16 AM James DeMichele < > > > > james.demich...@redfin.com> > > > > wrote: > > > > > > > > > That sounds like a great solution to me if that works for y'all! > > > > > > > > > > Note too, the Java 11 and yarn 3 module need to only use the Scala > > 2.12 > > > > > version of the build. > > > > > > > > > > Jamie > > > > > > > > > > > > > > > On Wed, Sep 14, 2022, 2:38 AM Yi Pan wrote: > > > > > > > > > >> Hi, James, > > > > >> > > > > >> Sorry to reply late. I just came back from a trip and had a > > discussion > > > > >> with > > > > >> our internal team as well. So, there is one proposal other than > > > > creating a > > > > >> branch. Let me elaborate it below: > > > > >> a) creating a new module samza-yarn3 that depends on YARN 3.3.0 > and > > be > > > > the > > > > >> hosting module for most of the jdk11 related changes. > > > > >> b) modify the build script s.t. samza-yarn3 will only compile and > > > build > > > > >> with jdk11 and samza-yarn only compile and build with jdk8. > > > > >> Thus, we can have two builds: jdk8 build that builds with > samza-yarn > > > w/ > > > > >> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN > > > 3.3.0. > > > > >> We > > > > >> can manage to publish both jdk8 and jdk11 artifacts if needed. > > > > >> The benefit of this approach is that we can still maintain the > trunk > > > > >> release while opening up the jdk11 support. > > > > >> > > > > >> Let me know if that works for you and we can work together to get > > the > > > > code > > > > >> in. > > > > >> > > > > >> Best! > > > > >> > > > > >> -Yi > > > > >> > > > > >> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele > > > > >> wrote: > > > > >> > > > > >> > Hey Yi, I wanted to follow up here and figure what a path > forward > > is > > > > >> here. > > > > >> > We need to move to Java 11, and Samza currently is our only > > blocking > > > > >> issue. > > > > >> > In order to move to Java 11, the Yarn Cluster would need to run > on > > > > Java > > > > >> 11 > > > > >> > correct? If that's the case, then it would need to be 3.3+. I > > don't > > > > know > > > > >> > what it entails on your end to have a new Major version, but > that > > > > seems > > > > >> > like a good option here right? Version 2 could be where we can > > move > > > > this > > > > >> > project forward to Java 11, while Version 1 can still remain, > and > > > > would > > > > >> not > > > > >> > break people that can't/won't upgrade to Java 11. > > > > >> > > > > > >> > -Jamie > > > > >> > > > > > >> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele < > > > > >> james.demich...@redfin.com > > > > >> > > > > > > >> > wrote: > > > > >> > > > > > >> > > Yeah I mean if Samza works fine with the hadoop-yarn library > > > running > > > > >> > > against a 3.3.x YARN cluster, then I don't mind keeping that > > > library > > > > >> of > > > >
Re: Java 11 Checkin again
Hi, James, Thanks a lot for reporting this. I will take a look this week. Best! -Yi On Mon, Sep 19, 2022 at 8:20 AM James DeMichele wrote: > Hey Yi. I can take a look at this. I do want to point out that your current > "master" branch is actually broken for using Scala 2.11. > > You can repro by just going into the master branch using Java 8 and > compiling like this > > $ java -version > openjdk version "1.8.0_332" > OpenJDK Runtime Environment (Temurin)(build 1.8.0_332-b09) > OpenJDK 64-Bit Server VM (Temurin)(build 25.332-b09, mixed mode) > > ./gradlew build -PscalaSuffix=2.11 > > The build fails with this command using that version of Java 8 ^. > > Anyway, just wanted to point that out since I hit this in my branch > trying to utilize the "bin/check-all.sh" script. That doesn't block > me/us, but just wanted to call it out. > > -Jamie > > > > On Wed, Sep 14, 2022 at 5:52 PM Yi Pan wrote: > > > Hey, James, > > > > In order to merge your PR without breaking the jdk8 older modules, we > will > > need the changes proposed here. Can you try to add those build script > > changes in the same PR? We will definitely help review and merge it. > > > > Best! > > > > -Yi > > > > On Wed, Sep 14, 2022 at 8:00 AM James DeMichele > > wrote: > > > > > Also, do you have a timeline for when this could be completed? Thanks. > > > > > > On Wed, Sep 14, 2022 at 7:16 AM James DeMichele < > > > james.demich...@redfin.com> > > > wrote: > > > > > > > That sounds like a great solution to me if that works for y'all! > > > > > > > > Note too, the Java 11 and yarn 3 module need to only use the Scala > 2.12 > > > > version of the build. > > > > > > > > Jamie > > > > > > > > > > > > On Wed, Sep 14, 2022, 2:38 AM Yi Pan wrote: > > > > > > > >> Hi, James, > > > >> > > > >> Sorry to reply late. I just came back from a trip and had a > discussion > > > >> with > > > >> our internal team as well. So, there is one proposal other than > > > creating a > > > >> branch. Let me elaborate it below: > > > >> a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and > be > > > the > > > >> hosting module for most of the jdk11 related changes. > > > >> b) modify the build script s.t. samza-yarn3 will only compile and > > build > > > >> with jdk11 and samza-yarn only compile and build with jdk8. > > > >> Thus, we can have two builds: jdk8 build that builds with samza-yarn > > w/ > > > >> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN > > 3.3.0. > > > >> We > > > >> can manage to publish both jdk8 and jdk11 artifacts if needed. > > > >> The benefit of this approach is that we can still maintain the trunk > > > >> release while opening up the jdk11 support. > > > >> > > > >> Let me know if that works for you and we can work together to get > the > > > code > > > >> in. > > > >> > > > >> Best! > > > >> > > > >> -Yi > > > >> > > > >> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele > > > >> wrote: > > > >> > > > >> > Hey Yi, I wanted to follow up here and figure what a path forward > is > > > >> here. > > > >> > We need to move to Java 11, and Samza currently is our only > blocking > > > >> issue. > > > >> > In order to move to Java 11, the Yarn Cluster would need to run on > > > Java > > > >> 11 > > > >> > correct? If that's the case, then it would need to be 3.3+. I > don't > > > know > > > >> > what it entails on your end to have a new Major version, but that > > > seems > > > >> > like a good option here right? Version 2 could be where we can > move > > > this > > > >> > project forward to Java 11, while Version 1 can still remain, and > > > would > > > >> not > > > >> > break people that can't/won't upgrade to Java 11. > > > >> > > > > >> > -Jamie > > > >> > > > > >> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele < > > > >> james.demich...@redfin.com > > > >> > > > > > >> > wrote: > > > >> > > > > >> > > Yeah I mean if Samza works fine with the hadoop-yarn library > > running > > > >> > > against a 3.3.x YARN cluster, then I don't mind keeping that > > library > > > >> of > > > >> > > 2.10.x in Samza's code. But it is still a moot point in terms of > > > >> > upgrading > > > >> > > your YARN cluster, since it must be upgraded to 3.3.x+ in order > to > > > be > > > >> > able > > > >> > > to run the Cluster with Java 11. > > > >> > > > > > >> > > @Yi, I think that moving to a new major version might be the > > > solution > > > >> > > here. That way Linkedin can still have a pathway of upgrading > code > > > for > > > >> > the > > > >> > > old legacy 1.x version of Samza. While a new major version of > 2.x > > of > > > >> > Samza > > > >> > > could then make it a requirement that it runs with a YARN > cluster > > of > > > >> > 3.3.x > > > >> > > if you want to use Java 11. > > > >> > > > > > >> > > The only issue there is that you'll probably need to backport > > > changes > > > >> > > between the 2 versions. But in all honestly, this project does > not > > > >> look > > > >> > > extremely active with commits
Re: Java 11 Checkin again
Hey Yi. I can take a look at this. I do want to point out that your current "master" branch is actually broken for using Scala 2.11. You can repro by just going into the master branch using Java 8 and compiling like this $ java -version openjdk version "1.8.0_332" OpenJDK Runtime Environment (Temurin)(build 1.8.0_332-b09) OpenJDK 64-Bit Server VM (Temurin)(build 25.332-b09, mixed mode) ./gradlew build -PscalaSuffix=2.11 The build fails with this command using that version of Java 8 ^. Anyway, just wanted to point that out since I hit this in my branch trying to utilize the "bin/check-all.sh" script. That doesn't block me/us, but just wanted to call it out. -Jamie On Wed, Sep 14, 2022 at 5:52 PM Yi Pan wrote: > Hey, James, > > In order to merge your PR without breaking the jdk8 older modules, we will > need the changes proposed here. Can you try to add those build script > changes in the same PR? We will definitely help review and merge it. > > Best! > > -Yi > > On Wed, Sep 14, 2022 at 8:00 AM James DeMichele > wrote: > > > Also, do you have a timeline for when this could be completed? Thanks. > > > > On Wed, Sep 14, 2022 at 7:16 AM James DeMichele < > > james.demich...@redfin.com> > > wrote: > > > > > That sounds like a great solution to me if that works for y'all! > > > > > > Note too, the Java 11 and yarn 3 module need to only use the Scala 2.12 > > > version of the build. > > > > > > Jamie > > > > > > > > > On Wed, Sep 14, 2022, 2:38 AM Yi Pan wrote: > > > > > >> Hi, James, > > >> > > >> Sorry to reply late. I just came back from a trip and had a discussion > > >> with > > >> our internal team as well. So, there is one proposal other than > > creating a > > >> branch. Let me elaborate it below: > > >> a) creating a new module samza-yarn3 that depends on YARN 3.3.0 and be > > the > > >> hosting module for most of the jdk11 related changes. > > >> b) modify the build script s.t. samza-yarn3 will only compile and > build > > >> with jdk11 and samza-yarn only compile and build with jdk8. > > >> Thus, we can have two builds: jdk8 build that builds with samza-yarn > w/ > > >> YARN 2.10.0, and jdk11 build that builds with samza-yanr3 w/ YARN > 3.3.0. > > >> We > > >> can manage to publish both jdk8 and jdk11 artifacts if needed. > > >> The benefit of this approach is that we can still maintain the trunk > > >> release while opening up the jdk11 support. > > >> > > >> Let me know if that works for you and we can work together to get the > > code > > >> in. > > >> > > >> Best! > > >> > > >> -Yi > > >> > > >> On Tue, Sep 13, 2022 at 7:53 AM James DeMichele > > >> wrote: > > >> > > >> > Hey Yi, I wanted to follow up here and figure what a path forward is > > >> here. > > >> > We need to move to Java 11, and Samza currently is our only blocking > > >> issue. > > >> > In order to move to Java 11, the Yarn Cluster would need to run on > > Java > > >> 11 > > >> > correct? If that's the case, then it would need to be 3.3+. I don't > > know > > >> > what it entails on your end to have a new Major version, but that > > seems > > >> > like a good option here right? Version 2 could be where we can move > > this > > >> > project forward to Java 11, while Version 1 can still remain, and > > would > > >> not > > >> > break people that can't/won't upgrade to Java 11. > > >> > > > >> > -Jamie > > >> > > > >> > On Tue, Sep 6, 2022 at 9:55 AM James DeMichele < > > >> james.demich...@redfin.com > > >> > > > > >> > wrote: > > >> > > > >> > > Yeah I mean if Samza works fine with the hadoop-yarn library > running > > >> > > against a 3.3.x YARN cluster, then I don't mind keeping that > library > > >> of > > >> > > 2.10.x in Samza's code. But it is still a moot point in terms of > > >> > upgrading > > >> > > your YARN cluster, since it must be upgraded to 3.3.x+ in order to > > be > > >> > able > > >> > > to run the Cluster with Java 11. > > >> > > > > >> > > @Yi, I think that moving to a new major version might be the > > solution > > >> > > here. That way Linkedin can still have a pathway of upgrading code > > for > > >> > the > > >> > > old legacy 1.x version of Samza. While a new major version of 2.x > of > > >> > Samza > > >> > > could then make it a requirement that it runs with a YARN cluster > of > > >> > 3.3.x > > >> > > if you want to use Java 11. > > >> > > > > >> > > The only issue there is that you'll probably need to backport > > changes > > >> > > between the 2 versions. But in all honestly, this project does not > > >> look > > >> > > extremely active with commits so it might not be that big of a > > >> problem. > > >> > > > > >> > > -Jamie > > >> > > > > >> > > On Fri, Sep 2, 2022 at 9:08 PM Malcolm McFarland < > > >> mmcfarl...@cavulus.com > > >> > > > > >> > > wrote: > > >> > > > > >> > >> Hi all, > > >> > >> > > >> > >> I've been doing a little bit of testing with Samza and Hadoop > > 3.3.4; > > >> > >> afaict, in light testing, Samza seems to work fine using the > 2.10.x > > >> > >> hadoop-yarn
