Yes, Kenn I was talking about building artifacts (because this is the strictest way to validate the compatiblity of our code base and find issues with some of the classpath/code generation we use or with the dependencies). I agree 100% with you, the real deal is to validate that we can use Java 8 built jars with more recent Java versions without issues and examples could be a good target for this.
Supporting all non LTS java versions seems like a lot of work for a small return: (1) most users will probably stay in LTS versions, (2) not any single execution system has talked about supporting intermediary versions (some not even LTS) (3) this will augment the combination of tests we should execute to validate a release. Of course if we do care only about direct runner this could make sense. On Thu, Oct 18, 2018 at 9:30 AM Robert Bradshaw <rober...@google.com> wrote: > On Thu, Oct 18, 2018 at 4:55 AM Kenneth Knowles <k...@apache.org> wrote: > >> Just to add to what Luke said: The reason we had those Java 8-only >> modules was because some underlying tech (example: Gearpump) required Java >> 8. If an engine requires something then it is OK for a user who chooses the >> runner for that engine to also be subject to that requirement. Otherwise I >> would push back a bit on specialized modules. >> > > I, too, would avoid getting back into the situation where we have > different modules with different build/runtime requirements. (Java 8 was > special both because of Gearpump and because it introduced features such as > lambdas, type inference, and other functional libraries that were > particularly well suited to our programming model). > > >> Ismaël: Do we need to be thinking in terms of building artifacts? I think >> the important thing is if a user has a project and just pulls our jars from >> maven central that they can use JRE 11 to run it. I think what we need is a >> build matrix of user pipelines compiled and run w/ Java 8, 9, 10, 11 (or >> some subset). This can be obtained via the examples maven archetype, for >> example, having no relation at all to our own build tooling. I know Jenkins >> has a plugin for this, so we don't run into the "mega-Jenkins-job" problem. >> > > +1 for rephrasing it this way. In general one would expect our jars of > classfiles to work with later JREs. The magic we do with annotations and > bytecode generation may, however, require more support and testing. > > >> On Wed, Oct 17, 2018 at 2:03 PM Lukasz Cwik <lc...@google.com> wrote: >> >>> When we were supporting Java 7, there were Java 8 based modules in >>> Maven. We can follow a similar pattern where Java11 (or any other version) >>> have their own projects and as the base Java version moves up, we can merge >>> those modules back in to simplify the set of modules being developed. >>> >>> On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <ieme...@gmail.com> wrote: >>> >>>> I want to give more context on the previous work towards Java 11 >>>> support. The issue was not necessarily that gradle was not ok to support >>>> Java 11, but this is also a good point Lukasz. >>>> >>>> Months ago, we built a maven profile that basically set up Java to be >>>> strictly compatible with the ‘minimal’ java.base profile and used Java >>>> version (9/10) at the time. We found and fixed some missing dependencies >>>> (because Java 11 removed some of the Java EE stuff from the base profile), >>>> not sure if these changes were migrated to the gradle build. >>>> >>>> We also dealt with a good chunk of updates at the time of >>>> plugins/bytebuddy and other deps to guarantee that the execution was done >>>> strictly with the latest version of Java and its generated bytecode (the >>>> origin of the docker build images was to guarantee this isolation). At the >>>> time I run the tests of most IOs and most extensions with the existing >>>> (java 8 compiled) direct runner in JRE 10 with success but of course this >>>> was not the ultimate goal but a good way to evaluate the state of the code >>>> base (I remember some GCP tests were broken and the ApiSurfaceTests too). >>>> >>>> One important thing to remember is that the goal on this work was >>>> ‘limited’ to guarantee that a user that had ONLY Java 11 installed could >>>> build and generate Beam artifacts and run Beam programs in the direct >>>> runner. The limited scope was in part because of some issues of a full >>>> migration to Java 11: >>>> >>>> - Support for Java modules, this is a too much radical change and >>>> probably far from the scope of Beam at the moment. >>>> - Support for other runners, given that most execution systems do not >>>> support yet Java 11 this could imply a lot of extra work. In the meantime, >>>> this could be easier now due to the portability work, anyway, still not >>>> sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon. >>>> >>>> Let’s not forget that we still are and probably will be for a long time >>>> backwards compatible with Java 8 so we need to ensure that Java 11 only >>>> things don’t get into the code base. >>>> >>>> If you Lukasz or someone else is up to take this task, I think the >>>> immediate thing to do is to get back to the same state we were before the >>>> move to gradle, build the equivalent of the maven profile for the gradle >>>> build and tackle code generation and classpath issues, then try to fix >>>> missing parts and push this into the CI too (similar to the recent python 3 >>>> work) to guarantee that new changes don’t break Java 11 compatibility. I >>>> sadly lack of bandwidth to lead this task but I will gladly help with >>>> reviews, or feedback if someone else is up to the task. >>>> >>>> Not sure about more recent details, but probably worth evaluating at >>>> some moment is if we there is some interest on supporting multi-release >>>> jars with Java 11 classes or the dream scenario of having a AOT build of >>>> the direct runner. >>>> >>>> On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy <lukasz.gaj...@gmail.com> >>>> wrote: >>>> >>>>> FYI: If I'm correct, migrating to Java 11 may require Migrating to >>>>> Gradle 5, which to me is ok, but there are some issues on the way (such as >>>>> [1] or maybe more that I'm not aware of). >>>>> >>>>> I did some research and it seems that Groovy 2.4.15 (the default >>>>> version of Groovy for Gradle 4.10.2) is unable to work with Java 11 due to >>>>> Nest based access control problem [2]. The very same error happened when I >>>>> tried to compile "core" module with java11. This issue is solved in Groovy >>>>> 2.5.3 [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3 >>>>> (NoClassDefError) and found out that the support for Groovy 2.5.3 will be >>>>> added in Gradle 5.0RC1 (milestone) [4]. >>>>> >>>>> Other than that I had some issues with spottless and findbugs (I >>>>> turned them off for the sake of my experiment). >>>>> >>>>> Łukasz >>>>> >>>>> [1] https://issues.apache.org/jira/browse/BEAM-4042 >>>>> [2] >>>>> https://github.com/gradle/gradle/issues/5120#issuecomment-424610114 >>>>> [3] https://issues.apache.org/jira/browse/GROOVY-8727 >>>>> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw> >>>>> [4] https://github.com/gradle/gradle/pull/7376 >>>>> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q> >>>>> >>>>> >>>>> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lukasz.gaj...@gmail.com> >>>>> napisał(a): >>>>> >>>>>> FWIW, the issue regarding java 11 support to gradle might be helpful >>>>>> here [1]. Quoting: "there are only some minor corner cases that don't >>>>>> work. >>>>>> Overall Java 11 support is pretty solid already" but some users reported >>>>>> problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 >>>>>> (this >>>>>> might be important for us). >>>>>> >>>>>> Łukasz >>>>>> >>>>>> [1] https://github.com/gradle/gradle/issues/5120 >>>>>> >>>>>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pabl...@google.com> >>>>>> napisał(a): >>>>>> >>>>>>> Hello all, >>>>>>> If I understand you correctly Ismael, a good amount of >>>>>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the >>>>>>> amount >>>>>>> of work necessary on the core module should be relatively small. Is this >>>>>>> correct? Are there improvements that may be missing in terms of >>>>>>> modularization? >>>>>>> >>>>>>> There is also the work necessary to build/run tests with Gradle.... >>>>>>> >>>>>>> I am also curious... how much work do you estimate is necessary to >>>>>>> support Java 11 with some of the existing sources? I understand that we >>>>>>> have many, many sources, but perhaps some of the more popular ones (e.g. >>>>>>> TextIO)? >>>>>>> >>>>>>> Thanks! >>>>>>> -P. >>>>>>> >>>>>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <arifka...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks for the clarification Ismaël. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> * • **Arif Kasim* >>>>>>>> * • * Strategic Cloud Engineer >>>>>>>> * • *Google, Inc. >>>>>>>> • arifka...@google.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ieme...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Just wanted to clarify, there is already a JIRA for ongoing work on >>>>>>>>> Java 11 support. >>>>>>>>> https://issues.apache.org/jira/browse/BEAM-2530 >>>>>>>>> >>>>>>>>> I led the initial work on supporting what at the time was Java >>>>>>>>> 9/10, >>>>>>>>> so far the biggest blockers were around the ApiSurface tests (not >>>>>>>>> at >>>>>>>>> all compatible with these versions) but at the time we were at 5 >>>>>>>>> tests >>>>>>>>> from getting sdks/core passing. Notice also that the scope of this >>>>>>>>> JIRA evolved to support only the LTS version (Java 11), and >>>>>>>>> specifically to support only sdks/core + direct runner. Supporting >>>>>>>>> all >>>>>>>>> IOs or runners really is more a question of the dependencies >>>>>>>>> working >>>>>>>>> nicely with Java 11 so this will probably take long time. Also the >>>>>>>>> idea so far does NOT include supporting the Java module system at >>>>>>>>> all. >>>>>>>>> >>>>>>>>> I stopped working on this during the move to gradle because it was >>>>>>>>> too >>>>>>>>> hard to tackle both Java evolving and all the ongoing changes in >>>>>>>>> the >>>>>>>>> build system. If somebody in the community wants to contribute in >>>>>>>>> this >>>>>>>>> area it will be greatly appreciated, notice that all the work we >>>>>>>>> did >>>>>>>>> on the build system for this needs to be implemented now in gradle >>>>>>>>> too. >>>>>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau < >>>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>>> > >>>>>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to inject >>>>>>>>> the proxy class is. There are other strategies you can use in >>>>>>>>> bytebuddy >>>>>>>>> which work. >>>>>>>>> > >>>>>>>>> > Romain Manni-Bucau >>>>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a >>>>>>>>> écrit : >>>>>>>>> >> >>>>>>>>> >> Romain, do you have any more details on the ByteBuddy >>>>>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or >>>>>>>>> just >>>>>>>>> with new language features? >>>>>>>>> >> >>>>>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau < >>>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>>> >>> >>>>>>>>> >>> Hi Arif, >>>>>>>>> >>> >>>>>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs >>>>>>>>> (but it means your pipeline is very very simple since it does not >>>>>>>>> have a >>>>>>>>> dofn ;)) if your engine supports it. Also note that the modules not >>>>>>>>> being >>>>>>>>> named you can have to use some weird import names or even unstable >>>>>>>>> ones if >>>>>>>>> you want to use modules (but there is no real reason to do that yet in >>>>>>>>> java). >>>>>>>>> >>> >>>>>>>>> >>> Romain Manni-Bucau >>>>>>>>> >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <arifka...@google.com> >>>>>>>>> a écrit : >>>>>>>>> >>>> >>>>>>>>> >>>> Hello, >>>>>>>>> >>>> What's the status of java version > 8 support for beam? >>>>>>>>> Thanks. >>>>>>>>> >>>> >>>>>>>>> >>>> -Arif. >>>>>>>>> >>>>>>>>
