Just to add to what Luke said: The reason we had those Java 8-only modules was because some underlying tech (example: Gearpump) required Java 8. If an engine requires something then it is OK for a user who chooses the runner for that engine to also be subject to that requirement. Otherwise I would push back a bit on specialized modules.
Ismaël: Do we need to be thinking in terms of building artifacts? I think the important thing is if a user has a project and just pulls our jars from maven central that they can use JRE 11 to run it. I think what we need is a build matrix of user pipelines compiled and run w/ Java 8, 9, 10, 11 (or some subset). This can be obtained via the examples maven archetype, for example, having no relation at all to our own build tooling. I know Jenkins has a plugin for this, so we don't run into the "mega-Jenkins-job" problem. Kenn On Wed, Oct 17, 2018 at 2:03 PM Lukasz Cwik <lc...@google.com> wrote: > When we were supporting Java 7, there were Java 8 based modules in Maven. > We can follow a similar pattern where Java11 (or any other version) have > their own projects and as the base Java version moves up, we can merge > those modules back in to simplify the set of modules being developed. > > On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <ieme...@gmail.com> wrote: > >> I want to give more context on the previous work towards Java 11 support. >> The issue was not necessarily that gradle was not ok to support Java 11, >> but this is also a good point Lukasz. >> >> Months ago, we built a maven profile that basically set up Java to be >> strictly compatible with the ‘minimal’ java.base profile and used Java >> version (9/10) at the time. We found and fixed some missing dependencies >> (because Java 11 removed some of the Java EE stuff from the base profile), >> not sure if these changes were migrated to the gradle build. >> >> We also dealt with a good chunk of updates at the time of >> plugins/bytebuddy and other deps to guarantee that the execution was done >> strictly with the latest version of Java and its generated bytecode (the >> origin of the docker build images was to guarantee this isolation). At the >> time I run the tests of most IOs and most extensions with the existing >> (java 8 compiled) direct runner in JRE 10 with success but of course this >> was not the ultimate goal but a good way to evaluate the state of the code >> base (I remember some GCP tests were broken and the ApiSurfaceTests too). >> >> One important thing to remember is that the goal on this work was >> ‘limited’ to guarantee that a user that had ONLY Java 11 installed could >> build and generate Beam artifacts and run Beam programs in the direct >> runner. The limited scope was in part because of some issues of a full >> migration to Java 11: >> >> - Support for Java modules, this is a too much radical change and >> probably far from the scope of Beam at the moment. >> - Support for other runners, given that most execution systems do not >> support yet Java 11 this could imply a lot of extra work. In the meantime, >> this could be easier now due to the portability work, anyway, still not >> sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon. >> >> Let’s not forget that we still are and probably will be for a long time >> backwards compatible with Java 8 so we need to ensure that Java 11 only >> things don’t get into the code base. >> >> If you Lukasz or someone else is up to take this task, I think the >> immediate thing to do is to get back to the same state we were before the >> move to gradle, build the equivalent of the maven profile for the gradle >> build and tackle code generation and classpath issues, then try to fix >> missing parts and push this into the CI too (similar to the recent python 3 >> work) to guarantee that new changes don’t break Java 11 compatibility. I >> sadly lack of bandwidth to lead this task but I will gladly help with >> reviews, or feedback if someone else is up to the task. >> >> Not sure about more recent details, but probably worth evaluating at some >> moment is if we there is some interest on supporting multi-release jars >> with Java 11 classes or the dream scenario of having a AOT build of the >> direct runner. >> >> On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy <lukasz.gaj...@gmail.com> >> wrote: >> >>> FYI: If I'm correct, migrating to Java 11 may require Migrating to >>> Gradle 5, which to me is ok, but there are some issues on the way (such as >>> [1] or maybe more that I'm not aware of). >>> >>> I did some research and it seems that Groovy 2.4.15 (the default version >>> of Groovy for Gradle 4.10.2) is unable to work with Java 11 due to Nest >>> based access control problem [2]. The very same error happened when I tried >>> to compile "core" module with java11. This issue is solved in Groovy 2.5.3 >>> [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3 >>> (NoClassDefError) and found out that the support for Groovy 2.5.3 will be >>> added in Gradle 5.0RC1 (milestone) [4]. >>> >>> Other than that I had some issues with spottless and findbugs (I turned >>> them off for the sake of my experiment). >>> >>> Łukasz >>> >>> [1] https://issues.apache.org/jira/browse/BEAM-4042 >>> [2] https://github.com/gradle/gradle/issues/5120#issuecomment-424610114 >>> [3] https://issues.apache.org/jira/browse/GROOVY-8727 >>> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw> >>> [4] https://github.com/gradle/gradle/pull/7376 >>> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q> >>> >>> >>> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lukasz.gaj...@gmail.com> >>> napisał(a): >>> >>>> FWIW, the issue regarding java 11 support to gradle might be helpful >>>> here [1]. Quoting: "there are only some minor corner cases that don't work. >>>> Overall Java 11 support is pretty solid already" but some users reported >>>> problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 (this >>>> might be important for us). >>>> >>>> Łukasz >>>> >>>> [1] https://github.com/gradle/gradle/issues/5120 >>>> >>>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pabl...@google.com> >>>> napisał(a): >>>> >>>>> Hello all, >>>>> If I understand you correctly Ismael, a good amount of >>>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the >>>>> amount >>>>> of work necessary on the core module should be relatively small. Is this >>>>> correct? Are there improvements that may be missing in terms of >>>>> modularization? >>>>> >>>>> There is also the work necessary to build/run tests with Gradle.... >>>>> >>>>> I am also curious... how much work do you estimate is necessary to >>>>> support Java 11 with some of the existing sources? I understand that we >>>>> have many, many sources, but perhaps some of the more popular ones (e.g. >>>>> TextIO)? >>>>> >>>>> Thanks! >>>>> -P. >>>>> >>>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <arifka...@google.com> >>>>> wrote: >>>>> >>>>>> Thanks for the clarification Ismaël. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> * • **Arif Kasim* >>>>>> * • * Strategic Cloud Engineer >>>>>> * • *Google, Inc. >>>>>> • arifka...@google.com >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ieme...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Just wanted to clarify, there is already a JIRA for ongoing work on >>>>>>> Java 11 support. >>>>>>> https://issues.apache.org/jira/browse/BEAM-2530 >>>>>>> >>>>>>> I led the initial work on supporting what at the time was Java 9/10, >>>>>>> so far the biggest blockers were around the ApiSurface tests (not at >>>>>>> all compatible with these versions) but at the time we were at 5 >>>>>>> tests >>>>>>> from getting sdks/core passing. Notice also that the scope of this >>>>>>> JIRA evolved to support only the LTS version (Java 11), and >>>>>>> specifically to support only sdks/core + direct runner. Supporting >>>>>>> all >>>>>>> IOs or runners really is more a question of the dependencies working >>>>>>> nicely with Java 11 so this will probably take long time. Also the >>>>>>> idea so far does NOT include supporting the Java module system at >>>>>>> all. >>>>>>> >>>>>>> I stopped working on this during the move to gradle because it was >>>>>>> too >>>>>>> hard to tackle both Java evolving and all the ongoing changes in the >>>>>>> build system. If somebody in the community wants to contribute in >>>>>>> this >>>>>>> area it will be greatly appreciated, notice that all the work we did >>>>>>> on the build system for this needs to be implemented now in gradle >>>>>>> too. >>>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau < >>>>>>> rmannibu...@gmail.com> wrote: >>>>>>> > >>>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to inject >>>>>>> the proxy class is. There are other strategies you can use in bytebuddy >>>>>>> which work. >>>>>>> > >>>>>>> > Romain Manni-Bucau >>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>> > >>>>>>> > >>>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a >>>>>>> écrit : >>>>>>> >> >>>>>>> >> Romain, do you have any more details on the ByteBuddy >>>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just >>>>>>> with new language features? >>>>>>> >> >>>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau < >>>>>>> rmannibu...@gmail.com> wrote: >>>>>>> >>> >>>>>>> >>> Hi Arif, >>>>>>> >>> >>>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs >>>>>>> (but it means your pipeline is very very simple since it does not have a >>>>>>> dofn ;)) if your engine supports it. Also note that the modules not >>>>>>> being >>>>>>> named you can have to use some weird import names or even unstable ones >>>>>>> if >>>>>>> you want to use modules (but there is no real reason to do that yet in >>>>>>> java). >>>>>>> >>> >>>>>>> >>> Romain Manni-Bucau >>>>>>> >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <arifka...@google.com> >>>>>>> a écrit : >>>>>>> >>>> >>>>>>> >>>> Hello, >>>>>>> >>>> What's the status of java version > 8 support for beam? Thanks. >>>>>>> >>>> >>>>>>> >>>> -Arif. >>>>>>> >>>>>>
