When we were supporting Java 7, there were Java 8 based modules in Maven. We can follow a similar pattern where Java11 (or any other version) have their own projects and as the base Java version moves up, we can merge those modules back in to simplify the set of modules being developed.
On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <ieme...@gmail.com> wrote: > I want to give more context on the previous work towards Java 11 support. > The issue was not necessarily that gradle was not ok to support Java 11, > but this is also a good point Lukasz. > > Months ago, we built a maven profile that basically set up Java to be > strictly compatible with the ‘minimal’ java.base profile and used Java > version (9/10) at the time. We found and fixed some missing dependencies > (because Java 11 removed some of the Java EE stuff from the base profile), > not sure if these changes were migrated to the gradle build. > > We also dealt with a good chunk of updates at the time of > plugins/bytebuddy and other deps to guarantee that the execution was done > strictly with the latest version of Java and its generated bytecode (the > origin of the docker build images was to guarantee this isolation). At the > time I run the tests of most IOs and most extensions with the existing > (java 8 compiled) direct runner in JRE 10 with success but of course this > was not the ultimate goal but a good way to evaluate the state of the code > base (I remember some GCP tests were broken and the ApiSurfaceTests too). > > One important thing to remember is that the goal on this work was > ‘limited’ to guarantee that a user that had ONLY Java 11 installed could > build and generate Beam artifacts and run Beam programs in the direct > runner. The limited scope was in part because of some issues of a full > migration to Java 11: > > - Support for Java modules, this is a too much radical change and probably > far from the scope of Beam at the moment. > - Support for other runners, given that most execution systems do not > support yet Java 11 this could imply a lot of extra work. In the meantime, > this could be easier now due to the portability work, anyway, still not > sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon. > > Let’s not forget that we still are and probably will be for a long time > backwards compatible with Java 8 so we need to ensure that Java 11 only > things don’t get into the code base. > > If you Lukasz or someone else is up to take this task, I think the > immediate thing to do is to get back to the same state we were before the > move to gradle, build the equivalent of the maven profile for the gradle > build and tackle code generation and classpath issues, then try to fix > missing parts and push this into the CI too (similar to the recent python 3 > work) to guarantee that new changes don’t break Java 11 compatibility. I > sadly lack of bandwidth to lead this task but I will gladly help with > reviews, or feedback if someone else is up to the task. > > Not sure about more recent details, but probably worth evaluating at some > moment is if we there is some interest on supporting multi-release jars > with Java 11 classes or the dream scenario of having a AOT build of the > direct runner. > > On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy <lukasz.gaj...@gmail.com> > wrote: > >> FYI: If I'm correct, migrating to Java 11 may require Migrating to Gradle >> 5, which to me is ok, but there are some issues on the way (such as [1] or >> maybe more that I'm not aware of). >> >> I did some research and it seems that Groovy 2.4.15 (the default version >> of Groovy for Gradle 4.10.2) is unable to work with Java 11 due to Nest >> based access control problem [2]. The very same error happened when I tried >> to compile "core" module with java11. This issue is solved in Groovy 2.5.3 >> [3]. I was unable to run Gradle 4.10.2 with Groovy 2.5.3 >> (NoClassDefError) and found out that the support for Groovy 2.5.3 will be >> added in Gradle 5.0RC1 (milestone) [4]. >> >> Other than that I had some issues with spottless and findbugs (I turned >> them off for the sake of my experiment). >> >> Łukasz >> >> [1] https://issues.apache.org/jira/browse/BEAM-4042 >> [2] https://github.com/gradle/gradle/issues/5120#issuecomment-424610114 >> [3] https://issues.apache.org/jira/browse/GROOVY-8727 >> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/GROOVY-8727&sa=D&ust=1539701479825000&usg=AFQjCNGwndnRaDx9NAr0cJ-isZpRWngcjw> >> [4] https://github.com/gradle/gradle/pull/7376 >> <https://www.google.com/url?q=https://github.com/gradle/gradle/pull/7376&sa=D&ust=1539701479825000&usg=AFQjCNE9DOokgVxojvsvjsaJ8_6KY2M92Q> >> >> >> czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lukasz.gaj...@gmail.com> >> napisał(a): >> >>> FWIW, the issue regarding java 11 support to gradle might be helpful >>> here [1]. Quoting: "there are only some minor corner cases that don't work. >>> Overall Java 11 support is pretty solid already" but some users reported >>> problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 (this >>> might be important for us). >>> >>> Łukasz >>> >>> [1] https://github.com/gradle/gradle/issues/5120 >>> >>> czw., 11 paź 2018 o 02:06 Pablo Estrada <pabl...@google.com> napisał(a): >>> >>>> Hello all, >>>> If I understand you correctly Ismael, a good amount of >>>> 'beam-sdks-java-core' tests are already passing with Java 11, so the amount >>>> of work necessary on the core module should be relatively small. Is this >>>> correct? Are there improvements that may be missing in terms of >>>> modularization? >>>> >>>> There is also the work necessary to build/run tests with Gradle.... >>>> >>>> I am also curious... how much work do you estimate is necessary to >>>> support Java 11 with some of the existing sources? I understand that we >>>> have many, many sources, but perhaps some of the more popular ones (e.g. >>>> TextIO)? >>>> >>>> Thanks! >>>> -P. >>>> >>>> On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <arifka...@google.com> >>>> wrote: >>>> >>>>> Thanks for the clarification Ismaël. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> * • **Arif Kasim* >>>>> * • * Strategic Cloud Engineer >>>>> * • *Google, Inc. >>>>> • arifka...@google.com >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <ieme...@gmail.com> >>>>> wrote: >>>>> >>>>>> Just wanted to clarify, there is already a JIRA for ongoing work on >>>>>> Java 11 support. >>>>>> https://issues.apache.org/jira/browse/BEAM-2530 >>>>>> >>>>>> I led the initial work on supporting what at the time was Java 9/10, >>>>>> so far the biggest blockers were around the ApiSurface tests (not at >>>>>> all compatible with these versions) but at the time we were at 5 tests >>>>>> from getting sdks/core passing. Notice also that the scope of this >>>>>> JIRA evolved to support only the LTS version (Java 11), and >>>>>> specifically to support only sdks/core + direct runner. Supporting all >>>>>> IOs or runners really is more a question of the dependencies working >>>>>> nicely with Java 11 so this will probably take long time. Also the >>>>>> idea so far does NOT include supporting the Java module system at all. >>>>>> >>>>>> I stopped working on this during the move to gradle because it was too >>>>>> hard to tackle both Java evolving and all the ongoing changes in the >>>>>> build system. If somebody in the community wants to contribute in this >>>>>> area it will be greatly appreciated, notice that all the work we did >>>>>> on the build system for this needs to be implemented now in gradle >>>>>> too. >>>>>> On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau < >>>>>> rmannibu...@gmail.com> wrote: >>>>>> > >>>>>> > @Reuven: bytebuddy by itself no but the way beam tries to inject >>>>>> the proxy class is. There are other strategies you can use in bytebuddy >>>>>> which work. >>>>>> > >>>>>> > Romain Manni-Bucau >>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>> > >>>>>> > >>>>>> > Le sam. 6 oct. 2018 à 17:51, Reuven Lax <re...@google.com> a écrit >>>>>> : >>>>>> >> >>>>>> >> Romain, do you have any more details on the ByteBuddy >>>>>> incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just >>>>>> with new language features? >>>>>> >> >>>>>> >> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau < >>>>>> rmannibu...@gmail.com> wrote: >>>>>> >>> >>>>>> >>> Hi Arif, >>>>>> >>> >>>>>> >>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs >>>>>> (but it means your pipeline is very very simple since it does not have a >>>>>> dofn ;)) if your engine supports it. Also note that the modules not being >>>>>> named you can have to use some weird import names or even unstable ones >>>>>> if >>>>>> you want to use modules (but there is no real reason to do that yet in >>>>>> java). >>>>>> >>> >>>>>> >>> Romain Manni-Bucau >>>>>> >>> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book >>>>>> >>> >>>>>> >>> >>>>>> >>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <arifka...@google.com> a >>>>>> écrit : >>>>>> >>>> >>>>>> >>>> Hello, >>>>>> >>>> What's the status of java version > 8 support for beam? Thanks. >>>>>> >>>> >>>>>> >>>> -Arif. >>>>>> >>>>>
