On Tue, Jul 14, 2020 at 2:22 PM Ismaël Mejía <[email protected]> wrote:
> We should read the ticket Kyle mentions with a grain of salt. Most of the > sub-tasks in that ticket are NOT about allowing users to run pipelines > with Java > 11 but about been able to fully build and run the tests and the source code > ofBeam with Java 11 which is a different goal (important but probably less > for > end users) and a task with lots of extra issues because of plugins / > dependent > systems etc. > Actually only the tests! Pawel set it up so that the main jars are built with Java 8 and then just the tests build and run with Java 11. So this is more like what a "real" user would do while Beam is still building with Java 8. Most of them have been fixed except runners, which we would not expect anyhow, and the core SDK was not broken, only the tests. Happily, there aren't IO unit tests failing except XmlO. I think to get the runners working we need to re-organize. Right now gradle project properties control how the tests are run, but what we really need for this is independent tests for different JREs that we can opt-in or opt-out of per module. I agree with Ismaël that releasing early is OK. I would propose an easier criteria of having continuous integration tests running. The code might have bugs, but at least we know that we won't break it more, and when we fix a bug it stays fixed. I think also a simple opt-in flag like --experiments=java11 could help users to know that things might break or be weird. But really we are so close maybe we don't need all that formality. Kenn > > For the Java 11 harness what we need is to guarantee is that users can run > their > code without issues with Java 11 and we can do this now for example by > checking > that portable runners that support Java 11 pass ValidatesRunner with the > Java 11 > harness. Since some classic runners [1] already pass these tests, it > should be > relatively 'easy' to do so for portable runners. > > [1] > https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/ > > > > > On Sat, Jul 11, 2020 at 12:43 AM Ahmet Altay <[email protected]> wrote: > > > > Related to the naming question, +1 and this will be similar to the > python container naming (e.g. beam_python3.7_sdk). > > > > On Fri, Jul 10, 2020 at 1:46 PM Pablo Estrada <[email protected]> > wrote: > >> > >> I agree with Kenn. Dataflow already has some publishing of non-portable > JAva 11 containers, so I think it'll be great to formalize the process for > portable containers, and let users play with it, and know of its > availability. > >> Best > >> -P. > >> > >> On Fri, Jul 10, 2020 at 9:42 AM Kenneth Knowles <[email protected]> > wrote: > >>> > >>> To the initial question: I'm +1 on the rename. The container is > primarily something that the SDK should insert into the pipeline proto > during construction, and only user-facing in more specialized situations. > Given the state of Java and portability, it is a good time to get things > named properly and unambiguously. I think a brief announce to dev@ and > user@ when it happens is nice-to-have, but no need to give advance > warning. > >>> > >>> Kenn > >>> > >>> On Fri, Jul 10, 2020 at 7:58 AM Kenneth Knowles <[email protected]> > wrote: > >>>> > >>>> I believe Beam already has quite a few users that have forged ahead > and used Java 11 with various runners, pre-portability. Mostly I believe > the Java 11 limitations are with particular features (Schema codegen) and > extensions/IOs/transitive deps. > >>>> > >>>> When it comes to the container, I'd be interested in looking at test > coverage. The Flink & Spark portable ValidatesRunner suites use EMBEDDED > environment, so they don't exercise the container. The first testing of the > Java SDK harness container against the Python-based Universal Local Runner > is in pull request now [1]. Are there other test suites to highlight? How > hard would it be to run Flink & Spark against the container(s) too? > >>>> > >>>> Kenn > >>>> > >>>> [1] https://github.com/apache/beam/pull/11792 (despite the name > ValidatesRunner, in this case it is validating both the runner and harness, > since we don't have a compliance test suite for SDK harnesses) > >>>> > >>>> On Fri, Jul 10, 2020 at 7:54 AM Tyson Hamilton <[email protected]> > wrote: > >>>>> > >>>>> What do we consider 'ready'? > >>>>> > >>>>> Maybe the only required outstanding bugs are supporting the direct > runner (BEAM-10085), core tests (BEAM-10081), IO tests (BEAM-10084) to > start with? Notably this would exclude failing tests like those for GCP > core, GCPIOs, Dataflow runner, Spark runner, Flink runner, Samza. > >>>>> > >>>>> > >>>>> On Thu, Jul 9, 2020 at 4:44 PM Kyle Weaver <[email protected]> > wrote: > >>>>>> > >>>>>> My main question is, are we confident the Java 11 container is > ready to release? AFAIK there are still a number of issues blocking full > Java 11 support (cf [1]; not sure how many of these, if any, affect the SDK > harness specifically though.) > >>>>>> > >>>>>> For comparison, we recently decided to stop publishing Go SDK > containers until the Go SDK is considered mature [2]. In the meantime, > those who want to use the Go SDK can build their own container images from > source. > >>>>>> > >>>>>> Do we already have a Gradle task to build Java 11 containers? If > not, this would be a good intermediate step, letting users opt-in to Java > 11 without us overpromising support. > >>>>> > >>>>> > >>>>> We do not. From what I can tell, the build.gradele [1] for the Java > container is only for the one version. There is a docker file used for > Jenkins tests. > >>>>> > >>>>> [1] > https://github.com/apache/beam/blob/master/sdks/java/container/build.gradle > >>>>> > >>>>>> > >>>>>> > >>>>>> When we eventually do the renaming, we can add a note to CHANGES.md > [3]. > >>>>>> > >>>>>> [1] https://issues.apache.org/jira/browse/BEAM-10090 > >>>>>> [2] https://issues.apache.org/jira/browse/BEAM-9685 > >>>>>> [3] https://github.com/apache/beam/blob/master/CHANGES.md > >>>>>> > >>>>>> On Thu, Jul 9, 2020 at 3:44 PM Emily Ye <[email protected]> wrote: > >>>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I'm getting ramped up on contributing and was looking into adding > the Java 11 harness container to releases ( > https://issues.apache.org/jira/browse/BEAM-8106) - should I rename the > current java container so we have two new images `beam_java8_sdk` and > `beam_java11_sdk` or hold off on renaming? If we do rename it, what steps > should I take to announce/document the change? > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Emily >
