Thanks Robert for the explanation. Is there any progress into getting back, any ticket people can follow if interested?
On Wed, Jul 15, 2020 at 12:13 AM Robert Burke <[email protected]> wrote: > > Disallowing the go containers was largely due to not having a simple check on > the go boot code's licenses which is required for containers hosted under the > main Apache namespace. > > A manual verification reveals it's only either Go's standard library BSD > license and GRPCs Apache v2 licenses. Not impossible but not yet done by us. > The JIRA issue has a link to the appropriate license finder for go packages. > > The amusing bit is that very similar Go boot code is included in the Java and > Python containers too, so we're only accidentally in compliance with that > there, if at all. > > > > On Tue, Jul 14, 2020, 2:22 PM Ismaël Mejía <[email protected]> wrote: >> >> +1 for naming as python containers, and quick release so users can try it. >> >> Not related to this tnread but I am also curious about the reasons to remove >> the >> go docker images, was this discussed/voted in the ML (maybe I missed it) ? >> >> I don't think Beam has been historically a conservative project about >> releasing >> early in-progress versions and I have learnt to appreciate this because it >> helps >> for early user testing and bug reports which will be definitely a must for >> Java >> 11. >> >> We should read the ticket Kyle mentions with a grain of salt. Most of the >> sub-tasks in that ticket are NOT about allowing users to run pipelines with >> Java >> 11 but about been able to fully build and run the tests and the source code >> ofBeam with Java 11 which is a different goal (important but probably less >> for >> end users) and a task with lots of extra issues because of plugins / >> dependent >> systems etc. >> >> For the Java 11 harness what we need is to guarantee is that users can run >> their >> code without issues with Java 11 and we can do this now for example by >> checking >> that portable runners that support Java 11 pass ValidatesRunner with the >> Java 11 >> harness. Since some classic runners [1] already pass these tests, it should >> be >> relatively 'easy' to do so for portable runners. >> >> [1] >> https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/ >> >> >> >> >> On Sat, Jul 11, 2020 at 12:43 AM Ahmet Altay <[email protected]> wrote: >> > >> > Related to the naming question, +1 and this will be similar to the python >> > container naming (e.g. beam_python3.7_sdk). >> > >> > On Fri, Jul 10, 2020 at 1:46 PM Pablo Estrada <[email protected]> wrote: >> >> >> >> I agree with Kenn. Dataflow already has some publishing of non-portable >> >> JAva 11 containers, so I think it'll be great to formalize the process >> >> for portable containers, and let users play with it, and know of its >> >> availability. >> >> Best >> >> -P. >> >> >> >> On Fri, Jul 10, 2020 at 9:42 AM Kenneth Knowles <[email protected]> wrote: >> >>> >> >>> To the initial question: I'm +1 on the rename. The container is >> >>> primarily something that the SDK should insert into the pipeline proto >> >>> during construction, and only user-facing in more specialized >> >>> situations. Given the state of Java and portability, it is a good time >> >>> to get things named properly and unambiguously. I think a brief announce >> >>> to dev@ and user@ when it happens is nice-to-have, but no need to give >> >>> advance warning. >> >>> >> >>> Kenn >> >>> >> >>> On Fri, Jul 10, 2020 at 7:58 AM Kenneth Knowles <[email protected]> wrote: >> >>>> >> >>>> I believe Beam already has quite a few users that have forged ahead and >> >>>> used Java 11 with various runners, pre-portability. Mostly I believe >> >>>> the Java 11 limitations are with particular features (Schema codegen) >> >>>> and extensions/IOs/transitive deps. >> >>>> >> >>>> When it comes to the container, I'd be interested in looking at test >> >>>> coverage. The Flink & Spark portable ValidatesRunner suites use >> >>>> EMBEDDED environment, so they don't exercise the container. The first >> >>>> testing of the Java SDK harness container against the Python-based >> >>>> Universal Local Runner is in pull request now [1]. Are there other test >> >>>> suites to highlight? How hard would it be to run Flink & Spark against >> >>>> the container(s) too? >> >>>> >> >>>> Kenn >> >>>> >> >>>> [1] https://github.com/apache/beam/pull/11792 (despite the name >> >>>> ValidatesRunner, in this case it is validating both the runner and >> >>>> harness, since we don't have a compliance test suite for SDK harnesses) >> >>>> >> >>>> On Fri, Jul 10, 2020 at 7:54 AM Tyson Hamilton <[email protected]> >> >>>> wrote: >> >>>>> >> >>>>> What do we consider 'ready'? >> >>>>> >> >>>>> Maybe the only required outstanding bugs are supporting the direct >> >>>>> runner (BEAM-10085), core tests (BEAM-10081), IO tests (BEAM-10084) >> >>>>> to start with? Notably this would exclude failing tests like those for >> >>>>> GCP core, GCPIOs, Dataflow runner, Spark runner, Flink runner, Samza. >> >>>>> >> >>>>> >> >>>>> On Thu, Jul 9, 2020 at 4:44 PM Kyle Weaver <[email protected]> wrote: >> >>>>>> >> >>>>>> My main question is, are we confident the Java 11 container is ready >> >>>>>> to release? AFAIK there are still a number of issues blocking full >> >>>>>> Java 11 support (cf [1]; not sure how many of these, if any, affect >> >>>>>> the SDK harness specifically though.) >> >>>>>> >> >>>>>> For comparison, we recently decided to stop publishing Go SDK >> >>>>>> containers until the Go SDK is considered mature [2]. In the >> >>>>>> meantime, those who want to use the Go SDK can build their own >> >>>>>> container images from source. >> >>>>>> >> >>>>>> Do we already have a Gradle task to build Java 11 containers? If not, >> >>>>>> this would be a good intermediate step, letting users opt-in to Java >> >>>>>> 11 without us overpromising support. >> >>>>> >> >>>>> >> >>>>> We do not. From what I can tell, the build.gradele [1] for the Java >> >>>>> container is only for the one version. There is a docker file used for >> >>>>> Jenkins tests. >> >>>>> >> >>>>> [1] >> >>>>> https://github.com/apache/beam/blob/master/sdks/java/container/build.gradle >> >>>>> >> >>>>>> >> >>>>>> >> >>>>>> When we eventually do the renaming, we can add a note to CHANGES.md >> >>>>>> [3]. >> >>>>>> >> >>>>>> [1] https://issues.apache.org/jira/browse/BEAM-10090 >> >>>>>> [2] https://issues.apache.org/jira/browse/BEAM-9685 >> >>>>>> [3] https://github.com/apache/beam/blob/master/CHANGES.md >> >>>>>> >> >>>>>> On Thu, Jul 9, 2020 at 3:44 PM Emily Ye <[email protected]> wrote: >> >>>>>>> >> >>>>>>> Hi all, >> >>>>>>> >> >>>>>>> I'm getting ramped up on contributing and was looking into adding >> >>>>>>> the Java 11 harness container to releases >> >>>>>>> (https://issues.apache.org/jira/browse/BEAM-8106) - should I rename >> >>>>>>> the current java container so we have two new images >> >>>>>>> `beam_java8_sdk` and `beam_java11_sdk` or hold off on renaming? If >> >>>>>>> we do rename it, what steps should I take to announce/document the >> >>>>>>> change? >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> Emily
