To the initial question: I'm +1 on the rename. The container is primarily something that the SDK should insert into the pipeline proto during construction, and only user-facing in more specialized situations. Given the state of Java and portability, it is a good time to get things named properly and unambiguously. I think a brief announce to dev@ and user@ when it happens is nice-to-have, but no need to give advance warning.
Kenn On Fri, Jul 10, 2020 at 7:58 AM Kenneth Knowles <[email protected]> wrote: > I believe Beam already has quite a few users that have forged ahead and > used Java 11 with various runners, pre-portability. Mostly I believe the > Java 11 limitations are with particular features (Schema codegen) and > extensions/IOs/transitive deps. > > When it comes to the container, I'd be interested in looking at test > coverage. The Flink & Spark portable ValidatesRunner suites use EMBEDDED > environment, so they don't exercise the container. The first testing of the > Java SDK harness container against the Python-based Universal Local Runner > is in pull request now [1]. Are there other test suites to highlight? How > hard would it be to run Flink & Spark against the container(s) too? > > Kenn > > [1] https://github.com/apache/beam/pull/11792 (despite the name > ValidatesRunner, in this case it is validating both the runner and harness, > since we don't have a compliance test suite for SDK harnesses) > > On Fri, Jul 10, 2020 at 7:54 AM Tyson Hamilton <[email protected]> wrote: > >> What do we consider 'ready'? >> >> Maybe the only required outstanding bugs are supporting the direct runner >> (BEAM-10085), core tests (BEAM-10081), IO tests (BEAM-10084) to start >> with? Notably this would exclude failing tests like those for GCP core, >> GCPIOs, Dataflow runner, Spark runner, Flink runner, Samza. >> >> >> On Thu, Jul 9, 2020 at 4:44 PM Kyle Weaver <[email protected]> wrote: >> >>> My main question is, are we confident the Java 11 container is ready to >>> release? AFAIK there are still a number of issues blocking full Java 11 >>> support (cf [1] <https://issues.apache.org/jira/browse/BEAM-10090>; not >>> sure how many of these, if any, affect the SDK harness specifically though.) >>> >>> For comparison, we recently decided to stop publishing Go SDK containers >>> until the Go SDK is considered mature [2]. In the meantime, those who want >>> to use the Go SDK can build their own container images from source. >>> >>> Do we already have a Gradle task to build Java 11 containers? If not, >>> this would be a good intermediate step, letting users opt-in to Java >>> 11 without us overpromising support. >>> >> >> We do not. From what I can tell, the build.gradele [1] for the Java >> container is only for the one version. There is a docker file used for >> Jenkins tests. >> >> [1] >> https://github.com/apache/beam/blob/master/sdks/java/container/build.gradle >> >> >>> >> When we eventually do the renaming, we can add a note to CHANGES.md [3]. >>> >>> [1] https://issues.apache.org/jira/browse/BEAM-10090 >>> [2] https://issues.apache.org/jira/browse/BEAM-9685 >>> [3] https://github.com/apache/beam/blob/master/CHANGES.md >>> >>> On Thu, Jul 9, 2020 at 3:44 PM Emily Ye <[email protected]> wrote: >>> >>>> Hi all, >>>> >>>> I'm getting ramped up on contributing and was looking into adding the >>>> Java 11 harness container to releases ( >>>> https://issues.apache.org/jira/browse/BEAM-8106) - should I rename the >>>> current java container so we have two new images `beam_java8_sdk` and >>>> `beam_java11_sdk` or hold off on renaming? If we do rename it, what steps >>>> should I take to announce/document the change? >>>> >>>> Thanks, >>>> Emily >>>> >>>
