Good to know. I added an icon and link. Good pointer about trying to see what we can do to make these official.
On Tue, Nov 19, 2019 at 11:16 AM Kyle Weaver <[email protected]> wrote: > > We do link from https://beam.apache.org/documentation/runtime/environments/. > > On Tue, Nov 19, 2019 at 10:59 AM Robert Bradshaw <[email protected]> wrote: >> >> We should probably add a link to these from our site as well, for visibility. >> >> On Tue, Nov 19, 2019 at 10:56 AM Kyle Weaver <[email protected]> wrote: >> > >> > +1 Thanks for bringing that up Chad, I had the same problem locating the >> > docker images on Docker hub (searching "apachebeam" is the only way that >> > seems to work). >> > >> > Another little thing I noticed is that https://hub.docker.com/u/apachebeam >> > is mostly empty. It'd be good to at least add the Beam logo as a profile >> > picture and a link to the website in the description. >> > >> > On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova <[email protected]> wrote: >> >> >> >> Hi all, >> >> I think it might be good to update the description of the beam docker >> >> images and add some descriptive tags, because searching for "apache beam" >> >> in docker hub does not turn up anything: >> >> https://hub.docker.com/search?q=apache%20beam&type=image. >> >> >> >> I clicked through 10 pages worth and couldn't find it. Maybe I missed >> >> something, but it clearly shouldn't be this hard. I did eventually >> >> manage to find it through the docs. >> >> >> >> Also, googling "apache beam python docker" also does not yield anything >> >> useful. In fact, it turns up an unofficial apache beam docker hub image. >> >> >> >> One thing I noticed is that images designated as "Official Images" get >> >> listed first, so it would be good to get that done as well. >> >> >> >> thanks! >> >> chad >> >> >> >> >> >> >> >> >> >> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang <[email protected]> >> >> wrote: >> >>> >> >>> Hi team >> >>> >> >>> I haven't received any objections, so will proceed with settings >> >>> mentioned in a previous email. >> >>> >> >>> A reminder to PMC members, please let me know your docker hub id if you >> >>> want to be an admin. >> >>> >> >>> Thanks, >> >>> Hannah >> >>> >> >>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka <[email protected]> wrote: >> >>>> >> >>>> Please ignore the previous email. I was looking at the older document >> >>>> in the mail thread. >> >>>> >> >>>> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka <[email protected]> wrote: >> >>>>> >> >>>>> I think sdk in the name is obsolete as they are all under sdks name >> >>>>> space. >> >>>>> >> >>>>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <[email protected]> >> >>>>> wrote: >> >>>>>> >> >>>>>> Hi Team >> >>>>>> >> >>>>>> Thanks for all the comments about beam containers. >> >>>>>> After considering various opinions and investigating gcr and docker >> >>>>>> hub, we decided to push images to docker hub. >> >>>>>> >> >>>>>> Each image will have two tags, {version}_rc and {version}. {version} >> >>>>>> tag will be added after the release candidate image is verified. >> >>>>>> Meanwhile, we will have latest tag for each repository, which always >> >>>>>> points to the most recent verified release image, so users can pull >> >>>>>> it by default. >> >>>>>> >> >>>>>> Docker hub doesn't support leveled repository, which means we should >> >>>>>> follow repository:tag format. >> >>>>>> it's too general if we use {language_version} as repository for SDK >> >>>>>> images. (version is added when we support multiple versions.) >> >>>>>> So I would like to include sdk to repository. Images generated at >> >>>>>> local will also have the same name. >> >>>>>> Here are some examples: >> >>>>>> >> >>>>>> python2.7_sdk:2.15.0 >> >>>>>> java_sdk:2.15.0_rc >> >>>>>> go_sdk:latest >> >>>>>> >> >>>>>> I will proceed with this format if there is no strong opposition by >> >>>>>> tomorrow noon(PST). >> >>>>>> >> >>>>>> To PMC members: >> >>>>>> Permission control will follow the pypi model. All interested PMC >> >>>>>> members will be added as admins and release managers will be granted >> >>>>>> push permission. >> >>>>>> Please let me know your docker id if you want to be added as an admin. >> >>>>>> >> >>>>>> Thanks, >> >>>>>> Hannah >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <[email protected]> wrote: >> >>>>>>> >> >>>>>>> This will greatly simplify trying out portable runners: >> >>>>>>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster >> >>>>>>> >> >>>>>>> Can't wait for following to disappear from the instructions page: >> >>>>>>> ./gradlew :sdks:python:container:docker >> >>>>>>> >> >>>>>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <[email protected]> wrote: >> >>>>>>>> >> >>>>>>>> Awesome, thank you! >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang >> >>>>>>>> <[email protected]> wrote: >> >>>>>>>>> >> >>>>>>>>> Hi Thomas >> >>>>>>>>> >> >>>>>>>>> I created snapshot images from head as of around 2PM today. >> >>>>>>>>> You can pull images from >> >>>>>>>>> gcr.io/apache-beam-testing/beam/sdks/snapshot. >> >>>>>>>>> >> >>>>>>>>> Thanks, >> >>>>>>>>> Hannah >> >>>>>>>>> >> >>>>>>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <[email protected]> >> >>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>> Hi Hannah, >> >>>>>>>>>> >> >>>>>>>>>> Thank you, I know how to build the containers locally, but not >> >>>>>>>>>> how to publish them! >> >>>>>>>>>> >> >>>>>>>>>> The cwiki says "Publishing images to gcr.io/beam requires >> >>>>>>>>>> permissions in apache-beam-testing project." >> >>>>>>>>>> >> >>>>>>>>>> Can I get access to the testing project (at least temporarily) >> >>>>>>>>>> and what would I need to setup to run the publish target that is >> >>>>>>>>>> shown on cwiki? >> >>>>>>>>>> >> >>>>>>>>>> Thanks, >> >>>>>>>>>> Thomas >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang >> >>>>>>>>>> <[email protected]> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>> Hi Thomas >> >>>>>>>>>>> >> >>>>>>>>>>> I haven't uploaded any snapshot images yet. Here is how you can >> >>>>>>>>>>> create one from head. >> >>>>>>>>>>> > cd [...]/beam/ >> >>>>>>>>>>> # For Python >> >>>>>>>>>>> > ./gradlew :sdks:python:container:py{version}:docker where >> >>>>>>>>>>> > version is {2,35,36,37} >> >>>>>>>>>>> # For Java >> >>>>>>>>>>> > ./gradlew -p sdks/java/container docker >> >>>>>>>>>>> # For Go >> >>>>>>>>>>> > ./gradlew -p sdks/go/container docker >> >>>>>>>>>>> >> >>>>>>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a >> >>>>>>>>>>> snapshot from head. >> >>>>>>>>>>> >> >>>>>>>>>>> Please let me know if you have any questions. >> >>>>>>>>>>> Hannah >> >>>>>>>>>>> >> >>>>>>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <[email protected]> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>> I actually found something in [1], but it is 2.15 unfortunately. >> >>>>>>>>>>>> >> >>>>>>>>>>>> [1] >> >>>>>>>>>>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30 >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <[email protected]> >> >>>>>>>>>>>> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Thanks for working on this. Do you happen to have publicly >> >>>>>>>>>>>>> accessible snapshots published for your testing currently >> >>>>>>>>>>>>> (even when the final location isn't sorted out)? >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> I would like to use a 2.16 based Python SDK image for working >> >>>>>>>>>>>>> on my downstream project, but could not find anything in >> >>>>>>>>>>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Thanks, >> >>>>>>>>>>>>> Thomas >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev >> >>>>>>>>>>>>> <[email protected]> wrote: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang >> >>>>>>>>>>>>>> <[email protected]> wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Hi team >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I am working on improving docker container support for Beam. >> >>>>>>>>>>>>>>> We would like to publish prebuilt containers for each >> >>>>>>>>>>>>>>> release version and daily snapshot. Current work focuses on >> >>>>>>>>>>>>>>> release images only and it would be part of the release >> >>>>>>>>>>>>>>> process. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> The release images will be pushed to GCR which is publicly >> >>>>>>>>>>>>>>> accessible(pullable). We will use the following locations. >> >>>>>>>>>>>>>>> Repository: gcr.io/beam >> >>>>>>>>>>>>>>> Project: apache-beam-testing >> >>>>>>>>>>>>>>> More details, including naming and tagging scheme, can be >> >>>>>>>>>>>>>>> found at wiki which is written by several contributors. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I would like to discuss these two questions. >> >>>>>>>>>>>>>>> 1. How many tests do we need to run before pushing images to >> >>>>>>>>>>>>>>> gcr? >> >>>>>>>>>>>>>>> Publishing artifacts is the last step of the release >> >>>>>>>>>>>>>>> process, so at this moment, we already verified all >> >>>>>>>>>>>>>>> codebase. In addition, many Jenkins tests use containers, so >> >>>>>>>>>>>>>>> it is already verified several times. Do we need to run it >> >>>>>>>>>>>>>>> again? >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> In a docker repository, one container image can have multiple >> >>>>>>>>>>>>>> tags. One possibility is that on the last step of the >> >>>>>>>>>>>>>> release process, after sufficient testing, we place a >> >>>>>>>>>>>>>> production tag on an image that was already pushed with a dev >> >>>>>>>>>>>>>> tag. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> For example a dev tag may look like: >> >>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag >> >>>>>>>>>>>>>> may look like: >> >>>>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the >> >>>>>>>>>>>>>> same image at the end. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> We should also plan what the process of updating the >> >>>>>>>>>>>>>> container image will look like, if we need to release the >> >>>>>>>>>>>>>> image with additional changes, and how we will test these >> >>>>>>>>>>>>>> changes before the final push (or placing production tag). >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> 2. How many tests do we need to run to validate pushed >> >>>>>>>>>>>>>>> images? >> >>>>>>>>>>>>>>> When we push the images, we assume the images would work and >> >>>>>>>>>>>>>>> pass all the tests. After pushing, we should confirm the >> >>>>>>>>>>>>>>> images are pullable and useable. I suggest we run several >> >>>>>>>>>>>>>>> tests on dataflow with each pushed image. What do you think? >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> I think it makes sense to do - Beam runners that use SDK >> >>>>>>>>>>>>>> container images should have some continuously running tests, >> >>>>>>>>>>>>>> which periodically check that all supported images are >> >>>>>>>>>>>>>> pullable and still compatible with the runner. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> This work can be refined later as we explore more during our >> >>>>>>>>>>>>>>> release process. >> >>>>>>>>>>>>>>> Please comment or edit the wiki page or reply to this email >> >>>>>>>>>>>>>>> with your opinions. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Thanks, >> >>>>>>>>>>>>>>> Hannah
