Hi team
I haven't received any objections, so will proceed with settings mentioned
in a previous email.
A reminder to PMC members, please let me know your docker hub id if you
want to be an admin.
Thanks,
Hannah
On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka <goe...@google.com> wrote:
> Please ignore the previous email. I was looking at the older document in
> the mail thread.
>
> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka <goe...@google.com> wrote:
>
>> I think sdk in the name is obsolete as they are all under sdks name space.
>>
>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang <hannahji...@google.com>
>> wrote:
>>
>>> Hi Team
>>>
>>> Thanks for all the comments about beam containers.
>>> After considering various opinions and investigating gcr and docker hub,
>>> we decided to push images to docker hub.
>>>
>>> Each image will have two tags, {version}_rc and {version}. {version} tag
>>> will be added after the release candidate image is verified.
>>> Meanwhile, we will have* latest* tag for each repository, which always
>>> points to the most recent verified release image, so users can pull it by
>>> default.
>>>
>>> Docker hub doesn't support leveled repository, which means we should
>>> follow *repository:tag* format.
>>> it's too general if we use {language_version} as repository for SDK
>>> images. (version is added when we support multiple versions.)
>>> So I would like to include *sdk* to repository. Images generated at
>>> local will also have the same name.
>>> Here are some examples:
>>>
>>> - python2.7_sdk:2.15.0
>>> - java_sdk:2.15.0_rc
>>> - go_sdk:latest
>>>
>>> I will proceed with this format if there is no strong opposition by
>>> tomorrow noon(PST).
>>>
>>> *To PMC members*:
>>> Permission control will follow the pypi model. All interested PMC
>>> members will be added as admins and release managers will be granted push
>>> permission.
>>> Please let me know your *docker id* if you want to be added as an admin.
>>>
>>> Thanks,
>>> Hannah
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise <t...@apache.org> wrote:
>>>
>>>> This will greatly simplify trying out portable runners:
>>>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>>>
>>>> Can't wait for following to disappear from the instructions page: ./gradlew
>>>> :sdks:python:container:docker
>>>>
>>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise <t...@apache.org> wrote:
>>>>
>>>>> Awesome, thank you!
>>>>>
>>>>>
>>>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <hannahji...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Thomas
>>>>>>
>>>>>> I created snapshot images from head as of around 2PM today.
>>>>>> You can pull images from
>>>>>> gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>>>>
>>>>>> Thanks,
>>>>>> Hannah
>>>>>>
>>>>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise <t...@apache.org> wrote:
>>>>>>
>>>>>>> Hi Hannah,
>>>>>>>
>>>>>>> Thank you, I know how to build the containers locally, but not how
>>>>>>> to publish them!
>>>>>>>
>>>>>>> The cwiki says "Publishing images to gcr.io/beam requires
>>>>>>> permissions in apache-beam-testing project."
>>>>>>>
>>>>>>> Can I get access to the testing project (at least temporarily) and
>>>>>>> what would I need to setup to run the publish target that is shown on
>>>>>>> cwiki?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <hannahji...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Thomas
>>>>>>>>
>>>>>>>> I haven't uploaded any snapshot images yet. Here is how you can
>>>>>>>> create one from head.
>>>>>>>> > cd [...]/beam/
>>>>>>>> # For Python
>>>>>>>> > ./gradlew :sdks:python:container:py{version}:docker *where
>>>>>>>> version is {2,35,36,37}*
>>>>>>>> # For Java
>>>>>>>> > ./gradlew -p sdks/java/container docker
>>>>>>>> # For Go
>>>>>>>> > ./gradlew -p sdks/go/container docker
>>>>>>>>
>>>>>>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
>>>>>>>> from head.
>>>>>>>>
>>>>>>>> Please let me know if you have any questions.
>>>>>>>> Hannah
>>>>>>>>
>>>>>>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise <t...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>>>>>>>
>>>>>>>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise <t...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for working on this. Do you happen to have publicly
>>>>>>>>>> accessible snapshots published for your testing currently (even when
>>>>>>>>>> the
>>>>>>>>>> final location isn't sorted out)?
>>>>>>>>>>
>>>>>>>>>> I would like to use a 2.16 based Python SDK image for working on
>>>>>>>>>> my downstream project, but could not find anything in
>>>>>>>>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>>>>>>>>> valen...@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>>>>>>> hannahji...@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi team
>>>>>>>>>>>>
>>>>>>>>>>>> I am working on improving docker container support for Beam. We
>>>>>>>>>>>> would like to publish prebuilt containers for each release version
>>>>>>>>>>>> and
>>>>>>>>>>>> daily snapshot. Current work focuses on release images only and it
>>>>>>>>>>>> would be
>>>>>>>>>>>> part of the release process.
>>>>>>>>>>>>
>>>>>>>>>>>> The release images will be pushed to GCR which is publicly
>>>>>>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>>>>>>> *Repository*: gcr.io/beam
>>>>>>>>>>>> *Project*: apache-beam-testing
>>>>>>>>>>>> More details, including naming and tagging scheme, can be found
>>>>>>>>>>>> at wiki
>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/%5BWIP%5D+SDKHarness+Container+Image+Release+Process>
>>>>>>>>>>>> which
>>>>>>>>>>>> is written by several contributors.
>>>>>>>>>>>>
>>>>>>>>>>>> I would like to discuss these two questions.
>>>>>>>>>>>> *1. How many tests do we need to run before pushing images to
>>>>>>>>>>>> gcr*?
>>>>>>>>>>>> Publishing artifacts is the last step of the release process,
>>>>>>>>>>>> so at this moment, we already verified all codebase. In addition,
>>>>>>>>>>>> many
>>>>>>>>>>>> Jenkins tests use containers, so it is already verified several
>>>>>>>>>>>> times. Do
>>>>>>>>>>>> we need to run it again?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In a docker repository, one container image can have multiple
>>>>>>>>>>> tags. One possibility is that on the last step of the release
>>>>>>>>>>> process,
>>>>>>>>>>> after sufficient testing, we place a production tag on an image
>>>>>>>>>>> that was
>>>>>>>>>>> already pushed with a dev tag.
>>>>>>>>>>>
>>>>>>>>>>> For example a dev tag may look like:
>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may
>>>>>>>>>>> look like:
>>>>>>>>>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the
>>>>>>>>>>> same image at the end.
>>>>>>>>>>>
>>>>>>>>>>> We should also plan what the process of updating the container
>>>>>>>>>>> image will look like, if we need to release the image with
>>>>>>>>>>> additional changes, and how we will test these changes before the
>>>>>>>>>>> final
>>>>>>>>>>> push (or placing production tag).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *2. How many tests do we need to run to validate pushed images?*
>>>>>>>>>>>> When we push the images, we assume the images would work and
>>>>>>>>>>>> pass all the tests. After pushing, we should confirm the images are
>>>>>>>>>>>> pullable and useable. I suggest we run several tests on dataflow
>>>>>>>>>>>> with each
>>>>>>>>>>>> pushed image. What do you think?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I think it makes sense to do - Beam runners that use SDK
>>>>>>>>>>> container images should have some continuously running tests, which
>>>>>>>>>>> periodically check that all supported images are pullable and still
>>>>>>>>>>> compatible with the runner.
>>>>>>>>>>>
>>>>>>>>>>> This work can be refined later as we explore more during our
>>>>>>>>>>>> release process.
>>>>>>>>>>>> Please comment or edit the wiki page or reply to this email
>>>>>>>>>>>> with your opinions.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Hannah
>>>>>>>>>>>>
>>>>>>>>>>>
