Re: Improve container support

[Robert Bradshaw]

Good to know. I added an icon and link. Good pointer about trying to
see what we can do to make these official.

On Tue, Nov 19, 2019 at 11:16 AM Kyle Weaver  wrote:
>
> We do link from https://beam.apache.org/documentation/runtime/environments/.
>
> On Tue, Nov 19, 2019 at 10:59 AM Robert Bradshaw  wrote:
>>
>> We should probably add a link to these from our site as well, for visibility.
>>
>> On Tue, Nov 19, 2019 at 10:56 AM Kyle Weaver  wrote:
>> >
>> > +1 Thanks for bringing that up Chad, I had the same problem locating the 
>> > docker images on Docker hub (searching "apachebeam" is the only way that 
>> > seems to work).
>> >
>> > Another little thing I noticed is that https://hub.docker.com/u/apachebeam 
>> > is mostly empty. It'd be good to at least add the Beam logo as a profile 
>> > picture and a link to the website in the description.
>> >
>> > On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova  wrote:
>> >>
>> >> Hi all,
>> >> I think it might be good to update the description of the beam docker 
>> >> images and add some descriptive tags, because searching for "apache beam" 
>> >> in docker hub does not turn up anything: 
>> >> https://hub.docker.com/search?q=apache%20beam=image.
>> >>
>> >> I clicked through 10 pages worth and couldn't find it.  Maybe I missed 
>> >> something, but it clearly shouldn't be this hard.  I did eventually 
>> >> manage to find it through the docs.
>> >>
>> >> Also, googling "apache beam python docker" also does not yield anything 
>> >> useful.  In fact, it turns up an unofficial apache beam docker hub image.
>> >>
>> >> One thing I noticed is that images designated as "Official Images" get 
>> >> listed first, so it would be good to get that done as well.
>> >>
>> >> thanks!
>> >> chad
>> >>
>> >>
>> >>
>> >>
>> >> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang  
>> >> wrote:
>> >>>
>> >>> Hi team
>> >>>
>> >>> I haven't received any objections, so will proceed with settings 
>> >>> mentioned in a previous email.
>> >>>
>> >>> A reminder to PMC members, please let me know your docker hub id if you 
>> >>> want to be an admin.
>> >>>
>> >>> Thanks,
>> >>> Hannah
>> >>>
>> >>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka  wrote:
>> 
>>  Please ignore the previous email. I was looking at the older document 
>>  in the mail thread.
>> 
>>  On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka  wrote:
>> >
>> > I think sdk in the name is obsolete as they are all under sdks name 
>> > space.
>> >
>> > On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang  
>> > wrote:
>> >>
>> >> Hi Team
>> >>
>> >> Thanks for all the comments about beam containers.
>> >> After considering various opinions and investigating gcr and docker 
>> >> hub, we decided to push images to docker hub.
>> >>
>> >> Each image will have two tags, {version}_rc and {version}. {version} 
>> >> tag will be added after the release candidate image is verified.
>> >> Meanwhile, we will have latest tag for each repository, which always 
>> >> points to the most recent verified release image, so users can pull 
>> >> it by default.
>> >>
>> >> Docker hub doesn't support leveled repository, which means we should 
>> >> follow repository:tag format.
>> >> it's too general if we use {language_version} as repository for SDK 
>> >> images. (version is added when we support multiple versions.)
>> >> So I would like to include sdk to repository. Images generated at 
>> >> local will also have the same name.
>> >> Here are some examples:
>> >>
>> >> python2.7_sdk:2.15.0
>> >> java_sdk:2.15.0_rc
>> >> go_sdk:latest
>> >>
>> >> I will proceed with this format if there is no strong opposition by 
>> >> tomorrow noon(PST).
>> >>
>> >> To PMC members:
>> >> Permission control will follow the pypi model. All interested PMC 
>> >> members will be added as admins and release managers will be granted 
>> >> push permission.
>> >> Please let me know your docker id if you want to be added as an admin.
>> >>
>> >> Thanks,
>> >> Hannah
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
>> >>>
>> >>> This will greatly simplify trying out portable runners: 
>> >>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>> >>>
>> >>> Can't wait for following to disappear from the instructions page: 
>> >>> ./gradlew :sdks:python:container:docker
>> >>>
>> >>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>> 
>>  Awesome, thank you!
>> 
>> 
>>  On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
>>   wrote:
>> >
>> > Hi Thomas
>> >
>> > I created snapshot images from head as of around 2PM today.
>> > You can pull images from 

Re: Improve container support

[Kyle Weaver]

We do link from https://beam.apache.org/documentation/runtime/environments/.

On Tue, Nov 19, 2019 at 10:59 AM Robert Bradshaw 
wrote:

> We should probably add a link to these from our site as well, for
> visibility.
>
> On Tue, Nov 19, 2019 at 10:56 AM Kyle Weaver  wrote:
> >
> > +1 Thanks for bringing that up Chad, I had the same problem locating the
> docker images on Docker hub (searching "apachebeam" is the only way that
> seems to work).
> >
> > Another little thing I noticed is that
> https://hub.docker.com/u/apachebeam is mostly empty. It'd be good to at
> least add the Beam logo as a profile picture and a link to the website in
> the description.
> >
> > On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova 
> wrote:
> >>
> >> Hi all,
> >> I think it might be good to update the description of the beam docker
> images and add some descriptive tags, because searching for "apache beam"
> in docker hub does not turn up anything:
> https://hub.docker.com/search?q=apache%20beam=image.
> >>
> >> I clicked through 10 pages worth and couldn't find it.  Maybe I missed
> something, but it clearly shouldn't be this hard.  I did eventually manage
> to find it through the docs.
> >>
> >> Also, googling "apache beam python docker" also does not yield anything
> useful.  In fact, it turns up an unofficial apache beam docker hub image.
> >>
> >> One thing I noticed is that images designated as "Official Images" get
> listed first, so it would be good to get that done as well.
> >>
> >> thanks!
> >> chad
> >>
> >>
> >>
> >>
> >> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang 
> wrote:
> >>>
> >>> Hi team
> >>>
> >>> I haven't received any objections, so will proceed with settings
> mentioned in a previous email.
> >>>
> >>> A reminder to PMC members, please let me know your docker hub id if
> you want to be an admin.
> >>>
> >>> Thanks,
> >>> Hannah
> >>>
> >>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka  wrote:
> 
>  Please ignore the previous email. I was looking at the older document
> in the mail thread.
> 
>  On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka 
> wrote:
> >
> > I think sdk in the name is obsolete as they are all under sdks name
> space.
> >
> > On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang 
> wrote:
> >>
> >> Hi Team
> >>
> >> Thanks for all the comments about beam containers.
> >> After considering various opinions and investigating gcr and docker
> hub, we decided to push images to docker hub.
> >>
> >> Each image will have two tags, {version}_rc and {version}.
> {version} tag will be added after the release candidate image is verified.
> >> Meanwhile, we will have latest tag for each repository, which
> always points to the most recent verified release image, so users can pull
> it by default.
> >>
> >> Docker hub doesn't support leveled repository, which means we
> should follow repository:tag format.
> >> it's too general if we use {language_version} as repository for SDK
> images. (version is added when we support multiple versions.)
> >> So I would like to include sdk to repository. Images generated at
> local will also have the same name.
> >> Here are some examples:
> >>
> >> python2.7_sdk:2.15.0
> >> java_sdk:2.15.0_rc
> >> go_sdk:latest
> >>
> >> I will proceed with this format if there is no strong opposition by
> tomorrow noon(PST).
> >>
> >> To PMC members:
> >> Permission control will follow the pypi model. All interested PMC
> members will be added as admins and release managers will be granted push
> permission.
> >> Please let me know your docker id if you want to be added as an
> admin.
> >>
> >> Thanks,
> >> Hannah
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
> >>>
> >>> This will greatly simplify trying out portable runners:
> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
> >>>
> >>> Can't wait for following to disappear from the instructions page:
> ./gradlew :sdks:python:container:docker
> >>>
> >>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise 
> wrote:
> 
>  Awesome, thank you!
> 
> 
>  On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <
> hannahji...@google.com> wrote:
> >
> > Hi Thomas
> >
> > I created snapshot images from head as of around 2PM today.
> > You can pull images from
> gcr.io/apache-beam-testing/beam/sdks/snapshot.
> >
> > Thanks,
> > Hannah
> >
> > On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise 
> wrote:
> >>
> >> Hi Hannah,
> >>
> >> Thank you, I know how to build the containers locally, but not
> how to publish them!
> >>
> >> The cwiki says "Publishing images to gcr.io/beam requires
> permissions in apache-beam-testing 

Re: Improve container support

[Robert Bradshaw]

We should probably add a link to these from our site as well, for visibility.

On Tue, Nov 19, 2019 at 10:56 AM Kyle Weaver  wrote:
>
> +1 Thanks for bringing that up Chad, I had the same problem locating the 
> docker images on Docker hub (searching "apachebeam" is the only way that 
> seems to work).
>
> Another little thing I noticed is that https://hub.docker.com/u/apachebeam is 
> mostly empty. It'd be good to at least add the Beam logo as a profile picture 
> and a link to the website in the description.
>
> On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova  wrote:
>>
>> Hi all,
>> I think it might be good to update the description of the beam docker images 
>> and add some descriptive tags, because searching for "apache beam" in docker 
>> hub does not turn up anything: 
>> https://hub.docker.com/search?q=apache%20beam=image.
>>
>> I clicked through 10 pages worth and couldn't find it.  Maybe I missed 
>> something, but it clearly shouldn't be this hard.  I did eventually manage 
>> to find it through the docs.
>>
>> Also, googling "apache beam python docker" also does not yield anything 
>> useful.  In fact, it turns up an unofficial apache beam docker hub image.
>>
>> One thing I noticed is that images designated as "Official Images" get 
>> listed first, so it would be good to get that done as well.
>>
>> thanks!
>> chad
>>
>>
>>
>>
>> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang  wrote:
>>>
>>> Hi team
>>>
>>> I haven't received any objections, so will proceed with settings mentioned 
>>> in a previous email.
>>>
>>> A reminder to PMC members, please let me know your docker hub id if you 
>>> want to be an admin.
>>>
>>> Thanks,
>>> Hannah
>>>
>>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka  wrote:

 Please ignore the previous email. I was looking at the older document in 
 the mail thread.

 On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka  wrote:
>
> I think sdk in the name is obsolete as they are all under sdks name space.
>
> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang  
> wrote:
>>
>> Hi Team
>>
>> Thanks for all the comments about beam containers.
>> After considering various opinions and investigating gcr and docker hub, 
>> we decided to push images to docker hub.
>>
>> Each image will have two tags, {version}_rc and {version}. {version} tag 
>> will be added after the release candidate image is verified.
>> Meanwhile, we will have latest tag for each repository, which always 
>> points to the most recent verified release image, so users can pull it 
>> by default.
>>
>> Docker hub doesn't support leveled repository, which means we should 
>> follow repository:tag format.
>> it's too general if we use {language_version} as repository for SDK 
>> images. (version is added when we support multiple versions.)
>> So I would like to include sdk to repository. Images generated at local 
>> will also have the same name.
>> Here are some examples:
>>
>> python2.7_sdk:2.15.0
>> java_sdk:2.15.0_rc
>> go_sdk:latest
>>
>> I will proceed with this format if there is no strong opposition by 
>> tomorrow noon(PST).
>>
>> To PMC members:
>> Permission control will follow the pypi model. All interested PMC 
>> members will be added as admins and release managers will be granted 
>> push permission.
>> Please let me know your docker id if you want to be added as an admin.
>>
>> Thanks,
>> Hannah
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
>>>
>>> This will greatly simplify trying out portable runners: 
>>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>>
>>> Can't wait for following to disappear from the instructions page: 
>>> ./gradlew :sdks:python:container:docker
>>>
>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:

 Awesome, thank you!


 On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang  
 wrote:
>
> Hi Thomas
>
> I created snapshot images from head as of around 2PM today.
> You can pull images from 
> gcr.io/apache-beam-testing/beam/sdks/snapshot.
>
> Thanks,
> Hannah
>
> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:
>>
>> Hi Hannah,
>>
>> Thank you, I know how to build the containers locally, but not how 
>> to publish them!
>>
>> The cwiki says "Publishing images to gcr.io/beam requires 
>> permissions in apache-beam-testing project."
>>
>> Can I get access to the testing project (at least temporarily) and 
>> what would I need to setup to run the publish target that is shown 
>> on cwiki?
>>
>> Thanks,
>> Thomas

Re: Improve container support

[Kyle Weaver]

+1 Thanks for bringing that up Chad, I had the same problem locating the
docker images on Docker hub (searching "apachebeam" is the only way that
seems to work).

Another little thing I noticed is that https://hub.docker.com/u/apachebeam is
mostly empty. It'd be good to at least add the Beam logo as a profile
picture and a link to the website in the description.

On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova  wrote:

> Hi all,
> I think it might be good to update the description of the beam docker
> images and add some descriptive tags, because searching for "apache beam"
> in docker hub does not turn up anything:
> https://hub.docker.com/search?q=apache%20beam=image.
>
> I clicked through 10 pages worth and couldn't find it.  Maybe I missed
> something, but it clearly shouldn't be this hard.  I did eventually manage
> to find it through the docs.
>
> Also, googling "apache beam python docker" also does not yield anything
> useful.  In fact, it turns up an *unofficial* apache beam docker hub
> image.
>
> One thing I noticed is that images designated as "Official Images" get
> listed first, so it would be good to get that done as well.
>
> thanks!
> chad
>
>
>
>
> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang 
> wrote:
>
>> Hi team
>>
>> I haven't received any objections, so will proceed with settings
>> mentioned in a previous email.
>>
>> A reminder to PMC members, please let me know your docker hub id if you
>> want to be an admin.
>>
>> Thanks,
>> Hannah
>>
>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka  wrote:
>>
>>> Please ignore the previous email. I was looking at the older document in
>>> the mail thread.
>>>
>>> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka  wrote:
>>>
 I think sdk in the name is obsolete as they are all under sdks name
 space.

 On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang 
 wrote:

> Hi Team
>
> Thanks for all the comments about beam containers.
> After considering various opinions and investigating gcr and docker
> hub, we decided to push images to docker hub.
>
> Each image will have two tags, {version}_rc and {version}. {version}
> tag will be added after the release candidate image is verified.
> Meanwhile, we will have* latest* tag for each repository, which
> always points to the most recent verified release image, so users can pull
> it by default.
>
> Docker hub doesn't support leveled repository, which means we should
> follow *repository:tag* format.
> it's too general if we use {language_version} as repository for SDK
> images. (version is added when we support multiple versions.)
> So I would like to include *sdk* to repository. Images generated at
> local will also have the same name.
> Here are some examples:
>
>- python2.7_sdk:2.15.0
>- java_sdk:2.15.0_rc
>- go_sdk:latest
>
> I will proceed with this format if there is no strong opposition by
> tomorrow noon(PST).
>
> *To PMC members*:
> Permission control will follow the pypi model. All interested PMC
> members will be added as admins and release managers will be granted push
> permission.
> Please let me know your *docker id* if you want to be added as an
> admin.
>
> Thanks,
> Hannah
>
>
>
>
>
>
>
>
> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
>
>> This will greatly simplify trying out portable runners:
>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>
>> Can't wait for following to disappear from the instructions page: 
>> ./gradlew
>> :sdks:python:container:docker
>>
>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>>
>>> Awesome, thank you!
>>>
>>>
>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
>>> wrote:
>>>
 Hi Thomas

 I created snapshot images from head as of around 2PM today.
 You can pull images from
 gcr.io/apache-beam-testing/beam/sdks/snapshot.

 Thanks,
 Hannah

 On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:

> Hi Hannah,
>
> Thank you, I know how to build the containers locally, but not how
> to publish them!
>
> The cwiki says "Publishing images to gcr.io/beam requires
> permissions in apache-beam-testing project."
>
> Can I get access to the testing project (at least temporarily) and
> what would I need to setup to run the publish target that is shown on 
> cwiki?
>
> Thanks,
> Thomas
>
>
> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <
> hannahji...@google.com> wrote:
>
>> Hi Thomas
>>
>> I haven't uploaded any snapshot images yet. Here is how you can
>> create one from head.

Re: Improve container support

[Chad Dombrova]

Hi all,
I think it might be good to update the description of the beam docker
images and add some descriptive tags, because searching for "apache beam"
in docker hub does not turn up anything:
https://hub.docker.com/search?q=apache%20beam=image.

I clicked through 10 pages worth and couldn't find it.  Maybe I missed
something, but it clearly shouldn't be this hard.  I did eventually manage
to find it through the docs.

Also, googling "apache beam python docker" also does not yield anything
useful.  In fact, it turns up an *unofficial* apache beam docker hub image.

One thing I noticed is that images designated as "Official Images" get
listed first, so it would be good to get that done as well.

thanks!
chad




On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang  wrote:

> Hi team
>
> I haven't received any objections, so will proceed with settings mentioned
> in a previous email.
>
> A reminder to PMC members, please let me know your docker hub id if you
> want to be an admin.
>
> Thanks,
> Hannah
>
> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka  wrote:
>
>> Please ignore the previous email. I was looking at the older document in
>> the mail thread.
>>
>> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka  wrote:
>>
>>> I think sdk in the name is obsolete as they are all under sdks name
>>> space.
>>>
>>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang 
>>> wrote:
>>>
 Hi Team

 Thanks for all the comments about beam containers.
 After considering various opinions and investigating gcr and docker
 hub, we decided to push images to docker hub.

 Each image will have two tags, {version}_rc and {version}. {version}
 tag will be added after the release candidate image is verified.
 Meanwhile, we will have* latest* tag for each repository, which always
 points to the most recent verified release image, so users can pull it by
 default.

 Docker hub doesn't support leveled repository, which means we should
 follow *repository:tag* format.
 it's too general if we use {language_version} as repository for SDK
 images. (version is added when we support multiple versions.)
 So I would like to include *sdk* to repository. Images generated at
 local will also have the same name.
 Here are some examples:

- python2.7_sdk:2.15.0
- java_sdk:2.15.0_rc
- go_sdk:latest

 I will proceed with this format if there is no strong opposition by
 tomorrow noon(PST).

 *To PMC members*:
 Permission control will follow the pypi model. All interested PMC
 members will be added as admins and release managers will be granted push
 permission.
 Please let me know your *docker id* if you want to be added as an
 admin.

 Thanks,
 Hannah








 On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:

> This will greatly simplify trying out portable runners:
> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>
> Can't wait for following to disappear from the instructions page: 
> ./gradlew
> :sdks:python:container:docker
>
> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>
>> Awesome, thank you!
>>
>>
>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
>> wrote:
>>
>>> Hi Thomas
>>>
>>> I created snapshot images from head as of around 2PM today.
>>> You can pull images from
>>> gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>
>>> Thanks,
>>> Hannah
>>>
>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:
>>>
 Hi Hannah,

 Thank you, I know how to build the containers locally, but not how
 to publish them!

 The cwiki says "Publishing images to gcr.io/beam requires
 permissions in apache-beam-testing project."

 Can I get access to the testing project (at least temporarily) and
 what would I need to setup to run the publish target that is shown on 
 cwiki?

 Thanks,
 Thomas


 On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <
 hannahji...@google.com> wrote:

> Hi Thomas
>
> I haven't uploaded any snapshot images yet. Here is how you can
> create one from head.
> > cd [...]/beam/
> # For Python
> > ./gradlew :sdks:python:container:py{version}:docker *where
> version is {2,35,36,37}*
> # For Java
> > ./gradlew -p sdks/java/container docker
> # For Go
> > ./gradlew -p sdks/go/container docker
>
> The 2.15 one is just for testing, not a real 2.15.0, nor a
> snapshot from head.
>
> Please let me know if you have any questions.
> Hannah
>
> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise 
> wrote:
>

Re: Improve container support

[Hannah Jiang]

Hi team

I haven't received any objections, so will proceed with settings mentioned
in a previous email.

A reminder to PMC members, please let me know your docker hub id if you
want to be an admin.

Thanks,
Hannah

On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka  wrote:

> Please ignore the previous email. I was looking at the older document in
> the mail thread.
>
> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka  wrote:
>
>> I think sdk in the name is obsolete as they are all under sdks name space.
>>
>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang 
>> wrote:
>>
>>> Hi Team
>>>
>>> Thanks for all the comments about beam containers.
>>> After considering various opinions and investigating gcr and docker hub,
>>> we decided to push images to docker hub.
>>>
>>> Each image will have two tags, {version}_rc and {version}. {version} tag
>>> will be added after the release candidate image is verified.
>>> Meanwhile, we will have* latest* tag for each repository, which always
>>> points to the most recent verified release image, so users can pull it by
>>> default.
>>>
>>> Docker hub doesn't support leveled repository, which means we should
>>> follow *repository:tag* format.
>>> it's too general if we use {language_version} as repository for SDK
>>> images. (version is added when we support multiple versions.)
>>> So I would like to include *sdk* to repository. Images generated at
>>> local will also have the same name.
>>> Here are some examples:
>>>
>>>- python2.7_sdk:2.15.0
>>>- java_sdk:2.15.0_rc
>>>- go_sdk:latest
>>>
>>> I will proceed with this format if there is no strong opposition by
>>> tomorrow noon(PST).
>>>
>>> *To PMC members*:
>>> Permission control will follow the pypi model. All interested PMC
>>> members will be added as admins and release managers will be granted push
>>> permission.
>>> Please let me know your *docker id* if you want to be added as an admin.
>>>
>>> Thanks,
>>> Hannah
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
>>>
 This will greatly simplify trying out portable runners:
 https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster

 Can't wait for following to disappear from the instructions page: ./gradlew
 :sdks:python:container:docker

 On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:

> Awesome, thank you!
>
>
> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
> wrote:
>
>> Hi Thomas
>>
>> I created snapshot images from head as of around 2PM today.
>> You can pull images from
>> gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>
>> Thanks,
>> Hannah
>>
>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:
>>
>>> Hi Hannah,
>>>
>>> Thank you, I know how to build the containers locally, but not how
>>> to publish them!
>>>
>>> The cwiki says "Publishing images to gcr.io/beam requires
>>> permissions in apache-beam-testing project."
>>>
>>> Can I get access to the testing project (at least temporarily) and
>>> what would I need to setup to run the publish target that is shown on 
>>> cwiki?
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang 
>>> wrote:
>>>
 Hi Thomas

 I haven't uploaded any snapshot images yet. Here is how you can
 create one from head.
 > cd [...]/beam/
 # For Python
 > ./gradlew :sdks:python:container:py{version}:docker *where
 version is {2,35,36,37}*
 # For Java
 > ./gradlew -p sdks/java/container docker
 # For Go
 > ./gradlew -p sdks/go/container docker

 The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
 from head.

 Please let me know if you have any questions.
 Hannah

 On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise 
 wrote:

> I actually found something in [1], but it is 2.15 unfortunately.
>
> [1]
> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>
> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise 
> wrote:
>
>> Thanks for working on this. Do you happen to have publicly
>> accessible snapshots published for your testing currently (even when 
>> the
>> final location isn't sorted out)?
>>
>> I would like to use a 2.16 based Python SDK image for working on
>> my downstream project, but could not find anything in
>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>
>> Thanks,
>> Thomas
>>
>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> On Tue, Aug 27, 2019 at 3:35 PM 

Re: Improve container support

[Ankur Goenka]

Please ignore the previous email. I was looking at the older document in
the mail thread.

On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka  wrote:

> I think sdk in the name is obsolete as they are all under sdks name space.
>
> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang 
> wrote:
>
>> Hi Team
>>
>> Thanks for all the comments about beam containers.
>> After considering various opinions and investigating gcr and docker hub,
>> we decided to push images to docker hub.
>>
>> Each image will have two tags, {version}_rc and {version}. {version} tag
>> will be added after the release candidate image is verified.
>> Meanwhile, we will have* latest* tag for each repository, which always
>> points to the most recent verified release image, so users can pull it by
>> default.
>>
>> Docker hub doesn't support leveled repository, which means we should
>> follow *repository:tag* format.
>> it's too general if we use {language_version} as repository for SDK
>> images. (version is added when we support multiple versions.)
>> So I would like to include *sdk* to repository. Images generated at
>> local will also have the same name.
>> Here are some examples:
>>
>>- python2.7_sdk:2.15.0
>>- java_sdk:2.15.0_rc
>>- go_sdk:latest
>>
>> I will proceed with this format if there is no strong opposition by
>> tomorrow noon(PST).
>>
>> *To PMC members*:
>> Permission control will follow the pypi model. All interested PMC members
>> will be added as admins and release managers will be granted push
>> permission.
>> Please let me know your *docker id* if you want to be added as an admin.
>>
>> Thanks,
>> Hannah
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
>>
>>> This will greatly simplify trying out portable runners:
>>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>>
>>> Can't wait for following to disappear from the instructions page: ./gradlew
>>> :sdks:python:container:docker
>>>
>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>>>
 Awesome, thank you!


 On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
 wrote:

> Hi Thomas
>
> I created snapshot images from head as of around 2PM today.
> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot
> .
>
> Thanks,
> Hannah
>
> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:
>
>> Hi Hannah,
>>
>> Thank you, I know how to build the containers locally, but not how to
>> publish them!
>>
>> The cwiki says "Publishing images to gcr.io/beam requires
>> permissions in apache-beam-testing project."
>>
>> Can I get access to the testing project (at least temporarily) and
>> what would I need to setup to run the publish target that is shown on 
>> cwiki?
>>
>> Thanks,
>> Thomas
>>
>>
>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang 
>> wrote:
>>
>>> Hi Thomas
>>>
>>> I haven't uploaded any snapshot images yet. Here is how you can
>>> create one from head.
>>> > cd [...]/beam/
>>> # For Python
>>> > ./gradlew :sdks:python:container:py{version}:docker *where
>>> version is {2,35,36,37}*
>>> # For Java
>>> > ./gradlew -p sdks/java/container docker
>>> # For Go
>>> > ./gradlew -p sdks/go/container docker
>>>
>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
>>> from head.
>>>
>>> Please let me know if you have any questions.
>>> Hannah
>>>
>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise  wrote:
>>>
 I actually found something in [1], but it is 2.15 unfortunately.

 [1]
 https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30

 On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise 
 wrote:

> Thanks for working on this. Do you happen to have publicly
> accessible snapshots published for your testing currently (even when 
> the
> final location isn't sorted out)?
>
> I would like to use a 2.16 based Python SDK image for working on
> my downstream project, but could not find anything in
> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>
> Thanks,
> Thomas
>
> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>> hannahji...@google.com> wrote:
>>
>>> Hi team
>>>
>>> I am working on improving docker container support for Beam. We
>>> would like to publish prebuilt containers for each release version 
>>> and
>>> daily snapshot. Current work focuses on release images only and it 
>>> would be
>>> part of the release process.

Re: Improve container support

[Ankur Goenka]

I think sdk in the name is obsolete as they are all under sdks name space.

On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang  wrote:

> Hi Team
>
> Thanks for all the comments about beam containers.
> After considering various opinions and investigating gcr and docker hub,
> we decided to push images to docker hub.
>
> Each image will have two tags, {version}_rc and {version}. {version} tag
> will be added after the release candidate image is verified.
> Meanwhile, we will have* latest* tag for each repository, which always
> points to the most recent verified release image, so users can pull it by
> default.
>
> Docker hub doesn't support leveled repository, which means we should
> follow *repository:tag* format.
> it's too general if we use {language_version} as repository for SDK
> images. (version is added when we support multiple versions.)
> So I would like to include *sdk* to repository. Images generated at local
> will also have the same name.
> Here are some examples:
>
>- python2.7_sdk:2.15.0
>- java_sdk:2.15.0_rc
>- go_sdk:latest
>
> I will proceed with this format if there is no strong opposition by
> tomorrow noon(PST).
>
> *To PMC members*:
> Permission control will follow the pypi model. All interested PMC members
> will be added as admins and release managers will be granted push
> permission.
> Please let me know your *docker id* if you want to be added as an admin.
>
> Thanks,
> Hannah
>
>
>
>
>
>
>
>
> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
>
>> This will greatly simplify trying out portable runners:
>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>
>> Can't wait for following to disappear from the instructions page: ./gradlew
>> :sdks:python:container:docker
>>
>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>>
>>> Awesome, thank you!
>>>
>>>
>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
>>> wrote:
>>>
 Hi Thomas

 I created snapshot images from head as of around 2PM today.
 You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.

 Thanks,
 Hannah

 On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:

> Hi Hannah,
>
> Thank you, I know how to build the containers locally, but not how to
> publish them!
>
> The cwiki says "Publishing images to gcr.io/beam requires permissions
> in apache-beam-testing project."
>
> Can I get access to the testing project (at least temporarily) and
> what would I need to setup to run the publish target that is shown on 
> cwiki?
>
> Thanks,
> Thomas
>
>
> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang 
> wrote:
>
>> Hi Thomas
>>
>> I haven't uploaded any snapshot images yet. Here is how you can
>> create one from head.
>> > cd [...]/beam/
>> # For Python
>> > ./gradlew :sdks:python:container:py{version}:docker *where version
>> is {2,35,36,37}*
>> # For Java
>> > ./gradlew -p sdks/java/container docker
>> # For Go
>> > ./gradlew -p sdks/go/container docker
>>
>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
>> from head.
>>
>> Please let me know if you have any questions.
>> Hannah
>>
>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise  wrote:
>>
>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>
>>> [1]
>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>
>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise  wrote:
>>>
 Thanks for working on this. Do you happen to have publicly
 accessible snapshots published for your testing currently (even when 
 the
 final location isn't sorted out)?

 I would like to use a 2.16 based Python SDK image for working on my
 downstream project, but could not find anything in
 gcr.io/apache-beam-testing/beam/sdks/rc/snapshot

 Thanks,
 Thomas

 On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
> hannahji...@google.com> wrote:
>
>> Hi team
>>
>> I am working on improving docker container support for Beam. We
>> would like to publish prebuilt containers for each release version 
>> and
>> daily snapshot. Current work focuses on release images only and it 
>> would be
>> part of the release process.
>>
>> The release images will be pushed to GCR which is publicly
>> accessible(pullable). We will use the following locations.
>> *Repository*: gcr.io/beam
>> *Project*: apache-beam-testing
>> More details, including naming and tagging scheme, can be found
>> at 

Re: Improve container support

[Hannah Jiang]

Hi Team

Thanks for all the comments about beam containers.
After considering various opinions and investigating gcr and docker hub, we
decided to push images to docker hub.

Each image will have two tags, {version}_rc and {version}. {version} tag
will be added after the release candidate image is verified.
Meanwhile, we will have* latest* tag for each repository, which always
points to the most recent verified release image, so users can pull it by
default.

Docker hub doesn't support leveled repository, which means we should follow
*repository:tag* format.
it's too general if we use {language_version} as repository for SDK images.
(version is added when we support multiple versions.)
So I would like to include *sdk* to repository. Images generated at local
will also have the same name.
Here are some examples:

   - python2.7_sdk:2.15.0
   - java_sdk:2.15.0_rc
   - go_sdk:latest

I will proceed with this format if there is no strong opposition by
tomorrow noon(PST).

*To PMC members*:
Permission control will follow the pypi model. All interested PMC members
will be added as admins and release managers will be granted push
permission.
Please let me know your *docker id* if you want to be added as an admin.

Thanks,
Hannah








On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:

> This will greatly simplify trying out portable runners:
> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>
> Can't wait for following to disappear from the instructions page: ./gradlew
> :sdks:python:container:docker
>
> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>
>> Awesome, thank you!
>>
>>
>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
>> wrote:
>>
>>> Hi Thomas
>>>
>>> I created snapshot images from head as of around 2PM today.
>>> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>
>>> Thanks,
>>> Hannah
>>>
>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:
>>>
 Hi Hannah,

 Thank you, I know how to build the containers locally, but not how to
 publish them!

 The cwiki says "Publishing images to gcr.io/beam requires permissions
 in apache-beam-testing project."

 Can I get access to the testing project (at least temporarily) and what
 would I need to setup to run the publish target that is shown on cwiki?

 Thanks,
 Thomas


 On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang 
 wrote:

> Hi Thomas
>
> I haven't uploaded any snapshot images yet. Here is how you can create
> one from head.
> > cd [...]/beam/
> # For Python
> > ./gradlew :sdks:python:container:py{version}:docker *where version
> is {2,35,36,37}*
> # For Java
> > ./gradlew -p sdks/java/container docker
> # For Go
> > ./gradlew -p sdks/go/container docker
>
> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
> from head.
>
> Please let me know if you have any questions.
> Hannah
>
> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise  wrote:
>
>> I actually found something in [1], but it is 2.15 unfortunately.
>>
>> [1]
>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>
>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise  wrote:
>>
>>> Thanks for working on this. Do you happen to have publicly
>>> accessible snapshots published for your testing currently (even when the
>>> final location isn't sorted out)?
>>>
>>> I would like to use a 2.16 based Python SDK image for working on my
>>> downstream project, but could not find anything in
>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>
>>> Thanks,
>>> Thomas
>>>
>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
 hannahji...@google.com> wrote:

> Hi team
>
> I am working on improving docker container support for Beam. We
> would like to publish prebuilt containers for each release version and
> daily snapshot. Current work focuses on release images only and it 
> would be
> part of the release process.
>
> The release images will be pushed to GCR which is publicly
> accessible(pullable). We will use the following locations.
> *Repository*: gcr.io/beam
> *Project*: apache-beam-testing
> More details, including naming and tagging scheme, can be found at
> wiki
> 
>  which
> is written by several contributors.
>
> I would like to discuss these two questions.
> *1. How many tests do we need to run before pushing images to gcr*
> 

Re: Improve container support

[Thomas Weise]

This will greatly simplify trying out portable runners:
https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster

Can't wait for following to disappear from the instructions page: ./gradlew
:sdks:python:container:docker

On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:

> Awesome, thank you!
>
>
> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
> wrote:
>
>> Hi Thomas
>>
>> I created snapshot images from head as of around 2PM today.
>> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>
>> Thanks,
>> Hannah
>>
>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:
>>
>>> Hi Hannah,
>>>
>>> Thank you, I know how to build the containers locally, but not how to
>>> publish them!
>>>
>>> The cwiki says "Publishing images to gcr.io/beam requires permissions
>>> in apache-beam-testing project."
>>>
>>> Can I get access to the testing project (at least temporarily) and what
>>> would I need to setup to run the publish target that is shown on cwiki?
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang 
>>> wrote:
>>>
 Hi Thomas

 I haven't uploaded any snapshot images yet. Here is how you can create
 one from head.
 > cd [...]/beam/
 # For Python
 > ./gradlew :sdks:python:container:py{version}:docker *where version
 is {2,35,36,37}*
 # For Java
 > ./gradlew -p sdks/java/container docker
 # For Go
 > ./gradlew -p sdks/go/container docker

 The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot
 from head.

 Please let me know if you have any questions.
 Hannah

 On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise  wrote:

> I actually found something in [1], but it is 2.15 unfortunately.
>
> [1]
> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>
> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise  wrote:
>
>> Thanks for working on this. Do you happen to have publicly accessible
>> snapshots published for your testing currently (even when the final
>> location isn't sorted out)?
>>
>> I would like to use a 2.16 based Python SDK image for working on my
>> downstream project, but could not find anything in
>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>
>> Thanks,
>> Thomas
>>
>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang 
>>> wrote:
>>>
 Hi team

 I am working on improving docker container support for Beam. We
 would like to publish prebuilt containers for each release version and
 daily snapshot. Current work focuses on release images only and it 
 would be
 part of the release process.

 The release images will be pushed to GCR which is publicly
 accessible(pullable). We will use the following locations.
 *Repository*: gcr.io/beam
 *Project*: apache-beam-testing
 More details, including naming and tagging scheme, can be found at
 wiki
 
  which
 is written by several contributors.

 I would like to discuss these two questions.
 *1. How many tests do we need to run before pushing images to gcr*?
 Publishing artifacts is the last step of the release process, so at
 this moment, we already verified all codebase. In addition, many 
 Jenkins
 tests use containers, so it is already verified several times. Do we 
 need
 to run it again?

>>>
>>> In a docker repository, one container image can have multiple tags.
>>> One possibility is that  on the last step of the release process, after
>>> sufficient testing,  we place a production tag on an image that was 
>>> already
>>> pushed with a dev tag.
>>>
>>> For example a dev tag may look like:
>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>>> like:
>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
>>> image at the end.
>>>
>>> We should also plan what the process of updating the container image
>>> will look like, if we need to release the image with additional changes,
>>> and how we will test these changes before the final push (or placing
>>> production tag).
>>>
>>>

 *2. How many tests do we need to run to validate pushed images?*
 When we push the images, we assume the images would work and pass
 all the tests. After pushing, we should confirm the images are 
 pullable and
 useable. I suggest we run several tests on dataflow with each pushed 

Re: Improve container support

[Thomas Weise]

Awesome, thank you!


On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang  wrote:

> Hi Thomas
>
> I created snapshot images from head as of around 2PM today.
> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>
> Thanks,
> Hannah
>
> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:
>
>> Hi Hannah,
>>
>> Thank you, I know how to build the containers locally, but not how to
>> publish them!
>>
>> The cwiki says "Publishing images to gcr.io/beam requires permissions in
>> apache-beam-testing project."
>>
>> Can I get access to the testing project (at least temporarily) and what
>> would I need to setup to run the publish target that is shown on cwiki?
>>
>> Thanks,
>> Thomas
>>
>>
>> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang 
>> wrote:
>>
>>> Hi Thomas
>>>
>>> I haven't uploaded any snapshot images yet. Here is how you can create
>>> one from head.
>>> > cd [...]/beam/
>>> # For Python
>>> > ./gradlew :sdks:python:container:py{version}:docker *where version is
>>> {2,35,36,37}*
>>> # For Java
>>> > ./gradlew -p sdks/java/container docker
>>> # For Go
>>> > ./gradlew -p sdks/go/container docker
>>>
>>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from
>>> head.
>>>
>>> Please let me know if you have any questions.
>>> Hannah
>>>
>>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise  wrote:
>>>
 I actually found something in [1], but it is 2.15 unfortunately.

 [1]
 https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30

 On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise  wrote:

> Thanks for working on this. Do you happen to have publicly accessible
> snapshots published for your testing currently (even when the final
> location isn't sorted out)?
>
> I would like to use a 2.16 based Python SDK image for working on my
> downstream project, but could not find anything in
> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>
> Thanks,
> Thomas
>
> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang 
>> wrote:
>>
>>> Hi team
>>>
>>> I am working on improving docker container support for Beam. We
>>> would like to publish prebuilt containers for each release version and
>>> daily snapshot. Current work focuses on release images only and it 
>>> would be
>>> part of the release process.
>>>
>>> The release images will be pushed to GCR which is publicly
>>> accessible(pullable). We will use the following locations.
>>> *Repository*: gcr.io/beam
>>> *Project*: apache-beam-testing
>>> More details, including naming and tagging scheme, can be found at
>>> wiki
>>> 
>>>  which
>>> is written by several contributors.
>>>
>>> I would like to discuss these two questions.
>>> *1. How many tests do we need to run before pushing images to gcr*?
>>> Publishing artifacts is the last step of the release process, so at
>>> this moment, we already verified all codebase. In addition, many Jenkins
>>> tests use containers, so it is already verified several times. Do we 
>>> need
>>> to run it again?
>>>
>>
>> In a docker repository, one container image can have multiple tags.
>> One possibility is that  on the last step of the release process, after
>> sufficient testing,  we place a production tag on an image that was 
>> already
>> pushed with a dev tag.
>>
>> For example a dev tag may look like:
>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>> like:
>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
>> image at the end.
>>
>> We should also plan what the process of updating the container image
>> will look like, if we need to release the image with additional changes,
>> and how we will test these changes before the final push (or placing
>> production tag).
>>
>>
>>>
>>> *2. How many tests do we need to run to validate pushed images?*
>>> When we push the images, we assume the images would work and pass
>>> all the tests. After pushing, we should confirm the images are pullable 
>>> and
>>> useable. I suggest we run several tests on dataflow with each pushed 
>>> image.
>>> What do you think?
>>>
>>
>> I think it makes sense to do -  Beam runners that use SDK container
>> images should have some continuously running tests, which periodically
>> check that all supported images  are pullable and still compatible with 
>> the
>> runner.
>>
>> This work can be refined later as we explore more during our release
>>> process.
>>> Please comment or edit the wiki page 

Re: Improve container support

[Hannah Jiang]

Hi Thomas

I created snapshot images from head as of around 2PM today.
You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.

Thanks,
Hannah

On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:

> Hi Hannah,
>
> Thank you, I know how to build the containers locally, but not how to
> publish them!
>
> The cwiki says "Publishing images to gcr.io/beam requires permissions in
> apache-beam-testing project."
>
> Can I get access to the testing project (at least temporarily) and what
> would I need to setup to run the publish target that is shown on cwiki?
>
> Thanks,
> Thomas
>
>
> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang 
> wrote:
>
>> Hi Thomas
>>
>> I haven't uploaded any snapshot images yet. Here is how you can create
>> one from head.
>> > cd [...]/beam/
>> # For Python
>> > ./gradlew :sdks:python:container:py{version}:docker *where version is
>> {2,35,36,37}*
>> # For Java
>> > ./gradlew -p sdks/java/container docker
>> # For Go
>> > ./gradlew -p sdks/go/container docker
>>
>> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from
>> head.
>>
>> Please let me know if you have any questions.
>> Hannah
>>
>> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise  wrote:
>>
>>> I actually found something in [1], but it is 2.15 unfortunately.
>>>
>>> [1]
>>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>>
>>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise  wrote:
>>>
 Thanks for working on this. Do you happen to have publicly accessible
 snapshots published for your testing currently (even when the final
 location isn't sorted out)?

 I would like to use a 2.16 based Python SDK image for working on my
 downstream project, but could not find anything in
 gcr.io/apache-beam-testing/beam/sdks/rc/snapshot

 Thanks,
 Thomas

 On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang 
> wrote:
>
>> Hi team
>>
>> I am working on improving docker container support for Beam. We would
>> like to publish prebuilt containers for each release version and daily
>> snapshot. Current work focuses on release images only and it would be 
>> part
>> of the release process.
>>
>> The release images will be pushed to GCR which is publicly
>> accessible(pullable). We will use the following locations.
>> *Repository*: gcr.io/beam
>> *Project*: apache-beam-testing
>> More details, including naming and tagging scheme, can be found at
>> wiki
>> 
>>  which
>> is written by several contributors.
>>
>> I would like to discuss these two questions.
>> *1. How many tests do we need to run before pushing images to gcr*?
>> Publishing artifacts is the last step of the release process, so at
>> this moment, we already verified all codebase. In addition, many Jenkins
>> tests use containers, so it is already verified several times. Do we need
>> to run it again?
>>
>
> In a docker repository, one container image can have multiple tags.
> One possibility is that  on the last step of the release process, after
> sufficient testing,  we place a production tag on an image that was 
> already
> pushed with a dev tag.
>
> For example a dev tag may look like:
> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
> like:
> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
> image at the end.
>
> We should also plan what the process of updating the container image
> will look like, if we need to release the image with additional changes,
> and how we will test these changes before the final push (or placing
> production tag).
>
>
>>
>> *2. How many tests do we need to run to validate pushed images?*
>> When we push the images, we assume the images would work and pass all
>> the tests. After pushing, we should confirm the images are pullable and
>> useable. I suggest we run several tests on dataflow with each pushed 
>> image.
>> What do you think?
>>
>
> I think it makes sense to do -  Beam runners that use SDK container
> images should have some continuously running tests, which periodically
> check that all supported images  are pullable and still compatible with 
> the
> runner.
>
> This work can be refined later as we explore more during our release
>> process.
>> Please comment or edit the wiki page or reply to this email with your
>> opinions.
>>
>> Thanks,
>> Hannah
>>
>

Re: Improve container support

[Thomas Weise]

Hi Hannah,

Thank you, I know how to build the containers locally, but not how to
publish them!

The cwiki says "Publishing images to gcr.io/beam requires permissions in
apache-beam-testing project."

Can I get access to the testing project (at least temporarily) and what
would I need to setup to run the publish target that is shown on cwiki?

Thanks,
Thomas


On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang  wrote:

> Hi Thomas
>
> I haven't uploaded any snapshot images yet. Here is how you can create one
> from head.
> > cd [...]/beam/
> # For Python
> > ./gradlew :sdks:python:container:py{version}:docker *where version is
> {2,35,36,37}*
> # For Java
> > ./gradlew -p sdks/java/container docker
> # For Go
> > ./gradlew -p sdks/go/container docker
>
> The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from
> head.
>
> Please let me know if you have any questions.
> Hannah
>
> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise  wrote:
>
>> I actually found something in [1], but it is 2.15 unfortunately.
>>
>> [1]
>> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>>
>> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise  wrote:
>>
>>> Thanks for working on this. Do you happen to have publicly accessible
>>> snapshots published for your testing currently (even when the final
>>> location isn't sorted out)?
>>>
>>> I would like to use a 2.16 based Python SDK image for working on my
>>> downstream project, but could not find anything in
>>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>>
>>> Thanks,
>>> Thomas
>>>
>>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang 
 wrote:

> Hi team
>
> I am working on improving docker container support for Beam. We would
> like to publish prebuilt containers for each release version and daily
> snapshot. Current work focuses on release images only and it would be part
> of the release process.
>
> The release images will be pushed to GCR which is publicly
> accessible(pullable). We will use the following locations.
> *Repository*: gcr.io/beam
> *Project*: apache-beam-testing
> More details, including naming and tagging scheme, can be found at
> wiki
> 
>  which
> is written by several contributors.
>
> I would like to discuss these two questions.
> *1. How many tests do we need to run before pushing images to gcr*?
> Publishing artifacts is the last step of the release process, so at
> this moment, we already verified all codebase. In addition, many Jenkins
> tests use containers, so it is already verified several times. Do we need
> to run it again?
>

 In a docker repository, one container image can have multiple tags. One
 possibility is that  on the last step of the release process, after
 sufficient testing,  we place a production tag on an image that was already
 pushed with a dev tag.

 For example a dev tag may look like:
 gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
 like:
 gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
 image at the end.

 We should also plan what the process of updating the container image
 will look like, if we need to release the image with additional changes,
 and how we will test these changes before the final push (or placing
 production tag).


>
> *2. How many tests do we need to run to validate pushed images?*
> When we push the images, we assume the images would work and pass all
> the tests. After pushing, we should confirm the images are pullable and
> useable. I suggest we run several tests on dataflow with each pushed 
> image.
> What do you think?
>

 I think it makes sense to do -  Beam runners that use SDK container
 images should have some continuously running tests, which periodically
 check that all supported images  are pullable and still compatible with the
 runner.

 This work can be refined later as we explore more during our release
> process.
> Please comment or edit the wiki page or reply to this email with your
> opinions.
>
> Thanks,
> Hannah
>

Re: Improve container support

[Hannah Jiang]

Hi Thomas

I haven't uploaded any snapshot images yet. Here is how you can create one
from head.
> cd [...]/beam/
# For Python
> ./gradlew :sdks:python:container:py{version}:docker *where version is
{2,35,36,37}*
# For Java
> ./gradlew -p sdks/java/container docker
# For Go
> ./gradlew -p sdks/go/container docker

The 2.15 one is just for testing, not a real 2.15.0, nor a snapshot from
head.

Please let me know if you have any questions.
Hannah

On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise  wrote:

> I actually found something in [1], but it is 2.15 unfortunately.
>
> [1]
> https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
>
> On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise  wrote:
>
>> Thanks for working on this. Do you happen to have publicly accessible
>> snapshots published for your testing currently (even when the final
>> location isn't sorted out)?
>>
>> I would like to use a 2.16 based Python SDK image for working on my
>> downstream project, but could not find anything in
>> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>>
>> Thanks,
>> Thomas
>>
>> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev 
>> wrote:
>>
>>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang 
>>> wrote:
>>>
 Hi team

 I am working on improving docker container support for Beam. We would
 like to publish prebuilt containers for each release version and daily
 snapshot. Current work focuses on release images only and it would be part
 of the release process.

 The release images will be pushed to GCR which is publicly
 accessible(pullable). We will use the following locations.
 *Repository*: gcr.io/beam
 *Project*: apache-beam-testing
 More details, including naming and tagging scheme, can be found at wiki
 
  which
 is written by several contributors.

 I would like to discuss these two questions.
 *1. How many tests do we need to run before pushing images to gcr*?
 Publishing artifacts is the last step of the release process, so at
 this moment, we already verified all codebase. In addition, many Jenkins
 tests use containers, so it is already verified several times. Do we need
 to run it again?

>>>
>>> In a docker repository, one container image can have multiple tags. One
>>> possibility is that  on the last step of the release process, after
>>> sufficient testing,  we place a production tag on an image that was already
>>> pushed with a dev tag.
>>>
>>> For example a dev tag may look like:
>>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>>> like:
>>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same
>>> image at the end.
>>>
>>> We should also plan what the process of updating the container image
>>> will look like, if we need to release the image with additional changes,
>>> and how we will test these changes before the final push (or placing
>>> production tag).
>>>
>>>

 *2. How many tests do we need to run to validate pushed images?*
 When we push the images, we assume the images would work and pass all
 the tests. After pushing, we should confirm the images are pullable and
 useable. I suggest we run several tests on dataflow with each pushed image.
 What do you think?

>>>
>>> I think it makes sense to do -  Beam runners that use SDK container
>>> images should have some continuously running tests, which periodically
>>> check that all supported images  are pullable and still compatible with the
>>> runner.
>>>
>>> This work can be refined later as we explore more during our release
 process.
 Please comment or edit the wiki page or reply to this email with your
 opinions.

 Thanks,
 Hannah

>>>

Re: Improve container support

[Thomas Weise]

I actually found something in [1], but it is 2.15 unfortunately.

[1]
https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30

On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise  wrote:

> Thanks for working on this. Do you happen to have publicly accessible
> snapshots published for your testing currently (even when the final
> location isn't sorted out)?
>
> I would like to use a 2.16 based Python SDK image for working on my
> downstream project, but could not find anything in
> gcr.io/apache-beam-testing/beam/sdks/rc/snapshot
>
> Thanks,
> Thomas
>
> On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev 
> wrote:
>
>> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang 
>> wrote:
>>
>>> Hi team
>>>
>>> I am working on improving docker container support for Beam. We would
>>> like to publish prebuilt containers for each release version and daily
>>> snapshot. Current work focuses on release images only and it would be part
>>> of the release process.
>>>
>>> The release images will be pushed to GCR which is publicly
>>> accessible(pullable). We will use the following locations.
>>> *Repository*: gcr.io/beam
>>> *Project*: apache-beam-testing
>>> More details, including naming and tagging scheme, can be found at wiki
>>> 
>>>  which
>>> is written by several contributors.
>>>
>>> I would like to discuss these two questions.
>>> *1. How many tests do we need to run before pushing images to gcr*?
>>> Publishing artifacts is the last step of the release process, so at this
>>> moment, we already verified all codebase. In addition, many Jenkins tests
>>> use containers, so it is already verified several times. Do we need to run
>>> it again?
>>>
>>
>> In a docker repository, one container image can have multiple tags. One
>> possibility is that  on the last step of the release process, after
>> sufficient testing,  we place a production tag on an image that was already
>> pushed with a dev tag.
>>
>> For example a dev tag may look like:
>> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look
>> like:
>> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same image
>> at the end.
>>
>> We should also plan what the process of updating the container image will
>> look like, if we need to release the image with additional changes, and how
>> we will test these changes before the final push (or placing production
>> tag).
>>
>>
>>>
>>> *2. How many tests do we need to run to validate pushed images?*
>>> When we push the images, we assume the images would work and pass all
>>> the tests. After pushing, we should confirm the images are pullable and
>>> useable. I suggest we run several tests on dataflow with each pushed image.
>>> What do you think?
>>>
>>
>> I think it makes sense to do -  Beam runners that use SDK container
>> images should have some continuously running tests, which periodically
>> check that all supported images  are pullable and still compatible with the
>> runner.
>>
>> This work can be refined later as we explore more during our release
>>> process.
>>> Please comment or edit the wiki page or reply to this email with your
>>> opinions.
>>>
>>> Thanks,
>>> Hannah
>>>
>>

Re: Improve container support

[Thomas Weise]

Thanks for working on this. Do you happen to have publicly accessible
snapshots published for your testing currently (even when the final
location isn't sorted out)?

I would like to use a 2.16 based Python SDK image for working on my
downstream project, but could not find anything in
gcr.io/apache-beam-testing/beam/sdks/rc/snapshot

Thanks,
Thomas

On Fri, Aug 30, 2019 at 10:56 AM Valentyn Tymofieiev 
wrote:

> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang 
> wrote:
>
>> Hi team
>>
>> I am working on improving docker container support for Beam. We would
>> like to publish prebuilt containers for each release version and daily
>> snapshot. Current work focuses on release images only and it would be part
>> of the release process.
>>
>> The release images will be pushed to GCR which is publicly
>> accessible(pullable). We will use the following locations.
>> *Repository*: gcr.io/beam
>> *Project*: apache-beam-testing
>> More details, including naming and tagging scheme, can be found at wiki
>> 
>>  which
>> is written by several contributors.
>>
>> I would like to discuss these two questions.
>> *1. How many tests do we need to run before pushing images to gcr*?
>> Publishing artifacts is the last step of the release process, so at this
>> moment, we already verified all codebase. In addition, many Jenkins tests
>> use containers, so it is already verified several times. Do we need to run
>> it again?
>>
>
> In a docker repository, one container image can have multiple tags. One
> possibility is that  on the last step of the release process, after
> sufficient testing,  we place a production tag on an image that was already
> pushed with a dev tag.
>
> For example a dev tag may look like:
> gcr.io/apache-beam/python37:2.16.0-RC4, and production tag may look like:
> gcr.io/apache-beam/python37:2.16.0 and both will refer to the same image
> at the end.
>
> We should also plan what the process of updating the container image will
> look like, if we need to release the image with additional changes, and how
> we will test these changes before the final push (or placing production
> tag).
>
>
>>
>> *2. How many tests do we need to run to validate pushed images?*
>> When we push the images, we assume the images would work and pass all the
>> tests. After pushing, we should confirm the images are pullable and
>> useable. I suggest we run several tests on dataflow with each pushed image.
>> What do you think?
>>
>
> I think it makes sense to do -  Beam runners that use SDK container images
> should have some continuously running tests, which periodically check that
> all supported images  are pullable and still compatible with the runner.
>
> This work can be refined later as we explore more during our release
>> process.
>> Please comment or edit the wiki page or reply to this email with your
>> opinions.
>>
>> Thanks,
>> Hannah
>>
>

Re: Improve container support

[Hannah Jiang]

Hi team

Thanks for discussions and inputs.
It is a kind reminder that I will wait until next Tue (Sep 3) to close the
first round of discussion. I will revise plan according to discussion and
may revisit again if there are something more to discuss.
FYI: We would like to include docker images from v2.16, whose cutoff date
is Sep 11.

Thanks,
Hannah

On Wed, Aug 28, 2019 at 4:39 PM Sam Bourne  wrote:

> If docker hub is the defacto place let's use that. bintray, or gcr.io (with
>> a new GCP project) also sounds like good options. I am impartial to the
>> choice of the service. Does anyone have a strong preference here?
>
>
> Not a strong preference, but it would simplify stuff for us if you choose
> dockerhub to limit the number of places we need access to behind our
> firewall. Also, the "Official Docker Images" of Flink are there
> https://hub.docker.com/_/flink.
>
> On Thu, Aug 29, 2019 at 6:45 AM Hannah Jiang 
> wrote:
>
>> Thanks for letting me know about acces issue and sharing solution.
>> Here I created a new one
>> <https://docs.google.com/document/d/1IKE_aEkrAzkzUE4pD_r_zVuL5amHGetJ1efnbTfmunM/edit#heading=h.5irk4csrpu0y>
>> with gmail.com account.
>> Please let me know if you still see the problems.
>>
>> On Wed, Aug 28, 2019 at 6:08 AM Lukasz Cwik  wrote:
>>
>>> Google locks down docs created wtih @google.com addresses. Hannah
>>> please recreate the doc using a non @google.com address and share it
>>> with the community. You'll want to replace Google short link with an Apache
>>> short link (s.apache.org).
>>>
>>> On Wed, Aug 28, 2019 at 5:40 AM Gleb Kanterov  wrote:
>>>
>>>> Google Doc doesn't seem to be shared with dev@. Can anybody
>>>> double-check?
>>>>
>>>> On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang 
>>>> wrote:
>>>>
>>>>> add dev@
>>>>>
>>>>> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang 
>>>>> wrote:
>>>>>
>>>>>> Thanks for commenting and discussions.
>>>>>> I created a Google Docs
>>>>>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing>
>>>>>>  for
>>>>>> easy commenting and reviewing. From this moment, all changes will be
>>>>>> updated to the Google Docs and I will sync to wiki after finalize all 
>>>>>> plans.
>>>>>>
>>>>>> Thanks,
>>>>>> Hannah
>>>>>>
>>>>>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay  wrote:
>>>>>>
>>>>>>> Hi datapls-engprod,
>>>>>>>
>>>>>>> I have a question. Do you know what would it take to create a new
>>>>>>> gcp project similar to apache-beam-testing for purposes of distributing 
>>>>>>> gcr
>>>>>>> packages? We can use the same billing account.
>>>>>>>
>>>>>>> Hannah, Robert, depending on the complexity of creating another gcp
>>>>>>> project we can go with that, or simply create a new bintray account. 
>>>>>>> Either
>>>>>>> way would give us a clean new project to publish artifacts.
>>>>>>>
>>>>>>> Ahmet
>>>>>>>
>>>>>>> -- Forwarded message -
>>>>>>> From: Robert Bradshaw 
>>>>>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>>>>>> Subject: Re: Improve container support
>>>>>>> To: dev 
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay 
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <
>>>>>>> rober...@google.com> wrote:
>>>>>>> >>
>>>>>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>>> hannahji...@google.com> wrote:
>>>>>>> >> >
>>>>>>> >> > Hi team
>>>>>>> >> >
>>>>>>> >> > I am working on improving docker container support for Beam. We
>>>>>>> would like to publish prebuilt containers for each release version and
>>>>>>> daily snapshot. Current work focuses on release images only and it 
>>>>>>> would be
>>>>>>> part of the release process.
>>>>>>> >>
>>>>>>> >> This would be great!
>>>>>>> >>
>>>>>>> >> > The release images will be pushed to GCR which is publicly
>>>>>>> accessible(pullable). We will use the following locations.
>>>>>>> >> > Repository: gcr.io/beam
>>>>>>> >> > Project: apache-beam-testing
>>>>>>> >>
>>>>>>> >> Given that these are release artifacts, we should use a project
>>>>>>> with
>>>>>>> >> more restricted access than "anyone who opens a PR on github."
>>>>>>> >
>>>>>>> >
>>>>>>> > We have two options:
>>>>>>> > -  gcr.io works based on the permissions of the gcs bucket that
>>>>>>> is backing it. GCS supports bucket only permissions. These permissions
>>>>>>> needs to be explicitly granted and the service accounts used by jenkins
>>>>>>> jobs does not have these explicit permissions today.
>>>>>>> > - we can create a new project in gcr, bintray or anything else
>>>>>>> that offers the same service.
>>>>>>>
>>>>>>> I think the cleanest is to simply have a new project whose membership
>>>>>>> consists of (interested) PMC members. If we have to populate this
>>>>>>> manually I think that'd still be OK as the churn is quite low.
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Gleb
>>>>
>>>

Re: Improve container support

[Sam Bourne]

>
> If docker hub is the defacto place let's use that. bintray, or gcr.io (with
> a new GCP project) also sounds like good options. I am impartial to the
> choice of the service. Does anyone have a strong preference here?


Not a strong preference, but it would simplify stuff for us if you choose
dockerhub to limit the number of places we need access to behind our
firewall. Also, the "Official Docker Images" of Flink are there
https://hub.docker.com/_/flink.

On Thu, Aug 29, 2019 at 6:45 AM Hannah Jiang  wrote:

> Thanks for letting me know about acces issue and sharing solution.
> Here I created a new one
> <https://docs.google.com/document/d/1IKE_aEkrAzkzUE4pD_r_zVuL5amHGetJ1efnbTfmunM/edit#heading=h.5irk4csrpu0y>
> with gmail.com account.
> Please let me know if you still see the problems.
>
> On Wed, Aug 28, 2019 at 6:08 AM Lukasz Cwik  wrote:
>
>> Google locks down docs created wtih @google.com addresses. Hannah please
>> recreate the doc using a non @google.com address and share it with the
>> community. You'll want to replace Google short link with an Apache short
>> link (s.apache.org).
>>
>> On Wed, Aug 28, 2019 at 5:40 AM Gleb Kanterov  wrote:
>>
>>> Google Doc doesn't seem to be shared with dev@. Can anybody
>>> double-check?
>>>
>>> On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang 
>>> wrote:
>>>
>>>> add dev@
>>>>
>>>> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang 
>>>> wrote:
>>>>
>>>>> Thanks for commenting and discussions.
>>>>> I created a Google Docs
>>>>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing>
>>>>>  for
>>>>> easy commenting and reviewing. From this moment, all changes will be
>>>>> updated to the Google Docs and I will sync to wiki after finalize all 
>>>>> plans.
>>>>>
>>>>> Thanks,
>>>>> Hannah
>>>>>
>>>>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay  wrote:
>>>>>
>>>>>> Hi datapls-engprod,
>>>>>>
>>>>>> I have a question. Do you know what would it take to create a new gcp
>>>>>> project similar to apache-beam-testing for purposes of distributing gcr
>>>>>> packages? We can use the same billing account.
>>>>>>
>>>>>> Hannah, Robert, depending on the complexity of creating another gcp
>>>>>> project we can go with that, or simply create a new bintray account. 
>>>>>> Either
>>>>>> way would give us a clean new project to publish artifacts.
>>>>>>
>>>>>> Ahmet
>>>>>>
>>>>>> -- Forwarded message -
>>>>>> From: Robert Bradshaw 
>>>>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>>>>> Subject: Re: Improve container support
>>>>>> To: dev 
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay  wrote:
>>>>>> >
>>>>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <
>>>>>> rober...@google.com> wrote:
>>>>>> >>
>>>>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>> hannahji...@google.com> wrote:
>>>>>> >> >
>>>>>> >> > Hi team
>>>>>> >> >
>>>>>> >> > I am working on improving docker container support for Beam. We
>>>>>> would like to publish prebuilt containers for each release version and
>>>>>> daily snapshot. Current work focuses on release images only and it would 
>>>>>> be
>>>>>> part of the release process.
>>>>>> >>
>>>>>> >> This would be great!
>>>>>> >>
>>>>>> >> > The release images will be pushed to GCR which is publicly
>>>>>> accessible(pullable). We will use the following locations.
>>>>>> >> > Repository: gcr.io/beam
>>>>>> >> > Project: apache-beam-testing
>>>>>> >>
>>>>>> >> Given that these are release artifacts, we should use a project
>>>>>> with
>>>>>> >> more restricted access than "anyone who opens a PR on github."
>>>>>> >
>>>>>> >
>>>>>> > We have two options:
>>>>>> > -  gcr.io works based on the permissions of the gcs bucket that is
>>>>>> backing it. GCS supports bucket only permissions. These permissions needs
>>>>>> to be explicitly granted and the service accounts used by jenkins jobs 
>>>>>> does
>>>>>> not have these explicit permissions today.
>>>>>> > - we can create a new project in gcr, bintray or anything else that
>>>>>> offers the same service.
>>>>>>
>>>>>> I think the cleanest is to simply have a new project whose membership
>>>>>> consists of (interested) PMC members. If we have to populate this
>>>>>> manually I think that'd still be OK as the churn is quite low.
>>>>>>
>>>>>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>

Re: Improve container support

[Ahmet Altay]

+1 to Pablo's comment. I believe any place that offers this service would
be fine and we would need to figure out how to set it up correctly. Our
pypi model seems to work fine so far (PMC members are owners, release
managers are added as maintainers as needed.). To me, the important part is
that we make progress so that we can add the containers to the release
process. We postponed this decision in the past 2 releases because the
conversation started late along with the release branch cut emails.

If docker hub is the defacto place let's use that. bintray, or gcr.io (with
a new GCP project) also sounds like good options. I am impartial to the
choice of the service. Does anyone have a strong preference here?

Ahmet

On Wed, Aug 28, 2019 at 8:25 AM Pablo Estrada  wrote:

> A question about the repository we're using. It seems that Docker Hub is
> the de-facto repo for docker images? Is GCR pretty much the same in terms
> of access, authentication, etc?
> We don't have to figure out the repository immediately, and it's fine to
> iterate - but I wanted to make sure we think about that : )
>
> On Wed, Aug 28, 2019 at 6:08 AM Lukasz Cwik  wrote:
>
>> Google locks down docs created wtih @google.com addresses. Hannah please
>> recreate the doc using a non @google.com address and share it with the
>> community. You'll want to replace Google short link with an Apache short
>> link (s.apache.org).
>>
>> On Wed, Aug 28, 2019 at 5:40 AM Gleb Kanterov  wrote:
>>
>>> Google Doc doesn't seem to be shared with dev@. Can anybody
>>> double-check?
>>>
>>> On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang 
>>> wrote:
>>>
>>>> add dev@
>>>>
>>>> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang 
>>>> wrote:
>>>>
>>>>> Thanks for commenting and discussions.
>>>>> I created a Google Docs
>>>>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing>
>>>>>  for
>>>>> easy commenting and reviewing. From this moment, all changes will be
>>>>> updated to the Google Docs and I will sync to wiki after finalize all 
>>>>> plans.
>>>>>
>>>>> Thanks,
>>>>> Hannah
>>>>>
>>>>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay  wrote:
>>>>>
>>>>>> Hi datapls-engprod,
>>>>>>
>>>>>> I have a question. Do you know what would it take to create a new gcp
>>>>>> project similar to apache-beam-testing for purposes of distributing gcr
>>>>>> packages? We can use the same billing account.
>>>>>>
>>>>>> Hannah, Robert, depending on the complexity of creating another gcp
>>>>>> project we can go with that, or simply create a new bintray account. 
>>>>>> Either
>>>>>> way would give us a clean new project to publish artifacts.
>>>>>>
>>>>>> Ahmet
>>>>>>
>>>>>> -- Forwarded message -
>>>>>> From: Robert Bradshaw 
>>>>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>>>>> Subject: Re: Improve container support
>>>>>> To: dev 
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay  wrote:
>>>>>> >
>>>>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw <
>>>>>> rober...@google.com> wrote:
>>>>>> >>
>>>>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>>> hannahji...@google.com> wrote:
>>>>>> >> >
>>>>>> >> > Hi team
>>>>>> >> >
>>>>>> >> > I am working on improving docker container support for Beam. We
>>>>>> would like to publish prebuilt containers for each release version and
>>>>>> daily snapshot. Current work focuses on release images only and it would 
>>>>>> be
>>>>>> part of the release process.
>>>>>> >>
>>>>>> >> This would be great!
>>>>>> >>
>>>>>> >> > The release images will be pushed to GCR which is publicly
>>>>>> accessible(pullable). We will use the following locations.
>>>>>> >> > Repository: gcr.io/beam
>>>>>> >> > Project: apache-beam-testing
>>>>>> >>
>>>>>> >> Given that these are release artifacts, we should use a project
>>>>>> with
>>>>>> >> more restricted access than "anyone who opens a PR on github."
>>>>>> >
>>>>>> >
>>>>>> > We have two options:
>>>>>> > -  gcr.io works based on the permissions of the gcs bucket that is
>>>>>> backing it. GCS supports bucket only permissions. These permissions needs
>>>>>> to be explicitly granted and the service accounts used by jenkins jobs 
>>>>>> does
>>>>>> not have these explicit permissions today.
>>>>>> > - we can create a new project in gcr, bintray or anything else that
>>>>>> offers the same service.
>>>>>>
>>>>>> I think the cleanest is to simply have a new project whose membership
>>>>>> consists of (interested) PMC members. If we have to populate this
>>>>>> manually I think that'd still be OK as the churn is quite low.
>>>>>>
>>>>>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>

Re: Improve container support

[Pablo Estrada]

A question about the repository we're using. It seems that Docker Hub is
the de-facto repo for docker images? Is GCR pretty much the same in terms
of access, authentication, etc?
We don't have to figure out the repository immediately, and it's fine to
iterate - but I wanted to make sure we think about that : )

On Wed, Aug 28, 2019 at 6:08 AM Lukasz Cwik  wrote:

> Google locks down docs created wtih @google.com addresses. Hannah please
> recreate the doc using a non @google.com address and share it with the
> community. You'll want to replace Google short link with an Apache short
> link (s.apache.org).
>
> On Wed, Aug 28, 2019 at 5:40 AM Gleb Kanterov  wrote:
>
>> Google Doc doesn't seem to be shared with dev@. Can anybody double-check?
>>
>> On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang 
>> wrote:
>>
>>> add dev@
>>>
>>> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang 
>>> wrote:
>>>
>>>> Thanks for commenting and discussions.
>>>> I created a Google Docs
>>>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing>
>>>>  for
>>>> easy commenting and reviewing. From this moment, all changes will be
>>>> updated to the Google Docs and I will sync to wiki after finalize all 
>>>> plans.
>>>>
>>>> Thanks,
>>>> Hannah
>>>>
>>>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay  wrote:
>>>>
>>>>> Hi datapls-engprod,
>>>>>
>>>>> I have a question. Do you know what would it take to create a new gcp
>>>>> project similar to apache-beam-testing for purposes of distributing gcr
>>>>> packages? We can use the same billing account.
>>>>>
>>>>> Hannah, Robert, depending on the complexity of creating another gcp
>>>>> project we can go with that, or simply create a new bintray account. 
>>>>> Either
>>>>> way would give us a clean new project to publish artifacts.
>>>>>
>>>>> Ahmet
>>>>>
>>>>> -- Forwarded message -
>>>>> From: Robert Bradshaw 
>>>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>>>> Subject: Re: Improve container support
>>>>> To: dev 
>>>>>
>>>>>
>>>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay  wrote:
>>>>> >
>>>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw 
>>>>> wrote:
>>>>> >>
>>>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang <
>>>>> hannahji...@google.com> wrote:
>>>>> >> >
>>>>> >> > Hi team
>>>>> >> >
>>>>> >> > I am working on improving docker container support for Beam. We
>>>>> would like to publish prebuilt containers for each release version and
>>>>> daily snapshot. Current work focuses on release images only and it would 
>>>>> be
>>>>> part of the release process.
>>>>> >>
>>>>> >> This would be great!
>>>>> >>
>>>>> >> > The release images will be pushed to GCR which is publicly
>>>>> accessible(pullable). We will use the following locations.
>>>>> >> > Repository: gcr.io/beam
>>>>> >> > Project: apache-beam-testing
>>>>> >>
>>>>> >> Given that these are release artifacts, we should use a project with
>>>>> >> more restricted access than "anyone who opens a PR on github."
>>>>> >
>>>>> >
>>>>> > We have two options:
>>>>> > -  gcr.io works based on the permissions of the gcs bucket that is
>>>>> backing it. GCS supports bucket only permissions. These permissions needs
>>>>> to be explicitly granted and the service accounts used by jenkins jobs 
>>>>> does
>>>>> not have these explicit permissions today.
>>>>> > - we can create a new project in gcr, bintray or anything else that
>>>>> offers the same service.
>>>>>
>>>>> I think the cleanest is to simply have a new project whose membership
>>>>> consists of (interested) PMC members. If we have to populate this
>>>>> manually I think that'd still be OK as the churn is quite low.
>>>>>
>>>>
>>
>> --
>> Cheers,
>> Gleb
>>
>

Re: Improve container support

[Lukasz Cwik]

Google locks down docs created wtih @google.com addresses. Hannah please
recreate the doc using a non @google.com address and share it with the
community. You'll want to replace Google short link with an Apache short
link (s.apache.org).

On Wed, Aug 28, 2019 at 5:40 AM Gleb Kanterov  wrote:

> Google Doc doesn't seem to be shared with dev@. Can anybody double-check?
>
> On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang 
> wrote:
>
>> add dev@
>>
>> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang 
>> wrote:
>>
>>> Thanks for commenting and discussions.
>>> I created a Google Docs
>>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing>
>>>  for
>>> easy commenting and reviewing. From this moment, all changes will be
>>> updated to the Google Docs and I will sync to wiki after finalize all plans.
>>>
>>> Thanks,
>>> Hannah
>>>
>>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay  wrote:
>>>
>>>> Hi datapls-engprod,
>>>>
>>>> I have a question. Do you know what would it take to create a new gcp
>>>> project similar to apache-beam-testing for purposes of distributing gcr
>>>> packages? We can use the same billing account.
>>>>
>>>> Hannah, Robert, depending on the complexity of creating another gcp
>>>> project we can go with that, or simply create a new bintray account. Either
>>>> way would give us a clean new project to publish artifacts.
>>>>
>>>> Ahmet
>>>>
>>>> -- Forwarded message -
>>>> From: Robert Bradshaw 
>>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>>> Subject: Re: Improve container support
>>>> To: dev 
>>>>
>>>>
>>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay  wrote:
>>>> >
>>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw 
>>>> wrote:
>>>> >>
>>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang 
>>>> wrote:
>>>> >> >
>>>> >> > Hi team
>>>> >> >
>>>> >> > I am working on improving docker container support for Beam. We
>>>> would like to publish prebuilt containers for each release version and
>>>> daily snapshot. Current work focuses on release images only and it would be
>>>> part of the release process.
>>>> >>
>>>> >> This would be great!
>>>> >>
>>>> >> > The release images will be pushed to GCR which is publicly
>>>> accessible(pullable). We will use the following locations.
>>>> >> > Repository: gcr.io/beam
>>>> >> > Project: apache-beam-testing
>>>> >>
>>>> >> Given that these are release artifacts, we should use a project with
>>>> >> more restricted access than "anyone who opens a PR on github."
>>>> >
>>>> >
>>>> > We have two options:
>>>> > -  gcr.io works based on the permissions of the gcs bucket that is
>>>> backing it. GCS supports bucket only permissions. These permissions needs
>>>> to be explicitly granted and the service accounts used by jenkins jobs does
>>>> not have these explicit permissions today.
>>>> > - we can create a new project in gcr, bintray or anything else that
>>>> offers the same service.
>>>>
>>>> I think the cleanest is to simply have a new project whose membership
>>>> consists of (interested) PMC members. If we have to populate this
>>>> manually I think that'd still be OK as the churn is quite low.
>>>>
>>>
>
> --
> Cheers,
> Gleb
>

Re: Improve container support

[Gleb Kanterov]

Google Doc doesn't seem to be shared with dev@. Can anybody double-check?

On Wed, Aug 28, 2019 at 7:36 AM Hannah Jiang  wrote:

> add dev@
>
> On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang 
> wrote:
>
>> Thanks for commenting and discussions.
>> I created a Google Docs
>> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing>
>>  for
>> easy commenting and reviewing. From this moment, all changes will be
>> updated to the Google Docs and I will sync to wiki after finalize all plans.
>>
>> Thanks,
>> Hannah
>>
>> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay  wrote:
>>
>>> Hi datapls-engprod,
>>>
>>> I have a question. Do you know what would it take to create a new gcp
>>> project similar to apache-beam-testing for purposes of distributing gcr
>>> packages? We can use the same billing account.
>>>
>>> Hannah, Robert, depending on the complexity of creating another gcp
>>> project we can go with that, or simply create a new bintray account. Either
>>> way would give us a clean new project to publish artifacts.
>>>
>>> Ahmet
>>>
>>> -- Forwarded message -
>>> From: Robert Bradshaw 
>>> Date: Tue, Aug 27, 2019 at 6:48 PM
>>> Subject: Re: Improve container support
>>> To: dev 
>>>
>>>
>>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay  wrote:
>>> >
>>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw 
>>> wrote:
>>> >>
>>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang 
>>> wrote:
>>> >> >
>>> >> > Hi team
>>> >> >
>>> >> > I am working on improving docker container support for Beam. We
>>> would like to publish prebuilt containers for each release version and
>>> daily snapshot. Current work focuses on release images only and it would be
>>> part of the release process.
>>> >>
>>> >> This would be great!
>>> >>
>>> >> > The release images will be pushed to GCR which is publicly
>>> accessible(pullable). We will use the following locations.
>>> >> > Repository: gcr.io/beam
>>> >> > Project: apache-beam-testing
>>> >>
>>> >> Given that these are release artifacts, we should use a project with
>>> >> more restricted access than "anyone who opens a PR on github."
>>> >
>>> >
>>> > We have two options:
>>> > -  gcr.io works based on the permissions of the gcs bucket that is
>>> backing it. GCS supports bucket only permissions. These permissions needs
>>> to be explicitly granted and the service accounts used by jenkins jobs does
>>> not have these explicit permissions today.
>>> > - we can create a new project in gcr, bintray or anything else that
>>> offers the same service.
>>>
>>> I think the cleanest is to simply have a new project whose membership
>>> consists of (interested) PMC members. If we have to populate this
>>> manually I think that'd still be OK as the churn is quite low.
>>>
>>

-- 
Cheers,
Gleb

Re: Improve container support

[Hannah Jiang]

add dev@

On Tue, Aug 27, 2019 at 9:29 PM Hannah Jiang  wrote:

> Thanks for commenting and discussions.
> I created a Google Docs
> <https://docs.google.com/document/d/1LOraxUHdmuykgOPsjRC00iDqvSapoCuCjZcZm-Vh8o0/edit?usp=sharing>
>  for
> easy commenting and reviewing. From this moment, all changes will be
> updated to the Google Docs and I will sync to wiki after finalize all plans.
>
> Thanks,
> Hannah
>
> On Tue, Aug 27, 2019 at 9:24 PM Ahmet Altay  wrote:
>
>> Hi datapls-engprod,
>>
>> I have a question. Do you know what would it take to create a new gcp
>> project similar to apache-beam-testing for purposes of distributing gcr
>> packages? We can use the same billing account.
>>
>> Hannah, Robert, depending on the complexity of creating another gcp
>> project we can go with that, or simply create a new bintray account. Either
>> way would give us a clean new project to publish artifacts.
>>
>> Ahmet
>>
>> ------ Forwarded message -
>> From: Robert Bradshaw 
>> Date: Tue, Aug 27, 2019 at 6:48 PM
>> Subject: Re: Improve container support
>> To: dev 
>>
>>
>> On Tue, Aug 27, 2019 at 6:20 PM Ahmet Altay  wrote:
>> >
>> > On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw 
>> wrote:
>> >>
>> >> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang 
>> wrote:
>> >> >
>> >> > Hi team
>> >> >
>> >> > I am working on improving docker container support for Beam. We
>> would like to publish prebuilt containers for each release version and
>> daily snapshot. Current work focuses on release images only and it would be
>> part of the release process.
>> >>
>> >> This would be great!
>> >>
>> >> > The release images will be pushed to GCR which is publicly
>> accessible(pullable). We will use the following locations.
>> >> > Repository: gcr.io/beam
>> >> > Project: apache-beam-testing
>> >>
>> >> Given that these are release artifacts, we should use a project with
>> >> more restricted access than "anyone who opens a PR on github."
>> >
>> >
>> > We have two options:
>> > -  gcr.io works based on the permissions of the gcs bucket that is
>> backing it. GCS supports bucket only permissions. These permissions needs
>> to be explicitly granted and the service accounts used by jenkins jobs does
>> not have these explicit permissions today.
>> > - we can create a new project in gcr, bintray or anything else that
>> offers the same service.
>>
>> I think the cleanest is to simply have a new project whose membership
>> consists of (interested) PMC members. If we have to populate this
>> manually I think that'd still be OK as the churn is quite low.
>>
>

Re: Improve container support

[Ahmet Altay]

On Tue, Aug 27, 2019 at 5:50 PM Robert Bradshaw  wrote:

> On Tue, Aug 27, 2019 at 3:35 PM Hannah Jiang 
> wrote:
> >
> > Hi team
> >
> > I am working on improving docker container support for Beam. We would
> like to publish prebuilt containers for each release version and daily
> snapshot. Current work focuses on release images only and it would be part
> of the release process.
>
> This would be great!
>
> > The release images will be pushed to GCR which is publicly
> accessible(pullable). We will use the following locations.
> > Repository: gcr.io/beam
> > Project: apache-beam-testing
>
> Given that these are release artifacts, we should use a project with
> more restricted access than "anyone who opens a PR on github."
>

We have two options:
-  gcr.io works based on the permissions of the gcs bucket that is backing
it. GCS supports bucket only permissions. These permissions needs to be
explicitly granted and the service accounts used by jenkins jobs does not
have these explicit permissions today.
- we can create a new project in gcr, bintray or anything else that offers
the same service.


>
> > More details, including naming and tagging scheme, can be found at wiki
> which is written by several contributors.
>
> Would it make sense to put this in a format more amenable to commenting?
>
> > I would like to discuss these two questions.
> > 1. How many tests do we need to run before pushing images to gcr?
> > Publishing artifacts is the last step of the release process, so at this
> moment, we already verified all codebase. In addition, many Jenkins tests
> use containers, so it is already verified several times. Do we need to run
> it again?
> >
> > 2. How many tests do we need to run to validate pushed images?
> > When we push the images, we assume the images would work and pass all
> the tests. After pushing, we should confirm the images are pullable and
> useable. I suggest we run several tests on dataflow with each pushed image.
> What do you think?
>
> I think the release manager publishing these images as part of the
> release process (just like publishing to the maven repo and svn) and
> validation happens as part of validating the artifacts during the
> vote.
>