Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
I agree, I think their are few things which have to be though through as part of Portable image release. * Where to host the images. We can ofcourse have an alias for the image which can point to a different location but the hosting location have to be sort through. * Validation process for the images. * Backward compatibility for the images. Though we can just tag them with release name. I might not have immediate bandwidth to do this so we need to prioritize based on other items we have. On Tue, May 28, 2019 at 5:24 PM Ahmet Altay wrote: > Could we first figure out the process (where to push, how to push, > permissions needed, how to validate etc.) as part of the snapshots and > update the release guide based on that? > > On Tue, May 28, 2019 at 2:43 AM Robert Bradshaw > wrote: > >> In the future (read, next release) the SDK will likely have reference >> to the containers, so this will have to be part of the release. > > > Who is working on this change? Could they help with figuring out the > publishing the containers part? > > >> But I >> agree for 2.13 it should be more about figuring out the process and >> not necessarily holding back. >> >> On Mon, May 27, 2019 at 7:42 PM Ankur Goenka wrote: >> > >> > +1 >> > We can release the images with 2.13 but we should not block 2.13 >> release for this. >> > >> > On Mon, May 27, 2019, 8:39 AM Thomas Weise wrote: >> >> >> >> +1 >> >> >> >> >> >> On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía >> wrote: >> >>> >> >>> +1 >> >>> >> >>> On Mon, May 27, 2019 at 3:35 PM Maximilian Michels >> wrote: >> >>> > >> >>> > +1 >> >>> > >> >>> > On 27.05.19 14:04, Robert Bradshaw wrote: >> >>> > > Sounds like everyone's onboard with the plan. Any chance we could >> >>> > > publish these for the upcoming 2.13 release? >> >>> > > >> >>> > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy >> wrote: >> >>> > >> >> >>> > >> +1 to have a registry for images accessible to anyone. For >> snapshot images, I agree that gcr + apache-beam-testing project seems a >> good and easy way to start with. >> >>> > >> >> >>> > >> Łukasz >> >>> > >> >> >>> > >> wt., 22 sty 2019 o 19:43 Mark Liu >> napisał(a): >> >>> > >>> >> >>> > >>> +1 to have an official Beam released container image. >> >>> > >>> >> >>> > >>> Also I would propose to add a verification step to (or after) >> the release process to do smoke check. Python have ValidatesContainer test >> that runs basic pipeline using newly built container for verification. >> Other sdk languages can do similar thing or add a common framework. >> >>> > >>> >> >>> > >>> Mark >> >>> > >>> >> >>> > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold < >> amyrv...@google.com> wrote: >> >>> > >> >>> > +1 This would be great. gcr.io seems like a good option for >> snapshots due to the permissions from jenkins to upload and ability to keep >> snapshots around. >> >>> > >> >>> > On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang < >> ruo...@google.com> wrote: >> >>> > > >> >>> > > +1 This would be a great thing to have. >> >>> > > >> >>> > > On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka < >> goe...@google.com> wrote: >> >>> > >> >> >>> > >> grc.io seems to be a good option. Given that we don't need >> the hosting server name in the image name makes it easily changeable later. >> >>> > >> >> >>> > >> Docker container for Apache Flink is named "flink" and they >> have different tags for different releases and configurations >> https://hub.docker.com/_/flink .We can follow a similar model and can >> name the image as "beam" (beam doesn't seem to be taken on docker hub) and >> use tags to distinguish Java/Python/Go and versions etc. >> >>> > >> >> >>> > >> Tags will look like: >> >>> > >> java-SNAPSHOT >> >>> > >> java-2.10.1 >> >>> > >> python2-SNAPSHOT >> >>> > >> python2-2.10.1 >> >>> > >> go-SNAPSHOT >> >>> > >> go-2.10.1 >> >>> > >> >> >>> > >> >> >>> > >> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay < >> al...@google.com> wrote: >> >>> > >>> >> >>> > >>> For snapshots, we could use gcr.io. Permission would not >> be a problem since Jenkins is already correctly setup. The cost will be >> covered under apache-beam-testing project. And since this is only for >> snapshots, it will be only for temporary artifacts not for release >> artifacts. >> >>> > >>> >> >>> > >>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> >>> > >> >>> > +1, releasing containers is a useful process that we need >> to build in Beam and it is required for FnApi users. Among other reasons, >> having officially-released Beam SDK harness container images will make it >> easier for users to do simple customizations to container images, as they >> will be able to use container image released by Beam as a base image. >> >>> > >> >>> > Good point about potential storage limitations on Bintray. >> With Beam
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
Could we first figure out the process (where to push, how to push, permissions needed, how to validate etc.) as part of the snapshots and update the release guide based on that? On Tue, May 28, 2019 at 2:43 AM Robert Bradshaw wrote: > In the future (read, next release) the SDK will likely have reference > to the containers, so this will have to be part of the release. Who is working on this change? Could they help with figuring out the publishing the containers part? > But I > agree for 2.13 it should be more about figuring out the process and > not necessarily holding back. > > On Mon, May 27, 2019 at 7:42 PM Ankur Goenka wrote: > > > > +1 > > We can release the images with 2.13 but we should not block 2.13 release > for this. > > > > On Mon, May 27, 2019, 8:39 AM Thomas Weise wrote: > >> > >> +1 > >> > >> > >> On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía wrote: > >>> > >>> +1 > >>> > >>> On Mon, May 27, 2019 at 3:35 PM Maximilian Michels > wrote: > >>> > > >>> > +1 > >>> > > >>> > On 27.05.19 14:04, Robert Bradshaw wrote: > >>> > > Sounds like everyone's onboard with the plan. Any chance we could > >>> > > publish these for the upcoming 2.13 release? > >>> > > > >>> > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy > wrote: > >>> > >> > >>> > >> +1 to have a registry for images accessible to anyone. For > snapshot images, I agree that gcr + apache-beam-testing project seems a > good and easy way to start with. > >>> > >> > >>> > >> Łukasz > >>> > >> > >>> > >> wt., 22 sty 2019 o 19:43 Mark Liu > napisał(a): > >>> > >>> > >>> > >>> +1 to have an official Beam released container image. > >>> > >>> > >>> > >>> Also I would propose to add a verification step to (or after) > the release process to do smoke check. Python have ValidatesContainer test > that runs basic pipeline using newly built container for verification. > Other sdk languages can do similar thing or add a common framework. > >>> > >>> > >>> > >>> Mark > >>> > >>> > >>> > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold < > amyrv...@google.com> wrote: > >>> > > >>> > +1 This would be great. gcr.io seems like a good option for > snapshots due to the permissions from jenkins to upload and ability to keep > snapshots around. > >>> > > >>> > On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang > wrote: > >>> > > > >>> > > +1 This would be a great thing to have. > >>> > > > >>> > > On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka < > goe...@google.com> wrote: > >>> > >> > >>> > >> grc.io seems to be a good option. Given that we don't need > the hosting server name in the image name makes it easily changeable later. > >>> > >> > >>> > >> Docker container for Apache Flink is named "flink" and they > have different tags for different releases and configurations > https://hub.docker.com/_/flink .We can follow a similar model and can > name the image as "beam" (beam doesn't seem to be taken on docker hub) and > use tags to distinguish Java/Python/Go and versions etc. > >>> > >> > >>> > >> Tags will look like: > >>> > >> java-SNAPSHOT > >>> > >> java-2.10.1 > >>> > >> python2-SNAPSHOT > >>> > >> python2-2.10.1 > >>> > >> go-SNAPSHOT > >>> > >> go-2.10.1 > >>> > >> > >>> > >> > >>> > >> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay > wrote: > >>> > >>> > >>> > >>> For snapshots, we could use gcr.io. Permission would not be > a problem since Jenkins is already correctly setup. The cost will be > covered under apache-beam-testing project. And since this is only for > snapshots, it will be only for temporary artifacts not for release > artifacts. > >>> > >>> > >>> > >>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev < > valen...@google.com> wrote: > >>> > > >>> > +1, releasing containers is a useful process that we need > to build in Beam and it is required for FnApi users. Among other reasons, > having officially-released Beam SDK harness container images will make it > easier for users to do simple customizations to container images, as they > will be able to use container image released by Beam as a base image. > >>> > > >>> > Good point about potential storage limitations on Bintray. > With Beam Release cadence we may quickly exceed the 10 GB quota. It may > also affect our decisions as to which images we want to release, for > example: do we want to only release one container image with Python 3 > interpreter, or do we want to release a container image for each Python 3 > minor version that Beam is compatible with. > >>> > >>> > >>> > >>> > >>> > >>> Probably worth a separate discussion. I would favor first > releasing a python 3 compatible version before figuring out how we would > target multiple python 3 versions. > >>> > >>> > >>> > >>> > >>> > > >>> > > >>> > On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka < > goe...@google.com> wrote: > >>> > >
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
In the future (read, next release) the SDK will likely have reference to the containers, so this will have to be part of the release. But I agree for 2.13 it should be more about figuring out the process and not necessarily holding back. On Mon, May 27, 2019 at 7:42 PM Ankur Goenka wrote: > > +1 > We can release the images with 2.13 but we should not block 2.13 release for > this. > > On Mon, May 27, 2019, 8:39 AM Thomas Weise wrote: >> >> +1 >> >> >> On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía wrote: >>> >>> +1 >>> >>> On Mon, May 27, 2019 at 3:35 PM Maximilian Michels wrote: >>> > >>> > +1 >>> > >>> > On 27.05.19 14:04, Robert Bradshaw wrote: >>> > > Sounds like everyone's onboard with the plan. Any chance we could >>> > > publish these for the upcoming 2.13 release? >>> > > >>> > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy wrote: >>> > >> >>> > >> +1 to have a registry for images accessible to anyone. For snapshot >>> > >> images, I agree that gcr + apache-beam-testing project seems a good >>> > >> and easy way to start with. >>> > >> >>> > >> Łukasz >>> > >> >>> > >> wt., 22 sty 2019 o 19:43 Mark Liu napisał(a): >>> > >>> >>> > >>> +1 to have an official Beam released container image. >>> > >>> >>> > >>> Also I would propose to add a verification step to (or after) the >>> > >>> release process to do smoke check. Python have ValidatesContainer >>> > >>> test that runs basic pipeline using newly built container for >>> > >>> verification. Other sdk languages can do similar thing or add a >>> > >>> common framework. >>> > >>> >>> > >>> Mark >>> > >>> >>> > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold >>> > >>> wrote: >>> > >>> > +1 This would be great. gcr.io seems like a good option for >>> > snapshots due to the permissions from jenkins to upload and ability >>> > to keep snapshots around. >>> > >>> > On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang >>> > wrote: >>> > > >>> > > +1 This would be a great thing to have. >>> > > >>> > > On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka >>> > > wrote: >>> > >> >>> > >> grc.io seems to be a good option. Given that we don't need the >>> > >> hosting server name in the image name makes it easily changeable >>> > >> later. >>> > >> >>> > >> Docker container for Apache Flink is named "flink" and they have >>> > >> different tags for different releases and configurations >>> > >> https://hub.docker.com/_/flink .We can follow a similar model and >>> > >> can name the image as "beam" (beam doesn't seem to be taken on >>> > >> docker hub) and use tags to distinguish Java/Python/Go and >>> > >> versions etc. >>> > >> >>> > >> Tags will look like: >>> > >> java-SNAPSHOT >>> > >> java-2.10.1 >>> > >> python2-SNAPSHOT >>> > >> python2-2.10.1 >>> > >> go-SNAPSHOT >>> > >> go-2.10.1 >>> > >> >>> > >> >>> > >> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay >>> > >> wrote: >>> > >>> >>> > >>> For snapshots, we could use gcr.io. Permission would not be a >>> > >>> problem since Jenkins is already correctly setup. The cost will >>> > >>> be covered under apache-beam-testing project. And since this is >>> > >>> only for snapshots, it will be only for temporary artifacts not >>> > >>> for release artifacts. >>> > >>> >>> > >>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev >>> > >>> wrote: >>> > >>> > +1, releasing containers is a useful process that we need to >>> > build in Beam and it is required for FnApi users. Among other >>> > reasons, having officially-released Beam SDK harness container >>> > images will make it easier for users to do simple customizations >>> > to container images, as they will be able to use container >>> > image released by Beam as a base image. >>> > >>> > Good point about potential storage limitations on Bintray. With >>> > Beam Release cadence we may quickly exceed the 10 GB quota. It >>> > may also affect our decisions as to which images we want to >>> > release, for example: do we want to only release one container >>> > image with Python 3 interpreter, or do we want to release a >>> > container image for each Python 3 minor version that Beam is >>> > compatible with. >>> > >>> >>> > >>> >>> > >>> Probably worth a separate discussion. I would favor first >>> > >>> releasing a python 3 compatible version before figuring out how >>> > >>> we would target multiple python 3 versions. >>> > >>> >>> > >>> >>> > >>> > >>> > On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka >>> > wrote: >>> > > >>> > > >>> > > >>> > > On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay >>> > > wrote: >>> >
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
+1 We can release the images with 2.13 but we should not block 2.13 release for this. On Mon, May 27, 2019, 8:39 AM Thomas Weise wrote: > +1 > > > On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía wrote: > >> +1 >> >> On Mon, May 27, 2019 at 3:35 PM Maximilian Michels >> wrote: >> > >> > +1 >> > >> > On 27.05.19 14:04, Robert Bradshaw wrote: >> > > Sounds like everyone's onboard with the plan. Any chance we could >> > > publish these for the upcoming 2.13 release? >> > > >> > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy >> wrote: >> > >> >> > >> +1 to have a registry for images accessible to anyone. For snapshot >> images, I agree that gcr + apache-beam-testing project seems a good and >> easy way to start with. >> > >> >> > >> Łukasz >> > >> >> > >> wt., 22 sty 2019 o 19:43 Mark Liu napisał(a): >> > >>> >> > >>> +1 to have an official Beam released container image. >> > >>> >> > >>> Also I would propose to add a verification step to (or after) the >> release process to do smoke check. Python have ValidatesContainer test that >> runs basic pipeline using newly built container for verification. Other sdk >> languages can do similar thing or add a common framework. >> > >>> >> > >>> Mark >> > >>> >> > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold >> wrote: >> > >> > +1 This would be great. gcr.io seems like a good option for >> snapshots due to the permissions from jenkins to upload and ability to keep >> snapshots around. >> > >> > On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang >> wrote: >> > > >> > > +1 This would be a great thing to have. >> > > >> > > On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka >> wrote: >> > >> >> > >> grc.io seems to be a good option. Given that we don't need the >> hosting server name in the image name makes it easily changeable later. >> > >> >> > >> Docker container for Apache Flink is named "flink" and they have >> different tags for different releases and configurations >> https://hub.docker.com/_/flink .We can follow a similar model and can >> name the image as "beam" (beam doesn't seem to be taken on docker hub) and >> use tags to distinguish Java/Python/Go and versions etc. >> > >> >> > >> Tags will look like: >> > >> java-SNAPSHOT >> > >> java-2.10.1 >> > >> python2-SNAPSHOT >> > >> python2-2.10.1 >> > >> go-SNAPSHOT >> > >> go-2.10.1 >> > >> >> > >> >> > >> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay >> wrote: >> > >>> >> > >>> For snapshots, we could use gcr.io. Permission would not be a >> problem since Jenkins is already correctly setup. The cost will be covered >> under apache-beam-testing project. And since this is only for snapshots, it >> will be only for temporary artifacts not for release artifacts. >> > >>> >> > >>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> > >> > +1, releasing containers is a useful process that we need to >> build in Beam and it is required for FnApi users. Among other reasons, >> having officially-released Beam SDK harness container images will make it >> easier for users to do simple customizations to container images, as they >> will be able to use container image released by Beam as a base image. >> > >> > Good point about potential storage limitations on Bintray. >> With Beam Release cadence we may quickly exceed the 10 GB quota. It may >> also affect our decisions as to which images we want to release, for >> example: do we want to only release one container image with Python 3 >> interpreter, or do we want to release a container image for each Python 3 >> minor version that Beam is compatible with. >> > >>> >> > >>> >> > >>> Probably worth a separate discussion. I would favor first >> releasing a python 3 compatible version before figuring out how we would >> target multiple python 3 versions. >> > >>> >> > >>> >> > >> > >> > On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka < >> goe...@google.com> wrote: >> > > >> > > >> > > >> > > On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay >> wrote: >> > >> >> > >> >> > >> >> > >> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka < >> goe...@google.com> wrote: >> > >>> >> > >>> - Could we start from snapshots first and then do it for >> releases? >> > >>> +1, releasing snapsots first makes sense to me. >> > >>> - For snapshots, do we need to clean old containers after a >> while? Otherwise I guess we will accumulate lots of containers. >> > >>> For snap shots we can maintain a single snapshot image from >> git HEAD daily. Docker has the internal image container id which changes >> everytime an image is changed and pulls new images as needed. >> > >> >> > >> >> > >> There is a potential use this may not work with. If a user >> picks up a
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
+1 On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía wrote: > +1 > > On Mon, May 27, 2019 at 3:35 PM Maximilian Michels wrote: > > > > +1 > > > > On 27.05.19 14:04, Robert Bradshaw wrote: > > > Sounds like everyone's onboard with the plan. Any chance we could > > > publish these for the upcoming 2.13 release? > > > > > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy > wrote: > > >> > > >> +1 to have a registry for images accessible to anyone. For snapshot > images, I agree that gcr + apache-beam-testing project seems a good and > easy way to start with. > > >> > > >> Łukasz > > >> > > >> wt., 22 sty 2019 o 19:43 Mark Liu napisał(a): > > >>> > > >>> +1 to have an official Beam released container image. > > >>> > > >>> Also I would propose to add a verification step to (or after) the > release process to do smoke check. Python have ValidatesContainer test that > runs basic pipeline using newly built container for verification. Other sdk > languages can do similar thing or add a common framework. > > >>> > > >>> Mark > > >>> > > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold > wrote: > > > > +1 This would be great. gcr.io seems like a good option for > snapshots due to the permissions from jenkins to upload and ability to keep > snapshots around. > > > > On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang > wrote: > > > > > > +1 This would be a great thing to have. > > > > > > On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka > wrote: > > >> > > >> grc.io seems to be a good option. Given that we don't need the > hosting server name in the image name makes it easily changeable later. > > >> > > >> Docker container for Apache Flink is named "flink" and they have > different tags for different releases and configurations > https://hub.docker.com/_/flink .We can follow a similar model and can > name the image as "beam" (beam doesn't seem to be taken on docker hub) and > use tags to distinguish Java/Python/Go and versions etc. > > >> > > >> Tags will look like: > > >> java-SNAPSHOT > > >> java-2.10.1 > > >> python2-SNAPSHOT > > >> python2-2.10.1 > > >> go-SNAPSHOT > > >> go-2.10.1 > > >> > > >> > > >> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay > wrote: > > >>> > > >>> For snapshots, we could use gcr.io. Permission would not be a > problem since Jenkins is already correctly setup. The cost will be covered > under apache-beam-testing project. And since this is only for snapshots, it > will be only for temporary artifacts not for release artifacts. > > >>> > > >>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev < > valen...@google.com> wrote: > > > > +1, releasing containers is a useful process that we need to > build in Beam and it is required for FnApi users. Among other reasons, > having officially-released Beam SDK harness container images will make it > easier for users to do simple customizations to container images, as they > will be able to use container image released by Beam as a base image. > > > > Good point about potential storage limitations on Bintray. With > Beam Release cadence we may quickly exceed the 10 GB quota. It may also > affect our decisions as to which images we want to release, for example: do > we want to only release one container image with Python 3 interpreter, or > do we want to release a container image for each Python 3 minor version > that Beam is compatible with. > > >>> > > >>> > > >>> Probably worth a separate discussion. I would favor first > releasing a python 3 compatible version before figuring out how we would > target multiple python 3 versions. > > >>> > > >>> > > > > > > On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka > wrote: > > > > > > > > > > > > On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay > wrote: > > >> > > >> > > >> > > >> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka < > goe...@google.com> wrote: > > >>> > > >>> - Could we start from snapshots first and then do it for > releases? > > >>> +1, releasing snapsots first makes sense to me. > > >>> - For snapshots, do we need to clean old containers after a > while? Otherwise I guess we will accumulate lots of containers. > > >>> For snap shots we can maintain a single snapshot image from > git HEAD daily. Docker has the internal image container id which changes > everytime an image is changed and pulls new images as needed. > > >> > > >> > > >> There is a potential use this may not work with. If a user > picks up a snaphsot build and want to use it until the next release > arrives. I guess in that case the user can copy the snapshotted container > image and rely on that. > > >> > > > > > > Yes, that should be reasonable. > > >>> > > >>> - Do we also need
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
+1 On Mon, May 27, 2019 at 3:35 PM Maximilian Michels wrote: > > +1 > > On 27.05.19 14:04, Robert Bradshaw wrote: > > Sounds like everyone's onboard with the plan. Any chance we could > > publish these for the upcoming 2.13 release? > > > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy wrote: > >> > >> +1 to have a registry for images accessible to anyone. For snapshot > >> images, I agree that gcr + apache-beam-testing project seems a good and > >> easy way to start with. > >> > >> Łukasz > >> > >> wt., 22 sty 2019 o 19:43 Mark Liu napisał(a): > >>> > >>> +1 to have an official Beam released container image. > >>> > >>> Also I would propose to add a verification step to (or after) the release > >>> process to do smoke check. Python have ValidatesContainer test that runs > >>> basic pipeline using newly built container for verification. Other sdk > >>> languages can do similar thing or add a common framework. > >>> > >>> Mark > >>> > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold wrote: > > +1 This would be great. gcr.io seems like a good option for snapshots > due to the permissions from jenkins to upload and ability to keep > snapshots around. > > On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang wrote: > > > > +1 This would be a great thing to have. > > > > On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka wrote: > >> > >> grc.io seems to be a good option. Given that we don't need the hosting > >> server name in the image name makes it easily changeable later. > >> > >> Docker container for Apache Flink is named "flink" and they have > >> different tags for different releases and configurations > >> https://hub.docker.com/_/flink .We can follow a similar model and can > >> name the image as "beam" (beam doesn't seem to be taken on docker hub) > >> and use tags to distinguish Java/Python/Go and versions etc. > >> > >> Tags will look like: > >> java-SNAPSHOT > >> java-2.10.1 > >> python2-SNAPSHOT > >> python2-2.10.1 > >> go-SNAPSHOT > >> go-2.10.1 > >> > >> > >> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay wrote: > >>> > >>> For snapshots, we could use gcr.io. Permission would not be a problem > >>> since Jenkins is already correctly setup. The cost will be covered > >>> under apache-beam-testing project. And since this is only for > >>> snapshots, it will be only for temporary artifacts not for release > >>> artifacts. > >>> > >>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev > >>> wrote: > > +1, releasing containers is a useful process that we need to build > in Beam and it is required for FnApi users. Among other reasons, > having officially-released Beam SDK harness container images will > make it easier for users to do simple customizations to container > images, as they will be able to use container image released by Beam > as a base image. > > Good point about potential storage limitations on Bintray. With Beam > Release cadence we may quickly exceed the 10 GB quota. It may also > affect our decisions as to which images we want to release, for > example: do we want to only release one container image with Python > 3 interpreter, or do we want to release a container image for each > Python 3 minor version that Beam is compatible with. > >>> > >>> > >>> Probably worth a separate discussion. I would favor first releasing a > >>> python 3 compatible version before figuring out how we would target > >>> multiple python 3 versions. > >>> > >>> > > > On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka > wrote: > > > > > > > > On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay > > wrote: > >> > >> > >> > >> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka > >> wrote: > >>> > >>> - Could we start from snapshots first and then do it for releases? > >>> +1, releasing snapsots first makes sense to me. > >>> - For snapshots, do we need to clean old containers after a > >>> while? Otherwise I guess we will accumulate lots of containers. > >>> For snap shots we can maintain a single snapshot image from git > >>> HEAD daily. Docker has the internal image container id which > >>> changes everytime an image is changed and pulls new images as > >>> needed. > >> > >> > >> There is a potential use this may not work with. If a user picks > >> up a snaphsot build and want to use it until the next release > >> arrives. I guess in that case the user can copy the snapshotted > >> container image and rely on that. > >> > > > >
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
+1 On 27.05.19 14:04, Robert Bradshaw wrote: Sounds like everyone's onboard with the plan. Any chance we could publish these for the upcoming 2.13 release? On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy wrote: +1 to have a registry for images accessible to anyone. For snapshot images, I agree that gcr + apache-beam-testing project seems a good and easy way to start with. Łukasz wt., 22 sty 2019 o 19:43 Mark Liu napisał(a): +1 to have an official Beam released container image. Also I would propose to add a verification step to (or after) the release process to do smoke check. Python have ValidatesContainer test that runs basic pipeline using newly built container for verification. Other sdk languages can do similar thing or add a common framework. Mark On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold wrote: +1 This would be great. gcr.io seems like a good option for snapshots due to the permissions from jenkins to upload and ability to keep snapshots around. On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang wrote: +1 This would be a great thing to have. On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka wrote: grc.io seems to be a good option. Given that we don't need the hosting server name in the image name makes it easily changeable later. Docker container for Apache Flink is named "flink" and they have different tags for different releases and configurations https://hub.docker.com/_/flink .We can follow a similar model and can name the image as "beam" (beam doesn't seem to be taken on docker hub) and use tags to distinguish Java/Python/Go and versions etc. Tags will look like: java-SNAPSHOT java-2.10.1 python2-SNAPSHOT python2-2.10.1 go-SNAPSHOT go-2.10.1 On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay wrote: For snapshots, we could use gcr.io. Permission would not be a problem since Jenkins is already correctly setup. The cost will be covered under apache-beam-testing project. And since this is only for snapshots, it will be only for temporary artifacts not for release artifacts. On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev wrote: +1, releasing containers is a useful process that we need to build in Beam and it is required for FnApi users. Among other reasons, having officially-released Beam SDK harness container images will make it easier for users to do simple customizations to container images, as they will be able to use container image released by Beam as a base image. Good point about potential storage limitations on Bintray. With Beam Release cadence we may quickly exceed the 10 GB quota. It may also affect our decisions as to which images we want to release, for example: do we want to only release one container image with Python 3 interpreter, or do we want to release a container image for each Python 3 minor version that Beam is compatible with. Probably worth a separate discussion. I would favor first releasing a python 3 compatible version before figuring out how we would target multiple python 3 versions. On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka wrote: On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: - Could we start from snapshots first and then do it for releases? +1, releasing snapsots first makes sense to me. - For snapshots, do we need to clean old containers after a while? Otherwise I guess we will accumulate lots of containers. For snap shots we can maintain a single snapshot image from git HEAD daily. Docker has the internal image container id which changes everytime an image is changed and pulls new images as needed. There is a potential use this may not work with. If a user picks up a snaphsot build and want to use it until the next release arrives. I guess in that case the user can copy the snapshotted container image and rely on that. Yes, that should be reasonable. - Do we also need additional code changes for snapshots and releases to default to these specific containers? There could be a version based mechanism to resolve the correct container to use. The current image defaults have username in it. We should be ok by just updating the default image url to published image url. We should also check for pricing and details about Apache-Bintray agreement before pushing images and changing defaults. There is information on bintray's pricing page about open source projects [1]. I do not know if there is a special apache-bintray agreement or not. If there is no special agreement there is a 10GB storage limit for using bintray. As each image can easily run into Gigs, 10GB might not be sufficient for future proofing. We can also register docker image to docker image registry and not have bintray in the name to later host images on a different vendor for future proofing. [1] https://bintray.com/account/pricing?tab=account=pricing On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay wrote: This sounds like a good idea. Some
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
Sounds like everyone's onboard with the plan. Any chance we could publish these for the upcoming 2.13 release? On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy wrote: > > +1 to have a registry for images accessible to anyone. For snapshot images, I > agree that gcr + apache-beam-testing project seems a good and easy way to > start with. > > Łukasz > > wt., 22 sty 2019 o 19:43 Mark Liu napisał(a): >> >> +1 to have an official Beam released container image. >> >> Also I would propose to add a verification step to (or after) the release >> process to do smoke check. Python have ValidatesContainer test that runs >> basic pipeline using newly built container for verification. Other sdk >> languages can do similar thing or add a common framework. >> >> Mark >> >> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold wrote: >>> >>> +1 This would be great. gcr.io seems like a good option for snapshots due >>> to the permissions from jenkins to upload and ability to keep snapshots >>> around. >>> >>> On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang wrote: +1 This would be a great thing to have. On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka wrote: > > grc.io seems to be a good option. Given that we don't need the hosting > server name in the image name makes it easily changeable later. > > Docker container for Apache Flink is named "flink" and they have > different tags for different releases and configurations > https://hub.docker.com/_/flink .We can follow a similar model and can > name the image as "beam" (beam doesn't seem to be taken on docker hub) > and use tags to distinguish Java/Python/Go and versions etc. > > Tags will look like: > java-SNAPSHOT > java-2.10.1 > python2-SNAPSHOT > python2-2.10.1 > go-SNAPSHOT > go-2.10.1 > > > On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay wrote: >> >> For snapshots, we could use gcr.io. Permission would not be a problem >> since Jenkins is already correctly setup. The cost will be covered under >> apache-beam-testing project. And since this is only for snapshots, it >> will be only for temporary artifacts not for release artifacts. >> >> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev >> wrote: >>> >>> +1, releasing containers is a useful process that we need to build in >>> Beam and it is required for FnApi users. Among other reasons, having >>> officially-released Beam SDK harness container images will make it >>> easier for users to do simple customizations to container images, as >>> they will be able to use container image released by Beam as a base >>> image. >>> >>> Good point about potential storage limitations on Bintray. With Beam >>> Release cadence we may quickly exceed the 10 GB quota. It may also >>> affect our decisions as to which images we want to release, for >>> example: do we want to only release one container image with Python 3 >>> interpreter, or do we want to release a container image for each Python >>> 3 minor version that Beam is compatible with. >> >> >> Probably worth a separate discussion. I would favor first releasing a >> python 3 compatible version before figuring out how we would target >> multiple python 3 versions. >> >> >>> >>> >>> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka wrote: On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: > > > > On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka > wrote: >> >> - Could we start from snapshots first and then do it for releases? >> +1, releasing snapsots first makes sense to me. >> - For snapshots, do we need to clean old containers after a while? >> Otherwise I guess we will accumulate lots of containers. >> For snap shots we can maintain a single snapshot image from git HEAD >> daily. Docker has the internal image container id which changes >> everytime an image is changed and pulls new images as needed. > > > There is a potential use this may not work with. If a user picks up a > snaphsot build and want to use it until the next release arrives. I > guess in that case the user can copy the snapshotted container image > and rely on that. > Yes, that should be reasonable. >> >> - Do we also need additional code changes for snapshots and releases >> to default to these specific containers? There could be a version >> based mechanism to resolve the correct container to use. >> The current image defaults have username in it. We should be ok by >> just updating the default image url to published image url. >> >> We should also check for pricing and details about
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
+1 to have a registry for images accessible to anyone. For snapshot images, I agree that gcr + apache-beam-testing project seems a good and easy way to start with. Łukasz wt., 22 sty 2019 o 19:43 Mark Liu napisał(a): > +1 to have an official Beam released container image. > > Also I would propose to add a verification step to (or after) the release > process to do smoke check. Python have ValidatesContainer test that runs > basic pipeline using newly built container for verification. Other sdk > languages can do similar thing or add a common framework. > > Mark > > On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold wrote: > >> +1 This would be great. gcr.io seems like a good option for snapshots >> due to the permissions from jenkins to upload and ability to keep snapshots >> around. >> >> On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang wrote: >> >>> +1 This would be a great thing to have. >>> >>> On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka wrote: >>> grc.io seems to be a good option. Given that we don't need the hosting server name in the image name makes it easily changeable later. Docker container for Apache Flink is named "flink" and they have different tags for different releases and configurations https://hub.docker.com/_/flink .We can follow a similar model and can name the image as "beam" (beam doesn't seem to be taken on docker hub) and use tags to distinguish Java/Python/Go and versions etc. Tags will look like: java-SNAPSHOT java-2.10.1 python2-SNAPSHOT python2-2.10.1 go-SNAPSHOT go-2.10.1 On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay wrote: > For snapshots, we could use gcr.io. Permission would not be a problem > since Jenkins is already correctly setup. The cost will be covered under > apache-beam-testing project. And since this is only for snapshots, it will > be only for temporary artifacts not for release artifacts. > > On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev < > valen...@google.com> wrote: > >> +1, releasing containers is a useful process that we need to build in >> Beam and it is required for FnApi users. Among other reasons, having >> officially-released Beam SDK harness container images will make it easier >> for users to do simple customizations to container images, as they will >> be >> able to use container image released by Beam as a base image. >> >> Good point about potential storage limitations on Bintray. With Beam >> Release cadence we may quickly exceed the 10 GB quota. It may also affect >> our decisions as to which images we want to release, for example: do we >> want to only release one container image with Python 3 interpreter, or do >> we want to release a container image for each Python 3 minor version that >> Beam is compatible with. >> > > Probably worth a separate discussion. I would favor first releasing a > python 3 compatible version before figuring out how we would target > multiple python 3 versions. > > >> >> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka >> wrote: >> >>> >>> >>> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay >>> wrote: >>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: > - Could we start from snapshots first and then do it for releases? > +1, releasing snapsots first makes sense to me. > - For snapshots, do we need to clean old containers after a while? > Otherwise I guess we will accumulate lots of containers. > For snap shots we can maintain a single snapshot image from git > HEAD daily. Docker has the internal image container id which changes > everytime an image is changed and pulls new images as needed. > There is a potential use this may not work with. If a user picks up a snaphsot build and want to use it until the next release arrives. I guess in that case the user can copy the snapshotted container image and rely on that. >>> Yes, that should be reasonable. >>> - Do we also need additional code changes for snapshots and > releases to default to these specific containers? There could be a > version > based mechanism to resolve the correct container to use. > The current image defaults have username in it. We should be ok by > just updating the default image url to published image url. > > We should also check for pricing and details about Apache-Bintray > agreement before pushing images and changing defaults. > There is information on bintray's pricing page about open source projects [1]. I do not know if there is a special apache-bintray agreement or
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
+1 to have an official Beam released container image. Also I would propose to add a verification step to (or after) the release process to do smoke check. Python have ValidatesContainer test that runs basic pipeline using newly built container for verification. Other sdk languages can do similar thing or add a common framework. Mark On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold wrote: > +1 This would be great. gcr.io seems like a good option for snapshots due > to the permissions from jenkins to upload and ability to keep snapshots > around. > > On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang wrote: > >> +1 This would be a great thing to have. >> >> On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka wrote: >> >>> grc.io seems to be a good option. Given that we don't need the hosting >>> server name in the image name makes it easily changeable later. >>> >>> Docker container for Apache Flink is named "flink" and they have >>> different tags for different releases and configurations >>> https://hub.docker.com/_/flink .We can follow a similar model and can >>> name the image as "beam" (beam doesn't seem to be taken on docker hub) and >>> use tags to distinguish Java/Python/Go and versions etc. >>> >>> Tags will look like: >>> java-SNAPSHOT >>> java-2.10.1 >>> python2-SNAPSHOT >>> python2-2.10.1 >>> go-SNAPSHOT >>> go-2.10.1 >>> >>> >>> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay wrote: >>> For snapshots, we could use gcr.io. Permission would not be a problem since Jenkins is already correctly setup. The cost will be covered under apache-beam-testing project. And since this is only for snapshots, it will be only for temporary artifacts not for release artifacts. On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev < valen...@google.com> wrote: > +1, releasing containers is a useful process that we need to build in > Beam and it is required for FnApi users. Among other reasons, having > officially-released Beam SDK harness container images will make it easier > for users to do simple customizations to container images, as they will > be > able to use container image released by Beam as a base image. > > Good point about potential storage limitations on Bintray. With Beam > Release cadence we may quickly exceed the 10 GB quota. It may also affect > our decisions as to which images we want to release, for example: do we > want to only release one container image with Python 3 interpreter, or do > we want to release a container image for each Python 3 minor version that > Beam is compatible with. > Probably worth a separate discussion. I would favor first releasing a python 3 compatible version before figuring out how we would target multiple python 3 versions. >>> > > On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka > wrote: > >> >> >> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: >> >>> >>> >>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka >>> wrote: >>> - Could we start from snapshots first and then do it for releases? +1, releasing snapsots first makes sense to me. - For snapshots, do we need to clean old containers after a while? Otherwise I guess we will accumulate lots of containers. For snap shots we can maintain a single snapshot image from git HEAD daily. Docker has the internal image container id which changes everytime an image is changed and pulls new images as needed. >>> >>> There is a potential use this may not work with. If a user picks up >>> a snaphsot build and want to use it until the next release arrives. I >>> guess >>> in that case the user can copy the snapshotted container image and rely >>> on >>> that. >>> >>> >> Yes, that should be reasonable. >> >>> - Do we also need additional code changes for snapshots and releases to default to these specific containers? There could be a version based mechanism to resolve the correct container to use. The current image defaults have username in it. We should be ok by just updating the default image url to published image url. We should also check for pricing and details about Apache-Bintray agreement before pushing images and changing defaults. >>> >>> There is information on bintray's pricing page about open source >>> projects [1]. I do not know if there is a special apache-bintray >>> agreement >>> or not. If there is no special agreement there is a 10GB storage limit >>> for >>> using bintray. >>> >> As each image can easily run into Gigs, 10GB might not be sufficient >> for future proofing. >> We can also register docker image to docker image registry and not >> have bintray in the name to later host images on a different
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
+1 This would be great. gcr.io seems like a good option for snapshots due to the permissions from jenkins to upload and ability to keep snapshots around. On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang wrote: > +1 This would be a great thing to have. > > On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka wrote: > >> grc.io seems to be a good option. Given that we don't need the hosting >> server name in the image name makes it easily changeable later. >> >> Docker container for Apache Flink is named "flink" and they have >> different tags for different releases and configurations >> https://hub.docker.com/_/flink .We can follow a similar model and can >> name the image as "beam" (beam doesn't seem to be taken on docker hub) and >> use tags to distinguish Java/Python/Go and versions etc. >> >> Tags will look like: >> java-SNAPSHOT >> java-2.10.1 >> python2-SNAPSHOT >> python2-2.10.1 >> go-SNAPSHOT >> go-2.10.1 >> >> >> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay wrote: >> >>> For snapshots, we could use gcr.io. Permission would not be a problem >>> since Jenkins is already correctly setup. The cost will be covered under >>> apache-beam-testing project. And since this is only for snapshots, it will >>> be only for temporary artifacts not for release artifacts. >>> >>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev >>> wrote: >>> +1, releasing containers is a useful process that we need to build in Beam and it is required for FnApi users. Among other reasons, having officially-released Beam SDK harness container images will make it easier for users to do simple customizations to container images, as they will be able to use container image released by Beam as a base image. Good point about potential storage limitations on Bintray. With Beam Release cadence we may quickly exceed the 10 GB quota. It may also affect our decisions as to which images we want to release, for example: do we want to only release one container image with Python 3 interpreter, or do we want to release a container image for each Python 3 minor version that Beam is compatible with. >>> >>> Probably worth a separate discussion. I would favor first releasing a >>> python 3 compatible version before figuring out how we would target >>> multiple python 3 versions. >>> >> >>> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka wrote: > > > On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: > >> >> >> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka >> wrote: >> >>> - Could we start from snapshots first and then do it for releases? >>> +1, releasing snapsots first makes sense to me. >>> - For snapshots, do we need to clean old containers after a while? >>> Otherwise I guess we will accumulate lots of containers. >>> For snap shots we can maintain a single snapshot image from git HEAD >>> daily. Docker has the internal image container id which changes >>> everytime >>> an image is changed and pulls new images as needed. >>> >> >> There is a potential use this may not work with. If a user picks up a >> snaphsot build and want to use it until the next release arrives. I guess >> in that case the user can copy the snapshotted container image and rely >> on >> that. >> >> > Yes, that should be reasonable. > >> - Do we also need additional code changes for snapshots and releases >>> to default to these specific containers? There could be a version based >>> mechanism to resolve the correct container to use. >>> The current image defaults have username in it. We should be ok by >>> just updating the default image url to published image url. >>> >>> We should also check for pricing and details about Apache-Bintray >>> agreement before pushing images and changing defaults. >>> >> >> There is information on bintray's pricing page about open source >> projects [1]. I do not know if there is a special apache-bintray >> agreement >> or not. If there is no special agreement there is a 10GB storage limit >> for >> using bintray. >> > As each image can easily run into Gigs, 10GB might not be sufficient > for future proofing. > We can also register docker image to docker image registry and not > have bintray in the name to later host images on a different vendor for > future proofing. > > >> [1] https://bintray.com/account/pricing?tab=account=pricing >> >> >>> >>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay >>> wrote: >>> This sounds like a good idea. Some questions: - Could we start from snapshots first and then do it for releases? - For snapshots, do we need to clean old containers after a while? Otherwise I guess we will accumulate lots of containers. - Do we also need additional code changes
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
+1 This would be a great thing to have. On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka wrote: > grc.io seems to be a good option. Given that we don't need the hosting > server name in the image name makes it easily changeable later. > > Docker container for Apache Flink is named "flink" and they have different > tags for different releases and configurations > https://hub.docker.com/_/flink .We can follow a similar model and can > name the image as "beam" (beam doesn't seem to be taken on docker hub) and > use tags to distinguish Java/Python/Go and versions etc. > > Tags will look like: > java-SNAPSHOT > java-2.10.1 > python2-SNAPSHOT > python2-2.10.1 > go-SNAPSHOT > go-2.10.1 > > > On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay wrote: > >> For snapshots, we could use gcr.io. Permission would not be a problem >> since Jenkins is already correctly setup. The cost will be covered under >> apache-beam-testing project. And since this is only for snapshots, it will >> be only for temporary artifacts not for release artifacts. >> >> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev >> wrote: >> >>> +1, releasing containers is a useful process that we need to build in >>> Beam and it is required for FnApi users. Among other reasons, having >>> officially-released Beam SDK harness container images will make it easier >>> for users to do simple customizations to container images, as they will be >>> able to use container image released by Beam as a base image. >>> >>> Good point about potential storage limitations on Bintray. With Beam >>> Release cadence we may quickly exceed the 10 GB quota. It may also affect >>> our decisions as to which images we want to release, for example: do we >>> want to only release one container image with Python 3 interpreter, or do >>> we want to release a container image for each Python 3 minor version that >>> Beam is compatible with. >>> >> >> Probably worth a separate discussion. I would favor first releasing a >> python 3 compatible version before figuring out how we would target >> multiple python 3 versions. >> > >> >>> >>> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka wrote: >>> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: > > > On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka > wrote: > >> - Could we start from snapshots first and then do it for releases? >> +1, releasing snapsots first makes sense to me. >> - For snapshots, do we need to clean old containers after a while? >> Otherwise I guess we will accumulate lots of containers. >> For snap shots we can maintain a single snapshot image from git HEAD >> daily. Docker has the internal image container id which changes everytime >> an image is changed and pulls new images as needed. >> > > There is a potential use this may not work with. If a user picks up a > snaphsot build and want to use it until the next release arrives. I guess > in that case the user can copy the snapshotted container image and rely on > that. > > Yes, that should be reasonable. > - Do we also need additional code changes for snapshots and releases >> to default to these specific containers? There could be a version based >> mechanism to resolve the correct container to use. >> The current image defaults have username in it. We should be ok by >> just updating the default image url to published image url. >> >> We should also check for pricing and details about Apache-Bintray >> agreement before pushing images and changing defaults. >> > > There is information on bintray's pricing page about open source > projects [1]. I do not know if there is a special apache-bintray agreement > or not. If there is no special agreement there is a 10GB storage limit for > using bintray. > As each image can easily run into Gigs, 10GB might not be sufficient for future proofing. We can also register docker image to docker image registry and not have bintray in the name to later host images on a different vendor for future proofing. > [1] https://bintray.com/account/pricing?tab=account=pricing > > >> >> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay wrote: >> >>> This sounds like a good idea. Some questions: >>> >>> - Could we start from snapshots first and then do it for releases? >>> - For snapshots, do we need to clean old containers after a while? >>> Otherwise I guess we will accumulate lots of containers. >>> - Do we also need additional code changes for snapshots and releases >>> to default to these specific containers? There could be a version based >>> mechanism to resolve the correct container to use. >>> >>> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka >>> wrote: >>> Hi All, As portability/FnApi is taking shape and are compatible with ULR and
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
grc.io seems to be a good option. Given that we don't need the hosting server name in the image name makes it easily changeable later. Docker container for Apache Flink is named "flink" and they have different tags for different releases and configurations https://hub.docker.com/_/flink .We can follow a similar model and can name the image as "beam" (beam doesn't seem to be taken on docker hub) and use tags to distinguish Java/Python/Go and versions etc. Tags will look like: java-SNAPSHOT java-2.10.1 python2-SNAPSHOT python2-2.10.1 go-SNAPSHOT go-2.10.1 On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay wrote: > For snapshots, we could use gcr.io. Permission would not be a problem > since Jenkins is already correctly setup. The cost will be covered under > apache-beam-testing project. And since this is only for snapshots, it will > be only for temporary artifacts not for release artifacts. > > On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev > wrote: > >> +1, releasing containers is a useful process that we need to build in >> Beam and it is required for FnApi users. Among other reasons, having >> officially-released Beam SDK harness container images will make it easier >> for users to do simple customizations to container images, as they will be >> able to use container image released by Beam as a base image. >> >> Good point about potential storage limitations on Bintray. With Beam >> Release cadence we may quickly exceed the 10 GB quota. It may also affect >> our decisions as to which images we want to release, for example: do we >> want to only release one container image with Python 3 interpreter, or do >> we want to release a container image for each Python 3 minor version that >> Beam is compatible with. >> > > Probably worth a separate discussion. I would favor first releasing a > python 3 compatible version before figuring out how we would target > multiple python 3 versions. > > >> >> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka wrote: >> >>> >>> >>> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: >>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: > - Could we start from snapshots first and then do it for releases? > +1, releasing snapsots first makes sense to me. > - For snapshots, do we need to clean old containers after a while? > Otherwise I guess we will accumulate lots of containers. > For snap shots we can maintain a single snapshot image from git HEAD > daily. Docker has the internal image container id which changes everytime > an image is changed and pulls new images as needed. > There is a potential use this may not work with. If a user picks up a snaphsot build and want to use it until the next release arrives. I guess in that case the user can copy the snapshotted container image and rely on that. >>> Yes, that should be reasonable. >>> - Do we also need additional code changes for snapshots and releases to > default to these specific containers? There could be a version based > mechanism to resolve the correct container to use. > The current image defaults have username in it. We should be ok by > just updating the default image url to published image url. > > We should also check for pricing and details about Apache-Bintray > agreement before pushing images and changing defaults. > There is information on bintray's pricing page about open source projects [1]. I do not know if there is a special apache-bintray agreement or not. If there is no special agreement there is a 10GB storage limit for using bintray. >>> As each image can easily run into Gigs, 10GB might not be sufficient for >>> future proofing. >>> We can also register docker image to docker image registry and not have >>> bintray in the name to later host images on a different vendor for future >>> proofing. >>> >>> [1] https://bintray.com/account/pricing?tab=account=pricing > > On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay wrote: > >> This sounds like a good idea. Some questions: >> >> - Could we start from snapshots first and then do it for releases? >> - For snapshots, do we need to clean old containers after a while? >> Otherwise I guess we will accumulate lots of containers. >> - Do we also need additional code changes for snapshots and releases >> to default to these specific containers? There could be a version based >> mechanism to resolve the correct container to use. >> >> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka >> wrote: >> >>> Hi All, >>> >>> As portability/FnApi is taking shape and are compatible with ULR and >>> Flink. I wanted to discuss the release plan release of SDKHarness Docker >>> images. Of-course users can create their own images but it will be >>> useful >>> to have a default image available out of box. >>> Pre build
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
For snapshots, we could use gcr.io. Permission would not be a problem since Jenkins is already correctly setup. The cost will be covered under apache-beam-testing project. And since this is only for snapshots, it will be only for temporary artifacts not for release artifacts. On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev wrote: > +1, releasing containers is a useful process that we need to build in Beam > and it is required for FnApi users. Among other reasons, having > officially-released Beam SDK harness container images will make it easier > for users to do simple customizations to container images, as they will be > able to use container image released by Beam as a base image. > > Good point about potential storage limitations on Bintray. With Beam > Release cadence we may quickly exceed the 10 GB quota. It may also affect > our decisions as to which images we want to release, for example: do we > want to only release one container image with Python 3 interpreter, or do > we want to release a container image for each Python 3 minor version that > Beam is compatible with. > Probably worth a separate discussion. I would favor first releasing a python 3 compatible version before figuring out how we would target multiple python 3 versions. > > On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka wrote: > >> >> >> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: >> >>> >>> >>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: >>> - Could we start from snapshots first and then do it for releases? +1, releasing snapsots first makes sense to me. - For snapshots, do we need to clean old containers after a while? Otherwise I guess we will accumulate lots of containers. For snap shots we can maintain a single snapshot image from git HEAD daily. Docker has the internal image container id which changes everytime an image is changed and pulls new images as needed. >>> >>> There is a potential use this may not work with. If a user picks up a >>> snaphsot build and want to use it until the next release arrives. I guess >>> in that case the user can copy the snapshotted container image and rely on >>> that. >>> >>> >> Yes, that should be reasonable. >> >>> - Do we also need additional code changes for snapshots and releases to default to these specific containers? There could be a version based mechanism to resolve the correct container to use. The current image defaults have username in it. We should be ok by just updating the default image url to published image url. We should also check for pricing and details about Apache-Bintray agreement before pushing images and changing defaults. >>> >>> There is information on bintray's pricing page about open source >>> projects [1]. I do not know if there is a special apache-bintray agreement >>> or not. If there is no special agreement there is a 10GB storage limit for >>> using bintray. >>> >> As each image can easily run into Gigs, 10GB might not be sufficient for >> future proofing. >> We can also register docker image to docker image registry and not have >> bintray in the name to later host images on a different vendor for future >> proofing. >> >> >>> [1] https://bintray.com/account/pricing?tab=account=pricing >>> >>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay wrote: > This sounds like a good idea. Some questions: > > - Could we start from snapshots first and then do it for releases? > - For snapshots, do we need to clean old containers after a while? > Otherwise I guess we will accumulate lots of containers. > - Do we also need additional code changes for snapshots and releases > to default to these specific containers? There could be a version based > mechanism to resolve the correct container to use. > > On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka > wrote: > >> Hi All, >> >> As portability/FnApi is taking shape and are compatible with ULR and >> Flink. I wanted to discuss the release plan release of SDKHarness Docker >> images. Of-course users can create their own images but it will be useful >> to have a default image available out of box. >> Pre build image are a must for making FnApi available for users and >> not just the developers. >> The other purpose of these images is to be server as base image layer >> for building custom images. >> >> Apache already have bintray repositories for beam. >> https://bintray.com/apache/beam-snapshots-docker >> https://bintray.com/apache/beam-docker >> >> Shall we start pushing Python/Java/Go SDK Harness containers to >> https://bintray.com/apache/beam-docker for beam release and maintain >> daily snapshot at https://bintray.com/apache/beam-snapshots-docker ? >> >> Thanks, >> Ankur >> >
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
+1, releasing containers is a useful process that we need to build in Beam and it is required for FnApi users. Among other reasons, having officially-released Beam SDK harness container images will make it easier for users to do simple customizations to container images, as they will be able to use container image released by Beam as a base image. Good point about potential storage limitations on Bintray. With Beam Release cadence we may quickly exceed the 10 GB quota. It may also affect our decisions as to which images we want to release, for example: do we want to only release one container image with Python 3 interpreter, or do we want to release a container image for each Python 3 minor version that Beam is compatible with. On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka wrote: > > > On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: > >> >> >> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: >> >>> - Could we start from snapshots first and then do it for releases? >>> +1, releasing snapsots first makes sense to me. >>> - For snapshots, do we need to clean old containers after a while? >>> Otherwise I guess we will accumulate lots of containers. >>> For snap shots we can maintain a single snapshot image from git HEAD >>> daily. Docker has the internal image container id which changes everytime >>> an image is changed and pulls new images as needed. >>> >> >> There is a potential use this may not work with. If a user picks up a >> snaphsot build and want to use it until the next release arrives. I guess >> in that case the user can copy the snapshotted container image and rely on >> that. >> >> > Yes, that should be reasonable. > >> - Do we also need additional code changes for snapshots and releases to >>> default to these specific containers? There could be a version based >>> mechanism to resolve the correct container to use. >>> The current image defaults have username in it. We should be ok by just >>> updating the default image url to published image url. >>> >>> We should also check for pricing and details about Apache-Bintray >>> agreement before pushing images and changing defaults. >>> >> >> There is information on bintray's pricing page about open source projects >> [1]. I do not know if there is a special apache-bintray agreement or not. >> If there is no special agreement there is a 10GB storage limit for using >> bintray. >> > As each image can easily run into Gigs, 10GB might not be sufficient for > future proofing. > We can also register docker image to docker image registry and not have > bintray in the name to later host images on a different vendor for future > proofing. > > >> [1] https://bintray.com/account/pricing?tab=account=pricing >> >> >>> >>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay wrote: >>> This sounds like a good idea. Some questions: - Could we start from snapshots first and then do it for releases? - For snapshots, do we need to clean old containers after a while? Otherwise I guess we will accumulate lots of containers. - Do we also need additional code changes for snapshots and releases to default to these specific containers? There could be a version based mechanism to resolve the correct container to use. On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka wrote: > Hi All, > > As portability/FnApi is taking shape and are compatible with ULR and > Flink. I wanted to discuss the release plan release of SDKHarness Docker > images. Of-course users can create their own images but it will be useful > to have a default image available out of box. > Pre build image are a must for making FnApi available for users and > not just the developers. > The other purpose of these images is to be server as base image layer > for building custom images. > > Apache already have bintray repositories for beam. > https://bintray.com/apache/beam-snapshots-docker > https://bintray.com/apache/beam-docker > > Shall we start pushing Python/Java/Go SDK Harness containers to > https://bintray.com/apache/beam-docker for beam release and maintain > daily snapshot at https://bintray.com/apache/beam-snapshots-docker ? > > Thanks, > Ankur >
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: > > > On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: > >> - Could we start from snapshots first and then do it for releases? >> +1, releasing snapsots first makes sense to me. >> - For snapshots, do we need to clean old containers after a while? >> Otherwise I guess we will accumulate lots of containers. >> For snap shots we can maintain a single snapshot image from git HEAD >> daily. Docker has the internal image container id which changes everytime >> an image is changed and pulls new images as needed. >> > > There is a potential use this may not work with. If a user picks up a > snaphsot build and want to use it until the next release arrives. I guess > in that case the user can copy the snapshotted container image and rely on > that. > > Yes, that should be reasonable. > - Do we also need additional code changes for snapshots and releases to >> default to these specific containers? There could be a version based >> mechanism to resolve the correct container to use. >> The current image defaults have username in it. We should be ok by just >> updating the default image url to published image url. >> >> We should also check for pricing and details about Apache-Bintray >> agreement before pushing images and changing defaults. >> > > There is information on bintray's pricing page about open source projects > [1]. I do not know if there is a special apache-bintray agreement or not. > If there is no special agreement there is a 10GB storage limit for using > bintray. > As each image can easily run into Gigs, 10GB might not be sufficient for future proofing. We can also register docker image to docker image registry and not have bintray in the name to later host images on a different vendor for future proofing. > [1] https://bintray.com/account/pricing?tab=account=pricing > > >> >> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay wrote: >> >>> This sounds like a good idea. Some questions: >>> >>> - Could we start from snapshots first and then do it for releases? >>> - For snapshots, do we need to clean old containers after a while? >>> Otherwise I guess we will accumulate lots of containers. >>> - Do we also need additional code changes for snapshots and releases to >>> default to these specific containers? There could be a version based >>> mechanism to resolve the correct container to use. >>> >>> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka wrote: >>> Hi All, As portability/FnApi is taking shape and are compatible with ULR and Flink. I wanted to discuss the release plan release of SDKHarness Docker images. Of-course users can create their own images but it will be useful to have a default image available out of box. Pre build image are a must for making FnApi available for users and not just the developers. The other purpose of these images is to be server as base image layer for building custom images. Apache already have bintray repositories for beam. https://bintray.com/apache/beam-snapshots-docker https://bintray.com/apache/beam-docker Shall we start pushing Python/Java/Go SDK Harness containers to https://bintray.com/apache/beam-docker for beam release and maintain daily snapshot at https://bintray.com/apache/beam-snapshots-docker ? Thanks, Ankur >>>
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
Sounds good. I set up the bintray stuff a while ago but got stuck on perms to have Jenkins upload the snapshot, and the release was not really relevant. Kenn On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: > > > On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: > >> - Could we start from snapshots first and then do it for releases? >> +1, releasing snapsots first makes sense to me. >> - For snapshots, do we need to clean old containers after a while? >> Otherwise I guess we will accumulate lots of containers. >> For snap shots we can maintain a single snapshot image from git HEAD >> daily. Docker has the internal image container id which changes everytime >> an image is changed and pulls new images as needed. >> > > There is a potential use this may not work with. If a user picks up a > snaphsot build and want to use it until the next release arrives. I guess > in that case the user can copy the snapshotted container image and rely on > that. > > >> - Do we also need additional code changes for snapshots and releases to >> default to these specific containers? There could be a version based >> mechanism to resolve the correct container to use. >> The current image defaults have username in it. We should be ok by just >> updating the default image url to published image url. >> >> We should also check for pricing and details about Apache-Bintray >> agreement before pushing images and changing defaults. >> > > There is information on bintray's pricing page about open source projects > [1]. I do not know if there is a special apache-bintray agreement or not. > If there is no special agreement there is a 10GB storage limit for using > bintray. > > [1] https://bintray.com/account/pricing?tab=account=pricing > > >> >> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay wrote: >> >>> This sounds like a good idea. Some questions: >>> >>> - Could we start from snapshots first and then do it for releases? >>> - For snapshots, do we need to clean old containers after a while? >>> Otherwise I guess we will accumulate lots of containers. >>> - Do we also need additional code changes for snapshots and releases to >>> default to these specific containers? There could be a version based >>> mechanism to resolve the correct container to use. >>> >>> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka wrote: >>> Hi All, As portability/FnApi is taking shape and are compatible with ULR and Flink. I wanted to discuss the release plan release of SDKHarness Docker images. Of-course users can create their own images but it will be useful to have a default image available out of box. Pre build image are a must for making FnApi available for users and not just the developers. The other purpose of these images is to be server as base image layer for building custom images. Apache already have bintray repositories for beam. https://bintray.com/apache/beam-snapshots-docker https://bintray.com/apache/beam-docker Shall we start pushing Python/Java/Go SDK Harness containers to https://bintray.com/apache/beam-docker for beam release and maintain daily snapshot at https://bintray.com/apache/beam-snapshots-docker ? Thanks, Ankur >>>
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: > - Could we start from snapshots first and then do it for releases? > +1, releasing snapsots first makes sense to me. > - For snapshots, do we need to clean old containers after a while? > Otherwise I guess we will accumulate lots of containers. > For snap shots we can maintain a single snapshot image from git HEAD > daily. Docker has the internal image container id which changes everytime > an image is changed and pulls new images as needed. > There is a potential use this may not work with. If a user picks up a snaphsot build and want to use it until the next release arrives. I guess in that case the user can copy the snapshotted container image and rely on that. > - Do we also need additional code changes for snapshots and releases to > default to these specific containers? There could be a version based > mechanism to resolve the correct container to use. > The current image defaults have username in it. We should be ok by just > updating the default image url to published image url. > > We should also check for pricing and details about Apache-Bintray > agreement before pushing images and changing defaults. > There is information on bintray's pricing page about open source projects [1]. I do not know if there is a special apache-bintray agreement or not. If there is no special agreement there is a 10GB storage limit for using bintray. [1] https://bintray.com/account/pricing?tab=account=pricing > > On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay wrote: > >> This sounds like a good idea. Some questions: >> >> - Could we start from snapshots first and then do it for releases? >> - For snapshots, do we need to clean old containers after a while? >> Otherwise I guess we will accumulate lots of containers. >> - Do we also need additional code changes for snapshots and releases to >> default to these specific containers? There could be a version based >> mechanism to resolve the correct container to use. >> >> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka wrote: >> >>> Hi All, >>> >>> As portability/FnApi is taking shape and are compatible with ULR and >>> Flink. I wanted to discuss the release plan release of SDKHarness Docker >>> images. Of-course users can create their own images but it will be useful >>> to have a default image available out of box. >>> Pre build image are a must for making FnApi available for users and not >>> just the developers. >>> The other purpose of these images is to be server as base image layer >>> for building custom images. >>> >>> Apache already have bintray repositories for beam. >>> https://bintray.com/apache/beam-snapshots-docker >>> https://bintray.com/apache/beam-docker >>> >>> Shall we start pushing Python/Java/Go SDK Harness containers to >>> https://bintray.com/apache/beam-docker for beam release and maintain >>> daily snapshot at https://bintray.com/apache/beam-snapshots-docker ? >>> >>> Thanks, >>> Ankur >>> >>
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
- Could we start from snapshots first and then do it for releases? +1, releasing snapsots first makes sense to me. - For snapshots, do we need to clean old containers after a while? Otherwise I guess we will accumulate lots of containers. For snap shots we can maintain a single snapshot image from git HEAD daily. Docker has the internal image container id which changes everytime an image is changed and pulls new images as needed. - Do we also need additional code changes for snapshots and releases to default to these specific containers? There could be a version based mechanism to resolve the correct container to use. The current image defaults have username in it. We should be ok by just updating the default image url to published image url. We should also check for pricing and details about Apache-Bintray agreement before pushing images and changing defaults. On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay wrote: > This sounds like a good idea. Some questions: > > - Could we start from snapshots first and then do it for releases? > - For snapshots, do we need to clean old containers after a while? > Otherwise I guess we will accumulate lots of containers. > - Do we also need additional code changes for snapshots and releases to > default to these specific containers? There could be a version based > mechanism to resolve the correct container to use. > > On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka wrote: > >> Hi All, >> >> As portability/FnApi is taking shape and are compatible with ULR and >> Flink. I wanted to discuss the release plan release of SDKHarness Docker >> images. Of-course users can create their own images but it will be useful >> to have a default image available out of box. >> Pre build image are a must for making FnApi available for users and not >> just the developers. >> The other purpose of these images is to be server as base image layer for >> building custom images. >> >> Apache already have bintray repositories for beam. >> https://bintray.com/apache/beam-snapshots-docker >> https://bintray.com/apache/beam-docker >> >> Shall we start pushing Python/Java/Go SDK Harness containers to >> https://bintray.com/apache/beam-docker for beam release and maintain >> daily snapshot at https://bintray.com/apache/beam-snapshots-docker ? >> >> Thanks, >> Ankur >> >
Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.
This sounds like a good idea. Some questions: - Could we start from snapshots first and then do it for releases? - For snapshots, do we need to clean old containers after a while? Otherwise I guess we will accumulate lots of containers. - Do we also need additional code changes for snapshots and releases to default to these specific containers? There could be a version based mechanism to resolve the correct container to use. On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka wrote: > Hi All, > > As portability/FnApi is taking shape and are compatible with ULR and > Flink. I wanted to discuss the release plan release of SDKHarness Docker > images. Of-course users can create their own images but it will be useful > to have a default image available out of box. > Pre build image are a must for making FnApi available for users and not > just the developers. > The other purpose of these images is to be server as base image layer for > building custom images. > > Apache already have bintray repositories for beam. > https://bintray.com/apache/beam-snapshots-docker > https://bintray.com/apache/beam-docker > > Shall we start pushing Python/Java/Go SDK Harness containers to > https://bintray.com/apache/beam-docker for beam release and maintain > daily snapshot at https://bintray.com/apache/beam-snapshots-docker ? > > Thanks, > Ankur >
