Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-05-29 Thread Ankur Goenka
I agree, I think their are few things which have to be though through as part of Portable image release. * Where to host the images. We can ofcourse have an alias for the image which can point to a different location but the hosting location have to be sort through. * Validation process for the

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-05-28 Thread Ahmet Altay
Could we first figure out the process (where to push, how to push, permissions needed, how to validate etc.) as part of the snapshots and update the release guide based on that? On Tue, May 28, 2019 at 2:43 AM Robert Bradshaw wrote: > In the future (read, next release) the SDK will likely have

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-05-28 Thread Robert Bradshaw
In the future (read, next release) the SDK will likely have reference to the containers, so this will have to be part of the release. But I agree for 2.13 it should be more about figuring out the process and not necessarily holding back. On Mon, May 27, 2019 at 7:42 PM Ankur Goenka wrote: > > +1

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-05-27 Thread Ankur Goenka
+1 We can release the images with 2.13 but we should not block 2.13 release for this. On Mon, May 27, 2019, 8:39 AM Thomas Weise wrote: > +1 > > > On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía wrote: > >> +1 >> >> On Mon, May 27, 2019 at 3:35 PM Maximilian Michels >> wrote: >> > >> > +1 >> >

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-05-27 Thread Thomas Weise
+1 On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía wrote: > +1 > > On Mon, May 27, 2019 at 3:35 PM Maximilian Michels wrote: > > > > +1 > > > > On 27.05.19 14:04, Robert Bradshaw wrote: > > > Sounds like everyone's onboard with the plan. Any chance we could > > > publish these for the upcoming

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-05-27 Thread Ismaël Mejía
+1 On Mon, May 27, 2019 at 3:35 PM Maximilian Michels wrote: > > +1 > > On 27.05.19 14:04, Robert Bradshaw wrote: > > Sounds like everyone's onboard with the plan. Any chance we could > > publish these for the upcoming 2.13 release? > > > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy wrote: >

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-05-27 Thread Maximilian Michels
+1 On 27.05.19 14:04, Robert Bradshaw wrote: Sounds like everyone's onboard with the plan. Any chance we could publish these for the upcoming 2.13 release? On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy wrote: +1 to have a registry for images accessible to anyone. For snapshot images, I

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-05-27 Thread Robert Bradshaw
Sounds like everyone's onboard with the plan. Any chance we could publish these for the upcoming 2.13 release? On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy wrote: > > +1 to have a registry for images accessible to anyone. For snapshot images, I > agree that gcr + apache-beam-testing project

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-02-06 Thread Łukasz Gajowy
+1 to have a registry for images accessible to anyone. For snapshot images, I agree that gcr + apache-beam-testing project seems a good and easy way to start with. Łukasz wt., 22 sty 2019 o 19:43 Mark Liu napisał(a): > +1 to have an official Beam released container image. > > Also I would

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-01-22 Thread Mark Liu
+1 to have an official Beam released container image. Also I would propose to add a verification step to (or after) the release process to do smoke check. Python have ValidatesContainer test that runs basic pipeline using newly built container for verification. Other sdk languages can do similar

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-01-17 Thread Alan Myrvold
+1 This would be great. gcr.io seems like a good option for snapshots due to the permissions from jenkins to upload and ability to keep snapshots around. On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang wrote: > +1 This would be a great thing to have. > > On Wed, Jan 16, 2019 at 6:11 PM Ankur

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-01-16 Thread Ruoyun Huang
+1 This would be a great thing to have. On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka wrote: > grc.io seems to be a good option. Given that we don't need the hosting > server name in the image name makes it easily changeable later. > > Docker container for Apache Flink is named "flink" and they

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-01-16 Thread Ankur Goenka
grc.io seems to be a good option. Given that we don't need the hosting server name in the image name makes it easily changeable later. Docker container for Apache Flink is named "flink" and they have different tags for different releases and configurations https://hub.docker.com/_/flink .We can

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-01-16 Thread Ahmet Altay
For snapshots, we could use gcr.io. Permission would not be a problem since Jenkins is already correctly setup. The cost will be covered under apache-beam-testing project. And since this is only for snapshots, it will be only for temporary artifacts not for release artifacts. On Wed, Jan 16, 2019

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-01-16 Thread Valentyn Tymofieiev
+1, releasing containers is a useful process that we need to build in Beam and it is required for FnApi users. Among other reasons, having officially-released Beam SDK harness container images will make it easier for users to do simple customizations to container images, as they will be able to

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-01-16 Thread Ankur Goenka
On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: > > > On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: > >> - Could we start from snapshots first and then do it for releases? >> +1, releasing snapsots first makes sense to me. >> - For snapshots, do we need to clean old containers after a

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-01-16 Thread Kenneth Knowles
Sounds good. I set up the bintray stuff a while ago but got stuck on perms to have Jenkins upload the snapshot, and the release was not really relevant. Kenn On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay wrote: > > > On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: > >> - Could we start from

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-01-16 Thread Ahmet Altay
On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka wrote: > - Could we start from snapshots first and then do it for releases? > +1, releasing snapsots first makes sense to me. > - For snapshots, do we need to clean old containers after a while? > Otherwise I guess we will accumulate lots of

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-01-16 Thread Ankur Goenka
- Could we start from snapshots first and then do it for releases? +1, releasing snapsots first makes sense to me. - For snapshots, do we need to clean old containers after a while? Otherwise I guess we will accumulate lots of containers. For snap shots we can maintain a single snapshot image from

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

2019-01-16 Thread Ahmet Altay
This sounds like a good idea. Some questions: - Could we start from snapshots first and then do it for releases? - For snapshots, do we need to clean old containers after a while? Otherwise I guess we will accumulate lots of containers. - Do we also need additional code changes for snapshots and