+1 We can release the images with 2.13 but we should not block 2.13 release for this.
On Mon, May 27, 2019, 8:39 AM Thomas Weise <t...@apache.org> wrote: > +1 > > > On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía <ieme...@gmail.com> wrote: > >> +1 >> >> On Mon, May 27, 2019 at 3:35 PM Maximilian Michels <m...@apache.org> >> wrote: >> > >> > +1 >> > >> > On 27.05.19 14:04, Robert Bradshaw wrote: >> > > Sounds like everyone's onboard with the plan. Any chance we could >> > > publish these for the upcoming 2.13 release? >> > > >> > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy <lgaj...@apache.org> >> wrote: >> > >> >> > >> +1 to have a registry for images accessible to anyone. For snapshot >> images, I agree that gcr + apache-beam-testing project seems a good and >> easy way to start with. >> > >> >> > >> Łukasz >> > >> >> > >> wt., 22 sty 2019 o 19:43 Mark Liu <mark...@google.com> napisał(a): >> > >>> >> > >>> +1 to have an official Beam released container image. >> > >>> >> > >>> Also I would propose to add a verification step to (or after) the >> release process to do smoke check. Python have ValidatesContainer test that >> runs basic pipeline using newly built container for verification. Other sdk >> languages can do similar thing or add a common framework. >> > >>> >> > >>> Mark >> > >>> >> > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold <amyrv...@google.com> >> wrote: >> > >>>> >> > >>>> +1 This would be great. gcr.io seems like a good option for >> snapshots due to the permissions from jenkins to upload and ability to keep >> snapshots around. >> > >>>> >> > >>>> On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang <ruo...@google.com> >> wrote: >> > >>>>> >> > >>>>> +1 This would be a great thing to have. >> > >>>>> >> > >>>>> On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka <goe...@google.com> >> wrote: >> > >>>>>> >> > >>>>>> grc.io seems to be a good option. Given that we don't need the >> hosting server name in the image name makes it easily changeable later. >> > >>>>>> >> > >>>>>> Docker container for Apache Flink is named "flink" and they have >> different tags for different releases and configurations >> https://hub.docker.com/_/flink .We can follow a similar model and can >> name the image as "beam" (beam doesn't seem to be taken on docker hub) and >> use tags to distinguish Java/Python/Go and versions etc. >> > >>>>>> >> > >>>>>> Tags will look like: >> > >>>>>> java-SNAPSHOT >> > >>>>>> java-2.10.1 >> > >>>>>> python2-SNAPSHOT >> > >>>>>> python2-2.10.1 >> > >>>>>> go-SNAPSHOT >> > >>>>>> go-2.10.1 >> > >>>>>> >> > >>>>>> >> > >>>>>> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay <al...@google.com> >> wrote: >> > >>>>>>> >> > >>>>>>> For snapshots, we could use gcr.io. Permission would not be a >> problem since Jenkins is already correctly setup. The cost will be covered >> under apache-beam-testing project. And since this is only for snapshots, it >> will be only for temporary artifacts not for release artifacts. >> > >>>>>>> >> > >>>>>>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev < >> valen...@google.com> wrote: >> > >>>>>>>> >> > >>>>>>>> +1, releasing containers is a useful process that we need to >> build in Beam and it is required for FnApi users. Among other reasons, >> having officially-released Beam SDK harness container images will make it >> easier for users to do simple customizations to container images, as they >> will be able to use container image released by Beam as a base image. >> > >>>>>>>> >> > >>>>>>>> Good point about potential storage limitations on Bintray. >> With Beam Release cadence we may quickly exceed the 10 GB quota. It may >> also affect our decisions as to which images we want to release, for >> example: do we want to only release one container image with Python 3 >> interpreter, or do we want to release a container image for each Python 3 >> minor version that Beam is compatible with. >> > >>>>>>> >> > >>>>>>> >> > >>>>>>> Probably worth a separate discussion. I would favor first >> releasing a python 3 compatible version before figuring out how we would >> target multiple python 3 versions. >> > >>>>>>> >> > >>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka < >> goe...@google.com> wrote: >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay <al...@google.com> >> wrote: >> > >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka < >> goe...@google.com> wrote: >> > >>>>>>>>>>> >> > >>>>>>>>>>> - Could we start from snapshots first and then do it for >> releases? >> > >>>>>>>>>>> +1, releasing snapsots first makes sense to me. >> > >>>>>>>>>>> - For snapshots, do we need to clean old containers after a >> while? Otherwise I guess we will accumulate lots of containers. >> > >>>>>>>>>>> For snap shots we can maintain a single snapshot image from >> git HEAD daily. Docker has the internal image container id which changes >> everytime an image is changed and pulls new images as needed. >> > >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> There is a potential use this may not work with. If a user >> picks up a snaphsot build and want to use it until the next release >> arrives. I guess in that case the user can copy the snapshotted container >> image and rely on that. >> > >>>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> Yes, that should be reasonable. >> > >>>>>>>>>>> >> > >>>>>>>>>>> - Do we also need additional code changes for snapshots and >> releases to default to these specific containers? There could be a version >> based mechanism to resolve the correct container to use. >> > >>>>>>>>>>> The current image defaults have username in it. We should >> be ok by just updating the default image url to published image url. >> > >>>>>>>>>>> >> > >>>>>>>>>>> We should also check for pricing and details about >> Apache-Bintray agreement before pushing images and changing defaults. >> > >>>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> There is information on bintray's pricing page about open >> source projects [1]. I do not know if there is a special apache-bintray >> agreement or not. If there is no special agreement there is a 10GB storage >> limit for using bintray. >> > >>>>>>>>> >> > >>>>>>>>> As each image can easily run into Gigs, 10GB might not be >> sufficient for future proofing. >> > >>>>>>>>> We can also register docker image to docker image registry >> and not have bintray in the name to later host images on a different vendor >> for future proofing. >> > >>>>>>>>> >> > >>>>>>>>>> >> > >>>>>>>>>> [1] >> https://bintray.com/account/pricing?tab=account&type=pricing >> > >>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> >> > >>>>>>>>>>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay < >> al...@google.com> wrote: >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> This sounds like a good idea. Some questions: >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> - Could we start from snapshots first and then do it for >> releases? >> > >>>>>>>>>>>> - For snapshots, do we need to clean old containers after >> a while? Otherwise I guess we will accumulate lots of containers. >> > >>>>>>>>>>>> - Do we also need additional code changes for snapshots >> and releases to default to these specific containers? There could be a >> version based mechanism to resolve the correct container to use. >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka < >> goe...@google.com> wrote: >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> Hi All, >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> As portability/FnApi is taking shape and are compatible >> with ULR and Flink. I wanted to discuss the release plan release of >> SDKHarness Docker images. Of-course users can create their own images but >> it will be useful to have a default image available out of box. >> > >>>>>>>>>>>>> Pre build image are a must for making FnApi available for >> users and not just the developers. >> > >>>>>>>>>>>>> The other purpose of these images is to be server as base >> image layer for building custom images. >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> Apache already have bintray repositories for beam. >> > >>>>>>>>>>>>> https://bintray.com/apache/beam-snapshots-docker >> > >>>>>>>>>>>>> https://bintray.com/apache/beam-docker >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> Shall we start pushing Python/Java/Go SDK Harness >> containers to https://bintray.com/apache/beam-docker for beam release >> and maintain daily snapshot at >> https://bintray.com/apache/beam-snapshots-docker ? >> > >>>>>>>>>>>>> >> > >>>>>>>>>>>>> Thanks, >> > >>>>>>>>>>>>> Ankur >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> -- >> > >>>>> ================ >> > >>>>> Ruoyun Huang >> > >>>>> >> >
