+1 to have a registry for images accessible to anyone. For snapshot images, I agree that gcr + apache-beam-testing project seems a good and easy way to start with.
Łukasz wt., 22 sty 2019 o 19:43 Mark Liu <mark...@google.com> napisał(a): > +1 to have an official Beam released container image. > > Also I would propose to add a verification step to (or after) the release > process to do smoke check. Python have ValidatesContainer test that runs > basic pipeline using newly built container for verification. Other sdk > languages can do similar thing or add a common framework. > > Mark > > On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold <amyrv...@google.com> wrote: > >> +1 This would be great. gcr.io seems like a good option for snapshots >> due to the permissions from jenkins to upload and ability to keep snapshots >> around. >> >> On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang <ruo...@google.com> wrote: >> >>> +1 This would be a great thing to have. >>> >>> On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka <goe...@google.com> wrote: >>> >>>> grc.io seems to be a good option. Given that we don't need the hosting >>>> server name in the image name makes it easily changeable later. >>>> >>>> Docker container for Apache Flink is named "flink" and they have >>>> different tags for different releases and configurations >>>> https://hub.docker.com/_/flink .We can follow a similar model and can >>>> name the image as "beam" (beam doesn't seem to be taken on docker hub) and >>>> use tags to distinguish Java/Python/Go and versions etc. >>>> >>>> Tags will look like: >>>> java-SNAPSHOT >>>> java-2.10.1 >>>> python2-SNAPSHOT >>>> python2-2.10.1 >>>> go-SNAPSHOT >>>> go-2.10.1 >>>> >>>> >>>> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay <al...@google.com> wrote: >>>> >>>>> For snapshots, we could use gcr.io. Permission would not be a problem >>>>> since Jenkins is already correctly setup. The cost will be covered under >>>>> apache-beam-testing project. And since this is only for snapshots, it will >>>>> be only for temporary artifacts not for release artifacts. >>>>> >>>>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev < >>>>> valen...@google.com> wrote: >>>>> >>>>>> +1, releasing containers is a useful process that we need to build in >>>>>> Beam and it is required for FnApi users. Among other reasons, having >>>>>> officially-released Beam SDK harness container images will make it easier >>>>>> for users to do simple customizations to container images, as they will >>>>>> be >>>>>> able to use container image released by Beam as a base image. >>>>>> >>>>>> Good point about potential storage limitations on Bintray. With Beam >>>>>> Release cadence we may quickly exceed the 10 GB quota. It may also affect >>>>>> our decisions as to which images we want to release, for example: do we >>>>>> want to only release one container image with Python 3 interpreter, or do >>>>>> we want to release a container image for each Python 3 minor version that >>>>>> Beam is compatible with. >>>>>> >>>>> >>>>> Probably worth a separate discussion. I would favor first releasing a >>>>> python 3 compatible version before figuring out how we would target >>>>> multiple python 3 versions. >>>>> >>>> >>>>> >>>>>> >>>>>> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka <goe...@google.com> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay <al...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka <goe...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> - Could we start from snapshots first and then do it for releases? >>>>>>>>> +1, releasing snapsots first makes sense to me. >>>>>>>>> - For snapshots, do we need to clean old containers after a while? >>>>>>>>> Otherwise I guess we will accumulate lots of containers. >>>>>>>>> For snap shots we can maintain a single snapshot image from git >>>>>>>>> HEAD daily. Docker has the internal image container id which changes >>>>>>>>> everytime an image is changed and pulls new images as needed. >>>>>>>>> >>>>>>>> >>>>>>>> There is a potential use this may not work with. If a user picks up >>>>>>>> a snaphsot build and want to use it until the next release arrives. I >>>>>>>> guess >>>>>>>> in that case the user can copy the snapshotted container image and >>>>>>>> rely on >>>>>>>> that. >>>>>>>> >>>>>>>> >>>>>>> Yes, that should be reasonable. >>>>>>> >>>>>>>> - Do we also need additional code changes for snapshots and >>>>>>>>> releases to default to these specific containers? There could be a >>>>>>>>> version >>>>>>>>> based mechanism to resolve the correct container to use. >>>>>>>>> The current image defaults have username in it. We should be ok by >>>>>>>>> just updating the default image url to published image url. >>>>>>>>> >>>>>>>>> We should also check for pricing and details about Apache-Bintray >>>>>>>>> agreement before pushing images and changing defaults. >>>>>>>>> >>>>>>>> >>>>>>>> There is information on bintray's pricing page about open source >>>>>>>> projects [1]. I do not know if there is a special apache-bintray >>>>>>>> agreement >>>>>>>> or not. If there is no special agreement there is a 10GB storage limit >>>>>>>> for >>>>>>>> using bintray. >>>>>>>> >>>>>>> As each image can easily run into Gigs, 10GB might not be sufficient >>>>>>> for future proofing. >>>>>>> We can also register docker image to docker image registry and not >>>>>>> have bintray in the name to later host images on a different vendor for >>>>>>> future proofing. >>>>>>> >>>>>>> >>>>>>>> [1] https://bintray.com/account/pricing?tab=account&type=pricing >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay <al...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> This sounds like a good idea. Some questions: >>>>>>>>>> >>>>>>>>>> - Could we start from snapshots first and then do it for releases? >>>>>>>>>> - For snapshots, do we need to clean old containers after a >>>>>>>>>> while? Otherwise I guess we will accumulate lots of containers. >>>>>>>>>> - Do we also need additional code changes for snapshots and >>>>>>>>>> releases to default to these specific containers? There could be a >>>>>>>>>> version >>>>>>>>>> based mechanism to resolve the correct container to use. >>>>>>>>>> >>>>>>>>>> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka <goe...@google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> As portability/FnApi is taking shape and are compatible with ULR >>>>>>>>>>> and Flink. I wanted to discuss the release plan release of >>>>>>>>>>> SDKHarness >>>>>>>>>>> Docker images. Of-course users can create their own images but it >>>>>>>>>>> will be >>>>>>>>>>> useful to have a default image available out of box. >>>>>>>>>>> Pre build image are a must for making FnApi available for users >>>>>>>>>>> and not just the developers. >>>>>>>>>>> The other purpose of these images is to be server as base image >>>>>>>>>>> layer for building custom images. >>>>>>>>>>> >>>>>>>>>>> Apache already have bintray repositories for beam. >>>>>>>>>>> https://bintray.com/apache/beam-snapshots-docker >>>>>>>>>>> https://bintray.com/apache/beam-docker >>>>>>>>>>> >>>>>>>>>>> Shall we start pushing Python/Java/Go SDK Harness containers to >>>>>>>>>>> https://bintray.com/apache/beam-docker for beam release and >>>>>>>>>>> maintain daily snapshot at >>>>>>>>>>> https://bintray.com/apache/beam-snapshots-docker ? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Ankur >>>>>>>>>>> >>>>>>>>>> >>> >>> -- >>> ================ >>> Ruoyun Huang >>> >>>
