+1 This would be great. gcr.io seems like a good option for snapshots due to the permissions from jenkins to upload and ability to keep snapshots around.
On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang <ruo...@google.com> wrote: > +1 This would be a great thing to have. > > On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka <goe...@google.com> wrote: > >> grc.io seems to be a good option. Given that we don't need the hosting >> server name in the image name makes it easily changeable later. >> >> Docker container for Apache Flink is named "flink" and they have >> different tags for different releases and configurations >> https://hub.docker.com/_/flink .We can follow a similar model and can >> name the image as "beam" (beam doesn't seem to be taken on docker hub) and >> use tags to distinguish Java/Python/Go and versions etc. >> >> Tags will look like: >> java-SNAPSHOT >> java-2.10.1 >> python2-SNAPSHOT >> python2-2.10.1 >> go-SNAPSHOT >> go-2.10.1 >> >> >> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay <al...@google.com> wrote: >> >>> For snapshots, we could use gcr.io. Permission would not be a problem >>> since Jenkins is already correctly setup. The cost will be covered under >>> apache-beam-testing project. And since this is only for snapshots, it will >>> be only for temporary artifacts not for release artifacts. >>> >>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev <valen...@google.com> >>> wrote: >>> >>>> +1, releasing containers is a useful process that we need to build in >>>> Beam and it is required for FnApi users. Among other reasons, having >>>> officially-released Beam SDK harness container images will make it easier >>>> for users to do simple customizations to container images, as they will be >>>> able to use container image released by Beam as a base image. >>>> >>>> Good point about potential storage limitations on Bintray. With Beam >>>> Release cadence we may quickly exceed the 10 GB quota. It may also affect >>>> our decisions as to which images we want to release, for example: do we >>>> want to only release one container image with Python 3 interpreter, or do >>>> we want to release a container image for each Python 3 minor version that >>>> Beam is compatible with. >>>> >>> >>> Probably worth a separate discussion. I would favor first releasing a >>> python 3 compatible version before figuring out how we would target >>> multiple python 3 versions. >>> >> >>> >>>> >>>> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka <goe...@google.com> wrote: >>>> >>>>> >>>>> >>>>> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay <al...@google.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka <goe...@google.com> >>>>>> wrote: >>>>>> >>>>>>> - Could we start from snapshots first and then do it for releases? >>>>>>> +1, releasing snapsots first makes sense to me. >>>>>>> - For snapshots, do we need to clean old containers after a while? >>>>>>> Otherwise I guess we will accumulate lots of containers. >>>>>>> For snap shots we can maintain a single snapshot image from git HEAD >>>>>>> daily. Docker has the internal image container id which changes >>>>>>> everytime >>>>>>> an image is changed and pulls new images as needed. >>>>>>> >>>>>> >>>>>> There is a potential use this may not work with. If a user picks up a >>>>>> snaphsot build and want to use it until the next release arrives. I guess >>>>>> in that case the user can copy the snapshotted container image and rely >>>>>> on >>>>>> that. >>>>>> >>>>>> >>>>> Yes, that should be reasonable. >>>>> >>>>>> - Do we also need additional code changes for snapshots and releases >>>>>>> to default to these specific containers? There could be a version based >>>>>>> mechanism to resolve the correct container to use. >>>>>>> The current image defaults have username in it. We should be ok by >>>>>>> just updating the default image url to published image url. >>>>>>> >>>>>>> We should also check for pricing and details about Apache-Bintray >>>>>>> agreement before pushing images and changing defaults. >>>>>>> >>>>>> >>>>>> There is information on bintray's pricing page about open source >>>>>> projects [1]. I do not know if there is a special apache-bintray >>>>>> agreement >>>>>> or not. If there is no special agreement there is a 10GB storage limit >>>>>> for >>>>>> using bintray. >>>>>> >>>>> As each image can easily run into Gigs, 10GB might not be sufficient >>>>> for future proofing. >>>>> We can also register docker image to docker image registry and not >>>>> have bintray in the name to later host images on a different vendor for >>>>> future proofing. >>>>> >>>>> >>>>>> [1] https://bintray.com/account/pricing?tab=account&type=pricing >>>>>> >>>>>> >>>>>>> >>>>>>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay <al...@google.com> >>>>>>> wrote: >>>>>>> >>>>>>>> This sounds like a good idea. Some questions: >>>>>>>> >>>>>>>> - Could we start from snapshots first and then do it for releases? >>>>>>>> - For snapshots, do we need to clean old containers after a while? >>>>>>>> Otherwise I guess we will accumulate lots of containers. >>>>>>>> - Do we also need additional code changes for snapshots and >>>>>>>> releases to default to these specific containers? There could be a >>>>>>>> version >>>>>>>> based mechanism to resolve the correct container to use. >>>>>>>> >>>>>>>> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka <goe...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> As portability/FnApi is taking shape and are compatible with ULR >>>>>>>>> and Flink. I wanted to discuss the release plan release of SDKHarness >>>>>>>>> Docker images. Of-course users can create their own images but it >>>>>>>>> will be >>>>>>>>> useful to have a default image available out of box. >>>>>>>>> Pre build image are a must for making FnApi available for users >>>>>>>>> and not just the developers. >>>>>>>>> The other purpose of these images is to be server as base image >>>>>>>>> layer for building custom images. >>>>>>>>> >>>>>>>>> Apache already have bintray repositories for beam. >>>>>>>>> https://bintray.com/apache/beam-snapshots-docker >>>>>>>>> https://bintray.com/apache/beam-docker >>>>>>>>> >>>>>>>>> Shall we start pushing Python/Java/Go SDK Harness containers to >>>>>>>>> https://bintray.com/apache/beam-docker for beam release and >>>>>>>>> maintain daily snapshot at >>>>>>>>> https://bintray.com/apache/beam-snapshots-docker ? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ankur >>>>>>>>> >>>>>>>> > > -- > ================ > Ruoyun Huang > >
