For snapshots, we could use gcr.io. Permission would not be a problem since Jenkins is already correctly setup. The cost will be covered under apache-beam-testing project. And since this is only for snapshots, it will be only for temporary artifacts not for release artifacts.
On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev <valen...@google.com> wrote: > +1, releasing containers is a useful process that we need to build in Beam > and it is required for FnApi users. Among other reasons, having > officially-released Beam SDK harness container images will make it easier > for users to do simple customizations to container images, as they will be > able to use container image released by Beam as a base image. > > Good point about potential storage limitations on Bintray. With Beam > Release cadence we may quickly exceed the 10 GB quota. It may also affect > our decisions as to which images we want to release, for example: do we > want to only release one container image with Python 3 interpreter, or do > we want to release a container image for each Python 3 minor version that > Beam is compatible with. > Probably worth a separate discussion. I would favor first releasing a python 3 compatible version before figuring out how we would target multiple python 3 versions. > > On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka <goe...@google.com> wrote: > >> >> >> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay <al...@google.com> wrote: >> >>> >>> >>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka <goe...@google.com> wrote: >>> >>>> - Could we start from snapshots first and then do it for releases? >>>> +1, releasing snapsots first makes sense to me. >>>> - For snapshots, do we need to clean old containers after a while? >>>> Otherwise I guess we will accumulate lots of containers. >>>> For snap shots we can maintain a single snapshot image from git HEAD >>>> daily. Docker has the internal image container id which changes everytime >>>> an image is changed and pulls new images as needed. >>>> >>> >>> There is a potential use this may not work with. If a user picks up a >>> snaphsot build and want to use it until the next release arrives. I guess >>> in that case the user can copy the snapshotted container image and rely on >>> that. >>> >>> >> Yes, that should be reasonable. >> >>> - Do we also need additional code changes for snapshots and releases to >>>> default to these specific containers? There could be a version based >>>> mechanism to resolve the correct container to use. >>>> The current image defaults have username in it. We should be ok by just >>>> updating the default image url to published image url. >>>> >>>> We should also check for pricing and details about Apache-Bintray >>>> agreement before pushing images and changing defaults. >>>> >>> >>> There is information on bintray's pricing page about open source >>> projects [1]. I do not know if there is a special apache-bintray agreement >>> or not. If there is no special agreement there is a 10GB storage limit for >>> using bintray. >>> >> As each image can easily run into Gigs, 10GB might not be sufficient for >> future proofing. >> We can also register docker image to docker image registry and not have >> bintray in the name to later host images on a different vendor for future >> proofing. >> >> >>> [1] https://bintray.com/account/pricing?tab=account&type=pricing >>> >>> >>>> >>>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay <al...@google.com> wrote: >>>> >>>>> This sounds like a good idea. Some questions: >>>>> >>>>> - Could we start from snapshots first and then do it for releases? >>>>> - For snapshots, do we need to clean old containers after a while? >>>>> Otherwise I guess we will accumulate lots of containers. >>>>> - Do we also need additional code changes for snapshots and releases >>>>> to default to these specific containers? There could be a version based >>>>> mechanism to resolve the correct container to use. >>>>> >>>>> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka <goe...@google.com> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> As portability/FnApi is taking shape and are compatible with ULR and >>>>>> Flink. I wanted to discuss the release plan release of SDKHarness Docker >>>>>> images. Of-course users can create their own images but it will be useful >>>>>> to have a default image available out of box. >>>>>> Pre build image are a must for making FnApi available for users and >>>>>> not just the developers. >>>>>> The other purpose of these images is to be server as base image layer >>>>>> for building custom images. >>>>>> >>>>>> Apache already have bintray repositories for beam. >>>>>> https://bintray.com/apache/beam-snapshots-docker >>>>>> https://bintray.com/apache/beam-docker >>>>>> >>>>>> Shall we start pushing Python/Java/Go SDK Harness containers to >>>>>> https://bintray.com/apache/beam-docker for beam release and maintain >>>>>> daily snapshot at https://bintray.com/apache/beam-snapshots-docker ? >>>>>> >>>>>> Thanks, >>>>>> Ankur >>>>>> >>>>>
