+1
On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía <ieme...@gmail.com> wrote: > +1 > > On Mon, May 27, 2019 at 3:35 PM Maximilian Michels <m...@apache.org> wrote: > > > > +1 > > > > On 27.05.19 14:04, Robert Bradshaw wrote: > > > Sounds like everyone's onboard with the plan. Any chance we could > > > publish these for the upcoming 2.13 release? > > > > > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy <lgaj...@apache.org> > wrote: > > >> > > >> +1 to have a registry for images accessible to anyone. For snapshot > images, I agree that gcr + apache-beam-testing project seems a good and > easy way to start with. > > >> > > >> Łukasz > > >> > > >> wt., 22 sty 2019 o 19:43 Mark Liu <mark...@google.com> napisał(a): > > >>> > > >>> +1 to have an official Beam released container image. > > >>> > > >>> Also I would propose to add a verification step to (or after) the > release process to do smoke check. Python have ValidatesContainer test that > runs basic pipeline using newly built container for verification. Other sdk > languages can do similar thing or add a common framework. > > >>> > > >>> Mark > > >>> > > >>> On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold <amyrv...@google.com> > wrote: > > >>>> > > >>>> +1 This would be great. gcr.io seems like a good option for > snapshots due to the permissions from jenkins to upload and ability to keep > snapshots around. > > >>>> > > >>>> On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang <ruo...@google.com> > wrote: > > >>>>> > > >>>>> +1 This would be a great thing to have. > > >>>>> > > >>>>> On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka <goe...@google.com> > wrote: > > >>>>>> > > >>>>>> grc.io seems to be a good option. Given that we don't need the > hosting server name in the image name makes it easily changeable later. > > >>>>>> > > >>>>>> Docker container for Apache Flink is named "flink" and they have > different tags for different releases and configurations > https://hub.docker.com/_/flink .We can follow a similar model and can > name the image as "beam" (beam doesn't seem to be taken on docker hub) and > use tags to distinguish Java/Python/Go and versions etc. > > >>>>>> > > >>>>>> Tags will look like: > > >>>>>> java-SNAPSHOT > > >>>>>> java-2.10.1 > > >>>>>> python2-SNAPSHOT > > >>>>>> python2-2.10.1 > > >>>>>> go-SNAPSHOT > > >>>>>> go-2.10.1 > > >>>>>> > > >>>>>> > > >>>>>> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay <al...@google.com> > wrote: > > >>>>>>> > > >>>>>>> For snapshots, we could use gcr.io. Permission would not be a > problem since Jenkins is already correctly setup. The cost will be covered > under apache-beam-testing project. And since this is only for snapshots, it > will be only for temporary artifacts not for release artifacts. > > >>>>>>> > > >>>>>>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev < > valen...@google.com> wrote: > > >>>>>>>> > > >>>>>>>> +1, releasing containers is a useful process that we need to > build in Beam and it is required for FnApi users. Among other reasons, > having officially-released Beam SDK harness container images will make it > easier for users to do simple customizations to container images, as they > will be able to use container image released by Beam as a base image. > > >>>>>>>> > > >>>>>>>> Good point about potential storage limitations on Bintray. With > Beam Release cadence we may quickly exceed the 10 GB quota. It may also > affect our decisions as to which images we want to release, for example: do > we want to only release one container image with Python 3 interpreter, or > do we want to release a container image for each Python 3 minor version > that Beam is compatible with. > > >>>>>>> > > >>>>>>> > > >>>>>>> Probably worth a separate discussion. I would favor first > releasing a python 3 compatible version before figuring out how we would > target multiple python 3 versions. > > >>>>>>> > > >>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka <goe...@google.com> > wrote: > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay <al...@google.com> > wrote: > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka < > goe...@google.com> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>> - Could we start from snapshots first and then do it for > releases? > > >>>>>>>>>>> +1, releasing snapsots first makes sense to me. > > >>>>>>>>>>> - For snapshots, do we need to clean old containers after a > while? Otherwise I guess we will accumulate lots of containers. > > >>>>>>>>>>> For snap shots we can maintain a single snapshot image from > git HEAD daily. Docker has the internal image container id which changes > everytime an image is changed and pulls new images as needed. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> There is a potential use this may not work with. If a user > picks up a snaphsot build and want to use it until the next release > arrives. I guess in that case the user can copy the snapshotted container > image and rely on that. > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Yes, that should be reasonable. > > >>>>>>>>>>> > > >>>>>>>>>>> - Do we also need additional code changes for snapshots and > releases to default to these specific containers? There could be a version > based mechanism to resolve the correct container to use. > > >>>>>>>>>>> The current image defaults have username in it. We should be > ok by just updating the default image url to published image url. > > >>>>>>>>>>> > > >>>>>>>>>>> We should also check for pricing and details about > Apache-Bintray agreement before pushing images and changing defaults. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> There is information on bintray's pricing page about open > source projects [1]. I do not know if there is a special apache-bintray > agreement or not. If there is no special agreement there is a 10GB storage > limit for using bintray. > > >>>>>>>>> > > >>>>>>>>> As each image can easily run into Gigs, 10GB might not be > sufficient for future proofing. > > >>>>>>>>> We can also register docker image to docker image registry and > not have bintray in the name to later host images on a different vendor for > future proofing. > > >>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> [1] > https://bintray.com/account/pricing?tab=account&type=pricing > > >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay < > al...@google.com> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>> This sounds like a good idea. Some questions: > > >>>>>>>>>>>> > > >>>>>>>>>>>> - Could we start from snapshots first and then do it for > releases? > > >>>>>>>>>>>> - For snapshots, do we need to clean old containers after a > while? Otherwise I guess we will accumulate lots of containers. > > >>>>>>>>>>>> - Do we also need additional code changes for snapshots and > releases to default to these specific containers? There could be a version > based mechanism to resolve the correct container to use. > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka < > goe...@google.com> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Hi All, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> As portability/FnApi is taking shape and are compatible > with ULR and Flink. I wanted to discuss the release plan release of > SDKHarness Docker images. Of-course users can create their own images but > it will be useful to have a default image available out of box. > > >>>>>>>>>>>>> Pre build image are a must for making FnApi available for > users and not just the developers. > > >>>>>>>>>>>>> The other purpose of these images is to be server as base > image layer for building custom images. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Apache already have bintray repositories for beam. > > >>>>>>>>>>>>> https://bintray.com/apache/beam-snapshots-docker > > >>>>>>>>>>>>> https://bintray.com/apache/beam-docker > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Shall we start pushing Python/Java/Go SDK Harness > containers to https://bintray.com/apache/beam-docker for beam release and > maintain daily snapshot at > https://bintray.com/apache/beam-snapshots-docker ? > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>> Ankur > > >>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> ================ > > >>>>> Ruoyun Huang > > >>>>> >
