+1
On 27.05.19 14:04, Robert Bradshaw wrote:
Sounds like everyone's onboard with the plan. Any chance we could
publish these for the upcoming 2.13 release?
On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy <lgaj...@apache.org> wrote:
+1 to have a registry for images accessible to anyone. For snapshot images, I
agree that gcr + apache-beam-testing project seems a good and easy way to start
with.
Łukasz
wt., 22 sty 2019 o 19:43 Mark Liu <mark...@google.com> napisał(a):
+1 to have an official Beam released container image.
Also I would propose to add a verification step to (or after) the release
process to do smoke check. Python have ValidatesContainer test that runs basic
pipeline using newly built container for verification. Other sdk languages can
do similar thing or add a common framework.
Mark
On Thu, Jan 17, 2019 at 5:56 AM Alan Myrvold <amyrv...@google.com> wrote:
+1 This would be great. gcr.io seems like a good option for snapshots due to
the permissions from jenkins to upload and ability to keep snapshots around.
On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang <ruo...@google.com> wrote:
+1 This would be a great thing to have.
On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka <goe...@google.com> wrote:
grc.io seems to be a good option. Given that we don't need the hosting server
name in the image name makes it easily changeable later.
Docker container for Apache Flink is named "flink" and they have different tags for
different releases and configurations https://hub.docker.com/_/flink .We can follow a similar model
and can name the image as "beam" (beam doesn't seem to be taken on docker hub) and use
tags to distinguish Java/Python/Go and versions etc.
Tags will look like:
java-SNAPSHOT
java-2.10.1
python2-SNAPSHOT
python2-2.10.1
go-SNAPSHOT
go-2.10.1
On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay <al...@google.com> wrote:
For snapshots, we could use gcr.io. Permission would not be a problem since
Jenkins is already correctly setup. The cost will be covered under
apache-beam-testing project. And since this is only for snapshots, it will be
only for temporary artifacts not for release artifacts.
On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev <valen...@google.com> wrote:
+1, releasing containers is a useful process that we need to build in Beam and
it is required for FnApi users. Among other reasons, having officially-released
Beam SDK harness container images will make it easier for users to do simple
customizations to container images, as they will be able to use container
image released by Beam as a base image.
Good point about potential storage limitations on Bintray. With Beam Release
cadence we may quickly exceed the 10 GB quota. It may also affect our decisions
as to which images we want to release, for example: do we want to only release
one container image with Python 3 interpreter, or do we want to release a
container image for each Python 3 minor version that Beam is compatible with.
Probably worth a separate discussion. I would favor first releasing a python 3
compatible version before figuring out how we would target multiple python 3
versions.
On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka <goe...@google.com> wrote:
On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay <al...@google.com> wrote:
On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka <goe...@google.com> wrote:
- Could we start from snapshots first and then do it for releases?
+1, releasing snapsots first makes sense to me.
- For snapshots, do we need to clean old containers after a while? Otherwise I
guess we will accumulate lots of containers.
For snap shots we can maintain a single snapshot image from git HEAD daily.
Docker has the internal image container id which changes everytime an image is
changed and pulls new images as needed.
There is a potential use this may not work with. If a user picks up a snaphsot
build and want to use it until the next release arrives. I guess in that case
the user can copy the snapshotted container image and rely on that.
Yes, that should be reasonable.
- Do we also need additional code changes for snapshots and releases to default
to these specific containers? There could be a version based mechanism to
resolve the correct container to use.
The current image defaults have username in it. We should be ok by just
updating the default image url to published image url.
We should also check for pricing and details about Apache-Bintray agreement
before pushing images and changing defaults.
There is information on bintray's pricing page about open source projects [1].
I do not know if there is a special apache-bintray agreement or not. If there
is no special agreement there is a 10GB storage limit for using bintray.
As each image can easily run into Gigs, 10GB might not be sufficient for future
proofing.
We can also register docker image to docker image registry and not have bintray
in the name to later host images on a different vendor for future proofing.
[1] https://bintray.com/account/pricing?tab=account&type=pricing
On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay <al...@google.com> wrote:
This sounds like a good idea. Some questions:
- Could we start from snapshots first and then do it for releases?
- For snapshots, do we need to clean old containers after a while? Otherwise I
guess we will accumulate lots of containers.
- Do we also need additional code changes for snapshots and releases to default
to these specific containers? There could be a version based mechanism to
resolve the correct container to use.
On Wed, Jan 16, 2019 at 4:42 PM Ankur Goenka <goe...@google.com> wrote:
Hi All,
As portability/FnApi is taking shape and are compatible with ULR and Flink. I
wanted to discuss the release plan release of SDKHarness Docker images.
Of-course users can create their own images but it will be useful to have a
default image available out of box.
Pre build image are a must for making FnApi available for users and not just
the developers.
The other purpose of these images is to be server as base image layer for
building custom images.
Apache already have bintray repositories for beam.
https://bintray.com/apache/beam-snapshots-docker
https://bintray.com/apache/beam-docker
Shall we start pushing Python/Java/Go SDK Harness containers to
https://bintray.com/apache/beam-docker for beam release and maintain daily
snapshot at https://bintray.com/apache/beam-snapshots-docker ?
Thanks,
Ankur
--
================
Ruoyun Huang
