Re: [ANNOUNCE] Beam 2.17.0 Released!

Thank you Mikhail! Yichi Zhang 于2020年1月11日 周六09:09写道： > Thank you Mikahil! > > On Fri, Jan 10, 2020 at 12:52 PM Ahmet Altay wrote: > >> Thank you Mikhail! >> >> On Fri, Jan 10, 2020 at 12:40 PM Kyle Weaver wrote: >> >>> Hooray! Thanks to Mikhail and everyone else who contributed. >>> >>> On

Re: [VOTE] Vendored Dependencies Release

+1 (non-binding) On Thu, Jan 9, 2020 at 8:48 PM jincheng sun wrote: > +1，checked list as follows： > - verified the hash and signature > - verified that there is no linkage errors > - verified that the content of the pom is expected: the shaded > dependencies are not exposed, the scope of the

Re: Go SplittableDoFn prototype and proposed changes

Thank you for sharing Daniel! Resolving SplittableDoFns for the Go SDK even just as far as initial splitting will take the SDK that much closer to exiting its experimental status. It's especially exciting seeing this work on Flink and on the Python direct runner! On Fri, Jan 10, 2020, 5:36 PM

Go SplittableDoFn prototype and proposed changes

Hey Beam devs, So several months ago I posted my Go SDF proposal and got a lot of good feedback (thread , doc ). Since then I've been

Re: [RELEASE] Tracking 2.18

RC1 is almost ready, but Nexus login is down due to LDAP issues with Apache. On Mon, Dec 16, 2019 at 9:53 AM Udi Meiri wrote: > The remaining 4 open blockers all have recently merged cherrypicks (at > least 1 blocker is waiting on verification since it's a release process > issue). > > Will

Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

On Fri, Jan 10, 2020 at 3:33 PM Ahmet Altay wrote: > > > On Fri, Jan 10, 2020 at 3:32 PM Ankur Goenka wrote: > >> Also curious to know if apache provide any infra support fro projects >> under Apache umbrella and any quota limits they might have. >> > Maybe Hannah can ask with an infra ticket?

Re: [ANNOUNCE] Beam 2.17.0 Released!

Thank you Mikahil! On Fri, Jan 10, 2020 at 12:52 PM Ahmet Altay wrote: > Thank you Mikhail! > > On Fri, Jan 10, 2020 at 12:40 PM Kyle Weaver wrote: > >> Hooray! Thanks to Mikhail and everyone else who contributed. >> >> On Fri, Jan 10, 2020 at 10:23 AM Maximilian Michels >> wrote: >> >>> At

Re: Cleaning up SDK docker image tagging

On Fri, Jan 10, 2020 at 3:30 PM Kyle Weaver wrote: > > > Does cloning a release, modifying the docker file, and building the > > containers create a "new" container with a default release tag? If so, > > we should discourage that > > Yes, and agreed. The doc you linked already mentions how to

Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

On Fri, Jan 10, 2020 at 3:32 PM Ankur Goenka wrote: > Also curious to know if apache provide any infra support fro projects > under Apache umbrella and any quota limits they might have. > > On Fri, Jan 10, 2020, 2:26 PM Robert Bradshaw wrote: > >> One downside is that, unlike many of these

Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

Also curious to know if apache provide any infra support fro projects under Apache umbrella and any quota limits they might have. On Fri, Jan 10, 2020, 2:26 PM Robert Bradshaw wrote: > One downside is that, unlike many of these projects, we release a > dozen or so containers. Is there exactly

Re: Cleaning up SDK docker image tagging

> Does cloning a release, modifying the docker file, and building the > containers create a "new" container with a default release tag? If so, > we should discourage that Yes, and agreed. The doc you linked already mentions how to customize tags, maybe we could also recommend the user always

Re: Custom window invariants and

Once again this is a great help, thank you Kenneth On Wed, Jan 8, 2020 at 3:03 PM Kenneth Knowles wrote: > Hmm. I've seen this manifest in some other tweaked versions of Sessions. > Your invariants are right. In fact, the Nexmark queries have auctions that > truncate in a similar way. This

Re: Cleaning up SDK docker image tagging

On Fri, Jan 10, 2020 at 12:48 PM Kyle Weaver wrote: > > > Shall we ALSO tag the image with git commit version for local build to keep > > track of obsolete images. > > This would mean we would have to be able to access the git commit from the > source, which might not be trivial (right now the

Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

One downside is that, unlike many of these projects, we release a dozen or so containers. Is there exactly (and only) one level of namespacing/nesting we can leverage here? (This isn't a blocker, but something to consider.) On Fri, Jan 10, 2020 at 2:06 PM Hannah Jiang wrote: > > Thanks Ahmet for

Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

Thanks Ahmet for proposing it. I will take it and work towards v2.19. Hannah On Fri, Jan 10, 2020 at 1:50 PM Kyle Weaver wrote: > It'd be nice to have the clout/official sheen of apache attached to our > containers. Although getting the required permissions might add some small > overhead to

Re: Cleaning up SDK docker image tagging

Thanks for pointing me to the thread. I agree with what discussed there, let's keep it as it is. I will proceed with cleaning up tags only. On Fri, Jan 10, 2020 at 12:48 PM Kyle Weaver wrote: > > Shall we ALSO tag the image with git commit version for local build to > keep track of obsolete

Re: Jenkins jobs not running for my PR 10438

Done. On Fri, Jan 10, 2020 at 12:59 PM Tomo Suzuki wrote: > Hi Bean developers, > > I appreciate a committer can trigger precommit build for > https://github.com/apache/beam/pull/10554. > > In addition to normal precommit checks, I want the followings: > Run Java PostCommit > Run Java

Re: [PROPOSAL] Transition released containers to the official ASF dockerhub organization

It'd be nice to have the clout/official sheen of apache attached to our containers. Although getting the required permissions might add some small overhead to the release process. For example, yesterday, when we needed to create new repositories (not just update existing ones), since we have

[PROPOSAL] Transition released containers to the official ASF dockerhub organization

Hi all, I saw recent progress on the containers and wanted to bring this question to the attention of the dev list. Would it be possible to use the official ASF dockerhub organization for new Beam container releases? Concretely, starting from 2.19 could we release Beam containers to

Re: Jenkins jobs not running for my PR 10438

Thank you, Andrew! On Fri, Jan 10, 2020 at 15:59 Tomo Suzuki wrote: > Hi Bean developers, > > I appreciate a committer can trigger precommit build for > https://github.com/apache/beam/pull/10554. > > In addition to normal precommit checks, I want the followings: > Run Java PostCommit > Run Java

Re: Jenkins jobs not running for my PR 10438

Hi Bean developers, I appreciate a committer can trigger precommit build for https://github.com/apache/beam/pull/10554. In addition to normal precommit checks, I want the followings: Run Java PostCommit Run Java HadoopFormatIO Performance Test Run BigQueryIO Streaming Performance Test Java Run

Re: [ANNOUNCE] Beam 2.17.0 Released!

Thank you Mikhail! On Fri, Jan 10, 2020 at 12:40 PM Kyle Weaver wrote: > Hooray! Thanks to Mikhail and everyone else who contributed. > > On Fri, Jan 10, 2020 at 10:23 AM Maximilian Michels > wrote: > >> At last :) Thank you for making it happen Mikhail! Also thanks to >> everyone else who

Re: Cleaning up SDK docker image tagging

> Shall we ALSO tag the image with git commit version for local build to keep track of obsolete images. This would mean we would have to be able to access the git commit from the source, which might not be trivial (right now the Beam version e.g. " 2.18.0.dev" is hard-coded in some properties

Re: Cleaning up SDK docker image tagging

Hi Hannah, +1 to standardize .dev suffixes across all SDKs. Whether to pull or not to pull was recently discussed in [1]. My personal preference would be to pull images before starting the containers, and instructing users who want to customize containers to tag them with a new tag, such as

Re: [ANNOUNCE] Beam 2.17.0 Released!

Hooray! Thanks to Mikhail and everyone else who contributed. On Fri, Jan 10, 2020 at 10:23 AM Maximilian Michels wrote: > At last :) Thank you for making it happen Mikhail! Also thanks to > everyone else who tested the release candidate. > > Cheers, > Max > > On 10.01.20 19:01, Mikhail

Re: [ANNOUNCE] Beam 2.17.0 Released!

Thanks for persistent and powering through all the issues. On Fri, Jan 10, 2020 at 10:23 AM Maximilian Michels wrote: > At last :) Thank you for making it happen Mikhail! Also thanks to > everyone else who tested the release candidate. > > Cheers, > Max > > On 10.01.20 19:01, Mikhail Gryzykhin

Re: release scripts as interactive notebooks?

I think we need to balance where we are with where we want to be. There are a couple layers of abstraction that are all independently useful. We have to acknowledge that currently the steps in the guide and scripts don't quite match and also are not quite right, and that things go wrong and

Re: [ANNOUNCE] Beam 2.17.0 Released!

At last :) Thank you for making it happen Mikhail! Also thanks to everyone else who tested the release candidate. Cheers, Max On 10.01.20 19:01, Mikhail Gryzykhin wrote: The Apache Beam team is pleased to announce the release of version 2.17.0. Apache Beam is an open source unified

Re: release scripts as interactive notebooks?

+1 to automating more, at least the creation and validation of release artifacts should all be completely automated. However signing should still be done by an individual--that's not something that (semantically) should be automated away. As much as I am a fan of jupyter notebooks, I think the

[ANNOUNCE] Beam 2.17.0 Released!

The Apache Beam team is pleased to announce the release of version 2.17.0. Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream (continuous) processing. See https://beam.apache.org You can download the release

Re: Cleaning up SDK docker image tagging

>> This has a minor downside for the users who are using unreleased versions. They need to build a local image first before using docker to run. > Isn't that the current behavior? Our current behavior is pull & run. So in case both local and remote images are available, the local image is getting

Re: Request for new dockerhub repos

Thank you the pushes were successful. On Fri, Jan 10, 2020 at 8:47 AM Hannah Jiang wrote: > Hi Udi > > The repositories are created. Were you added as a maintainer? If not, we > need your docker hub user ID. > > Thanks, > Hannah > > On Thu, Jan 9, 2020 at 5:48 PM Udi Meiri wrote: > >> Hi, >>

Re: release scripts as interactive notebooks?

I was always under the impression that artifact creation, signing and staging for voting we always wanted to be "automated" in some way. I believe we could have a jenkins job do this if we had a good way to transfer the release managers signing keys to a Jenkins worker (via cloud key management

release scripts as interactive notebooks?

What does the community think about converting our release scripts to be Jupyter notebooks using bash_kernel? Since these scripts frequently fail (especially for first time releasers), we often need to rerun parts manually. The

Re: Cleaning up SDK docker image tagging

> For the unreleased version of SDKs, the default tag will be version number + '.dev'. (ex: 2.18.0.dev) >> Shall we ALSO tag the image with git commit version for local build to keep track of obsolete images. I should clarify it more clearly. This is about release images. The dev images are only

Re: Poor Python 3.x performance on Dataflow?

Thanks, Kamil. I self-assigned the issue, but if anyone else is interested, feel free to take a look in parallel and post your findings on the Jira. On Fri, Jan 10, 2020 at 4:29 AM Kamil Wasilewski < kamil.wasilew...@polidea.com> wrote: > Our first Python3 performance test has just been

Re: Request for new dockerhub repos

Hi Udi The repositories are created. Were you added as a maintainer? If not, we need your docker hub user ID. Thanks, Hannah On Thu, Jan 9, 2020 at 5:48 PM Udi Meiri wrote: > Hi, > As part of the 2.18 release, we're adding 3 additional containers for > Flink. > I have write access but since I

Re: Poor Python 3.x performance on Dataflow?

Our first Python3 performance test has just been implemented and we have just started gathering results. Here[1] you can find dashboards with a side-by-side comparison. I also opened a Jira ticket to investigate the difference [2]. Anyone, please feel free to assign it to yourself. Thanks, Kamil