Re: Hazelcast Jet Runner

"Many APIs that have been in place for years and are used by most Beam users are still marked Experimental." Should there be a formal process in place to start 'graduating' features out of @Experimental? Perhaps even target an up coming release with a PR to remove the annotation from well establis

Re: Hazelcast Jet Runner

We generally use Experimental for two different things, which leads to confusion. 1. Features that work stably, but where we think we might still make some changes to the API. 2. New features that we think might not yet be stable. This dual usage leads to a lot of confusion IMO. The fact that

Re: Hazelcast Jet Runner

> Personally, I think that it is good that moving from experimental to > non-experimental is a breaking change in the dependency - one has > backwards-incompatible changes and the other does not. If artifacts had > separate versioning we could use 0.x for this. In theory it seems so, but in pra

Re: [DISCUSS] Autoformat python code with Black

I'm generally in favor of autoformatters, though I haven't looked at how well this particular one works. We might have to go with https://github.com/desbma/black-2spaces given https://github.com/python/black/issues/378 . On Mon, May 27, 2019 at 10:43 PM Pablo Estrada wrote: > > This looks pretty

Re: [DISCUSS] Autoformat python code with Black

This looks pretty good:) I know at least a couple people (myself included) who've been annoyed by having to take care of lint issues that maybe a code formatter could save us. Thanks for sharing Ismael. -P. On Mon, May 27, 2019, 12:24 PM Ismaël Mejía wrote: > I stumbled by chance into Black [1]

Re: RedisIO refactoring

> I’m not very familiar with Jedis, but, at the first glance, it seems that we can use List BinaryJedis.mget(final byte[]... keys) [1] in this case. Also, probably, it can be achieved by using "Response PipelineBase.get(byte[])” (we already use Pipeline for writing). In any case, Redis uses C char

[DISCUSS] Autoformat python code with Black

I stumbled by chance into Black [1] a python code auto formatter that is becoming the 'de-facto' auto-formatter for python, and wanted to bring to the ML Is there interest from the python people to get this into the build? The introduction of spotless for Java has been a good improvement and maybe

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

+1 We can release the images with 2.13 but we should not block 2.13 release for this. On Mon, May 27, 2019, 8:39 AM Thomas Weise wrote: > +1 > > > On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía wrote: > >> +1 >> >> On Mon, May 27, 2019 at 3:35 PM Maximilian Michels >> wrote: >> > >> > +1 >> > >>

Re: Hazelcast Jet Runner

(minor related tangent for additional perspective) +1 from the perspective of SDKs on moving from experimental to production versions being a breaking change. I've long posited that the Go SDK, as it's currently experimental, is v0.X, and some breaking changes have been made accordingly. Once the

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

+1 On Mon, May 27, 2019 at 6:56 AM Ismaël Mejía wrote: > +1 > > On Mon, May 27, 2019 at 3:35 PM Maximilian Michels wrote: > > > > +1 > > > > On 27.05.19 14:04, Robert Bradshaw wrote: > > > Sounds like everyone's onboard with the plan. Any chance we could > > > publish these for the upcoming 2.

Re: Hazelcast Jet Runner

I also favor explicit opt-in, especially when you're mixing mature and new components. A differently-named, but still published, artifact seems preferable IMHO to long lived branches. I don't have a handle on how problematic this would be in practice (e.g. how would a user know to update the name.

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

+1 On Mon, May 27, 2019 at 3:35 PM Maximilian Michels wrote: > > +1 > > On 27.05.19 14:04, Robert Bradshaw wrote: > > Sounds like everyone's onboard with the plan. Any chance we could > > publish these for the upcoming 2.13 release? > > > > On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy wrote: > >

Re: Environments for External Transforms

On Mon, May 27, 2019 at 1:59 PM Maximilian Michels wrote: > > > I think it makes a lot of sense for job servers to also act as > > expansion services, but one can't of course defer expansion until job > > submission. > > One could defer the expansion until job submission but it would be a > semant

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

+1 On 27.05.19 14:04, Robert Bradshaw wrote: Sounds like everyone's onboard with the plan. Any chance we could publish these for the upcoming 2.13 release? On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy wrote: +1 to have a registry for images accessible to anyone. For snapshot images, I agree

Beam Dependency Check Report (2019-05-27)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue google-cloud-bigquery 1.6.1 1.12.1

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

Sounds like everyone's onboard with the plan. Any chance we could publish these for the upcoming 2.13 release? On Wed, Feb 6, 2019 at 6:29 PM Łukasz Gajowy wrote: > > +1 to have a registry for images accessible to anyone. For snapshot images, I > agree that gcr + apache-beam-testing project seem

Re: Environments for External Transforms

I think it makes a lot of sense for job servers to also act as expansion services, but one can't of course defer expansion until job submission. One could defer the expansion until job submission but it would be a semantic change to how expansion works currently. In particular with respect to

Re: Environments for External Transforms

On Mon, May 27, 2019 at 12:38 PM Maximilian Michels wrote: > > > Which environment would be used to perform the expansion? I think this is > > an interesting option, as long as it does not introduce a hard dependency > > on docker. > > The same environment that the to-be-expanded transform requi

Re: Beam Summit Europe: speakers and schedule online!

Thanks Matthias! This is going to be a very special event for all attendees. -Max On 24.05.19 20:12, Joana Filipa Bernardo Carrasqueira wrote: Great! Thanks for spotting that. We'll update it in our page! On Fri, May 24, 2019 at 4:56 AM Aljoscha Krettek > wrote:

Re: Environments for External Transforms

Which environment would be used to perform the expansion? I think this is an interesting option, as long as it does not introduce a hard dependency on docker. The same environment that the to-be-expanded transform requires during runtime. Dataflow has been doing something similar in this r

Re: @RequireTimeSortedInput design draft

Hi, I have updated the proposal with gathered feedback, and I tried to resolve most of the issues that were mentioned. I'd be grateful for another round of comments. Thanks,  Jan [1] https://docs.google.com/document/d/1ObLVUFsf1NcG8ZuIZE4aVy2RYKx2FfyMhkZYWPnI9-c/ On 5/23/19 4:40 PM, Robe