Re: RFC: Assigning environments to transforms in a pipeline

Hi Robert, > Sounds nice. Is there a design doc (or, perhaps, you could just give an > example of what this would look like in this thread)? > I'll follow up shortly with something. The good news is that this first PR is quite straightforward and (I think) is independent of the semantics of how

Re: RFC: Assigning environments to transforms in a pipeline

Sounds nice. Is there a design doc (or, perhaps, you could just give an example of what this would look like in this thread)? On Wed, Oct 16, 2019 at 5:51 PM Chad Dombrova wrote: > Hi all, > One of our goals for the portability framework is to be able to assign > different environments to

RFC: Assigning environments to transforms in a pipeline

Hi all, One of our goals for the portability framework is to be able to assign different environments to different segments of a pipeline. This is not possible right now because environments are a concept that really only exist in the portable runner as protobuf messages: they lack a proper API

Re: beam_PostCommit_Java_Nexmark_Flink are failing continuously

I checked the logs of failure jobs. It might be related to the issue when vendoring calcite. Investigating on this issue. Best, Kai On Wed, Oct 16, 2019 at 11:06 AM Boyuan Zhang wrote: > Hey team, > > beam_PostCommit_Java_Nexmark_Flink >

Re: Feature addition to java CassandraIO connector

Hi Vincent, I think it makes sense to have some sort of `readAll` for CassandraIO that can receive multiple queries, and execute each one of them. This would also be consistent with other IOs that we have such as FileIOs. I suspect that doing this may require rearchitecting the whole IO from a

beam_PostCommit_Java_Nexmark_Flink are failing continuously

Hey team, beam_PostCommit_Java_Nexmark_Flink test target has failed for a long time. Is this something expected? If not, any insights on these failures? Boyuan

Re: MQTT to Python SDK

Till we have unbounded SDF support finalized for Python SDK (and supported by Beam runners) we don't have the framework to add an unbounded source to Python SDK. So if you want to add a Python version today it will be a runner native source (which is how Python PubSub source is implemented today,

Re: Python2 postcommit broken

Filed https://issues.apache.org/jira/browse/BEAM-8412 for this. On Wed, Oct 16, 2019 at 7:11 AM Kamil Wasilewski < kamil.wasilew...@polidea.com> wrote: > Hello all, > > I've noticed that since last two days all Python2 post commit tests have > failed[1]. Logs show it's because of an exception in

Re: [PROPOSAL] Preparing for Beam 2.17.0 release

+1. Thank you. On Tue, Oct 15, 2019 at 8:35 PM Jean-Baptiste Onofré wrote: > It sounds good to me. > > +1 > > Regards > JB > > On 15/10/2019 20:37, Mikhail Gryzykhin wrote: > > Hi all, > > > > Beam 2.17 release branch cut is scheduled on Oct 23 according to > > the release calendar [1]. I would

Re: MQTT to Python SDK

Hi Cham, How much work would it be to add a MQTT unbounded source directly to the Python SDK? I will look into the cross-language solution, but I'm wondering if contributing directly to the Python SDK would be a lot more work or not (keeping in mind that I have never contributed before, so there

Re: Gauge Metrics

Thank you for your answers! Fair points. Would you elaborate on what you are expecting the behaviour to look like? Ideally your runner would export gauges at a periodic interval. I'd expect all gauge values reported by the SDK to also be reported by the Runner. Instead of reducing the gauge