Re: AfterWatermarkEarlyAndLate

Thanks for parsing the formatting on my original mail.  :) What else is required for that change that I didn't outline?  I think there's a state machine class in runners core that would need to be deleted?  Is there anything else like a registry of triggers that might need to be edited? Do you

Re: Let's make Beam transforms comply with PTransform Style Guide

No problem ;) Happy to review if needed ;) Regards JB On 04/21/2017 07:50 AM, Eugene Kirpichov wrote: Guys, apologies, but I already have Kinesis in review, and Pubsub ready for review. I'm afraid there's not much left for volunteers to take on right now. On Thu, Apr 20, 2017 at 10:47 PM

Re: Let's make Beam transforms comply with PTransform Style Guide

No worries. Regards, Tarush On Fri, 21 Apr 2017 at 11:20 AM, Eugene Kirpichov wrote: > Guys, apologies, but I already have Kinesis in review, and Pubsub ready for > review. I'm afraid there's not much left for volunteers to take on right > now. > > On Thu, Apr 20,

Re: Let's make Beam transforms comply with PTransform Style Guide

Guys, apologies, but I already have Kinesis in review, and Pubsub ready for review. I'm afraid there's not much left for volunteers to take on right now. On Thu, Apr 20, 2017 at 10:47 PM Jean-Baptiste Onofré wrote: > Cool, I gonna take a look on PubSub later today (I would

Re: Let's make Beam transforms comply with PTransform Style Guide

Cool, I gonna take a look on PubSub later today (I would like to finish CassandraIO, HDFS refactoring and Spark 2 support first ;)). Regards JB On 04/21/2017 06:03 AM, tarush grover wrote: Hi, I can take kinesis one. Regards, Tarush On Thu, 20 Apr 2017 at 11:18 AM, Jean-Baptiste Onofré

Towards a spec for robust streaming SQL, Part 1

Hello Beam, Calcite, and Flink dev lists! Apologies for the big cross post, but I thought this might be something all three communities would find relevant. Beam is finally making progress on a SQL DSL utilizing Calcite, thanks to Mingmin Xu. As you can imagine, we need to come to some

Re: Can application specify how watermarks should be generated?

You want to use an existing source but just change the watermark tracking? You can't do this in your pipeline right now, but you could probably easily wrap a source and proxy every method except getWatermark, though I have never tried. The general feature that might address this is discussed a

Jenkins build is back to normal : beam_SeedJob #228

See

Can application specify how watermarks should be generated?

Hi, Can application developers provide classes/methods to specify how to generate watermarks from sources, and how to aggregate watermarks from multiple input PCollections? Say, emit at most 1 watermark per second, or create watermarks that are 5 seconds older than the latest tuple's timestamp?

Re: Should you always have a separate PTransform class for a new transform?

The discussion has been reignited on https://github.com/apache/beam/pull/2603 . I can see the pretty strong argument for not replicating all the features of Combine on the hypothetical helper transform like Count.Globally. I guess, then, I like Robert's option of having an interface like

AfterWatermarkEarlyAndLate

AfterWatermarkEarlyAndLate has:    public AfterWatermarkEarlyAndLate withEarlyFirings(OnceTrigger earlyTrigger)    public AfterWatermarkEarlyAndLate withLateFirings(OnceTrigger lateTrigger) FromEndOfWindow has:    public AfterWatermarkEarlyAndLate withEarlyFirings(OnceTrigger earlyFirings)