Re: KafkaIO Windowing Fn

In 0.2.0-incubating and beyond we've replaced the DirectPipelineRunner with the DirectRunner (formerly InProcessPipelineRunner), which is capable of handling Unbounded Pipelines. Is it possible for you to upgrade? On Wed, Aug 31, 2016 at 5:17 PM, Chawla,Sumit wrote: >

Re: KafkaIO Windowing Fn

@Ajioscha, My assumption is here that atleast one trigger should fire. Either the 100 elements or the 30 second since first element. (whichever happens first) @Thomas - here is the error i get: I am using 0.1.0-incubating *ava.lang.IllegalStateException: no evaluator registered for

Re: KafkaIO Windowing Fn

Sumit, I tried running the code you shared. I noticed that if MaxNumRecords is set to number N then KafkaIO doesn't return till it has read N messages. So either try setting a low value of MaxNumRecords or don't set it at all.. Another thing I observed was that while using anonymous DoFns I got

Re: KafkaIO Windowing Fn

Hi, could the reason for the second part of the trigger never firing be that there are never at least 100 elements per key. The trigger would only fire if it saw 100 elements and with only 540 elements that seems unlikely if you have more than 6 keys. Cheers, Aljoscha On Wed, 31 Aug 2016 at

Re: KafkaIO Windowing Fn

KafkaIO is implemented using the UnboundedRead API, which is supported by the DirectRunner. You should be able to run without the withMaxNumRecords; if you can't, I'd be very interested to see the stack trace that you get when you try to run the Pipeline. On Tue, Aug 30, 2016 at 11:24 PM,

Re: Add DistributedLog IO

Hi Khurrum, I already replied in the Jira this morning. To write the IO, the first question is bounded or unbounded and which features you want to provide. An IO could be a wrapper to a simple DoFn. If you want provide advanced features like: - watermark/skew management for unbounded source

Re: KafkaIO Windowing Fn

Yes. I added it only for DirectRunner as it cannot translate Read(UnboundedSourceOfKafka) Regards Sumit Chawla On Tue, Aug 30, 2016 at 11:18 PM, Aljoscha Krettek wrote: > Ah ok, this might be a stupid question but did you remove this line when > running it with Flink: >

Re: KafkaIO Windowing Fn

Ah ok, this might be a stupid question but did you remove this line when running it with Flink: .withMaxNumRecords(500) Cheers, Aljoscha On Wed, 31 Aug 2016 at 08:14 Chawla,Sumit wrote: > Hi Aljoscha > > The code is not different while running on Flink. It have