Re: KafkaIO Windowing Fn

2016-08-30 Thread Chawla,Sumit
Yes. I added it only for DirectRunner as it cannot translate Read(UnboundedSourceOfKafka) Regards Sumit Chawla On Tue, Aug 30, 2016 at 11:18 PM, Aljoscha Krettek wrote: > Ah ok, this might be a stupid question but did you remove this line when > running it with Flink: > .withMaxNumRec

Re: Anyone @scale tomorrow?

2016-08-30 Thread Jean-Baptiste Onofré
Hi Dan, unfortunately, I can't be there. Good luck for your talk ! And thanks to spread the Beam word ;) Regards JB On 08/31/2016 04:32 AM, Dan Halperin wrote: I'll be giving a talk at the Facebook @scale conference tomorrow. Sorry for the late notice, but if anyone is around to meet in the

Re: KafkaIO Windowing Fn

2016-08-30 Thread Aljoscha Krettek
Ah ok, this might be a stupid question but did you remove this line when running it with Flink: .withMaxNumRecords(500) Cheers, Aljoscha On Wed, 31 Aug 2016 at 08:14 Chawla,Sumit wrote: > Hi Aljoscha > > The code is not different while running on Flink. It have removed business > speci

Re: KafkaIO Windowing Fn

2016-08-30 Thread Chawla,Sumit
Hi Aljoscha The code is not different while running on Flink. It have removed business specific transformations only. Regards Sumit Chawla On Tue, Aug 30, 2016 at 4:49 AM, Aljoscha Krettek wrote: > Hi, > could you maybe also post the complete that you're using with the > FlinkRunner? I could

Anyone @scale tomorrow?

2016-08-30 Thread Dan Halperin
I'll be giving a talk at the Facebook @scale conference tomorrow. Sorry for the late notice, but if anyone is around to meet in the hallway track or have lunch or drinks, reach out. I'd love to connect. Dan

Re: KafkaIO Windowing Fn

2016-08-30 Thread Aljoscha Krettek
Hi, could you maybe also post the complete that you're using with the FlinkRunner? I could have a look into it. Cheers, Aljoscha On Tue, 30 Aug 2016 at 09:01 Chawla,Sumit wrote: > Hi Thomas > > Sorry i tried with DirectRunner but ran into some kafka issues. Following > is the snippet i am work

Re: [PROPOSAL] Splittable DoFn - Replacing the Source API with non-monolithic element processing in DoFn

2016-08-30 Thread Aljoscha Krettek
Thanks for the explanation Eugene and JB. By the way, I'm not trying to find holes in this, I really like the feature. I just sometimes wonder how a specific thing might be implemented with this. On Mon, 29 Aug 2016 at 18:00 Eugene Kirpichov wrote: > Hi Aljoscha, > > The watermark reporting is

Re: KafkaIO Windowing Fn

2016-08-30 Thread Chawla,Sumit
Hi Thomas Sorry i tried with DirectRunner but ran into some kafka issues. Following is the snippet i am working on, and will post more details once i get it working ( as of now i am unable to read messages from Kafka using DirectRunner) PipelineOptions pipelineOptions = PipelineOptionsFactory.c