Re: Read Kafka topic from the beginning

2015-12-03 Thread Vladimir Stoyak
specific offset. The offset is only unique per partition. You could modify the offsets in the Zookeeper state but you really have to know what you're doing then. Best regards, Max On Thu, Dec 3, 2015 at 4:01 PM, Vladimir Stoyak <vsto...@yahoo.com> wrote: > I see that Flink 0.10.1 now

Read Kafka topic from the beginning

2015-12-03 Thread Vladimir Stoyak
I see that Flink 0.10.1 now supports Keyed Schemas which allows us to rely on Kafka topics set to "compact" retention for data persistence. In our topology we wanted to set some topics with Log Compactions enabled and read topic from the beginning when the topology starts or component recovers.

Re: Read Kafka topic from the beginning

2015-12-03 Thread Vladimir Stoyak
Gave it a try, but does not seem to help. Is it working for you? Thanks Sent from my iPhone > On Dec 3, 2015, at 6:11 PM, Vladimir Stoyak <vsto...@yahoo.com> wrote: > > As far as I know "auto.offset.reset" what to do if offset it not available or >

Re: Session Based Windows

2015-11-18 Thread Vladimir Stoyak
We, were also trying to address session windowing but took slightly different approach as to what window we place the event into. We did not want "triggering event" to be purged as part of the window it triggered, but instead to create a new window for it and have the old window to fire and

Re: Issue with sharing state in CoFlatMapFunction

2015-11-17 Thread Vladimir Stoyak
Not that I necessarily need that for this particular example, but is there a Global State available?  IE, how can I make a state available across all parallel instances of an operator? On Tuesday, November 17, 2015 1:49 PM, Vladimir Stoyak <vsto...@yahoo.com> wrote: P

Re: Issue with sharing state in CoFlatMapFunction

2015-11-17 Thread Vladimir Stoyak
I know I can use broadcast, but was wondering if there is a better way DataStream control_stream = env.addSource(new FlinkKafkaConsumer082(control_topic, new AvroDeserializationSchema(Model.class), properties)).broadcast(); On Tuesday, November 17, 2015 2:45 PM, Vladimir Stoyak <v

Re: Issue with sharing state in CoFlatMapFunction

2015-11-17 Thread Vladimir Stoyak
ll keyBy() before both inputs to make sure that the records are properly routed... Greetings,Stephan On Tue, Nov 17, 2015 at 12:29 PM, Vladimir Stoyak <vsto...@yahoo.com> wrote: Got stuck a bit with CoFlatMapFunction. It seems to work fine if I place it on the DataStream before window but fa