Re: data loss after implementing checkpoint

2017-07-18 Thread Sridhar Chellappa
The CEP library runs Correction; the CEP implemented using the CEP library runs . On Wed, Jul 19, 2017 at 10:08 AM, Sridhar Chellappa wrote: > A follow up question on this. I have a Complex Event processor implemented > using the CEP library (1.3.0). The CEP

Re: data loss after implementing checkpoint

2017-07-18 Thread Sridhar Chellappa
A follow up question on this. I have a Complex Event processor implemented using the CEP library (1.3.0). The CEP library runs a variety of rules that are configured (enable/disable rule) VIA REST APIs. Now, if my application crashes and recovers (or is cancelled and restarted), will my

Re: AVRO Union type support in Flink

2017-07-18 Thread Martin Eden
Hey Vishnu, For those of us on the list that are not very familiar with Flink and Avro can you give a pointed to the docs you are referring to and how you intend to use it? Just so we gain understanding as well. Thanks, Martin On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath <

AVRO Union type support in Flink

2017-07-18 Thread Vishnu Viswanath
Hi All, Does Flink support AVRO union types - Documentation says it supports nullable types: {"name": "type_double_test", "type": ["null", "double"]} But my schema has something like : {"name": "union_field", "type": ["string", "double"]} Thanks Vishnu

Re: [POLL] Dropping savepoint format compatibility for 1.1.x in the Flink 1.4.0 release

2017-07-18 Thread Stephan Ewen
Hi Kanstantsin! Looking at other mail thread, I think you refer to the CEP library API, correct? The CEP library has not been labeled as stable yet. The stability guarantees that we introduced refer to the APIs marked with the `@Public` interface. The CEP library was still too much in

Re: Kafka control source in addition to Kafka data source

2017-07-18 Thread Konstantin Knauf
Hi Gabriele, I think this is actually a quite common pattern. Generally, you can `join` the two streams and then use a `CoFlatMapFunction`. A `CoFlatMapFunction` allows you to keep shared (checkpointed) state between two streams. It has two callbacks `flatMap1` and `flatMap2` which are called

Kafka control source in addition to Kafka data source

2017-07-18 Thread Gabriele Di Bernardo
Hello everyone, I am a Flink newcomer and I would like to implement a Flink application with two Kafka sources: one for the data stream to be processed and the other one for control purposes. The application should be able to read from the control stream and then apply the control operation to

Re: [POLL] Who still uses Java 7 with Flink ?

2017-07-18 Thread Stephan Ewen
All right, thanks everyone. I think the consensus here is clear :-) On Thu, Jul 13, 2017 at 5:17 PM, nragon wrote: > +1 dropping java 7 > > > > -- > View this message in context: http://apache-flink-user- >

Re: Flink rolling upgrade support

2017-07-18 Thread Moiz Jinia
Aljoscha Krettek wrote > Hi, > zero-downtime updates are currently not supported. What is supported in > Flink right now is a savepoint-shutdown-restore cycle. With this, you > first > draw a savepoint (which is essentially a checkpoint with some meta data), > then you cancel your job, then you do