Re: Query regarding tumbling event time windows with ingestion time

Hello, If you have events: A@10m, B@14m, Watermark@15m C@16m D@25m, Watermark@30m then the result will be: [A,B]@14.59 [C,D]@29.59 This means that for the next windowing, you will have the elements [A,B]@14.59 and [C,D]@29.59 to window in windows of 5min. Given that they are 15min apart,

Gelly simple program but: java.lang.RuntimeException: Memory ran out

I have a similar problem as this topic [1], this problem was caused by a bug in the software, afaik my problem isn't the same. *The error*: Caused by: java.lang.RuntimeException: Memory ran out. Compaction failed. numPartitions: 18 minPartition: 4 maxPartition: 5 number of overflow segments: 54

Time To Live-Setting for State/StateDescriptor

Hi everybody, does Flink offer any automated TTL-functionality for operator state? My current solution for deleting state after some time is using a SessionWindow which triggers after a timeout and sends messages to stateful operators which then call the clear() method on their state. This is my

Regarding windows and custom trigger

Hi everyone, I have a custom trigger implemented for Generic windows. The value which is responsible for triggering the window evaluation keeps on incrementing from some lower value anywhere between 0 to 50 to a certain maximum depending on the time of the day. The only solution which I can see

Re: Flink 1.1.3 OOME Permgen

Hi Stefan, unfortunately, I can not share any heap dumps with you. I was able to resolve some of the issues my self today, the root causes were different for different jobs. 1) Jackson 2.7.2 (which comes with Flink) has a known class loading issue (see

Re: Regarding windows and custom trigger

Hi Abdul, Probably the new enhanced evictors can help you do what you want. You can have a look here: https://cwiki.apache.org/confluence/display/FLINK/FLIP-4+%3A+Enhance+Window+Evictor and also in the related

Re: Query regarding tumbling event time windows with ingestion time

That makes it clear. Thanks On Wed, Nov 30, 2016 at 10:15 PM, Kostas Kloudas < k.klou...@data-artisans.com> wrote: > Hello, > > If you have events: > > A@10m, B@14m, Watermark@15m > C@16m D@25m, Watermark@30m > > then the result will be: > > [A,B]@14.59 > [C,D]@29.59 > > This means that for the

Save the date: ApacheCon Miami, May 15-19, 2017

Dear Apache enthusiast, ApacheCon and Apache Big Data will be held at the Intercontinental in Miami, Florida, May 16-18, 2017. Submit your talks, and register, at http://apachecon.com/ Talks aimed at the Big Data section of the event should go to

RE: Collect() freeze on yarn cluster on strange recover/deserialization error

Hi, Don't think so. I always delete the ZK path before launching the batch (with /usr/bin/zookeeper-client -server $FLINK_HA_ZOOKEEPER_SERVERS rmr $FLINK_HA_ZOOKEEPER_PATH_BATCH), and the "recovery" log line appears only before the collect() phase, not at the beginning. Full log is

Re: Query regarding tumbling event time windows with ingestion time

Hi Janardhan, After the first windowing operation, the timestamp of the emitted element for each window will be the (endOfWindow - 1). So in your case, in the second windowing operation (window by 5) there will be at most one element per window. I hope this answers your question. Kostas >

Re: Query regarding tumbling event time windows with ingestion time

HI i didn't get it , can you please clarify with an example in case each of operation A and B emit multiple elements. On Wed, Nov 30, 2016 at 3:34 PM, Kostas Kloudas wrote: > Hi Janardhan, > > After the first windowing operation, the timestamp of the emitted

Re: Executing graph algorithms on Gelly that are larger then memmory

Hi, can you give us some more details about the algorithm you are testing and your configuration? Flink DataSet operators like join, coGroup, reduce, etc. spill to disk if there is not enough memory. If you are using a delta iteration operator though, the state that is kept across iterations

separation of JVMs for different applications

Hi all, It seems tasks of different Flink applications can end up in the same JVM (TaskManager) in standalone mode. Isn't this fragile since errors in one application could crash another ? I checked FLIP-6 but didn't

Re: Query regarding tumbling event time windows with ingestion time

No problem! Kostas > On Nov 30, 2016, at 7:08 PM, Janardhan Reddy > wrote: > > That makes it clear. > > Thanks > > On Wed, Nov 30, 2016 at 10:15 PM, Kostas Kloudas > wrote: > Hello, > > If you