Re: Cassandra statebackend

2016-07-25 Thread Chen Qin
Hi Aljoscha, Cool! I created a JIRA for this. https://issues.apache.org/jira/browse/FLINK-4266 Some comments inline. Chen On Mon, Jul 25, 2016 at 2:41 AM, Aljoscha Krettek wrote: > Hi, > I thought there was a Jira for that but I looked and couldn't find it. If > you'd like you can create one a

[jira] [Created] (FLINK-4266) Cassandra Statebackend

2016-07-25 Thread Chen Qin (JIRA)
Chen Qin created FLINK-4266: --- Summary: Cassandra Statebackend Key: FLINK-4266 URL: https://issues.apache.org/jira/browse/FLINK-4266 Project: Flink Issue Type: New Feature Components: Stat

[jira] [Created] (FLINK-4265) Add a NoOpOperator

2016-07-25 Thread Greg Hogan (JIRA)
Greg Hogan created FLINK-4265: - Summary: Add a NoOpOperator Key: FLINK-4265 URL: https://issues.apache.org/jira/browse/FLINK-4265 Project: Flink Issue Type: New Feature Components: Data

[ANNOUNCE] Flink Forward 2016: First round of speakers and sessions is out

2016-07-25 Thread Kostas Tzoumas
Hi everyone, I wanted to share this with the community: we have announced the first round of speakers and sessions of Flink Forward 2016, and it looks amazing! Check it out here: http://flink-forward.org/program/sessions/ This year we have a great mix of use case talks (e.g., by Netflix, Alibaba

[jira] [Created] (FLINK-4264) New GraphMetrics driver

2016-07-25 Thread Greg Hogan (JIRA)
Greg Hogan created FLINK-4264: - Summary: New GraphMetrics driver Key: FLINK-4264 URL: https://issues.apache.org/jira/browse/FLINK-4264 Project: Flink Issue Type: New Feature Components:

[jira] [Created] (FLINK-4263) SQL's VALUES does not work properly

2016-07-25 Thread Timo Walther (JIRA)
Timo Walther created FLINK-4263: --- Summary: SQL's VALUES does not work properly Key: FLINK-4263 URL: https://issues.apache.org/jira/browse/FLINK-4263 Project: Flink Issue Type: Bug Com

[jira] [Created] (FLINK-4262) Consider null handling during sorting

2016-07-25 Thread Timo Walther (JIRA)
Timo Walther created FLINK-4262: --- Summary: Consider null handling during sorting Key: FLINK-4262 URL: https://issues.apache.org/jira/browse/FLINK-4262 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-4261) Setup atomic deployment of snapshots

2016-07-25 Thread Maximilian Michels (JIRA)
Maximilian Michels created FLINK-4261: - Summary: Setup atomic deployment of snapshots Key: FLINK-4261 URL: https://issues.apache.org/jira/browse/FLINK-4261 Project: Flink Issue Type: Bug

Re: [DISCUSS] Allowed Lateness in Flink

2016-07-25 Thread Aljoscha Krettek
Hi, yes, this is essentially the solution I had in my head but I went a bit further and generalized it. Basically, to make triggers composable they should have this interface, let's call it SimpleTrigger for now: class SimpleTrigger { void onElement(T element, long timestamp, W window, TriggerC

Re: [DISCUSS] Allowed Lateness in Flink

2016-07-25 Thread Kostas Kloudas
Hi Aljoscha, This was exactly one of the problems I also found. The way I was thinking about it is the following: Conceptually, time (event and processing) advances but state is a fixed property of the window. Given this, I modified the Count trigger to also ask for the current state (count)

Re: [DISCUSS] Allowed Lateness in Flink

2016-07-25 Thread Aljoscha Krettek
These are some very interesting thoughts! I have some more, based on these: What happens if you have for example this Trigger: All(Watermark.pastEndOfWindow(), Count.atLeast(10)) When would this even fire, i.e. what are the steps that lead to this combined trigger firing with the Trigger system t

[jira] [Created] (FLINK-4260) Allow SQL's LIKE ESCAPE

2016-07-25 Thread Timo Walther (JIRA)
Timo Walther created FLINK-4260: --- Summary: Allow SQL's LIKE ESCAPE Key: FLINK-4260 URL: https://issues.apache.org/jira/browse/FLINK-4260 Project: Flink Issue Type: Improvement Compone

Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

2016-07-25 Thread Aljoscha Krettek
Hi, the elements are currently not being removed from the buffers. That's a bug that we could fix while adding the new Evictor interface. Cheers, Aljoscha On Mon, 25 Jul 2016 at 13:00 Radu Tudoran wrote: > Hi Aljoscha, > > Can you point us to the way it is handled now. Is there anything else fo

RE: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

2016-07-25 Thread Radu Tudoran
Hi Aljoscha, Can you point us to the way it is handled now. Is there anything else for the removing of elements other than the skip in EvictingWindowOperator. Is there something as it was before version 1.x where you had an explicit remove from window buffers? Dr. Radu Tudoran Research Enginee

Re: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

2016-07-25 Thread Aljoscha Krettek
Hi, I think there is not yet a clear specification for how the actual removal of elements from the buffer will work. I think naively one can do: Iterable currentElements = state.get() evictor.evict(currentElements); // this will remove some stuff from there, or mark for removal state.clear() // t

Re: Cassandra statebackend

2016-07-25 Thread Aljoscha Krettek
Hi, I thought there was a Jira for that but I looked and couldn't find it. If you'd like you can create one and we can discuss the design. Do you have any ideas yet? The tricky things I see in this are: - Knowing which data is the current data. This will require some kind of timestamps or increas

RE: [DISCUSS][FLIP-4] Enhance Window Evictor in Flink

2016-07-25 Thread Radu Tudoran
Hi, Thanks for the clarification. Can someone point to where the events are removed from buffers - I am trying to understand the new logic of handling the eviction in this new API. Thanks -Original Message- From: Vishnu Viswanath [mailto:vishnu.viswanat...@gmail.com] Sent: Saturday,