Re: Flink flick cancel vs stop

2017-12-14 Thread Piotr Nowojski
Hi, Yes we are aware of this issue and we would like to have it soon, but at the moment it does not look like clean shutdown will be ready for Flink 1.5. Another solution is Kafka exactly-once producer implemented on top of the GenericWriteAheadSink. It could avoid this issue (at a cost of

Re: Flink flick cancel vs stop

2017-12-13 Thread Elias Levy
I am re-upping this thread now that FlinkKafkaProducer011 is out. The new producer, when used with the exactly once semantics, has the rather troublesome behavior that it will fallback to at-most-once, rather than at-least-once, if the job is down for longer than the Kafka broker's

Re: Flink flick cancel vs stop

2017-10-24 Thread Piotr Nowojski
I would propose implementations of NewSource to be not blocking/asynchronous. For example something like public abstract Future getCurrent(); Which would allow us to perform some certain actions while there are no data available to process (for example flush output buffers). Something like

Re: Flink flick cancel vs stop

2017-09-22 Thread Aljoscha Krettek
@Eron Yes, that would be the difference in characterisation. I think technically all sources could be transformed by that by pushing data into a (blocking) queue and having the "getElement()" method pull from that. > On 15. Sep 2017, at 20:17, Elias Levy wrote: >

Re: Flink flick cancel vs stop

2017-09-15 Thread Elias Levy
On Fri, Sep 15, 2017 at 10:02 AM, Eron Wright wrote: > Aljoscha, would it be correct to characterize your idea as a 'pull' source > rather than the current 'push'? It would be interesting to look at the > existing connectors to see how hard it would be to reverse their >

Re: Flink flick cancel vs stop

2017-09-15 Thread Eron Wright
Aljoscha, would it be correct to characterize your idea as a 'pull' source rather than the current 'push'? It would be interesting to look at the existing connectors to see how hard it would be to reverse their orientation. e.g. the source might require a buffer pool. On Fri, Sep 15, 2017 at

Re: Flink flick cancel vs stop

2017-09-15 Thread Aljoscha Krettek
Also relevant for this discussion: Several people (including me) by now were floating the idea of reworking the source interface to take away the responsibility of stopping/canceling/continuing from a specific source implementation and to instead give that power to the system. Currently each

Re: Flink flick cancel vs stop

2017-09-14 Thread Eron Wright
I too am curious about stop vs cancel. I'm trying to understand the motivations a bit more. The current behavior of stop is basically that the sources become bounded, leading to the job winding down. The interesting question is how best to support 'planned' maintenance procedures such as app

Re: Flink flick cancel vs stop

2017-09-14 Thread Ufuk Celebi
Hey Elias, sorry for the delay here. No, stop is not deprecated but not fully implemented yet. One missing part is migration of the existing source functions as you say. Let me pull in Till for more details on this. @Till: Is there more missing than migrating the sources? Here is the PR and

Re: Flink flick cancel vs stop

2017-09-13 Thread Elias Levy
Anyone? On Mon, Sep 11, 2017 at 6:17 PM, Elias Levy wrote: > I was wondering about the status of the flink stop command. At first > blush it would seem as the preferable way to shutdown a Flink job, but it > depends on StoppableFunction being implemented by sources

Flink flick cancel vs stop

2017-09-11 Thread Elias Levy
I was wondering about the status of the flink stop command. At first blush it would seem as the preferable way to shutdown a Flink job, but it depends on StoppableFunction being implemented by sources and I notice that the Kafka source does not seem to implement it. In addition, the command does