Re: Stopping a kafka consumer gracefully (no losing of inflight events, StoppableFunction)

Ok. Thanks a lot of the clarification. That was my initial understanding but then was confused by the "losing in-flight events" wording. On Wed, Feb 21, 2018 at 10:26 AM, Till Rohrmann wrote: > Hi Christophe, > > yes I think you misunderstood the thread. Cancel with

Re: Stopping a kafka consumer gracefully (no losing of inflight events, StoppableFunction)

@Bart, I think there is no Flip yet for the proper stop with savepoint implementation. My gut feeling is that the community will soon address this problem since it's a heavily requested feature. Cheers, Till On Wed, Feb 21, 2018 at 10:26 AM, Till Rohrmann wrote: > Hi

Re: Stopping a kafka consumer gracefully (no losing of inflight events, StoppableFunction)

Hi Christophe, yes I think you misunderstood the thread. Cancel with savepoint will never cause any data loss. The only problem which might arise if you have an operator which writes data to an external system immediately, then you might see some data in the external system which originates from

Re: Stopping a kafka consumer gracefully (no losing of inflight events, StoppableFunction)

Hmm, I did not realize that. I was planning when upgrading a job (consuming from Kafka) to cancel it with a savepoint and then start it back from the savedpoint. But this savedpoint thing was giving me the apparently false feeling I would not lose anything? My understanding was that maybe I would

Re: Stopping a kafka consumer gracefully (no losing of inflight events, StoppableFunction)

Thanks for the reply; is there a flip for this? - bart On Mon, Feb 19, 2018, at 5:50 PM, Till Rohrmann wrote: > Hi Bart, > > you're right that Flink currently does not support a graceful stop > mechanism for the Kafka source. The community has already a good > idea how to solve it in the

Re: Stopping a kafka consumer gracefully (no losing of inflight events, StoppableFunction)

Hi Bart, you're right that Flink currently does not support a graceful stop mechanism for the Kafka source. The community has already a good idea how to solve it in the general case and will hopefully soon add it to Flink. Concerning the StoppableFunction: This interface was introduced quite

Stopping a kafka consumer gracefully (no losing of inflight events, StoppableFunction)

In https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/cli.html it is shown that for gracefully stopping a job you need to implement the StoppableFunction interface. This appears not (yet) implemented for Kafka consumers. Am I missing something, or is there a different way to