Re: Clean shutdown of streaming job

Hi Ning, I don't think it is possible to pause a Kafka source upon taking a savepoint without making any changes to the implementation. I think your problem is that the Cassandra sink doesn't support exactly once guarantees when the Cassandra query isn't idempotent. If possible, the cleanest

Re: Manual trigger the window in fold operator or incremental aggregation

stinguish within apply function of 'RichWindowFunction' whether > it was called due to onElement trigger call or onProcessingtime trigger > call of a custom Trigger ? > > Thanks! > > On Wed, 17 Oct 2018 at 12:51, Niels van Kaam wrote: > >> Hi Zhen Li, >> >> Y

Re: Checkpointing when one of the sources has completed

ely > Joshua > > On Wed, Oct 17, 2018 at 4:58 PM Niels van Kaam wrote: > >> Hi All, >> >> I am debugging an issue where the periodic checkpointing has halted. I >> noticed that one of the sources of my job has completed (finished). The >> other sources and o

Re: Manual trigger the window in fold operator or incremental aggregation

Hi Zhen Li, You can control when a windowed stream emits data with "Triggers". See: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#triggers Flink comes with a couple of default triggers, but you can also create your own by implementing

Checkpointing when one of the sources has completed

Hi All, I am debugging an issue where the periodic checkpointing has halted. I noticed that one of the sources of my job has completed (finished). The other sources and operators would however still be able to produce output. Does anyone know if Flink's periodic checkpoints are supposed to

Re: Akka Http used in custom RichSourceFunction

Yes, this might be the cause of the issue, because indeed it looks like > your akka’s version is leaking to Flink’s classloader. > > Piotrek > > > On 25 May 2018, at 09:40, Niels van Kaam <ni...@vankaam.net> wrote: > > Hi Piotrek, > > Thank you for your response!

Re: Akka Http used in custom RichSourceFunction

gt; doesn’t help, probably you should shade your akka dependency. > > What is the full exception? Is it thrown when YOURS code tries to > shutdown, or when FLINK’s code tries to shutdown? > > Piotrek > > > On 24 May 2018, at 14:38, Niels van Kaam <ni...@vank

Akka Http used in custom RichSourceFunction

Hi All, I wrote a custom source function (RichSourceFunction) which connects to a web socket using the Akka Http Library. The job using this source runs fine on a local environment until upon shutdown I see the following error in the log: "Exception in thread "main" java.lang.NoSuchMethodError:

Re: "Close()" aborts last transaction in TwoPhaseCommitSinkFunction

g a source function or at least wrapping it and there might > be some corner cases that I haven’t thought about. > > Piotrek > > > On 9 Mar 2018, at 14:49, Niels van Kaam <ni...@vankaam.net> wrote: > > Hi, > > I'm working on a custom implementation of a sink which I w

"Close()" aborts last transaction in TwoPhaseCommitSinkFunction

Hi, I'm working on a custom implementation of a sink which I would like to use with exactly once semantics. Therefore I have implemented the TwoPhaseCommitSinkFunction class as mentioned in this recent post: https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apache-flink.html I