Let me see if I can do artificial throttle somewhere. Volume of data is really
high and hence trying to avoid rounds in Kafka too. Looks like options are “not
so elegant” until FLIP-15. Thanks for pointers again!!!
On Monday, May 4, 2020, 11:06 PM, Ken Krugler
wrote:
Hi Ashish,
The
Hi Ashish,
The workaround we did was to throttle data flowing in the iteration (in code),
though not sure if that’s possible for your situation.
You could remove the iteration by writing to a Kafka topic at the end of the
part of your workflow that is currently an iteration, and then consuming
Hi Ken,
Thanks for the quick response!
I came across FLIP-15 on my next google search after I sent email :) It
DEFINITELY looks that way. As I was watching logs and nature of how job gets
stuck it does look like buffer is blocked. But FLIP-15 has not moved further
though. So there are no
Hi Ashish,
Wondering if you’re running into the gridlock problem I mention on slide #25
here:
https://www.slideshare.net/FlinkForward/flink-forward-san-francisco-2018-ken-krugler-building-a-scalable-focused-web-crawler-with-flink
Could this be FLIP-15 related as well then?
> On May 4, 2020, at 9:41 PM, Ashish Pokharel wrote:
>
> Hi all,
>
> Hope everyone is doing well!
>
> I am running into what seems like a deadlock (application stalled) situation
> with a Flink streaming job upon restore from savepoint. Job has a
Hi all,
Hope everyone is doing well!
I am running into what seems like a deadlock (application stalled) situation
with a Flink streaming job upon restore from savepoint. Job has a slowly moving
stream (S1) that needs to be “stateful” and a continuous stream (S2) which is
“joined” with slow
