A possible workaround while waiting for FLINK-5479, if someone is hitting the same problem: we chose to send "heartbeat" messages periodically to all topics & partitions found on our Kafka. We do that through the service that normally writes to our Kafka. This way every partition always has some ~recent timestamps.
On Wed, Dec 13, 2017 at 1:06 PM, Gerard Garcia <ger...@talaia.io> wrote: > Thanks Gordon. > > Don't worry, I'll be careful to not have empty partitions until the next > release. > Also, I'll keep an eye to FLINK-5479 and if at some point I see that > there is a fix and the issue bothers us too much I'll try to apply the > patch myself to the latest stable release. > > Gerard > > On Wed, Dec 13, 2017 at 10:31 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org > > wrote: > >> Hi, >> >> I've just elevated FLINK-5479 to BLOCKER for 1.5. >> >> Unfortunately, AFAIK there is no easy workaround solution for this issue >> yet in the releases so far. >> The min watermark logic that controls per-partition watermark emission is >> hidden inside the consumer, making it hard to work around it. >> >> One possible solution I can imagine, but perhaps not that trivial, is to >> inject some special marker event into all partitions periodically. >> The watermark assigner should be able to recognize this special marker >> and try to provide some watermark for it. >> Another option is that I can provide some patch you can apply for a >> custom build of the Kafka connector that handles partition idleness >> properly. >> However, given that we're aiming for a faster release cycle for Flink 1.5 >> (proposed release date is Feb. 2018), it might not be worth the extra >> maintenance effort on your side of a custom build. >> >> Best, >> Gordon >> >> >> On Tue, Dec 12, 2017 at 9:28 PM, gerardg <ger...@talaia.io> wrote: >> >>> I'm also affected by this behavior. There are no updates in FLINK-5479 >>> but >>> did you manage to find a way to workaround this? >>> >>> Thanks, >>> >>> Gerard >>> >>> >>> >>> -- >>> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>> ble.com/ >>> >> >> >