Re: Stale watermark due to unconsumed Kafka partitions

2019-08-20 Thread Eduardo Winpenny Tejedor
OK that makes sense, I can see how sharing the same Assigner instance between partitions could be tricky, specially if you're handling state. Good to hear this is not expected behaviour, I'm using Flink's JDBCAppendTableSink so nothing particularly unconventional (and I get the same result with JD

Re: Stale watermark due to unconsumed Kafka partitions

2019-08-19 Thread Stephan Ewen
You can use the Timestamp Assigner / Watermark Generator in two different ways: Per Kafka Partition or per parallel source. I would usually recommend per Kafka Partition, because if the read position in the partitions drifts apart (for example some partitions are read at the tail, some are read a

Stale watermark due to unconsumed Kafka partitions

2019-08-16 Thread Eduardo Winpenny Tejedor
Hi all, It was a bit tricky to figure out what was going wrong here, hopefully someone can add the missing piece to the puzzle. I have a Kafka source with a custom AssignerWithPeriodicWatermarks timestamp assigner. It's a copy of the AscendingTimestampExtractor with a log statement printing each