Re: Only a single message processed

Hi Niclas, Glad that you got it working! Thanks for sharing the problem and solution. Best, Fabian 2018-02-19 9:29 GMT+01:00 Niclas Hedhman : > > (Sorry for the incoherent order and ramblings. I am writing this as I am > trying to sort out what is going on...) > > 1. It is

Re: Only a single message processed

(Sorry for the incoherent order and ramblings. I am writing this as I am trying to sort out what is going on...) 1. It is the first message to be processed in the Kafka topic. If I set the offset manually, it will pick up the message at that point, process it, and ignore all following messages.

Re: Only a single message processed

Hi Niclas, About the second point you mentioned, was the processed message a random one or a fixed one? The default startup mode for FlinkKafkaConsumer is StartupMode.GROUP_OFFSETS, maybe you could try StartupMode.EARLIST while debugging. Also, before that, you may try fetching the messages

Re: Only a single message processed

So, the producer is run (at the moment) manually (command-line) one message at a time. Kafka's tooling (different consumer group) shows that a message is added each time. Since my last post, I have also added a UUID as the key, and that didn't make a difference, so you are likely correct about

Re: Only a single message processed

Hi Niclas, Flink's Kafka consumer should not apply any deduplication. AFAIK, such a "feature" is not implemented. Do you produce into the topic that you want to read or is the data in the topic static? If you do not produce in the topic while the consuming application is running, this might be an