Thank you Ethan for looking into this! However, the partitioning etc is not really the problem here I think. It is more of an issue to understand in what order partitions are emptied and why there might be tuples left in the Kafka topic without being polled into the topology.
We tried to remote debug this and see that one partition at a time is polled, while the others are disabled. However, we cannot see why there would be tuples left unread in the partitions. Any thoughts? /peter G: neubauer.peter S: peter.neubauer P: +46 704 106975 L: http://www.linkedin.com/in/neubauer T: @peterneubauer Mapillary - Join the greatest expedition of our time. On Thu, Apr 9, 2020 at 6:15 AM Ethan Li <[email protected]> wrote: > > Hi Peter, > > I am not very familiar with trident kafka spout. I found something > https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md#manual-partition-assignment-advanced > related. See if it helps. > > Best > Ethan > > On Tue, Apr 7, 2020 at 2:07 AM Peter Neubauer <[email protected]> > wrote: >> >> Hi there, >> we are trying to upgrade our Storm 1.2.3 installation (with Trident >> Kafka spouts) to 2.1. In our tests with 2 topic partitions, we cannot >> really see a pattern on how exactly the different partitions are >> polled (we get messages randomly left in one or both partitions >> without the spout moving, sometimes it moves after a couple of >> minutes, sometimes after we send more messages, sometimes after we >> remote-debug and insert a breakpoint into the KafkaConsumer). The new >> record-size fetch ConsumerConfig.MAX_POLL_RECORDS_CONFIG is very >> welcome btw! >> >> Is there anyone that can help us (even on a consulting base) to go to >> the bottom with the spout fetch behavior so we are not in the blind >> when updating? >> >> Just let me know, or maybe point to relevant documentation in case we >> missed that - thanks a lot! >> >> /peter >> >> G: neubauer.peter >> S: peter.neubauer >> P: +46 704 106975 >> L: http://www.linkedin.com/in/neubauer >> T: @peterneubauer >> >> Mapillary - Join the greatest expedition of our time.
