Hi Peter, I am not very familiar with trident kafka spout. I found something https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md#manual-partition-assignment-advanced related. See if it helps.
Best Ethan On Tue, Apr 7, 2020 at 2:07 AM Peter Neubauer <[email protected]> wrote: > Hi there, > we are trying to upgrade our Storm 1.2.3 installation (with Trident > Kafka spouts) to 2.1. In our tests with 2 topic partitions, we cannot > really see a pattern on how exactly the different partitions are > polled (we get messages randomly left in one or both partitions > without the spout moving, sometimes it moves after a couple of > minutes, sometimes after we send more messages, sometimes after we > remote-debug and insert a breakpoint into the KafkaConsumer). The new > record-size fetch ConsumerConfig.MAX_POLL_RECORDS_CONFIG is very > welcome btw! > > Is there anyone that can help us (even on a consulting base) to go to > the bottom with the spout fetch behavior so we are not in the blind > when updating? > > Just let me know, or maybe point to relevant documentation in case we > missed that - thanks a lot! > > /peter > > G: neubauer.peter > S: peter.neubauer > P: +46 704 106975 > L: http://www.linkedin.com/in/neubauer > T: @peterneubauer > > Mapillary - Join the greatest expedition of our time. >
