Hi Ryan, The maxPollInterval waits for at-most the given time duration and returns ASAP even if a single record is available. If you want to collect data once 30-45 minutes, better to use the Consumer with `assign` mode and poll for records once in 30 minutes.
If you're using the consumer with `subscribe` mode, then you have to update the following configs: 1. session.timeout.ms 2. heartbeat.interval.ms and 3. group.max.session.timeout.ms in the broker configs. Increasing the session timeout will lead to delay in detecting the consumer failures, I would suggest to go with `assign` mode. On Tue, Mar 24, 2020 at 4:45 AM Ryan Schachte <coderyanschac...@gmail.com> wrote: > Hey guys, I'm getting a bit overwhelmed by the different variables used to > help enable batching for me. > > I have some custom batching logic that processes when either N records have > been buffered or my max timeout has been hit. It was working decently well, > but I hit this error: > > *This means that the time between subsequent calls to poll() was longer > than the configured max.poll.interval.ms <http://max.poll.interval.ms>, > which typically implies that the poll loop is spending too much time > message processing.* > > I ultimately want to wait for the buffer to fill up or sit and collect data > continuously for 30-45 mins at a time. Do I need to do anything with > heartbeat or session timeout as well? > > So now my question is.. Can I just bump my maxPollInterval to something > like: > > maxPollInterval: '2700000', >
