Don't I lose consumer group coordination with assign? On Mon, Mar 23, 2020 at 11:49 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote:
> Hi Ryan, > > The maxPollInterval waits for at-most the given time duration and returns > ASAP even if a single record is available. > If you want to collect data once 30-45 minutes, better to use the Consumer > with `assign` mode and poll for records > once in 30 minutes. > > If you're using the consumer with `subscribe` mode, then you have to update > the following configs: > 1. session.timeout.ms > 2. heartbeat.interval.ms and > 3. group.max.session.timeout.ms in the broker configs. > > Increasing the session timeout will lead to delay in detecting the consumer > failures, I would suggest to go with `assign` mode. > > > On Tue, Mar 24, 2020 at 4:45 AM Ryan Schachte <coderyanschac...@gmail.com> > wrote: > > > Hey guys, I'm getting a bit overwhelmed by the different variables used > to > > help enable batching for me. > > > > I have some custom batching logic that processes when either N records > have > > been buffered or my max timeout has been hit. It was working decently > well, > > but I hit this error: > > > > *This means that the time between subsequent calls to poll() was longer > > than the configured max.poll.interval.ms <http://max.poll.interval.ms>, > > which typically implies that the poll loop is spending too much time > > message processing.* > > > > I ultimately want to wait for the buffer to fill up or sit and collect > data > > continuously for 30-45 mins at a time. Do I need to do anything with > > heartbeat or session timeout as well? > > > > So now my question is.. Can I just bump my maxPollInterval to something > > like: > > > > maxPollInterval: '2700000', > > >
