Yes, with `assign` you'll lose the group coordination. You can still use the `subscribe` mode, update the above mentioned configs. You're ask is kind of Delay Queue. Kafka Consumer doesn't support that feature. You've to manually `sleep` in between the poll calls.
On Tue, Mar 24, 2020 at 11:56 PM Ryan Schachte <coderyanschac...@gmail.com> wrote: > Don't I lose consumer group coordination with assign? > > On Mon, Mar 23, 2020 at 11:49 PM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Hi Ryan, > > > > The maxPollInterval waits for at-most the given time duration and returns > > ASAP even if a single record is available. > > If you want to collect data once 30-45 minutes, better to use the > Consumer > > with `assign` mode and poll for records > > once in 30 minutes. > > > > If you're using the consumer with `subscribe` mode, then you have to > update > > the following configs: > > 1. session.timeout.ms > > 2. heartbeat.interval.ms and > > 3. group.max.session.timeout.ms in the broker configs. > > > > Increasing the session timeout will lead to delay in detecting the > consumer > > failures, I would suggest to go with `assign` mode. > > > > > > On Tue, Mar 24, 2020 at 4:45 AM Ryan Schachte < > coderyanschac...@gmail.com> > > wrote: > > > > > Hey guys, I'm getting a bit overwhelmed by the different variables used > > to > > > help enable batching for me. > > > > > > I have some custom batching logic that processes when either N records > > have > > > been buffered or my max timeout has been hit. It was working decently > > well, > > > but I hit this error: > > > > > > *This means that the time between subsequent calls to poll() was longer > > > than the configured max.poll.interval.ms <http://max.poll.interval.ms > >, > > > which typically implies that the poll loop is spending too much time > > > message processing.* > > > > > > I ultimately want to wait for the buffer to fill up or sit and collect > > data > > > continuously for 30-45 mins at a time. Do I need to do anything with > > > heartbeat or session timeout as well? > > > > > > So now my question is.. Can I just bump my maxPollInterval to something > > > like: > > > > > > maxPollInterval: '2700000', > > > > > >
