Re: How to keep consumers alive without polling new messages

[Yifan Ying]

Thanks Vincent and Kamal. That solves my problem. :)

Yifan

On Tue, Sep 27, 2016 at 11:36 PM, Kamal C  wrote:

> You can refer this example[1]
>
> [1]:
> https://github.com/omkreddy/kafka-examples/blob/master/
> consumer/src/main/java/kafka/examples/consumer/advanced/
> AdvancedConsumer.java
>
> - Kamal
>
> On Wed, Sep 28, 2016 at 11:33 AM, Vincent Dautremont <
> vincent.dautrem...@olamobile.com> wrote:
>
> > I had the same problem :
> > Call pause() on all partitions.
> > Then continue your loop that calls consume(), it will then poll without
> > consuming messages.
> >
> > When you want to consume again, call resume() on all partition
> >
> > It's not obvious at all, the doc should explain that in the documentation
> > of method consume() and put it in red and bold.
> > https://kafka.apache.org/090/javadoc/index.html?org/apache/
> > kafka/clients/consumer/KafkaConsumer.html
> >
> > > Le 28 sept. 2016 à 06:21, Yifan Ying  a écrit :
> > >
> > > Hi all,
> > >
> > > 0.10 consumers use poll() method to heartbeat Kafka brokers. Is there
> any
> > > way that I can make the consumer heartbeat but not poll any messages?
> The
> > > javadoc says, the recommended way is to move message processing to
> > another
> > > thread. But when message processing keeps failing(because a third party
> > > service goes down for a while), the thread that actually processes
> > messages
> > > could have too many messages accumulated. Maybe re-sending failed
> > messages
> > > to another queue(IMQ) and re-processing them later is a good option?
> > >
> > > Thanks!
> > > --
> > > Yifan
> >
>



-- 
Yifan

Re: How to keep consumers alive without polling new messages

[Kamal C]

You can refer this example[1]

[1]:
https://github.com/omkreddy/kafka-examples/blob/master/consumer/src/main/java/kafka/examples/consumer/advanced/AdvancedConsumer.java

- Kamal

On Wed, Sep 28, 2016 at 11:33 AM, Vincent Dautremont <
vincent.dautrem...@olamobile.com> wrote:

> I had the same problem :
> Call pause() on all partitions.
> Then continue your loop that calls consume(), it will then poll without
> consuming messages.
>
> When you want to consume again, call resume() on all partition
>
> It's not obvious at all, the doc should explain that in the documentation
> of method consume() and put it in red and bold.
> https://kafka.apache.org/090/javadoc/index.html?org/apache/
> kafka/clients/consumer/KafkaConsumer.html
>
> > Le 28 sept. 2016 à 06:21, Yifan Ying  a écrit :
> >
> > Hi all,
> >
> > 0.10 consumers use poll() method to heartbeat Kafka brokers. Is there any
> > way that I can make the consumer heartbeat but not poll any messages? The
> > javadoc says, the recommended way is to move message processing to
> another
> > thread. But when message processing keeps failing(because a third party
> > service goes down for a while), the thread that actually processes
> messages
> > could have too many messages accumulated. Maybe re-sending failed
> messages
> > to another queue(IMQ) and re-processing them later is a good option?
> >
> > Thanks!
> > --
> > Yifan
>

Re: How to keep consumers alive without polling new messages

[Vincent Dautremont]

I had the same problem :
Call pause() on all partitions.
Then continue your loop that calls consume(), it will then poll without 
consuming messages.

When you want to consume again, call resume() on all partition

It's not obvious at all, the doc should explain that in the documentation of 
method consume() and put it in red and bold.
https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

> Le 28 sept. 2016 à 06:21, Yifan Ying  a écrit :
> 
> Hi all,
> 
> 0.10 consumers use poll() method to heartbeat Kafka brokers. Is there any
> way that I can make the consumer heartbeat but not poll any messages? The
> javadoc says, the recommended way is to move message processing to another
> thread. But when message processing keeps failing(because a third party
> service goes down for a while), the thread that actually processes messages
> could have too many messages accumulated. Maybe re-sending failed messages
> to another queue(IMQ) and re-processing them later is a good option?
> 
> Thanks!
> -- 
> Yifan

How to keep consumers alive without polling new messages

[Yifan Ying]

Hi all,

0.10 consumers use poll() method to heartbeat Kafka brokers. Is there any
way that I can make the consumer heartbeat but not poll any messages? The
javadoc says, the recommended way is to move message processing to another
thread. But when message processing keeps failing(because a third party
service goes down for a while), the thread that actually processes messages
could have too many messages accumulated. Maybe re-sending failed messages
to another queue(IMQ) and re-processing them later is a good option?

Thanks!
-- 
Yifan