Thank you for the explanation.
I tried using them in the same thread because there is a need for message
prioritization,
thus I tried polling from high priority and then from lower priority topic.
On Wed, Dec 11, 2019 at 8:24 AM Matthias J. Sax <matth...@confluent.io>
wrote:
> First of all, from a performance point of view, it does not make sense
> to have two consumers of the same group in one thread. They will share
> the same network and same CPU resource (but are not able to utilize them
> more efficiently as both are on the same thread). Each consumer within a
> group will get some partitions assigned, thus, instead of having two
> consumers (with partition set A and B assigned), just use one consumer
> that gets A+B assigned and it will perform equally efficient.
>
> Why does it block? Well, after the first consumer calls poll(), it will
> join the group and the group will have 1 member. When the second
> consumer calls poll() it will try to join the group. The broker side
> group coordinator knows that was already one consumer in the group and
> it will wait until the first consumer say "I am still here and still
> part of the group" -- however, this will never happen, because the first
> consumer would do this via calling poll(), but it can't, because the
> second consumer is stuck in its own poll() call to actually join the
> group. Eventually the first consumer would time out and be remove from
> the group and the second consumer unblocks (the group has still one
> member only). Afterward the first consumer will retry to join the
> group... etc. etc. This ping pong game will continue forever...
>
> -Matthias
>
> On 12/10/19 5:52 AM, Sasa Trifunovic wrote:
> > I have two Kafka consumers, subscribed to different topics and belonging
> > to the same consumer group, having different consumer ids, running in the
> > same thread. They are executing poll sequentially but after the first is
> > done second seems to be stuck in poll. I tried using proper Kafka broker
> > and embedded one for testing and nothing changes.I tried associating them
> > with different consumer groups and that seems to be working but
> > unfortunately, that is not a viable solution for me.
> >
> > I found this in "Kafka: The Definitive Guide":
> >
> > "You can’t have multiple consumers that belong to the same group in one
> > thread and you can’t have multiple threads safely use the same consumer.
> > One consumer per thread is the rule."
> >
> > That quote directs me towards some form of thread cooperation due to the
> > specific order of message processing I need to do.
> >
> > Can someone provide an explanation of why is it necessary to run
> different
> > consumers belonging to the same consumer group, subscribed to different
> > topics in separate threads?
> >
> > Test code with embedded kafka:
> >
> > @Test
> > public void pollTest() {
> > kafkaAdmin.createTopics(topics1);
> > kafkaAdmin.createTopics(topics2);
> >
> > Consumer<String, String> consumer1 = createConsumer(topics1);
> > Consumer<String, String> consumer2 = createConsumer(topics2);
> >
> > consumer1.poll(200);
> > consumer2.poll(200);
> > }
> >
> > Version of kafka - 2.3.0
> >
> > Thank you.
> >
>
>
--
www.fractalcastle.com
