First of all, from a performance point of view, it does not make sense to have two consumers of the same group in one thread. They will share the same network and same CPU resource (but are not able to utilize them more efficiently as both are on the same thread). Each consumer within a group will get some partitions assigned, thus, instead of having two consumers (with partition set A and B assigned), just use one consumer that gets A+B assigned and it will perform equally efficient.
Why does it block? Well, after the first consumer calls poll(), it will
join the group and the group will have 1 member. When the second
consumer calls poll() it will try to join the group. The broker side
group coordinator knows that was already one consumer in the group and
it will wait until the first consumer say "I am still here and still
part of the group" -- however, this will never happen, because the first
consumer would do this via calling poll(), but it can't, because the
second consumer is stuck in its own poll() call to actually join the
group. Eventually the first consumer would time out and be remove from
the group and the second consumer unblocks (the group has still one
member only). Afterward the first consumer will retry to join the
group... etc. etc. This ping pong game will continue forever...
-Matthias
On 12/10/19 5:52 AM, Sasa Trifunovic wrote:
> I have two Kafka consumers, subscribed to different topics and belonging
> to the same consumer group, having different consumer ids, running in the
> same thread. They are executing poll sequentially but after the first is
> done second seems to be stuck in poll. I tried using proper Kafka broker
> and embedded one for testing and nothing changes.I tried associating them
> with different consumer groups and that seems to be working but
> unfortunately, that is not a viable solution for me.
>
> I found this in "Kafka: The Definitive Guide":
>
> "You can’t have multiple consumers that belong to the same group in one
> thread and you can’t have multiple threads safely use the same consumer.
> One consumer per thread is the rule."
>
> That quote directs me towards some form of thread cooperation due to the
> specific order of message processing I need to do.
>
> Can someone provide an explanation of why is it necessary to run different
> consumers belonging to the same consumer group, subscribed to different
> topics in separate threads?
>
> Test code with embedded kafka:
>
> @Test
> public void pollTest() {
> kafkaAdmin.createTopics(topics1);
> kafkaAdmin.createTopics(topics2);
>
> Consumer<String, String> consumer1 = createConsumer(topics1);
> Consumer<String, String> consumer2 = createConsumer(topics2);
>
> consumer1.poll(200);
> consumer2.poll(200);
> }
>
> Version of kafka - 2.3.0
>
> Thank you.
>
signature.asc
Description: OpenPGP digital signature
