Kane,
I'm the creator of kafka-python, just thought I'd give some insight.
Consumer rebalancing is actually pretty tricky to get right. It requires
interaction with ZooKeeper which (though possible via kazoo) is
something I've tried to avoid in kafka-python. It also seems a little
strange to me to mix your consumers between Java/Scala and Python. If
you really need rebalancing between Python consumers, you'd have to
implement that on top of kafka-python.
Once the coordinator API is finalized for 0.9, I (or someone) will work
on implementing it in kafka-python
Cheers
-David
On 10/1/13 11:56 AM, Kane Kane wrote:
The reason i was asking is that this library seems to have support only for
SimpleConsumer https://github.com/mumrah/kafka-python/, i was curious if
all should be implemented on client or kafka has some rebalancing logic and
prevent consuming from the same queue on server side in case of
SimpleConsumer api, but I see now that everything should implemented on the
client side.
Thanks.
On Tue, Oct 1, 2013 at 8:52 AM, Guozhang Wang <wangg...@gmail.com> wrote:
I do not understand your question, what are you trying to implement?
On Tue, Oct 1, 2013 at 8:42 AM, Kane Kane <kane.ist...@gmail.com> wrote:
So essentially you can't do "queue" pattern, unless you somehow implement
locking on the client?
On Tue, Oct 1, 2013 at 8:35 AM, Guozhang Wang <wangg...@gmail.com>
wrote:
SimpleConsumer do not have any concept of group management, only the
high-level consumers have. So multiple simple consumers can
independently
consume from the same partition(s).
Guozhang
On Tue, Oct 1, 2013 at 8:11 AM, Kane Kane <kane.ist...@gmail.com>
wrote:
Yeah, I noticed that, i'm curious how balancing happens if
SimpleConsumer
is used. I.e. i can provide a partition to read from if i use
SimpleConsumer, but what if someone else already attached to that
partition, what would happen? Also what would happen if one
SimpleConsumer
attached to all partitions? Noone would be able to join?
On Tue, Oct 1, 2013 at 6:33 AM, Neha Narkhede <
neha.narkh...@gmail.com
wrote:
There are 2 types of consumer clients in Kafka -
ZookeeperConsumerConnector
and SimpleConsumer. Only the former has the re balancing logic.
Thanks,
Neha
On Oct 1, 2013 6:30 AM, "Kane Kane" <kane.ist...@gmail.com> wrote:
But it looks like some clients don't implement it?
--
-- Guozhang
--
-- Guozhang
