Stevo, Both of the main solutions used by the high-level consumer are standardized and supported directly by the kafka client libraries (e.g. maintaining offsets in zookeeper or in kafka itself). And for the zk case, there is the consumer offset checker (which is good for monitoring). Consumer offset checker still needs to be extended for offsets stored in kafka _consumer_offset topics though.
Anyway, I'm not sure I understand your question, you want something for better monitoring of all possible clients (some of which might choose to manage offsets in their own way)? It's just not part of the kafka design to directly track individual consumers. Jason On Wed, May 27, 2015 at 7:42 AM, Shady Xu <shad...@gmail.com> wrote: > I guess adding a new component will increase the complexity of the system > structure. And if the new component consists of one or a few nodes, it may > becomes the bottleneck of the whole system, if it consists of many nodes, > it will make the system even more complex. > > Although every solution has its downsides, I think the current one is > decent. > > 2015-05-27 17:10 GMT+08:00 Stevo Slavić <ssla...@gmail.com>: > > > It could be a separate server component, does not have to be > > monolith/coupled with broker. > > Such solution would have benefits - single API, pluggable > implementations. > > > > On Wed, May 27, 2015 at 8:57 AM, Shady Xu <shad...@gmail.com> wrote: > > > > > Storing and managing offsets by broker will leave high pressure on the > > > brokers which will affect the performance of the cluster. > > > > > > You can use the advanced consumer APIs, then you can get the offsets > > either > > > from zookeeper or the __consumer_offsets__ topic. On the other hand, if > > you > > > use the simple consumer APIs, you mean to manage offsets yourself, then > > you > > > should monitor them yourself, simple and plain, right? > > > > > > 2015-04-22 14:36 GMT+08:00 Stevo Slavić <ssla...@gmail.com>: > > > > > > > Hello Apache Kafka community, > > > > > > > > Please correct me if wrong, AFAIK currently (Kafka 0.8.2.x) offset > > > > management responsibility is mainly client/consumer side > > responsibility. > > > > > > > > Wouldn't it be better if it was broker side only responsibility? > > > > > > > > E.g. now if one wants to use custom offset management, any of the > Kafka > > > > monitoring tools cannot see the offsets - they would need to use same > > > > custom client implementation which is practically not possible. > > > > > > > > Kind regards, > > > > Stevo Slavic. > > > > > > > > > >
