Hi, I haven't followed the changes to offset tracking closely, other than that storing them in ZK is not the only option any more. I think what Stevo is asking about/suggesting is that there there be a single API from which offset information can be retrieved (e.g. by monitoring tools), so that monitoring tools work regardless of where one chose to store offsets. I know we'd love to have this for SPM's Kafka monitoring and can tell you that adding support for N different APIs for N different offset storage systems would be hard/time-consuming/expensive. But maybe this single API already exists?
Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Mon, Jun 1, 2015 at 4:41 PM, Jason Rosenberg <j...@squareup.com> wrote: > Stevo, > > Both of the main solutions used by the high-level consumer are standardized > and supported directly by the kafka client libraries (e.g. maintaining > offsets in zookeeper or in kafka itself). And for the zk case, there is > the consumer offset checker (which is good for monitoring). Consumer > offset checker still needs to be extended for offsets stored in kafka > _consumer_offset topics though. > > Anyway, I'm not sure I understand your question, you want something for > better monitoring of all possible clients (some of which might choose to > manage offsets in their own way)? > > It's just not part of the kafka design to directly track individual > consumers. > > Jason > > On Wed, May 27, 2015 at 7:42 AM, Shady Xu <shad...@gmail.com> wrote: > > > I guess adding a new component will increase the complexity of the system > > structure. And if the new component consists of one or a few nodes, it > may > > becomes the bottleneck of the whole system, if it consists of many nodes, > > it will make the system even more complex. > > > > Although every solution has its downsides, I think the current one is > > decent. > > > > 2015-05-27 17:10 GMT+08:00 Stevo Slavić <ssla...@gmail.com>: > > > > > It could be a separate server component, does not have to be > > > monolith/coupled with broker. > > > Such solution would have benefits - single API, pluggable > > implementations. > > > > > > On Wed, May 27, 2015 at 8:57 AM, Shady Xu <shad...@gmail.com> wrote: > > > > > > > Storing and managing offsets by broker will leave high pressure on > the > > > > brokers which will affect the performance of the cluster. > > > > > > > > You can use the advanced consumer APIs, then you can get the offsets > > > either > > > > from zookeeper or the __consumer_offsets__ topic. On the other hand, > if > > > you > > > > use the simple consumer APIs, you mean to manage offsets yourself, > then > > > you > > > > should monitor them yourself, simple and plain, right? > > > > > > > > 2015-04-22 14:36 GMT+08:00 Stevo Slavić <ssla...@gmail.com>: > > > > > > > > > Hello Apache Kafka community, > > > > > > > > > > Please correct me if wrong, AFAIK currently (Kafka 0.8.2.x) offset > > > > > management responsibility is mainly client/consumer side > > > responsibility. > > > > > > > > > > Wouldn't it be better if it was broker side only responsibility? > > > > > > > > > > E.g. now if one wants to use custom offset management, any of the > > Kafka > > > > > monitoring tools cannot see the offsets - they would need to use > same > > > > > custom client implementation which is practically not possible. > > > > > > > > > > Kind regards, > > > > > Stevo Slavic. > > > > > > > > > > > > > > >
