Jason, So, this sounds like a real issue. Perhaps we can fix it just by setting the tag name as the scope. For example, for mbean kafka.server:type= BrokerTopicMetrics,name=BytesInPerSec,topic=test, we can have
group: "kafka.server" type: "BrokerTopicMetrics" name: "BytesInPerSec" scope: "topic=test" Do you know if scope can have characters like "=" and "," (e.g., for scope like "topic=test,partition=1")? The issue with using mytopic-BytesInPerSec as the name is what we are trying to fix in kafka-1481. Topic name (and clientId, etc) can have dash in it and it's hard to parse. Thanks, Jun On Tue, Jan 27, 2015 at 6:30 AM, Jason Rosenberg <j...@squareup.com> wrote: > Remember multiple people have reported this issue. Per topic metrics no > longer appear in graphite (or in any system modeled after the yammer > GraphiteReporter). They are not being seen as unique. > > While these metrics are registered in the registry as separate ‘MetricName’ > instances (varying only by mbeanName), the GraphiteReporter sends the > metrics to graphite using only the 4 fields I describe above. Consequently, > multiple metrics in the registry get sent to graphite under the same name. > Thus these metrics all end up in the same bucket in graphite, trampling > over each other making them useless. They aren’t ‘double counted’ so much > as flapping between multiple independent values. > > We actually have our own Reporter class (based off the yammer > GraphiteReporter). Our version sends metrics through kafka which is then > consumed downstream by multiple metric consumers. > > The ConsoleReporter isn’t useful for actually persisting metrics anywhere. > It’s just iterating through all the (identically named metrics in the > registry (save for the different mbeanNames))…. > > The mbeanName, as constructed, is not useful as a human readable metric > name, to be presented in a browsable tree of metrics, etc. The > ‘group’:’type’:’name’:’scope’ are the pieces that matter. > > The fix here is to produce MetricName instances similar to 0.8.1.1, etc. In > this case, it should probably be something like: > > group: kafka.server > type: BrokerTopicMetrics > name: mytopic-BytesInPerSec > group: <unused> > > Jason > > > On Tue, Jan 27, 2015 at 7:26 AM, Manikumar Reddy <ku...@nmsworks.co.in> > wrote: > > > I have enabled yammer's ConsoleReporter and I am getting all the metrics > > (including per-topic metrics). > > > > Yammer's MetricName object implements equals/hashcode methods using > > mBeanName . We are constructing a unique mBeanName for each metric, So we > > are not missing/overwriting any metrics. > > > > Current confusion is due to MetricName.name(). This will be same > > (BytesInPerSec) for both broker level and topic level metrics. We need to > > use MetricName.getMBeanName() to differentiate between broker level and > > topic level metrics. > > > > 0.8.1 MBeanName: > > "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec" > > "kafka.server":type="BrokerTopicMetrics",name="MYTOPIC-BytesInPerSec" > > > > 0.8.2 MBeanName: > > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec > > kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=MYTOPIC > > > > > > ConsoleReporter's O/P: > > > > BytesInPerSec: <- This is broker level > > count = 1521 > > mean rate = 3.63 bytes/s > > 1-minute rate = 0.35 bytes/s > > 5-minute rate = 2.07 bytes/s > > 15-minute rate = 1.25 bytes/s > > > > BytesInPerSec: <- This is for topic1 > > count = 626 > > mean rate = 1.89 bytes/s > > 1-minute rate = 0.42 bytes/s > > 5-minute rate = 31.53 bytes/s > > 15-minute rate = 64.66 bytes/s > > > > BytesInPerSec: <- This is for topic2 > > count = 895 > > mean rate = 3.62 bytes/s > > 1-minute rate = 1.39 bytes/s > > 5-minute rate = 30.08 bytes/s > > 15-minute rate = 50.27 bytes/s > > > > Manikumar > > > > On Tue, Jan 27, 2015 at 1:59 PM, Jason Rosenberg <j...@squareup.com> > wrote: > > > > > Ok, > > > > > > It looks like the yammer MetricName is not being created correctly for > > the > > > sub metrics that include a topic. E.g. a metric with an mbeanName like: > > > > > > > > > kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec,topic=mytopic > > > > > > appears to be malformed. A yammer MetricName has 4 fields that are used > > in > > > creating a graphite metric, that are included in the constructor: > > > group, type, name, scope. > > > > > > In this case, the metric with the above mbeanName has these fields set > in > > > the MetricName: > > > > > > group: "kafka.server" > > > type: "BrokerTopicMetrics" > > > name: "BytesInPerSec" > > > scope: null > > > > > > Thus, the topic metrics all look the same, and get lumped into the > > > top-level BrokerTopicMetrics (and thus that will now be double > counted). > > It > > > looks like the fix for kafka-1481 was where things got broken. It seems > > to > > > have introduced ‘tags’ in the building of metric names, and then those > > tags > > > only get applied to the mbeanName, but get excluded from the metric > name: > > > > > > > > > https://github.com/apache/kafka/commit/457744a820d806e546edebbd8ffd33f6772e519f > > > > > > This is a pretty severe issue, since the yammer metrics for these stats > > > will be double counted in aggregate, and the per-topic stats will be > > > removed. > > > > > > I should note too, in my previous email, I thought that only the > > per-topic > > > BrokerTopicMetrics were missing, but also several other per-topic > metrics > > > are missing too, e.g. under kafka.log, etc. > > > > > > Jason > > > > > > > > > On Tue, Jan 27, 2015 at 2:20 AM, Jason Rosenberg <j...@squareup.com> > > wrote: > > > > > > > I can confirm that the per topic metrics are not coming through to > the > > > > yammer metrics registry. I do see them in jmx (via jconsole), but > the > > > > MetricsRegistry does not have them. > > > > All the other metrics are coming through that appear in jmx. > > > > > > > > This is with single node instance running locally. > > > > > > > > Jason > > > > > > > > > > > > > > > > On Mon, Jan 26, 2015 at 8:30 PM, Manikumar Reddy < > ku...@nmsworks.co.in > > > > > > > wrote: > > > > > > > >> If you are using multi-node cluster, then metrics may be reported > from > > > >> other servers. > > > >> pl check all the servers in the cluster. > > > >> > > > >> On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker <kyleban...@gmail.com> > > > >> wrote: > > > >> > > > >> > I've been using a custom KafkaMetricsReporter to report Kafka > broker > > > >> > metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and > > > >> messages in > > > >> > and out for all topics together and for each individual topic. > > > >> > > > > >> > After upgrading to v0.8.2.0, these metrics are no longer being > > > reported. > > > >> > > > > >> > I'm only seeing the following: > > > >> > BrokerTopicMetrics > > > >> > - BytesInPerSec > > > >> > - BytesOutPerSec > > > >> > - BytesRejectedPerSec > > > >> > - MessagesInPerSec > > > >> > > > > >> > What's more, despite lots of successful writes to the cluster, the > > > >> values > > > >> > for these remaining metrics are all zero. > > > >> > > > > >> > I saw that there was some refactoring of metric naming code. Was > the > > > >> > behavior supposed to have changed? > > > >> > > > > >> > Many thanks in advance. > > > >> > > > > >> > > > > > > > > > > > > > >
