Hi again, few more questions from me:
*1.*
What I see in JMX:
kafka.consumer:type="ZookeeperConsumerConnector",name="af_servers-af_servers-spm_new_cluster_topic-af_servers_wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize"
>From code:
newGauge(
config.clientId + "-" + config.groupId + "-" + topicThreadId._1 +
"-" + topicThreadId._2 + "-FetchQueueSize",
new Gauge[Int] {
def value = q.size
}
)
I've tried to parse part as I've understood they.
config.clientId >> af_servers
topicThreadId._1 >> af_servers-spm_new_cluster_topic
topicThreadId._2 >> af_servers_wawanawna-Dell-1401353748289-fcaaea29-0
Yes I can suppose that this topicThreadId._1 will always looks like
GROUP_ID+TOPIC and topicThreadId._2 will contain CONSUMER HOST, but will it
always true?
*2.*
>From code I see that sometimes Kafka uses "_" as separator, not only "-":
val consumerIdString = {
var consumerUuid : String = null
config.consumerId match {
case Some(consumerId) // for testing only
=> consumerUuid = consumerId
case None // generate unique consumerId automatically
=> val uuid = UUID.randomUUID()
consumerUuid = "%s-%d-%s".format(
InetAddress.getLocalHost.getHostName, System.currentTimeMillis,
uuid.getMostSignificantBits().toHexString.substring(0,8))
}
config.groupId + "_" + consumerUuid
}
That means if user will use "_" as part of his host/topic/groupId name it
maybe be a problem to parse string like:
kafka.consumer:type="ZookeeperConsumerConnector",name="af_servers-af_servers-spm_new_cluster_topic-af_servers_wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize"
Look at part: "spm_new_cluster_topic-af_servers_wawanawna-Dell", what is
host name here "servers_wawanawna-Dell" or "wawanawna-Dell" ?
So from one side if we want to be able parse name without any problems we
have to avoid using "-" and "_" in host/topic/groupId/clientId, but at the
same time I see (from
http://grokbase.com/t/kafka/users/133xfsnpdh/cant-use-in-client-name):
*"Client id is used for registering jmx beans for monitoring. Because of
the*
*restrictions in bean names, we limit the client id to be only
alpha-numeric*
*plus "-" and "_"."*
Does that mean user can use only camelCase in his
host/topic/groupId/clientId for distinguish one part of name from another?
Is this a problem? Or I didn't understand something?
Best regards from Sematext.
On Tue, Jun 3, 2014 at 3:24 AM, Otis Gospodnetic <[email protected]
> wrote:
> Hi Guozhang,
>
> On Mon, Jun 2, 2014 at 7:18 PM, Guozhang Wang <[email protected]> wrote:
>
> > That is indeed a problem, for now, we recommend group name and topic
> names
> > to use "_" when there is a need for "-", but this should be fixed
> > systematically.
> >
>
> Right!
>
> For you use case, could you change your topic/group name using "_"?
>
>
> Our own Kafka doesn't use topics with "-" characters, so we don't have a
> problem.
>
> The problem, in our case, is that we have a general (Kafka) monitoring tool
> that other people use to monitor Kafka - see http://sematext.com/spm/ .
> So
> we can't really tell people "hey, our tool will work but only if you don't
> have a dash in your topic names and hosts and ... because if you use dashes
> we won't know how to parse your Kafka's MBean names" :)
>
>
> > Also, do you mind to file a JIRA ticket to keep track of this issue?
>
>
> Here it is: https://issues.apache.org/jira/browse/KAFKA-1481
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
>
>
> >
> > On Mon, Jun 2, 2014 at 5:18 AM, Vladimir Tretyakov <
> > [email protected]> wrote:
> >
> > > Hello everyone,
> > >
> > > We are adding Kafka 0.8.x monitoring support to SPM
> > > <http://sematext.com/spm/> here at Sematext. Unfortunately, we quickly
> > hit
> > > an issue caused by the new bean naming convention that embeds things
> like
> > > topic and host names in the beans along with metrics, separated by
> > dashes,
> > > making it hard to parse these beans.
> > >
> > > To put it simply: it is hard/impossible to automatically figure out
> which
> > > part of the bean name is e.g. consumer group, which is the topic, which
> > is
> > > the host name, and which is the name of the metric.
> > >
> > > Let me show you what I mean:
> > >
> > > kafka.consumer:type="ConsumerTopicMetrics",
> > >
> > > name="af_servers-spm_topic-BytesPerSec"
> > >
> > > Here we actually CAN extract:
> > >
> > > * consumer group ('af_servers')
> > >
> > > * topic ('spm_topic')
> > >
> > > * metric (‘BytesPerSec’)
> > >
> > > BUT what if the consumer group id and/or topic name contain '-'?
> > >
> > > Then how would we extract consumer group and topic?
> > >
> > > Here is a concrete example of this problem:
> > >
> > > kafka.consumer:type="ConsumerTopicMetrics",
> > >
> > > name="af-servers-spm-topic-BytesPerSec"
> > >
> > > How can we know what is group id or topic name here?
> > >
> > > This looks like a problem to me, but maybe I’m missing something?
> > >
> > > Is it possible to have all these values (group id, topic name) as
> > separate
> > > attributes inside JMX bean?
> > >
> > > Or maybe the problem could be solved if a different delimiter was used,
> > > such as the pipe (“I”)?
> > >
> > > It is really needed things and will be nice to have it to build good
> tool
> > > for monitoring.
> > >
> > > Thx and best regards from Sematext.
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
