subject:"\[jira\] \[Commented\] \(KAFKA\-1481\) Stop using dashes AND underscores as separators in MBean names"

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2015-03-18 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367878#comment-14367878
 ] 

Guozhang Wang commented on KAFKA-1481:
--

We hit an issue related to this ticket, which adds the brokerHost / 
brokerPort into FetchRequestAndResponseMetrics. The root cause is that when 
server starts up, it gets local host string by calling:

{code}
InetAddress.getLocalHost.getCanonicalHostName
{code}

which, will possibly just return the textual representation of the IP address 
if somehow accessing local hostname is not allowed:

http://docs.oracle.com/javase/7/docs/api/java/net/InetAddress.html#getCanonicalHostName%28%29

In our case, the IPV6 address string is returned, which is registered in ZK, 
read by controller and propogated to brokers through metadata update, and 
eventually read by consumers. And when that happens, we got the following error:

{code}
2015-03-18 09:46:30 JmxReporter [WARN] Error processing 
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=samza_checkpoint_manager-wikipedia_parser-1-1426697189628-0,brokerHost=fe80:0:0:0:7ed1:c3ff:fee0:c60f%4,brokerPort=9092
javax.management.MalformedObjectNameException: Invalid character ':' in value 
part of property
at javax.management.ObjectName.construct(ObjectName.java:618)
at javax.management.ObjectName.init(ObjectName.java:1382)
at com.yammer.metrics.reporting.JmxReporter.onMetricAdded(JmxReporter.java:395)
at 
com.yammer.metrics.core.MetricsRegistry.notifyMetricAdded(MetricsRegistry.java:516)
at com.yammer.metrics.core.MetricsRegistry.getOrAdd(MetricsRegistry.java:491)
at com.yammer.metrics.core.MetricsRegistry.newTimer(MetricsRegistry.java:320)
at kafka.metrics.KafkaMetricsGroup$class.newTimer(KafkaMetricsGroup.scala:85)
at 
kafka.consumer.FetchRequestAndResponseMetrics.newTimer(FetchRequestAndResponseStats.scala:26)
at 
kafka.consumer.FetchRequestAndResponseMetrics.init(FetchRequestAndResponseStats.scala:35)
at 
kafka.consumer.FetchRequestAndResponseStats$$anonfun$1.apply(FetchRequestAndResponseStats.scala:44)
at 
kafka.consumer.FetchRequestAndResponseStats$$anonfun$1.apply(FetchRequestAndResponseStats.scala:44)
at kafka.utils.Pool.getAndMaybePut(Pool.scala:61)
at 
kafka.consumer.FetchRequestAndResponseStats.getFetchRequestAndResponseStats(FetchRequestAndResponseStats.scala:51)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108)
at 
org.apache.samza.checkpoint.kafka.KafkaCheckpointManager$$anonfun$readLog$1.apply(KafkaCheckpointManager.scala:283)
at 
org.apache.samza.checkpoint.kafka.KafkaCheckpointManager$$anonfun$readLog$1.apply(KafkaCheckpointManager.scala:256)
at 
org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:82)
at 
org.apache.samza.checkpoint.kafka.KafkaCheckpointManager.readLog(KafkaCheckpointManager.scala:255)
at 
org.apache.samza.checkpoint.kafka.KafkaCheckpointManager.readChangeLogPartitionMapping(KafkaCheckpointManager.scala:242)
at 
org.apache.samza.coordinator.JobCoordinator$.buildJobModel(JobCoordinator.scala:136)
at 
org.apache.samza.coordinator.JobCoordinator.buildJobModel(JobCoordinator.scala)
at 
org.apache.samza.job.standalone.controller.StandaloneZkCoordinatorController.refreshOwnership(StandaloneZkCoordinatorController.java:161)
at 
org.apache.samza.job.standalone.controller.StandaloneZkCoordinatorController.access$900(StandaloneZkCoordinatorController.java:49)
at 
org.apache.samza.job.standalone.controller.StandaloneZkCoordinatorController$ContainerPathListener.handleChildChange(StandaloneZkCoordinatorController.java:256)
at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
{code}

I think the right solution here is that BrokerHost string should also be 
canonized before adding to sensor tags. [~vladimir.tretyakov] [~junrao] what do 
you think?

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Assignee: Vladimir Tretyakov
Priority: Critical
  Labels: patch
 Fix For: 0.8.2.0, 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch,

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2015-01-09 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272057#comment-14272057
 ] 

Jun Rao commented on KAFKA-1481:


Also committed the doc change to 0.8.2 documentation.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Assignee: Vladimir Tretyakov
Priority: Critical
  Labels: patch
 Fix For: 0.8.2, 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-17_14-33-03.patch, 
 KAFKA-1481_2014-11-19_16-03-03_trunk.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, logflushes.png, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-12-24 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258109#comment-14258109
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi Jun, thx for answer, it helps (log.flush.interval.messages property), I 
see data on charts
!https://issues.apache.org/jira/secure/attachment/12689014/logFlushes.png!

Maybe you also have a receipt how to force displaying of these metrics 
kafka.log.LogCleaner:
{code}
newGauge(max-buffer-utilization-percent, 
   new Gauge[Int] {
 def value: Int = cleaners.map(_.lastStats).map(100 * 
_.bufferUtilization).max.toInt
   })
  /* a metric to track the recopy rate of each thread's last cleaning */
  newGauge(cleaner-recopy-percent, 
   new Gauge[Int] {
 def value: Int = {
   val stats = cleaners.map(_.lastStats)
   val recopyRate = stats.map(_.bytesWritten).sum.toDouble / 
math.max(stats.map(_.bytesRead).sum, 1)
   (100 * recopyRate).toInt
 }
   })
  /* a metric to track the maximum cleaning time for the last cleaning from 
each thread */
  newGauge(max-clean-time-secs,
   new Gauge[Int] {
 def value: Int = 
cleaners.map(_.lastStats).map(_.elapsedSecs).max.toInt
   })
{code}

?

Thx again.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Assignee: Vladimir Tretyakov
Priority: Critical
  Labels: patch
 Fix For: 0.8.2, 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-17_14-33-03.patch, 
 KAFKA-1481_2014-11-19_16-03-03_trunk.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, logFlushes.png, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-12-22 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255722#comment-14255722
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi guys, maybe it is not a best place for this question, but it is related to 
metric, so I will try:

In code ( kafka/core/src/main/scala/kafka/log/FileMessageSet.scala ) I see:
 {code}
object LogFlushStats extends KafkaMetricsGroup {
  val logFlushTimer = new KafkaTimer(newTimer(LogFlushRateAndTimeMs, 
TimeUnit.MILLISECONDS, TimeUnit.SECONDS))
}
 {code}

But I don't see this metric in JMX when I attach JConsole to my broker.

Is it ok?
What I have to do to see this metric?
Maybe this bean/metric exists short time and recreates each time after/during 
log flushes?
Can somebody explain me?

Best regards. Vladimir.


 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Assignee: Vladimir Tretyakov
Priority: Critical
  Labels: patch
 Fix For: 0.8.2, 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-17_14-33-03.patch, 
 KAFKA-1481_2014-11-19_16-03-03_trunk.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-12-22 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255845#comment-14255845
 ] 

Jun Rao commented on KAFKA-1481:


The JMX will be registered the first time that the flush stat is recorded. So, 
try producing some messages to a topic.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Assignee: Vladimir Tretyakov
Priority: Critical
  Labels: patch
 Fix For: 0.8.2, 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-17_14-33-03.patch, 
 KAFKA-1481_2014-11-19_16-03-03_trunk.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-12-22 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255875#comment-14255875
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Already did, waited more than 1 day (on our test env), nothing, no bean with 
such name is in broker process.


 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Assignee: Vladimir Tretyakov
Priority: Critical
  Labels: patch
 Fix For: 0.8.2, 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-17_14-33-03.patch, 
 KAFKA-1481_2014-11-19_16-03-03_trunk.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-12-22 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255956#comment-14255956
 ] 

Jun Rao commented on KAFKA-1481:


Ok, the metric does show up. By default, the flush is only done in a background 
thread and it only flushes old log segments. So, a flush is only called when a 
new log segment is rolled. If you want to see the metric quicker, you can try 
setting log.flush.interval.messages to a small value.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Assignee: Vladimir Tretyakov
Priority: Critical
  Labels: patch
 Fix For: 0.8.2, 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-17_14-33-03.patch, 
 KAFKA-1481_2014-11-19_16-03-03_trunk.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-17 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14214559#comment-14214559
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi, provided new patch, last one?:)


80.1 Done
80.2 Done
80.3 Done
80.4 Do you mean 'new producer'? Can I just reformulate KAFKA-1768? Something 
like: Expose version via JMX in new client?
81. Done
82. Done
83. Left as is.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-17_14-33-03.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-17 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215304#comment-14215304
 ] 

Jun Rao commented on KAFKA-1481:


Thanks for the patch. +1 from me. Just a few minor comments from me.

90. Kafka: No need to call AppInfo.registerInfo().

91. ZookeeperConsumerConnector: Could we rename the following method to 
ownedPartitionsCountMetricTags?
def ownedPartitionsCountMetricName

Could you also provide a patch for trunk?

[~jjkoshy], do you want to look at the patch again?

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-17_14-33-03.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-17 Thread Joel Koshy (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215371#comment-14215371
 ] 

Joel Koshy commented on KAFKA-1481:
---

+1  (for 0.8.2)

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-17_14-33-03.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-17 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14215645#comment-14215645
 ] 

Jun Rao commented on KAFKA-1481:


[~vladimir.tretyakov], thanks a lot for your work. Committed to 0.8.2 after 
fixing 90 and 91.

Will leave the jira open until trunk is patched too.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-17_14-33-03.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-14 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212264#comment-14212264
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi, added new patches (code + doc), go by way 3:
(3) kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec


Also added Kafka version MBean, it exposes only Kafka version now (from 
gradle.properties file). I didn't find easy way where I can get build hash, so 
only version for now.

I hope it will be my last patches, it is a time consumption to change things 
many times and test everything each time and prepare patched, so I really hope 
these patches are good enough and I will not do additional iterations, thx.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-14 Thread Jun Rao (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212536#comment-14212536
]

Jun Rao commented on KAFKA-1481:

Thanks for the patch. Appreciate your persistence. A few comments below.

80. AppInfo.registerInfo()
80.1 On the server side, this needs to be called in
KafkaServerStartable.startup(). Some users will start up a Kafka broker using
KafkaServerStartable in a container and not from the command line.
80.2 On the client side, if there are multiple instances of clients running in
the same jvm, registerInfo() will be called multiple times. It would be good if
we make sure registerInfo() only register the mbean once no matter how many
times it's called. We can maintain an isRegistered flag internally and only
register the mbean if the flag is not set. We can also make this a synchronized
method.
80.3 There is no need to call registerInfo() in ConsoleConsumer and
ProducerPerformance since the mbean will be registered by the consumer/producer
client.
80.4 We will need to add the same version mbean in the new java client. We
don't need to do that in this jira. Could you file a separate jira to track
that?

81. KafkaServer: remove unused import AppInfo

82. TestUtils: Could you fix the indentation in the following?
def sendMessagesToPartition(configs: Seq[KafkaConfig],
topic: String,
partition: Int,
numMessages: Int,
compression: CompressionCodec =
NoCompressionCodec): List[String] = {

83. As I was reviewing KAFKA-1684, I realized that in the future, a broker may
have multiple ports: plain text, SSL, SASL, etc. In this patch, the
broker-specific mbeans have the tag of brokerHost and brokerPort. This is going
to be inconvenient once the broker has more than one port. I was thinking it's
simpler if we just add the brokerId tag or both the brokerId and the brokerHost
tag. What do you think?

Stop using dashes AND underscores as separators in MBean names
--

Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
Labels: patch
Fix For: 0.8.3

Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch,
KAFKA-1481_2014-10-31_14-35-43.patch,
KAFKA-1481_2014-11-03_16-39-41_doc.patch,
KAFKA-1481_2014-11-03_17-02-23.patch,
KAFKA-1481_2014-11-10_20-39-41_doc.patch,
KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch,
KAFKA-1481_2014-11-14_16-39-41_doc.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png,
alternateLayout2.png, diff-for-alternate-layout1.patch,
diff-for-alternate-layout2.patch, originalLayout.png

MBeans should not use dashes or underscores as separators because these
characters are allowed in hostnames, topics, group and consumer IDs, etc.,
and these are embedded in MBeans names making it impossible to parse out
individual bits from MBeans.
Perhaps a pipe character should be used to avoid the conflict.
This looks like a major blocker because it means nobody can write Kafka 0.8.x
monitoring tools unless they are doing it for themselves AND do not use
dashes AND do not use underscores.
See: http://search-hadoop.com/m/4TaT4lonIW

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-14 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212563#comment-14212563
 ] 

Jun Rao commented on KAFKA-1481:


83. On another thought, the port tag may be ok since a client is only going to 
connect to one port any way.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-14 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212598#comment-14212598
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Thx Jun, will try to fix everything according your last comments.
re 83, yeah host:port is unique pair, so it will work even with KAFKA-1684

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-14 Thread Joel Koshy (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212873#comment-14212873
 ] 

Joel Koshy commented on KAFKA-1481:
---

[~junrao] For 80.2, I believe the additional registration will not create any 
new mbeans. i.e., it should be a no-op.
For 83, we could have one set of mbeans per port right? Or do you think that 
would be too much? Your suggestion is to drop the port and just unify right? 
That should also be good.

Also, [~vladimir.tretyakov] your patch needs a rebase as mentioned earlier.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-14 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212987#comment-14212987
 ] 

Jun Rao commented on KAFKA-1481:


[~jjkoshy], yes, re-registering an existing mbean is a no-op. However, it would 
probably be good not to depend on this and to avoid the unnecessary checks on 
resources.

For 83, chances are a given application is going to use one type of port. So, 
we can leave this as it is.

The patch is actually intended for 0.8.2 and it applies.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-14 Thread Otis Gospodnetic (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14213409#comment-14213409
 ] 

Otis Gospodnetic commented on KAFKA-1481:
-

bq. 80.4 We will need to add the same version mbean in the new java client. We 
don't need to do that in this jira. Could you file a separate jira to track 
that?

I created KAFKA-1768 a few days ago and have linked it to this issue.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
 KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-13 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14210144#comment-14210144
 ] 

Jun Rao commented on KAFKA-1481:


Thinking about his a bit more, I agree with [~jjkoshy] that (3) is probably the 
best choice. It's the simplest. Also, if you look at a metric like the 
following, it should be clear that the request rate is for a particular client, 
aggregated. There are different ways that this metrics can be broken down. 
However, that information shouldn't need to be included in the aggregate 
metrics.

kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=console-consumer-50964

We can probably keep the name BrokerTopicMetrics so that it can be 
distinguished from ConsumerTopicMetrics and ProducerTopicMetrics.

[~vladimir.tretyakov], are you ok with this approach?

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-13 Thread Joel Koshy (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14211341#comment-14211341
 ] 

Joel Koshy commented on KAFKA-1481:
---

Re: BrokerTopicMetrics so that it can be distinguished from 
ConsumerTopicMetrics and ProducerTopicMetrics

Why do we need to distinguish it? i.e., BrokerTopicMetrics would only be on the 
server right? Or are you concerned about the possibility of a producer or 
consumer that happens to run in the same VM as a broker?

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-13 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14211555#comment-14211555
 ] 

Jun Rao commented on KAFKA-1481:


My concern is that in the code, if you have a class TopicMetrics, it's not 
clear what it is for (broker, consumer or producer). Also, if you have a 
centralized monitoring system that pulls all Kafka metrics (for both the client 
and the broker), TopicMetrics will be a bit out of context.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-12 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14208284#comment-14208284
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Maybe somebody can answer my last questions? Have to finish with this patch and 
moving forward! Thx.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-12 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14208497#comment-14208497
 ] 

Jun Rao commented on KAFKA-1481:



65. The following are some of the choices that we have.
(1) kafka.server:type=BrokerTopicMetrics,name=AggregateBytesOutPerSec
(2) kafka.server:type=AggregateBrokerTopicMetrics,name=BytesOutPerSec
(3) kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec
(4) kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesOutPerSec
(5) kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,allTopics=true
(6) kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topics=aggregate
The following is my take. The issue with (1), (2) and (3) is that it's not 
obvious which dimension is being aggregated upon. I also don't quite like (2) 
since it breaks the convention that type is the class name. If we do go with 
this route, I'd prefer that we explicitly create an AggregateBrokerTopicMetrics 
class instead of sneaking in the prefix in KafkaMetricsGroup. (4), (5) and (6) 
will all make it clear which dimension is being aggregated upon. (4) is a bit 
weird now that we support tags since the main purpose of tags is that we don't 
have to squeeze everything into a single name. So, either (5) and (6) looks 
reasonable to me. Also, I am not sure how jconsole displays mbeans, but the 
key/value pairs in the mbean name are supposed to be unordered.

[~jjkoshy], what's your take?

As for the mbean for the Kafka version, could we do that in a separate jira? 
The approach seems reasonable.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-12 Thread Joel Koshy (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209055#comment-14209055
 ] 

Joel Koshy commented on KAFKA-1481:
---

(1) and (4) seem equivalent (i.e., AllTopics vs Aggregate) - or are you saying 
that (4) will be AllTopics or AllBrokers as appropriate?

I'm +0 on (5) for the reason I stated above. i.e., it is odd to see true when 
browsing mbeans

I'm +0 on (6) as well as topics=aggregate is a bit odd. The field name 
suggests it is a list of topics but it is more like a boolean. Between this and 
(5) I prefer (5).

(3) seems reasonable to me although it is not as clear as having an explicit 
aggregate term in the type. However, I think (1), (2) and (3) do make it clear 
enough what is being aggregated: i.e., bytes-out-per-sec aggregated on topic. I 
actually think Broker should not be there since this is a broker-side mbean 
already. i.e., if we had kafka.server:type=TopicMetrics,name=BytesOutPerSec 
(wouldn't it be clear that the dimension of aggregation is across topics?) 
i.e., I think we can just make the dimension clear from the typename.

Likewise, it should be clear (for consumers) that 
FetchRequestAndResponseMetrics is really a broker-level aggregation.


 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-11 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206303#comment-14206303
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi [~junrao], 
{quote}
Vladimir Tretyakov, we could probably just add a new MBean to expose the Kafka 
version number. Any value in exposing other things like build hash and build 
timestamp?
{quote}
Version will be ok I think, but my question was also about where I can get this 
Version? Should I hardcoded this Version in MBean? What is the best place for 
such information? Maybe you already have some special property file?

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-11 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206333#comment-14206333
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi again, 
65. Honestly I'd prefer allBrokers=true and allTopics=true or maybe 
brokers=aggregated, topics=aggregated.
From first view user will not know what this is about:
{code}
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=console-consumer-50964
{code}

and only after user will see something like:

{code}
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=console-consumer-50964,
 broker=0
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=console-consumer-50964,
 broker=1
{code}

user can understand (not sure) that like without 'broker=..' is aggregated 
value, I'd prefer explicit definition.

[~jjkoshy] what do you think about:
brokers=aggregated, topics=aggregated ? In jconsole user will see 
aggregated instead of true, handy from my point of view.


 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
 KAFKA-1481_2014-11-10_21-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-10 Thread Jun Rao (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205937#comment-14205937
]

Jun Rao commented on KAFKA-1481:

[~vladimir.tretyakov], we could probably just add a new MBean to expose the
Kafka version number. Any value in exposing other things like build hash and
build timestamp?

Also, could you address my last few comments? For example, it seem #62 is still
not addressed. A few more comments on the new patch.

64. ConsumerTopicMetrics: Could you merge the following into a single line?
val tags = metricId
match {

65. About the change to the aggregate metric name. It seems that we now have
the following MBean name. A couple of comments on this.
kafka.consumer:type=AggregateFetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=console-consumer-50964
65.1 We only include the following in KafkaMetricsGroup.consumerMetricNameList.
That won't match the above aggregate metric's name. So, I am not sure how this
metric is removed when closing the consumer. The unit test does pass. So, I am
not sure if it's testing the right thing.
new MetricName(kafka.consumer, FetchRequestAndResponseMetrics,
FetchResponseSize),
65.1 I think adding Aggregate in front of the class name is a bit weird. The
way that we add it in KafkaMetricsGroup is also a bit hacky since it
essentially changed the typeName for aggregate metrics w/o actually changing
it. I was thinking for aggregate metrics, would it be simpler just to have the
following? The fact that it doesn't have any broker level labels is enough an
indication that it's an aggregate across all brokers.
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=console-consumer-50964

Stop using dashes AND underscores as separators in MBean names
--

Attachments: KAFKA-1481_2014-06-06_13-06-35.patch,
KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch,
KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch,
KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch,
KAFKA-1481_2014-10-31_14-35-43.patch,
KAFKA-1481_2014-11-03_16-39-41_doc.patch,
KAFKA-1481_2014-11-03_17-02-23.patch,
KAFKA-1481_2014-11-10_20-39-41_doc.patch,
KAFKA-1481_2014-11-10_21-02-23.patch,
KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch,
KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png,
alternateLayout2.png, diff-for-alternate-layout1.patch,
diff-for-alternate-layout2.patch, originalLayout.png

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-09 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204467#comment-14204467
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi [~jjkoshy] thx for your feedback, will change, one question from me:

What if I include Kafka version in any of the MBeans?  It will easy for 
external tools to detect Kafka version and use parsing rules for particular 
version. Something like Elasticsearch provides by http (Kafka will provide by 
JMX).

{code}
curl localhost:9200 
{
...
  version : {
number : 1.3.2,
build_hash : dee175dbe2f254f3f26992f5d7591939aaefd12f,
build_timestamp : 2014-08-13T14:29:30Z

  }
}
{code}

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
 alternateLayout2.png, diff-for-alternate-layout1.patch, 
 diff-for-alternate-layout2.patch, originalLayout.png


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-04 Thread Joel Koshy (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14197412#comment-14197412
 ] 

Joel Koshy commented on KAFKA-1481:
---

Sorry I ran out of time - will look tomorrow.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-03 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14194551#comment-14194551
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi, added new patch

50. I think it is better to have separate case class for 'all' things. It is 
real 'other' case.
51. Done
52.1 Done
52.2 Done
52.3 Done
52.4 Done
52.5 Done
53. 
https://issues.apache.org/jira/secure/attachment/12678927/KAFKA-1481_2014-11-03_16-39-41_doc.patch

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-03 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195548#comment-14195548
 ] 

Jun Rao commented on KAFKA-1481:


[~jjkoshy], do you want to take another look at the patch?

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-03 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195545#comment-14195545
 ] 

Jun Rao commented on KAFKA-1481:


Thanks for the patch. Just some minor comments below. Otherwise, +1 from me.

60. AbstractFetcherManager: In the following, we don't need to wrap new Gauge 
in {}.
  MinFetchRate, {
new Gauge[Double] {
  // current min fetch rate across all fetchers/topics/partitions
  def value = {
val headRate: Double =
  
fetcherThreadMap.headOption.map(_._2.fetcherStats.requestRate.oneMinuteRate).getOrElse(0)

fetcherThreadMap.foldLeft(headRate)((curMinAll, fetcherThreadMapEntry) 
= {
  
fetcherThreadMapEntry._2.fetcherStats.requestRate.oneMinuteRate.min(curMinAll)
})
  }
}
  },

61. AbstractFetcherThread: Could we align topic and partition to the same 
column as clientId? Ditto in a few other places.
Map(clientId - metricId.clientId,
  topic - metricId.topic,
  partition - metricId.partitionId.toString)
  )

62. FetchRequestAndResponseMetrics: In the following, could we put Map in a 
separate line?
  val tags = metricId match {
case ClientIdAndBroker(clientId, brokerHost, brokerPort) = Map(clientId 
- clientId, brokerHost - brokerHost,
  brokerPort - brokerPort.toString)
case ClientIdAllBrokers(clientId) = Map(clientId - clientId, 
allBrokers - true)
  }

62. TestUtils: It's a bit weird that sendMessagesToBrokerPartition() has both a 
config and configs. We actually don't need the config any more. When sending a 
message to a partition, we actually don't know which broker the partition is 
on. So, the message can just be of the form test + - + partition + - + x. 
We can also rename the method to sendMessagesToPartition().


 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-03 Thread Joel Koshy (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195573#comment-14195573
 ] 

Joel Koshy commented on KAFKA-1481:
---

Yes thanks - I'll take a look at this tomorrow.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
 KAFKA-1481_2014-11-03_17-02-23.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-31 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191698#comment-14191698
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi Jun, added new one, changed according to your last comments.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-31 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192034#comment-14192034
 ] 

Jun Rao commented on KAFKA-1481:


Thanks for the patch. A few more comments.

50. ClientIdBroker: Instead of having 2 subclasses, would it be better to have 
just one class
  case class ClientIdAndBroker(clientId: String, brokerHost: Option[String], 
brokerPort: Option[Int])?
Ditto to ClientIdTopic.

51. TopicPartitionRequestKey: Can this just be TopicAndPartition?

52. MetricsTest:
52.1 Could we remove the extra empty lines after the class?
52.2 remove unnecessary {} in the following (a few other files have a similar 
issue)
import java.util.{Properties}
52.3 In testMetricsLeak(), you don't need to create a new zkClient. You can get 
one from the base class in ZooKeeperTestHarness.
52.4 Instead of duplicating the createAndShutdownStep() calls, could we use a 
loop instead?
52.5 Instead of duplicating sendMessages() and getMessages() from 
ZookeeperConsumerConnectorTest, could we extract those methods to TestUtils and 
add comments to describe what they do? Then, we can reuse those methods.

53. Could you also include a patch to the 0.8.2 doc 
(https://svn.apache.org/repos/asf/kafka/site/082/ops.html) with the metric name 
changes?






 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_2014-10-31_14-35-43.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-30 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14190581#comment-14190581
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi Jun, added new patch, removed Taggable, did everything as you have suggested.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, 
 KAFKA-1481_2014-10-24_14-14-35.patch.patch, 
 KAFKA-1481_2014-10-30_21-35-43.patch.patch.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-30 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191115#comment-14191115
 ] 

Jun Rao commented on KAFKA-1481:


Thanks for the patch. Looks good overall. Some minor comments below.

40. KafkaMetricsGroup:
40.1 Since we already match the metric name directly, shouldn't the following 
pattern
   val pattern = (.* + metric.getName + .* + clientId + .*).r
   be
   val pattern = (.*clientId= + clientId + .*).r
40.2 Can we make toMBeanName() private?
40.3 According to 
http://docs.oracle.com/javase/7/docs/api/javax/management/ObjectName.html,  the 
key properties in an Mbean objectname are a set of unordered keys. So, we don't 
need the toMap() method. The caller can just create the Map using 
Map(tag1-val1, tag1-va2). 

41. FetchRequestAndResponseMetrics: The following two lines should be in the 
same line.
  val tags = metricId
  match {

42. Log, Partition,AbstractFetcherThread,ProducerRequestPurgatory: 
partitionId = partition

43. TopicPartitionRequestKey: We don't need keyLabel() any more. Instead, we 
can override the toString() method to get the string in the same way as that in 
TopicAndPartition.toString().

44. RequestChannel: Lower case processor in the following.
  toMap(Processor - i.toString)

45. SocketServer: NetworkProcessor =  networkProcessor

46. ZookeeperConsumerConnector: Could we put the new Gauge and the toMap() into 
a separate line?
newGauge(OwnedPartitionsCount, new Gauge[Int] {
  def value() = allTopicsOwnedPartitionsCount
}, toMap(clientId - config.clientId, groupId - config.groupId, 
allTopics - true))

  newGauge(OwnedPartitionsCount, new Gauge[Int] {
def value() = partitionThreadPairs.size
  }, ownedPartitionsCountMetricName(topic))

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-29 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188553#comment-14188553
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi Jun, thx for juicy feedback again, one question:
{quote}
31. AbstractFetcherThread:
31.1 You changed the meaning of clientId. clientId is used in the fetch request 
and we want to leave it as just the clientId string. Since the clientId should 
be uniquely representing a particular consumer client, we just need to include 
the clientId in the metric name. We don't need to include the consumer id in 
either the fetch request or the metric name since it's too long and has 
redundant info. 
{quote}

I didn't change meaning of clientId here, look (all code without my changes):

consumerIdString string is:
{code}
val consumerIdString = {
var consumerUuid : String = null
config.consumerId match {
  case Some(consumerId) // for testing only
  = consumerUuid = consumerId
  case None // generate unique consumerId automatically
  = val uuid = UUID.randomUUID()
  consumerUuid = %s-%d-%s.format(
InetAddress.getLocalHost.getHostName, System.currentTimeMillis,
uuid.getMostSignificantBits().toHexString.substring(0,8))
}
config.groupId + _ + consumerUuid
  }
{code}

thread name is (consumerIdString + fetcherId + sourceBroker.id):
{code}
 override def createFetcherThread(fetcherId: Int, sourceBroker: Broker): 
AbstractFetcherThread = {
new ConsumerFetcherThread(
  ConsumerFetcherThread-%s-%d-%d.format(consumerIdString, fetcherId, 
sourceBroker.id),
  config, sourceBroker, partitionMap, this)
  }
{code}

clientId inside AbstractFetcherThread is: config.clientId + consumerIdString + 
fetcherId + sourceBroker.id
{code}
class ConsumerFetcherThread(name: String,
val config: ConsumerConfig,
sourceBroker: Broker,
partitionMap: Map[TopicAndPartition, 
PartitionTopicInfo],
val consumerFetcherManager: ConsumerFetcherManager)
extends AbstractFetcherThread(name = name, 
  clientId = config.clientId + - + name,
  sourceBroker = sourceBroker,
  socketTimeout = config.socketTimeoutMs,
  socketBufferSize = 
config.socketReceiveBufferBytes,
  fetchSize = config.fetchMessageMaxBytes,
  fetcherBrokerId = 
Request.OrdinaryConsumerId,
  maxWait = config.fetchWaitMaxMs,
  minBytes = config.fetchMinBytes,
  isInterruptible = true) {
{code}

As you see there is no clean clientId inside AbstractFetcherThread class and it 
is not unique situation, and this is main goal why I added Taggable.
Now I am trying to remove Taggable, but I have no idea what to do with such 
cases I've described. Can I add new 'clientId' parameter to all these classes 
and use only this new/clean clientId as part of matric name? Any other 
suggestions? Thx.


 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, 
 KAFKA-1481_2014-10-24_14-14-35.patch.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-27 Thread Jun Rao (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14186124#comment-14186124
]

Jun Rao commented on KAFKA-1481:

Vladimir,

Thanks for the patch. Really appreciate your help. I realized that this is one
of the biggest technical debt that we have accumulated over time. So, it may
take some time to sort this out. So, bear with me. Some more comments.

30. About Taggable, I still have mixed feelings. I can see why you created it.
However, my reasoning is that for a lot of the case classes (ClientIdTopic,
CliendIdAndBroker) that we create, it's weird that some of them are taggable
and some of them are not, depending whether they are used for tagging metric
names or not. Those classes have no direct relationships with the metrics.
Similarly, we only need to be aware of tags when creating metrics. Also,
because of this, we change the constructor of SimpleConsumer. Since this is an
API change, we should really try to avoid it.

My feeling is that it's probably simpler if we just create regular case classes
as before and generate metric tags explicitly when we create the metric. For
example, in AbstractFetcherThread, we can do

class FetcherStats(clientIdAndBroker: ClientIdAndBroker) extends
KafkaMetricsGroup {
val requestRate = newMeter(RequestsPerSec, requests, TimeUnit.SECONDS,
Map(cliendId -
clientIdAndBroker.clientId,
brokerHost -
clientIdAndBroker.host,
brokerPort -
clientIdAndBroker.port))

and just have ClientIdAndBroker be the following case class.

case class ClientIdAndBroker(clientId: String, host: String, port: Int)

This way, the code is a bit cleaner since all the metric tag related stuff are
isolated to those places when the metrics are created. So, I'd suggest that we
remove Taggable.

31. AbstractFetcherThread:
31.1 You changed the meaning of clientId. clientId is used in the fetch request
and we want to leave it as just the clientId string. Since the clientId should
be uniquely representing a particular consumer client, we just need to include
the clientId in the metric name. We don't need to include the consumer id in
either the fetch request or the metric name since it's too long and has
redundant info.
31.2 FetcherLagStats: This is an existing problem. FetcherLagMetrics shouldn't
be keyed off ClientIdBrokerTopicPartition. It should be keyed off
ClientIdTopicPartition. This way, the metric name remains the same independent
of the current leader of those partitions.

32. ZookeeperConsumerConnector:
32.1 FetchQueueSize: I agree that the metric name just needs to be tagged with
clientId, topic and threadId. We don't need to include the consumerId since
it's too long (note that topicThread._2 includes both the consumerId and the
threadId).

33. KafkaMetricsGroup: Duplicate entries.
// kafka.consumer.ConsumerTopicStats -- kafka.consumer.{ConsumerIterator,
PartitionTopicInfo}
explicitMetricName(kafka.consumer, ConsumerTopicMetrics,
MessagesPerSec),
explicitMetricName(kafka.consumer, ConsumerTopicMetrics,
MessagesPerSec),

// kafka.consumer.ConsumerTopicStats
explicitMetricName(kafka.consumer, ConsumerTopicMetrics, BytesPerSec),
explicitMetricName(kafka.consumer, ConsumerTopicMetrics, BytesPerSec),

// kafka.consumer.FetchRequestAndResponseStats --
kafka.consumer.SimpleConsumer
explicitMetricName(kafka.consumer, FetchRequestAndResponseMetrics,
FetchResponseSize),
explicitMetricName(kafka.consumer, FetchRequestAndResponseMetrics,
FetchRequestRateAndTimeMs),
explicitMetricName(kafka.consumer, FetchRequestAndResponseMetrics,
FetchResponseSize),
explicitMetricName(kafka.consumer, FetchRequestAndResponseMetrics,
FetchRequestRateAndTimeMs),

/**
* ProducerRequestStats -- SyncProducer
* metric for SyncProducer in fetchTopicMetaData() needs to be removed when
consumer is closed.
*/
explicitMetricName(kafka.producer, ProducerRequestMetrics,
ProducerRequestRateAndTimeMs),
explicitMetricName(kafka.producer, ProducerRequestMetrics,
ProducerRequestSize),
explicitMetricName(kafka.producer, ProducerRequestMetrics,
ProducerRequestRateAndTimeMs),
explicitMetricName(kafka.producer, ProducerRequestMetrics,
ProducerRequestSize)

34. AbstractFetcherManager: Could you put the followings in 2 separate lines?
Similar things happen in a few other files. Perhaps you need to change the
formatting in your IDE?

}, metricPrefix.toTags

private def getFetcherId(topic: String, partitionId: Int) : Int = {
Utils.abs(31 * topic.hashCode() + partitionId) % numFetchers

Stop using dashes AND underscores as separators in MBean names
--

Key:

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-24 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182670#comment-14182670
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi Jun, thx for your feedback again, attached new patch.

20.1 done.
20.2 clientId is Taggable here, it will convert to something like 
clientId=XXX automatically
21 tags should be Map which iterators iterate in the same order elements were 
inserted (final string must be stable): from LinkedHashMap docs The iterator 
and all traversal methods of this class visit elements in the order they were 
inserted.. I am new in scala, if you know better candidate with predictable 
iterators please let me know
22.1 done
22.2 clientId is not always just clientId=XXX, look at:
{code}
class ConsumerFetcherThread(consumerFetcherThreadId: ConsumerFetcherThreadId,
val config: ConsumerConfig,
sourceBroker: Broker,
partitionMap: Map[TopicAndPartition, 
PartitionTopicInfo],
val consumerFetcherManager: ConsumerFetcherManager)
extends AbstractFetcherThread(name = consumerFetcherThreadId,
  clientId = new TaggableInfo(new 
TaggableInfo(clientId - config.clientId), consumerFetcherThreadId),  
  sourceBroker = sourceBroker,
  socketTimeout = config.socketTimeoutMs,
  socketBufferSize = 
config.socketReceiveBufferBytes,
  fetchSize = config.fetchMessageMaxBytes,
  fetcherBrokerId = 
Request.OrdinaryConsumerId,
  maxWait = config.fetchWaitMaxMs,
  minBytes = config.fetchMinBytes,
  isInterruptible = true) {
{code}


this clientId = new TaggableInfo(new TaggableInfo(clientId - 
config.clientId), consumerFetcherThreadId) part was in code before my changes 
(was manipulation with strings, not Taggable of course). Yeah, somewhere in 
code we can use string instead of Taggable, but maybe it is better to has 
Taggable everywhere what is related to metrics. At least it was not easy to 
me decide where I have to use Taggable, but where I can leave string. I'd 
prefer Taggable everywhere in this case, sorry:).

23. done everything as described by link, checked how it works/applies on 
'origin/0.8.2' from scratch.
24. see 22.2
25. agree, added second constructor with 'String' as clientId, can't remove 
constructor with Taggable as clientId because as you can see in 22.2 
sometimes clientId is not just clientId but something more complex.
26. added 'MetricsLeakTest', basic idea is start/stop many producers/consumes 
and observe count of metrics in Metrics.defaultRegistry(), count should not 
grow.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, 
 KAFKA-1481_2014-10-24_14-14-35.patch.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-22 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180056#comment-14180056
 ] 

Jun Rao commented on KAFKA-1481:


Otis,

I expect that we will have about another 4 weeks before 0.8.2 final is 
released. That should give us enough time to iterate on this, right? Since the 
patch touches many files, I'd prefer that we get a clean version upfront.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-21 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178039#comment-14178039
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi, added last patch (KAFKA-1481_2014-10-21_09-14-35.patch). Worked for me 
locally (previous worked too:( ).
There are commands I've used:
{code}
PATCH CREATION
wawanawna@wawanawna:/home/storage/sematext/src/kfk2/kafka$ git status
On branch 0.8.2
Your branch is up-to-date with 'origin/0.8.2'.

Changes to be committed:
  (use git reset HEAD file... to unstage)

new file:   core/src/main/scala/kafka/common/BrokerInfo.scala
new file:   core/src/main/scala/kafka/common/ClientIdTopic.scala
new file:   core/src/main/scala/kafka/common/Taggable.scala
renamed:core/src/main/scala/kafka/common/ClientIdAndTopic.scala - 
core/src/main/scala/kafka/common/TopicInfo.scala
new file:   
core/src/main/scala/kafka/consumer/ConsumerFetcherThreadId.scala
new file:   core/src/main/scala/kafka/consumer/ConsumerId.scala

Changes not staged for commit:
  (use git add file... to update what will be committed)
  (use git checkout -- file... to discard changes in working directory)

modified:   core/src/main/scala/kafka/admin/AdminUtils.scala
modified:   
core/src/main/scala/kafka/api/HeartbeatRequestAndHeader.scala
modified:   
core/src/main/scala/kafka/api/HeartbeatResponseAndHeader.scala
modified:   
core/src/main/scala/kafka/api/JoinGroupRequestAndHeader.scala
modified:   
core/src/main/scala/kafka/api/JoinGroupResponseAndHeader.scala
modified:   core/src/main/scala/kafka/api/RequestKeys.scala
modified:   core/src/main/scala/kafka/cluster/Partition.scala
...
...
...
wawanawna@wawanawna:/home/storage/sematext/src/kfk2/kafka$ git commit -a -m 
'kafka-1481; JMX renaming'
[0.8.2 a232af6] kafka-1481; JMX renaming
 73 files changed, 666 insertions(+), 435 deletions(-)
 create mode 100644 core/src/main/scala/kafka/common/BrokerInfo.scala
 create mode 100644 core/src/main/scala/kafka/common/ClientIdTopic.scala
 create mode 100644 core/src/main/scala/kafka/common/Taggable.scala
 rename core/src/main/scala/kafka/common/{ClientIdAndTopic.scala = 
TopicInfo.scala} (68%)
 create mode 100644 
core/src/main/scala/kafka/consumer/ConsumerFetcherThreadId.scala
 create mode 100644 core/src/main/scala/kafka/consumer/ConsumerId.scala
wawanawna@wawanawna:/home/storage/sematext/src/kfk2/kafka$ git log | head -n 20
commit a232af63b8a1f43054994a74520b31ef7b9b347c
Author: wawanawna sema...@mail.com
Date:   Tue Oct 21 09:29:31 2014 +0300

kafka-1481; JMX renaming

commit eb7ac9eb7eebc4e0655b65e07cae594e61a6c05e
Author: Jun Rao jun...@gmail.com
Date:   Mon Oct 20 11:09:31 2014 -0700

kafka-1717; remove netty dependency through ZK 3.4.x; patched by Jun Rao; 
reviewed by Sriharsha Chintalapani and Neha Narkhede


wawanawna@wawanawna:/home/storage/sematext/src/kfk2/kafka$ git diff 
eb7ac9eb7eebc4e0655b65e07cae594e61a6c05e 
a232af63b8a1f43054994a74520b31ef7b9b347c  
/tmp/KAFKA-1481_2014-10-21_09-14-35.patch

PATCH APPLYING

wawanawna@wawanawna$ git clone https://github.com/apache/kafka.git


wawanawna@wawanawna:/home/storage/sematext/src/kfk3/kafka$ git checkout -b 
0.8.2 origin/0.8.2
Branch 0.8.2 set up to track remote branch 0.8.2 from origin.
Switched to a new branch '0.8.2'


wawanawna@wawanawna:/home/storage/sematext/src/kfk3/kafka$ git apply 
/tmp/KAFKA-1481_2014-10-21_09-14-35.patch
/tmp/KAFKA-1481_2014-10-21_09-14-35.patch:1410: trailing whitespace.
   })  
warning: 1 line adds whitespace errors.
{code}

There is warning, but everything works (compilation/tests passed without errors)

PS: please tell me what I am doing wrong.
wawanawna@wawanawna:/home/storage/sematext/src/kfk3/kafka$ git --version
git version 1.9.1


 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc.,

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-21 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178739#comment-14178739
 ] 

Jun Rao commented on KAFKA-1481:


Otis,

Yes, we can include this in the 0.8.2 final release. Does that sound good? 
Thanks,

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-21 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178815#comment-14178815
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi, Jun, will be perfect!

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-21 Thread Bhavesh Mistry (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179092#comment-14179092
]

Bhavesh Mistry commented on KAFKA-1481:
---

Hi [~junrao],

Can you please let me know if this will also address the [New Java Producer]
metrics() method and which client.id or topic has special chars ? So we have
consistent naming across all JMX name bean or metrics() methods ?

Here is background on this:

{code}
Bhavesh,

Yes, allowing dot in clientId and topic makes it a bit harder to define the
JMX bean names. I see a couple of solutions here.

1. Disable dot in clientId and topic names. The issue is that dot may
already be used in existing deployment.

2. We can represent the JMX bean name differently in the new producer.
Instead of
kafka.producer.myclientid:type=mytopic
we could change it to
kafka.producer:clientId=myclientid,topic=mytopic

I felt that option 2 is probably better since it doesn't affect existing
users.

Otis,

We probably can also use option 2 to address KAFKA-1481. For topic/clientid
specific metrics, we could explicitly specify the metric name so that it
contains topic=mytopic,clientid=myclientid. That seems to be a much
cleaner way than having all parts included in a single string separated by
'|'.

Thanks,

Jun
{code}

Thanks,

Bhavesh

Stop using dashes AND underscores as separators in MBean names
--

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-21 Thread Jun Rao (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179102#comment-14179102
]

Jun Rao commented on KAFKA-1481:

Thanks for the patch. Some comments below.

20. KafkaMetricsGroup:
20.1 In the following, instead of doing map(kv = ), could we do map {
case(key, value) = }?
.filter(_._2 != ).map(kv = %s=%s.format(kv._1, kv._2))
20.2 In the following, shouldn't the pattern be (.* + clientId= + clientId
+ .*).r
val pattern = (.* + clientId + .*).r

21. TaggableInfo: tags should be an immutable hashmap, right?
case class TaggableInfo(tags: mutable.LinkedHashMap[String, String]) extends
Taggable {

22. New files:
22.1 The license header should be at the very beginning, before the package and
import.
22.2 I am not sure if we really need to have the Taggable trait. For example,
in ConsumerTopicMetrics, we can just take the original clientIdAndTopic and
explicitly create the tag for clientId and topic when creating those metrics.
The tags are really specific to the metrics. So, we probably don't need to
expose them in places other than where the metrics are created.

23. ReplicaFetcherManager: The first statement in shutdown() is on the same
line as the method. A similar issue happens in a few other classes like
RequestChannel and ConsumerFetcherManager. Perhaps you can follow the patch
creation process in
https://cwiki.apache.org/confluence/display/KAFKA/Patch+submission+and+review#Patchsubmissionandreview-Simplecontributorworkflow

24.
FetchRequestAndResponseStatsRegistry.removeConsumerFetchRequestAndResponseStats():
Should we use clientId= + clientId in pattern?

25. SimpleConsumer: Ideally, we shouldn't change the constructor since this
will be an api change.

26. Could you add a unit test to make sure that after a producer/consumer is
closed, all metrics specific to the producer/consumer are removed?

Stop using dashes AND underscores as separators in MBean names
--

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-21 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179132#comment-14179132
 ] 

Jun Rao commented on KAFKA-1481:


Bhavesh,

Created a subtask to track the metric name in the new producer. Thanks,

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.3

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_2014-10-21_09-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-20 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177149#comment-14177149
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi, fuf:), created new patch (for 0.8.2 branch).

re 2.1, 2.3 - didn't change this part because we use this part in our 
monitoring, it is more handy for user to see something like 'localhost:9092' 
instead of '2 (where 2 is broker id)'. For us it is easiest to see/have this 
information directly in mBean.

With this patch mBeanNames look like:
{code}
kafka.producer:type=ProducerTopicMetrics,name=MessagesPerSec,topic=spm_alerts_topic
kafka.producer:type=ProducerTopicMetrics,name=MessagesPerSec,allTopics=true
kafka.log:type=Log,name=LogEndOffset,topic=spm_alerts_topic,partitionId=0
kafka.consumer:type=ZookeeperConsumerConnector,name=FetchQueueSize,timestamp=1413817796508,clientId=af_servers,uuid=9f99df40,groupId=af_servers,topicThreadId=spm_topic,consumerHostName=wawanawna,threadId=0
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=af_servers,threadName=ConsumerFetcherThread,fetcherId=0,sourceBrokerId=0,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413817796508,uuid=9f99df40,brokerHost=wawanawna,brokerPort=9092
{code}

There is not everything clear for me in code, so I've tried to avoid deep 
refactoring. All tests passed for me locally.

PS: note that if 'value' part is empty (in key-value structure) we will not 
see this part in attributes.
example: key1-value1,key2-null,key3-value3 will be converted to 
key1=value1,key3=value3.







 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-20 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177250#comment-14177250
 ] 

Jun Rao commented on KAFKA-1481:


Thanks for the patch. It doesn't seem to apply. Could you rebase?

git apply -p0 ~/Downloads/KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch 
error: core/src/main/scala/kafka/common/ClientIdTopic.scala: No such file or 
directory
error: patch failed: core/src/main/scala/kafka/server/KafkaApis.scala:285
error: core/src/main/scala/kafka/server/KafkaApis.scala: patch does not apply
error: patch failed: 
core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala:74
error: core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala: 
patch does not apply
error: patch failed: 
core/src/main/scala/kafka/consumer/PartitionAssignor.scala:101
error: core/src/main/scala/kafka/consumer/PartitionAssignor.scala: patch does 
not apply
error: patch failed: 
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala:17
error: 
core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala: 
patch does not apply

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-20 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177423#comment-14177423
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi, thx for quick reply, did rebase, added new patches (1 was created by git, 1 
by IDEA IDE).

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-20 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177717#comment-14177717
 ] 

Jun Rao commented on KAFKA-1481:


It still doesn't apply.

git apply -p0 ~/Downloads/KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch 
/Users/junrao/Downloads/KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch:311: 
trailing whitespace.
   })  
error: core/src/main/scala/kafka/common/TopicInfo.scala: No such file or 
directory

Does the patch apply on a fresh checkout for you?

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-16 Thread Vladimir Tretyakov (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173425#comment-14173425
]

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi Jun, thx for juicy feedback, few comments fro me:

1-2. Did it in this way because I didn't want to change existing code a lot,
tried got what we need based on current method signatures. I've thought about
your approach with Map too, reject it because of larger changes. Now I see that
I will do it.

3. Removed 'mBeanName' param, was legacy code.
4. As I've understood from code this method must be called before shutdown in
hook, right? We have to remove only metrics related to particular clientId and
we shouldn't touch others, right?

Stop using dashes AND underscores as separators in MBean names
--

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-15 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172118#comment-14172118
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Added 2 patches (created with git diff and from IDEA IDE) with a small 
refactoring in KafkaMetricsGroup.

Jun can you please double check and change part related to metrics removing 
(KAFKA-1567) I can't easily test this code locally, thx a lot.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-15 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173157#comment-14173157
 ] 

Jun Rao commented on KAFKA-1481:


Thanks for the patch. The IDEA one applies successfully for me. However, it 
seems to cause compilation failure since in a few cases, two lines are 
incorrectly merged into a single one. Do you think you can try our patch review 
tool 
(https://cwiki.apache.org/confluence/display/KAFKA/Patch+submission+and+review)?
 Some comments.

1. KafkaMetricsGroup:
1.1 We now have 2
   new MetricName(kafka.consumer, ConsumerTopicMetrics, 
MessagesPerSec),
and 2
   new MetricName(kafka.consumer, ConsumerTopicMetrics, BytesPerSec),
1.2 Could we combine the following two methods
  def newGauge[T](name: String, mBeanName: String, metric: Gauge[T])
 
  def newGauge[T](name: String, metric: Gauge[T]) : Gauge[T]
 
  to
  def newGauge[T](name: String, metric: Gauge[T], tags: Map[String, String] = 
Map.empty)? We can then general the metric name based on name and the tags. 
This will consolidate the metric name generation in a single place. In this 
patch, we have too many places generating the key/value string for the metric 
name.

2. AbstractFetcherThread: 
2.1 name in this class should be just used for the thread name and it shouldn't 
be included in the metric name.
2.2. The way that we get the metric name in classes like ClientIdAndBroker, 
ClientIdBrokerTopicPartition and ClientIdAndTopic is not consistently. 
Sometimes, we generate the key/value in the toString(). Some other times, we 
rely on the key/value string to be passed in. It's easier to understand if 
those classes are constructed by just passing in the clientId string, the 
broker object, the topic string and the partition number, etc. We can generate 
the metric name by using what's described in 1.2.
2.3 FetcherLagStats just needs to take clientId, instead of ClientIdAndBroker 
since the per partition lag metric is unique per client id.
2.4 The metric name for a Broker can just use the brokerId, instead of its host 
and port.

3. Throttler: Do we need to add mBeanName to the constructor? It seems that 
nobody is using it at the moment.

4. KafkaMetricsGroup: We need to change the way how removeAllMetricsInList() 
works. We need to do equal comparison on group, type and name and do regex 
matching with clientId on MetricName.mBeanName.


 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
 KAFKA-1481_2014-10-15_10-23-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
 KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-14 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171371#comment-14171371
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi, Jun, 

Thx that you found a time to look at patch, I've added another regarding your 
suggestion, now in JMX you will see (hope it is what you need):
{code}
kafka.server:type=FetcherStats,name=RequestsPerSec,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092,topic=spm_topic,partitionId=0
kafka.consumer:type=ZookeeperConsumerConnector,name=OwnedPartitionsCount,clientId=af_servers,groupId=af_servers,topic=spm_topic
kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchResponseSize,clientId=af_servers,threadName=ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413306414731,uuid=4624cb0f,fetcherId=0,sourceBrokerId=0,allBrokers=true
{code}

{quote}
Also, your patch doesn't seem to apply to latest trunk.
git apply ~/Downloads/KAFKA-1481_2014-10-13_18-23-35.patch 
error: core/src/main/scala/kafka/common/ClientIdTopic.scala: No such file or 
directory
{quote}
Interesting...
1. I've worked with 0.8.2 branch, not trunk
2. ClientIdTopic.scala - I've added this file (must be in patch). 

Added 2 equal patches (created them in different way):
1. 'git diff' 
2. With help from IDEA IDE.

Try please, maybe one of them will work

Thx again.


 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-13 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169396#comment-14169396
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Added another patch which introduce metrics name like pairs (key=value), 
examples:
{code}
kafka.consumer:type=ConsumerTopicMetrics,name=clientId=af_servers,AllTopics,BytesPerSec
kafka.cluster:type=Partition,name=topic=spm_alerts_topic,partitionId=0,UnderReplicated
kafka.network:type=SocketServer,name=NetworkProcessorNum=1,IdlePercent
kafka.server:type=BrokerTopicMetrics,name=topic=spm_topic,MessagesInPerSec

kafka.server:type=FetcherStats,name=clientId=af_servers,ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413199781501,uuid=58a5cc70,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092,BytesPerSec
kafka.server:type=FetcherLagMetrics,name=clientId=af_servers,ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413199781501,uuid=58a5cc70,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092,topic=spm_topic,partitionId=0,ConsumerLag
kafka.consumer:type=FetchRequestAndResponseMetrics,name=clientId=af_servers,ConsumerFetcherThread,groupId=af_servers,consumerHostName=wawanawna,timestamp=1413199781501,uuid=58a5cc70,fetcherId=0,sourceBrokerId=0,brokerHost=wawanawna,brokerPort=9092,FetchResponseSize
{code}

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-10-13 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170275#comment-14170275
 ] 

Jun Rao commented on KAFKA-1481:


Vladimir,

Thanks for the patch. My suggestion is actually slightly different from what 
you did. Instead of using

kafka.consumer:type=ConsumerTopicMetrics,name=clientId=af_servers,AllTopics,BytesPerSec

I was suggesting

kafka.consumer:type=ConsumerTopicMetrics,clientId=af_servers,topic=AllTopics,name=BytesPerSec

This is probably the more standard mbean name.

We can do that by using the following method to create MetricName and pass in 
the mBeanName that we want.
public MetricName(String group, String type, String name, String scope, 
String mBeanName).

We also need to extend KafkaMetricsGroup by adding new helper functions that 
take a MetricName explicitly.
  def newMeter(name: MetricName, eventType: String, timeUnit: TimeUnit)

Also, your patch doesn't seem to apply to latest trunk.
git apply ~/Downloads/KAFKA-1481_2014-10-13_18-23-35.patch 
error: core/src/main/scala/kafka/common/ClientIdTopic.scala: No such file or 
directory





 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
 KAFKA-1481_2014-10-13_18-23-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-09-22 Thread Otis Gospodnetic (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143500#comment-14143500
 ] 

Otis Gospodnetic commented on KAFKA-1481:
-

This is [~junrao]'s new suggestion for dealing with the problem this issue is 
meant to address: http://search-hadoop.com/m/4TaT4aHH1S

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-09-15 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134039#comment-14134039
 ] 

Jun Rao commented on KAFKA-1481:


Ok, I am fine with taking this patch as a stop-gap solution. Since this is 
going to affect how people monitor Kafka, we probably should have a discussion 
in the mailing list.

Otis/Vladimir,
Do you want to start a discussion thread in our dev and user mailing list that 
describes the problem, the solution and the impact? If there are no serious 
concerns, we can commit the patch.

Thanks,

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Priority: Critical
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-09-11 Thread Neha Narkhede (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131073#comment-14131073
 ] 

Neha Narkhede commented on KAFKA-1481:
--

This issue has popped up enough times on the mailing list that we should pay 
attention to it and fix it. [~junrao], what plan would you suggest for fixing 
this so that the change works for everyone?

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-09-09 Thread Otis Gospodnetic (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127899#comment-14127899
 ] 

Otis Gospodnetic commented on KAFKA-1481:
-

Here is a Kafka user who said:

{quote}
 I have topic name with “.”  So, it is hard to distinguish metric name and topic
{quote}
c.f. http://search-hadoop.com/m/4TaT4IRD0t1

That's the sort of problem this patch solves.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-09-04 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122025#comment-14122025
 ] 

Guozhang Wang commented on KAFKA-1481:
--

[~junrao] do you think we can check in this ticket in time for 0.8.2?

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-08-28 Thread Otis Gospodnetic (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113641#comment-14113641
]

Otis Gospodnetic commented on KAFKA-1481:
-

bq. if you know the clientId, you can parse the rest of the string.

I think this makes parsers unnecessarily difficult to write and probably harder
to maintain. Parsers have to remember things this way.

bq. If this patch is going to cause existing users that are not having a
problem issues

I think it won't cause them issues. I *think* they would just need to change
which delimiter character they use to break MBean names into parts.

bq. If it is causing existing users issues by not patching it

I think it is causing issues to anyone who has any of the multiple characters
Kafka uses as delimiters in their hostnames or topic names. So I think these
people have issues when they first try parsing stuff from JMX, and then they
have to go modify their hostnames, topic names and such.

Stop using dashes AND underscores as separators in MBean names
--

Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Labels: patch
Fix For: 0.8.2

Attachments: KAFKA-1481_2014-06-06_13-06-35.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-08-12 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093822#comment-14093822
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi Jun Rao, I understood what you suggest, thx, but I am afraid it will not 
work everywhere or I've missed something. Look at example from my local env:

1) 
kafka.producer:type=ProducerRequestMetrics,name=af-servers-AllBrokersProducerRequestRateAndTimeMs

2) 
kafka.consumer:type=ZookeeperConsumerConnector,name=af-servers-af-servers-spm-new-cluster-topic-af-servers-wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize

From 1 I can parse af-servers as clientId, ok.

From 2 I have to parse: topic ,groupId, consumer host, timestamp, uuid, 
num_streams.

If I remove af-servers from 2 I will get 
spm-new-cluster-topic-af-servers-wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize.

I can extract timestamp, uuid, num_streams without problems, now let me talk 
about this part spm-new-cluster-topic-af-servers-wawanawna-Dell I see 
af-servers  in the middle but can I be sure that clientId will be always 
here?

Why initial bean name contains af-servers 3 times? I have a feeling sometimes 
all these 3 parts may be different and I will not able to parse everything in 
right way. 

It is painful in general as you mentioned above because we have to remember 
context for parse each line, I am pretty sure nobody like it.
Even if somebody parse names in way you have described I think they will glad 
to change its parse mechanism with  more easiest.

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-08-12 Thread Joe Stein (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093830#comment-14093830
 ] 

Joe Stein commented on KAFKA-1481:
--

[~junrao] If this patch is going to cause existing users that are not having a 
problem issues and it is causing existing users issues by not patching it then 
can we introduce a configuration so we can accommodate both (default to how it 
is working now)? Then we can get it in 0.8.2. Thoughts?

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-08-12 Thread Jun Rao (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094147#comment-14094147
]

Jun Rao commented on KAFKA-1481:

The name of FetchQueueSize is of the following format. If the clientId is not
explicitly specified, the groupId will be used. So if you know the clientId,
you can parse the rest of the string. You probably don't need to figure out
every part of the mbean name. The most important ones are clientId, groupId and
topic.

config.clientId + - + config.groupId + - + topicThreadId._1 + - +
topicThreadId._2 + -FetchQueueSize

Joe,

I am not sure having a config is necessarily better. If we want to fix the
mbean names, I'd rather that we fix them once into the end state that we want,
instead of patching little by little and changing the names multiple times.

Stop using dashes AND underscores as separators in MBean names
--

Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Labels: patch
Fix For: 0.8.2

Attachments: KAFKA-1481_2014-06-06_13-06-35.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-08-11 Thread Otis Gospodnetic (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092501#comment-14092501
 ] 

Otis Gospodnetic commented on KAFKA-1481:
-

Looks like 0.9 has been pushed to December 2014.  Any chance this can go into 
0.8.2 then?  It's a blocker for us :(


 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-08-11 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092863#comment-14092863
 ] 

Jun Rao commented on KAFKA-1481:


I am wondering if you can get around the problem by just parsing the mbean 
names a bit differently. For example, for an mbean like the following in which 
topic-1 is the topic name. When parsing the name part, you can find the last 
- to break the name into two parts.

kafka.server:type=BrokerTopicMetrics,name=topic-1-BytesInPerSec

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-08-11 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14092910#comment-14092910
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Hi Jun Rao, look at http://search-hadoop.com/m/4TaT4lonIW

Example:

kafka.consumer:type=ZookeeperConsumerConnector,name=af_servers-af_servers-spm_new_cluster_topic-af_servers_wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize

Look at part: spm_new_cluster_topic-af_servers_wawanawna-Dell, what is
host name here servers_wawanawna-Dell or wawanawna-Dell ?

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-08-11 Thread Jun Rao (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093709#comment-14093709
 ] 

Jun Rao commented on KAFKA-1481:


I am not sure exactly what your use cases are, but I was thinking of the 
following. For the AllTopics metric like below, you can figure out the 
clientId. Then, you can use that info to parse other beans more reliably.

kafka.consumer:type=ConsumerTopicMetrics,name=test-consumer-group-AllTopicsBytesPerSec

I understand that this is still painful. However, my concern is that this patch 
may fix the issues for some users, but may force other users to change the way 
they extract mbeans. 

 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-06-23 Thread Otis Gospodnetic (JIRA)

[
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14040652#comment-14040652
]

Otis Gospodnetic commented on KAFKA-1481:
-

Hi Jun, thanks for having a look.

bq. 1. While this may help some users, it may equally affect some other users
who now would need to change their parsing of the jmx names.

Won't they have to do that sooner or later anyway, because dashes as separators
obviously can't stay?

bq. 2. I am not sure how wide-spread the separator issue. If it only helps a
small number of users, I'd prefer that we only do that on the new consumer.

Nobody knows one way or the other though. So in absence of info I think it
makes sense to at least make a move in the right direction.

bq. Other than the separator issue, there are other things that are not ideal.
For example, the bean name is just unnecessarily too long. If we do want to
change the bean names, perhaps we should fix other problems together as well.

Long bean names may not be pretty, but they are not blockers. The dashes are
likely blockers for some portion of users and definitely blockers for any Kafka
monitoring tool makers. If long bean names are a problem (are they?) let's
open a separate issue, ha?

Stop using dashes AND underscores as separators in MBean names
--

Key: KAFKA-1481
URL: https://issues.apache.org/jira/browse/KAFKA-1481
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
Labels: patch
Fix For: 0.8.2

Attachments: KAFKA-1481_2014-06-06_13-06-35.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-06-20 Thread Otis Gospodnetic (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038721#comment-14038721
 ] 

Otis Gospodnetic commented on KAFKA-1481:
-

Nice and simple patch.  Question about this:
{code}
   def validateChars(prop: String, value: String) {
-val legalChars = [a-zA-Z0-9\\._\\-]
+val legalChars = [a-zA-Z0-9\\._\\-\\|]
{code}

I didn't check the source code to understand the bigger context, but the above 
looks like | is now a valid character, but should it really be valid/allowed?


 Stop using dashes AND underscores as separators in MBean names
 --

 Key: KAFKA-1481
 URL: https://issues.apache.org/jira/browse/KAFKA-1481
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Otis Gospodnetic
  Labels: patch
 Fix For: 0.8.2

 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch


 MBeans should not use dashes or underscores as separators because these 
 characters are allowed in hostnames, topics, group and consumer IDs, etc., 
 and these are embedded in MBeans names making it impossible to parse out 
 individual bits from MBeans.
 Perhaps a pipe character should be used to avoid the conflict. 
 This looks like a major blocker because it means nobody can write Kafka 0.8.x 
 monitoring tools unless they are doing it for themselves AND do not use 
 dashes AND do not use underscores.
 See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.2#6252)

72 matches

Mail list logo