[jira] [Commented] (CASSANDRA-6423) Some histogram metrics of long[] type are unusable with Graphite
[ https://issues.apache.org/jira/browse/CASSANDRA-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13839219#comment-13839219 ] mat gomes commented on CASSANDRA-6423: -- We are also having the same issue. invalid line received from client IP ignoring. Flooding the logs Please share resolutions to this issue. Thanks. Some histogram metrics of long[] type are unusable with Graphite Key: CASSANDRA-6423 URL: https://issues.apache.org/jira/browse/CASSANDRA-6423 Project: Cassandra Issue Type: Bug Components: Core Environment: Cassandra 2.0.3, Oracle Linux 6.3, graphite Reporter: Nikolai Grigoriev Priority: Minor I am not entirely sure if this is a Cassandra issue or the limitation of the graphite reporter agent. But since the metrics in question are created by Cassandra itself I have decided that it may be appropriate to report it here. I am using Graphite with Cassandra 2.0.x and I have recently noticed frequent 'invalid line' messages in Graphite log. Unfortunately Graphite did not provide enough details so I hacked it a bit to get more verbose error message with the metric name. And here is what I have found: {code} 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@17c26ba5 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedColumnCountHistogram.value [J@5d2929d2 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@3978c9c6 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedColumnCountHistogram.value [J@290703a4 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@b801907 1385741236received from client 192.168.20.157:3, ignoring {code} Then a quick search through Cassandre code confirmed that there is a number of histograms created as Gauge with long[] data. So, when they are serialized by GraphiteReporter they are just printed as long[].toString(), making these metrics useless. I am not sure what would be the best solution to it. I do see some histograms (LiveScannedHistogram etc) that are implemented differently and they are properly sent to Graphite. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6423) Some histogram metrics of long[] type are unusable with Graphite
[ https://issues.apache.org/jira/browse/CASSANDRA-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13839449#comment-13839449 ] Mikhail Stepura commented on CASSANDRA-6423: I wonder if it's possible to convert those {{Gaugelong[]}} to {{Histogram}}, i.e. populate metric's {{Histogram}} from {{EstimatedHistogram}} Some histogram metrics of long[] type are unusable with Graphite Key: CASSANDRA-6423 URL: https://issues.apache.org/jira/browse/CASSANDRA-6423 Project: Cassandra Issue Type: Bug Components: Core Environment: Cassandra 2.0.3, Oracle Linux 6.3, graphite Reporter: Nikolai Grigoriev Priority: Minor I am not entirely sure if this is a Cassandra issue or the limitation of the graphite reporter agent. But since the metrics in question are created by Cassandra itself I have decided that it may be appropriate to report it here. I am using Graphite with Cassandra 2.0.x and I have recently noticed frequent 'invalid line' messages in Graphite log. Unfortunately Graphite did not provide enough details so I hacked it a bit to get more verbose error message with the metric name. And here is what I have found: {code} 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@17c26ba5 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedColumnCountHistogram.value [J@5d2929d2 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@3978c9c6 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedColumnCountHistogram.value [J@290703a4 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@b801907 1385741236received from client 192.168.20.157:3, ignoring {code} Then a quick search through Cassandre code confirmed that there is a number of histograms created as Gauge with long[] data. So, when they are serialized by GraphiteReporter they are just printed as long[].toString(), making these metrics useless. I am not sure what would be the best solution to it. I do see some histograms (LiveScannedHistogram etc) that are implemented differently and they are properly sent to Graphite. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6423) Some histogram metrics of long[] type are unusable with Graphite
[ https://issues.apache.org/jira/browse/CASSANDRA-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835475#comment-13835475 ] Nikolai Grigoriev commented on CASSANDRA-6423: -- I think the consumers cannot really be expected to understand [J@290703a4...Knowing how the serialization of the metrics works, can you really expect that your metric defined as long[] would be serialized as something meaningful? Some histogram metrics of long[] type are unusable with Graphite Key: CASSANDRA-6423 URL: https://issues.apache.org/jira/browse/CASSANDRA-6423 Project: Cassandra Issue Type: Bug Components: Core Environment: Cassandra 2.0.3, Oracle Linux 6.3, graphite Reporter: Nikolai Grigoriev Priority: Minor I am not entirely sure if this is a Cassandra issue or the limitation of the graphite reporter agent. But since the metrics in question are created by Cassandra itself I have decided that it may be appropriate to report it here. I am using Graphite with Cassandra 2.0.x and I have recently noticed frequent 'invalid line' messages in Graphite log. Unfortunately Graphite did not provide enough details so I hacked it a bit to get more verbose error message with the metric name. And here is what I have found: {code} 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@17c26ba5 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedColumnCountHistogram.value [J@5d2929d2 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@3978c9c6 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedColumnCountHistogram.value [J@290703a4 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@b801907 1385741236received from client 192.168.20.157:3, ignoring {code} Then a quick search through Cassandre code confirmed that there is a number of histograms created as Gauge with long[] data. So, when they are serialized by GraphiteReporter they are just printed as long[].toString(), making these metrics useless. I am not sure what would be the best solution to it. I do see some histograms (LiveScannedHistogram etc) that are implemented differently and they are properly sent to Graphite. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6423) Some histogram metrics of long[] type are unusable with Graphite
[ https://issues.apache.org/jira/browse/CASSANDRA-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835552#comment-13835552 ] Mikhail Stepura commented on CASSANDRA-6423: https://github.com/codahale/metrics/issues is a good place to start this discussion. At the moment GraphiteReporter doesn't handle Gauges which return an array (which is perfectly valid use case) instead of a single value. I think checking {{object.getClass().isArray()}} and using {{Arrays.toString(object)}} should help Some histogram metrics of long[] type are unusable with Graphite Key: CASSANDRA-6423 URL: https://issues.apache.org/jira/browse/CASSANDRA-6423 Project: Cassandra Issue Type: Bug Components: Core Environment: Cassandra 2.0.3, Oracle Linux 6.3, graphite Reporter: Nikolai Grigoriev Priority: Minor I am not entirely sure if this is a Cassandra issue or the limitation of the graphite reporter agent. But since the metrics in question are created by Cassandra itself I have decided that it may be appropriate to report it here. I am using Graphite with Cassandra 2.0.x and I have recently noticed frequent 'invalid line' messages in Graphite log. Unfortunately Graphite did not provide enough details so I hacked it a bit to get more verbose error message with the metric name. And here is what I have found: {code} 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@17c26ba5 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedColumnCountHistogram.value [J@5d2929d2 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@3978c9c6 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedColumnCountHistogram.value [J@290703a4 1385741236received from client 192.168.20.157:3, ignoring 29/11/2013 16:07:16 :: invalid line .org.apache.cassandra.metrics.ColumnFamily.mykeyspace.some_cf.EstimatedRowSizeHistogram.value [J@b801907 1385741236received from client 192.168.20.157:3, ignoring {code} Then a quick search through Cassandre code confirmed that there is a number of histograms created as Gauge with long[] data. So, when they are serialized by GraphiteReporter they are just printed as long[].toString(), making these metrics useless. I am not sure what would be the best solution to it. I do see some histograms (LiveScannedHistogram etc) that are implemented differently and they are properly sent to Graphite. -- This message was sent by Atlassian JIRA (v6.1#6144)