[
https://issues.apache.org/jira/browse/CASSANDRA-11751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280143#comment-15280143
]
Jeff Griffith commented on CASSANDRA-11751:
-------------------------------------------
Thanks [~tjake]. Sorry for the duplicate.
> Histogram overflow in metrics
> -----------------------------
>
> Key: CASSANDRA-11751
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11751
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.2.6 on Linux
> Reporter: Jeff Griffith
>
> One particular histogram in the cassandra metrics seems to overflow
> preventing the calculation of the mean on the dropwizard "Snapshot". Here is
> the exception that comes from the metrics library:
> {code}
> java.lang.IllegalStateException: Unable to compute ceiling for max when
> histogram overflowed
> at
> org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:232)
> ~[apache-cassandra-2.2.6.jar:2.2.6-SNAPSHOT]
> at
> org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getMean(EstimatedHistogramReservoir.java:103)
> ~[apache-cassandra-2.2.6.jar:2.2.6-SNAPSHOT]
> at
> com.addthis.metrics3.reporter.config.SplunkReporter.reportHistogram(SplunkReporter.java:155)
> ~[reporter-config3-3.0.0.jar:3.0.0]
> at
> com.addthis.metrics3.reporter.config.SplunkReporter.report(SplunkReporter.java:101)
> ~[reporter-config3-3.0.0.jar:3.0.0]
> at
> com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162)
> ~[metrics-core-3.1.0.jar:3.1.0]
> at
> com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117)
> ~[metrics-core-3.1.0.jar:3.1.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_72]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_72]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_72]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_72]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_72]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_72]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
> {code}
> On deeper analysis, it seems like this is happening specifically on this
> metric:
> {code}
> ColUpdateTimeDeltaHistogram
> {code}
> I think this is where it is updated in ColumnFamilyStore.java
> {code}
> public void apply(DecoratedKey key, ColumnFamily columnFamily,
> SecondaryIndexManager.Updater indexer, OpOrder.Group opGroup, ReplayPosition
> replayPosition)
> {
> long start = System.nanoTime();
> Memtable mt = data.getMemtableFor(opGroup, replayPosition);
> final long timeDelta = mt.put(key, columnFamily, indexer, opGroup);
> maybeUpdateRowCache(key);
> metric.samplers.get(Sampler.WRITES).addSample(key.getKey(),
> key.hashCode(), 1);
> metric.writeLatency.addNano(System.nanoTime() - start);
> if(timeDelta < Long.MAX_VALUE)
> metric.colUpdateTimeDeltaHistogram.update(timeDelta);
> }
> {code}
> Considering it's calculating a mean, i don't know if perhaps a large sum
> might be overflowing? But that "if (timeDelta < Long.MAX_VALUE)" looks
> suspect, doesn't it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
