[
https://issues.apache.org/jira/browse/CASSANDRA-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15875287#comment-15875287
]
Nate McCall commented on CASSANDRA-13223:
-----------------------------------------
Hi [~vladimir.bukhtoyarov] thank you for the analysis and effort on the fix.
Please see the following instructions for submitting a patch:
http://cassandra.apache.org/doc/latest/development/patches.html
> Unable to compute when histogram overflowed
> -------------------------------------------
>
> Key: CASSANDRA-13223
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13223
> Project: Cassandra
> Issue Type: Bug
> Reporter: Vladimir Bukhtoyarov
> Priority: Minor
>
> DecayingEstimatedHistogramReservoir throws exception when value upper max
> recorded to reservoir. It is very undesired behavior, because functionality
> like logging or monitoring should never fail with exception. Current behavior
> of DecayingEstimatedHistogramReservoir violates contract for
> [Reservoir|https://github.com/dropwizard/metrics/blob/3.2-development/metrics-core/src/main/java/com/codahale/metrics/Reservoir.java],
> as you can see javadocs for Reservoir says nothing that implementation can
> throw exception in getSnapshot method. As result all Dropwizzard/Metrics
> reporters are broken, because nobody expect that metric will throw exception
> on get, for example our monitoring pipeline is broken with exception:
> {noformat}
> com.fasterxml.jackson.databind.JsonMappingException: Unable to compute when
> histogram overflowed (through reference chain:
> java.util.UnmodifiableSortedMap["org.apache.cassandra.metrics.Table
> .ColUpdateTimeDeltaHistogram.all"])
> at
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:339)
> at
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:299)
> at
> com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:342)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:620)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:519)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:31)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2436)
> at
> com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:355)
> at
> com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1442)
> at
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:188)
> at
> com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:171)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1428)
> at
> com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1129)
> at
> com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:967)
> at
> com.codahale.metrics.servlets.MetricsServlet.doGet(MetricsServlet.java:176)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
> at
> com.ringcentral.slf4j.CleanMDCFilter.doFilter(CleanMDCFilter.java:18)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:524)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> com.ringcentral.concurrent.executors.MonitoredExecutor$MonitoredRunnable.run(MonitoredExecutor.java:220)
> at
> com.ringcentral.concurrent.executors.ContextAwareExecutor$ContextAwareRunnable.run(ContextAwareExecutor.java:34)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Unable to compute when histogram
> overflowed
> at
> org.apache.cassandra.metrics.DecayingEstimatedHistogramReservoir$EstimatedHistogramReservoirSnapshot.getMean(DecayingEstimatedHistogramReservoir.java:472)
> at
> com.codahale.metrics.json.MetricsModule$HistogramSerializer.serialize(MetricsModule.java:72)
> at
> com.codahale.metrics.json.MetricsModule$HistogramSerializer.serialize(MetricsModule.java:56)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:616)
> ... 40 common frames omitted
> {noformat}
> The most oblivious solution to resolve overflow, will be replacing the
> exceeded value by highest trackable value, [similar to
> this|https://github.com/vladimir-bukhtoyarov/rolling-metrics/blob/master/src/main/java/com/github/rollingmetrics/histogram/OverflowResolver.java#L34]
> I have implemented the fix in the [branch on
> github|https://github.com/vladimir-bukhtoyarov/cassandra/tree/fix-reservoir-overflow],
> see [particular commit |
> https://github.com/vladimir-bukhtoyarov/cassandra/commit/4dca54c1000576a892a77bc716f87adc4bc05ecc]
> for details.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
