Vladimir Bukhtoyarov created CASSANDRA-13223:
------------------------------------------------
Summary: Unable to compute when histogram overflowed
Key: CASSANDRA-13223
URL: https://issues.apache.org/jira/browse/CASSANDRA-13223
Project: Cassandra
Issue Type: Bug
Reporter: Vladimir Bukhtoyarov
Priority: Minor
DecayingEstimatedHistogramReservoir throws exception when value upper max
recorded to reservoir. It is very undesired behavior, because functionality
like logging or monitoring should never fail with exception. Current behavior
of DecayingEstimatedHistogramReservoir violates contract for
[Reservoir|https://github.com/dropwizard/metrics/blob/3.2-development/metrics-core/src/main/java/com/codahale/metrics/Reservoir.java],
as you can see javadocs for Reservoir says nothing that implementation can
throw exception in getSnapshot method. As result all Dropwizzard/Metrics
reporters are broken, because nobody expect that metric will throw exception on
get, for example our monitoring pipeline is broken with exception:
{noformat}
com.fasterxml.jackson.databind.JsonMappingException: Unable to compute when
histogram overflowed (through reference chain:
java.util.UnmodifiableSortedMap["org.apache.cassandra.metrics.Table
.ColUpdateTimeDeltaHistogram.all"])
at
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:339)
at
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:299)
at
com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:342)
at
com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:620)
at
com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:519)
at
com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:31)
at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
at
com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2436)
at
com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:355)
at
com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1442)
at
com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:188)
at
com.codahale.metrics.json.MetricsModule$MetricRegistrySerializer.serialize(MetricsModule.java:171)
at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
at
com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1428)
at
com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1129)
at
com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:967)
at
com.codahale.metrics.servlets.MetricsServlet.doGet(MetricsServlet.java:176)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
at com.ringcentral.slf4j.CleanMDCFilter.doFilter(CleanMDCFilter.java:18)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:524)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
com.ringcentral.concurrent.executors.MonitoredExecutor$MonitoredRunnable.run(MonitoredExecutor.java:220)
at
com.ringcentral.concurrent.executors.ContextAwareExecutor$ContextAwareRunnable.run(ContextAwareExecutor.java:34)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Unable to compute when histogram
overflowed
at
org.apache.cassandra.metrics.DecayingEstimatedHistogramReservoir$EstimatedHistogramReservoirSnapshot.getMean(DecayingEstimatedHistogramReservoir.java:472)
at
com.codahale.metrics.json.MetricsModule$HistogramSerializer.serialize(MetricsModule.java:72)
at
com.codahale.metrics.json.MetricsModule$HistogramSerializer.serialize(MetricsModule.java:56)
at
com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:616)
... 40 common frames omitted
{noformat}
The most oblivious solution to resolve overflow, will be replacing the exceeded
value by highest trackable value, [similar to
this|https://github.com/vladimir-bukhtoyarov/rolling-metrics/blob/master/src/main/java/com/github/rollingmetrics/histogram/OverflowResolver.java#L34]
I have implemented the fix in the [branch on
github|https://github.com/vladimir-bukhtoyarov/cassandra/tree/fix-reservoir-overflow],
see [particular commit |
https://github.com/vladimir-bukhtoyarov/cassandra/commit/4dca54c1000576a892a77bc716f87adc4bc05ecc]
for details.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
