Rob Emery created CASSANDRA-12978:
-------------------------------------
Summary: mx4j -> HTTP 500 -> ConcurrentModificationException
Key: CASSANDRA-12978
URL: https://issues.apache.org/jira/browse/CASSANDRA-12978
Project: Cassandra
Issue Type: Bug
Components: Tools
Environment: Debian, Single cluster, 2 data centres, E5-2620 v3, 16GB,
RAID1 SSD Commit log, RAID10 15k HDD data
Reporter: Rob Emery
Fix For: 2.1.6
We run some checks from our Monitoring software that rely on mx4j.
The checks typically grab some xml via HTTP request and parse it. For
example, CF Stats on 'MyKeySpace' and 'MyColumnFamily' are retrieved
using:
http://cassandra001:8081/mbean?template=identity&objectname=org.apache.cassandra.db%3Atype%3DColumnFamilies%2Ckeyspace%3DMyKeySpace%2Ccolumnfamily%3DMyColumnFamily
The checks run each minute. Periodically they result in a "HTTP 500 internal
server error". The HTML body returned is empty.
Experimentally we ran Cassandra in the foreground on one node and reproduced
the problem. this elicited the following stack trace:
javax.management.RuntimeMBeanException:
java.util.ConcurrentModificationException
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
at
mx4j.tools.adaptor.http.MBeanCommandProcessor.createMBeanElement(MBeanCommandProcessor.java:119)
at
mx4j.tools.adaptor.http.MBeanCommandProcessor.executeRequest(MBeanCommandProcessor.java:56)
at
mx4j.tools.adaptor.http.HttpAdaptor$HttpClient.run(HttpAdaptor.java:980)
Caused by: java.util.ConcurrentModificationException
at
java.util.TreeMap$NavigableSubMap$SubMapIterator.nextEntry(TreeMap.java:1594)
at
java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1642)
at
java.util.TreeMap$NavigableSubMap$SubMapEntryIterator.next(TreeMap.java:1636)
at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
at
org.apache.cassandra.utils.StreamingHistogram.sum(StreamingHistogram.java:160)
at
org.apache.cassandra.io.sstable.metadata.StatsMetadata.getDroppableTombstonesBefore(StatsMetadata.java:113)
at
org.apache.cassandra.io.sstable.SSTableReader.getDroppableTombstonesBefore(SSTableReader.java:2004)
at
org.apache.cassandra.db.DataTracker.getDroppableTombstoneRatio(DataTracker.java:507)
at
org.apache.cassandra.db.ColumnFamilyStore.getDroppableTombstoneRatio(ColumnFamilyStore.java:3089)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
... 4 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
