[ 
https://issues.apache.org/jira/browse/CASSANDRA-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13905861#comment-13905861
 ] 

Joshua McKenzie commented on CASSANDRA-6577:
--------------------------------------------

Regarding the netstats exception:  the SessionInfo objects being operated on 
inside the creation of CompositeData aren't thread-safe, so the following use 
presents an opportunity for the Exception you're seeing.

Declaration:
{code:title=SessionInfo.java|borderstyle=solid}
        this.receivingFiles = new HashMap<>();
        this.sendingFiles = new HashMap<>();
{code}

These are iterated across creating the various CompositeData types requested, 
however updateProgress within SessionInfo would cause invalidation of any open 
iterators on the collection if any are active.

Changing these to ConcurrentHashMaps should prevent the concurrent modification 
exceptions race though it's going to have a performance impact I'll need to 
think about some more.  We could go the NonBlockingHashMap route (CliffC route) 
but we don't have strong guarantees of consistency there.



> ConcurrentModificationException during nodetool netstats
> --------------------------------------------------------
>
>                 Key: CASSANDRA-6577
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6577
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>         Environment: AWS EC2 machines, 
> java version "1.7.0_45"
> Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
> Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
> Each node has 32 vnodes.
>            Reporter: Shao-Chuan Wang
>            Assignee: Joshua McKenzie
>            Priority: Minor
>              Labels: decommission, nodetool
>             Fix For: 2.0.6
>
>
> The node is leaving and I wanted to check its netstats, but it raises 
> ConcurrentModificationException.
> {code}
> [ubuntu@ip-10-4-202-48 :~]# /mnt/cassandra_latest/bin/nodetool netstats
> Mode: LEAVING
> Exception in thread "main" java.util.ConcurrentModificationException
>       at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
>       at java.util.HashMap$ValueIterator.next(HashMap.java:954)
>       at 
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
>       at com.google.common.collect.Iterators.addAll(Iterators.java:357)
>       at com.google.common.collect.Lists.newArrayList(Lists.java:146)
>       at com.google.common.collect.Lists.newArrayList(Lists.java:128)
>       at 
> org.apache.cassandra.streaming.management.SessionInfoCompositeData.toArrayOfCompositeData(SessionInfoCompositeData.java:161)
>       at 
> org.apache.cassandra.streaming.management.SessionInfoCompositeData.toCompositeData(SessionInfoCompositeData.java:98)
>       at 
> org.apache.cassandra.streaming.management.StreamStateCompositeData$1.apply(StreamStateCompositeData.java:82)
>       at 
> org.apache.cassandra.streaming.management.StreamStateCompositeData$1.apply(StreamStateCompositeData.java:79)
>       at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
>       at 
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
>       at com.google.common.collect.Iterators.addAll(Iterators.java:357)
>       at com.google.common.collect.Lists.newArrayList(Lists.java:146)
>       at com.google.common.collect.Lists.newArrayList(Lists.java:128)
>       at 
> org.apache.cassandra.streaming.management.StreamStateCompositeData.toCompositeData(StreamStateCompositeData.java:78)
>       at 
> org.apache.cassandra.streaming.StreamManager$1.apply(StreamManager.java:87)
>       at 
> org.apache.cassandra.streaming.StreamManager$1.apply(StreamManager.java:84)
>       at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
>       at 
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
>       at com.google.common.collect.Iterators.addAll(Iterators.java:357)
>       at com.google.common.collect.Sets.newHashSet(Sets.java:238)
>       at com.google.common.collect.Sets.newHashSet(Sets.java:218)
>       at 
> org.apache.cassandra.streaming.StreamManager.getCurrentStreams(StreamManager.java:83)
>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>       at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>       at java.lang.reflect.Method.invoke(Method.java:606)
>       at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
>       at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
>       at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>       at java.lang.reflect.Method.invoke(Method.java:606)
>       at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
>       at 
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
>       at 
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
>       at 
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
>       at 
> com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
>       at 
> com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
>       at 
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
>       at 
> com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
>       at 
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1464)
>       at 
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
>       at 
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
>       at 
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
>       at 
> javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:657)
>       at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
>       at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>       at java.lang.reflect.Method.invoke(Method.java:606)
>       at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
>       at sun.rmi.transport.Transport$1.run(Transport.java:177)
>       at sun.rmi.transport.Transport$1.run(Transport.java:174)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
>       at 
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
>       at 
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
>       at 
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:724)
> {code}
> The gossip info is as follows:
> {code}
> /10.215.114.239
>   SCHEMA:e2a4b7b2-df50-3cc4-97f6-2ce7fb77982b
>   NET_VERSION:7
>   LOAD:6.42655250606E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1305722878370288637
>   HOST_ID:d13374d8-e4ae-466d-ad5a-44229a2fa190
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.122.218.80
>   SCHEMA:548308e6-bd1c-388f-ad1e-5ecf39f994fd
>   NET_VERSION:7
>   LOAD:2.71405175303E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1796000685025988745
>   HOST_ID:7746a875-1e53-4ebe-aef6-ad59fadd6ea7
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.178.13.230
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.34997966591E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:LEAVING,-1646882803236172485
>   HOST_ID:160d83bf-4cc0-4872-aec6-7908446eccbf
>   DC:pagedb-backend
>   SEVERITY:1.5584415197372437
> /10.71.141.179
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.21492186385E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1277850891915823266
>   HOST_ID:baa81888-6417-4e7c-8d7b-2bb79c38ca40
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.251.114.31
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.95280061394E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1b
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1866613386904756042
>   HOST_ID:a71e7037-c4f8-4ff9-864d-e83b6d3037d7
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.95.128.214
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.90628204861E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1e
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1896996972976095280
>   HOST_ID:3e5e6e75-56cb-4664-abf4-de8c2801bd5d
>   DC:pagedb-backend
>   SEVERITY:0.0
> /10.228.26.240
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.46469437383E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1359255731430490566
>   HOST_ID:8df816c9-7875-4106-aeda-b372b9f1fdc9
>   DC:pagedb-frontend
>   SEVERITY:1.5037593841552734
> /10.185.67.195
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:4.6531989096E10
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1295072457639851651
>   HOST_ID:37fe7cc7-3481-48b3-96ff-c92df17a4132
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.220.195.198
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:7.8265401144E10
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1514821029440891529
>   HOST_ID:0bd7f3ab-d347-4c59-9f1a-1b7104e34a6b
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.62.39.130
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.7020827919E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1089815038805941976
>   HOST_ID:13b12ee3-36a4-462c-9013-ccc1b83a70ff
>   DC:pagedb-frontend
>   SEVERITY:0.9411764144897461
> /10.137.11.197
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.17329728527E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:LEAVING,-1708596056219631840
>   HOST_ID:76d339f4-ac4a-4f95-bada-1ec17f67f4e3
>   DC:pagedb-backend
>   SEVERITY:0.5555555820465088
> /10.87.145.85
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:5.78714900637E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-2223601356751123985
>   HOST_ID:bdad1b28-3fb7-4788-8f34-01caace03ca9
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.97.135.36
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:9.3081789914E10
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1146411778648768253
>   HOST_ID:40fb13d6-0803-45cc-9b4f-45b3ad7194fa
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.95.128.6
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:3.56500409995E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1e
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1025262993714352739
>   HOST_ID:80b583a5-8f7d-4fee-91db-b948c090d055
>   DC:pagedb-backend
>   SEVERITY:0.7518796920776367
> /10.154.136.39
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.70485777643E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   HOST_ID:0e391fea-e4e9-4a46-b9af-87948459876c
>   STATUS:LEAVING,-2690692580263318876
>   DC:pagedb-backend
>   SEVERITY:0.0
> /10.185.9.84
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.87054930947E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:LEAVING,-1468213189211385280
>   HOST_ID:080241a9-eadd-48b4-8f94-efbe35bfd6e1
>   DC:pagedb-backend
>   SEVERITY:0.2557544708251953
> /10.251.114.15
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.86231038729E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1b
>   RPC_ADDRESS:0.0.0.0
>   HOST_ID:99e3e672-8410-4850-8257-edc7b814a776
>   STATUS:NORMAL,-1117031336072704821
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.97.135.32
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:3.1249908096E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1176341277720804881
>   HOST_ID:e9ac8ad9-c3cc-44ff-81e7-af28a4d5f576
>   DC:pagedb-backend
>   SEVERITY:0.0
> /10.99.144.60
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:8.9296442414E10
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1695890443918261320
>   HOST_ID:01b323c1-3f0e-40de-8ef7-3e1d33b391c5
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.121.13.216
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:3.70269351421E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1081159627867819835
>   HOST_ID:05f06b6f-ddb7-42a9-9e56-8efd41cf1545
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.99.144.97
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.95197366549E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1188304416367949186
>   HOST_ID:7a84853a-2c0a-448c-84e2-cceaa98c7523
>   DC:pagedb-backend
>   SEVERITY:0.26178011298179626
> /10.95.132.5
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:9.8325409989E10
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1e
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1168449340133292844
>   HOST_ID:6480a3a2-cbbe-44f4-b67b-7310f885b307
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.178.2.177
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:3.8547129498E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1337943409437095727
>   HOST_ID:aa7fd551-8a47-4e59-869e-ae0afaadf27e
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.201.206.166
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.02276920825E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-2302424866842171501
>   HOST_ID:f9651587-8f34-4dbc-af5c-64f19bc85ad6
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.93.6.87
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:5.55645040614E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1767037199861003446
>   HOST_ID:2ca596b1-fa0a-40a8-ace1-bc7b21e82c45
>   DC:pagedb-frontend
>   SEVERITY:1.0230178833007812
> /10.123.74.248
>   SCHEMA:b2c3e6ec-3b3f-3ad9-baef-e9e311a24eb9
>   NET_VERSION:7
>   LOAD:3.09054824961E11
>   REMOVAL_COORDINATOR:REMOVER,76d339f4-ac4a-4f95-bada-1ec17f67f4e3
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   STATUS:removed,94daf57d-838d-4285-8048-e60f97629dba,1389650189822
>   HOST_ID:94daf57d-838d-4285-8048-e60f97629dba
>   DC:pagedb-backend
>   SEVERITY:0.2493765652179718
> /10.71.141.12
>   SCHEMA:26cf5f29-f7f9-321d-b5c4-ac9314d90653
>   NET_VERSION:7
>   LOAD:1.91108857556E11
>   REMOVAL_COORDINATOR:REMOVER,c7a0edfd-629e-47c3-b79c-32a9c2c98801
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:removed,4b9f63e5-8561-4e59-8011-e11fb3e8d627,1389562451886
>   HOST_ID:4b9f63e5-8561-4e59-8011-e11fb3e8d627
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.185.0.80
>   SCHEMA:039db936-c2e7-3f53-9c94-efdf2b861d53
>   NET_VERSION:7
>   LOAD:2.22049506986E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:removed,9885866e-af4d-4129-ad33-d465c8a7cd58,1389732020477
>   HOST_ID:9885866e-af4d-4129-ad33-d465c8a7cd58
>   DC:pagedb-backend
>   SEVERITY:2.7707808017730713
> /10.185.47.50
>   SCHEMA:b13a987f-adbe-3c19-afc9-419e595d9002
>   NET_VERSION:7
>   LOAD:2.19790952307E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   HOST_ID:73c27c92-1f3e-4628-9c91-a1e0a03a9e21
>   STATUS:removed,73c27c92-1f3e-4628-9c91-a1e0a03a9e21,1389731586723
>   DC:pagedb-backend
>   SEVERITY:6.6137566566467285
> /10.93.49.164
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.03359005906E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1102475930390387446
>   HOST_ID:19f401cb-b418-4ce3-b0e0-ea1530ab2121
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.206.97.184
>   SCHEMA:ede9498d-befd-3153-87ee-2ea329001466
>   NET_VERSION:7
>   LOAD:2.34212215287E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   HOST_ID:8d3d9e79-5a72-4561-99f0-0817cd6ad6cc
>   STATUS:LEAVING,-1397201050646007911
>   DC:pagedb-backend
>   SEVERITY:2.236421823501587
> /10.71.141.42
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:6.0594679857E10
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1008058669507434673
>   HOST_ID:2113b135-8566-4ed9-b74b-5e302d240c42
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.154.148.5
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:4.91232885735E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1261154574760184756
>   HOST_ID:86642aef-f625-4561-be7b-ab8f58066294
>   DC:pagedb-frontend
>   SEVERITY:0.7672634720802307
> /10.183.153.179
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:3.17986161259E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1e
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1486604450347152783
>   HOST_ID:5988d85d-663c-4597-9422-7f5bc195ec5b
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.97.131.247
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.43618302855E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1087382214101550882
>   HOST_ID:811a2966-b6f0-41fd-ba29-fc980763a69c
>   DC:pagedb-backend
>   SEVERITY:0.0
> /10.99.144.54
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:8.9936707476E10
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-3327961952466683302
>   HOST_ID:675e8de7-641f-4914-beeb-fad29942381f
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.210.91.83
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:3.19028851441E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1375725999837634215
>   HOST_ID:b0726b39-1587-4443-853e-6e668658b1cc
>   DC:pagedb-frontend
>   SEVERITY:17.39130401611328
> /10.71.141.158
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.11308980882E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1306761067597910068
>   HOST_ID:31c7690b-30fa-455c-9f69-e98f11d4c235
>   DC:pagedb-backend
>   SEVERITY:0.0
> /10.38.169.32
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.3296808983E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:LEAVING,-137637605179813169
>   HOST_ID:67ec4857-c867-46c8-9c62-778fd26360eb
>   DC:pagedb-backend
>   SEVERITY:1.671309232711792
> /10.87.94.230
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:3.52091238152E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   STATUS:LEAVING,-1519665921822534263
>   HOST_ID:604b0268-187a-4a5f-a51e-b94608111825
>   DC:pagedb-backend
>   SEVERITY:0.27397260069847107
> /10.99.144.91
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.96583444544E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-3103769032004455685
>   HOST_ID:010613b8-9d0b-487d-bdd1-64e80ddc60e6
>   DC:pagedb-backend
>   SEVERITY:1.0204081535339355
> /10.71.141.130
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.45733317443E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1099965132011727160
>   HOST_ID:3c30b5fb-bf8c-4169-a60f-6ffad2c28598
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.99.146.244
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.6041541835E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1404826323347485057
>   HOST_ID:6a96c2e0-6376-461d-bac5-0dcbe3cd1eb5
>   DC:pagedb-backend
>   SEVERITY:0.0
> /10.95.129.124
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.61026008599E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1e
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-2498420976414553971
>   HOST_ID:2a987ea1-4037-4b9e-a403-27a6d7995621
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.122.50.31
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.18914643786E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   HOST_ID:b9e40521-bf2e-4833-8f6a-a8e3297df042
>   STATUS:NORMAL,-1277185879546496035
>   DC:pagedb-backend
>   SEVERITY:0.0
> /10.4.197.53
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:3.15526897917E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:LEAVING,-1388643925813439514
>   HOST_ID:9f76061b-43a7-432b-9e6d-612a628534cf
>   DC:pagedb-backend
>   SEVERITY:0.7537688612937927
> /10.44.183.111
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.10331330719E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   STATUS:LEAVING,-283520605333269313
>   HOST_ID:1e0e3870-5026-452b-8050-dad7fcd5bd88
>   DC:pagedb-backend
>   SEVERITY:7.416880130767822
> /10.238.138.32
>   SCHEMA:b13a987f-adbe-3c19-afc9-419e595d9002
>   NET_VERSION:7
>   REMOVAL_COORDINATOR:REMOVER,3c30b5fb-bf8c-4169-a60f-6ffad2c28598
>   LOAD:1.92772802904E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   HOST_ID:e225c748-ffd8-4905-91bf-4e745b26fa44
>   STATUS:removed,e225c748-ffd8-4905-91bf-4e745b26fa44,1389731792774
>   DC:pagedb-backend
>   SEVERITY:0.0
> /10.95.129.103
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.50386165884E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1e
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1389777754931898083
>   HOST_ID:18031026-e20c-40eb-8bb3-d771b641fb26
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.125.10.14
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:7.2903902818E10
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1a
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1822845329295703853
>   HOST_ID:5764492f-31f8-4c3f-8bec-d8a2c1af4b4d
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.4.202.48
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:3.05065515518E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   HOST_ID:76f98456-2029-4bb6-b3de-e08157220fc9
>   STATUS:LEAVING,-2184837513740789816
>   DC:pagedb-backend
>   SEVERITY:28.76344108581543
> /10.95.132.44
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.97019107429E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1e
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-243162178282959980
>   HOST_ID:c7a0edfd-629e-47c3-b79c-32a9c2c98801
>   DC:pagedb-backend
>   SEVERITY:0.5076141953468323
> /10.96.31.80
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:2.29408976299E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1c
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1236217677726412210
>   HOST_ID:fa3ea99e-f9b6-4f83-9234-21d8059e6eb4
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.40.245.193
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:1.27656537169E11
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1237220198988914368
>   HOST_ID:d2383010-fd55-4364-8868-5dc8a937778f
>   DC:pagedb-frontend
>   SEVERITY:0.0
> /10.99.144.75
>   SCHEMA:f9832569-f987-3545-8c36-196342655a7f
>   NET_VERSION:7
>   LOAD:8.2445952994E10
>   RELEASE_VERSION:2.0.1
>   RACK:us-east-1d
>   RPC_ADDRESS:0.0.0.0
>   STATUS:NORMAL,-1022810197871922800
>   HOST_ID:c085c62d-74ea-45e5-9ed6-6a26a72ec3bb
>   DC:pagedb-frontend
>   SEVERITY:0.0
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Reply via email to