hello, I have a 4 node cassandra cluster with 0.6.1 installed. We've been running a mixed read / write workload test how it works in our environment, we run about 4M bath mutations and 40M get_range_slice requests over 6 to 8 hours that load about 10 to 15 GB of data.
Yesterday while there was no activity I noticed 2 nodes sitting at 200% CPU on 8 Core machine. Thought nothing of it. Checked again this morning and they are still sitting at that level of activity with no requests going into them. Checking the streams using node tool I see node 3 is streaming to node 0 and 2, and appears to have stalled. The information in the JMX console for streams matches the info below. I cannot see any errors in the logs. This is just a test system, so am happy to bounce the JVM's. Before I do is there anything else I should be looking for to understand why this happened? Also, sorry for the previous empty email. Node 0 Mode: Normal Nothing streaming to /192.168.34.27 Nothing streaming to /192.168.34.28 Nothing streaming to /192.168.34.29 Streaming from: /192.168.34.29 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Filter.db 0/22765 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Data.db 0/10750717 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Index.db 0/58 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Filter.db 0/325 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-108-Data.db 0/695 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Index.db 0/58 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Filter.db 0/325 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-119-Data.db 0/695 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Index.db 0/587164 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Filter.db 0/22765 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-163-Data.db 0/5159652 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-124-Data.db 22765/4966927 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Index.db 22765/1223053 Node 1 Mode: Normal Nothing streaming to /192.168.34.26 Nothing streaming to /192.168.34.28 Nothing streaming to /192.168.34.29 Not receiving any streams. Node 2 Mode: Normal Nothing streaming to /192.168.34.26 Nothing streaming to /192.168.34.27 Nothing streaming to /192.168.34.29 Streaming from: /192.168.34.29 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Filter.db 0/22765 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-137-Data.db 0/2161660 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Index.db 0/787524 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Filter.db 0/22765 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-147-Data.db 0/6917064 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Index.db 0/58 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Filter.db 0/565 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Databases-tmp-130-Data.db 0/695 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Index.db 0/581779 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Filter.db 0/22765 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-168-Data.db 0/5111887 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-125-Data.db 361367/3173057 junkbox.mycompany: /local1/junkbox/cassandra/data/junkbox.mycompany/Buckets-tmp-125-Index.db 695/361367 Node 3 ode: Normal Streaming to: /192.168.34.26 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-69-Filter.db 22765/22765 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-69-Data.db 0/4966927 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Index.db 0/58 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Filter.db 0/325 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-42-Data.db 0/695 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Index.db 0/1223053 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Filter.db 0/22765 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-82-Data.db 0/10750717 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Index.db 0/58 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Filter.db 0/325 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-52-Data.db 0/695 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Index.db 0/58 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Filter.db 0/325 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-61-Data.db 0/695 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Index.db 0/587164 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Filter.db 0/22765 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-111-Data.db 0/5159652 Nothing streaming to /192.168.34.27 Streaming to: /192.168.34.28 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Index.db 361367/361367 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Filter.db 0/22765 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-68-Data.db 0/3173057 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Index.db 0/58 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Filter.db 0/325 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-43-Data.db 0/695 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Index.db 0/246014 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Filter.db 0/22765 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-83-Data.db 0/2161660 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Index.db 0/787524 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Filter.db 0/22765 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-93-Data.db 0/6917064 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Index.db 0/58 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Filter.db 0/565 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Databases-63-Data.db 0/695 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Index.db 0/581779 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Filter.db 0/22765 /local1/junkbox/cassandra/data/junkbox.mycompany/stream/Buckets-112-Data.db 0/5111887 Not receiving any streams.
