Encountering some new issues while running clients against Cassandra.
For some reason as time goes Cassandra in my case starts to throw
timeout exceptions to the point where even if one client is connected
it processes things *very* slow and often throws the timeout exception
frequently. My clients do: Write ops: ~2.5-4.5k, Read ops: ~7-10ku
Does anyone have a clue what might cause the timeout errors? Things
seem to get progressively worse as time goes on.
I wrote my own client using Thrift's python output of the thrift
interface. Pretty standard and close to the example. open transport ->
execute query -> close transport. I am wondering if possibly I
shouldn't close transport on every query and leave it open.
Potentially opening and closing the socket to the thrift server could
cause problems at scale?
I checked the usual suspects: iostat, vmstat -- all is fine. Plenty of
memory on the 2 64-bit 8 G machines.
First run
----------------------
total,interval_op_rate,avg_latency,elapsed_time28727,2872,0.0175149779964,1061622,3289,0.0146141170531,20101748,4012,0.0119166439863,30137757,3600,0.0135236120405,40175646,3788,0.012693485803,50213738,3809,0.0126799201372,60245012,3127,0.0155579636391,70285269,4025,0.0118225181672,80324749,3948,0.0121112306852,90355683,3093,0.0157108579878,100388473,3279,0.0148143836004,110426829,3835,0.0125468635425,120463487,3665,0.0133001198517,130502343,3885,0.0124575762634,140533697,3135,0.0154061204143,150567714,3401,0.0135120152761,160603673,3595,0.0141025309853,170643477,3980,0.0120242987251,180683208,3973,0.0120188054736,190723563,4035,0.0118774686127,200753554,2999,0.0160202514827,210787872,3431,0.0141030247447,220828373,4050,0.0118379161928,230868246,3987,0.012024458323,240907380,3913,0.0122560267615,250937451,3007,0.01239524122,260975023,3757,0.0156890653541,2701000000,2497,0.0130479447468,280
Second run (ho later)
----------------------
total,interval_op_rate,avg_latency,elapsed_time25631,2563,0.0192476335427,1059575,3394,0.0139286767445,2095670,3609,0.0132010291332,30111244,1557,0.0317069589671,40131045,1980,0.0245688458379,50141317,1027,0.0364191305368,60150975,965,0.0632473327625,70183505,3253,0.0147324700155,80215452,3194,0.0148266515631,90252952,3750,0.0128544898542,100286385,3343,0.0143946225061,110322792,3640,0.0128910038463,120338928,1613,0.0288048155041,130369871,3094,0.0167357261351,140398297,2842,0.017077029762,150415342,1704,0.0288302706917,160429611,1426,0.0346701375921,170461401,3179,0.0147932940138,180Process
Inserter-22:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()478302,1690,0.0294986360419,190511967,3366,0.014146656114,200544990,3302,0.0143403996201,210579351,3436,0.0138775690627,220612898,3354,0.0141493124013,230647257,3435,0.0137730139321,240669652,2239,0.0131938768354,250685008,1535,0.0435601793934,260695135,1012,0.0371799632852,270719213,2407,0.0244178672953,280753296,3408,0.0139683472405,290786566,3327,0.0142738517283,300812512,2594,0.0146942751213,310831476,1896,0.0302928054768,320862522,3104,0.0150721740732,330893463,3094,0.0155329019027,340910865,1740,0.0276735682636,350915065,420,0.0715707530862,360927336,1227,0.0543709511221,370944975,1763,0.0258861654665,380963617,1864,0.0244650796636,390981487,1787,0.028689940935,400989352,786,0.00684437982155,410
Third run (3 hours later)Note: According to jconsole write operations
were < 10,000 but during the first run I saw them peak at 80,000. Huge
difference.----------------------
total,interval_op_rate,avg_latency,elapsed_timeProcess
Inserter-26:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-4:Traceback (most recent
call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 600, in recv_batch_insert
raise result.ueUnavailableException: UnavailableException()Process
Inserter-6:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 600, in recv_batch_insert
raise result.ueUnavailableException: UnavailableException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 600, in recv_batch_insert
raise result.ueUnavailableException:
UnavailableException()9892,989,0.0452791625702,1013884,399,0.0905211865902,20Process
Inserter-9:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()16919,303,0.113362641861,3017398,47,1.34807682635,4017735,33,0.934142259179,5018103,36,1.70099910122,6018596,49,0.666715461874,7018867,27,2.30892864716,80Process
Inserter-45:Traceback (most recent call last):Process
Inserter-23:Traceback (most recent call last):Process
Inserter-29:Process Inserter-32:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.run()
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-20:Traceback (most recent
call last):Process Inserter-42:Process Inserter-33:Traceback (most
recent call last):Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.run()
File "stress.py", line 167, in run
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTraceback (most recent call
last):TimedOutException: TimedOutException()
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in runProcess Inserter-34:Traceback
(most recent call last):Process Inserter-8:Traceback (most recent call
last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-24:Traceback (most recent
call last):
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-47:Traceback (most recent call
last):
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()Process
Inserter-36:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-1:Traceback (most recent
call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-2:Traceback (most recent call
last):
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
raise result.teTimedOutException: TimedOutException()
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()Process
Inserter-37:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.run()
File "stress.py", line 167, in run
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insertProcess Inserter-38:Traceback (most
recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
raise result.teTimedOutException: TimedOutException()
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-7:Traceback (most recent
call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()19660,79,0.259248053742,9026066,640,0.0368563048809,10034222,815,0.0542649867606,11037799,357,0.0611899578275,12042595,479,0.047750012292,13042814,21,1.82856085638,14042863,4,4.05655763101,150Process
Inserter-18:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in runProcess Inserter-39:Traceback
(most recent call last):
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()Process
Inserter-12:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-19:Traceback (most recent
call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()Process
Inserter-35:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-41:Traceback (most recent
call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()Process Inserter-15:
self.run()Traceback (most recent call last):
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()43380,51,0.592962222588,16043556,17,0.85316223448,17043582,2,5.76186679877,18044110,52,0.574297892776,19044143,3,4.52656103625,20044799,65,0.46613632397,210
44828,2,5.20553808377,22044976,14,2.0302237675,230Process
Inserter-31:Traceback (most recent call last):Process
Inserter-11:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-30:Traceback (most recent
call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-27:Traceback (most recent call
last):
self.run()
File "stress.py", line 167, in run
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()Process
Inserter-46:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()45324,34,0.361710271616,240Process
Inserter-44:Traceback (most recent call last):Process
Inserter-21:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-22:Traceback (most recent
call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()Process
Inserter-49:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()45394,7,2.14389359951,250Process
Inserter-10:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()45413,1,6.80838800731,26045836,42,0.212721344427,270Process
Inserter-13:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()45872,3,4.39313924313,28045899,2,2.80208562039,29046025,12,1.03094763226,30046036,1,6.91312122345,310Process
Inserter-5:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()46041,0,9.6405497551,32046937,89,0.210266448291,33047675,73,0.0936150796368,34049555,188,0.0744233962069,35050292,73,0.0938777962404,36051306,101,0.136950609717,37052649,134,0.0525897449286,38053247,59,0.229411067771,39053888,64,0.107282243541,40054971,108,0.0673501024836,410Process
Inserter-28:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-3:Traceback (most recent
call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-16:Traceback (most recent
call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.te
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insertTimedOutException: TimedOutException()
raise result.teTimedOutException:
TimedOutException()55637,66,0.174130035234,42055875,23,0.201759393476,43057258,138,0.0706880513258,44057522,26,0.182813448436,450Process
Inserter-25:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in runProcess Inserter-17:Traceback
(most recent call last):
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()58338,81,0.102476425615,46058565,22,0.151250916956,47058695,13,0.266876726884,48059481,78,0.0896081578641,49059767,28,0.121027883116,50061782,201,0.0350987749123,51066830,504,0.00852425800262,52067753,92,0.0383770502014,53067789,3,1.9088866512,54067789,0,NAN,550Process
Inserter-50:Process Inserter-40:Traceback (most recent call
last):Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in runProcess Inserter-14:
self.run()
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File "stress.py", line 167, in run
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertTraceback (most recent call last):
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException: TimedOutException()
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()67902,11,0.370109583424,560Process
Inserter-48:Traceback (most recent call last):
File
"/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File "stress.py", line 167, in run
self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
self.recv_batch_insert()
File
"/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
raise result.teTimedOutException:
TimedOutException()67903,0,6.85239100456,57068002,9,0.13895741376,58068003,0,6.84438991547,590
Extra information:
1st no clients were connected when it was run
2nd run, stress test was conducted when there were 50+ clients doing
7k read, 3-4k write operations
3rd was run after clients were killed as problems showed up
After 3rd run looked at tpstats for seed-node:Pool Name
Active Pending CompletedFILEUTILS-DELETE-POOL 0
0 442STREAM-STAGE 0 0
0RESPONSE-STAGE 0 0
1427177ROW-READ-STAGE 0 0
3705481LB-OPERATIONS 0 0
0COMMITLOG 1 0
5133614MESSAGE-DESERIALIZER-POOL 0 0
3456220GMFD 0 0
34882LB-TARGET 0 0
0CONSISTENCY-MANAGER 0 0
0ROW-MUTATION-STAGE 0 0
5452268MESSAGE-STREAMING-POOL 0 0
0LOAD-BALANCER-STAGE 0 0
0FLUSH-SORTER-POOL 0 0
118MEMTABLE-POST-FLUSHER 0 0
118FLUSH-WRITER-POOL 0 0
118AE-SERVICE-STAGE 0 0
1HINTED-HANDOFF-POOL 0 0 41
