One last thing, timeouts start happening to happen more frequently either
with time or just more reads/writes.
Suhail
On Mon, Feb 1, 2010 at 9:31 AM, Suhail Doshi <suh...@mixpanel.com> wrote:
> So it turns out it's not socket related and it seems to only happen on my
> second new node only not the first one with a higher load.
>
> I decommissioned the new node and replaced it with another new node from
> scratch but the issue persists: clients continue to timeout on it and
> eventual the node becomes unavailable after a while requiring a restart.
>
> I've gone as far as I can trying to debug it so would greatly appreciate
> any insight. Let me know if there's anything else you need to know.
>
> Suhail
>
> On Sun, Jan 31, 2010 at 2:55 PM, Suhail Doshi <suh...@mixpanel.com>wrote:
>
>> Encountering some new issues while running clients against Cassandra. For
>> some reason as time goes Cassandra in my case starts to throw timeout
>> exceptions to the point where even if one client is connected it processes
>> things *very* slow and often throws the timeout exception frequently. My
>> clients do: Write ops: ~2.5-4.5k, Read ops: ~7-10ku
>>
>> Does anyone have a clue what might cause the timeout errors? Things seem to
>> get progressively worse as time goes on.
>>
>> I wrote my own client using Thrift's python output of the thrift interface.
>> Pretty standard and close to the example. open transport -> execute query ->
>> close transport. I am wondering if possibly I shouldn't close transport on
>> every query and leave it open. Potentially opening and closing the socket to
>> the thrift server could cause problems at scale?
>>
>> I checked the usual suspects: iostat, vmstat -- all is fine. Plenty of
>> memory on the 2 64-bit 8 G machines.
>>
>>
>>
>>
>> First run
>>
>> ----------------------
>> total,interval_op_rate,avg_latency,elapsed_time28727,2872,0.0175149779964,1061622,3289,0.0146141170531,20101748,4012,0.0119166439863,30137757,3600,0.0135236120405,40175646,3788,0.012693485803,50213738,3809,0.0126799201372,60245012,3127,0.0155579636391,70285269,4025,0.0118225181672,80324749,3948,0.0121112306852,90355683,3093,0.0157108579878,100388473,3279,0.0148143836004,110426829,3835,0.0125468635425,120463487,3665,0.0133001198517,130502343,3885,0.0124575762634,140533697,3135,0.0154061204143,150567714,3401,0.0135120152761,160603673,3595,0.0141025309853,170643477,3980,0.0120242987251,180683208,3973,0.0120188054736,190723563,4035,0.0118774686127,200753554,2999,0.0160202514827,210787872,3431,0.0141030247447,220828373,4050,0.0118379161928,230868246,3987,0.012024458323,240907380,3913,0.0122560267615,250937451,3007,0.01239524122,260975023,3757,0.0156890653541,2701000000,2497,0.0130479447468,280
>> Second run (ho later)
>>
>> ----------------------
>> total,interval_op_rate,avg_latency,elapsed_time25631,2563,0.0192476335427,1059575,3394,0.0139286767445,2095670,3609,0.0132010291332,30111244,1557,0.0317069589671,40131045,1980,0.0245688458379,50141317,1027,0.0364191305368,60150975,965,0.0632473327625,70183505,3253,0.0147324700155,80215452,3194,0.0148266515631,90252952,3750,0.0128544898542,100286385,3343,0.0143946225061,110322792,3640,0.0128910038463,120338928,1613,0.0288048155041,130369871,3094,0.0167357261351,140398297,2842,0.017077029762,150415342,1704,0.0288302706917,160429611,1426,0.0346701375921,170461401,3179,0.0147932940138,180Process
>> Inserter-22:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()478302,1690,0.0294986360419,190511967,3366,0.014146656114,200544990,3302,0.0143403996201,210579351,3436,0.0138775690627,220612898,3354,0.0141493124013,230647257,3435,0.0137730139321,240669652,2239,0.0131938768354,250685008,1535,0.0435601793934,260695135,1012,0.0371799632852,270719213,2407,0.0244178672953,280753296,3408,0.0139683472405,290786566,3327,0.0142738517283,300812512,2594,0.0146942751213,310831476,1896,0.0302928054768,320862522,3104,0.0150721740732,330893463,3094,0.0155329019027,340910865,1740,0.0276735682636,350915065,420,0.0715707530862,360927336,1227,0.0543709511221,370944975,1763,0.0258861654665,380963617,1864,0.0244650796636,390981487,1787,0.028689940935,400989352,786,0.00684437982155,410
>> Third run (3 hours later)Note: According to jconsole write operations were <
>> 10,000 but during the first run I saw them peak at 80,000. Huge
>> difference.----------------------
>> total,interval_op_rate,avg_latency,elapsed_timeProcess Inserter-26:Traceback
>> (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertProcess Inserter-4:Traceback (most recent call
>> last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 600, in recv_batch_insert
>> raise result.ueUnavailableException: UnavailableException()Process
>> Inserter-6:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 600, in recv_batch_insert
>> raise result.ueUnavailableException: UnavailableException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 600, in recv_batch_insert
>> raise result.ueUnavailableException:
>> UnavailableException()9892,989,0.0452791625702,1013884,399,0.0905211865902,20Process
>> Inserter-9:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()16919,303,0.113362641861,3017398,47,1.34807682635,4017735,33,0.934142259179,5018103,36,1.70099910122,6018596,49,0.666715461874,7018867,27,2.30892864716,80Process
>> Inserter-45:Traceback (most recent call last):Process Inserter-23:Traceback
>> (most recent call last):Process Inserter-29:Process Inserter-32:Traceback
>> (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.run()
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertProcess Inserter-20:Traceback (most recent call
>> last):Process Inserter-42:Process Inserter-33:Traceback (most recent call
>> last):Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.run()
>> File "stress.py", line 167, in run
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTraceback (most recent call last):TimedOutException:
>> TimedOutException()
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in runProcess Inserter-34:Traceback (most
>> recent call last):Process Inserter-8:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertProcess Inserter-24:Traceback (most recent call
>> last):
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrapProcess Inserter-47:Traceback (most recent call
>> last):
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process
>> Inserter-36:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertProcess Inserter-1:Traceback (most recent call
>> last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrapProcess Inserter-2:Traceback (most recent call last):
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> raise result.teTimedOutException: TimedOutException()
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process
>> Inserter-37:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.run()
>> File "stress.py", line 167, in run
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insertProcess Inserter-38:Traceback (most recent
>> call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> raise result.teTimedOutException: TimedOutException()
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertProcess Inserter-7:Traceback (most recent call
>> last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()19660,79,0.259248053742,9026066,640,0.0368563048809,10034222,815,0.0542649867606,11037799,357,0.0611899578275,12042595,479,0.047750012292,13042814,21,1.82856085638,14042863,4,4.05655763101,150Process
>> Inserter-18:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in runProcess Inserter-39:Traceback (most
>> recent call last):
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process
>> Inserter-12:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertProcess Inserter-19:Traceback (most recent call
>> last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process
>> Inserter-35:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertProcess Inserter-41:Traceback (most recent call
>> last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process Inserter-15:
>> self.run()Traceback (most recent call last):
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()43380,51,0.592962222588,16043556,17,0.85316223448,17043582,2,5.76186679877,18044110,52,0.574297892776,19044143,3,4.52656103625,20044799,65,0.46613632397,210
>>
>>
>> 44828,2,5.20553808377,22044976,14,2.0302237675,230Process
>> Inserter-31:Traceback (most recent call last):Process Inserter-11:Traceback
>> (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertProcess Inserter-30:Traceback (most recent call
>> last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrapProcess Inserter-27:Traceback (most recent call
>> last):
>> self.run()
>> File "stress.py", line 167, in run
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process
>> Inserter-46:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()45324,34,0.361710271616,240Process Inserter-44:Traceback
>> (most recent call last):Process Inserter-21:Traceback (most recent call
>> last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertProcess Inserter-22:Traceback (most recent call
>> last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()Process
>> Inserter-49:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()45394,7,2.14389359951,250Process Inserter-10:Traceback
>> (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()45413,1,6.80838800731,26045836,42,0.212721344427,270Process
>> Inserter-13:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()45872,3,4.39313924313,28045899,2,2.80208562039,29046025,12,1.03094763226,30046036,1,6.91312122345,310Process
>> Inserter-5:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()46041,0,9.6405497551,32046937,89,0.210266448291,33047675,73,0.0936150796368,34049555,188,0.0744233962069,35050292,73,0.0938777962404,36051306,101,0.136950609717,37052649,134,0.0525897449286,38053247,59,0.229411067771,39053888,64,0.107282243541,40054971,108,0.0673501024836,410Process
>> Inserter-28:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertProcess Inserter-3:Traceback (most recent call
>> last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertProcess Inserter-16:Traceback (most recent call
>> last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.te
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insertTimedOutException: TimedOutException()
>> raise result.teTimedOutException:
>> TimedOutException()55637,66,0.174130035234,42055875,23,0.201759393476,43057258,138,0.0706880513258,44057522,26,0.182813448436,450Process
>> Inserter-25:Traceback (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in runProcess Inserter-17:Traceback (most
>> recent call last):
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()58338,81,0.102476425615,46058565,22,0.151250916956,47058695,13,0.266876726884,48059481,78,0.0896081578641,49059767,28,0.121027883116,50061782,201,0.0350987749123,51066830,504,0.00852425800262,52067753,92,0.0383770502014,53067789,3,1.9088866512,54067789,0,NAN,550Process
>> Inserter-50:Process Inserter-40:Traceback (most recent call last):Traceback
>> (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in runProcess Inserter-14:
>> self.run()
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File "stress.py", line 167, in run
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insertTraceback (most recent call last):
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException: TimedOutException()
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()67902,11,0.370109583424,560Process Inserter-48:Traceback
>> (most recent call last):
>> File
>> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
>> line 236, in _bootstrap
>> self.run()
>> File "stress.py", line 167, in run
>> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 574, in batch_insert
>> self.recv_batch_insert()
>> File
>> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
>> line 602, in recv_batch_insert
>> raise result.teTimedOutException:
>> TimedOutException()67903,0,6.85239100456,57068002,9,0.13895741376,58068003,0,6.84438991547,590
>>
>> Extra information:
>>
>> 1st no clients were connected when it was run
>> 2nd run, stress test was conducted when there were 50+ clients doing 7k
>> read, 3-4k write operations
>> 3rd was run after clients were killed as problems showed up
>> After 3rd run looked at tpstats for seed-node:Pool Name
>> Active Pending CompletedFILEUTILS-DELETE-POOL 0 0
>> 442STREAM-STAGE 0 0
>> 0RESPONSE-STAGE 0 0 1427177ROW-READ-STAGE
>> 0 0 3705481LB-OPERATIONS
>> 0 0 0COMMITLOG 1 0
>> 5133614MESSAGE-DESERIALIZER-POOL 0 0 3456220GMFD
>> 0 0 34882LB-TARGET
>> 0 0 0CONSISTENCY-MANAGER 0
>> 0 0ROW-MUTATION-STAGE 0 0
>> 5452268MESSAGE-STREAMING-POOL 0 0
>> 0LOAD-BALANCER-STAGE 0 0
>> 0FLUSH-SORTER-POOL 0 0
>> 118MEMTABLE-POST-FLUSHER 0 0
>> 118FLUSH-WRITER-POOL 0 0
>> 118AE-SERVICE-STAGE 0 0
>> 1HINTED-HANDOFF-POOL 0 0 41
>>
>>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
