So it turns out it's not socket related and it seems to only happen on my
second new node only not the first one with a higher load.
I decommissioned the new node and replaced it with another new node from
scratch but the issue persists: clients continue to timeout on it and
eventual the node becomes unavailable after a while requiring a restart.
I've gone as far as I can trying to debug it so would greatly appreciate any
insight. Let me know if there's anything else you need to know.
Suhail
On Sun, Jan 31, 2010 at 2:55 PM, Suhail Doshi <suh...@mixpanel.com> wrote:
> Encountering some new issues while running clients against Cassandra. For
> some reason as time goes Cassandra in my case starts to throw timeout
> exceptions to the point where even if one client is connected it processes
> things *very* slow and often throws the timeout exception frequently. My
> clients do: Write ops: ~2.5-4.5k, Read ops: ~7-10ku
>
> Does anyone have a clue what might cause the timeout errors? Things seem to
> get progressively worse as time goes on.
>
> I wrote my own client using Thrift's python output of the thrift interface.
> Pretty standard and close to the example. open transport -> execute query ->
> close transport. I am wondering if possibly I shouldn't close transport on
> every query and leave it open. Potentially opening and closing the socket to
> the thrift server could cause problems at scale?
>
> I checked the usual suspects: iostat, vmstat -- all is fine. Plenty of memory
> on the 2 64-bit 8 G machines.
>
>
>
> First run
>
> ----------------------
> total,interval_op_rate,avg_latency,elapsed_time28727,2872,0.0175149779964,1061622,3289,0.0146141170531,20101748,4012,0.0119166439863,30137757,3600,0.0135236120405,40175646,3788,0.012693485803,50213738,3809,0.0126799201372,60245012,3127,0.0155579636391,70285269,4025,0.0118225181672,80324749,3948,0.0121112306852,90355683,3093,0.0157108579878,100388473,3279,0.0148143836004,110426829,3835,0.0125468635425,120463487,3665,0.0133001198517,130502343,3885,0.0124575762634,140533697,3135,0.0154061204143,150567714,3401,0.0135120152761,160603673,3595,0.0141025309853,170643477,3980,0.0120242987251,180683208,3973,0.0120188054736,190723563,4035,0.0118774686127,200753554,2999,0.0160202514827,210787872,3431,0.0141030247447,220828373,4050,0.0118379161928,230868246,3987,0.012024458323,240907380,3913,0.0122560267615,250937451,3007,0.01239524122,260975023,3757,0.0156890653541,2701000000,2497,0.0130479447468,280
> Second run (ho later)
>
> ----------------------
> total,interval_op_rate,avg_latency,elapsed_time25631,2563,0.0192476335427,1059575,3394,0.0139286767445,2095670,3609,0.0132010291332,30111244,1557,0.0317069589671,40131045,1980,0.0245688458379,50141317,1027,0.0364191305368,60150975,965,0.0632473327625,70183505,3253,0.0147324700155,80215452,3194,0.0148266515631,90252952,3750,0.0128544898542,100286385,3343,0.0143946225061,110322792,3640,0.0128910038463,120338928,1613,0.0288048155041,130369871,3094,0.0167357261351,140398297,2842,0.017077029762,150415342,1704,0.0288302706917,160429611,1426,0.0346701375921,170461401,3179,0.0147932940138,180Process
> Inserter-22:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()478302,1690,0.0294986360419,190511967,3366,0.014146656114,200544990,3302,0.0143403996201,210579351,3436,0.0138775690627,220612898,3354,0.0141493124013,230647257,3435,0.0137730139321,240669652,2239,0.0131938768354,250685008,1535,0.0435601793934,260695135,1012,0.0371799632852,270719213,2407,0.0244178672953,280753296,3408,0.0139683472405,290786566,3327,0.0142738517283,300812512,2594,0.0146942751213,310831476,1896,0.0302928054768,320862522,3104,0.0150721740732,330893463,3094,0.0155329019027,340910865,1740,0.0276735682636,350915065,420,0.0715707530862,360927336,1227,0.0543709511221,370944975,1763,0.0258861654665,380963617,1864,0.0244650796636,390981487,1787,0.028689940935,400989352,786,0.00684437982155,410
> Third run (3 hours later)Note: According to jconsole write operations were <
> 10,000 but during the first run I saw them peak at 80,000. Huge
> difference.----------------------
> total,interval_op_rate,avg_latency,elapsed_timeProcess Inserter-26:Traceback
> (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertProcess Inserter-4:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 600, in recv_batch_insert
> raise result.ueUnavailableException: UnavailableException()Process
> Inserter-6:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 600, in recv_batch_insert
> raise result.ueUnavailableException: UnavailableException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 600, in recv_batch_insert
> raise result.ueUnavailableException:
> UnavailableException()9892,989,0.0452791625702,1013884,399,0.0905211865902,20Process
> Inserter-9:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()16919,303,0.113362641861,3017398,47,1.34807682635,4017735,33,0.934142259179,5018103,36,1.70099910122,6018596,49,0.666715461874,7018867,27,2.30892864716,80Process
> Inserter-45:Traceback (most recent call last):Process Inserter-23:Traceback
> (most recent call last):Process Inserter-29:Process Inserter-32:Traceback
> (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.run()
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertProcess Inserter-20:Traceback (most recent call
> last):Process Inserter-42:Process Inserter-33:Traceback (most recent call
> last):Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.run()
> File "stress.py", line 167, in run
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTraceback (most recent call last):TimedOutException:
> TimedOutException()
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in runProcess Inserter-34:Traceback (most
> recent call last):Process Inserter-8:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertProcess Inserter-24:Traceback (most recent call
> last):
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrapProcess Inserter-47:Traceback (most recent call last):
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process
> Inserter-36:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertProcess Inserter-1:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrapProcess Inserter-2:Traceback (most recent call last):
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> raise result.teTimedOutException: TimedOutException()
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process
> Inserter-37:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.run()
> File "stress.py", line 167, in run
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insertProcess Inserter-38:Traceback (most recent call
> last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> raise result.teTimedOutException: TimedOutException()
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertProcess Inserter-7:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()19660,79,0.259248053742,9026066,640,0.0368563048809,10034222,815,0.0542649867606,11037799,357,0.0611899578275,12042595,479,0.047750012292,13042814,21,1.82856085638,14042863,4,4.05655763101,150Process
> Inserter-18:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in runProcess Inserter-39:Traceback (most
> recent call last):
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process
> Inserter-12:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertProcess Inserter-19:Traceback (most recent call
> last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process
> Inserter-35:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertProcess Inserter-41:Traceback (most recent call
> last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process Inserter-15:
> self.run()Traceback (most recent call last):
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()43380,51,0.592962222588,16043556,17,0.85316223448,17043582,2,5.76186679877,18044110,52,0.574297892776,19044143,3,4.52656103625,20044799,65,0.46613632397,210
>
>
> 44828,2,5.20553808377,22044976,14,2.0302237675,230Process
> Inserter-31:Traceback (most recent call last):Process Inserter-11:Traceback
> (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertProcess Inserter-30:Traceback (most recent call
> last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrapProcess Inserter-27:Traceback (most recent call last):
> self.run()
> File "stress.py", line 167, in run
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process
> Inserter-46:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()45324,34,0.361710271616,240Process Inserter-44:Traceback
> (most recent call last):Process Inserter-21:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertProcess Inserter-22:Traceback (most recent call
> last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()Process
> Inserter-49:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()45394,7,2.14389359951,250Process Inserter-10:Traceback
> (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()45413,1,6.80838800731,26045836,42,0.212721344427,270Process
> Inserter-13:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()45872,3,4.39313924313,28045899,2,2.80208562039,29046025,12,1.03094763226,30046036,1,6.91312122345,310Process
> Inserter-5:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()46041,0,9.6405497551,32046937,89,0.210266448291,33047675,73,0.0936150796368,34049555,188,0.0744233962069,35050292,73,0.0938777962404,36051306,101,0.136950609717,37052649,134,0.0525897449286,38053247,59,0.229411067771,39053888,64,0.107282243541,40054971,108,0.0673501024836,410Process
> Inserter-28:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertProcess Inserter-3:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertProcess Inserter-16:Traceback (most recent call
> last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.te
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insertTimedOutException: TimedOutException()
> raise result.teTimedOutException:
> TimedOutException()55637,66,0.174130035234,42055875,23,0.201759393476,43057258,138,0.0706880513258,44057522,26,0.182813448436,450Process
> Inserter-25:Traceback (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in runProcess Inserter-17:Traceback (most
> recent call last):
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()58338,81,0.102476425615,46058565,22,0.151250916956,47058695,13,0.266876726884,48059481,78,0.0896081578641,49059767,28,0.121027883116,50061782,201,0.0350987749123,51066830,504,0.00852425800262,52067753,92,0.0383770502014,53067789,3,1.9088866512,54067789,0,NAN,550Process
> Inserter-50:Process Inserter-40:Traceback (most recent call last):Traceback
> (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in runProcess Inserter-14:
> self.run()
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File "stress.py", line 167, in run
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insertTraceback (most recent call last):
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException: TimedOutException()
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()67902,11,0.370109583424,560Process Inserter-48:Traceback
> (most recent call last):
> File
> "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
> line 236, in _bootstrap
> self.run()
> File "stress.py", line 167, in run
> self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 574, in batch_insert
> self.recv_batch_insert()
> File
> "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
> line 602, in recv_batch_insert
> raise result.teTimedOutException:
> TimedOutException()67903,0,6.85239100456,57068002,9,0.13895741376,58068003,0,6.84438991547,590
>
> Extra information:
>
> 1st no clients were connected when it was run
> 2nd run, stress test was conducted when there were 50+ clients doing 7k read,
> 3-4k write operations
> 3rd was run after clients were killed as problems showed up
> After 3rd run looked at tpstats for seed-node:Pool Name
> Active Pending CompletedFILEUTILS-DELETE-POOL 0 0
> 442STREAM-STAGE 0 0
> 0RESPONSE-STAGE 0 0 1427177ROW-READ-STAGE
> 0 0 3705481LB-OPERATIONS
> 0 0 0COMMITLOG 1 0
> 5133614MESSAGE-DESERIALIZER-POOL 0 0 3456220GMFD
> 0 0 34882LB-TARGET
> 0 0 0CONSISTENCY-MANAGER 0 0
> 0ROW-MUTATION-STAGE 0 0
> 5452268MESSAGE-STREAMING-POOL 0 0
> 0LOAD-BALANCER-STAGE 0 0
> 0FLUSH-SORTER-POOL 0 0
> 118MEMTABLE-POST-FLUSHER 0 0
> 118FLUSH-WRITER-POOL 0 0
> 118AE-SERVICE-STAGE 0 0
> 1HINTED-HANDOFF-POOL 0 0 41
>
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
