Yulian,
Maybe other people have other clues, but I think if you could monitor the
behavior in tpstats after activity "Seeking to partition beginning in data
file" it could help to find the problem. Which type of thread is getting stuck?
Do you see any number increasing continuously during the request?
Best regards,
Marcelo.
From: user@cassandra.apache.org
Subject: Re:Cassandra Read Timeout
Hello to all
I have single node cassandra on amazon ec2.
Currently i am having a read timeout problem on single CF , single raw.
Raw size is aroung 190MB.There are bigger raws with similar structure ( its
index raws , which actually stores keys ) and everything is working fine on
them, everything is working also fine on this cf but on other raw.
Tables data from CFStats ( First table has bigger raws but works fine , where
second has timeout ) :
Column Family: pendindexes
SSTable count: 5
Space used (live): 462298352
Space used (total): 462306752
SSTable Compression Ratio: 0.3511107495795905
Number of Keys (estimate): 640
Memtable Columns Count: 63339
Memtable Data Size: 12328802
Memtable Switch Count: 78
Read Count: 10
Read Latency: NaN ms.
Write Count: 1530113
Write Latency: 0.022 ms.
Pending Tasks: 0
Bloom Filter False Positives: 0
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 3920
Compacted row minimum size: 73
Compacted row maximum size: 223875792
Compacted row mean size: 42694982
Average live cells per slice (last five minutes): 21.0
Average tombstones per slice (last five minutes): 0.0
Column Family: statuspindexes
SSTable count: 1
Space used (live): 99602136
Space used (total): 99609360
SSTable Compression Ratio: 0.34278775390997873
Number of Keys (estimate): 128
Memtable Columns Count: 6250
Memtable Data Size: 6061097
Memtable Switch Count: 65
Read Count: 1000
Read Latency: NaN ms.
Write Count: 1193142
Write Latency: 3.616 ms.
Pending Tasks: 0
Bloom Filter False Positives: 0
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 656
Compacted row minimum size: 180
Compacted row maximum size: 186563160
Compacted row mean size: 63225562
Average live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
I have tried to debug it with cql , thats what i get:
activity
| timestamp | source | source_elapsed
-------------------------------------------------------------------------------------------------+--------------+--------------+----------------
execute_cql3_query | 15:39:53,120 | 172.31.6.173 | 0
Parsing Select * from
statuspindexes LIMIT 10000; | 15:39:53,120 | 172.31.6.173 | 875
Preparing statement | 15:39:53,121 | 172.31.6.173 | 1643
Determining
replicas to query | 15:39:53,121 | 172.31.6.173 | 1740
Executing seq scan across 1 sstables for [min(-9223372036854775808),
min(-9223372036854775808)] | 15:39:53,122 | 172.31.6.173 | 2581
Seeking to partition
beginning in data file | 15:39:53,123 | 172.31.6.173 | 3118
Timed out; received 0 of 1 responses
for range 2 of 2 | 15:40:03,121 | 172.31.6.173 | 10001370
Request complete | 15:40:03,121 | 172.31.6.173 | 10001513
I have executed compaction on that cf.
What could lead to that problem?
Best regards
Yulian Oifa
