I have a large Cassandra 2.1.13 ring (60 nodes) in AWS that has consistently random high read times. In general most reads are under 10 milliseconds but with in the 30 request there is usually a read time that is a couple of seconds.
Instance type: r4.8xlarge EBS GP2 volumes, 3tb with 9000 IOPS 30 Gig Heap Data per node is about 170 gigs The keyspace is an id & a blob. When I check the data the slow reads don’t seem to have anything to do with size of the blobs This system has repairs run once a weeks because it takes a lot of updates. The client makes a call and does 30 request serially to Cassandra and the response times look like this in milliseconds. What could make these so slow and what can I do to diagnosis this? Responses Get Person time: 3 319746229:9009:66 Get Person time: 7 1830093695:9009:66 Get Person time: 4 30072253:9009:66 Get Person time: 4 2303790089:9009:66 Get Person time: 2 156792066:9009:66 Get Person time: 8 491230624:9009:66 Get Person time: 7 284904599:9009:66 Get Person time: 4 600370489:9009:66 Get Person time: 2 281007386:9009:66 Get Person time: 4 971178094:9009:66 Get Person time: 1 1322259885:9009:66 Get Person time: 2 1937958542:9009:66 Get Person time: 9 286536648:9009:66 Get Person time: 9 1835633470:9009:66 Get Person time: 2 300867513:9009:66 Get Person time: 3 178975468:9009:66 Get Person time: 2900 293043081:9009:66 Get Person time: 8 214913830:9009:66 Get Person time: 2 1956710764:9009:66 Get Person time: 4 237673776:9009:66 Get Person time: 17 68942206:9009:66 Get Person time: 1800 20072145:9009:66 Get Person time: 2 304698506:9009:66 Get Person time: 2 308177320:9009:66 Get Person time: 2 998436038:9009:66 Get Person time: 10 1036890112:9009:66 Get Person time: 1 1629649548:9009:66 Get Person time: 6 1595339706:9009:66 Get Person time: 4 1079637599:9009:66 Get Person time: 3 556342855:9009:66 Get Person time: 5 1856382256:9009:66 Get Person time: 3 1891737174:9009:66 Get Person time: 2 1179373651:9009:66 Get Person time: 2 1482602756:9009:66 Get Person time: 3 1236458510:9009:66 Get Person time: 11 1003159823:9009:66 Get Person time: 2 1264952556:9009:66 Get Person time: 2 1662234295:9009:66 Get Person time: 1 246108569:9009:66 Get Person time: 5 1709881651:9009:66 Get Person time: 3213 11878078:9009:66 Get Person time: 2 112866483:9009:66 Get Person time: 2 201870153:9009:66 Get Person time: 6 227696684:9009:66 Get Person time: 2 1946780190:9009:66 Get Person time: 2 2197987101:9009:66 Get Person time: 18 1838959725:9009:66 Get Person time: 3 1782937802:9009:66 Get Person time: 3 1692530939:9009:66 Get Person time: 9 1765654196:9009:66 Get Person time: 2 1597757121:9009:66 Get Person time: 2 1853127153:9009:66 Get Person time: 3 1533599253:9009:66 Get Person time: 6 1693244112:9009:66 Get Person time: 6 82047537:9009:66 Get Person time: 2 96221961:9009:66 Get Person time: 4 98202209:9009:66 Get Person time: 9 12952388:9009:66 Get Person time: 2 300118652:9009:66 Get Person time: 10 78801084:9009:66 Get Person time: 13 1856424913:9009:66 Get Person time: 2 255814186:9009:66 Get Person time: 2 1183397424:9009:66 Get Person time: 5 1828603730:9009:66 Get Person time: 9 132965919:9009:66 Get Person time: 4 1616190071:9009:66 Get Person time: 2 15929337:9009:66 Get Person time: 10 297005427:9009:66 Get Person time: 2 1306460047:9009:66 Get Person time: 5 620139216:9009:66 Get Person time: 2 1364349058:9009:66 Get Person time: 3 629543403:9009:66 Get Person time: 5 1299827034:9009:66 Get Person time: 4 1593205912:9009:66 Get Person time: 2 1755460077:9009:66 Get Person time: 2 1906388666:9009:66 Get Person time: 1 1838653952:9009:66 Get Person time: 2 2249662508:9009:66 Get Person time: 3 1931708432:9009:66 Get Person time: 2 2177004948:9009:66 Get Person time: 2 2042756682:9009:66 Get Person time: 5 41764865:9009:66 Get Person time: 4023 1733384704:9009:66 Get Person time: 1 1614842189:9009:66 Get Person time: 2 2194211396:9009:66 Get Person time: 3 1711330834:9009:66 Get Person time: 2 2264849689:9009:66 Get Person time: 3 1819027970:9009:66 Get Person time: 2 1978614851:9009:66 Get Person time: 1 1863483129:9009:66
