I didn't run nodetool scrub. My app uses c++ thrift client (0.5.0 and 0.6.1) . As this is production environment I get a lot of messages "collecting %s of %s", but there is no row key. I've matched it by uuid and thread - hope it is ok:
[ReadStage:3][org.apache.cassandra.db.filter.SliceQueryFilter] collecting 0 of 1000000: SuperColumn(455470c2-60e6-11e0-acae-e41f13798a50 [4554853a-60e6-11e0-9342-e41f13798a50:false:361745@0,aa92b386-60e6-11e0-bc58-e41f13798a50:false:159@0,ac346cc0-60e6-11e0-bf8e-e41f13798a50:false:53@0,ad16d362-60e6-11e0-ad9c-e41f13798a50:false:66@0,ae57076a-60e6-11e0-8982-e41f13798a50:false:48@0,afc042ba-60e6-11e0-8320-e41f13798a50:false:63@0,b38d912c-60e6-11e0-b3b7-e41f13798a50:false:164@0,49d7dc00-60e7-11e0-9694-e41f13798a50:false:100@0,94e99fa8-60e7-11e0-b621-e41f13798a50:false:233@0,9612c3a0-60e7-11e0-8292-e41f13798a50:false:4049@0,ec245880-60e7-11e0-85be-e41f13798a50:false:148@0,110ac968-60e8-11e0-b325-e41f13798a50:false:125@0,64a45b4c-60e9-11e0-9628-e41f13798a50:false:160@0,cfc39a00-60e9-11e0-9539-e41f13798a50:false:105@0,2a21ab22-60ea-11e0-a1f6-e41f13798a50:false:146@0,95f2d2d6-60ea-11e0-b763-e41f13798a50:false:53@0,97972362-60ea-11e0-9275-e41f13798a50:false:134@0,ce980606-60ea-11e0-bd03-e41f13798a50:false:195@0,517e02dc-60eb-11e0-b0a8-e41f13798a50:false:53@0,5694c74c-60eb-11e0-941f-e41f13798a50:false:170@0,8d48cca2-60eb-11e0-bdbc-e41f13798a50:false:187@0,fc5e0148-60eb-11e0-ac4c-e41f13798a50:false:558@0,fc7e9476-60eb-11e0-bbfe-e41f13798a50:false:161@0,22a860a0-60ec-11e0-8d1b-e41f13798a50:false:138@0,bf054b52-60ec-11e0-a4b1-e41f13798a50:false:56@0,fd3b4822-60ec-11e0-8612-e41f13798a50:false:234@0,0c1d6fe0-60ee-11e0-8bb0-e41f13798a50:false:79@0,43bbddec-60ee-11e0-848f-e41f13798a50:false:79@0,ec50ed8e-60ef-11e0-a8c1-e41f13798a50:false:60@0,57ae0534-60f1-11e0-8fd5-e41f13798a50:false:60@0,81d7bbe8-60f1-11e0-9586-e41f13798a50:false:143@0,08ce8852-60f2-11e0-bfeb-e41f13798a50:false:150@0,c51bf170-60f2-11e0-8a60-e41f13798a50:false:192@0,e98b181e-60f3-11e0-b20b-e41f13798a50:false:108@0,ed931d76-60f3-11e0-90cf-e41f13798a50:false:146@0,f754541a-60f3-11e0-84db-e41f13798a50:false:60@0,24a73220-60f4-11e0-ae03-e41f13798a50:false:139@0,33485db8-60f4-11e0-9f86-e41f13798a50:false:196@0,42fe380e-60f4-11e0-9f0f-e41f13798a50:false:150@0,440afbb0-60f4-11e0-a066-e41f13798a50:false:122@0,64d9d13a-60f5-11e0-a30c-e41f13798a50:false:60@0,9e1cefc2-60f5-11e0-826e-e41f13798a50:false:205@0,1f642a46-60f6-11e0-8aa3-e41f13798a50:false:298@0,1f6c946a-60f6-11e0-9eab-e41f13798a50:false:117@0,d1a4e8c6-60f6-11e0-9bfa-e41f13798a50:false:58@0,0a40ba5c-60f7-11e0-893f-e41f13798a50:false:170@0,a2e0f740-60f7-11e0-93b0-e41f13798a50:false:108@0,3922060e-60f8-11e0-b850-e41f13798a50:false:147@0,3cdcdf08-60f8-11e0-8320-e41f13798a50:false:60@0,79b33a26-60f8-11e0-b151-e41f13798a50:false:187@0,aae281c8-60f9-11e0-9d14-e41f13798a50:false:60@0,e1b3295a-60f9-11e0-b367-e41f13798a50:false:81@0,8149870c-60fa-11e0-b6d4-e41f13798a50:false:128@0,b483680e-60fa-11e0-a56f-e41f13798a50:false:164@0,3dfdfe78-60fb-11e0-8582-e41f13798a50:false:143@0,15a9b830-60fc-11e0-be81-e41f13798a50:false:217@0,46ae8aae-60fd-11e0-908e-e41f13798a50:false:60@0,7ee58490-60fd-11e0-9072-e41f13798a50:false:81@0,bf4fba30-60ff-11e0-9b5f-e41f13798a50:false:60@0,26201952-6101-11e0-a4cb-e41f13798a50:false:60@0,8e5e995c-6102-11e0-9bb3-e41f13798a50:false:60@0,a4ef01c0-6102-11e0-9bff-e41f13798a50:false:214@0,a5034784-6102-11e0-96ee-e41f13798a50:false:133@0,ad9f3682-6102-11e0-8df5-e41f13798a50:false:172@0,e2e01b36-6102-11e0-8489-e41f13798a50:false:155@0,25561204-6103-11e0-a14b-e41f13798a50:false:155@0,4d39d2a6-6103-11e0-844a-e41f13798a50:false:152@0,74aeb4fa-6103-11e0-9f65-e41f13798a50:false:140@0,7e8802ba-6103-11e0-81f1-e41f13798a50:false:140@0,8e6532f2-6103-11e0-acd4-e41f13798a50:false:141@0,fceadee8-6103-11e0-8beb-e41f13798a50:false:313@0,2d319618-6105-11e0-9efa-e41f13798a50:false:60@0,4d4540bc-6105-11e0-96d1-e41f13798a50:false:128@0,63fcaa02-6105-11e0-ac35-e41f13798a50:false:81@0,67c70114-6105-11e0-9ab8-e41f13798a50:false:167@0,3a104306-6106-11e0-8be9-e41f13798a50:false:145@0,ce884cf4-6106-11e0-92ba-e41f13798a50:false:208@0,fe230516-6107-11e0-a9fa-e41f13798a50:false:60@0,354f86e0-6108-11e0-970f-e41f13798a50:false:81@0,9fbcaef8-6109-11e0-9b5a-e41f13798a50:false:60@0,55b4cca4-610a-11e0-a82d-e41f13798a50:false:129@0,ddc1395c-610a-11e0-9541-e41f13798a50:false:166@0,0934dec2-610b-11e0-a63b-e41f13798a50:false:192@0,39562880-610c-11e0-af41-e41f13798a50:false:310@0,74296bb6-610c-11e0-bdc4-e41f13798a50:false:192@0,cfe50906-610c-11e0-9fc9-e41f13798a50:false:108@0,dd594b9c-610c-11e0-9db1-e41f13798a50:false:60@0,dd5ce428-610c-11e0-8962-e41f13798a50:false:92@0,1a0804d4-610d-11e0-b783-e41f13798a50:false:178@0,5135e390-610d-11e0-9d71-e41f13798a50:false:62@0,a3f68ada-610d-11e0-a6ef-e41f13798a50:false:60@0,db2a37a4-610d-11e0-b795-e41f13798a50:false:62@0,15802f0c-610f-11e0-abfe-e41f13798a50:false:128@0,42d810be-610f-11e0-8f4d-e41f13798a50:false:62@0,54456e96-610f-11e0-88a0-e41f13798a50:false:165@0,7b7b9e54-610f-11e0-b1e2-e41f13798a50:false:139@0,20b637bc-6110-11e0-b107-e41f13798a50:false:180@0,746d9be8-6110-11e0-9c9a-e41f13798a50:false:60@0,af7f87be-6110-11e0-8c1b-e41f13798a50:false:62@0,50e10916-6111-11e0-9d9a-e41f13798a50:false:128@0,1815bcde-6112-11e0-900e-e41f13798a50:false:160@0,4539798a-6112-11e0-84a8-e41f13798a50:false:217@0,45451d1c-6112-11e0-a6f5-e41f13798a50:false:110@0,cb4ae90a-6112-11e0-baab-e41f13798a50:false:146@0,cd86943a-6112-11e0-893b-e41f13798a50:false:119@0,490a443a-6113-11e0-a2f5-e41f13798a50:false:60@0,8cbfb584-6113-11e0-91d3-e41f13798a50:false:190@0,15215c34-6114-11e0-aec4-e41f13798a50:false:108@0,bdd0c716-6114-11e0-b8b0-e41f13798a50:false:60@0,f4e4446c-6114-11e0-a772-e41f13798a50:false:165@0,26b972d6-6116-11e0-bb90-e41f13798a50:false:60@0,7814059c-6116-11e0-bac6-e41f13798a50:false:62@0,94e63e9c-6116-11e0-b28b-e41f13798a50:false:127@0,e27c6a68-6117-11e0-b4b3-e41f13798a50:false:164@0,131754e8-6119-11e0-a9b0-e41f13798a50:false:60@0,4b32a4d6-6119-11e0-adda-e41f13798a50:false:62@0,7df5d5d6-611a-11e0-be7e-e41f13798a50:false:62@0,6012816c-611b-11e0-9795-e41f13798a50:false:128@0,620fab98-611b-11e0-b2e0-e41f13798a50:false:164@0,6325e7d6-611b-11e0-802a-e41f13798a50:false:139@0,65258b9a-611b-11e0-b598-e41f13798a50:false:154@0,655f9146-611b-11e0-ac19-e41f13798a50:false:155@0,67c337c6-611b-11e0-bc49-e41f13798a50:false:144@0,e8e8ff48-611b-11e0-be42-e41f13798a50:false:253@0,610ad578-611c-11e0-b7f7-e41f13798a50:false:242@0,6120cb1c-611c-11e0-9308-e41f13798a50:false:135@0,75bbf9c0-611c-11e0-b8b3-e41f13798a50:false:148@0,19ed7636-611d-11e0-adc5-e41f13798a50:false:61@0,515b2078-611d-11e0-afa7-e41f13798a50:false:178@0,9e68ba2e-611d-11e0-9fca-e41f13798a50:false:110@0,82113080-611e-11e0-9aee-e41f13798a50:false:61@0,bcbfb760-611e-11e0-bee8-e41f13798a50:false:165@0,1b236b44-611f-11e0-9f97-e41f13798a50:false:110@0,35f553b0-611f-11e0-a0c2-e41f13798a50:false:61@0,5c9072fc-611f-11e0-b81c-e41f13798a50:false:166@0,70a9b898-611f-11e0-a4c4-e41f13798a50:false:173@0,432459c0-6122-11e0-b93a-e41f13798a50:false:147@0,8a11c9d0-6122-11e0-9ae7-e41f13798a50:false:123@0,8a3081e0-6122-11e0-949a-e41f13798a50:false:137@0,b6ca7206-6122-11e0-9b27-e41f13798a50:false:125@0,c30ae4c4-6122-11e0-b50e-e41f13798a50:false:138@0,79f580fe-6123-11e0-ac09-e41f13798a50:false:120@0,bb123550-6123-11e0-8eaf-e41f13798a50:false:137@0,bca00ad2-6123-11e0-87f1-e41f13798a50:false:120@0,c34c7730-6123-11e0-9d3c-e41f13798a50:false:137@0,dc6de510-6127-11e0-b924-e41f13798a50:false:120@0,aa9c885a-6129-11e0-9bb0-e41f13798a50:false:136@0,30aa16e2-612a-11e0-9823-e41f13798a50:false:120@0,5911474e-612b-11e0-a19e-e41f13798a50:false:137@0,59acbcf6-612b-11e0-890d-e41f13798a50:false:120@0,5a4d3960-612b-11e0-9f6e-e41f13798a50:false:136@0,5af0d048-612b-11e0-81ef-e41f13798a50:false:120@0,5cbc91e6-612b-11e0-85a5-e41f13798a50:false:137@0,9b09d3fa-612b-11e0-86d7-e41f13798a50:false:120@0,9bd9527e-612b-11e0-984c-e41f13798a50:false:137@0,6d7436d6-612d-11e0-a7eb-e41f13798a50:false:119@0,8e928714-612d-11e0-910f-e41f13798a50:false:137@0,04db8efc-612e-11e0-bd08-e41f13798a50:false:120@0,092da8be-612e-11e0-96d1-e41f13798a50:false:137@0,0c471170-612e-11e0-9fab-e41f13798a50:false:120@0,0e2f2fa4-612e-11e0-a03d-e41f13798a50:false:137@0,116bb444-612e-11e0-96b0-e41f13798a50:false:120@0,152c9a4e-612e-11e0-a302-e41f13798a50:false:137@0,18d207f6-612e-11e0- Such msg repeats 2 times after "LocalReadRunnable reading SliceFromReadCommand" with the big row key. Switching to IRC... Regards, Wojtek From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Wednesday, May 11, 2011 12:20 PM To: user@cassandra.apache.org Subject: Re: Finding big rows Couple of questions to ask. You may also get some value from the #cassandra chat room where you can have a bit more of a conversation. - checking you ran nodetool scrub when upgrading to 0.7.3 ? (not related to the current problem, just asking) - what client library was using to write the data ? - when you have DEBUG logging and run the get that fails do you see any log messages that say "collecting %s of %s" ? (these mean the columns are been read by the query even if not returned). - not sure how easy it's going to be to pull 400MB of data through the server in one call. Take a look at thrift_max_message_length_in_mb and thrift_framed_transport_size_in_mb in the config. Hope that helps. ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com "WIRTUALNA POLSKA" Spolka Akcyjna z siedziba w Gdansku przy ul. Traugutta 115 C, wpisana do Krajowego Rejestru Sadowego - Rejestru Przedsiebiorcow prowadzonego przez Sad Rejonowy Gdansk - Polnoc w Gdansku pod numerem KRS 0000068548, o kapitale zakladowym 67.980.024,00 zlotych oplaconym w calosci oraz Numerze Identyfikacji Podatkowej 957-07-51-216.
