Let me know if you get anywhere, I'm on there as aaron_morton but I'm also way over in New Zealand.
If you are using your own client and writing data you cannot read back check that the byte encoding is always the same and that you are setting appropriate timestamps for every call. In the log message all the columns have a 0 time stamp. Is that deliberate ? Does the count in "collecting 0 of 1000000" increase ? If it does it means the query is collecting rows that a "live" and match the query filer. If not it means all that data under a (probably row level) tombstone. Which make me ask is this a row that gets a lot of inserts and then deletes ? Cheers ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 11 May 2011, at 23:30, Meler Wojciech wrote: > I didn’t run nodetool scrub. My app uses c++ thrift client (0.5.0 and 0.6.1) . > As this is production environment I get a lot of messages "collecting %s of > %s", but there is no row key. > I’ve matched it by uuid and thread – hope it is ok: > > [ReadStage:3][org.apache.cassandra.db.filter.SliceQueryFilter] collecting 0 > of 1000000: SuperColumn(455470c2-60e6-11e0-acae-e41f13798a50 > [4554853a-60e6-11e0-9342-e41f13798a50:false:361745@0,aa92b386-60e6-11e0-bc58-e41f13798a50:false:159@0,ac346cc0-60e6-11e0-bf8e-e41f13798a50:false:53@0,ad16d362-60e6-11e0-ad9c-e41f13798a50:false:66@0,ae57076a-60e6-11e0-8982-e41f13798a50:false:48@0,afc042ba-60e6-11e0-8320-e41f13798a50:false:63@0,b38d912c-60e6-11e0-b3b7-e41f13798a50:false:164@0,49d7dc00-60e7-11e0-9694-e41f13798a50:false:100@0,94e99fa8-60e7-11e0-b621-e41f13798a50:false:233@0,9612c3a0-60e7-11e0-8292-e41f13798a50:false:4049@0,ec245880-60e7-11e0-85be-e41f13798a50:false:148@0,110ac968-60e8-11e0-b325-e41f13798a50:false:125@0,64a45b4c-60e9-11e0-9628-e41f13798a50:false:160@0,cfc39a00-60e9-11e0-9539-e41f13798a50:false:105@0,2a21ab22-60ea-11e0-a1f6-e41f13798a50:false:146@0,95f2d2d6-60ea-11e0-b763-e41f13798a50:false:53@0,97972362-60ea-11e0-9275-e41f13798a50:false:134@0,ce980606-60ea-11e0-bd03-e41f13798a50:false:195@0,517e02dc-60eb-11e0-b0a8-e41f13798a50:false:53@0,5694c74c-60eb-11e0-941f-e41f13798a50:false:170@0,8d48cca2-60eb-11e0-bdbc-e41f13798a50:false:187@0,fc5e0148-60eb-11e0-ac4c-e41f13798a50:false:558@0,fc7e9476-60eb-11e0-bbfe-e41f13798a50:false:161@0,22a860a0-60ec-11e0-8d1b-e41f13798a50:false:138@0,bf054b52-60ec-11e0-a4b1-e41f13798a50:false:56@0,fd3b4822-60ec-11e0-8612-e41f13798a50:false:234@0,0c1d6fe0-60ee-11e0-8bb0-e41f13798a50:false:79@0,43bbddec-60ee-11e0-848f-e41f13798a50:false:79@0,ec50ed8e-60ef-11e0-a8c1-e41f13798a50:false:60@0,57ae0534-60f1-11e0-8fd5-e41f13798a50:false:60@0,81d7bbe8-60f1-11e0-9586-e41f13798a50:false:143@0,08ce8852-60f2-11e0-bfeb-e41f13798a50:false:150@0,c51bf170-60f2-11e0-8a60-e41f13798a50:false:192@0,e98b181e-60f3-11e0-b20b-e41f13798a50:false:108@0,ed931d76-60f3-11e0-90cf-e41f13798a50:false:146@0,f754541a-60f3-11e0-84db-e41f13798a50:false:60@0,24a73220-60f4-11e0-ae03-e41f13798a50:false:139@0,33485db8-60f4-11e0-9f86-e41f13798a50:false:196@0,42fe380e-60f4-11e0-9f0f-e41f13798a50:false:150@0,440afbb0-60f4-11e0-a066-e41f13798a50:false:122@0,64d9d13a-60f5-11e0-a30c-e41f13798a50:false:60@0,9e1cefc2-60f5-11e0-826e-e41f13798a50:false:205@0,1f642a46-60f6-11e0-8aa3-e41f13798a50:false:298@0,1f6c946a-60f6-11e0-9eab-e41f13798a50:false:117@0,d1a4e8c6-60f6-11e0-9bfa-e41f13798a50:false:58@0,0a40ba5c-60f7-11e0-893f-e41f13798a50:false:170@0,a2e0f740-60f7-11e0-93b0-e41f13798a50:false:108@0,3922060e-60f8-11e0-b850-e41f13798a50:false:147@0,3cdcdf08-60f8-11e0-8320-e41f13798a50:false:60@0,79b33a26-60f8-11e0-b151-e41f13798a50:false:187@0,aae281c8-60f9-11e0-9d14-e41f13798a50:false:60@0,e1b3295a-60f9-11e0-b367-e41f13798a50:false:81@0,8149870c-60fa-11e0-b6d4-e41f13798a50:false:128@0,b483680e-60fa-11e0-a56f-e41f13798a50:false:164@0,3dfdfe78-60fb-11e0-8582-e41f13798a50:false:143@0,15a9b830-60fc-11e0-be81-e41f13798a50:false:217@0,46ae8aae-60fd-11e0-908e-e41f13798a50:false:60@0,7ee58490-60fd-11e0-9072-e41f13798a50:false:81@0,bf4fba30-60ff-11e0-9b5f-e41f13798a50:false:60@0,26201952-6101-11e0-a4cb-e41f13798a50:false:60@0,8e5e995c-6102-11e0-9bb3-e41f13798a50:false:60@0,a4ef01c0-6102-11e0-9bff-e41f13798a50:false:214@0,a5034784-6102-11e0-96ee-e41f13798a50:false:133@0,ad9f3682-6102-11e0-8df5-e41f13798a50:false:172@0,e2e01b36-6102-11e0-8489-e41f13798a50:false:155@0,25561204-6103-11e0-a14b-e41f13798a50:false:155@0,4d39d2a6-6103-11e0-844a-e41f13798a50:false:152@0,74aeb4fa-6103-11e0-9f65-e41f13798a50:false:140@0,7e8802ba-6103-11e0-81f1-e41f13798a50:false:140@0,8e6532f2-6103-11e0-acd4-e41f13798a50:false:141@0,fceadee8-6103-11e0-8beb-e41f13798a50:false:313@0,2d319618-6105-11e0-9efa-e41f13798a50:false:60@0,4d4540bc-6105-11e0-96d1-e41f13798a50:false:128@0,63fcaa02-6105-11e0-ac35-e41f13798a50:false:81@0,67c70114-6105-11e0-9ab8-e41f13798a50:false:167@0,3a104306-6106-11e0-8be9-e41f13798a50:false:145@0,ce884cf4-6106-11e0-92ba-e41f13798a50:false:208@0,fe230516-6107-11e0-a9fa-e41f13798a50:false:60@0,354f86e0-6108-11e0-970f-e41f13798a50:false:81@0,9fbcaef8-6109-11e0-9b5a-e41f13798a50:false:60@0,55b4cca4-610a-11e0-a82d-e41f13798a50:false:129@0,ddc1395c-610a-11e0-9541-e41f13798a50:false:166@0,0934dec2-610b-11e0-a63b-e41f13798a50:false:192@0,39562880-610c-11e0-af41-e41f13798a50:false:310@0,74296bb6-610c-11e0-bdc4-e41f13798a50:false:192@0,cfe50906-610c-11e0-9fc9-e41f13798a50:false:108@0,dd594b9c-610c-11e0-9db1-e41f13798a50:false:60@0,dd5ce428-610c-11e0-8962-e41f13798a50:false:92@0,1a0804d4-610d-11e0-b783-e41f13798a50:false:178@0,5135e390-610d-11e0-9d71-e41f13798a50:false:62@0,a3f68ada-610d-11e0-a6ef-e41f13798a50:false:60@0,db2a37a4-610d-11e0-b795-e41f13798a50:false:62@0,15802f0c-610f-11e0-abfe-e41f13798a50:false:128@0,42d810be-610f-11e0-8f4d-e41f13798a50:false:62@0,54456e96-610f-11e0-88a0-e41f13798a50:false:165@0,7b7b9e54-610f-11e0-b1e2-e41f13798a50:false:139@0,20b637bc-6110-11e0-b107-e41f13798a50:false:180@0,746d9be8-6110-11e0-9c9a-e41f13798a50:false:60@0,af7f87be-6110-11e0-8c1b-e41f13798a50:false:62@0,50e10916-6111-11e0-9d9a-e41f13798a50:false:128@0,1815bcde-6112-11e0-900e-e41f13798a50:false:160@0,4539798a-6112-11e0-84a8-e41f13798a50:false:217@0,45451d1c-6112-11e0-a6f5-e41f13798a50:false:110@0,cb4ae90a-6112-11e0-baab-e41f13798a50:false:146@0,cd86943a-6112-11e0-893b-e41f13798a50:false:119@0,490a443a-6113-11e0-a2f5-e41f13798a50:false:60@0,8cbfb584-6113-11e0-91d3-e41f13798a50:false:190@0,15215c34-6114-11e0-aec4-e41f13798a50:false:108@0,bdd0c716-6114-11e0-b8b0-e41f13798a50:false:60@0,f4e4446c-6114-11e0-a772-e41f13798a50:false:165@0,26b972d6-6116-11e0-bb90-e41f13798a50:false:60@0,7814059c-6116-11e0-bac6-e41f13798a50:false:62@0,94e63e9c-6116-11e0-b28b-e41f13798a50:false:127@0,e27c6a68-6117-11e0-b4b3-e41f13798a50:false:164@0,131754e8-6119-11e0-a9b0-e41f13798a50:false:60@0,4b32a4d6-6119-11e0-adda-e41f13798a50:false:62@0,7df5d5d6-611a-11e0-be7e-e41f13798a50:false:62@0,6012816c-611b-11e0-9795-e41f13798a50:false:128@0,620fab98-611b-11e0-b2e0-e41f13798a50:false:164@0,6325e7d6-611b-11e0-802a-e41f13798a50:false:139@0,65258b9a-611b-11e0-b598-e41f13798a50:false:154@0,655f9146-611b-11e0-ac19-e41f13798a50:false:155@0,67c337c6-611b-11e0-bc49-e41f13798a50:false:144@0,e8e8ff48-611b-11e0-be42-e41f13798a50:false:253@0,610ad578-611c-11e0-b7f7-e41f13798a50:false:242@0,6120cb1c-611c-11e0-9308-e41f13798a50:false:135@0,75bbf9c0-611c-11e0-b8b3-e41f13798a50:false:148@0,19ed7636-611d-11e0-adc5-e41f13798a50:false:61@0,515b2078-611d-11e0-afa7-e41f13798a50:false:178@0,9e68ba2e-611d-11e0-9fca-e41f13798a50:false:110@0,82113080-611e-11e0-9aee-e41f13798a50:false:61@0,bcbfb760-611e-11e0-bee8-e41f13798a50:false:165@0,1b236b44-611f-11e0-9f97-e41f13798a50:false:110@0,35f553b0-611f-11e0-a0c2-e41f13798a50:false:61@0,5c9072fc-611f-11e0-b81c-e41f13798a50:false:166@0,70a9b898-611f-11e0-a4c4-e41f13798a50:false:173@0,432459c0-6122-11e0-b93a-e41f13798a50:false:147@0,8a11c9d0-6122-11e0-9ae7-e41f13798a50:false:123@0,8a3081e0-6122-11e0-949a-e41f13798a50:false:137@0,b6ca7206-6122-11e0-9b27-e41f13798a50:false:125@0,c30ae4c4-6122-11e0-b50e-e41f13798a50:false:138@0,79f580fe-6123-11e0-ac09-e41f13798a50:false:120@0,bb123550-6123-11e0-8eaf-e41f13798a50:false:137@0,bca00ad2-6123-11e0-87f1-e41f13798a50:false:120@0,c34c7730-6123-11e0-9d3c-e41f13798a50:false:137@0,dc6de510-6127-11e0-b924-e41f13798a50:false:120@0,aa9c885a-6129-11e0-9bb0-e41f13798a50:false:136@0,30aa16e2-612a-11e0-9823-e41f13798a50:false:120@0,5911474e-612b-11e0-a19e-e41f13798a50:false:137@0,59acbcf6-612b-11e0-890d-e41f13798a50:false:120@0,5a4d3960-612b-11e0-9f6e-e41f13798a50:false:136@0,5af0d048-612b-11e0-81ef-e41f13798a50:false:120@0,5cbc91e6-612b-11e0-85a5-e41f13798a50:false:137@0,9b09d3fa-612b-11e0-86d7-e41f13798a50:false:120@0,9bd9527e-612b-11e0-984c-e41f13798a50:false:137@0,6d7436d6-612d-11e0-a7eb-e41f13798a50:false:119@0,8e928714-612d-11e0-910f-e41f13798a50:false:137@0,04db8efc-612e-11e0-bd08-e41f13798a50:false:120@0,092da8be-612e-11e0-96d1-e41f13798a50:false:137@0,0c471170-612e-11e0-9fab-e41f13798a50:false:120@0,0e2f2fa4-612e-11e0-a03d-e41f13798a50:false:137@0,116bb444-612e-11e0-96b0-e41f13798a50:false:120@0,152c9a4e-612e-11e0-a302-e41f13798a50:false:137@0,18d207f6-612e-11e0- > > Such msg repeats 2 times after „LocalReadRunnable reading > SliceFromReadCommand” with the big row key. > > Switching to IRC… > > Regards, > Wojtek > > From: aaron morton [mailto:aa...@thelastpickle.com] > Sent: Wednesday, May 11, 2011 12:20 PM > To: user@cassandra.apache.org > Subject: Re: Finding big rows > > Couple of questions to ask. You may also get some value from the #cassandra > chat room where you can have a bit more of a conversation. > > - checking you ran nodetool scrub when upgrading to 0.7.3 ? (not related to > the current problem, just asking) > - what client library was using to write the data ? > - when you have DEBUG logging and run the get that fails do you see any log > messages that say "collecting %s of %s" ? (these mean the columns are been > read by the query even if not returned). > - not sure how easy it's going to be to pull 400MB of data through the server > in one call. Take a look at thrift_max_message_length_in_mb and > thrift_framed_transport_size_in_mb in the config. > > Hope that helps. > > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > > "WIRTUALNA POLSKA" Spolka Akcyjna z siedziba w Gdansku przy ul. Traugutta 115 > C, wpisana do Krajowego Rejestru Sadowego - Rejestru Przedsiebiorcow > prowadzonego przez Sad Rejonowy Gdansk - Polnoc w Gdansku pod numerem KRS > 0000068548, o kapitale zakladowym 67.980.024,00 zlotych oplaconym w calosci > oraz Numerze Identyfikacji Podatkowej 957-07-51-216. >
