Check, I understand. Thanks! The cluster certainly was overloaded and I did not realize that truncate does not tombstone or have a timestamp. Some 'feature' for future implementation, maybe? It seems odd if you expect the same behaviour of "delete from usertable" (in SQL, not yet in CQL, I presume), especially because the truncate is synced over all nodes before it returns to client, so a truncate may rightfully discard its handoffs, right? btw, it was very hard to replicate this behaviour, seems to be a rare occurrence...
I wonder though that you don't know YCSB, what do you use to do performance testing? wrote your own or use another tool? In the latter I would like to know what you use :) Ciao Peter Dijkshoorn Adyen - Payments Made Easy www.adyen.com Visiting address: Mail Address: Simon Carmiggeltstraat 6-50 P.O. Box 10095 1011 DJ Amsterdam 1001 EB Amsterdam The Netherlands The Netherlands Office +31.20.240.1240 Email peter.dijksho...@adyen.com On 05/07/2012 12:59 PM, aaron morton wrote: > I don't know the YCSB code, but one theory would be... > > 1) The cluster is overloaded by the test. > 2) A write at CL ALL fails because a node does not respond in time. > 3) The coordinator stores the hint and returns failure to the client. > 4) The client gets an UnavailableException and retries the operation. > > Did the nodes show any dropped messages ? Either in nodetool tpstats > or in the logs? > > Truncate is meta data operation, unlike deleting columns or rows. When > a column is deleted a Tombstone column is written, when row is deleted > information is associated with the key, in the context of the CF. > Truncate snapshots and then deletes the SSTables on disk, it does not > write to the SSTables. So it is possible for a write to be stored with > a lower timestamp than the truncate, because truncate does not have a > timestamp. > > cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 4/05/2012, at 1:28 AM, Peter Dijkshoorn wrote: > >> Hi guys, >> >> I got a weird thingy popping up twice today, I run a test where I insert >> a milion records via YCSB and edited it to allow me to adjust the >> consistency level: the write operations are done with >> ConsistencyLevel.ALL. >> This is send to a 4 (virtual) node cluster with a keyspace 'test' set up >> with replication factor 3. >> Now I expect that because of the ConsistencyLevel.ALL there is no hinted >> handoff active, since writes are to be accepted by all nodes before the >> operation returns to the client. The client gets only OK back, none >> fails. >> After the test I run a truncate, and a count which reveals still active >> records, time does not matter, I have to re-invoke the truncate to >> remove the records. >> >> [cqlsh 2.0.0 | Cassandra 1.0.8 | CQL spec 2.0.0 | Thrift protocol >> 19.20.0] >> cqlsh> use test; >> cqlsh:test> truncate usertable; >> cqlsh:test> select count(*) from usertable ; >> count >> ------- >> 3 >> >> >> On the cassandra output (-f) I can see that there is some handoff-ing >> active, which I did not expect. >> >> Has anyone an idea why the handoff is active while issuing opperations >> with ConsistencyLevel.ALL? >> Why is the truncate not correctly put in sync and allows subsequent >> handoff's delivered of records originally set before the truncate? >> >> Thanks if you can clarify these thing, I did not expect this at all. >> >> Cheers, >> >> Peter >> >> -- >> Peter Dijkshoorn >> Adyen - Payments Made Easy >> www.adyen.com <http://www.adyen.com> >> >> Visiting address: Mail Address: >> Simon Carmiggeltstraat 6-50 P.O. Box 10095 >> 1011 DJ Amsterdam 1001 EB Amsterdam >> The Netherlands The Netherlands >> >> Office +31.20.240.1240 >> Email peter.dijksho...@adyen.com >> >
