Thanks Christian can you reproduce the behaviour with a single node?
I tried my test with a single node. But I can't. This behaviour is seems to be CQL only, or at least has gotten worse with > CQL. I did not experience this with Thrift. I truncate tables with CQL. I've never tried with Thrift. I think that my problem can happen when truncating even succeeds. That's because I check all records after truncating. I checked the source code. ReplayPosition.segment and position become -1 and 0 (ReplayPosition.NONE) in dscardSSTables() at truncating a table when there is no SSTable. I guess that ReplayPosition.segment shouldn't be -1 at truncating a table in this case. replayMutation() can request unexpected replay mutations because of this segment's value. Is there anyone familiar with truncate and replay? Regards, Yuji On Mon, Aug 8, 2016 at 6:36 PM, horschi <hors...@gmail.com> wrote: > Hi Yuji, > > can you reproduce the behaviour with a single node? > > The reason I ask is because I probably have the same issue with my > automated tests (which run truncate between every test), which run on my > local laptop. > > Maybe around 5 tests randomly fail out of my 1800. I can see that the > failed tests sometimes show data from other tests, which I think must be > because of a failed truncate. This behaviour is seems to be CQL only, or at > least has gotten worse with CQL. I did not experience this with Thrift. > > regards, > Christian > > > > On Mon, Aug 8, 2016 at 7:34 AM, Yuji Ito <y...@imagine-orb.com> wrote: > >> Hi all, >> >> I have a question about clearing table and commit log replay. >> After some tables were truncated consecutively, I got some stale values. >> This problem doesn't occur when I clear keyspaces with DROP (and CREATE). >> >> I'm testing the following test with node failure. >> Some stale values appear at checking phase. >> >> Test iteration: >> 1. initialize tables as below >> 2. request a lot of read/write concurrently >> 3. check all records >> 4. repeat from the beginning >> >> I use C* 2.2.6. There are 3 nodes (replication_factor: 3). >> Each node kills cassandra process at random intervals and restarts it >> immediately. >> >> My initialization: >> 1. clear tables with TRUNCATE >> 2. INSERT initial records >> 3. check if all values are correct >> >> If any phase fails (because of node failure), the initialization starts >> all over again. >> So, tables are sometimes truncated consecutively. >> Though the check in the initialization is OK, stale data appears when I >> execute "SELECT * FROM mykeyspace.mytable;" after a lot of requests are >> completed. >> >> The problem is likely to occur when the ReplayPosition's value in >> "truncated_at" is initialized as below after an empty table is truncated. >> >> Column Family ID: truncated_at >> XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX: 0xffffffffffffffff000000000000 >> 0156597cd4c7 >> (this value was acquired just after phase 1 in my initialization) >> >> I guess some unexpected replays occur. >> Does anyone know the behavior? >> >> Thanks, >> Yuji >> > >
