I'm trying to track down some really worrying behavior. It appears that writing
multiple columns while a table flush is occurring can result in Cassandra
recording its data in a way that makes columns visible only to some queries but
not others.
Ie. Query for a single column works but the column does not appear in slice
queries depending on the other columns in the query
cfq.getKey("foo").getColumn("A") returns "A"
cfq.getKey("foo").withColumnSlice("A", "B") returns "B" only
cfq.getKey("foo").withColumnSlice("A","B","C") returns "A","B" and "C"
This is a permanent condition meaning that even hours later with no reads or
writes the DB will return the same results. I can reproduce this 100% of the
time by writing multiple columns and then reading a different set of multiple
columns. Columns written during the flush may or may not appear.
Details
# There are no log errors
# All single column queries return correct data.
# Slice queries may or may not return the column depending on which other
columns are in the query.
# This is on a stock "unzip and run" installation of Cassandra using default
options only; basically doing the cassandra getting started tutorial and using
the Demo table described in that tutorial.
# Cassandra 1.2.0 using Astynax and Java 1.6.0_37.
# There are no errors but there is always a "flushing high traffic column
family" that happens right before the incoherent state occurs
# to reproduce just update multiple columns at the same time, using random rows
and then verify the writes by reading multiple columns. I get can generate the
error on 100% of runs. Once the state is screwed up, the multi column read will
not contain the column but the single column read will.
Log snippet
INFO 15:47:49,066 GC for ParNew: 320 ms for 1 collections, 207199992 used; max
is 1052770304
INFO 15:47:58,076 GC for ParNew: 330 ms for 1 collections, 232839680 used; max
is 1052770304
INFO 15:48:00,374 flushing high-traffic column family CFS(Keyspace='BUGS',
ColumnFamily='Test') (estimated 50416978 bytes)
INFO 15:48:00,374 Enqueuing flush of Memtable-Test@1575891161(4529586/50416978
serialized/live bytes, 279197 ops)
INFO 15:48:00,378 Writing Memtable-Test@1575891161(4529586/50416978
serialized/live bytes, 279197 ops)
INFO 15:48:01,142 GC for ParNew: 654 ms for 1 collections, 239478568 used; max
is 1052770304
INFO 15:48:01,474 Completed flushing
/var/lib/cassandra/data/BUGS/Test/BUGS-Test-ia-45-Data.db (4580066 bytes) for
commitlog position ReplayPosition(segmentId=1359415964165, position=7462737)
Any ideas on what could be going on? I could not find anything like this in the
open bugs and the only workaround seems to be never doing multi-column reads or
writes. I'm concerned that the DB can get into a state where different queries
can return such inconsistent results. All with no warning or errors. There is
no way to even verify data correctness; every column can seem correct when
queried and then disappear during slice queries depending on the other columns
in the query.
Thanks
