We are facing an issue on one of our production systems where after we delete
the data
the data doesn't seem to get deleted. We have a Get call just after the delete
call.
The data shows up.
Versions
cassandra : 3.11.6
gocqlx : v2 v2.1.0
1. Client Settings: LocalQuorum
2. Number of Nodes : 3
3. All 3 nodes up and running for weeks.
4. Inserts were done few days earlier. So there is good amount of time
difference
between Inserts and Deletes and Inserts have made through successfully.
The Delete Call :
q := s.session.Query(stmt, names).BindStruct(*customModel)
err := q.ExecRelease()
We do check the error and it is Nil.
There are no exceptions during that time either on the client side or server
side.
The Get Call :
q := s.session.Query(stmt, names).BindStruct(*customModel)
err := q.GetRelease(customModel)
This returns the data successfully.
We do have these two options enabled.
1.
https://docs.datastax.com/en/dse/6.8/dse-dev/datastax_enterprise/config/configCassandra_yaml.html#configCassandra_yaml__commitlog_sync
batch - Send ACK signal for writes after the commit log has been flushed to
disk. Each incoming write triggers the flush task.
2. only_purge_repaired_tombstones
This does not happen for all the delete operations. For many of them, the
delete seems
to go through. This does not seem to be timing-related and the successful and
unsuccessful ones
are spread out.
CASSANDRA-15690
Single partition queries can mistakenly omit partition deletions and resurrect
data
I am trying to go through this PR and ticket. If you have any suggestions,
please do let me know.
The table structure is the following
CREATE KEYSPACE cycling WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '3'} AND durable_writes = true;
CREATE TABLE cycling.rider (
uuid text,
created_at timestamp,
PRIMARY KEY (uuid, created_at)
) WITH CLUSTERING ORDER BY (created_at DESC)
AND WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4', 'only_purge_repaired_tombstones':
'true'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 3600
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
Thanks
