Caleb Rackliffe created CASSANDRA-21854:
-------------------------------------------
Summary: SAI vector index corruption when an insert is interrupted
at the new graph-insert permit wait
Key: CASSANDRA-21854
URL: https://issues.apache.org/jira/browse/CASSANDRA-21854
Project: Apache Cassandra
Issue Type: Bug
Components: Feature/SAI, Feature/Vector Search
Reporter: Caleb Rackliffe
Assignee: Sunil Ramchandra Pawar
h2. Summary
Commit
[{{bf06dbd}}|https://github.com/cscotta/cassandra/commit/bf06dbdd9aafb6b1d4ef17f1e06ec2b720d40d2b]
bounds concurrent SAI graph inserts with a semaphore. {{OnHeapGraph.add}} now
registers the vector and ordinal before it waits for a permit. An interrupt
during that wait corrupts the memtable index permanently.
{{{}postingsMap.putIfAbsent{}}}, the ordinal assignment,
{{{}vectorValues.add{}}}, and {{postingsByOrdinal.put}} all run before
{{{}graphInsertPermits.acquireThrowUncheckedOnInterrupt(1){}}}. The thrown
{{UncheckedInterruptedException}} lands before the {{{}try{}}}block. No
rollback runs, so the vector stays registered and the graph node never gets
linked.
h2. Trigger conditions
# A table has a Storage-Attached Index vector index on a memtable-backed index.
# Concurrent inserts exceed {{{}MAX_CONCURRENT_GRAPH_INSERTS{}}}, or the
memtable reports {{{}limitsConcurrentWritesTo{}}}.
# A writer thread receives an interrupt while it waits for a graph-insert
permit.
h2. Failure scenario
The interrupted writer throws {{UncheckedInterruptedException}} and abandons
the insert. {{vectorValues}} and {{{}postingsByOrdinal{}}}keep the ordinal.
{{graph.size()}} stays 0. A retry of the same row only updates the postings
list. It never calls {{{}builder.addGraphNode{}}}, so nothing repairs the graph.
At flush time, {{OnHeapGraph.writeData}} asserts {{nextOrdinal ==
builder.getGraph().size()}} and fails. With assertions disabled, the index
stays silent and vector searches return no results for that vector.
h2. Reproducer
Path:
{{~/.local/share/cassandra-watch/reports/reproducers/bf06dbdd9aaf/InterruptPermitUnlinkedNodeRepro.java}}
(executed against a full patched build with {{{}-ea{}}}).
Observed output on
[{{bf06dbd}}|https://github.com/cscotta/cassandra/commit/bf06dbdd9aafb6b1d4ef17f1e06ec2b720d40d2b]:
the interrupted {{add()}} throws. {{{}vectorValues.size()==1{}}}.
{{{}graph.size()==0{}}}. The retry succeeds but does not link the node. Vector
search returns 0 results. The flush invariant fails.
On parent
[{{b40e292}}|https://github.com/cscotta/cassandra/commit/b40e29225c50c96f10c4836d68d37174204cddd6]
the same interrupt path cannot occur. The parent has no blocking wait between
registration and {{{}addGraphNode{}}}.
h2. Proposed resolution
Acquire the permit before the ordinal assignment and registration, or roll back
the registration when {{acquireThrowUncheckedOnInterrupt}} throws.
Alternatively, add a failure path that unregisters the ordinal and the postings
entry.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
