[
https://issues.apache.org/jira/browse/CASSANDRA-21854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138376#comment-18138376
]
Caleb Rackliffe commented on CASSANDRA-21854:
---------------------------------------------
Full details at https://github.com/cscotta/cassandra/issues/21
> SAI vector index corruption when an insert is interrupted at the new
> graph-insert permit wait
> ---------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21854
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21854
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Feature/SAI, Feature/Vector Search
> Reporter: Caleb Rackliffe
> Assignee: Sunil Ramchandra Pawar
> Priority: Normal
>
> h2. Summary
> Commit
> [{{bf06dbd}}|https://github.com/cscotta/cassandra/commit/bf06dbdd9aafb6b1d4ef17f1e06ec2b720d40d2b]
> bounds concurrent SAI graph inserts with a semaphore. {{OnHeapGraph.add}}
> now registers the vector and ordinal before it waits for a permit. An
> interrupt during that wait corrupts the memtable index permanently.
> {{{}postingsMap.putIfAbsent{}}}, the ordinal assignment,
> {{{}vectorValues.add{}}}, and {{postingsByOrdinal.put}} all run before
> {{{}graphInsertPermits.acquireThrowUncheckedOnInterrupt(1){}}}. The thrown
> {{UncheckedInterruptedException}} lands before the {{{}try{}}}block. No
> rollback runs, so the vector stays registered and the graph node never gets
> linked.
> h2. Trigger conditions
> # A table has a Storage-Attached Index vector index on a memtable-backed
> index.
> # Concurrent inserts exceed {{{}MAX_CONCURRENT_GRAPH_INSERTS{}}}, or the
> memtable reports {{{}limitsConcurrentWritesTo{}}}.
> # A writer thread receives an interrupt while it waits for a graph-insert
> permit.
> h2. Failure scenario
> The interrupted writer throws {{UncheckedInterruptedException}} and abandons
> the insert. {{vectorValues}} and {{{}postingsByOrdinal{}}}keep the ordinal.
> {{graph.size()}} stays 0. A retry of the same row only updates the postings
> list. It never calls {{{}builder.addGraphNode{}}}, so nothing repairs the
> graph.
> At flush time, {{OnHeapGraph.writeData}} asserts {{nextOrdinal ==
> builder.getGraph().size()}} and fails. With assertions disabled, the index
> stays silent and vector searches return no results for that vector.
> h2. Reproducer
> Path:
> {{~/.local/share/cassandra-watch/reports/reproducers/bf06dbdd9aaf/InterruptPermitUnlinkedNodeRepro.java}}
> (executed against a full patched build with {{{}-ea{}}}).
> Observed output on
> [{{bf06dbd}}|https://github.com/cscotta/cassandra/commit/bf06dbdd9aafb6b1d4ef17f1e06ec2b720d40d2b]:
> the interrupted {{add()}} throws. {{{}vectorValues.size()==1{}}}.
> {{{}graph.size()==0{}}}. The retry succeeds but does not link the node.
> Vector search returns 0 results. The flush invariant fails.
> On parent
> [{{b40e292}}|https://github.com/cscotta/cassandra/commit/b40e29225c50c96f10c4836d68d37174204cddd6]
> the same interrupt path cannot occur. The parent has no blocking wait
> between registration and {{{}addGraphNode{}}}.
> h2. Proposed resolution
> Acquire the permit before the ordinal assignment and registration, or roll
> back the registration when {{acquireThrowUncheckedOnInterrupt}} throws.
> Alternatively, add a failure path that unregisters the ordinal and the
> postings entry.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
