Jeremiah Jordan created CASSANDRA-21727:
-------------------------------------------
Summary: SAI vector graph is broken after an interrupt during the
wait for a permit
Key: CASSANDRA-21727
URL: https://issues.apache.org/jira/browse/CASSANDRA-21727
Project: Apache Cassandra
Issue Type: Bug
Components: Feature/SAI
Reporter: Jeremiah Jordan
h2. Problem
CASSANDRA-21644 added a semaphore to {{OnHeapGraph.add}}. The writer gets the
semaphore too late. Before the wait, the writer already added the new vector to
{{postingsMap}}, {{vectorValues}} and {{postingsByOrdinal}}. It also already
wrote the vector to {{vectorsByKey}}.
If the writer receives an interrupt during the wait, the wait throws
{{UncheckedInterruptedException}}. The insert stops before {{addGraphNode}}.
The result is as follows:
* The vector is in {{postingsMap}}, {{vectorValues}} and {{postingsByOrdinal}},
but it is not in the graph.
* A retry of the same row finds the vector in {{postingsMap}}. The retry adds
only the key. It does not add the vector to the graph.
* A vector search does not find the row.
* At flush, the assertion in {{OnHeapGraph.writeData}} fails. This assertion
checks that each ordinal is in the graph. If the JVM runs without assertions,
no error occurs and the index is incorrect.
* For an update, {{vectorsByKey}} contains the new vector, but the row keeps
the old vector. The brute-force search for {{ORDER BY}} uses the new vector to
score the row.
h2. Steps to reproduce
# Create an {{OnHeapGraph}} with a memtable for which
{{limitsConcurrentWritesTo}} returns false.
# Set the interrupt flag of the thread.
# Call {{add}} with a new vector. The call throws
{{UncheckedInterruptedException}}.
# Clear the interrupt flag. Call {{add}} again for the same row and vector.
# Search the graph for the vector.
Expected result: the search finds the vector. Actual result: {{size()}} returns
1, but the search finds no vector.
h2. Proposed fix
Change {{OnHeapGraph.add}} as follows:
# Read the entry for the vector from {{postingsMap}}.
# If the vector is new, get the permit before any change to the data. This
includes {{vectorsByKey}}.
# Release the permit in one {{finally}} block.
With this order, an interrupt during the wait does not change the graph or
{{vectorsByKey}}. A retry then adds the vector to the graph.
Do not try to roll back the changes after the interrupt. Before the rollback,
other writers can add keys for the same vector and send acknowledgements to
clients. Also, the graph cannot reuse an ordinal after it assigns that ordinal.
Add tests for these conditions:
* After an interrupt during the wait for a permit, the graph, {{vectorsByKey}}
and the number of permits do not change. A retry adds the vector to the graph.
* If two writers add the same new vector, the writer that does not add the
vector returns its permit.
* If more writers than permits add the same vectors at the same time, all
inserts complete.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
