Joel Knighton created CASSANDRA-10413:
-----------------------------------------
Summary: Replaying materialized view updates from commitlog after
node decommission crashes Cassandra
Key: CASSANDRA-10413
URL: https://issues.apache.org/jira/browse/CASSANDRA-10413
Project: Cassandra
Issue Type: Bug
Reporter: Joel Knighton
Priority: Critical
Fix For: 3.0.0 rc2
This issue is reproducible through a Jepsen test, runnable as
{code}
lein with-profile +trunk test :only
cassandra.mv-test/mv-crash-subset-decommission
{code}
This test crashes/restarts nodes while decommissioning nodes. These actions are
not coordinated.
In [10164|https://issues.apache.org/jira/browse/CASSANDRA-10164], we introduced
a change to re-apply materialized view updates on commitlog replay.
Some nodes, upon restart, will crash in commitlog replay. They throw the
"Trying to get the view natural endpoint on a non-data replica" exception. I
added logging to getViewNaturalEndpoint to show the results of
replicationStrategy.getNaturalEndpoints for the baseToken and viewToken.
It can be seen that these problems occur when the baseEndpoints and
viewEndpoints are identical but do not contain the broadcast address of the
local node.
For example, a node at 10.0.0.5 crashes on replay of a write whose base token
and view token replicas are both [10.0.0.2, 10.0.0.4, 10.0.0.6]. It seems we
try to guard against this by considering pendingEndpoints for the viewToken,
but this does not appear to be sufficient.
I've attached the system.logs for a test run with added logging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
