David Capwell created CASSANDRA-18471:
-----------------------------------------
Summary: CEP-15 Accord: BurnTest fails with Received replies from
a node that must have known the route, but that did not include it
Key: CASSANDRA-18471
URL: https://issues.apache.org/jira/browse/CASSANDRA-18471
Project: Cassandra
Issue Type: Bug
Components: Accord
Reporter: David Capwell
While working on CASSANDRA-18451 I hit the following failure
{code}
Failed on seed -5929214838499924343
accord.burn.SimulationException: Failed on seed -5929214838499924343
Caused by: java.lang.AssertionError: Unexpected exception encountered
at
accord.impl.basic.PropagatingPendingQueue.poll(PropagatingPendingQueue.java:73)
at accord.impl.basic.Cluster.processPending(Cluster.java:179)
at accord.impl.basic.Cluster.run(Cluster.java:296)
at accord.burn.BurnTest.burn(BurnTest.java:309)
at accord.burn.BurnTest.run(BurnTest.java:386)
at accord.burn.BurnTest.testOne(BurnTest.java:372)
Suppressed: java.lang.IllegalStateException: Received replies from a
node that must have known the route, but that did not include it
at accord.coordinate.Invalidate.invalidate(Invalidate.java:204)
at accord.coordinate.Invalidate.handle(Invalidate.java:131)
at accord.coordinate.Invalidate.onSuccess(Invalidate.java:105)
at accord.coordinate.Invalidate.onSuccess(Invalidate.java:51)
at
accord.impl.basic.Cluster.lambda$processNext$1(Cluster.java:209)
at accord.impl.basic.Cluster.now(Cluster.java:260)
at accord.impl.basic.Cluster.processNext(Cluster.java:206)
at accord.impl.basic.Cluster.processPending(Cluster.java:183)
{code}
In a debugger was able to figure out the state and create a unit test to hit
the same situation
{code}
class InvalidateTest
{
@Test
void test() throws ExecutionException
{
try (MockCluster cluster =
MockCluster.builder().replication(2).nodes(2).build())
{
Node n1 = cluster.get(1);
Node n2 = cluster.get(2);
RoutingKey n1RoutingKey =
n1.topology().current().get(0).range.end();
IntKey.Raw n1key = IntKey.key(((IntKey.Routing) n1RoutingKey).key);
RoutingKey n2RoutingKey =
n1.topology().current().get(1).range.end();
IntKey.Raw n2key = IntKey.key(((IntKey.Routing) n2RoutingKey).key);
Keys keys = Keys.of(n1key, n2key);
Node coordinator = n1;
TxnId txnId = coordinator.nextTxnId(Txn.Kind.Read,
Routable.Domain.Key);
Txn txn = readOnly(keys);
AsyncChains.getUninterruptibly(n2.commandStores().unsafeForKey(n2key).execute(PreLoadContext.contextFor(txnId,
keys), store -> {
Ranges ranges = store.ranges().currentRanges();
PartialTxn partial = txn.slice(ranges, true);
FullKeyRoute route = keys.toRoute(n2RoutingKey);
// RoutingKey progressKey = n2RoutingKey.toUnseekable(); // if
this is non-null this passes
RoutingKey progressKey = null;
CheckedCommands.preaccept(store, txnId, partial, route,
progressKey);
CheckedCommands.accept(store, txnId, Ballot.ZERO,
route.slice(ranges), partial.keys().slice(ranges), progressKey, txnId,
PartialDeps.builder(ranges).build());
}));
AsyncChains.getUninterruptibly(new AsyncChains.Head<Outcome>() {
@Override
protected void start(BiConsumer<? super Outcome, Throwable>
callback) {
Invalidate.invalidate(coordinator, txnId,
keys.toUnseekables(), callback);
}
});
}
}
private static Txn readOnly(Seekables<?, ?> keys)
{
Read read = MockStore.read(keys);
Query query = Mockito.mock(Query.class);
return new Txn.InMemory(keys, read, query);
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
