[
https://issues.apache.org/jira/browse/CASSANDRA-18320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699095#comment-17699095
]
David Capwell commented on CASSANDRA-18320:
-------------------------------------------
agree with [~smiklosovic] here, this isn't a simulator issue but a jvm-dtest
Cluster.close issue
{code}
at
org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1047)
at
org.apache.cassandra.simulator.ClusterSimulation.close(ClusterSimulation.java:816)
at
org.apache.cassandra.simulator.SimulationRunner$Run.run(SimulationRunner.java:370)
at
org.apache.cassandra.simulator.SimulationRunner$BasicCommand.run(SimulationRunner.java:345)
at
org.apache.cassandra.simulator.paxos.PaxosSimulationRunner.main(PaxosSimulationRunner.java:148)
{code}
Since we got a "Caused by: java.nio.file.DirectoryNotEmptyException:
/cassandra/node1/commitlog" we then rethrow the error, but this looks like we
have a close bug...
{code}
@Override
public void close()
{
FBUtilities.waitOnFutures(instances.stream()
.filter(i -> !i.isShutdown())
.map(IInstance::shutdown)
.collect(Collectors.toList()),
1L, TimeUnit.MINUTES);
instances.clear();
instanceMap.clear();
PathUtils.setDeletionListener(ignore -> {});
// Make sure to only delete directory when threads are stopped
if (Files.exists(root))
PathUtils.deleteRecursive(root);
Thread.setDefaultUncaughtExceptionHandler(previousHandler);
previousHandler = null;
checkAndResetUncaughtExceptions();
//checkForThreadLeaks();
//withThreadLeakCheck(futures);
}
{code}
All our close futures were success, yet commit log touched a file while we were
deleting... so we have a concurrency issue with PathUtils.deleteRecursive(root);
bq. This property is set only in IDEA test run config but you see that there is
the intention to set it by default to true in tests without setting any flag.
Its hard for me to test this as the issue looks to be a concurrency issue, so
if `rm -rd` fails when the dir adds a file AFTER the dentry was loaded by rm I
can't say... but I don't think changing the implementation actually fixes this
bug, it looks like commit log said it was fully closed when it wasn't, so we
have concurrent file creation while we are deleting...
> simulationTest is flaky
> ------------------------
>
> Key: CASSANDRA-18320
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18320
> Project: Cassandra
> Issue Type: Bug
> Components: CI
> Reporter: Ekaterina Dimitrova
> Priority: Normal
> Fix For: 4.1.x, 5.x
>
>
> {code}
> java.io.UncheckedIOException
> at
> org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:831)
> at
> org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:816)
> at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:257)
> at
> org.apache.cassandra.io.util.PathUtils.deleteRecursive(PathUtils.java:381)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:395)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> at org.apache.cassandra.io.util.PathUtils.forEach(PathUtils.java:155)
> at
> org.apache.cassandra.io.util.PathUtils.deleteRecursive(PathUtils.java:378)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:395)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> at org.apache.cassandra.io.util.PathUtils.forEach(PathUtils.java:155)
> at
> org.apache.cassandra.io.util.PathUtils.deleteRecursive(PathUtils.java:378)
> at
> org.apache.cassandra.distributed.impl.AbstractCluster.close(AbstractCluster.java:1047)
> at
> org.apache.cassandra.simulator.ClusterSimulation.close(ClusterSimulation.java:816)
> at
> org.apache.cassandra.simulator.SimulationRunner$Run.run(SimulationRunner.java:370)
> at
> org.apache.cassandra.simulator.SimulationRunner$BasicCommand.run(SimulationRunner.java:345)
> at
> org.apache.cassandra.simulator.paxos.PaxosSimulationRunner.main(PaxosSimulationRunner.java:148)
> at
> org.apache.cassandra.simulator.test.ShortPaxosSimulationTest.simulationTest(ShortPaxosSimulationTest.java:33)
> Caused by: java.nio.file.DirectoryNotEmptyException:
> /cassandra/node1/commitlog
> at
> com.google.common.jimfs.FileSystemView.checkEmpty(FileSystemView.java:535)
> at
> com.google.common.jimfs.FileSystemView.checkDeletable(FileSystemView.java:517)
> at
> com.google.common.jimfs.FileSystemView.delete(FileSystemView.java:479)
> at
> com.google.common.jimfs.FileSystemView.deleteFile(FileSystemView.java:465)
> at
> com.google.common.jimfs.JimfsFileSystemProvider.delete(JimfsFileSystemProvider.java:261)
> at java.nio.file.Files.delete(Files.java:1126)
> at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:252)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
