[
https://issues.apache.org/jira/browse/CASSANDRA-8743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568892#comment-14568892
]
Nicolas Guyomar commented on CASSANDRA-8743:
HI,
I just reproduced this 'exact' same issue on 2.1.5 while doing a repair
(nodetool repair) after changing my cluster snitch
My cluster is also on an NFS mounted drive
I ran again the repair and got the same exception (keyspace name and table are
voluntary changed in the stack)
INFO [AntiEntropyStage:1] 2015-06-02 10:56:29,133 Validator.java:257 - [repair
#412b6e80-0905-11e5-aa92-f327be1d597d] Sending completed merkle tree to
/10.106.199.9 for mykeyspace/mytable2
ERROR [ValidationExecutor:3] 2015-06-02 10:56:29,142 Validator.java:245 -
Failed creating a merkle tree for [repair #412b6e80-0905-11e5-aa92-f327be1d597d
on mykeyspace/mytable, (1997653793039594215,2135846523050696776]],
/10.106.199.9 (see log for details)
ERROR [ValidationExecutor:3] 2015-06-02 10:56:29,158 CassandraDaemon.java:223 -
Exception in thread Thread[ValidationExecutor:3,1,main]
org.apache.cassandra.io.FSWriteError: java.nio.file.DirectoryNotEmptyException:
/var/opt/data/flat/cassandra/mykeyspace/mytable-648d9b70abc011e49f8747338c540143/snapshots/412b6e80-0905-11e5-aa92-f327be1d597d
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135)
~[apache-cassandra-2.1.5.jar:2.1.5]
at
org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:381)
~[apache-cassandra-2.1.5.jar:2.1.5]
at
org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:616)
~[apache-cassandra-2.1.5.jar:2.1.5]
at
org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:2341)
~[apache-cassandra-2.1.5.jar:2.1.5]
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1013)
~[apache-cassandra-2.1.5.jar:2.1.5]
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:95)
~[apache-cassandra-2.1.5.jar:2.1.5]
at
org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:617)
~[apache-cassandra-2.1.5.jar:2.1.5]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.7.0_67]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_67]
Caused by: java.nio.file.DirectoryNotEmptyException:
/var/opt/data/flat/cassandra/mykeyspace/mytable-648d9b70abc011e49f8747338c540143/snapshots/412b6e80-0905-11e5-aa92-f327be1d597d
at sun.nio.fs.UnixFileSystemProvider.implDelete(Unknown Source)
~[na:1.7.0_67]
at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
~[na:1.7.0_67]
at java.nio.file.Files.delete(Unknown Source) ~[na:1.7.0_67]
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131)
~[apache-cassandra-2.1.5.jar:2.1.5]
... 10 common frames omitted
ERROR [ValidationExecutor:3] 2015-06-02 10:56:29,166 StorageService.java:445 -
Stopping gossiper
WARN [ValidationExecutor:3] 2015-06-02 10:56:29,167 StorageService.java:351 -
Stopping gossip by operator request
INFO [ValidationExecutor:3] 2015-06-02 10:56:29,167 Gossiper.java:1410 -
Announcing shutdown
Repair on NFS in version 2.1.2
--
Key: CASSANDRA-8743
URL: https://issues.apache.org/jira/browse/CASSANDRA-8743
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Tamar Nirenberg
Assignee: Joshua McKenzie
Priority: Minor
Running repair over NFS in Cassandra 2.1.2 encounters this error and crashes
the ring:
ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,811 Validator.java:232 -
Failed creating a merkle tree for [repair
#c84c7c70-a21b-11e4-aeca-19e6d7fa2595 on ATTRIBUTES/LINKS,
(11621838520493020277529637175352775759,11853478749048239324667887059881170862]],
/10.1.234.63 (see log for details)
ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,827 CassandraDaemon.java:153
- Exception in thread Thread[ValidationExecutor:2,1,main]
org.apache.cassandra.io.FSWriteError:
java.nio.file.DirectoryNotEmptyException:
/exlibris/cassandra/local/data/data/ATTRIBUTES/LINKS/snapshots/c84c7c70-a21b-11e4-aeca-19e6d7fa2595
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:381)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:547)
~[apache-cassandra-2.1.2.jar:2.1.2]
at