[ https://issues.apache.org/jira/browse/CASSANDRA-8743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16454794#comment-16454794 ]
Joshua McKenzie commented on CASSANDRA-8743: -------------------------------------------- Most of the issue on the Windows side was that, if anyone anywhere has a file handle open to *anything* linked to the original root file handle (i.e. any other hard links, etc), none of them are allowed to delete any of them. Not cool, and no way around it, just how NTFS rolls. So files over a network share probably had some kernel hook accessing / caching / who knows what (I wouldn't put it past the [Cache Manager|https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/subsystem/cache-memory-management/] to be up to something) on the remote side, so boom. As such, I didn't move on it any more because a) we recommended not using network shares for data file storage anyway, and b) my digging kept hitting hard places that I couldn't work around. Somewhat surprised you see that behavior on Linux. Linux tends to move handles to the /proc/$pid subsystem when things are deleted but a process still has a ref to the handle open somewhere, but I'm not sure what/how NFS does that on the remote end when asked to delete something. Your hypothesis doesn't seem that out of line to me, but yeah - parallel repair would be the immediate workaround for this. > NFS doesn't behave on Windows > ----------------------------- > > Key: CASSANDRA-8743 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8743 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Streaming and Messaging > Reporter: Tamar Nirenberg > Priority: Minor > Fix For: 2.2.x > > Attachments: docker-system.log > > > Running repair over NFS in Cassandra 2.1.2 encounters this error and crashes > the ring: > ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,811 Validator.java:232 - > Failed creating a merkle tree for [repair > #c84c7c70-a21b-11e4-aeca-19e6d7fa2595 on ATTRIBUTES/LINKS, > (11621838520493020277529637175352775759,11853478749048239324667887059881170862]], > /10.1.234.63 (see log for details) > ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,827 CassandraDaemon.java:153 > - Exception in thread Thread[ValidationExecutor:2,1,main] > org.apache.cassandra.io.FSWriteError: > java.nio.file.DirectoryNotEmptyException: > /exlibris/cassandra/local/data/data/ATTRIBUTES/LINKS/snapshots/c84c7c70-a21b-11e4-aeca-19e6d7fa2595 > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:381) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:547) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:2223) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:939) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:97) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at > org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:557) > ~[apache-cassandra-2.1.2.jar:2.1.2] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_71] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_71] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_71] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71] > Caused by: java.nio.file.DirectoryNotEmptyException: > /exlibris/cassandra/local/data/data/ATTRIBUTES/LINKS/snapshots/c84c7c70-a21b-11e4-aeca-19e6d7fa2595 > at > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) > ~[na:1.7.0_71] > at > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > ~[na:1.7.0_71] > at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71] > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131) > ~[apache-cassandra-2.1.2.jar:2.1.2] > ... 10 common frames omitted > ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,829 StorageService.java:383 > - Stopping gossiper > WARN [ValidationExecutor:2] 2015-01-22 11:48:14,829 StorageService.java:291 > - Stopping gossip by operator request > INFO [ValidationExecutor:2] 2015-01-22 11:48:14,829 Gossiper.java:1318 - > Announcing shutdown -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org