[
https://issues.apache.org/jira/browse/CASSANDRA-8743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16454794#comment-16454794
]
Joshua McKenzie commented on CASSANDRA-8743:
--------------------------------------------
Most of the issue on the Windows side was that, if anyone anywhere has a file
handle open to *anything* linked to the original root file handle (i.e. any
other hard links, etc), none of them are allowed to delete any of them. Not
cool, and no way around it, just how NTFS rolls. So files over a network share
probably had some kernel hook accessing / caching / who knows what (I wouldn't
put it past the [Cache
Manager|https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/subsystem/cache-memory-management/]
to be up to something) on the remote side, so boom.
As such, I didn't move on it any more because a) we recommended not using
network shares for data file storage anyway, and b) my digging kept hitting
hard places that I couldn't work around.
Somewhat surprised you see that behavior on Linux. Linux tends to move handles
to the /proc/$pid subsystem when things are deleted but a process still has a
ref to the handle open somewhere, but I'm not sure what/how NFS does that on
the remote end when asked to delete something. Your hypothesis doesn't seem
that out of line to me, but yeah - parallel repair would be the immediate
workaround for this.
> NFS doesn't behave on Windows
> -----------------------------
>
> Key: CASSANDRA-8743
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8743
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Streaming and Messaging
> Reporter: Tamar Nirenberg
> Priority: Minor
> Fix For: 2.2.x
>
> Attachments: docker-system.log
>
>
> Running repair over NFS in Cassandra 2.1.2 encounters this error and crashes
> the ring:
> ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,811 Validator.java:232 -
> Failed creating a merkle tree for [repair
> #c84c7c70-a21b-11e4-aeca-19e6d7fa2595 on ATTRIBUTES/LINKS,
> (11621838520493020277529637175352775759,11853478749048239324667887059881170862]],
> /10.1.234.63 (see log for details)
> ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,827 CassandraDaemon.java:153
> - Exception in thread Thread[ValidationExecutor:2,1,main]
> org.apache.cassandra.io.FSWriteError:
> java.nio.file.DirectoryNotEmptyException:
> /exlibris/cassandra/local/data/data/ATTRIBUTES/LINKS/snapshots/c84c7c70-a21b-11e4-aeca-19e6d7fa2595
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:381)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:547)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:2223)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:939)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:557)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> Caused by: java.nio.file.DirectoryNotEmptyException:
> /exlibris/cassandra/local/data/data/ATTRIBUTES/LINKS/snapshots/c84c7c70-a21b-11e4-aeca-19e6d7fa2595
> at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
> ~[na:1.7.0_71]
> at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
> ~[na:1.7.0_71]
> at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71]
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> ... 10 common frames omitted
> ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,829 StorageService.java:383
> - Stopping gossiper
> WARN [ValidationExecutor:2] 2015-01-22 11:48:14,829 StorageService.java:291
> - Stopping gossip by operator request
> INFO [ValidationExecutor:2] 2015-01-22 11:48:14,829 Gossiper.java:1318 -
> Announcing shutdown
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
