[
https://issues.apache.org/jira/browse/CASSANDRA-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13835321#comment-13835321
]
Andreas Schnitzerling edited comment on CASSANDRA-6283 at 11/29/13 11:51 AM:
-----------------------------------------------------------------------------
I think I can cancel repair without -par. After repairing one keyspace I got al
lot of errors but not the leak-detect-messages. I think, not RAR is responsible
during repair. The error on the Validation Node:
{panel:title=system.log}
ERROR [ValidationExecutor:1] 2013-11-29 12:15:38,370 Validator.java (line 242)
Failed creating a merkle tree for [repair #92b6ccb0-58e7-11e3-aac3-b13a5fe180aa
on nieste/niesteplants, (-5215786285174483271,-5206407297765302700]],
/10.6.8.78 (see log for details)
ERROR [ValidationExecutor:1] 2013-11-29 12:15:38,370 CassandraDaemon.java (line
187) Exception in thread Thread[ValidationExecutor:1,1,main]
FSWriteError in
D:\Programme\cassandra\data\nieste\niesteplants\snapshots\92b6ccb0-58e7-11e3-aac3-b13a5fe180aa\nieste-niesteplants-jb-19-Data.db
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:120)
at
org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:382)
at
org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:378)
at
org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:416)
at
org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1801)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:810)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:397)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.FileSystemException:
D:\Programme\cassandra\data\nieste\niesteplants\snapshots\92b6ccb0-58e7-11e3-aac3-b13a5fe180aa\nieste-niesteplants-jb-19-Data.db:
Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen
Prozess verwendet wird.
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
at java.nio.file.Files.delete(Unknown Source)
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:116)
... 11 more
{panel}
The error on the repair-node:
{panel:title=system.log}
ERROR [AntiEntropySessions:1] 2013-11-29 12:15:38,419 RepairSession.java (line
278) [repair #92b6ccb0-58e7-11e3-aac3-b13a5fe180aa] session completed with the
following error
org.apache.cassandra.exceptions.RepairException: [repair
#92b6ccb0-58e7-11e3-aac3-b13a5fe180aa on nieste/niesteplants,
(-5215786285174483271,-5206407297765302700]] Validation failed in /10.9.9.69
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:188)
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
ERROR [AntiEntropySessions:1] 2013-11-29 12:15:38,429 CassandraDaemon.java
(line 187) Exception in thread Thread[AntiEntropySessions:1,5,RMI Runtime]
java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
[repair #92b6ccb0-58e7-11e3-aac3-b13a5fe180aa on nieste/niesteplants,
(-5215786285174483271,-5206407297765302700]] Validation failed in /10.9.9.69
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#92b6ccb0-58e7-11e3-aac3-b13a5fe180aa on nieste/niesteplants,
(-5215786285174483271,-5206407297765302700]] Validation failed in /10.9.9.69
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:152)
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:188)
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:59)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
... 3 more
{panel}
{panel:title=NodeTool}
Starting NodeTool
[2013-11-29 12:15:37,329] Starting repair command #1, repairing 256 ranges for
keyspace nieste
[2013-11-29 12:19:46,219] Repair session 92b6ccb0-58e7-11e3-aac3-b13a5fe180aa
for range (-5215786285174483271,-5206407297765302700] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#92b6ccb0-58e7-11e3-aac3-b13a5fe180aa on nieste/niesteplants,
(-5215786285174483271,-5206407297765302700]] Validation failed in /10.9.9.69
[2013-11-29 12:19:46,219] Repair session 935b9830-58e7-11e3-aac3-b13a5fe180aa
for range (8782854129978293476,8784492934430401685] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#935b9830-58e7-11e3-aac3-b13a5fe180aa on nieste/niesteplants,
(8782854129978293476,8784492934430401685]] Validation failed in /10.9.9.69
[2013-11-29 12:19:46,219] Repair session 93b725b0-58e7-11e3-aac3-b13a5fe180aa
for range (-5799639987122737930,-5786898749415113092] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#93b725b0-58e7-11e3-aac3-b13a5fe180aa on nieste/nfiles,
(-5799639987122737930,-5786898749415113092]] Validation failed in /10.9.9.240
[2013-11-29 12:19:46,229] Repair session 94b16430-58e7-11e3-aac3-b13a5fe180aa
for range (2664807773952357126,2669403005419855407] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#94b16430-58e7-11e3-aac3-b13a5fe180aa on nieste/nfiles,
(2664807773952357126,2669403005419855407]] Validation failed in /10.9.9.240
[2013-11-29 12:19:46,229] Repair session 9503c9f0-58e7-11e3-aac3-b13a5fe180aa
for range (-6420115574437655437,-6410524043851626540] finished
{panel}
Im curious, if cache-file-deleting will write leak-messages...
I'm using C* 2.0.3-release with leak-detection.patch.
was (Author: andie78):
I think I can cancel repair without -par. After repairing one keyspace I got al
lot of errors but not the leak-detect-messages. I think, not RAR is responsible
during repair. The error on the Validation Node:
{panel:title=system.log}
ERROR [ValidationExecutor:1] 2013-11-29 12:15:38,370 Validator.java (line 242)
Failed creating a merkle tree for [repair #92b6ccb0-58e7-11e3-aac3-b13a5fe180aa
on nieste/niesteplants, (-5215786285174483271,-5206407297765302700]],
/10.6.8.78 (see log for details)
ERROR [ValidationExecutor:1] 2013-11-29 12:15:38,370 CassandraDaemon.java (line
187) Exception in thread Thread[ValidationExecutor:1,1,main]
FSWriteError in
D:\Programme\cassandra\data\nieste\niesteplants\snapshots\92b6ccb0-58e7-11e3-aac3-b13a5fe180aa\nieste-niesteplants-jb-19-Data.db
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:120)
at
org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:382)
at
org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:378)
at
org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:416)
at
org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1801)
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:810)
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:397)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.FileSystemException:
D:\Programme\cassandra\data\nieste\niesteplants\snapshots\92b6ccb0-58e7-11e3-aac3-b13a5fe180aa\nieste-niesteplants-jb-19-Data.db:
Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen
Prozess verwendet wird.
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
at java.nio.file.Files.delete(Unknown Source)
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:116)
... 11 more
{panel}
The error on the repair-node:
{panel:title=system.log}
Starting NodeTool
[2013-11-29 12:15:37,329] Starting repair command #1, repairing 256 ranges for
keyspace nieste
[2013-11-29 12:19:46,219] Repair session 92b6ccb0-58e7-11e3-aac3-b13a5fe180aa
for range (-5215786285174483271,-5206407297765302700] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#92b6ccb0-58e7-11e3-aac3-b13a5fe180aa on nieste/niesteplants,
(-5215786285174483271,-5206407297765302700]] Validation failed in /10.9.9.69
[2013-11-29 12:19:46,219] Repair session 935b9830-58e7-11e3-aac3-b13a5fe180aa
for range (8782854129978293476,8784492934430401685] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#935b9830-58e7-11e3-aac3-b13a5fe180aa on nieste/niesteplants,
(8782854129978293476,8784492934430401685]] Validation failed in /10.9.9.69
[2013-11-29 12:19:46,219] Repair session 93b725b0-58e7-11e3-aac3-b13a5fe180aa
for range (-5799639987122737930,-5786898749415113092] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#93b725b0-58e7-11e3-aac3-b13a5fe180aa on nieste/nfiles,
(-5799639987122737930,-5786898749415113092]] Validation failed in /10.9.9.240
[2013-11-29 12:19:46,229] Repair session 94b16430-58e7-11e3-aac3-b13a5fe180aa
for range (2664807773952357126,2669403005419855407] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#94b16430-58e7-11e3-aac3-b13a5fe180aa on nieste/nfiles,
(2664807773952357126,2669403005419855407]] Validation failed in /10.9.9.240
[2013-11-29 12:19:46,229] Repair session 9503c9f0-58e7-11e3-aac3-b13a5fe180aa
for range (-6420115574437655437,-6410524043851626540] finished
{panel}
Im curious, if cache-file-deleting will write leak-messages...
I'm using C* 2.0.3-release with leak-detection.patch.
> Windows 7 data files keept open / can't be deleted after compaction.
> --------------------------------------------------------------------
>
> Key: CASSANDRA-6283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Windows 7 (32) / Java 1.7.0.45
> Reporter: Andreas Schnitzerling
> Priority: Critical
> Labels: newbie, patch, test
> Fix For: 2.0.4
>
> Attachments: leakdetect.patch, screenshot-1.jpg, system.log
>
>
> Files cannot be deleted, patch CASSANDRA-5383 (Win7 deleting problem) doesn't
> help on Win-7 on Cassandra 2.0.2. Even 2.1 Snapshot is not running. The cause
> is: Opened file handles seem to be lost and not closed properly. Win 7
> blames, that another process is still using the file (but its obviously
> cassandra). Only restart of the server makes the files deleted. But after
> heavy using (changes) of tables, there are about 24K files in the data folder
> (instead of 35 after every restart) and Cassandra crashes. I experiminted and
> I found out, that a finalizer fixes the problem. So after GC the files will
> be deleted (not optimal, but working fine). It runs now 2 days continously
> without problem. Possible fix/test:
> I wrote the following finalizer at the end of class
> org.apache.cassandra.io.util.RandomAccessReader:
> {code:title=RandomAccessReader.java|borderStyle=solid}
> @Override
> protected void finalize() throws Throwable {
> deallocate();
> super.finalize();
> }
> {code}
> Can somebody test / develop / patch it? Thx.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
