Hello, It seems that is a problem with Cassandra trying to delete a directory that is not empty. First time seeing this error. Are your harddrives with enough space and working correctly?
Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo <http://linkedin.com/in/carlosjuzarterolo>* Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com On Thu, Jul 16, 2015 at 2:45 PM, Modha, Digant < digant.mo...@tdsecurities.com> wrote: > Hi, > > > > We’re using Cassandra 2.0.10 ( 2 DC, 3 Nodes each RF=3 for each DC). > During one of the weekly repairs, we received the following error: > > > > ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,992 Validator.java > (line 242) Failed creating a merkle tree for [repair > #d2178ba0-2902-11e5-bd95-f14c61d86b85 on dmds/curve_dates, > (-1942303675502999131,-1890400428284965630]], / (see log for details) > > ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,992 > CassandraDaemon.java (line 199) Exception in thread > Thread[ValidationExecutor:1280,1,main] > > FSWriteError in > /apps/data/cassandra/dmds/data/dmds/curve_dates/snapshots/d2178ba0-2902-11e5-bd95-f14c61d86b85 > > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:122) > > at > org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:384) > > at > org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:488) > > at > org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1877) > > at > org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:811) > > at > org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63) > > at > org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:398) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.nio.file.DirectoryNotEmptyException: > /apps/data/cassandra/dmds/data/dmds/curve_dates/snapshots/d2178ba0-2902-11e5-bd95-f14c61d86b85 > > at > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) > > at > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > > at java.nio.file.Files.delete(Files.java:1126) > > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:118) > > ... 10 more > > ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,993 > StorageService.java (line 364) Stopping gossiper > > WARN [ValidationExecutor:1280] 2015-07-12 22:18:10,993 StorageService.java > (line 278) Stopping gossip by operator request > > INFO [ValidationExecutor:1280] 2015-07-12 22:18:10,993 Gossiper.java (line > 1279) Announcing shutdown > > Has anybody seen this error? The drives are local. Once this happened, > the other node performing repair maxed out the CPU and cluster became > unresponsive. > > > > > > Thanks, > > dm > > > > > TD Securities disclaims any liability or losses either direct or > consequential caused by the use of this information. This communication is > for informational purposes only and is not intended as an offer or > solicitation for the purchase or sale of any financial instrument or as an > official confirmation of any transaction. TD Securities is neither making > any investment recommendation nor providing any professional or advisory > services relating to the activities described herein. All market prices, > data and other information are not warranted as to completeness or accuracy > and are subject to change without notice Any products described herein are > (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or > guaranteed by, an insured depository institution and (iii) subject to > investment risks, including possible loss of the principal amount invested. > The information shall not be further distributed or duplicated in whole or > in part by any means without the prior written consent of TD Securities. TD > Securities is a trademark of The Toronto-Dominion Bank and represents TD > Securities (USA) LLC and certain investment banking activities of The > Toronto-Dominion Bank and its subsidiaries. > -- --
