Hi All. Taking snapshots sometimes works, sometimes don't. Following is the stacktrace whenever the process fails ::
############################################################################################## ajay@ajay-HP-15-Notebook-PC:/var/lib/cassandra/data/instamsg$ nodetool -h localhost -p 7199 snapshot instamsgRequested creating snapshot(s) for [instamsg] with snapshot name [1442298538121] error: /var/lib/cassandra/data/instamsg/clients-b32f01b02eec11e5866887c3880d7c45/snapshots/1442298538121/instamsg-clients-ka-15-TOC.txt -> /var/lib/cassandra/data/instamsg/clients-b32f01b02eec11e5866887c3880d7c45/instamsg-clients-ka-15-TOC.txt: Operation not permitted -- StackTrace -- java.nio.file.FileSystemException: /var/lib/cassandra/data/instamsg/clients-b32f01b02eec11e5866887c3880d7c45/snapshots/1442298538121/instamsg-clients-ka-15-TOC.txt -> /var/lib/cassandra/data/instamsg/clients-b32f01b02eec11e5866887c3880d7c45/instamsg-clients-ka-15-TOC.txt: Operation not permitted at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476) at java.nio.file.Files.createLink(Files.java:1086) at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:94) at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1842) at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:2279) at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2361) at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2355) at org.apache.cassandra.db.Keyspace.snapshot(Keyspace.java:207) at org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:2388) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1466) at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:828) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$251(TCPTransport.java:683) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$1/13812661.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ############################################################################################## For above example, I have checked that the following exist :: (file) :: /var/lib/cassandra/data/instamsg/clients-b32f01b02eec11e5866887c3880d7c45/instamsg-clients-ka-15-TOC.txt (directory) :: /var/lib/cassandra/data/instamsg/clients-b32f01b02eec11e5866887c3880d7c45/snapshots/1442298538121 My machine is Ubuntu-14.04, 32-bit processor+OS. Trying the above using "sudo" does not make any difference. What I am doing wrong? The thing that worries me is that things sometimes work, and sometimes don't :-\ Thanks and Regards, Ajay