[
https://issues.apache.org/jira/browse/CASSANDRA-13575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691872#comment-16691872
]
C. Scott Andreas commented on CASSANDRA-13575:
----------------------------------------------
Hi [~hkroger], thanks for replying and writing back. The challenge is typically
working backward from the stacktrace to identifying the set of conditions that
trigger it. Sometimes that can happen more quickly on the users' list as many
more people are watching who may have encountered a similar issue.
I see a similar report in CASSANDRA-6716, in which [~snazy] reported the same
issue occurring after `sstablescrub` had been run after the Cassandra process
was launched.
The stacktrace you've shared shows that snapshot's trying to create a hardlink
to a file that's not present on disk. Is there any chance a tool outside the
Cassandra daemon's process may have modified the files present on disk while
the process was running? (Also, can you confirm whether the path listed was
actually present on disk; and if so, do any logs show when or the cause of its
removal)?
(An earlier issue may have resulted in the same symptoms if a Keyspace was
dropped/recreated with the same name, but this was resolved in CASSANDRA-5202).
> Snapshot fails on IndexInfo
> ---------------------------
>
> Key: CASSANDRA-13575
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13575
> Project: Cassandra
> Issue Type: Bug
> Reporter: Hannu Kröger
> Priority: Major
>
> Snapshot creation fails on IndexInfo table. This has happened in several
> Cassandra environments.
> There is also Stratio lucene index installed 2.2.3.1. I don't know if that
> matters.
> {code}
> [root@host1 IndexInfo-9f5c6374d48532299a0a5094af9ad1e3]# nodetool snapshot -t
> testsnapshot
> Requested creating snapshot(s) for [all keyspaces] with snapshot name
> [testsnapshot]
> error: Tried to hard link to file that does not exist
> /cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/la-264-big-Filter.db
> -- StackTrace --
> java.lang.RuntimeException: Tried to hard link to file that does not exist
> /cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/la-264-big-Filter.db
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:85)
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.createLinks(SSTableReader.java:1763)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:2328)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2453)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2443)
> at org.apache.cassandra.db.Keyspace.snapshot(Keyspace.java:198)
> at
> org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:2604)
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> [root@host1 IndexInfo-9f5c6374d48532299a0a5094af9ad1e3]#
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
