Jason Kania created CASSANDRA-11426:
---------------------------------------
Summary: End of File Exception (EOF) on startup due to
HintsDescriptor read failure
Key: CASSANDRA-11426
URL: https://issues.apache.org/jira/browse/CASSANDRA-11426
Project: Cassandra
Issue Type: Bug
Environment: Debian Jesse patch current, Cassandra 3.0.3
Reporter: Jason Kania
I had a node go down and after restart, Cassandra failed to start with the
following error popping up. Since the error does not specify the file, it is
hard to correct. Running 'nodetool repair' also fails with an unhelpful error,
following this error below.
ERROR [main] 2016-03-24 10:49:12,347 CassandraDaemon.java:692 - Exception
encountered during startup
org.apache.cassandra.io.FSReadError: java.io.EOFException
at
org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:142)
~[apache-cassandra-3.0.3.jar:3.0.3]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[na:1.8.0_74]
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
~[na:1.8.0_74]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_74]
at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
~[na:1.8.0_74]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
~[na:1.8.0_74]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
~[na:1.8.0_74]
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
~[na:1.8.0_74]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[na:1.8.0_74]
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
~[na:1.8.0_74]
at org.apache.cassandra.hints.HintsCatalog.load(HintsCatalog.java:65)
~[apache-cassandra-3.0.3.jar:3.0.3]
at org.apache.cassandra.hints.HintsService.<init>(HintsService.java:87)
~[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.hints.HintsService.<clinit>(HintsService.java:62)
~[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.service.StorageProxy.<clinit>(StorageProxy.java:120)
~[apache-cassandra-3.0.3.jar:3.0.3]
at java.lang.Class.forName0(Native Method) ~[na:1.8.0_74]
at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_74]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:625)
~[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:610)
~[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:333)
[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679)
[apache-cassandra-3.0.3.jar:3.0.3]
Caused by: java.io.EOFException: null
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:803)
~[na:1.8.0_74]
at
org.apache.cassandra.hints.HintsDescriptor.deserialize(HintsDescriptor.java:237)
~[apache-cassandra-3.0.3.jar:3.0.3]
at
org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:138)
~[apache-cassandra-3.0.3.jar:3.0.3]
... 20 common frames omitted
nodetool repair error
root@bull:~# nodetool repair
error: null
-- StackTrace --
java.lang.AssertionError
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:105)
at
org.apache.cassandra.service.StorageService.getRangesForEndpoint(StorageService.java:3234)
at
org.apache.cassandra.service.StorageService.getLocalRanges(StorageService.java:230)
at
org.apache.cassandra.service.StorageService.repairAsync(StorageService.java:2935)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
