[
https://issues.apache.org/jira/browse/CASSANDRA-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531121#comment-15531121
]
Sharvanath Pathak edited comment on CASSANDRA-11678 at 9/28/16 10:55 PM:
-------------------------------------------------------------------------
We have reproduced the same case, and found out that the hint file was
truncated because there was a hard reboot around the time of last write to the
file. I think we basically need to handle partially written hint files. Also,
the CRC file does not exist in this case (probably because it crashed while
writing the hints file). May be ignoring the hints file if the CRC file is
absent can be a way to fix this?
was (Author: sharvanath):
We have reproduced the same case, and found out that the hint file truncated
because there was a hard reboot around the time of last write to the file. I
think we basically need to handle partially written hint files. Also, the CRC
file does not exist in this case (probably because it crashed while writing the
hints file). May be ignoring the hints file if the CRC file is absent can be a
way to fix this?
> cassandra crush when enable hints_compression
> ---------------------------------------------
>
> Key: CASSANDRA-11678
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11678
> Project: Cassandra
> Issue Type: Bug
> Components: Core, Local Write-Read Paths
> Environment: Centos 7
> Reporter: Weijian Lin
> Assignee: Blake Eggleston
> Priority: Critical
>
> When I enable hints_compression and set the compression class to
> LZ4Compressor，the
> cassandra (v3.05, V3.5.0) will crush。That is a bug, or any conf is wrong?
> *Exception in V 3.5.0 *
> {code}
> ERROR [HintsDispatcher:2] 2016-04-26 15:02:56,970
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> abc4dda2-b551-427e-bb0b-e383d4a392e1-1461654138963-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.EOFException: null
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readInt(RebufferingInputStream.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:297)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 15 common frames omitted
> {code}
> *Exception in V 3.0.5 *
> {code}
> ERROR [HintsDispatcher:2] 2016-04-26 15:54:46,294
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> 8603be13-6878-4de3-8bc3-a7a7146b0376-1461657251205-1.hints: file is
> corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
> [apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
> [apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
> [apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
> [apache-cassandra-3.0.5.jar:3.0.5]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.EOFException: null
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readInt(RebufferingInputStream.java:188)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:295)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:278)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> ... 15 common frames omitted
> ERROR [HintsDispatcher:2] 2016-04-26 15:54:46,296 CassandraDaemon.java:195
> Exception in thread Thread[HintsDispatcher:2,1,main]
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:282)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:252)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_65]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: java.io.EOFException: null
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readInt(RebufferingInputStream.java:188)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:295)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:278)
> ~[apache-cassandra-3.0.5.jar:3.0.5]
> ... 15 common frames omitted
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
