[
https://issues.apache.org/jira/browse/CASSANDRA-11345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280328#comment-15280328
]
Vincent de Lagabbe commented on CASSANDRA-11345:
------------------------------------------------
Ok, I'm sorry, I completely missed this, there was a timeout error prior to the
"Memory was freed" error for this particular stream session:
{noformat}
ERROR [STREAM-IN-/172.31.45.28] 2016-05-11 13:10:43,842 StreamSession.java:505
- [Stream #ecfe0390-1763-11e6-b6c8-c1820b05e9ae] Streaming error occurred
java.net.SocketTimeoutException: null
at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
~[na:1.7.0_80]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
~[na:1.7.0_80]
at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
~[na:1.7.0_80]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:257)
~[cassandra-all-2.1.14.1272.jar:2.1.14.1272]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
{noformat}
This particular file is mentioned only once in the log prior to the "Finished
streaming" message, when is started to be streamed (with apparently the same
size):
{noformat}
DEBUG [STREAM-OUT-/172.31.45.28] 2016-05-11 12:10:23,960
CompressedStreamWriter.java:60 - [Stream #ecfe0390-1763-11e6-b6c8-c1820b05e9ae]
Start streaming file /raid0/cassandra/data/SSTABLEFILE-Data.db to
/172.31.45.28, repairedAt = 0, totalSize = 5854462750
{noformat}
There are compactions for sstables of the same keyspace/table but not for this
particular file.
It still exist, weighting 52595900317 bytes and was last modified on Apr 16
> Assertion Errors "Memory was freed" during streaming
> ----------------------------------------------------
>
> Key: CASSANDRA-11345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11345
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Reporter: Jean-Francois Gosselin
> Assignee: Paulo Motta
>
> We encountered the following AssertionError (twice on the same node) during a
> repair :
> On node /172.16.63.41
> {noformat}
> INFO [STREAM-IN-/10.174.216.160] 2016-03-09 02:38:13,900
> StreamResultFuture.java:180 - [Stream #f6980580-e55f-11e5-8f08-ef9e099ce99e]
> Session with /10.174.216.160 is complete
>
> WARN [STREAM-IN-/10.174.216.160] 2016-03-09 02:38:13,900
> StreamResultFuture.java:207 - [Stream #f6980580-e55f-11e5-8f08-ef9e099ce99e]
> Stream failed
> ERROR [STREAM-OUT-/10.174.216.160] 2016-03-09 02:38:13,906
> StreamSession.java:505 - [Stream #f6980580-e55f-11e5-8f08-ef9e099ce99e]
> Streaming error occurred
> java.lang.AssertionError: Memory was freed
>
>
> at
> org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:97)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
>
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:249)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
>
> at
> org.apache.cassandra.io.compress.CompressionMetadata.getTotalSizeForSections(CompressionMetadata.java:247)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.messages.FileMessageHeader.size(FileMessageHeader.java:112)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.StreamSession.fileSent(StreamSession.java:546)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
>
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
>
>
> {noformat}
> On node /10.174.216.160
>
> {noformat}
> ERROR [STREAM-OUT-/172.16.63.41] 2016-03-09 02:38:14,140
> StreamSession.java:505 - [Stream #f6980580-e55f-11e5-8f08-ef9e099ce99e]
> Streaming error occurred
> java.io.IOException: Connection reset by peer
>
>
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_65]
>
>
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> ~[na:1.7.0_65]
>
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> ~[na:1.7.0_65]
>
> at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_65]
>
>
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
> ~[na:1.7.0_65]
>
> at
> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
> [apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:323)
> [apache-cassandra-2.1.13.jar:2.1.13]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
>
>
> INFO [STREAM-IN-/172.16.63.41] 2016-03-09 02:38:14,142
> StreamResultFuture.java:180 - [Stream #f6980580-e55f-11e5-8f08-ef9e099ce99e]
> Session with /172.16.63.41 is complete
> WARN [STREAM-IN-/172.16.63.41] 2016-03-09 02:38:14,142
> StreamResultFuture.java:207 - [Stream #f6980580-e55f-11e5-8f08-ef9e099ce99e]
> Stream failed
> ERROR [STREAM-OUT-/172.16.63.41] 2016-03-09 02:38:14,143
> StreamSession.java:505 - [Stream #f6980580-e55f-11e5-8f08-ef9e099ce99e]
> Streaming error occurred
> java.io.IOException: Broken pipe
>
>
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_65]
>
>
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> ~[na:1.7.0_65]
>
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> ~[na:1.7.0_65]
>
> at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_65]
>
>
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
> ~[na:1.7.0_65]
>
> at
> org.apache.cassandra.io.util.DataOutputStreamAndChannel.write(DataOutputStreamAndChannel.java:48)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:44)
> ~[apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:351)
> [apache-cassandra-2.1.13.jar:2.1.13]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
> [apache-cassandra-2.1.13.jar:2.1.13]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
