Kenneth Failbus created CASSANDRA-8898:
------------------------------------------
Summary: sstableloader utility should allow loading of data from
mounted filesystem
Key: CASSANDRA-8898
URL: https://issues.apache.org/jira/browse/CASSANDRA-8898
Project: Cassandra
Issue Type: Improvement
Components: Tools
Environment: 2.0.12
Reporter: Kenneth Failbus
When trying to load data from a mounted filesystem onto a new cluster,
following exceptions is observed intermittently, and at some point the
sstableloader process gets hung without completing the loading process.
Please note that in my case the scenario was loading the existing sstables from
an existing cluster to a brand new cluster.
Finally, it was found that there were some hard assumptions been made by
sstableloader utility w.r.t response from the filesystem, which was not working
with mounted filesystem.
The work-around was to copy each existing nodes sstable data files locally and
then point sstableloader to that local filesystem to then load data.
In case of restoring during disaster recovery from backups the data using
sstableloader, this copying to local filesystem of data files and then loading
would take a long time.
It would be a good enhancement of the sstableloader utility to enable use of
mounted filesystem as copying data locally and then loading is time consuming.
Below is the exception seen during the use of mounted filesystem.
{code}
java.lang.AssertionError: Reference counter -1 for
/opt/tmp/casapp-c1-c00053-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5449-Data.db
at
org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)
at
org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)
at
org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)
at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)
at java.lang.Thread.run(Thread.java:744)
WARN 21:07:16,853 [Stream #3e5a5ba0-bdef-11e4-a975-5777dbff0945] Stream failed
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:59)
at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1406)
at
org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:55)
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:59)
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)
at
org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:311)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException:
/opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5997-Data.db
(No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55)
... 8 more
Exception in thread "STREAM-OUT-/96.115.88.196" java.lang.NullPointerException
at
org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
at java.lang.Thread.run(Thread.java:744)
ERROR 20:49:35,646 [Stream #d9fce650-bdf3-11e4-b6c0-252cb9b3e9f3] Streaming
error occurred
java.lang.AssertionError: Reference counter -3 for
/opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-4897-Data.db
at
org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)
at
org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)
at
org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)
at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)
at java.lang.Thread.run(Thread.java:744)
Exception in thread "STREAM-IN-/96.115.88.196" java.lang.NullPointerException
at
org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:260)
at java.lang.Thread.run(Thread.java:744)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
