Jens Rantil created CASSANDRA-11583:
---------------------------------------
Summary: Exception when streaming sstables using `sstableloader`
Key: CASSANDRA-11583
URL: https://issues.apache.org/jira/browse/CASSANDRA-11583
Project: Cassandra
Issue Type: Bug
Components: Tools
Environment: $ uname -a
Linux bigdb-100 3.2.0-99-virtual #139-Ubuntu SMP Mon Feb 1 23:52:21 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux
I am using Datastax Enterprise 4.7.8-1 which is based on 2.1.13.
Reporter: Jens Rantil
This bug came out of CASSANDRA-11562.
I have have a keyspace snapshotted from a 2.1.11 (DSE 4.7.5-1) node. When I'm
running the `sstableloader` I get the following output/exception:
{noformat}
# sstableloader --nodes X.X.X.20 --username YYY --password ZZZ --ignore XXX
/var/lib/cassandra/data/XXX/ZZZ-f7ebdf0daa3a3062828fddebc109a3b2
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of
/var/lib/cassandra/data/XXX/ZZZ-f7ebdf0daa3a3062828fddebc109a3b2/XXX-ZZZ-ka-6463-Data.db
/var/lib/cassandra/data/XXX/ZZZ-f7ebdf0daa3a3062828fddebc109a3b2/tink-ZZZ-ka-6464-Data.db
to [/X.X.X.33, /X.X.X.113, /X.X.X.32, /X.X.X.20, /X.X.X.122, /X.X.X.176,
/X.X.X.143, /X.X.X.172, /X.X.X.50, /X.X.X.51, /X.X.X.52, /X.X.X.71, /X.X.X.53,
/X.X.X.54, /X.X.X.47, /X.X.X.31, /X.X.X.8]
progress: [/X.X.X.113]0:0/2 0 % [/X.X.X.143]0:0/2 0 % [/X.X.X.172]0:0/2 0 %
[/X.X.X.20]0:0/2 0 % [/X.X.X.71]0:0/2 0 % [/X.X.X.122]0:0/2 0 %
[/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 0 % [/X.X.X.143]0:0/2 0 %
[/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 1 % [/X.X.X.71]0:0/2 0 %
[/X.X.X.122]0:0/2 0 % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 0 %
[/X.X.X.143]0:0/2 1 % [/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 1 %
[/X.X.X.71]0:0/2 0 % [/X.X.X.122]0:0/2 0 % [/X.X.X.47]0:0/2 progress:
[/X.X.X.113]0:0/2 0 % [/X.X.X.143]0:1/2 1 % [/X.X.X.172]0:0/2 0 %
[/X.X.X.20]0:1/2 1 % [/X.X.X.71]0:0/2 0 % [/X.X.X.122]0:0/2 0 %
[/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 0 % [/X.X.X.143]0:1/2 1 %
[/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 1 % [/X.X.X.71]0:1/2 1 %
[/X.X.X.122]0:0/2 0 % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 0 %
[/X.X.X.143]0:1/2 1 % [/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 1 %
[/X.X.X.71]0:1/2 1 % [/X.X.X.122]0:1/2 1 % [/X.X.X.47]0:0/2 progress:
[/X.X.X.113]0:0/2 0 % [/X.X.X.143]0:1/2 1 % [/X.X.X.172]0:0/2 0 %
[/X.X.X.20]0:1/2 1 % [/X.X.X.71]0:1/2 1 % [/X.X.X.122]0:1/2 1 %
[/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 7 % [/X.X.X.143]0:1/2 1 %
[/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 1 % [/X.X.X.71]0:1/2 1 %
[/X.X.X.122]0:1/2 1 % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 7 %
[/X.X.X.143]0:1/2 6 % [/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 1 %
[/X.X.X.71]0:1/2 1 % [/X.X.X.122]0:1/2 1 % [/X.X.X.47]0:0/2 progress:
[/X.X.X.113]0:0/2 12 % [/X.X.X.143]0:1/2 6 % [/X.X.X.172]0:0/2 0 %
[/X.X.X.20]0:1/2 1 % [/X.X.X.71]0:1/2 1 % [/X.X.X.122]0:1/2 1 %
[/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 12 % [/X.X.X.143]0:1/2 11 %
[/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 1 % [/X.X.X.71]0:1/2 1 %
[/X.X.X.122]0:1/2 1 % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 19 %
[/X.X.X.143]0:1/2 11 % [/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 1 %
[/X.X.X.71]0:1/2 1 % [/X.X.X.122]0:1/2 1 % [/X.X.X.47]0:0/2 progress:
[/X.X.X.113]0:0/2 19 % [/X.X.X.143]0:1/2 15 % [/X.X.X.172]0:0/2 0 %
[/X.X.X.20]0:1/2 1 % [/X.X.X.71]0:1/2 1 % [/X.X.X.122]0:1/2 1 %
[/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 26 % [/X.X.X.143]0:1/2 15 %
[/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 1 % [/X.X.X.71]0:1/2 1 %
[/X.X.X.122]0:1/2 1 % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 26 %
[/X.X.X.143]0:1/2 20 % [/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 1 %
[/X.X.X.71]0:1/2 1 % [/X.X.X.122]0:1/2 1 % [/X.X.X.47]0:0/2 progress:
[/X.X.X.113]0:0/2 26 % [/X.X.X.143]0:1/2 21 % [/X.X.X.172]0:0/2 0 %
[/X.X.X.20]0:1/2 1 % [/X.X.X.71]0:1/2 1 % [/X.X.X.122]0:1/2 1 %
[/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 26 % [/X.X.X.143]0:1/2 21 %
[/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 3 % [/X.X.X.71]0:1/2 1 %
[/X.X.X.122]0:1/2 1 % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:0/2 42 %
[/X.X.X.143]0:1/2 27 % [/X.X.X.172]0:0/2 0 % [/X.X.X.20]0:1/2 3 %
[/X.X.X.71]0:1/2 6 % [/X.X.X.122]0:1/2 1 % [/X.X.X.47]0:0/2
[...]
progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 78 %
[/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:1/2 97 %
[/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100%
[/X.X.X.172]0:0/2 78 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100%
[/X.X.X.122]0:1/2 97 % [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100%
[/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 86 % [/X.X.X.20]0:2/2 100%
[/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:0/2 progress:
[/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 86 %
[/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100%
[/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100%
[/X.X.X.172]0:0/2 86 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100%
[/X.X.X.122]0:2/2 100% [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100%
[/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 86 % [/X.X.X.20]0:2/2 100%
[/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:0/2 progress:
[/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 86 %
[/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100%
[/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100%
[/X.X.X.172]0:0/2 95 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100%
[/X.X.X.122]0:2/2 100% [/X.X.X.47]0:0/2 progress: [/X.X.X.113]0:2/2 100%
[/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 95 % [/X.X.X.20]0:2/2 100%
[/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:1/2 progress:
[/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:0/2 95 %
[/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100%
[/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100%
[/X.X.X.172]0:0/2 95 % [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100%
[/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100%
[/X.X.X.143]0:2/2 100% [/X.X.X.172]0:1/2 98 % [/X.X.X.20]0:2/2 100%
[/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress:
[/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:1/2 99 %
[/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100%
[/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100%
[/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100%
[/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100%
[/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100%
[/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress:
[/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100%
[/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100%
[/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100%
[/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100%
[/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100%
[/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100%
[/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress:
[/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100%
[/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100%
[/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100%
[/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100%
[/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100%
[/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100%
[/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 progress:
[/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100% [/X.X.X.172]0:2/2 100%
[/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100% [/X.X.X.122]0:2/2 100%
[/X.X.X.47]0:2/2 progress: [/X.X.X.113]0:2/2 100% [/X.X.X.143]0:2/2 100%
[/X.X.X.172]0:2/2 100% [/X.X.X.20]0:2/2 100% [/X.X.X.71]0:2/2 100%
[/X.X.X.122]0:2/2 100% [/X.X.X.47]0:2/2 100% [/X.X.X.176]0:2/2 100%
[/X.X.X.8]0:2/2 100% total: 100% 0 MB/s(avg: 23 MB/s)
Summary statistics:
Connections per host: : 1
Total files transferred: : 14
Total bytes transferred: : 142070241
Total duration (ms): : 5823
Average transfer rate (MB/s): : 23
Peak transfer rate (MB/s): : 33
ERROR 09:07:12 [Stream #6e345de0-02e9-11e6-b944-1fee10beb91b] Streaming error
occurred
java.lang.AssertionError: null
at
org.apache.cassandra.io.sstable.SSTableLoader.releaseReferences(SSTableLoader.java:208)
~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
at
org.apache.cassandra.io.sstable.SSTableLoader.onSuccess(SSTableLoader.java:193)
~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
at
org.apache.cassandra.io.sstable.SSTableLoader.onSuccess(SSTableLoader.java:48)
~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1181)
~[guava-16.0.1.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.1.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-16.0.1.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-16.0.1.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)
~[guava-16.0.1.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184)
~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:415)
~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
at
org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:607)
~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:471)
~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:263)
~[cassandra-all-2.1.13.1218.jar:2.1.13.1218]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
{noformat}
Question: Is it an issue that I dumped the sstables using a _slightly_ older
Cassandra node?
Let me know if you need anything else from me. Also, know of any possible for
workaround?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
