Paul Cheon created CASSANDRA-14437:
--------------------------------------
Summary: SSTableLoader does not work when "internode_encryption :
all" is set
Key: CASSANDRA-14437
URL: https://issues.apache.org/jira/browse/CASSANDRA-14437
Project: Cassandra
Issue Type: Bug
Components: Tools
Reporter: Paul Cheon
Fix For: 3.11.2
I am trying to use sstableloader to restore snapshot.
If "internode_encryption : all" is set, then it does not work and complain
with below error messages. I initiated sstableloader from 10.1.10.203
(yvr-paul-cas003), so 10.1.10.203 worked fine, but the the other two nodes
(10.1.10.201 & 10.1.10.202 failed)
{noformat}
pcheon@yvr-paul-cas003:~/t$ sstableloader -v -d 10.1.10.203 office_audit/log/
-f /etc/cassandra/cassandra.yaml -u pcheon -pw `cat .secret`
WARN 17:23:45,166 Small commitlog volume detected at
/var/lib/cassandra/commitlog; setting commitlog_total_space_in_mb to 2316. You
can override this in cassandra.yaml
WARN 17:23:45,170 Small cdc volume detected at /var/lib/cassandra/cdc_raw;
setting cdc_total_space_in_mb to 1158. You can override this in cassandra.yaml
WARN 17:23:45,285 Only 5.318GiB free across all data volumes. Consider adding
more capacity to your cluster or removing obsolete snapshots
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of /home/pcheon/t/office_audit/log/mc-1083-big-Data.db
/home/pcheon/t/office_audit/log/mc-1100-big-Data.db
/home/pcheon/t/office_audit/log/mc-1101-big-Data.db
/home/pcheon/t/office_audit/log/mc-257-big-Data.db
/home/pcheon/t/office_audit/log/mc-984-big-Data.db to [/10.1.10.201,
/10.1.10.203, /10.1.10.202]
ERROR 17:23:49,460 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming
error occurred on session with peer 10.1.10.201
java.net.SocketException: Invalid argument or cannot assign requested address
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
~[na:1.8.0_112]
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
~[na:1.8.0_112]
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
~[na:1.8.0_112]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
~[na:1.8.0_112]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
~[na:1.8.0_112]
at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495)
~[na:1.8.0_112]
at
sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169)
~[na:1.8.0_112]
at
org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263)
[apache-cassandra-3.11.2.jar:3.11.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_112]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.2.jar:3.11.2]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
ERROR 17:23:49,458 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming
error occurred on session with peer 10.1.10.202
java.net.SocketException: Invalid argument or cannot assign requested address
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
~[na:1.8.0_112]
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
~[na:1.8.0_112]
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
~[na:1.8.0_112]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
~[na:1.8.0_112]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
~[na:1.8.0_112]
at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495)
~[na:1.8.0_112]
at
sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169)
~[na:1.8.0_112]
at
org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269)
~[apache-cassandra-3.11.2.jar:3.11.2]
at
org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263)
[apache-cassandra-3.11.2.jar:3.11.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_112]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.2.jar:3.11.2]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
progress: [/10.1.10.203]0:0/5 7 % total: 7% 13.858MiB/s (avg: 2.147MiB/s)
progress: [/10.1.10.203]0:0/5 15 % total: 15% 41.587MiB/s (avg: 4.083MiB/s)
progress: [/10.1.10.203]0:0/5 23 % total: 23% 75.631MiB/s (avg: 5.964MiB/s)
progress: [/10.1.10.203]0:0/5 31 % total: 31% 34.837MiB/s (avg: 7.523MiB/s)
progress: [/10.1.10.203]0:0/5 39 % total: 39% 21.321MiB/s (avg: 8.641MiB/s)
progress: [/10.1.10.203]0:1/5 41 % total: 41% 17.665MiB/s (avg: 8.766MiB/s)
progress: [/10.1.10.203]0:1/5 49 % total: 49% 17.841MiB/s (avg: 9.557MiB/s)
progress: [/10.1.10.203]0:1/5 56 % total: 56% 75.654MiB/s (avg: 10.889MiB/s)
progress: [/10.1.10.203]0:1/5 64 % total: 64% 21.252MiB/s (avg: 11.583MiB/s)
progress: [/10.1.10.203]0:1/5 72 % total: 72% 33.267MiB/s (avg: 12.472MiB/s)
progress: [/10.1.10.203]0:1/5 80 % total: 80% 22.969MiB/s (avg: 13.060MiB/s)
progress: [/10.1.10.203]0:2/5 81 % total: 81% 6.191MiB/s (avg: 12.951MiB/s)
progress: [/10.1.10.203]0:3/5 81 % total: 81% 100.225MiB/s (avg: 12.994MiB/s)
progress: [/10.1.10.203]0:3/5 89 % total: 89% 12.939MiB/s (avg: 12.989MiB/s)
progress: [/10.1.10.203]0:3/5 97 % total: 97% 65.803MiB/s (avg: 13.899MiB/s)
progress: [/10.1.10.203]0:4/5 99 % total: 99% 69.864MiB/s (avg: 14.038MiB/s)
progress: [/10.1.10.203]0:5/5 100% total: 100% 56.930MiB/s (avg: 14.141MiB/s)
progress: [/10.1.10.203]0:5/5 100% total: 100% 0.000KiB/s (avg: 13.373MiB/s)
WARN 17:23:54,923 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Stream failed
Streaming to the following hosts failed:
[/10.1.10.201, /10.1.10.202]
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
at
org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "main" org.apache.cassandra.tools.BulkLoadException:
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:114)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
Caused by: java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
... 1 more
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
at
org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
at
org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
at java.lang.Thread.run(Thread.java:745)
{noformat}
However, if I set the "internode_encryption : none", then it works fine as
expected
I do not know why it does not work. I guess somehow sstableloader does not
know how to talk with other Cassandra through encrypted channel?
Would you help?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
