[ https://issues.apache.org/jira/browse/CASSANDRA-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16470220#comment-16470220 ]
Kurt Greaves commented on CASSANDRA-14437: ------------------------------------------ You specify it in the yaml, and specify the yaml to use by using the {{-f,--conf-path <path to config file> }} option. This should more or less be the same yaml you have on the nodes, with similar encryption settings. > SSTableLoader does not work when "internode_encryption : all" is set > -------------------------------------------------------------------- > > Key: CASSANDRA-14437 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14437 > Project: Cassandra > Issue Type: Bug > Components: Tools > Reporter: Paul Cheon > Priority: Major > Fix For: 3.11.2 > > > I am trying to use sstableloader to restore snapshot. > If "internode_encryption : all" is set, then it does not work and complain > with below error messages. I initiated sstableloader from 10.1.10.203 > (yvr-paul-cas003), so 10.1.10.203 worked fine, but the the other two nodes > (10.1.10.201 & 10.1.10.202 failed) > {noformat} > pcheon@yvr-paul-cas003:~/t$ sstableloader -v -d 10.1.10.203 office_audit/log/ > -f /etc/cassandra/cassandra.yaml -u pcheon -pw `cat .secret` > WARN 17:23:45,166 Small commitlog volume detected at > /var/lib/cassandra/commitlog; setting commitlog_total_space_in_mb to 2316. > You can override this in cassandra.yaml > WARN 17:23:45,170 Small cdc volume detected at /var/lib/cassandra/cdc_raw; > setting cdc_total_space_in_mb to 1158. You can override this in > cassandra.yaml > WARN 17:23:45,285 Only 5.318GiB free across all data volumes. Consider > adding more capacity to your cluster or removing obsolete snapshots > Established connection to initial hosts > Opening sstables and calculating sections to stream > Streaming relevant part of > /home/pcheon/t/office_audit/log/mc-1083-big-Data.db > /home/pcheon/t/office_audit/log/mc-1100-big-Data.db > /home/pcheon/t/office_audit/log/mc-1101-big-Data.db > /home/pcheon/t/office_audit/log/mc-257-big-Data.db > /home/pcheon/t/office_audit/log/mc-984-big-Data.db to [/10.1.10.201, > /10.1.10.203, /10.1.10.202] > ERROR 17:23:49,460 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming > error occurred on session with peer 10.1.10.201 > java.net.SocketException: Invalid argument or cannot assign requested address > at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112] > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) > ~[na:1.8.0_112] > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) > ~[na:1.8.0_112] > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) > ~[na:1.8.0_112] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > ~[na:1.8.0_112] > at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) > ~[na:1.8.0_112] > at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495) > ~[na:1.8.0_112] > at > sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169) > ~[na:1.8.0_112] > at > org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263) > [apache-cassandra-3.11.2.jar:3.11.2] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_112] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_112] > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) > [apache-cassandra-3.11.2.jar:3.11.2] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112] > ERROR 17:23:49,458 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming > error occurred on session with peer 10.1.10.202 > java.net.SocketException: Invalid argument or cannot assign requested address > at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112] > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) > ~[na:1.8.0_112] > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) > ~[na:1.8.0_112] > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) > ~[na:1.8.0_112] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > ~[na:1.8.0_112] > at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112] > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) > ~[na:1.8.0_112] > at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495) > ~[na:1.8.0_112] > at > sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169) > ~[na:1.8.0_112] > at > org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263) > [apache-cassandra-3.11.2.jar:3.11.2] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_112] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_112] > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) > [apache-cassandra-3.11.2.jar:3.11.2] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112] > progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s) > progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s) > progress: [/10.1.10.203]0:0/5 7 % total: 7% 13.858MiB/s (avg: 2.147MiB/s) > progress: [/10.1.10.203]0:0/5 15 % total: 15% 41.587MiB/s (avg: 4.083MiB/s) > progress: [/10.1.10.203]0:0/5 23 % total: 23% 75.631MiB/s (avg: 5.964MiB/s) > progress: [/10.1.10.203]0:0/5 31 % total: 31% 34.837MiB/s (avg: 7.523MiB/s) > progress: [/10.1.10.203]0:0/5 39 % total: 39% 21.321MiB/s (avg: 8.641MiB/s) > progress: [/10.1.10.203]0:1/5 41 % total: 41% 17.665MiB/s (avg: 8.766MiB/s) > progress: [/10.1.10.203]0:1/5 49 % total: 49% 17.841MiB/s (avg: 9.557MiB/s) > progress: [/10.1.10.203]0:1/5 56 % total: 56% 75.654MiB/s (avg: 10.889MiB/s) > progress: [/10.1.10.203]0:1/5 64 % total: 64% 21.252MiB/s (avg: 11.583MiB/s) > progress: [/10.1.10.203]0:1/5 72 % total: 72% 33.267MiB/s (avg: 12.472MiB/s) > progress: [/10.1.10.203]0:1/5 80 % total: 80% 22.969MiB/s (avg: 13.060MiB/s) > progress: [/10.1.10.203]0:2/5 81 % total: 81% 6.191MiB/s (avg: 12.951MiB/s) > progress: [/10.1.10.203]0:3/5 81 % total: 81% 100.225MiB/s (avg: 12.994MiB/s) > progress: [/10.1.10.203]0:3/5 89 % total: 89% 12.939MiB/s (avg: 12.989MiB/s) > progress: [/10.1.10.203]0:3/5 97 % total: 97% 65.803MiB/s (avg: 13.899MiB/s) > progress: [/10.1.10.203]0:4/5 99 % total: 99% 69.864MiB/s (avg: 14.038MiB/s) > progress: [/10.1.10.203]0:5/5 100% total: 100% 56.930MiB/s (avg: 14.141MiB/s) > progress: [/10.1.10.203]0:5/5 100% total: 100% 0.000KiB/s (avg: 13.373MiB/s) > WARN 17:23:54,923 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Stream > failed > Streaming to the following hosts failed: > [/10.1.10.201, /10.1.10.202] > java.util.concurrent.ExecutionException: > org.apache.cassandra.streaming.StreamException: Stream failed > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98) > at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48) > Caused by: org.apache.cassandra.streaming.StreamException: Stream failed > at > org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88) > at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) > at > com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) > at > com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) > at > com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) > at > com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) > at > org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215) > at > org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191) > at > org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481) > at > org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682) > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532) > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317) > at java.lang.Thread.run(Thread.java:745) > Exception in thread "main" org.apache.cassandra.tools.BulkLoadException: > java.util.concurrent.ExecutionException: > org.apache.cassandra.streaming.StreamException: Stream failed > at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:114) > at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48) > Caused by: java.util.concurrent.ExecutionException: > org.apache.cassandra.streaming.StreamException: Stream failed > at > com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) > at > com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98) > ... 1 more > Caused by: org.apache.cassandra.streaming.StreamException: Stream failed > at > org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88) > at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) > at > com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) > at > com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) > at > com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) > at > com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) > at > org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215) > at > org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191) > at > org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481) > at > org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682) > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532) > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317) > at java.lang.Thread.run(Thread.java:745) > {noformat} > However, if I set the "internode_encryption : none", then it works fine as > expected > I do not know why it does not work. I guess somehow sstableloader does not > know how to talk with other Cassandra through encrypted channel? > Would you help? -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org