[
https://issues.apache.org/jira/browse/CASSANDRA-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16476156#comment-16476156
]
Paul Cheon edited comment on CASSANDRA-14437 at 5/15/18 4:52 PM:
-----------------------------------------------------------------
The content of yaml file is exact same except the node keystore file
{code}
keystore: "/etc/ssl/visier/10.1.119.203.jks"
{code}
The keystore password is same though. Just the certificate inside of the
keystore is generated with the IP address of each node
Trust keystore is same file in every node
was (Author: paul.ch...@visiercorp.com):
The content of yaml file is exact same except the node keystore file
{code}
keystore: "/etc/ssl/visier/10.1.119.203.jks"
{code}
The keystore password is same though. Just the certificate inside of the
keystore is generated with the IP address of each node
> SSTableLoader does not work when "internode_encryption : all" is set
> --------------------------------------------------------------------
>
> Key: CASSANDRA-14437
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14437
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Paul Cheon
> Priority: Major
> Fix For: 3.11.2
>
>
> I am trying to use sstableloader to restore snapshot.
> If "internode_encryption : all" is set, then it does not work and complain
> with below error messages. I initiated sstableloader from 10.1.10.203
> (yvr-paul-cas003), so 10.1.10.203 worked fine, but the the other two nodes
> (10.1.10.201 & 10.1.10.202 failed)
> {noformat}
> pcheon@yvr-paul-cas003:~/t$ sstableloader -v -d 10.1.10.203 office_audit/log/
> -f /etc/cassandra/cassandra.yaml -u pcheon -pw `cat .secret`
> WARN 17:23:45,166 Small commitlog volume detected at
> /var/lib/cassandra/commitlog; setting commitlog_total_space_in_mb to 2316.
> You can override this in cassandra.yaml
> WARN 17:23:45,170 Small cdc volume detected at /var/lib/cassandra/cdc_raw;
> setting cdc_total_space_in_mb to 1158. You can override this in
> cassandra.yaml
> WARN 17:23:45,285 Only 5.318GiB free across all data volumes. Consider
> adding more capacity to your cluster or removing obsolete snapshots
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of
> /home/pcheon/t/office_audit/log/mc-1083-big-Data.db
> /home/pcheon/t/office_audit/log/mc-1100-big-Data.db
> /home/pcheon/t/office_audit/log/mc-1101-big-Data.db
> /home/pcheon/t/office_audit/log/mc-257-big-Data.db
> /home/pcheon/t/office_audit/log/mc-984-big-Data.db to [/10.1.10.201,
> /10.1.10.203, /10.1.10.202]
> ERROR 17:23:49,460 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming
> error occurred on session with peer 10.1.10.201
> java.net.SocketException: Invalid argument or cannot assign requested address
> at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> ~[na:1.8.0_112]
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> ~[na:1.8.0_112]
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> ~[na:1.8.0_112]
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> ~[na:1.8.0_112]
> at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
> ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495)
> ~[na:1.8.0_112]
> at
> sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169)
> ~[na:1.8.0_112]
> at
> org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_112]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
> ERROR 17:23:49,458 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming
> error occurred on session with peer 10.1.10.202
> java.net.SocketException: Invalid argument or cannot assign requested address
> at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> ~[na:1.8.0_112]
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> ~[na:1.8.0_112]
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> ~[na:1.8.0_112]
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> ~[na:1.8.0_112]
> at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
> ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495)
> ~[na:1.8.0_112]
> at
> sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169)
> ~[na:1.8.0_112]
> at
> org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269)
> ~[apache-cassandra-3.11.2.jar:3.11.2]
> at
> org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_112]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.2.jar:3.11.2]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
> progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
> progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
> progress: [/10.1.10.203]0:0/5 7 % total: 7% 13.858MiB/s (avg: 2.147MiB/s)
> progress: [/10.1.10.203]0:0/5 15 % total: 15% 41.587MiB/s (avg: 4.083MiB/s)
> progress: [/10.1.10.203]0:0/5 23 % total: 23% 75.631MiB/s (avg: 5.964MiB/s)
> progress: [/10.1.10.203]0:0/5 31 % total: 31% 34.837MiB/s (avg: 7.523MiB/s)
> progress: [/10.1.10.203]0:0/5 39 % total: 39% 21.321MiB/s (avg: 8.641MiB/s)
> progress: [/10.1.10.203]0:1/5 41 % total: 41% 17.665MiB/s (avg: 8.766MiB/s)
> progress: [/10.1.10.203]0:1/5 49 % total: 49% 17.841MiB/s (avg: 9.557MiB/s)
> progress: [/10.1.10.203]0:1/5 56 % total: 56% 75.654MiB/s (avg: 10.889MiB/s)
> progress: [/10.1.10.203]0:1/5 64 % total: 64% 21.252MiB/s (avg: 11.583MiB/s)
> progress: [/10.1.10.203]0:1/5 72 % total: 72% 33.267MiB/s (avg: 12.472MiB/s)
> progress: [/10.1.10.203]0:1/5 80 % total: 80% 22.969MiB/s (avg: 13.060MiB/s)
> progress: [/10.1.10.203]0:2/5 81 % total: 81% 6.191MiB/s (avg: 12.951MiB/s)
> progress: [/10.1.10.203]0:3/5 81 % total: 81% 100.225MiB/s (avg: 12.994MiB/s)
> progress: [/10.1.10.203]0:3/5 89 % total: 89% 12.939MiB/s (avg: 12.989MiB/s)
> progress: [/10.1.10.203]0:3/5 97 % total: 97% 65.803MiB/s (avg: 13.899MiB/s)
> progress: [/10.1.10.203]0:4/5 99 % total: 99% 69.864MiB/s (avg: 14.038MiB/s)
> progress: [/10.1.10.203]0:5/5 100% total: 100% 56.930MiB/s (avg: 14.141MiB/s)
> progress: [/10.1.10.203]0:5/5 100% total: 100% 0.000KiB/s (avg: 13.373MiB/s)
> WARN 17:23:54,923 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Stream
> failed
> Streaming to the following hosts failed:
> [/10.1.10.201, /10.1.10.202]
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
> at java.lang.Thread.run(Thread.java:745)
> Exception in thread "main" org.apache.cassandra.tools.BulkLoadException:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:114)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
> ... 1 more
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> However, if I set the "internode_encryption : none", then it works fine as
> expected
> I do not know why it does not work. I guess somehow sstableloader does not
> know how to talk with other Cassandra through encrypted channel?
> Would you help?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
