[jira] [Updated] (CASSANDRA-12886) Streaming failed due to SSL Socket connection reset
[
https://issues.apache.org/jira/browse/CASSANDRA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paulo Motta updated CASSANDRA-12886:
Resolution: Fixed
Fix Version/s: 3.0.12
2.2.10
Status: Resolved (was: Ready to Commit)
Committed to 2.2 as 06feaefba50301734c490521d720c8a482f638e4 and merged to 3.0.
Thanks!
> Streaming failed due to SSL Socket connection reset
> ---
>
> Key: CASSANDRA-12886
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12886
> Project: Cassandra
> Issue Type: Bug
>Reporter: Bing Wu
>Assignee: Paulo Motta
> Fix For: 2.2.10, 3.0.12
>
> Attachments: debug.log.2016-11-10_2319.gz
>
>
> While running "nodetool repair", I see many instances of
> "javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in
> system.logs on some nodes in the cluster. Timestamps correspond to streaming
> source/initiator's error messages of "sync failed between ..."
> Setup:
> - Cassandra 3.7.01
> - CentOS 6.7 in AWS (multi-region)
> - JDK version: {noformat}
> java version "1.8.0_102"
> Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
> {noformat}
> - cassandra.yaml:
> {noformat}
> server_encryption_options:
> internode_encryption: all
> keystore: [path]
> keystore_password: [password]
> truststore: [path]
> truststore_password: [password]
> # More advanced defaults below:
> # protocol: TLS
> # algorithm: SunX509
> # store_type: JKS
> # cipher_suites:
> [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
> require_client_auth: false
> {noformat}
> Error messages in system.log on the target host:
> {noformat}
> ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475
> StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e]
> Streaming error occurred on session with peer 54.247.111.232
> javax.net.ssl.SSLException: Connection has been shutdown:
> javax.net.ssl.SSLException: java.net.SocketException: Connection reset
> at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541)
> ~[na:1.8.0_102]
> at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553)
> ~[na:1.8.0_102]
> at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
> ~[na:1.8.0_102]
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[na:1.8.0_102]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[na:1.8.0_102]
> at
> org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection
> reset
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-12886) Streaming failed due to SSL Socket connection reset
[
https://issues.apache.org/jira/browse/CASSANDRA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuki Morishita updated CASSANDRA-12886:
---
Status: Ready to Commit (was: Patch Available)
> Streaming failed due to SSL Socket connection reset
> ---
>
> Key: CASSANDRA-12886
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12886
> Project: Cassandra
> Issue Type: Bug
>Reporter: Bing Wu
>Assignee: Paulo Motta
> Attachments: debug.log.2016-11-10_2319.gz
>
>
> While running "nodetool repair", I see many instances of
> "javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in
> system.logs on some nodes in the cluster. Timestamps correspond to streaming
> source/initiator's error messages of "sync failed between ..."
> Setup:
> - Cassandra 3.7.01
> - CentOS 6.7 in AWS (multi-region)
> - JDK version: {noformat}
> java version "1.8.0_102"
> Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
> {noformat}
> - cassandra.yaml:
> {noformat}
> server_encryption_options:
> internode_encryption: all
> keystore: [path]
> keystore_password: [password]
> truststore: [path]
> truststore_password: [password]
> # More advanced defaults below:
> # protocol: TLS
> # algorithm: SunX509
> # store_type: JKS
> # cipher_suites:
> [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
> require_client_auth: false
> {noformat}
> Error messages in system.log on the target host:
> {noformat}
> ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475
> StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e]
> Streaming error occurred on session with peer 54.247.111.232
> javax.net.ssl.SSLException: Connection has been shutdown:
> javax.net.ssl.SSLException: java.net.SocketException: Connection reset
> at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541)
> ~[na:1.8.0_102]
> at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553)
> ~[na:1.8.0_102]
> at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
> ~[na:1.8.0_102]
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[na:1.8.0_102]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[na:1.8.0_102]
> at
> org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection
> reset
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-12886) Streaming failed due to SSL Socket connection reset
[
https://issues.apache.org/jira/browse/CASSANDRA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joshua McKenzie updated CASSANDRA-12886:
Reviewer: Yuki Morishita
> Streaming failed due to SSL Socket connection reset
> ---
>
> Key: CASSANDRA-12886
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12886
> Project: Cassandra
> Issue Type: Bug
>Reporter: Bing Wu
>Assignee: Paulo Motta
> Attachments: debug.log.2016-11-10_2319.gz
>
>
> While running "nodetool repair", I see many instances of
> "javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in
> system.logs on some nodes in the cluster. Timestamps correspond to streaming
> source/initiator's error messages of "sync failed between ..."
> Setup:
> - Cassandra 3.7.01
> - CentOS 6.7 in AWS (multi-region)
> - JDK version: {noformat}
> java version "1.8.0_102"
> Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
> {noformat}
> - cassandra.yaml:
> {noformat}
> server_encryption_options:
> internode_encryption: all
> keystore: [path]
> keystore_password: [password]
> truststore: [path]
> truststore_password: [password]
> # More advanced defaults below:
> # protocol: TLS
> # algorithm: SunX509
> # store_type: JKS
> # cipher_suites:
> [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
> require_client_auth: false
> {noformat}
> Error messages in system.log on the target host:
> {noformat}
> ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475
> StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e]
> Streaming error occurred on session with peer 54.247.111.232
> javax.net.ssl.SSLException: Connection has been shutdown:
> javax.net.ssl.SSLException: java.net.SocketException: Connection reset
> at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541)
> ~[na:1.8.0_102]
> at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553)
> ~[na:1.8.0_102]
> at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
> ~[na:1.8.0_102]
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[na:1.8.0_102]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[na:1.8.0_102]
> at
> org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection
> reset
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-12886) Streaming failed due to SSL Socket connection reset
[
https://issues.apache.org/jira/browse/CASSANDRA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paulo Motta updated CASSANDRA-12886:
Status: Patch Available (was: Open)
This is hard to reproduce but I've got a similar issue when working on
CASSANDRA-11841, and what is probably happening is a race where the handler
sender thread is started and pushes a message to the socket before the init
message is sent, what breaks the connection in the receiver side. In order to
avoid this, we must first send the init message before starting the handler
thread. This should already be fixed on 3.10 by CASSANDRA-11841, but this fix
is for 2.2+.
||2.2||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...pauloricardomg:2.2-12886]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-12886-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-12886-dtest/lastCompletedBuild/testReport/]|
> Streaming failed due to SSL Socket connection reset
> ---
>
> Key: CASSANDRA-12886
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12886
> Project: Cassandra
> Issue Type: Bug
>Reporter: Bing Wu
>Assignee: Paulo Motta
> Attachments: debug.log.2016-11-10_2319.gz
>
>
> While running "nodetool repair", I see many instances of
> "javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in
> system.logs on some nodes in the cluster. Timestamps correspond to streaming
> source/initiator's error messages of "sync failed between ..."
> Setup:
> - Cassandra 3.7.01
> - CentOS 6.7 in AWS (multi-region)
> - JDK version: {noformat}
> java version "1.8.0_102"
> Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
> {noformat}
> - cassandra.yaml:
> {noformat}
> server_encryption_options:
> internode_encryption: all
> keystore: [path]
> keystore_password: [password]
> truststore: [path]
> truststore_password: [password]
> # More advanced defaults below:
> # protocol: TLS
> # algorithm: SunX509
> # store_type: JKS
> # cipher_suites:
> [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
> require_client_auth: false
> {noformat}
> Error messages in system.log on the target host:
> {noformat}
> ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475
> StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e]
> Streaming error occurred on session with peer 54.247.111.232
> javax.net.ssl.SSLException: Connection has been shutdown:
> javax.net.ssl.SSLException: java.net.SocketException: Connection reset
> at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541)
> ~[na:1.8.0_102]
> at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553)
> ~[na:1.8.0_102]
> at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
> ~[na:1.8.0_102]
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[na:1.8.0_102]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[na:1.8.0_102]
> at
> org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection
> reset
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-12886) Streaming failed due to SSL Socket connection reset
[
https://issues.apache.org/jira/browse/CASSANDRA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bing Wu updated CASSANDRA-12886:
Attachment: debug.log.2016-11-10_2319.gz
> Streaming failed due to SSL Socket connection reset
> ---
>
> Key: CASSANDRA-12886
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12886
> Project: Cassandra
> Issue Type: Bug
>Reporter: Bing Wu
> Attachments: debug.log.2016-11-10_2319.gz
>
>
> While running "nodetool repair", I see many instances of
> "javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in
> system.logs on some nodes in the cluster. Timestamps correspond to streaming
> source/initiator's error messages of "sync failed between ..."
> Setup:
> - Cassandra 3.7.01
> - CentOS 6.7 in AWS (multi-region)
> - JDK version: {noformat}
> java version "1.8.0_102"
> Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
> {noformat}
> - cassandra.yaml:
> {noformat}
> server_encryption_options:
> internode_encryption: all
> keystore: [path]
> keystore_password: [password]
> truststore: [path]
> truststore_password: [password]
> # More advanced defaults below:
> # protocol: TLS
> # algorithm: SunX509
> # store_type: JKS
> # cipher_suites:
> [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
> require_client_auth: false
> {noformat}
> Error messages in system.log on the target host:
> {noformat}
> ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475
> StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e]
> Streaming error occurred on session with peer 54.247.111.232
> javax.net.ssl.SSLException: Connection has been shutdown:
> javax.net.ssl.SSLException: java.net.SocketException: Connection reset
> at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541)
> ~[na:1.8.0_102]
> at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553)
> ~[na:1.8.0_102]
> at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
> ~[na:1.8.0_102]
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[na:1.8.0_102]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[na:1.8.0_102]
> at
> org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection
> reset
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12886) Streaming failed due to SSL Socket connection reset
[
https://issues.apache.org/jira/browse/CASSANDRA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bing Wu updated CASSANDRA-12886:
Description:
While running "nodetool repair", I see many instances of
"javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in
system.logs on some nodes in the cluster. Timestamps correspond to streaming
source/initiator's error messages of "sync failed between ..."
Setup:
- Cassandra 3.7.01
- CentOS 6.7 in AWS (multi-region)
- JDK version: {noformat}
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
{noformat}
- cassandra.yaml:
{noformat}
server_encryption_options:
internode_encryption: all
keystore: [path]
keystore_password: [password]
truststore: [path]
truststore_password: [password]
# More advanced defaults below:
# protocol: TLS
# algorithm: SunX509
# store_type: JKS
# cipher_suites:
[TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
require_client_auth: false
{noformat}
Error messages in system.log on the target host:
{noformat}
ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475
StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e]
Streaming error occurred on session with peer 54.247.111.232
javax.net.ssl.SSLException: Connection has been shutdown:
javax.net.ssl.SSLException: java.net.SocketException: Connection reset
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541)
~[na:1.8.0_102]
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553)
~[na:1.8.0_102]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
~[na:1.8.0_102]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[na:1.8.0_102]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[na:1.8.0_102]
at
org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
~[apache-cassandra-3.7.0.jar:3.7.0]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371)
[apache-cassandra-3.7.0.jar:3.7.0]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342)
[apache-cassandra-3.7.0.jar:3.7.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection
reset
{noformat}
was:
While running "nodetool repair", I see many instances of
"javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in
system.logs on some nodes in the cluster. Timestamp corresponds to streaming
source/initiator's error messages of "sync failed between ..."
Setup:
- Cassandra 3.7.01
- CentOS 6.7 in AWS (multi-region)
- JDK version: {noformat}
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
{noformat}
- cassandra.yaml:
{noformat}
server_encryption_options:
internode_encryption: all
keystore: [path]
keystore_password: [password]
truststore: [path]
truststore_password: [password]
# More advanced defaults below:
# protocol: TLS
# algorithm: SunX509
# store_type: JKS
# cipher_suites:
[TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
require_client_auth: false
{noformat}
Error messages in system.log on the target host:
{noformat}
ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475
StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e]
Streaming error occurred on session with peer 54.247.111.232
javax.net.ssl.SSLException: Connection has been shutdown:
javax.net.ssl.SSLException: java.net.SocketException: Connection reset
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541)
~[na:1.8.0_102]
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553)
~[na:1.8.0_102]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
~[na:1.8.0_102]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[na:1.8.0_102]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[na:1.8.0_102]
at
org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
~[apache-cassandra-3.7.0.jar:3.7.0]
at
[jira] [Updated] (CASSANDRA-12886) Streaming failed due to SSL Socket connection reset
[
https://issues.apache.org/jira/browse/CASSANDRA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bing Wu updated CASSANDRA-12886:
Description:
While running "nodetool repair", I see many instances of
"javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in
system.logs on some nodes in the cluster. Timestamp corresponds to streaming
source/initiator's error messages of "sync failed between ..."
Setup:
- Cassandra 3.7.01
- CentOS 6.7 in AWS (multi-region)
- JDK version: {noformat}
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
{noformat}
cassandra.yaml:
{noformat}
server_encryption_options:
internode_encryption: all
keystore: [path]
keystore_password: [password]
truststore: [path]
truststore_password: [password]
# More advanced defaults below:
# protocol: TLS
# algorithm: SunX509
# store_type: JKS
# cipher_suites:
[TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
require_client_auth: false
{noformat}
Error messages in system.log on the target host:
{noformat}
ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475
StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e]
Streaming error occurred on session with peer 54.247.111.232
javax.net.ssl.SSLException: Connection has been shutdown:
javax.net.ssl.SSLException: java.net.SocketException: Connection reset
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541)
~[na:1.8.0_102]
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553)
~[na:1.8.0_102]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
~[na:1.8.0_102]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[na:1.8.0_102]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[na:1.8.0_102]
at
org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
~[apache-cassandra-3.7.0.jar:3.7.0]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371)
[apache-cassandra-3.7.0.jar:3.7.0]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342)
[apache-cassandra-3.7.0.jar:3.7.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection
reset
{noformat}
was:
While running "nodetool repair", I see many instances of
"javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in
system.logs on some nodes in the cluster. Timestamp corresponds to streaming
source/initiator's error messages of "sync failed between ..."
Setup: we are running 3.7.01 on CentOS 6.7 in AWS (multi-region)
jdk version: {noformat}
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
{noformat}
cassandra.yaml:
{noformat}
server_encryption_options:
internode_encryption: all
keystore: [path]
keystore_password: [password]
truststore: [path]
truststore_password: [password]
# More advanced defaults below:
# protocol: TLS
# algorithm: SunX509
# store_type: JKS
# cipher_suites:
[TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
require_client_auth: false
{noformat}
Error messages in system.log on the target host:
{noformat}
ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475
StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e]
Streaming error occurred on session with peer 54.247.111.232
javax.net.ssl.SSLException: Connection has been shutdown:
javax.net.ssl.SSLException: java.net.SocketException: Connection reset
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541)
~[na:1.8.0_102]
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553)
~[na:1.8.0_102]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
~[na:1.8.0_102]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[na:1.8.0_102]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[na:1.8.0_102]
at
org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
~[apache-cassandra-3.7.0.jar:3.7.0]
at
[jira] [Updated] (CASSANDRA-12886) Streaming failed due to SSL Socket connection reset
[
https://issues.apache.org/jira/browse/CASSANDRA-12886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bing Wu updated CASSANDRA-12886:
Description:
While running "nodetool repair", I see many instances of
"javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in
system.logs on some nodes in the cluster. Timestamp corresponds to streaming
source/initiator's error messages of "sync failed between ..."
Setup:
- Cassandra 3.7.01
- CentOS 6.7 in AWS (multi-region)
- JDK version: {noformat}
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
{noformat}
- cassandra.yaml:
{noformat}
server_encryption_options:
internode_encryption: all
keystore: [path]
keystore_password: [password]
truststore: [path]
truststore_password: [password]
# More advanced defaults below:
# protocol: TLS
# algorithm: SunX509
# store_type: JKS
# cipher_suites:
[TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
require_client_auth: false
{noformat}
Error messages in system.log on the target host:
{noformat}
ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475
StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e]
Streaming error occurred on session with peer 54.247.111.232
javax.net.ssl.SSLException: Connection has been shutdown:
javax.net.ssl.SSLException: java.net.SocketException: Connection reset
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541)
~[na:1.8.0_102]
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553)
~[na:1.8.0_102]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
~[na:1.8.0_102]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[na:1.8.0_102]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[na:1.8.0_102]
at
org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
~[apache-cassandra-3.7.0.jar:3.7.0]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:371)
[apache-cassandra-3.7.0.jar:3.7.0]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:342)
[apache-cassandra-3.7.0.jar:3.7.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection
reset
{noformat}
was:
While running "nodetool repair", I see many instances of
"javax.net.ssl.SSLException: java.net.SocketException: Connection reset" in
system.logs on some nodes in the cluster. Timestamp corresponds to streaming
source/initiator's error messages of "sync failed between ..."
Setup:
- Cassandra 3.7.01
- CentOS 6.7 in AWS (multi-region)
- JDK version: {noformat}
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
{noformat}
cassandra.yaml:
{noformat}
server_encryption_options:
internode_encryption: all
keystore: [path]
keystore_password: [password]
truststore: [path]
truststore_password: [password]
# More advanced defaults below:
# protocol: TLS
# algorithm: SunX509
# store_type: JKS
# cipher_suites:
[TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
require_client_auth: false
{noformat}
Error messages in system.log on the target host:
{noformat}
ERROR [STREAM-OUT-/54.247.111.232:7001] 2016-11-07 07:30:56,475
StreamSession.java:529 - [Stream #e14abcb0-a4bb-11e6-9758-55b9ac38b78e]
Streaming error occurred on session with peer 54.247.111.232
javax.net.ssl.SSLException: Connection has been shutdown:
javax.net.ssl.SSLException: java.net.SocketException: Connection reset
at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1541)
~[na:1.8.0_102]
at sun.security.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1553)
~[na:1.8.0_102]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:71)
~[na:1.8.0_102]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[na:1.8.0_102]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[na:1.8.0_102]
at
org.apache.cassandra.io.util.WrappedDataOutputStreamPlus.flush(WrappedDataOutputStreamPlus.java:66)
~[apache-cassandra-3.7.0.jar:3.7.0]
at
