[jira] [Commented] (CASSANDRA-19508) Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false
[
https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833992#comment-17833992
]
Jon Meredith commented on CASSANDRA-19508:
--
[~Aburadeh]Thanks for updating the patch, [~brandon.williams] thanks for
rerunning CI, I ran out of time to kick off a run yesterday.
+1 from me.
> Getting tons of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796" when require_client_auth is set to false
> ---
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
>Reporter: Mohammad Aburadeh
>Assignee: Mohammad Aburadeh
>Priority: Urgent
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We
> started seeing thousands of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
> causing a huge problem for us because cassandra log files are growing very
> fast as our connections are short live connections, we open more than 1K
> connections per second and they stay live for 1-2 seconds.
> {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
> ServerConnection.java:140 - Failed to get peer certificates for peer
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at
> io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at
> org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at
> org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at
> org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at
> org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: x
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or
> at least it should be logged in TRACE not DEBUG.
> I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19508) Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false
[
https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833782#comment-17833782
]
Brandon Williams commented on CASSANDRA-19508:
--
Looks good to me. We probably don't need CI for this but I already got caught
once, so:
||Branch||CI||
|[4.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19508-4.0]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1563/workflows/3200ba88-f38d-41ad-99c2-65e6240fb9ee],
[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1563/workflows/99832cc7-a831-4074-aae5-2bd70783a408]|
|[4.1|https://github.com/driftx/cassandra/tree/CASSANDRA-19508-4.1]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1562/workflows/8029ecf4-e1a6-421f-8387-6c82b2ca58e0],
[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1562/workflows/ee01ef79-8541-4600-9764-1d43d8165d91]|
|[5.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19508-5.0]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1564/workflows/0ae12b67-291f-4da8-936b-0c4bdd5d5f45],
[j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1564/workflows/89ee38ea-6b17-42fa-92e6-51f472407088]|
|[trunk|https://github.com/driftx/cassandra/tree/CASSANDRA-19508-trunk]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1565/workflows/1979282a-89d0-44a5-a37f-7d237046eea0],
[j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1565/workflows/d9f97053-1541-4505-8ab3-2a0b50070ba3]|
> Getting tons of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796" when require_client_auth is set to false
> ---
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
>Reporter: Mohammad Aburadeh
>Assignee: Mohammad Aburadeh
>Priority: Urgent
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We
> started seeing thousands of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
> causing a huge problem for us because cassandra log files are growing very
> fast as our connections are short live connections, we open more than 1K
> connections per second and they stay live for 1-2 seconds.
> {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
> ServerConnection.java:140 - Failed to get peer certificates for peer
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at
> io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at
> org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at
> org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at
> org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at
> org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: x
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or
> at least it should be logged in TRACE not DEBUG.
> I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands,
[jira] [Commented] (CASSANDRA-19508) Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false
[
https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833781#comment-17833781
]
Mohammad Aburadeh commented on CASSANDRA-19508:
---
||Branch||
|[4.0\|https://github.com/apache/cassandra/pull/3219/]|
|[4.1\|https://github.com/apache/cassandra/pull/3216/]|
|[5.0\|https://github.com/apache/cassandra/pull/3217/]|
|[trunk\|https://github.com/apache/cassandra/pull/3218/]|
[~brandon.williams] Would you please review the patch and let me know ?
> Getting tons of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796" when require_client_auth is set to false
> ---
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
>Reporter: Mohammad Aburadeh
>Assignee: Mohammad Aburadeh
>Priority: Urgent
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We
> started seeing thousands of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
> causing a huge problem for us because cassandra log files are growing very
> fast as our connections are short live connections, we open more than 1K
> connections per second and they stay live for 1-2 seconds.
> {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
> ServerConnection.java:140 - Failed to get peer certificates for peer
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at
> io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at
> org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at
> org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at
> org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at
> org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: x
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or
> at least it should be logged in TRACE not DEBUG.
> I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19508) Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false
[
https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833776#comment-17833776
]
Mohammad Aburadeh commented on CASSANDRA-19508:
---
Thanks [~jonmeredith] .
I'll disable DEBUG logging for ServerConnection in the short term. I'll update
my PR to log the msg as TRACE instead of DEBUG.
> Getting tons of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796" when require_client_auth is set to false
> ---
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
>Reporter: Mohammad Aburadeh
>Assignee: Mohammad Aburadeh
>Priority: Urgent
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We
> started seeing thousands of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
> causing a huge problem for us because cassandra log files are growing very
> fast as our connections are short live connections, we open more than 1K
> connections per second and they stay live for 1-2 seconds.
> {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
> ServerConnection.java:140 - Failed to get peer certificates for peer
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at
> io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at
> org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at
> org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at
> org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at
> org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: x
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or
> at least it should be logged in TRACE not DEBUG.
> I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19508) Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false
[
https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833771#comment-17833771
]
Jon Meredith commented on CASSANDRA-19508:
--
That's a lot of logs to deal with. Have you tried adding something like this to
your {{logback.xml}} file to improve things in the short term?
{code:xml}
{code}
I don't think we should merge the patch as it stands because it disables
retrieving the certificate if not required and it may be used by
{{IAuthenticator}} implementions. We could drop the log level to {{TRACE}} --
although logging per socket connection event at {{DEBUG}} level doesn't seem
unreasonable and it seems like other log events at that level could be added in
the future.
something like this instead? It should be a simpler patch and not involve the
config subsystem.
{code}
diff --git a/src/java/org/apache/cassandra/transport/ServerConnection.java
b/src/java/org/apache/cassandra/transport/ServerConnection.java
index 21f2e0b0e6..b47d0d9c66 100644
--- a/src/java/org/apache/cassandra/transport/ServerConnection.java
+++ b/src/java/org/apache/cassandra/transport/ServerConnection.java
@@ -137,7 +137,8 @@ public class ServerConnection extends Connection
}
catch (SSLPeerUnverifiedException e)
{
-logger.debug("Failed to get peer certificates for peer {}",
channel().remoteAddress(), e);
+if (logger.isTraceEnabled())
+logger.trace("Failed to get peer certificates for peer
{}", channel().remoteAddress(), e);
}
}
return certificates;
{code}
> Getting tons of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796" when require_client_auth is set to false
> ---
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
>Reporter: Mohammad Aburadeh
>Assignee: Mohammad Aburadeh
>Priority: Urgent
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We
> started seeing thousands of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
> causing a huge problem for us because cassandra log files are growing very
> fast as our connections are short live connections, we open more than 1K
> connections per second and they stay live for 1-2 seconds.
> {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
> ServerConnection.java:140 - Failed to get peer certificates for peer
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at
> io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at
> org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at
> org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at
> org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at
> org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: x
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or
> at least it should be logged in TRACE not DEBUG.
> I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CASSANDRA-19508) Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false
[
https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833720#comment-17833720
]
Mohammad Aburadeh commented on CASSANDRA-19508:
---
Hi [~jonmeredith] ,
We have been enabling DEBUG logging for several years, it's helpful for us to
investigate in case of any issue.
After, we upgraded to Cassandra 4., we started seeing tons of strange messages
and this is causing two problems:
1- log files are getting full very fast, we usually keep the last 10 log files
( for around 1 week) but now, the log files are rotated many times per day.
This is because our connection to Cassandra are for a short time ( less than 2
seconds).
2- Performance impact due to printing many many messages to the log file.
If you think that it might be needed for migrations, then I would suggest
printing these msgs in TRACE, not DEBUG.
Please let me know what you think.
> Getting tons of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796" when require_client_auth is set to false
> ---
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
>Reporter: Mohammad Aburadeh
>Assignee: Mohammad Aburadeh
>Priority: Urgent
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We
> started seeing thousands of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
> causing a huge problem for us because cassandra log files are growing very
> fast as our connections are short live connections, we open more than 1K
> connections per second and they stay live for 1-2 seconds.
> {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
> ServerConnection.java:140 - Failed to get peer certificates for peer
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at
> io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at
> org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at
> org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at
> org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at
> org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: x
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or
> at least it should be logged in TRACE not DEBUG.
> I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19508) Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false
[
https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833585#comment-17833585
]
Jon Meredith commented on CASSANDRA-19508:
--
[~Aburadeh] sorry the logging is causing you issues on upgrade. Are you running
DEBUG level logs on your production servers - is there some other logging you
need access to that is not available at INFO level? If not, you could adjust
the logging configuration to switch to INFO for the ServerConnection logger.
I can see the temptation to disable the check if the client certificates aren't
required, but we don't know whether {{IAuthenticator}} implementations outside
the main source tree use that information -- one example could be during
configuration migrations to see whether it is safe to require client
authentication or not without breaking existing authentication flow.
> Getting tons of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796" when require_client_auth is set to false
> ---
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
>Reporter: Mohammad Aburadeh
>Assignee: Mohammad Aburadeh
>Priority: Urgent
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We
> started seeing thousands of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
> causing a huge problem for us because cassandra log files are growing very
> fast as our connections are short live connections, we open more than 1K
> connections per second and they stay live for 1-2 seconds.
> {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
> ServerConnection.java:140 - Failed to get peer certificates for peer
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at
> io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at
> org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at
> org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at
> org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at
> org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: x
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or
> at least it should be logged in TRACE not DEBUG.
> I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19508) Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false
[
https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833322#comment-17833322
]
Brandon Williams commented on CASSANDRA-19508:
--
That looks good except for trunk, where the check is not as simple, and I [took
a
shot|https://github.com/driftx/cassandra/commit/67b420f499c58519435a9a8e5702fdc3945f1d52]
which seemed correct but broke a ton of tests. [~jmeredithco] can you assist?
> Getting tons of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796" when require_client_auth is set to false
> ---
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
>Reporter: Mohammad Aburadeh
>Assignee: Mohammad Aburadeh
>Priority: Urgent
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We
> started seeing thousands of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
> causing a huge problem for us because cassandra log files are growing very
> fast as our connections are short live connections, we open more than 1K
> connections per second and they stay live for 1-2 seconds.
> {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
> ServerConnection.java:140 - Failed to get peer certificates for peer
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at
> io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at
> org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at
> org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at
> org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at
> org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: x
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or
> at least it should be logged in TRACE not DEBUG.
> I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19508) Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false
[
https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832935#comment-17832935
]
Brandon Williams commented on CASSANDRA-19508:
--
Looks good to me, let's check CI:
||Branch||CI||
|[4.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19508-4.0]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1557/workflows/c1ae034b-e2e5-441a-9e5d-cfbfedf092fb],
[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1557/workflows/e537d55a-3cc7-482a-aaeb-51d804c7b6b5]|
|[4.1|https://github.com/driftx/cassandra/tree/CASSANDRA-19508-4.1]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/1554/workflows/5a09fafb-756a-4c7f-af8b-8e8fc7707721],
[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1554/workflows/e6874734-01db-4eaa-867b-d38d9fdd6eeb]|
|[5.0|https://github.com/driftx/cassandra/tree/CASSANDRA-19508-5.0]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1555/workflows/d0c49666-4804-4e96-bce8-ff25945697ee],
[j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1555/workflows/1a3573c1-2cc7-4276-bf45-ce6a39237f6f]|
|[trunk|https://github.com/driftx/cassandra/tree/CASSANDRA-19508-trunk]|[j11|https://app.circleci.com/pipelines/github/driftx/cassandra/1556/workflows/ab0e8423-e1d0-480f-91e7-cad66509453c],
[j17|https://app.circleci.com/pipelines/github/driftx/cassandra/1556/workflows/380ed9fb-8c0f-4c24-9b66-d66442d84ad5]|
> Getting tons of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796" when require_client_auth is set to false
> ---
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
>Reporter: Mohammad Aburadeh
>Assignee: Mohammad Aburadeh
>Priority: Urgent
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We
> started seeing thousands of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
> causing a huge problem for us because cassandra log files are growing very
> fast as our connections are short live connections, we open more than 1K
> connections per second and they stay live for 1-2 seconds.
> {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
> ServerConnection.java:140 - Failed to get peer certificates for peer
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at
> io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at
> org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at
> org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at
> org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at
> org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: x
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or
> at least it should be logged in TRACE not DEBUG.
> I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19508) Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false
[
https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832929#comment-17832929
]
Mohammad Aburadeh commented on CASSANDRA-19508:
---
I submitted PR for this issue. I tested it on our clusters, it worked well.
My fix is simple, it just disabled checking peer certificates if
require_client_auth is disabled. The other option is logging the msg in TRACE
not DEBUG.
||Branch||
|[4.1\|[https://github.com/apache/cassandra/pull/3216]|
|[5.0\|https://github.com/apache/cassandra/pull/3217]|
|[Trunk\|https://github.com/apache/cassandra/pull/3218]|
Please review and let me know.
> Getting tons of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796" when require_client_auth is set to false
> ---
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
>Reporter: Mohammad Aburadeh
>Priority: Urgent
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We
> started seeing thousands of msgs "Failed to get peer certificates for peer
> /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled. This is
> causing a huge problem for us because cassandra log files are growing very
> fast as our connections are short live connections, we open more than 1K
> connections per second and they stay live for 1-2 seconds.
> {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026
> ServerConnection.java:140 - Failed to get peer certificates for peer
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at
> io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at
> org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at
> org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at
> org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at
> org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at
> org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at
> org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at
> org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: x
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or
> at least it should be logged in TRACE not DEBUG.
> I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
