[
https://issues.apache.org/jira/browse/SOLR-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-12988:
----------------------------
Labels: Java11 Java12 Java13 (was: Java11 Java12)
Description:
There are several known OpenJDK JVM bugs (begining with Java11, when TLS v1.3
support was first added) that are known to affect Solr's SSL support, and have
caused numerous test failures -- notably early "testing" builds of OpenJDK 11,
12, & 13, as well as the officially released OpenJDK 11, 11.0.1, and 11.0.2.
>From the standpoint of the Solr project, there is very little we can do to
>mitigate these bugs, and have taken steps to ensure any code using our
>{{SSLTestConfig}} / {{RandomizeSSL}} test-framework classes will be "SKIPed"
>with an {{AssumptionViolatedException}} when used on JVMs that are known to be
>problematic.
Users who encounter any of the types of failures described below, or developers
who encounter test runs that "SKIP" with a message refering to this issue ID,
are encouraged to Upgrade their JVM. (or as a last resort: try disabling
"TLSv1.3" in your JVM security properties)
----
Examples of known bugs as they have manifested in Solr tests...
* https://bugs.openjdk.java.net/browse/JDK-8212885
** "TLS 1.3 resumed session does not retain peer certificate chain"
** affects users with {{checkPeerNames=true}} in your SSL configuration
** causes 100% failure rate in Solr's
{{TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName}}
** can result in exceptions for SolrJ users, or in solr cloud server logs when
making intra-node requests, with a root cause of
"javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated"
** {noformat}
[junit4] 2> Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not
authenticated
[junit4] 2> at
java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:526)
[junit4] 2> at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:464)
[junit4] 2> at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:397)
[junit4] 2> at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
[junit4] 2> at
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
[junit4] 2> at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
[junit4] 2> at
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
[junit4] 2> at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
[junit4] 2> at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
[junit4] 2> at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
[junit4] 2> at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
[junit4] 2> at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
[junit4] 2> at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
[junit4] 2> at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
[junit4] 2> at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
{noformat}
* https://bugs.openjdk.java.net/browse/JDK-8213202
** "Possible race condition in TLS 1.3 session resumption"
** May affect any and all Solr SSL users, although noted only in tests when
"clientAuth" was configured to be false
** Causes non-reproducing test failures, and sporadic end user exceptions with
a root cause of "javax.net.ssl.SSLException: Received fatal alert:
internal_error "
** SSL Debugging may indicate "Fatal (INTERNAL_ERROR): Session has no PSK"
** {noformat}
[junit4] 2> Caused by: javax.net.ssl.SSLException: Received fatal alert:
internal_error
[junit4] 2> at
sun.security.ssl.Alert.createSSLException(Alert.java:129) ~[?:?]
[junit4] 2> at
sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
[junit4] 2> at
sun.security.ssl.TransportContext.fatal(TransportContext.java:308) ~[?:?]
[junit4] 2> at
sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:279) ~[?:?]
[junit4] 2> at
sun.security.ssl.TransportContext.dispatch(TransportContext.java:181) ~[?:?]
[junit4] 2> at
sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?]
[junit4] 2> at
sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) ~[?:?]
[junit4] 2> at
sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
~[?:?]
[junit4] 2> at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) ~[?:?]
[junit4] 2> at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
~[httpclient-4.5.6.jar:4.5.6]
[junit4] 2> at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:555)
~[java/:?]
[junit4] 2> ... 13 more
{noformat}
* https://bugs.openjdk.java.net/browse/JDK-8224829
** "AsyncSSLSocketClose.java has timing issue"
** May affect any and all Solr SSL users running early testing versions of java
13 or 14.
** Causes non-reproducing test failures, and sporadic end user exceptions with
a root cause of "javax.net.ssl.SSLException: Software caused connection abort:
recv failed"
** {noformat}
javax.net.ssl.SSLException: Software caused connection abort: recv failed
at
__randomizedtesting.SeedInfo.seed([AA73C7E858ABD2EE:88D2A395FDC7B4AB]:0)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
at
java.base/sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1501)
at
java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:935)
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
{noformat}
was:
HTTPCLIENT-1967 indicates that HttpClient can't be used properly with TLSv1.3.
It caused some test failures below, therefore we should skip running tests with
SSL on Java 11 to 11.0.2.
TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName seems to fail 100% of the
time when run with java11 (or java12), regardless of seed, on both master & 7x.
The nature of the problem and the way our htp stack works suggests it *may*
ultimately be a jetty bug (perhaps related to [jetty
issue#2711|https://github.com/eclipse/jetty.project/issues/2711]?)
*HOWEVER* ... as far as i can tell, whatever the root cause is, seems to have
been fixed on the {{jira/http2}} branch (as of
52bc163dc1804c31af09c1fba99647005da415ad) which should hopefully be getting
merged to master soon.
Filing this issue largely for tracking purpose, although we may also want to
use it for discussions/considerations of other backports/fixes to 7x
Summary: Known OpenJDK >= 11 SSL (TLSv1.3) bugs can cause problems with
Solr (was: Skip running tests with SSL on Java 11 to 11.0.2)
Updated summary & description in an attempt to be more helpful for end users
(who may encounter problems in the wild) to try and reflect the full scope of
known SSL issues they might encounter if they use one of these affected JVMs.
From an issue tracking standpoint, the fact that we're "skipping" tests on
these JVMs should really be secondary to *why* we're skipping these tests.
> Known OpenJDK >= 11 SSL (TLSv1.3) bugs can cause problems with Solr
> -------------------------------------------------------------------
>
> Key: SOLR-12988
> URL: https://issues.apache.org/jira/browse/SOLR-12988
> Project: Solr
> Issue Type: Test
> Reporter: Hoss Man
> Assignee: Cao Manh Dat
> Priority: Major
> Labels: Java11, Java12, Java13
> Attachments: SOLR-12988.patch, SOLR-12988.patch, SOLR-13413.patch
>
>
> There are several known OpenJDK JVM bugs (begining with Java11, when TLS v1.3
> support was first added) that are known to affect Solr's SSL support, and
> have caused numerous test failures -- notably early "testing" builds of
> OpenJDK 11, 12, & 13, as well as the officially released OpenJDK 11, 11.0.1,
> and 11.0.2.
> From the standpoint of the Solr project, there is very little we can do to
> mitigate these bugs, and have taken steps to ensure any code using our
> {{SSLTestConfig}} / {{RandomizeSSL}} test-framework classes will be "SKIPed"
> with an {{AssumptionViolatedException}} when used on JVMs that are known to
> be problematic.
> Users who encounter any of the types of failures described below, or
> developers who encounter test runs that "SKIP" with a message refering to
> this issue ID, are encouraged to Upgrade their JVM. (or as a last resort: try
> disabling "TLSv1.3" in your JVM security properties)
> ----
> Examples of known bugs as they have manifested in Solr tests...
> * https://bugs.openjdk.java.net/browse/JDK-8212885
> ** "TLS 1.3 resumed session does not retain peer certificate chain"
> ** affects users with {{checkPeerNames=true}} in your SSL configuration
> ** causes 100% failure rate in Solr's
> {{TestMiniSolrCloudClusterSSL.testSslWithCheckPeerName}}
> ** can result in exceptions for SolrJ users, or in solr cloud server logs
> when making intra-node requests, with a root cause of
> "javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated"
> ** {noformat}
> [junit4] 2> Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer
> not authenticated
> [junit4] 2> at
> java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:526)
> [junit4] 2> at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:464)
> [junit4] 2> at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:397)
> [junit4] 2> at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
> [junit4] 2> at
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> [junit4] 2> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
> [junit4] 2> at
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
> [junit4] 2> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
> [junit4] 2> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
> [junit4] 2> at
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> [junit4] 2> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
> [junit4] 2> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> [junit4] 2> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> [junit4] 2> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> [junit4] 2> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
> {noformat}
> * https://bugs.openjdk.java.net/browse/JDK-8213202
> ** "Possible race condition in TLS 1.3 session resumption"
> ** May affect any and all Solr SSL users, although noted only in tests when
> "clientAuth" was configured to be false
> ** Causes non-reproducing test failures, and sporadic end user exceptions
> with a root cause of "javax.net.ssl.SSLException: Received fatal alert:
> internal_error "
> ** SSL Debugging may indicate "Fatal (INTERNAL_ERROR): Session has no PSK"
> ** {noformat}
> [junit4] 2> Caused by: javax.net.ssl.SSLException: Received fatal alert:
> internal_error
> [junit4] 2> at
> sun.security.ssl.Alert.createSSLException(Alert.java:129) ~[?:?]
> [junit4] 2> at
> sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
> [junit4] 2> at
> sun.security.ssl.TransportContext.fatal(TransportContext.java:308) ~[?:?]
> [junit4] 2> at
> sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:279) ~[?:?]
> [junit4] 2> at
> sun.security.ssl.TransportContext.dispatch(TransportContext.java:181) ~[?:?]
> [junit4] 2> at
> sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?]
> [junit4] 2> at
> sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) ~[?:?]
> [junit4] 2> at
> sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
> ~[?:?]
> [junit4] 2> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) ~[?:?]
> [junit4] 2> at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> ~[httpclient-4.5.6.jar:4.5.6]
> [junit4] 2> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:555)
> ~[java/:?]
> [junit4] 2> ... 13 more
> {noformat}
> * https://bugs.openjdk.java.net/browse/JDK-8224829
> ** "AsyncSSLSocketClose.java has timing issue"
> ** May affect any and all Solr SSL users running early testing versions of
> java 13 or 14.
> ** Causes non-reproducing test failures, and sporadic end user exceptions
> with a root cause of "javax.net.ssl.SSLException: Software caused connection
> abort: recv failed"
> ** {noformat}
> javax.net.ssl.SSLException: Software caused connection abort: recv failed
> at
> __randomizedtesting.SeedInfo.seed([AA73C7E858ABD2EE:88D2A395FDC7B4AB]:0)
> at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
> at
> java.base/sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1501)
> at
> java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:935)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
> at
> org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
> at
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
> at
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
> at
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
> at
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
