[
https://issues.apache.org/jira/browse/SOLR-13778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994611#comment-16994611
]
Dawid Weiss commented on SOLR-13778:
------------------------------------
So, this one was fun. Please read if you like good suspense...
The bug reproduces for me on Windows without any problems on newest JDKs
(openjdk-11+28, openjdk-13+33).
I used an older JDK (11.0.4+11) to dump parallel logs from Linux and Windows.
Linux passes, Windows doesn't.
Disabling TLS1.3 at JVM level doesn't help (although you can see int the logs
that TLS13 is no longer used in the communication) - see
"adoptopenjdk-11.0.4+11-no-tls13" folder in the attached zipped logs.
I see a bunch of odd error messages in the log but I don't know the SSL layer
well enough to tell what they're actually doing and why. The oddest one to me
is the perceived assymmetry in inbound/ outbound sockets:
{code:java}
2> javax.net.ssl|ERROR|6B|closeThreadPool-31-thread-2|2019-12-12 10:49:45.234
WAT|TransportContext.java:312|Fatal (INTERNAL_ERROR): closing inbound before
receiving peer's close_notify
{code}
The stack trace of the recv exception contains so many frames that I decided to
dump full logs (so that apache's httpclient is included).
"adoptopenjdk-11.0.4+11-full-debug" contain these logs (with full debug from
javax.net and jetty/ httpclient loggers). The
problem/ difference starts around this moment:
{code:java}
o.a.h.i.e.MainClientExec Executing request GET
/solr/admin/collections?action=CLUSTERPROP&name=legacyCloud&val=false&wt=javabin&version=2
{code}
While the linux code receives a close_notify at SSL level and then proceeds to
write to a broken pipe:
{code:java}
[junit4] 2>
javax.net.ssl|WARNING|10|TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF]|2019-12-12
12:44:56.450 WAT|TransportContext.java:245|Warning: failed to send warning
alert CLOSE_NOTIFY (
[junit4] 2> "throwable" : {
[junit4] 2> java.net.SocketException: Broken pipe (Write failed)
[junit4] 2> at
java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
[junit4] 2> at
java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
{code}
...and httpclient retries:
{code:java}
[junit4] 2>
javax.net.ssl|DEBUG|10|TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF]|2019-12-12
12:44:56.450 WAT|SSLSocketImpl.java:636|close inbound of SSLSocket
[junit4] 2>
javax.net.ssl|DEBUG|10|TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF]|2019-12-12
12:44:56.450 WAT|SSLSocketImpl.java:473|duplex close of SSLSocket
[junit4] 2>
javax.net.ssl|DEBUG|10|TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF]|2019-12-12
12:44:56.450 WAT|SSLSocketImpl.java:1381|close the SSL connection (passive)
[junit4] 2> ## 7753 DEBUG
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.h.i.c.DefaultManagedHttpClientConnection http-outgoing-0: Shutdown
connection
[junit4] 2> ## 7753 DEBUG
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.h.i.e.MainClientExec Connection discarded
[junit4] 2> ## 7753 DEBUG
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.h.i.c.PoolingHttpClientConnectionManager Connection released: [id:
0][route: {s}->https://127.0.0.1:36121][state: class
org.apache.solr.client.solrj.impl.HttpSolrClient][total kept alive: 0; route
allocated: 0 of 10000; total allocated: 0 of 10000]
[junit4] 2> ## 7753 DEBUG
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.s.c.s.i.SolrHttpRequestRetryHandler Retry http request 1 out of 1
[junit4] 2> ## 7753 DEBUG
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.s.c.s.i.SolrHttpRequestRetryHandler Retry, request should be
idempotent
[junit4] 2> ## 7753 INFO
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.h.i.e.RetryExec I/O exception
(org.apache.http.NoHttpResponseException) caught when processing request to
{s}->https://127.0.0.1:36121: The target server failed to respond
{code}
the Windows counterpart fails on recv (?)...
{code:java}
[junit4] 2>
javax.net.ssl|WARNING|11|TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF]|2019-12-12
12:35:49.820 WAT|SSLSocketImpl.java:1289|handling exception (
[junit4] 2> "throwable" : {
[junit4] 2> java.net.SocketException: Software caused connection abort:
recv failed
[junit4] 2> at
java.base/java.net.SocketInputStream.socketRead0(Native Method)
[junit4] 2> at
java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
{code}
and then fails to retry (!).
{code:java}
[junit4] 2> ## 18393 DEBUG
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.h.i.c.DefaultManagedHttpClientConnection http-outgoing-0: Close
connection
[junit4] 2>
javax.net.ssl|DEBUG|11|TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF]|2019-12-12
12:35:49.825 WAT|SSLSocketImpl.java:670|close outbound of SSLSocket
[junit4] 2>
javax.net.ssl|DEBUG|11|TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF]|2019-12-12
12:35:49.825 WAT|SSLSocketImpl.java:636|close inbound of SSLSocket
[junit4] 2>
javax.net.ssl|WARNING|11|TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF]|2019-12-12
12:35:49.825 WAT|TransportContext.java:278|Closed transport, general or
untracked problem
[junit4] 2> ## 18394 DEBUG
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.h.i.c.DefaultManagedHttpClientConnection http-outgoing-0: Shutdown
connection
[junit4] 2> ## 18394 DEBUG
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.h.i.e.MainClientExec Connection discarded
[junit4] 2> ## 18394 DEBUG
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.h.i.c.PoolingHttpClientConnectionManager Connection released: [id:
0][route: {s}->https://127.0.0.1:56646][state: class
org.apache.solr.client.solrj.impl.HttpSolrClient][total kept alive: 0; route
allocated: 0 of 10000; total allocated: 0 of 10000]
[junit4] 2> ## 18394 DEBUG
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.s.c.s.i.SolrHttpRequestRetryHandler Retry http request 1 out of 1
[junit4] 2> ## 18394 DEBUG
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.s.c.s.i.SolrHttpRequestRetryHandler Do not retry, non retriable
class javax.net.ssl.SSLException
{code}
Look at the last line though: note the exception type:
{code:java}
SolrHttpRequestRetryHandler Do not retry, non retriable class
javax.net.ssl.SSLException
{code}
I've dumped the whole exception, including suppressions (sorry, it is long but
so is Ulysses and many people like it):
{code:java}
10810 INFO
(TEST-LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud-seed#[DEADBEEF])
[ ] o.a.s.c.s.i.SolrHttpRequestRetryHandler Do not retry, non retriable
class javax.net.ssl.SSLException
=> javax.net.ssl.SSLException: Software caused connection abort: recv
failed
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
javax.net.ssl.SSLException: Software caused connection abort: recv failed
at sun.security.ssl.Alert.createSSLException(Alert.java:127) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
~[?:?]
at
sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1315) ~[?:?]
at
sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:839)
~[?:?]
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
~[httpclient-4.5.6.jar:4.5.6]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:564)
~[main/:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
~[main/:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
~[main/:?]
at
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
~[main/:?]
at
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
~[main/:?]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1143)
~[main/:?]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:906)
~[main/:?]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:838)
~[main/:?]
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207)
~[main/:?]
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224)
~[main/:?]
at
org.apache.solr.cloud.LegacyCloudClusterPropTest.createAndTest(LegacyCloudClusterPropTest.java:88)
~[test/:?]
at
org.apache.solr.cloud.LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud(LegacyCloudClusterPropTest.java:80)
~[test/:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
~[randomizedtesting-runner-2.7.5.jar:?]
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
~[main/:?]
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
~[main/:?]
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
~[main/:?]
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
~[main/:?]
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
~[main/:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
~[randomizedtesting-runner-2.7.5.jar:?]
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
~[main/:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
~[main/:?]
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
~[main/:?]
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
~[main/:?]
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
~[main/:?]
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
~[main/:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
~[randomizedtesting-runner-2.7.5.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Suppressed: java.net.SocketException: Software caused connection abort:
socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
~[?:?]
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) ~[?:?]
at
java.net.SocketOutputStream.write(SocketOutputStream.java:150) ~[?:?]
at
sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
~[?:?]
at
sun.security.ssl.TransportContext.fatal(TransportContext.java:351) ~[?:?]
at
sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
at
sun.security.ssl.TransportContext.fatal(TransportContext.java:258) ~[?:?]
at
sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1315) ~[?:?]
at
sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:839)
~[?:?]
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
~[httpcore-4.4.10.jar:4.4.10]
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
~[httpclient-4.5.6.jar:4.5.6]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:564)
~[main/:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
~[main/:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
~[main/:?]
at
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
~[main/:?]
at
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
~[main/:?]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1143)
~[main/:?]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:906)
~[main/:?]
at
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:838)
~[main/:?]
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207)
~[main/:?]
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224)
~[main/:?]
at
org.apache.solr.cloud.LegacyCloudClusterPropTest.createAndTest(LegacyCloudClusterPropTest.java:88)
~[test/:?]
at
org.apache.solr.cloud.LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud(LegacyCloudClusterPropTest.java:80)
~[test/:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) ~[?:?]
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
~[randomizedtesting-runner-2.7.5.jar:?]
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
~[main/:?]
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
~[main/:?]
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
~[main/:?]
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
~[main/:?]
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
~[main/:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
~[randomizedtesting-runner-2.7.5.jar:?]
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
~[main/:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
~[main/:?]
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
~[main/:?]
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
~[main/:?]
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
~[main/:?]
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
~[main/:?]
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
~[randomizedtesting-runner-2.7.5.jar:?]
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
~[randomizedtesting-runner-2.7.5.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.net.SocketException: Software caused connection abort: recv
failed
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
~[?:?]
at java.net.SocketInputStream.read(SocketInputStream.java:168) ~[?:?]
at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
at
sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448) ~[?:?]
at
sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
~[?:?]
at
sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104)
~[?:?]
at
sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823)
~[?:?]
... 67 more
{code}
It's a weird one – with two suppressed socket exceptions (recv and socket write
error) and a top SSLException wrapping them both.
Now, the problem we see on the surface is caused by the fact that
javax.net.ssl.SSLException is *on the list of non-retriable classes* in
SolrHttpRequestRetryHandler so the code then follows a different path on
Windows (compared to Linux).
Adding javax.net.ssl.SSLException to retriable exceptions makes the test pass.
_But I don't know if it's a good fix;_ I don't know who came up with
"retriable" classes or why they're there in the first place. Neither can I
explain the difference between the two (seems like a different underlying close
operation on sockets).
Still, I think that's a decent step forward. Thoughts?
> Windows JDK SSL Test Failure trend: SSLException: Software caused connection
> abort: recv failed
> -----------------------------------------------------------------------------------------------
>
> Key: SOLR-13778
> URL: https://issues.apache.org/jira/browse/SOLR-13778
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Chris M. Hostetter
> Priority: Major
> Attachments: dumps-LegacyCloud.zip, logs-2019-12-12-1.zip
>
>
> Now that Uwe's jenkins build has been correctly reporting it's build results
> for my [automated
> reports|http://fucit.org/solr-jenkins-reports/failure-report.html] to pick
> up, I've noticed a pattern of failures that indicate a definite problem with
> using SSL on Windows (even with java 11.0.4
> )
> The symptommatic stack traces all contain...
> {noformat}
> ...
> [junit4] > Caused by: javax.net.ssl.SSLException: Software caused
> connection abort: recv failed
> [junit4] > at
> java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
> ...
> [junit4] > Caused by: java.net.SocketException: Software caused
> connection abort: recv failed
> [junit4] > at
> java.base/java.net.SocketInputStream.socketRead0(Native Method)
> ...
> {noformat}
> I suspect this may be related to
> [https://bugs.openjdk.java.net/browse/JDK-8209333] but i have no concrete
> evidence to back this up.
> I'll post some details of my analysis in comments...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
