Thanks for the info, I was able to reproduce the problem and it seems to be isolated to using two-way SSL on OSX. Check out my comment on SOLR-3854 for additional information. @Mark Miller seems very responsive for applying patches for this issue and will hopefully get this resolved pretty soon.
-Steve On Feb 11, 2014, at 4:02 PM, Per Steffensen <[email protected]> wrote: > I ran BasicDistributedZk2Test > * 5 times with SolrTestCaseJ4.ALLOW_SSL = false: 5 green runs > * 5 times with SolrTestCaseJ4.ALLOW_SSL = true: 2 green and 3 red runs > Do not know if that is statistics enough to conclude anything, but it smells > a little like a SSL issue, if you ask me. > > Mac OSX Maverick (10.9.1) > java -version > java version "1.7.0_51" > Java(TM) SE Runtime Environment (build 1.7.0_51-b13) > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) > > Regards, Per Steffensen > > On 11/02/14 17:14, Steve Davids wrote: >> A way to isolate if this is an SSL issue, you can turn off SSL for that >> specific test by adding: >> static { >> ALLOW_SSL = false; >> } >> to the test class. Although, I have seen that test error out every so often >> in non-SSL mode. Will take a look later on tonight though. >> >> -Steve >> >> >> On Tue, Feb 11, 2014 at 11:06 AM, Uwe Schindler <[email protected]> wrote: >> Hi, >> >> this looks like some bug in the MacOSX libc. Sometimes, Java 7 also crashes >> on OSX in this code part (because it segfaults when producing the error >> message). There is already a bug open at Oracle, but they have no idea how >> to fix. Currently it is impossible to run something like Tomcat or Jetty on >> an OSX server in production... >> >> https://bugs.openjdk.java.net/browse/JDK-8024045 >> >> Uwe >> >> ----- >> Uwe Schindler >> H.-H.-Meier-Allee 63, D-28213 Bremen >> http://www.thetaphi.de >> eMail: [email protected] >> >> >> > -----Original Message----- >> > From: Per Steffensen [mailto:[email protected]] >> > Sent: Tuesday, February 11, 2014 4:46 PM >> > To: [email protected] >> > Subject: Trunk tests fail (on Mac OSX) >> > >> > I am sure you have noticed it from Jenkins, but tests fail (most of the >> > time) on trunk. E.g. I have been running BasicDistributedZk2Test numerous >> > times from Eclipse on my Mac. Revision 1567049 on trunk. >> > Sometimes the test is green, but most of the times it is not. It is random >> > exactly where it fails, but there is an example below. I believe that when >> > it >> > fails it is always with the kind of exception >> > (java.net.SocketException: Invalid argument) shown at the bottom. My >> > guess it that is has to do with SOLR-3854. >> > >> > Anyone working on fixing this? Any current knowledge about what the >> > problem is? An estimate on when things are consistently green again on >> > trunk? >> > >> > Regards, Per Steffensen >> > >> > --------- An example of where the test fails ----------------------------- >> > org.apache.solr.client.solrj.SolrServerException: No live SolrServers >> > available >> > to handle this request:[https://127.0.0.1:58496/qls/z/collection1, >> > https://127.0.0.1:58500/qls/z/collection1, >> > https://127.0.0.1:58493/qls/z/collection1, >> > https://127.0.0.1:58490/qls/z/collection1] >> > at >> > __randomizedtesting.SeedInfo.seed([F0E9F84409868187:710F765C7ED9E1BB >> > ]:0) >> > at >> > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j >> > ava:352) >> > at >> > org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.jav >> > a:635) >> > at >> > org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.ja >> > va:90) >> > at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301) >> > at >> > org.apache.solr.cloud.AbstractFullDistribZkTestBase.queryServer(AbstractFul >> > lDistribZkTestBase.java:1356) >> > at >> > org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc >> > hTestCase.java:561) >> > at >> > org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc >> > hTestCase.java:543) >> > at >> > org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc >> > hTestCase.java:522) >> > at >> > org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkQueries(AbstractF >> > ullDistribZkTestBase.java:754) >> > at >> > org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Te >> > st.java:107) >> > at >> > org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistri >> > butedSearchTestCase.java:868) >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j >> > ava:57) >> > at >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces >> > sorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:606) >> > at >> > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomize >> > dRunner.java:1559) >> > at >> > com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(Rando >> > mizedRunner.java:79) >> > at >> > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Rando >> > mizedRunner.java:737) >> > at >> > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Rando >> > mizedRunner.java:773) >> > at >> > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Rando >> > mizedRunner.java:787) >> > at >> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1. >> > evaluate(SystemPropertiesRestoreRule.java:53) >> > at >> > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRule >> > SetupTeardownChained.java:50) >> > at >> > org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCa >> > cheSanity.java:51) >> > at >> > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA >> > fterRule.java:46) >> > at >> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1 >> > .evaluate(SystemPropertiesInvariantRule.java:55) >> > at >> > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleTh >> > readAndTestName.java:49) >> > at >> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule >> > IgnoreAfterMaxFailures.java:70) >> > at >> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure >> > .java:48) >> > at >> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat >> > ementAdapter.java:36) >> > at >> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner. >> > run(ThreadLeakControl.java:358) >> > at >> > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask >> > (ThreadLeakControl.java:782) >> > at >> > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadL >> > eakControl.java:442) >> > at >> > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran >> > domizedRunner.java:746) >> > at >> > com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(Rando >> > mizedRunner.java:648) >> > at >> > com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(Rando >> > mizedRunner.java:682) >> > at >> > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Rando >> > mizedRunner.java:693) >> > at >> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat >> > ementAdapter.java:36) >> > at >> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1. >> > evaluate(SystemPropertiesRestoreRule.java:53) >> > at >> > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA >> > fterRule.java:46) >> > at >> > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreCl >> > assName.java:42) >> > at >> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1 >> > .evaluate(SystemPropertiesInvariantRule.java:55) >> > at >> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet >> > hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> > at >> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet >> > hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> > at >> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat >> > ementAdapter.java:36) >> > at >> > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAss >> > ertionsRequired.java:43) >> > at >> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure >> > .java:48) >> > at >> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule >> > IgnoreAfterMaxFailures.java:70) >> > at >> > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnore >> > TestSuites.java:55) >> > at >> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat >> > ementAdapter.java:36) >> > at >> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner. >> > run(ThreadLeakControl.java:358) >> > at java.lang.Thread.run(Thread.java:744) >> > Caused by: >> > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: >> > org.apache.solr.client.solrj.SolrServerException: No live SolrServers >> > available >> > to handle this request:[https://127.0.0.1:58496/qls/z/collection1, >> > https://127.0.0.1:58500/qls/z/collection1, >> > https://127.0.0.1:58493/qls/z/collection1, >> > https://127.0.0.1:58490/qls/z/collection1] >> > at >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java: >> > 498) >> > at >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java: >> > 199) >> > at >> > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j >> > ava:283) >> > ... 50 more >> > >> > --------------------- What I saw in the log during this test-run >> > ---------------------- >> > 21901 T71 C29 P58493 oasc.SolrException.log ERROR >> > org.apache.solr.common.SolrException: >> > org.apache.solr.client.solrj.SolrServerException: No live SolrServers >> > available >> > to handle this request:[https://127.0.0.1:58496/qls/z/collection1, >> > https://127.0.0.1:58500/qls/z/collection1, >> > https://127.0.0.1:58493/qls/z/collection1, >> > https://127.0.0.1:58490/qls/z/collection1] >> > at >> > org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sea >> > rchHandler.java:309) >> > at >> > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandl >> > erBase.java:135) >> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1916) >> > at >> > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780 >> > ) >> > at >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427 >> > ) >> > at >> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217 >> > ) >> > at >> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle >> > r.java:1419) >> > at >> > org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter( >> > JettySolrRunner.java:134) >> > at >> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle >> > r.java:1419) >> > at >> > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) >> > at >> > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.ja >> > va:229) >> > at >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java: >> > 137) >> > at >> > org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301) >> > at >> > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.j >> > ava:1077) >> > at >> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) >> > at >> > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.jav >> > a:193) >> > at >> > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.ja >> > va:1009) >> > at >> > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java: >> > 135) >> > at >> > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.ja >> > va:116) >> > at org.eclipse.jetty.server.Server.handle(Server.java:368) >> > at >> > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHtt >> > pConnection.java:489) >> > at >> > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractH >> > ttpConnection.java:942) >> > at >> > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerCo >> > mplete(AbstractHttpConnection.java:1004) >> > at >> > org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) >> > at >> > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) >> > at >> > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection. >> > java:82) >> > at >> > org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) >> > at >> > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoi >> > nt.java:628) >> > at >> > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint >> > .java:52) >> > at >> > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.j >> > ava:608) >> > at >> > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.ja >> > va:543) >> > at java.lang.Thread.run(Thread.java:744) >> > Caused by: org.apache.solr.client.solrj.SolrServerException: No live >> > SolrServers available to handle this >> > request:[https://127.0.0.1:58496/qls/z/collection1, >> > https://127.0.0.1:58500/qls/z/collection1, >> > https://127.0.0.1:58493/qls/z/collection1, >> > https://127.0.0.1:58490/qls/z/collection1] >> > at >> > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j >> > ava:352) >> > at >> > org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBal >> > ancedRequest(HttpShardHandlerFactory.java:205) >> > at >> > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHand >> > ler.java:161) >> > at >> > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHand >> > ler.java:1) >> > at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> > at >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> > at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> > at >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav >> > a:1145) >> > at >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja >> > va:615) >> > ... 1 more >> > Caused by: org.apache.solr.client.solrj.SolrServerException: >> > IOException occured when talking to server at: >> > https://127.0.0.1:58490/qls/z/collection1 >> > at >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java: >> > 510) >> > at >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java: >> > 199) >> > at >> > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j >> > ava:283) >> > ... 9 more >> > Caused by: java.net.SocketException: Invalid argument >> > at java.net.SocketInputStream.socketRead0(Native Method) >> > at java.net.SocketInputStream.read(SocketInputStream.java:152) >> > at java.net.SocketInputStream.read(SocketInputStream.java:122) >> > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) >> > at sun.security.ssl.InputRecord.read(InputRecord.java:480) >> > at >> > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) >> > at >> > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) >> > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) >> > at >> > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessio >> > nInputBuffer.java:160) >> > at >> > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java: >> > 84) >> > at >> > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessio >> > nInputBuffer.java:273) >> > at >> > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultH >> > ttpResponseParser.java:140) >> > at >> > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultH >> > ttpResponseParser.java:57) >> > at >> > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessagePars >> > er.java:260) >> > at >> > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader >> > (AbstractHttpClientConnection.java:283) >> > at >> > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader >> > (DefaultClientConnection.java:251) >> > at >> > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponse >> > Header(ManagedClientConnectionImpl.java:197) >> > at >> > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRe >> > questExecutor.java:271) >> > at >> > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecut >> > or.java:123) >> > at >> > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequ >> > estDirector.java:682) >> > at >> > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequest >> > Director.java:486) >> > at >> > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient >> > .java:863) >> > at >> > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient >> > .java:82) >> > at >> > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient >> > .java:106) >> > at >> > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient >> > .java:57) >> > at >> > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java: >> > 395) >> > ... 11 more >> > >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] For additional >> > commands, e-mail: [email protected] >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >
