Hi, this looks like some bug in the MacOSX libc. Sometimes, Java 7 also crashes on OSX in this code part (because it segfaults when producing the error message). There is already a bug open at Oracle, but they have no idea how to fix. Currently it is impossible to run something like Tomcat or Jetty on an OSX server in production...
https://bugs.openjdk.java.net/browse/JDK-8024045 Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: [email protected] > -----Original Message----- > From: Per Steffensen [mailto:[email protected]] > Sent: Tuesday, February 11, 2014 4:46 PM > To: [email protected] > Subject: Trunk tests fail (on Mac OSX) > > I am sure you have noticed it from Jenkins, but tests fail (most of the > time) on trunk. E.g. I have been running BasicDistributedZk2Test numerous > times from Eclipse on my Mac. Revision 1567049 on trunk. > Sometimes the test is green, but most of the times it is not. It is random > exactly where it fails, but there is an example below. I believe that when it > fails it is always with the kind of exception > (java.net.SocketException: Invalid argument) shown at the bottom. My > guess it that is has to do with SOLR-3854. > > Anyone working on fixing this? Any current knowledge about what the > problem is? An estimate on when things are consistently green again on > trunk? > > Regards, Per Steffensen > > --------- An example of where the test fails ----------------------------- > org.apache.solr.client.solrj.SolrServerException: No live SolrServers > available > to handle this request:[https://127.0.0.1:58496/qls/z/collection1, > https://127.0.0.1:58500/qls/z/collection1, > https://127.0.0.1:58493/qls/z/collection1, > https://127.0.0.1:58490/qls/z/collection1] > at > __randomizedtesting.SeedInfo.seed([F0E9F84409868187:710F765C7ED9E1BB > ]:0) > at > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j > ava:352) > at > org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.jav > a:635) > at > org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.ja > va:90) > at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301) > at > org.apache.solr.cloud.AbstractFullDistribZkTestBase.queryServer(AbstractFul > lDistribZkTestBase.java:1356) > at > org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc > hTestCase.java:561) > at > org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc > hTestCase.java:543) > at > org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearc > hTestCase.java:522) > at > org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkQueries(AbstractF > ullDistribZkTestBase.java:754) > at > org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Te > st.java:107) > at > org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistri > butedSearchTestCase.java:868) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > ava:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces > sorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomize > dRunner.java:1559) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(Rando > mizedRunner.java:79) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Rando > mizedRunner.java:737) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Rando > mizedRunner.java:773) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Rando > mizedRunner.java:787) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1. > evaluate(SystemPropertiesRestoreRule.java:53) > at > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRule > SetupTeardownChained.java:50) > at > org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCa > cheSanity.java:51) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA > fterRule.java:46) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1 > .evaluate(SystemPropertiesInvariantRule.java:55) > at > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleTh > readAndTestName.java:49) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule > IgnoreAfterMaxFailures.java:70) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure > .java:48) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat > ementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner. > run(ThreadLeakControl.java:358) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask > (ThreadLeakControl.java:782) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadL > eakControl.java:442) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran > domizedRunner.java:746) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(Rando > mizedRunner.java:648) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(Rando > mizedRunner.java:682) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Rando > mizedRunner.java:693) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat > ementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1. > evaluate(SystemPropertiesRestoreRule.java:53) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeA > fterRule.java:46) > at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreCl > assName.java:42) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1 > .evaluate(SystemPropertiesInvariantRule.java:55) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet > hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMet > hodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat > ementAdapter.java:36) > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAss > ertionsRequired.java:43) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure > .java:48) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule > IgnoreAfterMaxFailures.java:70) > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnore > TestSuites.java:55) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat > ementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner. > run(ThreadLeakControl.java:358) > at java.lang.Thread.run(Thread.java:744) > Caused by: > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: > org.apache.solr.client.solrj.SolrServerException: No live SolrServers > available > to handle this request:[https://127.0.0.1:58496/qls/z/collection1, > https://127.0.0.1:58500/qls/z/collection1, > https://127.0.0.1:58493/qls/z/collection1, > https://127.0.0.1:58490/qls/z/collection1] > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java: > 498) > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java: > 199) > at > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j > ava:283) > ... 50 more > > --------------------- What I saw in the log during this test-run > ---------------------- > 21901 T71 C29 P58493 oasc.SolrException.log ERROR > org.apache.solr.common.SolrException: > org.apache.solr.client.solrj.SolrServerException: No live SolrServers > available > to handle this request:[https://127.0.0.1:58496/qls/z/collection1, > https://127.0.0.1:58500/qls/z/collection1, > https://127.0.0.1:58493/qls/z/collection1, > https://127.0.0.1:58490/qls/z/collection1] > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sea > rchHandler.java:309) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandl > erBase.java:135) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1916) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780 > ) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427 > ) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217 > ) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle > r.java:1419) > at > org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter( > JettySolrRunner.java:134) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle > r.java:1419) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.ja > va:229) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java: > 137) > at > org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.j > ava:1077) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.jav > a:193) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.ja > va:1009) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java: > 135) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.ja > va:116) > at org.eclipse.jetty.server.Server.handle(Server.java:368) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHtt > pConnection.java:489) > at > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractH > ttpConnection.java:942) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerCo > mplete(AbstractHttpConnection.java:1004) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection. > java:82) > at > org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoi > nt.java:628) > at > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint > .java:52) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.j > ava:608) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.ja > va:543) > at java.lang.Thread.run(Thread.java:744) > Caused by: org.apache.solr.client.solrj.SolrServerException: No live > SolrServers available to handle this > request:[https://127.0.0.1:58496/qls/z/collection1, > https://127.0.0.1:58500/qls/z/collection1, > https://127.0.0.1:58493/qls/z/collection1, > https://127.0.0.1:58490/qls/z/collection1] > at > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j > ava:352) > at > org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBal > ancedRequest(HttpShardHandlerFactory.java:205) > at > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHand > ler.java:161) > at > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHand > ler.java:1) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav > a:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja > va:615) > ... 1 more > Caused by: org.apache.solr.client.solrj.SolrServerException: > IOException occured when talking to server at: > https://127.0.0.1:58490/qls/z/collection1 > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java: > 510) > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java: > 199) > at > org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.j > ava:283) > ... 9 more > Caused by: java.net.SocketException: Invalid argument > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:152) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) > at sun.security.ssl.InputRecord.read(InputRecord.java:480) > at > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) > at > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) > at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessio > nInputBuffer.java:160) > at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java: > 84) > at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessio > nInputBuffer.java:273) > at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultH > ttpResponseParser.java:140) > at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultH > ttpResponseParser.java:57) > at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessagePars > er.java:260) > at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader > (AbstractHttpClientConnection.java:283) > at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader > (DefaultClientConnection.java:251) > at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponse > Header(ManagedClientConnectionImpl.java:197) > at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRe > questExecutor.java:271) > at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecut > or.java:123) > at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequ > estDirector.java:682) > at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequest > Director.java:486) > at > org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient > .java:863) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient > .java:82) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient > .java:106) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient > .java:57) > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java: > 395) > ... 11 more > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] For additional > commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
