Hi Mark,
last night this happened again (for the 3rd time). For the second time in a windows VM. Maybe it was another test than the one we have seen it first. I don’t think disabling SSL helps here. It happens together with SSL, that’s right, but from the log file there seems to be some bug in the test setup: The test tries to reconnect endless without stopping to try again and again and later failing the test. This fills disk space quite fast. This also makes the tests never end. The rate of reconnects is so high, that the log file is filled with megabytes in very short time. When disk is full, the carrot framework is no longer able to handle this case and the whole JVM setup hangs. Unfortunately, I have no data available anymore because I had to revert the Windows Virtualbox VM to the latest clean snapshot: [junit4] Could not serialize report for suite org.apache.solr.cloud.TestShortCircuitedRequests: java.io.IOException: There is not enough space on the disk [junit4] Mar 24, 2014 8:44:09 PM com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus$LoggingSubscriberExceptionHandler handleException [junit4] SEVERE: Could not dispatch event: com.carrotsearch.ant.tasks.junit4.listeners.TextReport@1290a22 to public void com.carrotsearch.ant.tasks.junit4.listeners.TextReport.onSuiteResult(com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatedSuiteResultEvent) I am not even sure if this is the test that caused this. Would it be possible to catch those cases while running tests (maybe before the disk is full) and fail the build? Maybe something that the event file is not allowed to grow beyond a specific size. If it grows, the test framework fails the whole build? We can have something like maximum size of 1 GB (configureable). Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen <http://www.thetaphi.de/> http://www.thetaphi.de eMail: u...@thetaphi.de From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Wednesday, March 19, 2014 6:51 PM To: dev@lucene.apache.org Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9828 - Failure! I'll disable SSL for that test for now. SSL in general has been hard to get working smoothly with tests unfortunately. I've got a JIRA issue to look at improving it, but not likely I'll look into it for some time, so until then, tests having issues with SSL should likely simply disable SSL for now. - Mark On Tue, Mar 18, 2014 at 4:54 AM, Dawid Weiss <dawid.we...@cs.put.poznan.pl> wrote: It's a lot of error messages like this one. I have the full syserr dump if needed. D. 2773140 T6223 oasc.ChaosMonkeyNothingIsSafeTest$FullThrottleStopableIndexingThread$1.handleError WARN suss error java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522) at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:232) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) On Tue, Mar 18, 2014 at 9:46 AM, Uwe Schindler <u...@thetaphi.de> wrote: > I dig "tail -10000" to extract the last 10000 lines. The file is also in the > archive at same place. > > It is indeed a loop. The code loops endless in a "Connection Refused" loop, > without any delay between the events. After approx. 2:50 hours this hit the > limits of the SSD file system. This test fails so often since it was "fixed", > we should revert to @BadApple. > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > >> -----Original Message----- >> From: dawid.we...@gmail.com [mailto:dawid.we...@gmail.com] On Behalf >> Of Dawid Weiss >> Sent: Tuesday, March 18, 2014 9:16 AM >> To: dev@lucene.apache.org >> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # >> 9828 - Failure! >> >> > junit4-J0-20140317_230107_233.events 8.17 GB [fingerprint] view >> > >> > This build created a 8.17 GB big events file and failed with out of space. >> How can this happen? >> >> Can you peek at it? It's probably something that logs in a loop or something. >> I'm fetching it right now, let's see if I can figure it out. >> >> D. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional >> commands, e-mail: dev-h...@lucene.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org -- - Mark http://about.me/markrmiller
