See <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/5020/>
------------------------------------------
[...truncated 14006 lines...]
[junit4:junit4] 2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket
connection for client /127.0.0.1:58646 which had sessionid 0x13aa99d68d00002
[junit4:junit4] 2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket
connection for client /127.0.0.1:58708 which had sessionid 0x13aa99d68d00004
[junit4:junit4] 2> 690757 T303 oaz.ClientCnxn$SendThread.run Unable to read
additional data from server sessionid 0x13aa99d68d00002, likely server has
closed socket, closing socket connection and attempting reconnect
[junit4:junit4] 2> 690757 T331 oaz.ClientCnxn$SendThread.run Unable to read
additional data from server sessionid 0x13aa99d68d00004, likely server has
closed socket, closing socket connection and attempting reconnect
[junit4:junit4] 2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket
connection for client /127.0.0.1:59242 which had sessionid 0x13aa99d68d00006
[junit4:junit4] 2> 690757 T355 oaz.ClientCnxn$SendThread.run Unable to read
additional data from server sessionid 0x13aa99d68d00006, likely server has
closed socket, closing socket connection and attempting reconnect
[junit4:junit4] 2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket
connection for client /127.0.0.1:58739 which had sessionid 0x13aa99d68d00005
[junit4:junit4] 2> 690757 T343 oaz.ClientCnxn$SendThread.run Unable to read
additional data from server sessionid 0x13aa99d68d00005, likely server has
closed socket, closing socket connection and attempting reconnect
[junit4:junit4] 2> 690757 T282 oazs.NIOServerCnxn.closeSock Closed socket
connection for client /127.0.0.1:58674 which had sessionid 0x13aa99d68d00003
[junit4:junit4] 2> 690757 T284 oazs.NIOServerCnxn$Factory.run NIOServerCnxn
factory exited run method
[junit4:junit4] 2> 690757 T317 oaz.ClientCnxn$SendThread.run Unable to read
additional data from server sessionid 0x13aa99d68d00003, likely server has
closed socket, closing socket connection and attempting reconnect
[junit4:junit4] 2> 690757 T282 oazs.FinalRequestProcessor.shutdown shutdown
of request processor complete
[junit4:junit4] 2> 690757 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
58639
[junit4:junit4] 2> 690757 T282 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=25232942
[junit4:junit4] 2> 690757 T282 oasc.SolrCore.close [collection1] CLOSING
SolrCore org.apache.solr.core.SolrCore@7e4f51
[junit4:junit4] 2> 690757 T282 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=0,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 690757 T282 oasc.SolrCore.decrefSolrCoreState Closing
SolrCoreState
[junit4:junit4] 2> 690757 T282 oasu.DefaultSolrCoreState.closeIndexWriter
SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 690757 T282 oasu.DefaultSolrCoreState.closeIndexWriter
closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 690757 T282 oasc.SolrCore.closeSearcher [collection1]
Closing main searcher on request.
[junit4:junit4] 2> 690757 T305 oasc.Overseer$ClusterStateUpdater.amILeader
According to ZK I (id=88569636220567554-127.0.0.1:58639_solr-n_0000000000) am
no longer a leader.
[junit4:junit4] 2> 690866 T318 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@4985d9 name:ZooKeeperConnection
Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
[junit4:junit4] 2> 690866 T344 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@13c22b0 name:ZooKeeperConnection
Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
[junit4:junit4] 2> 690866 T356 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@1b66ab4 name:ZooKeeperConnection
Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
[junit4:junit4] 2> 690866 T318 oascc.ConnectionManager.process zkClient has
disconnected
[junit4:junit4] 2> 690866 T344 oascc.ConnectionManager.process zkClient has
disconnected
[junit4:junit4] 2> 690866 T356 oascc.ConnectionManager.process zkClient has
disconnected
[junit4:junit4] 2> 690866 T332 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@b57ca1 name:ZooKeeperConnection
Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
[junit4:junit4] 2> 690866 T332 oascc.ConnectionManager.process zkClient has
disconnected
[junit4:junit4] 2> 690866 T304 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@14a65f8 name:ZooKeeperConnection
Watcher:127.0.0.1:58626/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
[junit4:junit4] 2> 690866 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00002
closed
[junit4:junit4] 2> 690866 T304 oascc.ConnectionManager.process
Client->ZooKeeper status change trigger but we are already closed
[junit4:junit4] 2> 690866 T304 oaz.ClientCnxn$EventThread.run EventThread
shut down
[junit4:junit4] 2> 690867 T282 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 690918 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
58669
[junit4:junit4] 2> 690918 T282 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=2908606
[junit4:junit4] 2> 690918 T282 oasc.SolrCore.close [collection1] CLOSING
SolrCore org.apache.solr.core.SolrCore@d75f7a
[junit4:junit4] 2> 690922 T282 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=0,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 690922 T282 oasc.SolrCore.decrefSolrCoreState Closing
SolrCoreState
[junit4:junit4] 2> 690922 T282 oasu.DefaultSolrCoreState.closeIndexWriter
SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 690923 T282 oasu.DefaultSolrCoreState.closeIndexWriter
closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 690923 T282 oasc.SolrCore.closeSearcher [collection1]
Closing main searcher on request.
[junit4:junit4] 2> 692065 T343 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:58626
[junit4:junit4] 2> 692236 T317 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:58626
[junit4:junit4] 2> 692580 T331 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:58626
[junit4:junit4] 2> 692845 T355 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:58626
[junit4:junit4] 2> 693063 T343 oaz.ClientCnxn$SendThread.run WARNING Session
0x13aa99d68d00005 for server null, unexpected error, closing socket connection
and attempting reconnect java.net.ConnectException: Connection refused: no
further information
[junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native
Method)
[junit4:junit4] 2> at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
[junit4:junit4] 2> at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
[junit4:junit4] 2>
[junit4:junit4] 2> 693314 T318 oaz.ClientCnxn$EventThread.run EventThread
shut down
[junit4:junit4] 2> 693314 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00003
closed
[junit4:junit4] 2> 693325 T282 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 693376 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
58703
[junit4:junit4] 2> 693376 T282 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=15813854
[junit4:junit4] 2> 693376 T282 oasc.SolrCore.close [collection1] CLOSING
SolrCore org.apache.solr.core.SolrCore@6b55c1
[junit4:junit4] 2> 693380 T282 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=1,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 693380 T282 oasc.SolrCore.decrefSolrCoreState Closing
SolrCoreState
[junit4:junit4] 2> 693381 T282 oasu.DefaultSolrCoreState.closeIndexWriter
SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 693381 T282 oasu.DefaultSolrCoreState.closeIndexWriter
closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 693383 T282 oasc.SolrCore.closeSearcher [collection1]
Closing main searcher on request.
[junit4:junit4] 2> 693634 T332 oaz.ClientCnxn$EventThread.run EventThread
shut down
[junit4:junit4] 2> 693634 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00004
closed
[junit4:junit4] 2> 693644 T282 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 693695 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
58734
[junit4:junit4] 2> 693695 T282 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=18062375
[junit4:junit4] 2> 693814 T355 oaz.ClientCnxn$SendThread.run WARNING Session
0x13aa99d68d00006 for server null, unexpected error, closing socket connection
and attempting reconnect java.net.ConnectException: Connection refused: no
further information
[junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native
Method)
[junit4:junit4] 2> at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
[junit4:junit4] 2> at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
[junit4:junit4] 2>
[junit4:junit4] 2> 695169 T355 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:58626
[junit4:junit4] 2> 695169 T343 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:58626
[junit4:junit4] 2> 696183 T355 oaz.ClientCnxn$SendThread.run WARNING Session
0x13aa99d68d00006 for server null, unexpected error, closing socket connection
and attempting reconnect java.net.ConnectException: Connection refused: no
further information
[junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native
Method)
[junit4:junit4] 2> at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
[junit4:junit4] 2> at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
[junit4:junit4] 2>
[junit4:junit4] 2> 696292 T344 oaz.ClientCnxn$EventThread.run EventThread
shut down
[junit4:junit4] 2> 696292 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00005
closed
[junit4:junit4] 2> 696293 T282 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 696344 T282 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
59235
[junit4:junit4] 2> 696344 T282 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=33028132
[junit4:junit4] 2> 696344 T282 oasc.SolrCore.close [collection1] CLOSING
SolrCore org.apache.solr.core.SolrCore@14c2b2f
[junit4:junit4] 2> 696353 T282 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=1,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 696354 T282 oasc.SolrCore.decrefSolrCoreState Closing
SolrCoreState
[junit4:junit4] 2> 696354 T282 oasu.DefaultSolrCoreState.closeIndexWriter
SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 696354 T282 oasu.DefaultSolrCoreState.closeIndexWriter
closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 696356 T282 oasc.SolrCore.closeSearcher [collection1]
Closing main searcher on request.
[junit4:junit4] 2> 698109 T355 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:58626
[junit4:junit4] 2> 699232 T356 oaz.ClientCnxn$EventThread.run EventThread
shut down
[junit4:junit4] 2> 699232 T282 oaz.ZooKeeper.close Session: 0x13aa99d68d00006
closed
[junit4:junit4] 2> 699248 T282 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 699501 T282 oas.SolrTestCaseJ4.tearDown ###Ending
testDistribSearch
[junit4:junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.method=testDistribSearch
-Dtests.seed=F1B077980FD36247 -Dtests.slow=true -Dtests.locale=bg
-Dtests.timezone=Indian/Mahe -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 700s J4 |
FullSolrCloudDistribCmdsTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: There are still
nodes recoverying - waited for 600 seconds
[junit4:junit4] > at
__randomizedtesting.SeedInfo.seed([F1B077980FD36247:7056F980788C027B]:0)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:156)
[junit4:junit4] > at
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:115)
[junit4:junit4] > at
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:110)
[junit4:junit4] > at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:553)
[junit4:junit4] > at
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:65)
[junit4:junit4] > at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:695)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit4:junit4] > at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit4:junit4] > at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4] > at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> 699517 T282 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 822211 T282 oas.SolrTestCaseJ4.endTrackingSearchers SEVERE
ERROR: SolrIndexSearcher opens=7 closes=6
[junit4:junit4] 2> 822211 T281 ccr.ThreadLeakControl.checkThreadLeaks WARNING
Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> 842366 T281 ccr.ThreadLeakControl.checkThreadLeaks SEVERE
1 thread leaked from SUITE scope at
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest:
[junit4:junit4] 2> 1) Thread[id=345,
name=searcherExecutor-163-thread-1, state=WAITING,
group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] 2> at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit4:junit4] 2> at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4] 2> at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> 842366 T281 ccr.ThreadLeakControl.tryToInterruptAll
Starting to interrupt leaked threads:
[junit4:junit4] 2> 1) Thread[id=345,
name=searcherExecutor-163-thread-1, state=WAITING,
group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4] 2> 845455 T281 ccr.ThreadLeakControl.tryToInterruptAll SEVERE
There are still zombie threads that couldn't be terminated:
[junit4:junit4] 2> 1) Thread[id=345,
name=searcherExecutor-163-thread-1, state=WAITING,
group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] 2> at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit4:junit4] 2> at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4] 2> at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> NOTE: test params are: codec=Asserting,
sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=bg,
timezone=Indian/Mahe
[junit4:junit4] 2> NOTE: Windows Vista 6.0 x86/Sun Microsystems Inc. 1.6.0_23
(32-bit)/cpus=4,threads=2,free=6564744,total=32903168
[junit4:junit4] 2> NOTE: All tests run in this JVM:
[XsltUpdateRequestHandlerTest, TestDocSet, TermVectorComponentTest,
SolrCmdDistributorTest, TestPseudoReturnFields, URLClassifyProcessorTest,
NoCacheHeaderTest, TestSystemIdResolver, FileBasedSpellCheckerTest,
TestIndexSearcher, TestStressReorder, FastVectorHighlighterTest,
TestLMJelinekMercerSimilarityFactory, StatsComponentTest, ScriptEngineTest,
LukeRequestHandlerTest, FullSolrCloudDistribCmdsTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.seed=F1B077980FD36247
-Dtests.slow=true -Dtests.locale=bg -Dtests.timezone=Indian/Mahe
-Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR 0.00s J4 | FullSolrCloudDistribCmdsTest (suite) <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: ERROR:
SolrIndexSearcher opens=7 closes=6
[junit4:junit4] > at
__randomizedtesting.SeedInfo.seed([F1B077980FD36247]:0)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at
org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:242)
[junit4:junit4] > at
org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:97)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit4:junit4] > at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit4:junit4] > at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] > Throwable #2:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE
scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest:
[junit4:junit4] > 1) Thread[id=345, name=searcherExecutor-163-thread-1,
state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] > at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit4:junit4] > at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4] > at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4] > at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4] > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] > at
__randomizedtesting.SeedInfo.seed([F1B077980FD36247]:0)
[junit4:junit4] > Throwable #3:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie
threads that couldn't be terminated:
[junit4:junit4] > 1) Thread[id=345, name=searcherExecutor-163-thread-1,
state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] > at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit4:junit4] > at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4] > at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4] > at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4] > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] > at
__randomizedtesting.SeedInfo.seed([F1B077980FD36247]:0)
[junit4:junit4] Completed on J4 in 845.50s, 1 test, 2 failures, 2 errors <<<
FAILURES!
[junit4:junit4]
[junit4:junit4]
[junit4:junit4] Tests with failures:
[junit4:junit4] -
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch
[junit4:junit4] - org.apache.solr.cloud.FullSolrCloudDistribCmdsTest (suite)
[junit4:junit4]
[junit4:junit4]
[junit4:junit4] JVM J0: 2.62 .. 273.28 = 270.66s
[junit4:junit4] JVM J1: 2.62 .. 269.64 = 267.02s
[junit4:junit4] JVM J2: 2.87 .. 341.30 = 338.43s
[junit4:junit4] JVM J3: 2.62 .. 270.27 = 267.65s
[junit4:junit4] JVM J4: 3.40 .. 974.88 = 971.47s
[junit4:junit4] JVM J5: 2.62 .. 280.39 = 277.77s
[junit4:junit4] JVM J6: 2.62 .. 275.62 = 273.00s
[junit4:junit4] JVM J7: 2.62 .. 270.00 = 267.38s
[junit4:junit4] Execution time total: 16 minutes 14 seconds
[junit4:junit4] Tests summary: 233 suites, 965 tests, 3 suite-level errors, 1
failure, 13 ignored (8 assumptions)
BUILD FAILED
<http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/build.xml>:39:
The following error occurred while executing this line:
<http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/solr/build.xml>:178:
The following error occurred while executing this line:
<http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/module-build.xml>:61:
The following error occurred while executing this line:
<http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/common-build.xml>:1159:
The following error occurred while executing this line:
<http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/common-build.xml>:827:
There were test failures: 233 suites, 965 tests, 3 suite-level errors, 1
failure, 13 ignored (8 assumptions)
Total time: 24 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
