See <http://sierranevada.servebeer.com:8080/job/slow-io-beasting/4870/>
------------------------------------------
[...truncated 14303 lines...]
[junit4:junit4] 2> 872363 T20 oazs.NIOServerCnxn.closeSock Closed socket
connection for client /127.0.0.1:52224 which had sessionid 0x13a9d32d6480002
[junit4:junit4] 2> 872363 T40 oaz.ClientCnxn$SendThread.run Unable to read
additional data from server sessionid 0x13a9d32d6480002, likely server has
closed socket, closing socket connection and attempting reconnect
[junit4:junit4] 2> 872363 T20 oazs.NIOServerCnxn.closeSock Closed socket
connection for client /127.0.0.1:52266 which had sessionid 0x13a9d32d6480003
[junit4:junit4] 2> 872363 T22 oazs.NIOServerCnxn$Factory.run NIOServerCnxn
factory exited run method
[junit4:junit4] 2> 872363 T54 oaz.ClientCnxn$SendThread.run Unable to read
additional data from server sessionid 0x13a9d32d6480003, likely server has
closed socket, closing socket connection and attempting reconnect
[junit4:junit4] 2> 872363 T20 oazs.FinalRequestProcessor.shutdown shutdown of
request processor complete
[junit4:junit4] 2> 872363 T20 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
52219
[junit4:junit4] 2> 872363 T20 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=26799663
[junit4:junit4] 2> 872363 T20 oasc.SolrCore.close [collection1] CLOSING
SolrCore org.apache.solr.core.SolrCore@3ba4f1
[junit4:junit4] 2> 872363 T20 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=0,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 872363 T20 oasc.SolrCore.decrefSolrCoreState Closing
SolrCoreState
[junit4:junit4] 2> 872363 T20 oasu.DefaultSolrCoreState.closeIndexWriter
SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 872363 T20 oasu.DefaultSolrCoreState.closeIndexWriter
closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 872363 T20 oasc.SolrCore.closeSearcher [collection1]
Closing main searcher on request.
[junit4:junit4] 2> 872363 T42 oasc.Overseer$ClusterStateUpdater.amILeader
According to ZK I (id=88555984356114434-127.0.0.1:52219_solr-n_0000000000) am
no longer a leader.
[junit4:junit4] 2> 872472 T93 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@13d0fea name:ZooKeeperConnection
Watcher:127.0.0.1:52195/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
[junit4:junit4] 2> 872472 T55 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@138ec91 name:ZooKeeperConnection
Watcher:127.0.0.1:52195/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
[junit4:junit4] 2> 872472 T93 oascc.ConnectionManager.process zkClient has
disconnected
[junit4:junit4] 2> 872472 T55 oascc.ConnectionManager.process zkClient has
disconnected
[junit4:junit4] 2> 872472 T67 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@1e3c2c6 name:ZooKeeperConnection
Watcher:127.0.0.1:52195/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
[junit4:junit4] 2> 872472 T79 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@14c02d4 name:ZooKeeperConnection
Watcher:127.0.0.1:52195/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
[junit4:junit4] 2> 872472 T20 oaz.ZooKeeper.close Session: 0x13a9d32d6480002
closed
[junit4:junit4] 2> 872472 T41 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@17a906e name:ZooKeeperConnection
Watcher:127.0.0.1:52195/solr got event WatchedEvent state:Disconnected
type:None path:null path:null type:None
[junit4:junit4] 2> 872472 T41 oascc.ConnectionManager.process
Client->ZooKeeper status change trigger but we are already closed
[junit4:junit4] 2> 872472 T67 oascc.ConnectionManager.process zkClient has
disconnected
[junit4:junit4] 2> 872472 T79 oascc.ConnectionManager.process zkClient has
disconnected
[junit4:junit4] 2> 872472 T41 oaz.ClientCnxn$EventThread.run EventThread shut
down
[junit4:junit4] 2> 872473 T20 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 872526 T20 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
52256
[junit4:junit4] 2> 872526 T20 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=29120938
[junit4:junit4] 2> 872528 T20 oasc.SolrCore.close [collection1] CLOSING
SolrCore org.apache.solr.core.SolrCore@f6852d
[junit4:junit4] 2> 872532 T20 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=1,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 872533 T20 oasc.SolrCore.decrefSolrCoreState Closing
SolrCoreState
[junit4:junit4] 2> 872533 T20 oasu.DefaultSolrCoreState.closeIndexWriter
SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 872533 T20 oasu.DefaultSolrCoreState.closeIndexWriter
closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 872536 T20 oasc.SolrCore.closeSearcher [collection1]
Closing main searcher on request.
[junit4:junit4] 2> 873608 T92 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:52195
[junit4:junit4] 2> 873623 T66 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:52195
[junit4:junit4] 2> 874247 T78 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:52195
[junit4:junit4] 2> 874434 T54 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:52195
[junit4:junit4] 2> 874606 T66 oaz.ClientCnxn$SendThread.run WARNING Session
0x13a9d32d6480004 for server null, unexpected error, closing socket connection
and attempting reconnect java.net.ConnectException: Connection refused: no
further information
[junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native
Method)
[junit4:junit4] 2> at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
[junit4:junit4] 2> at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
[junit4:junit4] 2>
[junit4:junit4] 2> 874606 T92 oaz.ClientCnxn$SendThread.run WARNING Session
0x13a9d32d6480006 for server null, unexpected error, closing socket connection
and attempting reconnect java.net.ConnectException: Connection refused: no
further information
[junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native
Method)
[junit4:junit4] 2> at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
[junit4:junit4] 2> at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
[junit4:junit4] 2>
[junit4:junit4] 2> 875217 T78 oaz.ClientCnxn$SendThread.run WARNING Session
0x13a9d32d6480005 for server null, unexpected error, closing socket connection
and attempting reconnect java.net.ConnectException: Connection refused: no
further information
[junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native
Method)
[junit4:junit4] 2> at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
[junit4:junit4] 2> at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
[junit4:junit4] 2>
[junit4:junit4] 2> 875507 T55 oaz.ClientCnxn$EventThread.run EventThread shut
down
[junit4:junit4] 2> 875507 T20 oaz.ZooKeeper.close Session: 0x13a9d32d6480003
closed
[junit4:junit4] 2> 875517 T20 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 875568 T20 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
52285
[junit4:junit4] 2> 875568 T20 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=4055775
[junit4:junit4] 2> 876288 T92 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:52195
[junit4:junit4] 2> 876444 T66 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:52195
[junit4:junit4] 2> 876694 T78 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:52195
[junit4:junit4] 2> 877287 T92 oaz.ClientCnxn$SendThread.run WARNING Session
0x13a9d32d6480006 for server null, unexpected error, closing socket connection
and attempting reconnect java.net.ConnectException: Connection refused: no
further information
[junit4:junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native
Method)
[junit4:junit4] 2> at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
[junit4:junit4] 2> at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
[junit4:junit4] 2>
[junit4:junit4] 2> 877554 T67 oaz.ClientCnxn$EventThread.run EventThread shut
down
[junit4:junit4] 2> 877554 T20 oaz.ZooKeeper.close Session: 0x13a9d32d6480004
closed
[junit4:junit4] 2> 877569 T20 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 877620 T20 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
53072
[junit4:junit4] 2> 877620 T20 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=14194572
[junit4:junit4] 2> 877620 T20 oasc.SolrCore.close [collection1] CLOSING
SolrCore org.apache.solr.core.SolrCore@6745b9
[junit4:junit4] 2> 877652 T20 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=1,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 877652 T20 oasc.SolrCore.decrefSolrCoreState Closing
SolrCoreState
[junit4:junit4] 2> 877652 T20 oasu.DefaultSolrCoreState.closeIndexWriter
SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 877652 T20 oasu.DefaultSolrCoreState.closeIndexWriter
closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 877652 T20 oasc.SolrCore.closeSearcher [collection1]
Closing main searcher on request.
[junit4:junit4] 2> 877808 T79 oaz.ClientCnxn$EventThread.run EventThread shut
down
[junit4:junit4] 2> 877808 T20 oaz.ZooKeeper.close Session: 0x13a9d32d6480005
closed
[junit4:junit4] 2> 877823 T20 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 877874 T20 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
53079
[junit4:junit4] 2> 877874 T20 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=1056169
[junit4:junit4] 2> 877874 T20 oasc.SolrCore.close [collection1] CLOSING
SolrCore org.apache.solr.core.SolrCore@146ac5a
[junit4:junit4] 2> 877905 T20 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=0,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 877905 T20 oasc.SolrCore.decrefSolrCoreState Closing
SolrCoreState
[junit4:junit4] 2> 877905 T20 oasu.DefaultSolrCoreState.closeIndexWriter
SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 877905 T20 oasu.DefaultSolrCoreState.closeIndexWriter
closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 877905 T20 oasc.SolrCore.closeSearcher [collection1]
Closing main searcher on request.
[junit4:junit4] 2> 878436 T92 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server 127.0.0.1/127.0.0.1:52195
[junit4:junit4] 2> 879543 T93 oaz.ClientCnxn$EventThread.run EventThread shut
down
[junit4:junit4] 2> 879543 T20 oaz.ZooKeeper.close Session: 0x13a9d32d6480006
closed
[junit4:junit4] 2> 879559 T20 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 879813 T20 oas.SolrTestCaseJ4.tearDown ###Ending
testDistribSearch
[junit4:junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=BasicDistributedZkTest -Dtests.method=testDistribSearch
-Dtests.seed=5931ACB8E3018C7A -Dtests.slow=true -Dtests.locale=es_PR
-Dtests.timezone=Europe/Guernsey -Dtests.file.encoding=Cp1252
[junit4:junit4] FAILURE 880s J0 | BasicDistributedZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: There are still
nodes recoverying
[junit4:junit4] > at
__randomizedtesting.SeedInfo.seed([5931ACB8E3018C7A:D8D722A0945EEC46]:0)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
[junit4:junit4] > at
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:115)
[junit4:junit4] > at
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:110)
[junit4:junit4] > at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:553)
[junit4:junit4] > at
org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:162)
[junit4:junit4] > at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:695)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit4:junit4] > at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit4:junit4] > at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4] > at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> 879860 T20 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 1002554 T20 oas.SolrTestCaseJ4.endTrackingSearchers SEVERE
ERROR: SolrIndexSearcher opens=7 closes=6
[junit4:junit4] 2> 1002554 T19 ccr.ThreadLeakControl.checkThreadLeaks WARNING
Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> 1022709 T19 ccr.ThreadLeakControl.checkThreadLeaks SEVERE
1 thread leaked from SUITE scope at
org.apache.solr.cloud.BasicDistributedZkTest:
[junit4:junit4] 2> 1) Thread[id=68, name=searcherExecutor-40-thread-1,
state=WAITING, group=TGRP-BasicDistributedZkTest]
[junit4:junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] 2> at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit4:junit4] 2> at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4] 2> at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> 1022709 T19 ccr.ThreadLeakControl.tryToInterruptAll
Starting to interrupt leaked threads:
[junit4:junit4] 2> 1) Thread[id=68, name=searcherExecutor-40-thread-1,
state=WAITING, group=TGRP-BasicDistributedZkTest]
[junit4:junit4] 2> 1025798 T19 ccr.ThreadLeakControl.tryToInterruptAll SEVERE
There are still zombie threads that couldn't be terminated:
[junit4:junit4] 2> 1) Thread[id=68, name=searcherExecutor-40-thread-1,
state=WAITING, group=TGRP-BasicDistributedZkTest]
[junit4:junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] 2> at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit4:junit4] 2> at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4] 2> at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> NOTE: test params are:
codec=Compressing(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION,
chunkSize=28, storedFieldsIndex=MEMORY_CHUNK)),
sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=es_PR,
timezone=Europe/Guernsey
[junit4:junit4] 2> NOTE: Windows Vista 6.0 x86/Sun Microsystems Inc. 1.6.0_23
(32-bit)/cpus=4,threads=2,free=4449480,total=16449536
[junit4:junit4] 2> NOTE: All tests run in this JVM:
[PathHierarchyTokenizerFactoryTest, SimplePostToolTest,
CoreContainerCoreInitFailuresTest, BasicDistributedZkTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=BasicDistributedZkTest -Dtests.seed=5931ACB8E3018C7A
-Dtests.slow=true -Dtests.locale=es_PR -Dtests.timezone=Europe/Guernsey
-Dtests.file.encoding=Cp1252
[junit4:junit4] ERROR 0.00s J0 | BasicDistributedZkTest (suite) <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: ERROR:
SolrIndexSearcher opens=7 closes=6
[junit4:junit4] > at
__randomizedtesting.SeedInfo.seed([5931ACB8E3018C7A]:0)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at
org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:242)
[junit4:junit4] > at
org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:97)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit4:junit4] > at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit4:junit4] > at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4] > at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] > Throwable #2:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE
scope at org.apache.solr.cloud.BasicDistributedZkTest:
[junit4:junit4] > 1) Thread[id=68, name=searcherExecutor-40-thread-1,
state=WAITING, group=TGRP-BasicDistributedZkTest]
[junit4:junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] > at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit4:junit4] > at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4] > at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4] > at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4] > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] > at
__randomizedtesting.SeedInfo.seed([5931ACB8E3018C7A]:0)
[junit4:junit4] > Throwable #3:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie
threads that couldn't be terminated:
[junit4:junit4] > 1) Thread[id=68, name=searcherExecutor-40-thread-1,
state=WAITING, group=TGRP-BasicDistributedZkTest]
[junit4:junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] > at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
[junit4:junit4] > at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4] > at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4] > at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4] > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] > at
__randomizedtesting.SeedInfo.seed([5931ACB8E3018C7A]:0)
[junit4:junit4] Completed on J0 in 1025.86s, 1 test, 2 failures, 2 errors <<<
FAILURES!
[junit4:junit4]
[junit4:junit4]
[junit4:junit4] Tests with failures:
[junit4:junit4] -
org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch
[junit4:junit4] - org.apache.solr.cloud.BasicDistributedZkTest (suite)
[junit4:junit4]
[junit4:junit4]
[junit4:junit4] JVM J0: 2.59 .. 1061.41 = 1058.82s
[junit4:junit4] JVM J1: 2.36 .. 212.27 = 209.91s
[junit4:junit4] JVM J2: 2.59 .. 225.12 = 222.53s
[junit4:junit4] JVM J3: 2.59 .. 281.84 = 279.25s
[junit4:junit4] JVM J4: 2.59 .. 227.87 = 225.28s
[junit4:junit4] JVM J5: 2.59 .. 214.53 = 211.94s
[junit4:junit4] JVM J6: 2.59 .. 228.34 = 225.75s
[junit4:junit4] JVM J7: 2.59 .. 213.99 = 211.40s
[junit4:junit4] Execution time total: 17 minutes 41 seconds
[junit4:junit4] Tests summary: 233 suites, 965 tests, 3 suite-level errors, 1
failure, 12 ignored (7 assumptions)
BUILD FAILED
<http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/build.xml>:39:
The following error occurred while executing this line:
<http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/solr/build.xml>:178:
The following error occurred while executing this line:
<http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/module-build.xml>:61:
The following error occurred while executing this line:
<http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/common-build.xml>:1171:
The following error occurred while executing this line:
<http://sierranevada.servebeer.com:8080/job/slow-io-beasting/ws/lucene/common-build.xml>:839:
There were test failures: 233 suites, 965 tests, 3 suite-level errors, 1
failure, 12 ignored (7 assumptions)
Total time: 24 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
