Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-master/47/
11 tests failed. FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test Error Message: Test abandoned because suite timeout was reached. Stack Trace: java.lang.Exception: Test abandoned because suite timeout was reached. at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest Error Message: Suite timeout exceeded (>= 7200000 msec). Stack Trace: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest Error Message: Captured an uncaught exception in thread: Thread[id=127710, name=Thread-11980, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=127710, name=Thread-11980, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest] Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:33776//control_collection at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6]:0) at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:638) Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:33776//control_collection at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:661) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:256) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224) at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:636) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:549) ... 5 more FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest Error Message: Captured an uncaught exception in thread: Thread[id=127711, name=Thread-11981, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=127711, name=Thread-11981, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest] Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:33776/collection2_shard7_replica_n36 at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6]:0) at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:638) Caused by: org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:33776/collection2_shard7_replica_n36 at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:567) at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1019) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224) at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:635) Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:33776/collection2_shard7_replica_n36 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:661) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:256) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245) at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368) at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:213) at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:561) ... 6 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:549) ... 12 more FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest Error Message: Captured an uncaught exception in thread: Thread[id=127044, name=OverseerStateUpdate-74080050190811140-127.0.0.1:33776_-n_0000000000, state=RUNNABLE, group=Overseer state updater.] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=127044, name=OverseerStateUpdate-74080050190811140-127.0.0.1:33776_-n_0000000000, state=RUNNABLE, group=Overseer state updater.] Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6]:0) at org.apache.zookeeper.ZooKeeper.prependChroot(ZooKeeper.java:709) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1211) at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) at org.apache.solr.common.cloud.SolrZkClient$$Lambda$36/1463209208.execute(Unknown Source) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:432) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:150) at java.lang.Thread.run(Thread.java:748) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest Error Message: Captured an uncaught exception in thread: Thread[id=127709, name=Thread-11979, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=127709, name=Thread-11979, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest] Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33776//control_collection: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 500 Server Error</title> </head> <body><h2>HTTP ERROR 500</h2> <p>Problem accessing /control_collection/update. Reason: <pre> Server Error</pre></p><h3>Caused by:</h3><pre>java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:452) at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:131) at org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:115) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:288) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:368) at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:113) at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172) at org.apache.lucene.index.BufferedUpdatesStream$SegmentState.<init>(BufferedUpdatesStream.java:271) at org.apache.lucene.index.FrozenBufferedUpdates.openSegmentStates(FrozenBufferedUpdates.java:329) at org.apache.lucene.index.FrozenBufferedUpdates.forceApply(FrozenBufferedUpdates.java:222) at org.apache.lucene.index.FrozenBufferedUpdates.tryApply(FrozenBufferedUpdates.java:160) at org.apache.lucene.index.IndexWriter.lambda$publishFrozenUpdates$3(IndexWriter.java:2589) at org.apache.lucene.index.IndexWriter$$Lambda$466/43019974.process(Unknown Source) at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5112) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1597) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586) at org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:970) at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:342) at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:289) at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:236) at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76) at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55) at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:995) at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1216) at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:700) at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103) at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:110) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:327) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:280) at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:333) at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:278) </pre> <hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.14.v20181114</a><hr/> </body> </html> at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6]:0) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:614) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:256) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224) at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:636) FAILED: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_stored Error Message: Some docs had errors -- check logs expected:<0> but was:<3> Stack Trace: java.lang.AssertionError: Some docs had errors -- check logs expected:<0> but was:<3> at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6:A3250ADE66ED130E]:0) at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:645) at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:345) at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_stored(TestStressCloudBlindAtomicUpdates.java:212) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.lang.Thread.run(Thread.java:748) FAILED: org.apache.solr.cloud.TriLevelCompositeIdRoutingTest.test Error Message: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /solr/collections/collection1/terms/shard2 Stack Trace: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /solr/collections/collection1/terms/shard2 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForActiveReplicaCount(AbstractFullDistribZkTestBase.java:601) at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:552) at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:358) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1080) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /solr/collections/collection1/terms/shard2 at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) at org.apache.solr.common.cloud.SolrZkClient.printLayout(SolrZkClient.java:615) at org.apache.solr.common.cloud.SolrZkClient.printLayout(SolrZkClient.java:640) at org.apache.solr.common.cloud.SolrZkClient.printLayout(SolrZkClient.java:640) at org.apache.solr.common.cloud.SolrZkClient.printLayout(SolrZkClient.java:640) at org.apache.solr.common.cloud.SolrZkClient.printLayout(SolrZkClient.java:640) at org.apache.solr.common.cloud.SolrZkClient.printLayout(SolrZkClient.java:640) at org.apache.solr.common.cloud.SolrZkClient.printLayoutToStream(SolrZkClient.java:660) at org.apache.solr.common.cloud.SolrZkClient.printLayoutToStdOut(SolrZkClient.java:655) at org.apache.solr.cloud.AbstractDistribZkTestBase.printLayout(AbstractDistribZkTestBase.java:315) at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForActiveReplicaCount(AbstractFullDistribZkTestBase.java:599) ... 35 more FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TriLevelCompositeIdRoutingTest Error Message: Suite timeout exceeded (>= 7200000 msec). Stack Trace: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6]:0) FAILED: org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload Error Message: Number of registered MBeans is not the same as the number of core metrics: 443 != 442 Stack Trace: java.lang.AssertionError: Number of registered MBeans is not the same as the number of core metrics: 443 != 442 at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6:33A0B036A9E9F21B]:0) at org.junit.Assert.fail(Assert.java:88) at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:260) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.lang.Thread.run(Thread.java:748) FAILED: junit.framework.TestSuite.org.apache.solr.metrics.reporters.SolrJmxReporterCloudTest Error Message: Clean up static fields (in @AfterClass?), your test seems to hang on to approximately 11,572,176 bytes (threshold is 10,485,760). Field reference sizes (counted individually): - 11,571,456 bytes, private static javax.management.MBeanServer org.apache.solr.metrics.reporters.SolrJmxReporterCloudTest.mBeanServer - 192 bytes, public static org.junit.rules.TestRule org.apache.solr.SolrTestCaseJ4.solrClassRules - 128 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.factoryProp - 112 bytes, private static java.lang.String org.apache.solr.metrics.reporters.SolrJmxReporterCloudTest.COLLECTION - 96 bytes, private static java.util.Map org.apache.solr.SolrTestCaseJ4.savedClassLogLevels - 80 bytes, protected static volatile java.util.concurrent.ExecutorService org.apache.solr.SolrTestCaseJ4.testExecutor - 64 bytes, protected static java.lang.String org.apache.solr.SolrTestCaseJ4.coreName - 48 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.initialRootLogLevel Stack Trace: junit.framework.AssertionFailedError: Clean up static fields (in @AfterClass?), your test seems to hang on to approximately 11,572,176 bytes (threshold is 10,485,760). Field reference sizes (counted individually): - 11,571,456 bytes, private static javax.management.MBeanServer org.apache.solr.metrics.reporters.SolrJmxReporterCloudTest.mBeanServer - 192 bytes, public static org.junit.rules.TestRule org.apache.solr.SolrTestCaseJ4.solrClassRules - 128 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.factoryProp - 112 bytes, private static java.lang.String org.apache.solr.metrics.reporters.SolrJmxReporterCloudTest.COLLECTION - 96 bytes, private static java.util.Map org.apache.solr.SolrTestCaseJ4.savedClassLogLevels - 80 bytes, protected static volatile java.util.concurrent.ExecutorService org.apache.solr.SolrTestCaseJ4.testExecutor - 64 bytes, protected static java.lang.String org.apache.solr.SolrTestCaseJ4.coreName - 48 bytes, private static java.lang.String org.apache.solr.SolrTestCaseJ4.initialRootLogLevel at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6]:0) at com.carrotsearch.randomizedtesting.rules.StaticFieldsInvariantRule$1.afterAlways(StaticFieldsInvariantRule.java:170) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.lang.Thread.run(Thread.java:748) Build Log: [...truncated 13889 lines...] [junit4] Suite: org.apache.solr.core.TestJmxIntegration [junit4] 2> 1301396 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom [junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestJmxIntegration_31196B5ADAAA2ED6-001/init-core-data-001 [junit4] 2> 1301397 WARN (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6 [junit4] 2> 1301397 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true [junit4] 2> 1301399 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN) [junit4] 2> 1301400 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore [junit4] 2> 1301401 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes] [junit4] 2> 1301435 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0 [junit4] 2> 1301478 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.s.IndexSchema [null] Schema name=test [junit4] 2> 1301673 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id [junit4] 2> 1301950 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true] [junit4] 2> 1301951 WARN (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.c.s.i.Http2SolrClient Create Http2SolrClient with HTTP/1.1 transport since Java 8 or lower versions does not support SSL + HTTP/2 [junit4] 2> 1301958 WARN (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@3a212e77[provider=null,keyStore=null,trustStore=null] [junit4] 2> 1302030 WARN (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.c.s.i.Http2SolrClient Create Http2SolrClient with HTTP/1.1 transport since Java 8 or lower versions does not support SSL + HTTP/2 [junit4] 2> 1302061 WARN (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@1fc8b9e[provider=null,keyStore=null,trustStore=null] [junit4] 2> 1302106 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores [junit4] 2> 1302106 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory. [junit4] 2> 1302142 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@617d483 [junit4] 2> 1302148 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@617d483 [junit4] 2> 1302148 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@617d483 [junit4] 2> 1302150 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes] [junit4] 2> 1302189 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0 [junit4] 2> 1302221 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test [junit4] 2> 1302415 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id [junit4] 2> 1302423 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1, trusted=true [junit4] 2> 1302423 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@617d483 [junit4] 2> 1302424 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestJmxIntegration_31196B5ADAAA2ED6-001/init-core-data-001/] [junit4] 2> 1302436 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=22.958984375, floorSegmentMB=1.3056640625, forceMergeDeletesPctAllowed=10.007169334770646, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.30752591381726735, deletesPctAllowed=32.41259637393401 [junit4] 2> 1302509 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1302509 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 1302511 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 1302512 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 1302526 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=91.642578125, floorSegmentMB=0.4658203125, forceMergeDeletesPctAllowed=21.025216847150237, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0, deletesPctAllowed=26.783711149399203 [junit4] 2> 1302526 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6cb9ab20[collection1] main] [junit4] 2> 1302527 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers [junit4] 2> 1302536 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3} [junit4] 2> 1302635 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms. [junit4] 2> 1302635 INFO (coreLoadExecutor-2557-thread-1) [ x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1623739922262261760 [junit4] 2> 1302651 INFO (searcherExecutor-2558-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default [junit4] 2> 1302654 INFO (SUITE-TestJmxIntegration-seed#[31196B5ADAAA2ED6]-worker) [ ] o.a.s.SolrTestCaseJ4 ####initCore end [junit4] 2> 1302656 INFO (searcherExecutor-2558-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct [junit4] 2> 1302656 INFO (searcherExecutor-2558-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak [junit4] 2> 1302656 INFO (searcherExecutor-2558-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields [junit4] 2> 1302656 INFO (searcherExecutor-2558-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler [junit4] 2> 1302657 INFO (searcherExecutor-2558-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external [junit4] 2> 1302657 INFO (searcherExecutor-2558-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq [junit4] 2> 1302658 INFO (searcherExecutor-2558-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn [junit4] 2> 1302658 INFO (searcherExecutor-2558-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict [junit4] 2> 1302666 INFO (searcherExecutor-2558-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6cb9ab20[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1302677 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testJmxOnCoreReload [junit4] 2> 1302677 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX [junit4] 2> 1302678 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@6e96300f[collection1] realtime] [junit4] 2> 1302679 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 1 [junit4] 2> 1302679 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1623739922308399104,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} [junit4] 2> 1302679 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2e0133a5 commitCommandVersion:1623739922308399104 [junit4] 2> 1302689 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2f4475ab[collection1] main] [junit4] 2> 1302691 INFO (searcherExecutor-2558-thread-1) [ ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2f4475ab[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1302691 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush [junit4] 2> 1302691 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=null path=null params={}{commit=} 0 12 [junit4] 2> 1302694 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=*:*&wt=xml} hits=0 status=0 QTime=0 [junit4] 2> 1302815 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.c.TestJmxIntegration Before Reload: size of all core metrics: 443 MBeans: 443 [junit4] 2> 1302815 ERROR (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.c.CorePropertiesLocator Couldn't load core descriptor from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/core.properties:java.nio.file.NoSuchFileException: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/core.properties [junit4] 2> 1302815 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes] [junit4] 2> 1302880 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0 [junit4] 2> 1302917 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.s.IndexSchema [collection1] Schema name=test [junit4] 2> 1303162 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id [junit4] 2> 1303187 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1 [junit4] 2> 1303287 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@617d483 [junit4] 2> 1303288 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.core.TestJmxIntegration_31196B5ADAAA2ED6-001/init-core-data-001/] [junit4] 2> 1303424 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 1303424 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 1303426 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5d72cff9[collection1] main] [junit4] 2> 1303426 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers [junit4] 2> 1303427 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3} [junit4] 2> 1303430 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms. [junit4] 2> 1303430 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1623739923095879680 [junit4] 2> 1303433 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default [junit4] 2> 1303433 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct [junit4] 2> 1303433 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak [junit4] 2> 1303433 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields [junit4] 2> 1303437 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler [junit4] 2> 1303437 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external [junit4] 2> 1303437 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq [junit4] 2> 1303437 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn [junit4] 2> 1303438 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict [junit4] 2> 1303441 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5d72cff9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1303442 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=403657774, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1] [junit4] 2> 1303442 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used. [junit4] 2> 1303443 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5cea7a4c[collection1] main] [junit4] 2> 1303453 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@2c2ecaf2 [junit4] 2> 1303453 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=2c2ecaf2 [junit4] 2> 1303453 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@375fd3d4: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@25b9fec6 [junit4] 2> 1303460 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5cea7a4c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1303468 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=*:*&wt=xml} hits=0 status=0 QTime=2 [junit4] 2> 1303548 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.c.TestJmxIntegration After Reload: size of all core metrics: 443 MBeans: 442 [junit4] 2> 1303548 ERROR (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.c.TestJmxIntegration Unexpected bean names: [] [junit4] 2> 1303548 ERROR (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.c.TestJmxIntegration Unexpected metric names: [SEARCHER.searcher.maxDoc] [junit4] 2> 1303549 INFO (TEST-TestJmxIntegration.testJmxOnCoreReload-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testJmxOnCoreReload [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestJmxIntegration -Dtests.method=testJmxOnCoreReload -Dtests.seed=31196B5ADAAA2ED6 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=en-PH -Dtests.timezone=Etc/Greenwich -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 [junit4] FAILURE 0.92s J0 | TestJmxIntegration.testJmxOnCoreReload <<< [junit4] > Throwable #1: java.lang.AssertionError: Number of registered MBeans is not the same as the number of core metrics: 443 != 442 [junit4] > at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6:33A0B036A9E9F21B]:0) [junit4] > at org.apache.solr.core.TestJmxIntegration.testJmxOnCoreReload(TestJmxIntegration.java:260) [junit4] > at java.lang.Thread.run(Thread.java:748) [junit4] 2> 1303582 INFO (TEST-TestJmxIntegration.testJmxUpdate-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.SolrTestCaseJ4 ###Starting testJmxUpdate [junit4] 2> 1303582 INFO (TEST-TestJmxIntegration.testJmxUpdate-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX [junit4] 2> 1303582 INFO (TEST-TestJmxIntegration.testJmxUpdate-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6c728831[collection1] realtime] [junit4] 2> 1303583 INFO (TEST-TestJmxIntegration.testJmxUpdate-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true}{deleteByQuery=*:* (-9223372036854775807)} 0 0 [junit4] 2> 1303583 INFO (TEST-TestJmxIntegration.testJmxUpdate-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1623739923256311808,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} [junit4] 2> 1303583 INFO (TEST-TestJmxIntegration.testJmxUpdate-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@5c1b1b0 commitCommandVersion:1623739923256311808 [junit4] 2> 1303583 INFO (TEST-TestJmxIntegration.testJmxUpdate-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@10c4ff6c[collection1] main] [junit4] 2> 1303583 INFO (TEST-TestJmxIntegration.testJmxUpdate-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush [junit4] 2> 1303585 INFO (searcherExecutor-2563-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@10c4ff6c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1303585 INFO (TEST-TestJmxIntegration.testJmxUpdate-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=null path=null params={}{commit=} 0 2 [junit4] 2> 1303609 INFO (TEST-TestJmxIntegration.testJmxUpdate-seed#[31196B5ADAAA2ED6]) [ x:collection1] o.a.s.c.TestJmxIntegration Mbeans in server: [solr_34558:dom1=solr,dom2=jvm,name0=system,name=properties, org.apache.logging.log4j2:type=2a139a55,component=StatusLogger, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=handlerStart, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Survivor-Space,name=max, java.lang:type=Threading, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=requests, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=errors, solr:dom1=jvm,name0=os,name=systemLoadAverage, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=requestTimes, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=requestTimes, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=shardRequests, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=threadPool.httpShardExecutor.duration, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=handlerStart, solr:dom1=node,category=QUERY,scope=/admin/metrics/history,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=handlerStart, solr:dom1=jvm,name0=buffers,name1=mapped,name=TotalCapacity, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=totalTime, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=rollbacks, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.recoveryExecutor.submitted, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=requestTimes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=clientErrors, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreContainerWorkExecutor.running, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=requestTimes, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=reader, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragmenter,name=gap.requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=requestTimes, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrBoundaryScanner,name=breakIterator.requests, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=shardRequests, org.apache.logging.log4j2:type=2a139a55,component=Loggers,name=org.eclipse.jetty, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=errors, solr:dom1=node,category=QUERY,scope=/admin/metrics/history,name=shardRequests, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Survivor-Space,name=usage, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=max, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=clientErrors, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=init, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/health,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=shardRequests, solr:dom1=core,dom2=collection1,category=TLOG,name=state, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=spins, solr:dom1=jvm,name0=buffers,name1=direct,name=MemoryUsed, solr:dom1=jvm,name0=classes,name=unloaded, solr:dom1=jvm,name0=threads,name1=waiting,name=count, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=errors, solr_44020:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=threadPool.parallelCoreAdminExecutor.submitted, solr:dom1=core,dom2=collection1,category=TLOG,scope=replay,name=remaining.bytes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=timeouts, solr:dom1=node,category=CONTAINER,scope=cores,name=loaded, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=handlerStart, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=clientErrors, solr_35452:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=totalTime, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=new,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=handlerStart, solr:dom1=node,category=QUERY,scope=httpShardHandler,name=threadPool.httpShardExecutor.completed, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=clientErrors, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=serverErrors, java.lang:type=Runtime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=cumulativeDeletesById, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=cumulativeDeletesByQuery, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=errors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.recoveryExecutor.duration, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=timeouts, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=totalSpace, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=handlerStart, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreContainerWorkExecutor.duration, solr_34426:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=clientErrors, solr_41390:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.updateOnlyExecutor.running, solr:dom1=node,category=CONTAINER,scope=cores,name=unloaded, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=serverErrors, java.lang:type=MemoryPool,name=PS Eden Space, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=timeouts, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=usableSpace, solr:dom1=jvm,name0=os,name=totalPhysicalMemorySize, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=serverErrors, solr:dom1=jvm,name0=os,name=totalSwapSpaceSize, solr:dom1=jvm,name0=memory,name1=pools,name2=Code-Cache,name=init, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=cumulativeErrors, java.lang:type=MemoryPool,name=PS Survivor Space, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=handlerStart, solr:dom1=jvm,name0=os,name=availableProcessors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.recoveryExecutor.running, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=errors, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=used, solr:dom1=node,category=CONTAINER,scope=fs,name=path, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=timeouts, solr:dom1=jvm,name0=threads,name=count, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=timeouts, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=caching, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper/status,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=serverErrors, solr:dom1=jvm,name0=memory,name1=total,name=used, solr:dom1=jvm,name0=memory,name1=pools,name2=Code-Cache,name=usage, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFormatter,name=html.requests, solr:dom1=node,category=CONTAINER,scope=fs,name=spins, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=deletesByQuery, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=serverErrors, solr:dom1=jvm,name0=threads,name1=new,name=count, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=handlerStart, solr:dom1=core,dom2=collection1,category=TLOG,scope=replay,name=ops, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=timeouts, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=committed, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=totalTime, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=openedAt, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=clientErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=requestTimes, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=initCount, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper/status,name=totalTime, solr:dom1=node,category=QUERY,scope=/admin/metrics/history,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=clientErrors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.updateOnlyExecutor.duration, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=requests, org.apache.logging.log4j2:type=2a139a55,component=Loggers,name=org.apache.directory, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=requests, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=softAutoCommits, solr:dom1=jvm,name0=memory,name1=non-heap,name=used, org.apache.logging.log4j2:type=2a139a55,component=Appenders,name=STDERR, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper/status,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=clientErrors, java.lang:type=MemoryPool,name=Code Cache, solr:dom1=core,dom2=collection1,category=CORE,name=refCount, solr_44904:dom1=solr,dom2=jvm,name0=system,name=properties, java.nio:type=BufferPool,name=direct, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=errors, solr:dom1=jvm,name0=threads,name1=daemon,name=count, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=serverErrors, solr:dom1=node,category=CONTAINER,scope=fs,name=coreRoot.path, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=errors, solr:dom1=jvm,name0=memory,name1=non-heap,name=init, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=requestTimes, solr:dom1=core,dom2=collection1,category=CACHE,scope=searcher,name=queryResultCache, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=handlerStart, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=searcherName, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/dismax,name=requests, solr:dom1=jvm,name0=memory,name1=heap,name=max, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=shardRequests, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreLoadExecutor.duration, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=shardRequests, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=requestTimes, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=timeouts, solr:dom1=core,dom2=c [junit4] 2> ollection1,category=REPLICATION,scope=/replication,name=indexSize, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/ping,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=shardRequests, solr_46471:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=jvm,name0=buffers,name1=mapped,name=Count, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=requestTimes, solr:dom1=core,dom2=collection1,category=QUERY,scope=/defaults,name=errors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/file,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-invariants,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=serverErrors, java.lang:type=Compilation, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=committed, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=requests, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=serverErrors, solr:dom1=core,dom2=collection1,category=TLOG,scope=buffered,name=ops, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=timeouts, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=adds, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=errors, solr_45662:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=jvm,name0=os,name=version, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=serverErrors, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=pendingConnections, solr:dom1=jvm,name0=memory,name1=pools,name2=Metaspace,name=usage, solr:dom1=core,dom2=collection1,category=CACHE,scope=searcher,name=documentCache, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=shardRequests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=totalTime, solr_43294:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=errors, solr_39351:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak_Direct,name=handlerStart, java.lang:type=MemoryPool,name=Compressed Class Space, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=requestTimes, solr_40605:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=errors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=requests, solr:dom1=jvm,name0=os,name=committedVirtualMemorySize, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=errors, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=usage, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=requests, solr:dom1=jvm,name0=memory,name1=non-heap,name=max, solr:dom1=jvm,name0=os,name=maxFileDescriptorCount, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragmentsBuilder,name=simple.requests, solr:dom1=core,dom2=collection1,category=CORE,name=coreName, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreLoadExecutor.running, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/health,name=handlerStart, solr:dom1=jvm,name0=memory,name1=heap,name=used, solr:dom1=core,dom2=collection1,category=CACHE,scope=searcher,name=perSegFilter, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=errors, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=confFilesToReplicate, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=requests, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=totalTime, solr:dom1=node,category=CONTAINER,scope=fs,name=coreRoot.usableSpace, solr:dom1=jvm,name0=os,name=systemCpuLoad, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragListBuilder,name=default.requests, solr:dom1=node,category=ADMIN,scope=/admin/info,name=shardRequests, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=splits, solr:dom1=node,category=UPDATE,scope=updateShardHandler,name=threadPool.updateOnlyExecutor.completed, solr:dom1=core,dom2=collection1,category=QUERY,scope=/terms,name=handlerStart, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=fetcher, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH_Direct,name=requestTimes, solr:dom1=jvm,name0=threads,name1=blocked,name=count, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=requestTimes, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=shardRequests, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=totalTime, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrBoundaryScanner,name=default.requests, solr:dom1=jvm,name0=gc,name1=PS-MarkSweep,name=time, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=totalTime, solr:dom1=core,dom2=collection1,category=TLOG,scope=applyingBuffered,name=ops, solr_45632:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=node,category=ADMIN,scope=/admin/info,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=totalTime, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/health,name=requestTimes, solr:dom1=jvm,name0=buffers,name1=direct,name=TotalCapacity, solr:dom1=node,category=ADMIN,scope=/admin/autoscaling/history,name=serverErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=clientErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=update,name=handlerStart, solr_35997:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=generation, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/health,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=errors, java.lang:type=OperatingSystem, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=handlerStart, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=timeouts, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=clientErrors, solr:dom1=jvm,name0=threads,name1=terminated,name=count, java.lang:type=MemoryManager,name=Metaspace Manager, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=handlerStart, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=shardRequests, solr:dom1=node,category=QUERY,scope=/admin/metrics/collector,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/metrics,name=totalTime, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=errors, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper,name=handlerStart, solr_36079:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=TLOG,scope=replay,name=remaining.logs, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=requestTimes, solr:dom1=core,dom2=collection1,category=CORE,scope=fs,name=path, solr:dom1=node,category=CONTAINER,scope=threadPool,name=coreContainerWorkExecutor.completed, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/properties,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=serverErrors, solr_46395:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=totalTime, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=threadPool.parallelCoreAdminExecutor.duration, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/luke,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH,name=totalTime, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json/docs,name=totalTime, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Eden-Space,name=max, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=used, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckWithWordbreak,name=serverErrors, solr:dom1=core,dom2=collection1,category=INDEX,name=sizeInBytes, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper/status,name=shardRequests, solr:dom1=jvm,name0=memory,name1=pools,name2=Metaspace,name=max, solr:dom1=core,dom2=collection1,category=QUERY,scope=/tvrh,name=shardRequests, solr:dom1=node,category=ADMIN,scope=/admin/info,name=requestTimes, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=commits, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=isSlave, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=requests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/sql,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/schema,name=serverErrors, solr:dom1=core,dom2=collection1,category=UPDATE,scope=updateHandler,name=cumulativeAdds, solr:dom1=node,category=ADMIN,scope=/admin/authorization,name=clientErrors, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/segments,name=shardRequests, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mlt,name=serverErrors, solr:dom1=core,dom2=collection1,category=QUERY,scope=/select,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/mbeans,name=timeouts, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/logging,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mltrh,name=clientErrors, solr:dom1=jvm,name0=os,name=freeSwapSpaceSize, solr:dom1=core,dom2=collection1,category=QUERY,scope=/get,name=totalTime, java.lang:type=MemoryManager,name=CodeCacheManager, solr:dom1=core,dom2=collection1,category=QUERY,scope=/stream,name=requests, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=indexPath, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/threads,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=requestTimes, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Eden-Space,name=committed, solr:dom1=core,dom2=collection1,category=QUERY,scope=/export,name=requests, solr:dom1=node,category=ADMIN,scope=/admin/configs,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/debug/dump,name=totalTime, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=requestTimes, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Old-Gen,name=used-after-gc, solr:dom1=node,category=ADMIN,scope=/admin/collections,name=timeouts, java.lang:type=GarbageCollector,name=PS Scavenge, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/system,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/graph,name=totalTime, solr:dom1=node,category=ADMIN,scope=/admin/info,name=clientErrors, solr:dom1=jvm,name0=memory,name1=heap,name=usage, solr:dom1=core,dom2=collection1,category=INDEX,name=size, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=requestTimes, java.nio:type=BufferPool,name=mapped, solr:dom1=jvm,name0=memory,name1=pools,name2=PS-Eden-Space,name=used-after-gc, com.sun.management:type=DiagnosticCommand, solr:dom1=core,dom2=collection1,category=REPLICATION,scope=/replication,name=replicationEnabled, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/config,name=serverErrors, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=new,name=maxReached, solr:dom1=node,category=ADMIN,scope=/admin/info,name=totalTime, solr:dom1=core,dom2=collection1,category=ADMIN,scope=/admin/plugins,name=clientErrors, solr:dom1=node,category=ADMIN,scope=/admin/cores,name=threadPool.parallelCoreAdminExecutor.running, org.apache.logging.log4j2:type=2a139a55,component=Loggers,name=org.apache.hadoop, solr:dom1=node,category=ADMIN,scope=/admin/zookeeper/status,name=requests, solr:dom1=jvm,name0=os,name=arch, solr_33523:dom1=solr,dom2=jvm,name0=system,name=properties, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/csv,name=serverErrors, solr:dom1=node,category=QUERY,scope=/admin/metrics/history,name=timeouts, solr:dom1=core,dom2=collection1,category=QUERY,scope=/mock,name=shardRequests, solr:dom1=node,category=QUERY,scope=/admin/metrics/history,name=handlerStart, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=requests, solr:dom1=jvm,name0=memory,name1=pools,name2=Compressed-Class-Space,name=init, solr:dom1=jvm,name0=os,name=name, solr:dom1=core,dom2=collection1,category=QUERY,scope=/search-facet-def,name=serverErrors, solr:dom1=core,dom2=collection1,category=SEARCHER,scope=searcher,name=maxDoc, solr:dom1=jvm,name0=os,name=processCpuLoad, solr:dom1=core,dom2=collection1,category=UPDATE,scope=/update/json,name=errors, solr:dom1=jvm,name0=memory,name1=pools,name2=Metaspace,name=committed, solr:dom1=core,dom2=collection1,category=HIGHLIGHTER,scope=SolrFragmentsBuilder,name=scoreOrder.requests, solr:dom1=node,category=CONTAINER,scope=fs,name=usableSpace, solr:dom1=core,dom2=collection1,category=QUERY,scope=/spellCheckCompRH1,name=serverErrors, solr:dom1=node,category=A [...truncated too long message...] .nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) [junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) [junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) [junit4] 2> at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) [junit4] 2> at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) [junit4] 2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) [junit4] 2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) [junit4] 2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [junit4] 2> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) [junit4] 2> at org.eclipse.jetty.io.ManagedSelector$$Lambda$41/621753608.run(Unknown Source) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$10/916963197.run(Unknown Source) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=2021, maxMBSortInHeap=5.0220498055603935, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5acf4961), locale=nl-BE, timezone=Europe/Ulyanovsk [junit4] 2> NOTE: Linux 4.4.0-112-generic amd64/Oracle Corporation 1.8.0_191 (64-bit)/cpus=4,threads=565,free=51552968,total=486539264 [junit4] 2> NOTE: All tests run in this JVM: [TestRequestStatusCollectionAPI, LeaderFailoverAfterPartitionTest, URLClassifyProcessorTest, PeerSyncWithLeaderTest, ConvertedLegacyTest, MoveReplicaHDFSTest, DistributedFacetPivotLargeTest, ExternalFileFieldSortTest, TestConfigSetImmutable, TestStressInPlaceUpdates, TestManagedSchema, TestHashQParserPlugin, TestXIncludeConfig, SchemaApiFailureTest, LeaderElectionContextKeyTest, InfoHandlerTest, TestQueryTypes, PropertiesRequestHandlerTest, SpellCheckComponentTest, TestSolrDeletionPolicy2, ScheduledTriggerIntegrationTest, AssignBackwardCompatibilityTest, HdfsUnloadDistributedZkTest, TestDistributedGrouping, LeaderVoteWaitTimeoutTest, TestLegacyFieldCache, TestFacetMethods, TestFieldCache, SearchRateTriggerTest, TestMiniSolrCloudClusterSSL, TestSSLRandomization, HdfsChaosMonkeyNothingIsSafeTest, TestReqParamsAPI, SortByFunctionTest, CdcrUpdateLogTest, TestSolrFieldCacheBean, BasicAuthIntegrationTest, TestCustomStream, TestLegacyNumericUtils, TestStressRecovery, CdcrRequestHandlerTest, TestWithCollection, AutoScalingHandlerTest, DistributedIntervalFacetingTest, RandomizedTaggerTest, OverriddenZkACLAndCredentialsProvidersTest, TestHighFrequencyDictionaryFactory, TransactionLogTest, ImplicitSnitchTest, ExplicitHLLTest, TestSimTriggerIntegration, IgnoreCommitOptimizeUpdateProcessorFactoryTest, ActionThrottleTest, TestAuthenticationFramework, RegexBytesRefFilterTest, TestCloudSearcherWarming, TestHdfsCloudBackupRestore, JWTAuthPluginTest, HdfsSyncSliceTest, TestRestManager, CreateRoutedAliasTest, TestRandomCollapseQParserPlugin, TestLuceneIndexBackCompat, TestCSVLoader, TestDynamicFieldCollectionResource, TestSolr4Spatial, XmlUpdateRequestHandlerTest, TestDeleteCollectionOnDownNodes, BigEndianAscendingWordDeserializerTest, VersionInfoTest, TestZkAclsWithHadoopAuth, HLLUtilTest, SignificantTermsQParserPluginTest, TestDownShardTolerantSearch, SolrMetricsIntegrationTest, CoreSorterTest, TestRemoteStreaming, TestReload, HealthCheckHandlerTest, OverseerTest, TestChildDocTransformerHierarchy, TestCloudJSONFacetJoinDomain, LargeFieldTest, DistributedQueryComponentCustomSortTest, TestMaxScoreQueryParser, TestCSVResponseWriter, TestSimpleTrackingShardHandler, TestJsonFacets, TestIndexingPerformance, TestFileDictionaryLookup, RecoveryZkTest, TestSafeXMLParsing, CoreAdminOperationTest, DateMathParserTest, TestNumericRangeQuery32, TestNestedDocsSort, ByteBuffersDirectoryFactoryTest, DistributedFacetPivotSmallTest, TestPartialUpdateDeduplication, TestLeaderElectionWithEmptyReplica, TestSchemaVersionResource, TestPushWriter, SimplePostToolTest, UpdateRequestProcessorFactoryTest, QueryResultKeyTest, BJQParserTest, MetricTriggerIntegrationTest, ConfigSetsAPITest, SearchHandlerTest, HdfsRecoverLeaseTest, SpatialRPTFieldTypeTest, SolrCoreTest, HdfsLockFactoryTest, JSONWriterTest, TriggerCooldownIntegrationTest, TestStressVersions, TestSizeLimitedDistributedMap, SolrRequestParserTest, CustomCollectionTest, ChangedSchemaMergeTest, TestPerFieldSimilarity, TestManagedStopFilterFactory, BadCopyFieldTest, NodeAddedTriggerTest, TestApiFramework, PreAnalyzedFieldTest, TestSchemalessBufferedUpdates, AssignTest, TemplateUpdateProcessorTest, HDFSCollectionsAPITest, TestSQLHandlerNonCloud, SolrIndexSplitterTest, TestTrie, TestMergePolicyConfig, SolrCoreMetricManagerTest, AnalysisAfterCoreReloadTest, BasicFunctionalityTest, CursorPagingTest, DisMaxRequestHandlerTest, OutputWriterTest, SolrTestCaseJ4Test, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestDocumentBuilder, TestJoin, TestSolrCoreProperties, TestTolerantSearch, TestCharFilters, TestReversedWildcardFilterFactory, TestWordDelimiterFilterFactory, TokenizerChainTest, TestEmbeddedSolrServerSchemaAPI, BasicDistributedZkTest, BasicZkTest, ChaosMonkeyNothingIsSafeTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, ChaosMonkeySafeLeaderTest, ChaosMonkeySafeLeaderWithPullReplicasTest, ConnectionManagerTest, DeleteNodeTest, DeleteShardTest, DeleteStatusTest, DistribJoinFromCollectionTest, DistributedQueueTest, DistributedVersionInfoTest, ForceLeaderTest, HttpPartitionOnCommitTest, HttpPartitionTest, LeaderFailureAfterFreshStartTest, LeaderTragicEventTest, LegacyCloudClusterPropTest, MetricsHistoryIntegrationTest, MigrateRouteKeyTest, MultiThreadedOCPTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerRolesTest, OverseerStatusTest, OverseerTaskQueueTest, RemoteQueryErrorTest, ReplaceNodeNoTargetTest, ReplaceNodeTest, ReplicationFactorTest, RestartWhileUpdatingTest, RollingRestartTest, ShardRoutingCustomTest, ShardRoutingTest, SharedFSAutoReplicaFailoverTest, SliceStateTest, SolrXmlInZkTest, TestCloudInspectUtil, TestCloudPseudoReturnFields, TestClusterProperties, TestConfigSetsAPI, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestCryptoKeys, TestDistribDocBasedVersion, TestDistributedMap, TestLeaderElectionZkExpiry, TestLockTree, TestOnReconnectListenerSupport, TestPrepRecovery, TestPullReplica, TestRandomRequestDistribution, TestRebalanceLeaders, TestRequestForwarding, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, TestStressCloudBlindAtomicUpdates, TestTolerantUpdateProcessorCloud, TestTolerantUpdateProcessorRandomCloud, TestUtilizeNode, TestZkChroot, TlogReplayBufferedWhileIndexingTest, TriLevelCompositeIdRoutingTest] [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TriLevelCompositeIdRoutingTest -Dtests.seed=31196B5ADAAA2ED6 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=nl-BE -Dtests.timezone=Europe/Ulyanovsk -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 [junit4] ERROR 0.00s J2 | TriLevelCompositeIdRoutingTest (suite) <<< [junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). [junit4] > at __randomizedtesting.SeedInfo.seed([31196B5ADAAA2ED6]:0) [junit4] Completed [842/842 (5!)] on J2 in 7233.32s, 1 test, 2 errors <<< FAILURES! [...truncated 1 lines...] [junit4] JVM J2: stdout was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20190126_154917_1152879976612986033288.sysout [junit4] >>> JVM J2 emitted unexpected output (verbatim) ---- [junit4] java.lang.OutOfMemoryError: Java heap space [junit4] Dumping heap to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/heapdumps/java_pid10040.hprof ... [junit4] Heap dump file created [791836890 bytes in 3.581 secs] [junit4] <<< JVM J2: EOF ---- [junit4] JVM J2: stderr was not empty, see: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20190126_154917_1151320486448861748079.syserr [junit4] >>> JVM J2 emitted unexpected output (verbatim) ---- [junit4] [junit4] Exception: java.lang.IllegalArgumentException thrown from the UncaughtExceptionHandler in thread "Scheduler-711239499" [junit4] <<< JVM J2: EOF ---- [...truncated 8928 lines...] BUILD FAILED /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/build.xml:661: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created: * java_pid10040.hprof * java_pid10047.hprof Total time: 747 minutes 19 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org