Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/2109/

2 tests failed.
FAILED:  
org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit

Error Message:
this IndexWriter is closed

Stack Trace:
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681)
        at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3392)
        at 
org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:228)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at 
org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:660)
        at org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
        at 
org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
        at 
org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:862)
        at 
org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:442)
        at 
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:406)
        at 
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:250)
        at 
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:495)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
        at 
org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:208)


FAILED:  org.apache.solr.cloud.RemoteQueryErrorTest.test

Error Message:
 Expected: a string containing "Document is missing mandatory uniqueKey field: 
id"      but: was "Error from server at http://127.0.0.1:34518/solr: No active 
replicas found for collection: collection"

Stack Trace:
java.lang.AssertionError: 
Expected: a string containing "Document is missing mandatory uniqueKey field: 
id"
     but: was "Error from server at http://127.0.0.1:34518/solr: No active 
replicas found for collection: collection"
        at 
__randomizedtesting.SeedInfo.seed([4E094E65EF364989:C65D71BF41CA2471]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at 
org.apache.solr.cloud.RemoteQueryErrorTest.test(RemoteQueryErrorTest.java:53)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)




Build Log:
[...truncated 842 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexingSequenceNumbers
   [junit4]   2> Feb 19, 2020 9:19:00 PM 
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
 uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: 
Thread[Thread-3343,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> 
   [junit4]   2> Feb 19, 2020 9:19:00 PM 
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
 uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: 
Thread[Thread-3345,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> 
   [junit4]   2> Feb 19, 2020 9:19:00 PM 
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
 uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: 
Thread[Thread-3344,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> 
   [junit4]   2> Feb 19, 2020 9:19:00 PM 
com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
 uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: 
Thread[Thread-3338,5,TGRP-TestIndexingSequenceNumbers]
   [junit4]   2> java.lang.AssertionError: pendingNumDocs 1837 != 1094 
totalMaxDoc
   [junit4]   2>        at 
__randomizedtesting.SeedInfo.seed([542FC45EC20059D4]:0)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2255)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:4894)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.tragicEvent(IndexWriter.java:4884)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1292)
   [junit4]   2>        at 
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1277)
   [junit4]   2>        at 
org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:206)
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestIndexingSequenceNumbers 
-Dtests.method=testStressConcurrentCommit -Dtests.seed=542FC45EC20059D4 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=en-FJ -Dtests.timezone=America/Coral_Harbour 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    668s J2 | 
TestIndexingSequenceNumbers.testStressConcurrentCommit <<<
   [junit4]    > Throwable #1: org.apache.lucene.store.AlreadyClosedException: 
this IndexWriter is closed
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3392)
   [junit4]    >        at 
org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:228)
   [junit4]    >        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    >        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    >        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    >        at 
java.base/java.lang.reflect.Method.invoke(Method.java:566)
   [junit4]    >        at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    >        at 
org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator.getIntBlock(DocumentsWriterPerThread.java:660)
   [junit4]    >        at 
org.apache.lucene.util.IntBlockPool.nextBuffer(IntBlockPool.java:155)
   [junit4]    >        at 
org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:159)
   [junit4]    >        at 
org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:862)
   [junit4]    >        at 
org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:442)
   [junit4]    >        at 
org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:406)
   [junit4]    >        at 
org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:250)
   [junit4]    >        at 
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:495)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
   [junit4]    >        at 
org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:208)Throwable
 #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an 
uncaught exception in thread: Thread[id=3430, name=Thread-3343, state=RUNNABLE, 
group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap 
spaceThrowable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: 
Captured an uncaught exception in thread: Thread[id=3431, name=Thread-3344, 
state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap 
spaceThrowable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: 
Captured an uncaught exception in thread: Thread[id=3432, name=Thread-3345, 
state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap 
spaceThrowable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: 
Captured an uncaught exception in thread: Thread[id=3425, name=Thread-3338, 
state=RUNNABLE, group=TGRP-TestIndexingSequenceNumbers]
   [junit4]    > Caused by: java.lang.AssertionError: pendingNumDocs 1837 != 
1094 totalMaxDoc
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([542FC45EC20059D4]:0)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2255)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.maybeCloseOnTragicEvent(IndexWriter.java:4894)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.tragicEvent(IndexWriter.java:4884)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1292)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1277)
   [junit4]    >        at 
org.apache.lucene.index.TestIndexingSequenceNumbers$2.run(TestIndexingSequenceNumbers.java:206)
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexingSequenceNumbers_542FC45EC20059D4-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene84): 
{id=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, 
docValues:{thread=DocValuesFormat(name=Lucene80), 
id=DocValuesFormat(name=Asserting), 
___soft_deletes=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=1417, 
maxMBSortInHeap=5.778054691683031, 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5b2e96e7),
 locale=en-FJ, timezone=America/Coral_Harbour
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 
(64-bit)/cpus=8,threads=1,free=40220120,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestDistributionLL, 
TestLatLonLineShapeQueries, TestFeatureSort, TestRateLimiter, 
TestAllFilesCheckIndexHeader, TestIntroSelector, TestDocIdSetIterator, 
TestCharArrayMap, TestAutomatonQueryUnicode, TestIndexWriterExceptions, 
TestMultiLevelSkipList, TestNot, TestStressIndexing2, TestDocIdSetBuilder, 
TestXYMultiPointShapeQueries, TestCompetitiveFreqNormAccumulator, 
TestMatchAllDocsQuery, TestBooleanOr, TestPoint2D, TestStringMSBRadixSorter, 
TestNewestSegment, TestElevationComparator, TestSpanTermQuery, 
TestIndexWriterOutOfFileDescriptors, TestDocsAndPositions, TestTryDelete, 
TestTermdocPerf, TestLongPostings, TestQueryRescorer, TestMaxTermFrequency, 
Test2BTerms, TestBytesRefArray, TestBoolean2, Test2BBinaryDocValues, 
TestFixedBitSet, TestTermsEnum2, TestIndexingSequenceNumbers]
   [junit4] Completed [172/560 (1!)] on J2 in 892.84s, 8 tests, 1 error <<< 
FAILURES!

[...truncated 1495 lines...]
   [junit4] JVM J2: stdout was not empty, see: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20200220_014607_80815275572708860027501.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid1913.hprof
 ...
   [junit4] Heap dump file created [629315008 bytes in 33.845 secs]
   [junit4] <<< JVM J2: EOF ----

   [junit4] JVM J2: stderr was not empty, see: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J2-20200220_014607_80813675417066397877257.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] 
   [junit4] Exception: java.lang.IllegalArgumentException thrown from the 
UncaughtExceptionHandler in thread "Thread-3339"
   [junit4] <<< JVM J2: EOF ----

[...truncated 12413 lines...]
   [junit4] Suite: org.apache.solr.cloud.RemoteQueryErrorTest
   [junit4]   2> 5896951 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to 
test-framework derived value of 
'/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/server/solr/configsets/_default/conf'
   [junit4]   2> 5896952 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.SolrTestCaseJ4 Created dataDir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.RemoteQueryErrorTest_4E094E65EF364989-001/data-dir-210-001
   [junit4]   2> 5896952 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 5896969 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, 
clientAuth=0.0/0.0)
   [junit4]   2> 5896969 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 5896970 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.RemoteQueryErrorTest_4E094E65EF364989-001/tempDir-001
   [junit4]   2> 5896970 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 5896971 INFO  (ZkTestServer Run Thread) [     ] 
o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 5896971 INFO  (ZkTestServer Run Thread) [     ] 
o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 5897078 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.ZkTestServer start zk server on port:43070
   [junit4]   2> 5897078 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.ZkTestServer waitForServerUp: 127.0.0.1:43070
   [junit4]   2> 5897078 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43070
   [junit4]   2> 5897078 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1 43070
   [junit4]   2> 5897080 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5897081 INFO  (zkConnectionManagerCallback-20719-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5897082 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5897084 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5897085 INFO  (zkConnectionManagerCallback-20721-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5897086 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5897086 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5897087 INFO  (zkConnectionManagerCallback-20723-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5897087 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5897191 WARN  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, 
break even is size 23
   [junit4]   2> 5897191 WARN  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, 
break even is size 23
   [junit4]   2> 5897191 WARN  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, 
break even is size 23
   [junit4]   2> 5897191 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 5897191 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 5897191 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 5897191 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 5897191 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 5897191 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 5897191 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 
363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 5897191 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 
363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 5897191 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 
363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 5897193 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 5897193 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 5897193 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 5897194 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@c49d1de{/solr,null,AVAILABLE}
   [junit4]   2> 5897194 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 5897194 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 5897194 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 5897195 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@7681c073{/solr,null,AVAILABLE}
   [junit4]   2> 5897195 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 5897195 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 5897195 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 5897195 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.s.AbstractConnector Started ServerConnector@132aa614{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:44398}
   [junit4]   2> 5897195 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.s.Server Started @5897234ms
   [junit4]   2> 5897195 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=44398}
   [junit4]   2> 5897195 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@754185a1{/solr,null,AVAILABLE}
   [junit4]   2> 5897196 ERROR (jetty-launcher-20724-thread-3) [     ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.s.AbstractConnector Started ServerConnector@8e71959{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:34518}
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.s.Server Started @5897234ms
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=34518}
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
9.0.0
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2020-02-20T07:44:05.198185Z
   [junit4]   2> 5897196 ERROR (jetty-launcher-20724-thread-2) [     ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.s.AbstractConnector Started ServerConnector@558413a6{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:33252}
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.s.Server Started @5897234ms
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
9.0.0
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=33252}
   [junit4]   2> 5897196 ERROR (jetty-launcher-20724-thread-1) [     ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
9.0.0
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr
   [junit4]   2> 5897196 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2020-02-20T07:44:05.198584Z
   [junit4]   2> 5897212 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 5897212 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr
   [junit4]   2> 5897212 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2020-02-20T07:44:05.214737Z
   [junit4]   2> 5897230 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5897230 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5897231 INFO  (zkConnectionManagerCallback-20726-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5897231 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5897232 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 5897238 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5897250 INFO  (zkConnectionManagerCallback-20730-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5897250 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5897250 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 5897310 INFO  (zkConnectionManagerCallback-20728-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5897310 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5897322 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 5899962 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 5899963 WARN  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.u.s.S.config Trusting all certificates configured for 
Client@18ee0a8f[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5899963 WARN  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
Client@18ee0a8f[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5900027 WARN  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.u.s.S.config Trusting all certificates configured for 
Client@5bf12d70[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5900027 WARN  (jetty-launcher-20724-thread-2) [     ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
Client@5bf12d70[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5900029 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43070/solr
   [junit4]   2> 5900071 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5900128 INFO  (zkConnectionManagerCallback-20739-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5900144 INFO  (jetty-launcher-20724-thread-2) [     ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5900424 INFO  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.c.ConnectionManager Waiting for client to 
connect to ZooKeeper
   [junit4]   2> 5900501 INFO  (zkConnectionManagerCallback-20741-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5900510 INFO  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.c.ConnectionManager Client is connected 
to ZooKeeper
   [junit4]   2> 5901158 INFO  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:34518_solr
   [junit4]   2> 5901175 INFO  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.Overseer Overseer 
(id=72317128431894535-127.0.0.1:34518_solr-n_0000000000) starting
   [junit4]   2> 5901204 INFO  
(OverseerStateUpdate-72317128431894535-127.0.0.1:34518_solr-n_0000000000) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.Overseer Starting to work on the main 
queue : 127.0.0.1:34518_solr
   [junit4]   2> 5901253 INFO  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:34518_solr
   [junit4]   2> 5901273 INFO  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.p.PackageLoader /packages.json updated to 
version -1
   [junit4]   2> 5901273 WARN  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.CoreContainer Not all security plugins 
configured!  authentication=disabled authorization=disabled.  Solr is only as 
secure as you make it. Consider configuring authentication/authorization before 
exposing Solr to users internal or external.  See 
https://s.apache.org/solrsecurity for more info
   [junit4]   2> 5901399 INFO  (zkCallback-20740-thread-1) [     ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 5901533 INFO  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.h.a.MetricsHistoryHandler No .system 
collection, keeping metrics history in memory.
   [junit4]   2> 5901786 INFO  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34518.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@74c51b81
   [junit4]   2> 5902164 INFO  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34518.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@74c51b81
   [junit4]   2> 5902165 INFO  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34518.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@74c51b81
   [junit4]   2> 5902183 INFO  (jetty-launcher-20724-thread-2) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.RemoteQueryErrorTest_4E094E65EF364989-001/tempDir-001/node2/.
   [junit4]   2> 5902709 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 5902731 WARN  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.u.s.S.config Trusting all certificates configured for 
Client@2f4a3a70[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5902731 WARN  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
Client@2f4a3a70[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5902738 WARN  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.u.s.S.config Trusting all certificates configured for 
Client@385a3118[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5902738 WARN  (jetty-launcher-20724-thread-3) [     ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
Client@385a3118[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5902760 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43070/solr
   [junit4]   2> 5902761 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5902762 INFO  (zkConnectionManagerCallback-20751-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5902762 INFO  (jetty-launcher-20724-thread-3) [     ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5902986 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.c.c.ConnectionManager Waiting for client to 
connect to ZooKeeper
   [junit4]   2> 5902998 INFO  (zkConnectionManagerCallback-20753-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5903010 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.c.c.ConnectionManager Client is connected 
to ZooKeeper
   [junit4]   2> 5903034 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 5903205 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.c.ZkController Publish 
node=127.0.0.1:44398_solr as DOWN
   [junit4]   2> 5903257 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 2147483647 transient cores
   [junit4]   2> 5903257 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:44398_solr
   [junit4]   2> 5903259 INFO  (zkCallback-20740-thread-1) [     ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 5903260 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.p.PackageLoader /packages.json updated to 
version -1
   [junit4]   2> 5903261 WARN  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.c.CoreContainer Not all security plugins 
configured!  authentication=disabled authorization=disabled.  Solr is only as 
secure as you make it. Consider configuring authentication/authorization before 
exposing Solr to users internal or external.  See 
https://s.apache.org/solrsecurity for more info
   [junit4]   2> 5903347 INFO  (zkCallback-20752-thread-1) [     ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 5903444 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.h.a.MetricsHistoryHandler No .system 
collection, keeping metrics history in memory.
   [junit4]   2> 5903576 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_44398.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@74c51b81
   [junit4]   2> 5903618 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_44398.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@74c51b81
   [junit4]   2> 5903618 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_44398.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@74c51b81
   [junit4]   2> 5903632 INFO  (jetty-launcher-20724-thread-3) 
[n:127.0.0.1:44398_solr     ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.RemoteQueryErrorTest_4E094E65EF364989-001/tempDir-001/node3/.
   [junit4]   2> 5904719 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: 
WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 5904720 WARN  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.u.s.S.config Trusting all certificates configured for 
Client@32e7ce77[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5904720 WARN  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
Client@32e7ce77[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5904723 WARN  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.u.s.S.config Trusting all certificates configured for 
Client@5f5b5fb[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5904724 WARN  (jetty-launcher-20724-thread-1) [     ] 
o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for 
Client@5f5b5fb[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 5904725 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43070/solr
   [junit4]   2> 5904726 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5904727 INFO  (zkConnectionManagerCallback-20763-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5904727 INFO  (jetty-launcher-20724-thread-1) [     ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5904830 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.c.c.ConnectionManager Waiting for client to 
connect to ZooKeeper
   [junit4]   2> 5904831 INFO  (zkConnectionManagerCallback-20765-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5904831 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.c.c.ConnectionManager Client is connected 
to ZooKeeper
   [junit4]   2> 5904846 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 5904851 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.c.ZkController Publish 
node=127.0.0.1:33252_solr as DOWN
   [junit4]   2> 5904852 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 2147483647 transient cores
   [junit4]   2> 5904852 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:33252_solr
   [junit4]   2> 5904853 INFO  (zkCallback-20752-thread-1) [     ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 5904853 INFO  (zkCallback-20740-thread-1) [     ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 5904853 INFO  (zkCallback-20764-thread-1) [     ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 5904854 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.p.PackageLoader /packages.json updated to 
version -1
   [junit4]   2> 5904855 WARN  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.c.CoreContainer Not all security plugins 
configured!  authentication=disabled authorization=disabled.  Solr is only as 
secure as you make it. Consider configuring authentication/authorization before 
exposing Solr to users internal or external.  See 
https://s.apache.org/solrsecurity for more info
   [junit4]   2> 5904989 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.h.a.MetricsHistoryHandler No .system 
collection, keeping metrics history in memory.
   [junit4]   2> 5905058 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_33252.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@74c51b81
   [junit4]   2> 5905072 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_33252.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@74c51b81
   [junit4]   2> 5905072 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_33252.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@74c51b81
   [junit4]   2> 5905074 INFO  (jetty-launcher-20724-thread-1) 
[n:127.0.0.1:33252_solr     ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.RemoteQueryErrorTest_4E094E65EF364989-001/tempDir-001/node1/.
   [junit4]   2> 5905123 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.MiniSolrCloudCluster waitForAllNodes: numServers=3
   [junit4]   2> 5905124 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5905125 INFO  (zkConnectionManagerCallback-20774-thread-1) [   
  ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 5905126 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5905127 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 5905129 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43070/solr ready
   [junit4]   2> 5905176 INFO  
(TEST-RemoteQueryErrorTest.test-seed#[4E094E65EF364989]) [     ] 
o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 5905203 INFO  (qtp1531180734-111674) [n:127.0.0.1:33252_solr   
  ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
collection.configName=conf&name=collection&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 5905207 INFO  
(OverseerThreadFactory-14112-thread-1-processing-n:127.0.0.1:34518_solr) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.a.c.CreateCollectionCmd Create collection 
collection
   [junit4]   2> 5905317 INFO  
(OverseerStateUpdate-72317128431894535-127.0.0.1:34518_solr-n_0000000000) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"collection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:34518/solr";,
   [junit4]   2>   "node_name":"127.0.0.1:34518_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 5905358 INFO  
(OverseerStateUpdate-72317128431894535-127.0.0.1:34518_solr-n_0000000000) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"collection_shard2_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:44398/solr";,
   [junit4]   2>   "node_name":"127.0.0.1:44398_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 5905562 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr    
x:collection_shard1_replica_n1 ] o.a.s.h.a.CoreAdminOperation core create 
command 
qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=collection_shard1_replica_n1&action=CREATE&numShards=2&collection=collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 5905562 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr    
x:collection_shard2_replica_n2 ] o.a.s.h.a.CoreAdminOperation core create 
command 
qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=collection_shard2_replica_n2&action=CREATE&numShards=2&collection=collection&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 5905563 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr    
x:collection_shard1_replica_n1 ] o.a.s.c.TransientSolrCoreCacheDefault 
Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 5905590 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0
   [junit4]   2> 5905629 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0
   [junit4]   2> 5905642 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.s.IndexSchema [collection_shard2_replica_n2] Schema name=minimal
   [junit4]   2> 5905653 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 5905653 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.CoreContainer Creating SolrCore 'collection_shard2_replica_n2' using 
configuration from configset conf, trusted=true
   [junit4]   2> 5905653 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_44398.solr.core.collection.shard2.replica_n2' (registry 
'solr.core.collection.shard2.replica_n2') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@74c51b81
   [junit4]   2> 5905654 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.SolrCore [[collection_shard2_replica_n2] ] Opening new SolrCore at 
[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.RemoteQueryErrorTest_4E094E65EF364989-001/tempDir-001/node3/collection_shard2_replica_n2],
 
dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.RemoteQueryErrorTest_4E094E65EF364989-001/tempDir-001/node3/./collection_shard2_replica_n2/data/]
   [junit4]   2> 5905656 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.s.IndexSchema [collection_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 5905659 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 5905659 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.CoreContainer Creating SolrCore 'collection_shard1_replica_n1' using 
configuration from configset conf, trusted=true
   [junit4]   2> 5905659 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_34518.solr.core.collection.shard1.replica_n1' (registry 
'solr.core.collection.shard1.replica_n1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@74c51b81
   [junit4]   2> 5905660 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.SolrCore [[collection_shard1_replica_n1] ] Opening new SolrCore at 
[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.RemoteQueryErrorTest_4E094E65EF364989-001/tempDir-001/node2/collection_shard1_replica_n1],
 
dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.RemoteQueryErrorTest_4E094E65EF364989-001/tempDir-001/node2/./collection_shard1_replica_n1/data/]
   [junit4]   2> 5906009 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 5906009 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 5906010 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 5906010 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 5906013 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 5906013 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 5906028 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 5906028 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 5906029 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@58978704[collection_shard1_replica_n1] main]
   [junit4]   2> 5906039 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@3d3e8efe[collection_shard2_replica_n2] main]
   [junit4]   2> 5906041 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 5906041 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 5906042 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 5906042 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1659040855798841344
   [junit4]   2> 5906046 INFO  
(searcherExecutor-14131-thread-1-processing-n:127.0.0.1:44398_solr 
x:collection_shard2_replica_n2 c:collection s:shard2 r:core_node4) 
[n:127.0.0.1:44398_solr c:collection s:shard2 r:core_node4 
x:collection_shard2_replica_n2 ] o.a.s.c.SolrCore 
[collection_shard2_replica_n2] Registered new searcher 
Searcher@3d3e8efe[collection_shard2_replica_n2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 5906068 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 5906068 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 5906069 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 5906069 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1659040855827152896
   [junit4]   2> 5906069 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.ZkShardTerms Successful update of terms at 
/collections/collection/terms/shard2 to Terms{values={core_node4=0}, version=0}
   [junit4]   2> 5906069 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.ShardLeaderElectionContextBase make sure parent is created 
/collections/collection/leaders/shard2
   [junit4]   2> 5906072 INFO  
(searcherExecutor-14132-thread-1-processing-n:127.0.0.1:34518_solr 
x:collection_shard1_replica_n1 c:collection s:shard1 r:core_node3) 
[n:127.0.0.1:34518_solr c:collection s:shard1 r:core_node3 
x:collection_shard1_replica_n1 ] o.a.s.c.SolrCore 
[collection_shard1_replica_n1] Registered new searcher 
Searcher@58978704[collection_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 5906073 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 5906073 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 5906073 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:44398/solr/collection_shard2_replica_n2/
   [junit4]   2> 5906074 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 5906074 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.SyncStrategy http://127.0.0.1:44398/solr/collection_shard2_replica_n2/ 
has no replicas
   [junit4]   2> 5906074 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node 
/collections/collection/leaders/shard2/leader after winning as 
/collections/collection/leader_elect/shard2/election/72317128431894537-core_node4-n_0000000000
   [junit4]   2> 5906091 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.ZkShardTerms Successful update of terms at 
/collections/collection/terms/shard1 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 5906091 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.ShardLeaderElectionContextBase make sure parent is created 
/collections/collection/leaders/shard1
   [junit4]   2> 5906092 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:44398/solr/collection_shard2_replica_n2/ shard2
   [junit4]   2> 5906103 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 5906103 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 5906103 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:34518/solr/collection_shard1_replica_n1/
   [junit4]   2> 5906104 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 5906104 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.SyncStrategy http://127.0.0.1:34518/solr/collection_shard1_replica_n1/ 
has no replicas
   [junit4]   2> 5906104 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node 
/collections/collection/leaders/shard1/leader after winning as 
/collections/collection/leader_elect/shard1/election/72317128431894535-core_node3-n_0000000000
   [junit4]   2> 5906106 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:34518/solr/collection_shard1_replica_n1/ shard1
   [junit4]   2> 5906208 INFO  (zkCallback-20740-thread-1) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906208 INFO  (zkCallback-20752-thread-1) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906208 INFO  (zkCallback-20740-thread-2) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906208 INFO  (zkCallback-20752-thread-2) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906210 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 5906211 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 5906213 INFO  (qtp102823917-111667) [n:127.0.0.1:34518_solr 
c:collection s:shard1 r:core_node3 x:collection_shard1_replica_n1 ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=collection_shard1_replica_n1&action=CREATE&numShards=2&collection=collection&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=651
   [junit4]   2> 5906215 INFO  (qtp502090713-111675) [n:127.0.0.1:44398_solr 
c:collection s:shard2 r:core_node4 x:collection_shard2_replica_n2 ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=collection_shard2_replica_n2&action=CREATE&numShards=2&collection=collection&shard=shard2&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=653
   [junit4]   2> 5906233 INFO  (qtp1531180734-111674) [n:127.0.0.1:33252_solr   
  ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at 
most 45 seconds. Check all shard replicas
   [junit4]   2> 5906315 INFO  (zkCallback-20752-thread-2) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906315 INFO  (zkCallback-20752-thread-1) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906315 INFO  (zkCallback-20764-thread-1) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906315 INFO  (zkCallback-20740-thread-1) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906315 INFO  (zkCallback-20752-thread-3) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906315 INFO  (zkCallback-20740-thread-2) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906315 INFO  (zkCallback-20740-thread-3) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906316 INFO  (qtp1531180734-111674) [n:127.0.0.1:33252_solr   
  ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={collection.configName=conf&name=collection&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2}
 status=0 QTime=1113
   [junit4]   2> 5906333 INFO  
(TEST-RemoteQueryErrorTest.test-seed#[4E094E65EF364989]) [     ] 
o.a.s.SolrTestCaseJ4 ###Ending test
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=RemoteQueryErrorTest -Dtests.method=test 
-Dtests.seed=4E094E65EF364989 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=en-PK -Dtests.timezone=Pacific/Bougainville 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 1.22s J1 | RemoteQueryErrorTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: 
   [junit4]    > Expected: a string containing "Document is missing mandatory 
uniqueKey field: id"
   [junit4]    >      but: was "Error from server at 
http://127.0.0.1:34518/solr: No active replicas found for collection: 
collection"
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([4E094E65EF364989:C65D71BF41CA2471]:0)
   [junit4]    >        at 
org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   [junit4]    >        at 
org.apache.solr.cloud.RemoteQueryErrorTest.test(RemoteQueryErrorTest.java:53)
   [junit4]    >        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]    >        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]    >        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]    >        at 
java.base/java.lang.reflect.Method.invoke(Method.java:566)
   [junit4]    >        at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> 5906453 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=568958718
   [junit4]   2> 5906453 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.c.ZkController Remove node as live in 
ZooKeeper:/live_nodes/127.0.0.1:34518_solr
   [junit4]   2> 5906454 INFO  (jetty-closer-20777-thread-2) [     ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=336465441
   [junit4]   2> 5906454 INFO  (jetty-closer-20777-thread-2) [     ] 
o.a.s.c.ZkController Remove node as live in 
ZooKeeper:/live_nodes/127.0.0.1:44398_solr
   [junit4]   2> 5906454 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.c.ZkController Publish this node as DOWN...
   [junit4]   2> 5906454 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.c.ZkController Publish node=127.0.0.1:34518_solr as DOWN
   [junit4]   2> 5906454 INFO  (jetty-closer-20777-thread-2) [     ] 
o.a.s.c.ZkController Publish this node as DOWN...
   [junit4]   2> 5906454 INFO  (jetty-closer-20777-thread-2) [     ] 
o.a.s.c.ZkController Publish node=127.0.0.1:44398_solr as DOWN
   [junit4]   2> 5906456 INFO  (coreCloseExecutor-14142-thread-1) 
[n:127.0.0.1:44398_solr c:collection s:shard2 r:core_node4 
x:collection_shard2_replica_n2 ] o.a.s.c.SolrCore 
[collection_shard2_replica_n2]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@a52692c
   [junit4]   2> 5906456 INFO  (coreCloseExecutor-14142-thread-1) 
[n:127.0.0.1:44398_solr c:collection s:shard2 r:core_node4 
x:collection_shard2_replica_n2 ] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.core.collection.shard2.replica_n2, 
tag=SolrCore@a52692c
   [junit4]   2> 5906456 INFO  (coreCloseExecutor-14142-thread-1) 
[n:127.0.0.1:44398_solr c:collection s:shard2 r:core_node4 
x:collection_shard2_replica_n2 ] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@466da6b4: rootName = 
solr_44398, domain = solr.core.collection.shard2.replica_n2, service url = 
null, agent id = null] for registry solr.core.collection.shard2.replica_n2 / 
com.codahale.metrics.MetricRegistry@4159c306
   [junit4]   2> 5906457 INFO  (coreCloseExecutor-14141-thread-1) 
[n:127.0.0.1:34518_solr c:collection s:shard1 r:core_node3 
x:collection_shard1_replica_n1 ] o.a.s.c.SolrCore 
[collection_shard1_replica_n1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@63aa86cf
   [junit4]   2> 5906457 INFO  (coreCloseExecutor-14141-thread-1) 
[n:127.0.0.1:34518_solr c:collection s:shard1 r:core_node3 
x:collection_shard1_replica_n1 ] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.core.collection.shard1.replica_n1, 
tag=SolrCore@63aa86cf
   [junit4]   2> 5906457 INFO  (coreCloseExecutor-14141-thread-1) 
[n:127.0.0.1:34518_solr c:collection s:shard1 r:core_node3 
x:collection_shard1_replica_n1 ] o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@d809492: rootName = 
solr_34518, domain = solr.core.collection.shard1.replica_n1, service url = 
null, agent id = null] for registry solr.core.collection.shard1.replica_n1 / 
com.codahale.metrics.MetricRegistry@3e671159
   [junit4]   2> 5906459 INFO  (jetty-closer-20777-thread-3) [     ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1017680840
   [junit4]   2> 5906459 INFO  (jetty-closer-20777-thread-3) [     ] 
o.a.s.c.ZkController Remove node as live in 
ZooKeeper:/live_nodes/127.0.0.1:33252_solr
   [junit4]   2> 5906459 INFO  (jetty-closer-20777-thread-3) [     ] 
o.a.s.c.ZkController Publish this node as DOWN...
   [junit4]   2> 5906459 INFO  (jetty-closer-20777-thread-3) [     ] 
o.a.s.c.ZkController Publish node=127.0.0.1:33252_solr as DOWN
   [junit4]   2> 5906651 INFO  (zkCallback-20752-thread-3) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906651 INFO  (zkCallback-20752-thread-1) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906651 INFO  (zkCallback-20752-thread-2) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906668 INFO  (zkCallback-20740-thread-3) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906668 INFO  (zkCallback-20740-thread-1) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906670 INFO  (zkCallback-20740-thread-4) [     ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection/state.json] for collection [collection] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 5906744 INFO  (jetty-closer-20777-thread-3) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 5906744 INFO  (jetty-closer-20777-thread-3) [     ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@483df448: rootName = 
solr_33252, domain = solr.node, service url = null, agent id = null] for 
registry solr.node / com.codahale.metrics.MetricRegistry@46ce6ea
   [junit4]   2> 5906745 INFO  (coreCloseExecutor-14141-thread-1) 
[n:127.0.0.1:34518_solr c:collection s:shard1 r:core_node3 
x:collection_shard1_replica_n1 ] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.collection.shard1.leader, 
tag=SolrCore@63aa86cf
   [junit4]   2> 5906777 INFO  (coreCloseExecutor-14141-thread-1) 
[n:127.0.0.1:34518_solr c:collection s:shard1 r:core_node3 
x:collection_shard1_replica_n1 ] o.a.s.u.DirectUpdateHandler2 Committing on 
IndexWriter.close()  ... SKIPPED (unnecessary).
   [junit4]   2> 5906784 INFO  (coreCloseExecutor-14142-thread-1) 
[n:127.0.0.1:44398_solr c:collection s:shard2 r:core_node4 
x:collection_shard2_replica_n2 ] o.a.s.m.SolrMetricManager Closing metric 
reporters for registry=solr.collection.collection.shard2.leader, 
tag=SolrCore@a52692c
   [junit4]   2> 5906806 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 5906806 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@3ed4c4e1: rootName = 
solr_34518, domain = solr.node, service url = null, agent id = null] for 
registry solr.node / com.codahale.metrics.MetricRegistry@15352b3d
   [junit4]   2> 5906824 INFO  (jetty-closer-20777-thread-3) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 5906825 INFO  (jetty-closer-20777-thread-3) [     ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@f6537c2: rootName = 
solr_33252, domain = solr.jvm, service url = null, agent id = null] for 
registry solr.jvm / com.codahale.metrics.MetricRegistry@5111173e
   [junit4]   2> 5906825 INFO  (coreCloseExecutor-14142-thread-1) 
[n:127.0.0.1:44398_solr c:collection s:shard2 r:core_node4 
x:collection_shard2_replica_n2 ] o.a.s.u.DirectUpdateHandler2 Committing on 
IndexWriter.close()  ... SKIPPED (unnecessary).
   [junit4]   2> 5906832 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 5906833 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@1adc00e1: rootName = 
solr_34518, domain = solr.jvm, service url = null, agent id = null] for 
registry solr.jvm / com.codahale.metrics.MetricRegistry@5111173e
   [junit4]   2> 5906845 INFO  (jetty-closer-20777-thread-3) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 5906845 INFO  (jetty-closer-20777-thread-3) [     ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@11fb85a9: rootName = 
solr_33252, domain = solr.jetty, service url = null, agent id = null] for 
registry solr.jetty / com.codahale.metrics.MetricRegistry@1bd00ff6
   [junit4]   2> 5906845 INFO  (jetty-closer-20777-thread-3) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 5906853 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 5906853 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@4f022e8c: rootName = 
solr_34518, domain = solr.jetty, service url = null, agent id = null] for 
registry solr.jetty / com.codahale.metrics.MetricRegistry@1bd00ff6
   [junit4]   2> 5906853 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 5906888 INFO  (closeThreadPool-20787-thread-1) [     ] 
o.a.s.c.Overseer Overseer 
(id=72317128431894535-127.0.0.1:34518_solr-n_0000000000) closing
   [junit4]   2> 5906899 INFO  
(OverseerAutoScalingTriggerThread-72317128431894535-127.0.0.1:34518_solr-n_0000000000)
 [     ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we 
are closed, exiting.
   [junit4]   2> 5906911 INFO  
(OverseerStateUpdate-72317128431894535-127.0.0.1:34518_solr-n_0000000000) 
[n:127.0.0.1:34518_solr     ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:34518_solr
   [junit4]   2> 5906942 INFO  (zkCallback-20764-thread-1) [     ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 5906942 INFO  (zkCallback-20752-thread-2) [     ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
   [junit4]   2> 5906943 INFO  (jetty-closer-20777-thread-2) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 5906943 INFO  (jetty-closer-20777-thread-2) [     ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@6818dd7f: rootName = 
solr_44398, domain = solr.node, service url = null, agent id = null] for 
registry solr.node / com.codahale.metrics.MetricRegistry@2d403c71
   [junit4]   2> 5906973 INFO  (closeThreadPool-20787-thread-2) [     ] 
o.a.s.c.Overseer Overseer 
(id=72317128431894535-127.0.0.1:34518_solr-n_0000000000) closing
   [junit4]   2> 5906995 INFO  (jetty-closer-20777-thread-2) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 5906995 INFO  (jetty-closer-20777-thread-2) [     ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@519bc49d: rootName = 
solr_44398, domain = solr.jvm, service url = null, agent id = null] for 
registry solr.jvm / com.codahale.metrics.MetricRegistry@5111173e
   [junit4]   2> 5907005 INFO  (zkCallback-20752-thread-1) [     ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 5907015 INFO  (jetty-closer-20777-thread-2) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 5907015 INFO  (jetty-closer-20777-thread-2) [     ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@3e2b6976: rootName = 
solr_44398, domain = solr.jetty, service url = null, agent id = null] for 
registry solr.jetty / com.codahale.metrics.MetricRegistry@1bd00ff6
   [junit4]   2> 5907015 INFO  (jetty-closer-20777-thread-2) [     ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 5907049 INFO  (jetty-closer-20777-thread-1) [     ] 
o.a.s.c.Overseer Overseer 
(id=72317128431894535-127.0.0.1:34518_solr-n_0000000000) closing
   [junit4]   2> 5907139 INFO  (jetty-closer-20777-thread-3) [     ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@558413a6{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:0}
   [junit4]   2> 5907139 INFO  (jetty-closer-20777-thread-3) [     ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@754185a1{/solr,null,UNAVAILABLE}
   [junit4]   2> 5907139 INFO  (jetty-closer-20777-thread-3) [     ] 
o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 5907140 INFO  (jetty-closer-20777-thread-1) [     ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@8e71959{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:0}
   [junit4]   2> 5907140 INFO  (jetty-closer-20777-thread-1) [     ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@c49d1de{/solr,null,UNAVAILABLE}
   [junit4]   2> 5907141 INFO  (jetty-closer-20777-thread-1) [     ] 
o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 5907707 INFO  (jetty-closer-20777-thread-2) [     ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@132aa614{HTTP/1.1,[http/1.1, 
h2c]}{127.0.0.1:0}
   [junit4]   2> 5907707 INFO  (jetty-closer-20777-thread-2) [     ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@7681c073{/solr,null,UNAVAILABLE}
   [junit4]   2> 5907752 INFO  (jetty-closer-20777-thread-2) [     ] 
o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 5907788 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.ZkTestServer Shutting down ZkTestServer.
   [junit4]   2> 5908094 WARN  (ZkTestServer Run Thread) [     ] 
o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        4       /solr/aliases.json
   [junit4]   2>        4       /solr/clusterprops.json
   [junit4]   2>        3       /solr/packages.json
   [junit4]   2>        3       /solr/security.json
   [junit4]   2>        2       /solr/collections/collection/terms/shard1
   [junit4]   2>        2       /solr/collections/collection/terms/shard2
   [junit4]   2>        2       /solr/configs/conf
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        26      /solr/collections/collection/state.json
   [junit4]   2>        4       /solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        21      /solr/overseer/queue
   [junit4]   2>        13      /solr/live_nodes
   [junit4]   2>        8       /solr/collections
   [junit4]   2>        4       /solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> 5908096 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.ZkTestServer waitForServerDown: 127.0.0.1:43070
   [junit4]   2> 5908096 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43070
   [junit4]   2> 5908096 INFO  
(SUITE-RemoteQueryErrorTest-seed#[4E094E65EF364989]-worker) [     ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1 43070
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.RemoteQueryErrorTest_4E094E65EF364989-001
   [junit4]   2> Feb 20, 2020 7:44:16 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: 
codec=FastCompressingStoredFieldsData(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST,
 chunkSize=7, maxDocsPerChunk=577, blockShift=8), 
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, 
chunkSize=7, blockSize=8)), 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@76b43195),
 locale=en-PK, timezone=Pacific/Bougainville
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 
(64-bit)/cpus=8,threads=1,free=365625032,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [AnalyticsQueryTest, 
TestHighlightDedupGrouping, PeerSyncWithLeaderAndIndexFingerprintCachingTest, 
TestSchemaField, JsonLoaderTest, DeleteReplicaTest, TestFieldCacheWithThreads, 
TestManagedStopFilterFactory, HdfsThreadLeakTest, LukeRequestHandlerTest, 
BigEndianAscendingWordDeserializerTest, LeaderTragicEventTest, 
CloudExitableDirectoryReaderTest, CollectionsAPISolrJTest, 
TestBlendedInfixSuggestions, IndexSizeTriggerSizeEstimationTest, 
TestPullReplica, HdfsAutoAddReplicasIntegrationTest, TestExactSharedStatsCache, 
TestSchemaVersionResource, TemplateUpdateProcessorTest, AddReplicaTest, 
TestFaceting, TestPayloadScoreQParserPlugin, TestShortCircuitedRequests, 
AlternateDirectoryTest, TestAuthenticationFramework, TestJettySolrRunner, 
TestCoreDiscovery, TestSQLHandler, CdcrOpsAndBoundariesTest, MetricTriggerTest, 
SuggesterWFSTTest, TestWithCollection, UninvertDocValuesMergePolicyTest, 
RAMDirectoryFactoryTest, TestMacroExpander, TestSimClusterStateProvider, 
TestQuerySenderNoQuery, SpatialFilterTest, DocValuesMultiTest, 
TestPerFieldSimilarity, TriggerEventQueueTest, HdfsDirectoryTest, 
TestSolrQueryResponse, TestDFRSimilarityFactory, SolrCLIZkUtilsTest, 
ByteBuffersDirectoryFactoryTest, SystemLogListenerTest, 
TestCloudNestedDocsSort, SignatureUpdateProcessorFactoryTest, 
AuditLoggerIntegrationTest, TestUseDocValuesAsStored2, TestLegacyField, 
TestDistributedSearch, TestHdfsBackupRestoreCore, TestNumericRangeQuery64, 
XsltUpdateRequestHandlerTest, JWTAuthPluginTest, TestSolrTestCaseJ4, 
TestJsonFacetsWithNestedObjects, TestQueryUtils, ConnectionManagerTest, 
HdfsCollectionsAPIDistributedZkTest, TestFuzzyAnalyzedSuggestions, 
TestTrieFacet, TestConfigSetsAPI, TestRequestStatusCollectionAPI, 
TestCollapseQParserPlugin, UtilsToolTest, MetricsHistoryIntegrationTest, 
ShardsWhitelistTest, SpatialHeatmapFacetsTest, TestNumericTerms64, 
CloneFieldUpdateProcessorFactoryTest, 
OverriddenZkACLAndCredentialsProvidersTest, TestCloudSchemaless, 
DebugComponentTest, TestScoreJoinQPScore, ReturnFieldsTest, 
DistributedFacetPivotSmallAdvancedTest, NestedShardedAtomicUpdateTest, 
TestElisionMultitermQuery, TestCSVLoader, TestReloadDeadlock, 
TestSimTriggerIntegration, ZkCollectionPropsCachingTest, 
TestSolrDeletionPolicy2, TestRandomFlRTGCloud, TestExceedMaxTermLength, 
ChaosMonkeySafeLeaderWithPullReplicasTest, TestNoOpRegenerator, 
TestDirectoryFactory, FuzzySearchTest, CoreAdminCreateDiscoverTest, 
BasicFunctionalityTest, ResponseBuilderTest, SuggesterFSTTest, 
ResolveAnalyzerByNameTest, TestTlogReplayVsRecovery, BasicDistributedZk2Test, 
TestRangeQuery, TestSolr4Spatial2, VersionInfoTest, 
TestTolerantUpdateProcessorRandomCloud, SpellCheckCollatorTest, 
DeleteInactiveReplicaTest, CloudReplicaSourceTest, 
TestSubQueryTransformerDistrib, ScheduledTriggerTest, TestXmlQParserPlugin, 
V2ApiIntegrationTest, DistributedQueryElevationComponentTest, HLLUtilTest, 
AnalysisErrorHandlingTest, TestJsonFacetRefinement, 
TestGraphTermsQParserPlugin, TestSimpleQParserPlugin, DirectSolrConnectionTest, 
CoreSorterTest, TaggerTest, TestDeprecatedFilters, MinimalSchemaTest, 
TestSimpleTextCodec, SimpleFacetsTest, ClusterStateUpdateTest, 
CdcrVersionReplicationTest, LeaderElectionTest, TestHalfAndHalfDocValues, 
TestSolrCloudWithHadoopAuthPlugin, RegexBoostProcessorTest, TestCoreAdminApis, 
BadCopyFieldTest, RequestLoggingTest, TestCryptoKeys, 
FastVectorHighlighterTest, TestStressRecovery, CollectionTooManyReplicasTest, 
RecoveryZkTest, TestStressVersions, TestSolrCoreSnapshots, 
OpenExchangeRatesOrgProviderTest, TestRecovery, 
TestSolrConfigHandlerConcurrent, TestLeaderElectionZkExpiry, CursorPagingTest, 
DisMaxRequestHandlerTest, DistributedIntervalFacetingTest, EchoParamsTest, 
SolrInfoBeanTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, 
TestDistributedGrouping, TestRandomDVFaceting, TestRandomFaceting, 
TestSimpleTrackingShardHandler, PathHierarchyTokenizerFactoryTest, 
ProtectedTermFilterFactoryTest, TestCharFilters, 
TestReversedWildcardFilterFactory, TestWordDelimiterFilterFactory, 
TokenizerChainTest, TestLuceneIndexBackCompat, 
TestEmbeddedSolrServerAdminHandler, TestEmbeddedSolrServerConstructors, 
TestEmbeddedSolrServerSchemaAPI, ActionThrottleTest, 
AssignBackwardCompatibilityTest, BasicZkTest, ChaosMonkeyNothingIsSafeTest, 
ChaosMonkeyNothingIsSafeWithPullReplicasTest, ChaosMonkeySafeLeaderTest, 
ClusterStateTest, CollectionStateFormat2Test, ConfigSetsAPITest, 
DeleteStatusTest, DistribCursorPagingTest, 
DistribDocExpirationUpdateProcessorTest, DistribJoinFromCollectionTest, 
DistributedQueueTest, DistributedVersionInfoTest, ForceLeaderTest, 
ForceLeaderWithTlogReplicasTest, HealthCheckHandlerTest, 
HttpPartitionOnCommitTest, HttpPartitionTest, 
HttpPartitionWithTlogReplicasTest, MoveReplicaTest, MultiSolrCloudTestCaseTest, 
OutOfBoxZkACLAndCredentialsProvidersTest, OverseerRolesTest, OverseerTest, 
RemoteQueryErrorTest]
   [junit4] Completed [505/904 (1!)] on J1 in 11.69s, 1 test, 1 failure <<< 
FAILURES!

[...truncated 10937 lines...]
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:653:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:587:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid1913.hprof

Total time: 454 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to