Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/350/

2 tests failed.
FAILED:  org.apache.lucene.index.TestIndexSorting.testRandom3

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
        at 
__randomizedtesting.SeedInfo.seed([C9B802A09D538A0A:6B604C7AF9A1A30C]:0)
        at 
org.apache.lucene.util.packed.Packed8ThreeBlocks.<init>(Packed8ThreeBlocks.java:41)
        at 
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:963)
        at 
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:939)
        at 
org.apache.lucene.util.packed.GrowableWriter.ensureCapacity(GrowableWriter.java:80)
        at 
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:88)
        at 
org.apache.lucene.util.packed.AbstractPagedMutable.set(AbstractPagedMutable.java:98)
        at org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:152)
        at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
        at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
        at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:214)
        at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:310)
        at org.apache.lucene.util.fst.Builder.add(Builder.java:414)
        at 
org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.writeFST(MemoryDocValuesConsumer.java:367)
        at 
org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:404)
        at 
org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:653)
        at 
org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:204)
        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:153)
        at 
org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:167)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:111)
        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
        at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2083)
        at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:5005)
        at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5043)
        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5034)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1574)
        at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1316)
        at 
org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2229)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)


FAILED:  
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testCollectionsAPI

Error Message:
expected:<3> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<3> but was:<1>
        at 
__randomizedtesting.SeedInfo.seed([1F377C89C3176F6:4986037C9A025963]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:522)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 1819 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexSorting
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexSorting 
-Dtests.method=testRandom3 -Dtests.seed=C9B802A09D538A0A -Dtests.multiplier=2 
-Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=ca-ES -Dtests.timezone=PNT -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] ERROR   3851s J1 | TestIndexSorting.testRandom3 <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([C9B802A09D538A0A:6B604C7AF9A1A30C]:0)
   [junit4]    >        at 
org.apache.lucene.util.packed.Packed8ThreeBlocks.<init>(Packed8ThreeBlocks.java:41)
   [junit4]    >        at 
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:963)
   [junit4]    >        at 
org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:939)
   [junit4]    >        at 
org.apache.lucene.util.packed.GrowableWriter.ensureCapacity(GrowableWriter.java:80)
   [junit4]    >        at 
org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:88)
   [junit4]    >        at 
org.apache.lucene.util.packed.AbstractPagedMutable.set(AbstractPagedMutable.java:98)
   [junit4]    >        at 
org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:152)
   [junit4]    >        at 
org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
   [junit4]    >        at 
org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
   [junit4]    >        at 
org.apache.lucene.util.fst.Builder.compileNode(Builder.java:214)
   [junit4]    >        at 
org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:310)
   [junit4]    >        at 
org.apache.lucene.util.fst.Builder.add(Builder.java:414)
   [junit4]    >        at 
org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.writeFST(MemoryDocValuesConsumer.java:367)
   [junit4]    >        at 
org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:404)
   [junit4]    >        at 
org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:653)
   [junit4]    >        at 
org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:204)
   [junit4]    >        at 
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:153)
   [junit4]    >        at 
org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:167)
   [junit4]    >        at 
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:111)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
   [junit4]    >        at 
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2083)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:5005)
   [junit4]    >        at 
org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5043)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5034)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1574)
   [junit4]    >        at 
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1316)
   [junit4]    >        at 
org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:2229)
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J1/temp/lucene.index.TestIndexSorting_C9B802A09D538A0A-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{docs=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 sparse_text=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
norms=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 positions=PostingsFormat(name=LuceneVarGapDocFreqInterval), 
id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 term_vectors=PostingsFormat(name=Memory doPackFST= true)}, 
docValues:{multi_valued_long=DocValuesFormat(name=Asserting), 
foo=DocValuesFormat(name=Asserting), numeric=DocValuesFormat(name=Direct), 
float=DocValuesFormat(name=Lucene54), 
multi_valued_numeric=DocValuesFormat(name=Memory), 
long=DocValuesFormat(name=Lucene54), points=DocValuesFormat(name=Memory), 
sparse_int=DocValuesFormat(name=Lucene54), bar=DocValuesFormat(name=Memory), 
multi_valued_double=DocValuesFormat(name=Memory), 
docs=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Memory), 
multi_valued_int=DocValuesFormat(name=Direct), 
multi_valued_bytes=DocValuesFormat(name=Direct), 
sparse_text=DocValuesFormat(name=Asserting), 
double=DocValuesFormat(name=Direct), positions=DocValuesFormat(name=Asserting), 
sparse_binary=DocValuesFormat(name=Asserting), 
int=DocValuesFormat(name=Memory), sorted=DocValuesFormat(name=Direct), 
sparse=DocValuesFormat(name=Asserting), 
multi_valued_string=DocValuesFormat(name=Memory), 
norms=DocValuesFormat(name=Memory), bytes=DocValuesFormat(name=Memory), 
binary=DocValuesFormat(name=Direct), dense_int=DocValuesFormat(name=Memory), 
multi_valued_float=DocValuesFormat(name=Lucene54), 
term_vectors=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=25, 
maxMBSortInHeap=5.2194053838463725, 
sim=RandomSimilarity(queryNorm=true,coord=yes): {positions=DFR I(ne)L3(800.0), 
id=DFR GL3(800.0), term_vectors=DFR I(n)1}, locale=ca-ES, timezone=PNT
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_121 (64-bit)/cpus=4,threads=1,free=335686232,total=422051840
   [junit4]   2> NOTE: All tests run in this JVM: [TestTermsEnum, 
TestBooleanQueryVisitSubscorers, TestFilterIterator, 
TestConcurrentMergeScheduler, TestWildcard, TestMultiTermConstantScore, 
TestUsageTrackingFilterCachingPolicy, TestIndexWriterCommit, 
TestLucene60PointsFormat, TestExternalCodecs, TestLSBRadixSorter, 
TestManyFields, TestSubScorerFreqs, TestTimSorterWorstCase, TestFilterWeight, 
TestReaderClosed, TestVirtualMethod, TestAllFilesHaveChecksumFooter, 
TestArrayUtil, TestIsCurrent, TestSloppyPhraseQuery, TestAutomaton, 
TestQueryRescorer, TestRecyclingByteBlockAllocator, 
TestReaderWrapperDVTypeCheck, TestDirectoryReader, TestNRTReaderCleanup, 
Test2BSortedDocValuesOrds, TestForTooMuchCloning, TestCheckIndex, 
TestSimpleAttributeImpl, TestRadixSelector, TestSpanNearQuery, 
TestParallelLeafReader, TestOmitNorms, TestBinaryDocument, 
TestIndexWriterOnVMError, TestSentinelIntSet, TestFilterLeafReader, 
TestDocValuesRewriteMethod, TestTopDocsCollector, TestDocIDMerger, 
TestAutomatonQuery, TestStressNRT, TestRAMDirectory, Test2BBKDPoints, 
TestIntArrayDocIdSet, TestNGramPhraseQuery, TestPagedBytes, 
TestUpgradeIndexMergePolicy, TestNorms, TestIndexWriterOnDiskFull, 
TestMultiThreadTermVectors, TestLucene50StoredFieldsFormat, TestNoMergePolicy, 
TestRegexpRandom, TestPostingsOffsets, TestLongRangeFieldQueries, 
TestUnicodeUtil, TestPriorityQueue, TestMultiMMap, TestSloppyMath, 
TestDuelingCodecsAtNight, TestCharsRefBuilder, TestPerFieldPostingsFormat, 
TestSameScoresWithThreads, TestIntBlockPool, TestNIOFSDirectory, TestCodecUtil, 
TestSizeBoundedForceMerge, TestGraphTokenizers, TestIndexSearcher, 
TestDeterminism, TestOmitTf, TestMathUtil, TestNeverDelete, 
TestComplexExplanations, TestFastDecompressionMode, 
TestParallelCompositeReader, TestFieldCacheRewriteMethod, TestTransactions, 
TestSegmentTermDocs, TestPerFieldPostingsFormat2, TestSort, TestMultiDocValues, 
TestDocValuesIndexing, TestBooleanMinShouldMatch, TestLegacyNumericUtils, 
TestSpans, TestCustomNorms, TestTransactionRollback, TestSpanExplanations, 
TestMultiPhraseQuery, TestMergeSchedulerExternal, TestBagOfPostings, 
TestFuzzyQuery, TestDoc, TestSimilarity2, TestNoDeletionPolicy, 
TestTermVectorsReader, TestMultiValuedNumericRangeQuery, 
TestComplexExplanationsOfNonMatches, TestSortRandom, TestPrefixCodedTerms, 
TestAllFilesHaveCodecHeader, TestParallelReaderEmptyIndex, TestDocument, 
TestMaxTermFrequency, TestTermScorer, TestTopFieldCollector, 
TestFileSwitchDirectory, TestSumDocFreq, TestIndexableField, 
TestWildcardRandom, TestDirectory, TestMultiLevelSkipList, 
TestIndexFileDeleter, TestConstantScoreQuery, TestMatchAllDocsQuery, 
TestBinaryTerms, TestConjunctions, TestMultiTermQueryRewrites, 
TestPositionIncrement, TestField, TestDateSort, TestSimilarityProvider, 
TestNumericTokenStream, TestSearch, TestCachingTokenFilter, TestDocCount, 
TestCachingCollector, TestAttributeSource, TestTotalHitCountCollector, 
TestBooleanScorer, TestRamUsageEstimator, TestSpanFirstQuery, 
TestIndexWriterLockRelease, TestPhrasePrefixQuery, TestByteBlockPool, 
TestBytesRef, TestDemo, TestCharFilter, TestRollback, TestTwoPhaseCommitTool, 
TestNot, TestCloseableThreadLocal, TestIndexWriterOnJRECrash, TestTermdocPerf, 
Test2BPositions, TestIndexCommit, TestTerm, TestPositiveScoresOnlyCollector, 
Test2BTerms, TestNewestSegment, TestByteArrayDataInput, Test2BNumericDocValues, 
Test2BBinaryDocValues, TestCharArrayMap, TestCharArraySet, TestCharacterUtils, 
TestReusableStringReader, TestStopFilter, TestBytesRefAttImpl, 
TestPackedTokenAttributeImpl, TestBlockPostingsFormat3, 
TestLucene50CompoundFormat, TestLucene50StoredFieldsFormatHighCompression, 
TestLucene50TermVectorsFormat, TestLucene60FieldInfoFormat, 
TestLucene53NormsFormat, TestFieldType, TestGeoEncodingUtils, TestPolygon, 
TestPolygon2D, TestExitableDirectoryReader, TestFieldReuse, 
TestFilterCodecReader, TestIndexReaderClose, TestIndexSorting]
   [junit4] Completed [456/456 (1!)] on J1 in 3879.00s, 50 tests, 1 error <<< 
FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J1: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/temp/junit4-J1-20170516_015011_9675172344729003322175.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/heapdumps/java_pid1694.hprof
 ...
   [junit4] Heap dump file created [391349138 bytes in 1.392 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 9968 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/init-core-data-001
   [junit4]   2> 513  INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 516  INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 881  INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/tempDir-001
   [junit4]   2> 892  INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 894  INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 894  INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 994  INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.s.c.ZkTestServer start zk server on port:42404
   [junit4]   2> 1069 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn Exception causing close of session 0x0 due to 
java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 1075 ERROR (Thread-1) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3427 INFO  (jetty-launcher-1-thread-4) [    ] o.e.j.s.Server 
jetty-9.3.14.v20161028
   [junit4]   2> 3429 INFO  (jetty-launcher-1-thread-1) [    ] o.e.j.s.Server 
jetty-9.3.14.v20161028
   [junit4]   2> 3429 INFO  (jetty-launcher-1-thread-2) [    ] o.e.j.s.Server 
jetty-9.3.14.v20161028
   [junit4]   2> 3430 INFO  (jetty-launcher-1-thread-3) [    ] o.e.j.s.Server 
jetty-9.3.14.v20161028
   [junit4]   2> 3490 INFO  (jetty-launcher-1-thread-4) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@26f6967a{/solr,null,AVAILABLE}
   [junit4]   2> 3494 INFO  (jetty-launcher-1-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@312f59a7{/solr,null,AVAILABLE}
   [junit4]   2> 3490 INFO  (jetty-launcher-1-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@6fa2f9{/solr,null,AVAILABLE}
   [junit4]   2> 3501 INFO  (jetty-launcher-1-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@66f66c72{/solr,null,AVAILABLE}
   [junit4]   2> 3547 INFO  (jetty-launcher-1-thread-1) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@3daf36b9{SSL,[ssl, 
http/1.1]}{127.0.0.1:50939}
   [junit4]   2> 3547 INFO  (jetty-launcher-1-thread-1) [    ] o.e.j.s.Server 
Started @6650ms
   [junit4]   2> 3547 INFO  (jetty-launcher-1-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=50939}
   [junit4]   2> 3554 INFO  (jetty-launcher-1-thread-2) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@7c347514{SSL,[ssl, 
http/1.1]}{127.0.0.1:56481}
   [junit4]   2> 3565 INFO  (jetty-launcher-1-thread-2) [    ] o.e.j.s.Server 
Started @6668ms
   [junit4]   2> 3554 INFO  (jetty-launcher-1-thread-4) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@7de2898e{SSL,[ssl, 
http/1.1]}{127.0.0.1:47037}
   [junit4]   2> 3567 INFO  (jetty-launcher-1-thread-4) [    ] o.e.j.s.Server 
Started @6670ms
   [junit4]   2> 3567 INFO  (jetty-launcher-1-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=47037}
   [junit4]   2> 3554 INFO  (jetty-launcher-1-thread-3) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@4fcf6af4{SSL,[ssl, 
http/1.1]}{127.0.0.1:38868}
   [junit4]   2> 3565 INFO  (jetty-launcher-1-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=56481}
   [junit4]   2> 3574 INFO  (jetty-launcher-1-thread-3) [    ] o.e.j.s.Server 
Started @6677ms
   [junit4]   2> 3574 INFO  (jetty-launcher-1-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=38868}
   [junit4]   2> 3635 ERROR (jetty-launcher-1-thread-1) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 3638 ERROR (jetty-launcher-1-thread-4) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 3639 INFO  (jetty-launcher-1-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.7.0
   [junit4]   2> 3645 INFO  (jetty-launcher-1-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3649 INFO  (jetty-launcher-1-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3649 ERROR (jetty-launcher-1-thread-2) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 3650 INFO  (jetty-launcher-1-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.7.0
   [junit4]   2> 3650 INFO  (jetty-launcher-1-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3650 INFO  (jetty-launcher-1-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3645 INFO  (jetty-launcher-1-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.7.0
   [junit4]   2> 3651 INFO  (jetty-launcher-1-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3651 INFO  (jetty-launcher-1-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3739 ERROR (jetty-launcher-1-thread-3) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 3739 INFO  (jetty-launcher-1-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.7.0
   [junit4]   2> 3739 INFO  (jetty-launcher-1-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3739 INFO  (jetty-launcher-1-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3764 INFO  (jetty-launcher-1-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-05-16T06:10:19.784Z
   [junit4]   2> 3766 INFO  (jetty-launcher-1-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-05-16T06:10:19.784Z
   [junit4]   2> 3766 INFO  (jetty-launcher-1-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-05-16T06:10:19.782Z
   [junit4]   2> 3766 INFO  (jetty-launcher-1-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-05-16T06:10:19.784Z
   [junit4]   2> 3817 INFO  (jetty-launcher-1-thread-2) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 3847 INFO  (jetty-launcher-1-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3871 INFO  (jetty-launcher-1-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3883 INFO  (jetty-launcher-1-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3883 INFO  (jetty-launcher-1-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4071 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15c0fe0e9330004, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 4408 INFO  (jetty-launcher-1-thread-2) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 4410 INFO  (jetty-launcher-1-thread-3) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 4412 INFO  (jetty-launcher-1-thread-1) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 4434 INFO  (jetty-launcher-1-thread-4) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 4532 INFO  (jetty-launcher-1-thread-4) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42404/solr
   [junit4]   2> 4534 INFO  (jetty-launcher-1-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42404/solr
   [junit4]   2> 4534 INFO  (jetty-launcher-1-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42404/solr
   [junit4]   2> 4546 INFO  (jetty-launcher-1-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42404/solr
   [junit4]   2> 4770 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15c0fe0e9330008, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 4800 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15c0fe0e9330006, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 5170 INFO  (jetty-launcher-1-thread-4) [n:127.0.0.1:47037_solr 
   ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 5170 INFO  (jetty-launcher-1-thread-1) [n:127.0.0.1:50939_solr 
   ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 5173 INFO  (jetty-launcher-1-thread-2) [n:127.0.0.1:56481_solr 
   ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 5176 INFO  (jetty-launcher-1-thread-4) [n:127.0.0.1:47037_solr 
   ] o.a.s.c.OverseerElectionContext I am going to be the leader 
127.0.0.1:47037_solr
   [junit4]   2> 5180 INFO  (jetty-launcher-1-thread-3) [n:127.0.0.1:38868_solr 
   ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 5182 INFO  (jetty-launcher-1-thread-1) [n:127.0.0.1:50939_solr 
   ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:50939_solr
   [junit4]   2> 5182 INFO  (jetty-launcher-1-thread-2) [n:127.0.0.1:56481_solr 
   ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:56481_solr
   [junit4]   2> 5185 INFO  (jetty-launcher-1-thread-4) [n:127.0.0.1:47037_solr 
   ] o.a.s.c.Overseer Overseer 
(id=97970750554832907-127.0.0.1:47037_solr-n_0000000000) starting
   [junit4]   2> 5242 INFO  
(zkCallback-22-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 5244 INFO  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 5242 INFO  
(zkCallback-21-thread-1-processing-n:127.0.0.1:56481_solr) 
[n:127.0.0.1:56481_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 5244 INFO  
(zkCallback-20-thread-1-processing-n:127.0.0.1:38868_solr) 
[n:127.0.0.1:38868_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 5274 INFO  (jetty-launcher-1-thread-3) [n:127.0.0.1:38868_solr 
   ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:38868_solr
   [junit4]   2> 5294 INFO  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 5295 INFO  
(zkCallback-20-thread-1-processing-n:127.0.0.1:38868_solr) 
[n:127.0.0.1:38868_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 5294 INFO  
(zkCallback-21-thread-1-processing-n:127.0.0.1:56481_solr) 
[n:127.0.0.1:56481_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 5302 INFO  
(zkCallback-22-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 5765 INFO  (jetty-launcher-1-thread-4) [n:127.0.0.1:47037_solr 
   ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:47037_solr
   [junit4]   2> 5769 INFO  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 5769 INFO  
(zkCallback-21-thread-1-processing-n:127.0.0.1:56481_solr) 
[n:127.0.0.1:56481_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 5769 INFO  
(zkCallback-20-thread-1-processing-n:127.0.0.1:38868_solr) 
[n:127.0.0.1:38868_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 5770 INFO  
(zkCallback-22-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 6009 INFO  (jetty-launcher-1-thread-2) [n:127.0.0.1:56481_solr 
   ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/tempDir-001/node2/.
   [junit4]   2> 6014 INFO  (jetty-launcher-1-thread-3) [n:127.0.0.1:38868_solr 
   ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/tempDir-001/node3/.
   [junit4]   2> 6020 INFO  (jetty-launcher-1-thread-1) [n:127.0.0.1:50939_solr 
   ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/tempDir-001/node1/.
   [junit4]   2> 6122 INFO  (jetty-launcher-1-thread-4) [n:127.0.0.1:47037_solr 
   ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/tempDir-001/node4/.
   [junit4]   2> 6245 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 6248 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42404/solr ready
   [junit4]   2> 7804 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your 
platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 9494 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 9941 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via 
org.mortbay.log.Slf4jLog
   [junit4]   2> 9992 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 10144 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 10257 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_51394_hdfs____gyg5jw/webapp
   [junit4]   2> 11670 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:51394
   [junit4]   2> 13415 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 13420 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 13436 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_36164_datanode____.xihzu5/webapp
   [junit4]   2> 13915 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36164
   [junit4]   2> 16335 INFO  (IPC Server handler 4 on 51007) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-b8064d89-1898-4366-a483-32e8cf956eed node 
DatanodeRegistration(127.0.0.1:49158, 
datanodeUuid=b9f1bc5b-fca8-40aa-8f19-9a16cb00a028, infoPort=45162, 
infoSecurePort=0, ipcPort=52400, 
storageInfo=lv=-56;cid=testClusterID;nsid=1417652438;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 3 msecs
   [junit4]   2> 16336 INFO  (IPC Server handler 4 on 51007) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-a6d39e80-814f-429b-8781-9cecbe0c824f node 
DatanodeRegistration(127.0.0.1:49158, 
datanodeUuid=b9f1bc5b-fca8-40aa-8f19-9a16cb00a028, infoPort=45162, 
infoSecurePort=0, ipcPort=52400, 
storageInfo=lv=-56;cid=testClusterID;nsid=1417652438;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 1 msecs
   [junit4]   2> 16810 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testDeleteNonExistentCollection-seed#[1F377C89C3176F6])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testDeleteNonExistentCollection
   [junit4]   2> 16818 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testDeleteNonExistentCollection-seed#[1F377C89C3176F6])
 [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 17427 INFO  (qtp748027989-56) [n:127.0.0.1:56481_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params 
name=unknown_collection&action=DELETE&wt=javabin&version=2 and 
sendToOCPQueue=true
   [junit4]   2> 17455 INFO  
(OverseerThreadFactory-18-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing 
Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 17456 ERROR 
(OverseerThreadFactory-18-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.OverseerCollectionMessageHandler 
Collection: unknown_collection operation: delete 
failed:org.apache.solr.common.SolrException: Could not find collection : 
unknown_collection
   [junit4]   2>        at 
org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:194)
   [junit4]   2>        at 
org.apache.solr.cloud.OverseerCollectionMessageHandler.collectionCmd(OverseerCollectionMessageHandler.java:846)
   [junit4]   2>        at 
org.apache.solr.cloud.DeleteCollectionCmd.call(DeleteCollectionCmd.java:88)
   [junit4]   2>        at 
org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:226)
   [junit4]   2>        at 
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:464)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 17549 INFO  (qtp748027989-56) [n:127.0.0.1:56481_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={name=unknown_collection&action=DELETE&wt=javabin&version=2} status=400 
QTime=172
   [junit4]   2> 17615 INFO  (qtp748027989-28) [n:127.0.0.1:56481_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&collection.configName=conf&name=acollectionafterbaddelete&action=CREATE&numShards=1&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 17646 INFO  
(OverseerCollectionConfigSetProcessor-97970750554832907-127.0.0.1:47037_solr-n_0000000000)
 [n:127.0.0.1:47037_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 17652 INFO  
(OverseerThreadFactory-18-thread-2-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
acollectionafterbaddelete
   [junit4]   2> 17881 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr    ] 
o.a.s.u.TestInjection Inject random core creation delay of 5s
   [junit4]   2> 17894 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr    ] 
o.a.s.u.TestInjection Inject random core creation delay of 5s
   [junit4]   2> 22881 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica2&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2
   [junit4]   2> 22887 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
   [junit4]   2> 22895 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica1&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2
   [junit4]   2> 22897 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
   [junit4]   2> 22904 INFO  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 22904 INFO  
(zkCallback-22-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 23015 INFO  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 23015 INFO  
(zkCallback-22-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 23985 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.SolrConfig Using Lucene 
MatchVersion: 6.7.0
   [junit4]   2> 23989 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.SolrConfig Using Lucene 
MatchVersion: 6.7.0
   [junit4]   2> 24118 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.s.IndexSchema 
[acollectionafterbaddelete_shard1_replica2] Schema name=minimal
   [junit4]   2> 24127 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.s.IndexSchema 
[acollectionafterbaddelete_shard1_replica1] Schema name=minimal
   [junit4]   2> 24234 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.s.IndexSchema Loaded schema 
minimal/1.1 with uniqueid field id
   [junit4]   2> 24235 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.s.IndexSchema Loaded schema 
minimal/1.1 with uniqueid field id
   [junit4]   2> 24235 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.CoreContainer Creating 
SolrCore 'acollectionafterbaddelete_shard1_replica1' using configuration from 
collection acollectionafterbaddelete, trusted=true
   [junit4]   2> 24235 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.CoreContainer Creating 
SolrCore 'acollectionafterbaddelete_shard1_replica2' using configuration from 
collection acollectionafterbaddelete, trusted=true
   [junit4]   2> 24262 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:51007/data
   [junit4]   2> 24262 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Solr 
Kerberos Authentication disabled
   [junit4]   2> 24263 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 24264 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:51007/data
   [junit4]   2> 24265 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Solr 
Kerberos Authentication disabled
   [junit4]   2> 24265 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 24270 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.SolrCore 
[[acollectionafterbaddelete_shard1_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/tempDir-001/node1/acollectionafterbaddelete_shard1_replica1],
 
dataDir=[hdfs://localhost:51007/data/acollectionafterbaddelete/core_node2/data/]
   [junit4]   2> 24270 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.SolrCore 
[[acollectionafterbaddelete_shard1_replica2] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/tempDir-001/node4/acollectionafterbaddelete_shard1_replica2],
 
dataDir=[hdfs://localhost:51007/data/acollectionafterbaddelete/core_node1/data/]
   [junit4]   2> 24290 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.JmxMonitoredMap JMX 
monitoring is enabled. Adding Solr mbeans to JMX Server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@5ac59ebc
   [junit4]   2> 24295 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.JmxMonitoredMap JMX 
monitoring is enabled. Adding Solr mbeans to JMX Server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@5ac59ebc
   [junit4]   2> 24370 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.HdfsDirectoryFactory 
creating directory factory for path 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node1/data/snapshot_metadata
   [junit4]   2> 24371 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.HdfsDirectoryFactory 
creating directory factory for path 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node2/data/snapshot_metadata
   [junit4]   2> 24438 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.HdfsDirectoryFactory 
Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 24438 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.HdfsDirectoryFactory 
Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 24446 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [4194304] will allocate [1] slabs and 
use ~[4194304] bytes
   [junit4]   2> 24446 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.HdfsDirectoryFactory 
Creating new single instance HDFS BlockCache
   [junit4]   2> 24446 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [4194304] will allocate [1] slabs and 
use ~[4194304] bytes
   [junit4]   2> 24446 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.HdfsDirectoryFactory 
Creating new single instance HDFS BlockCache
   [junit4]   2> 25573 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.s.b.BlockDirectory Block 
cache on write is disabled
   [junit4]   2> 25576 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.s.b.BlockDirectory Block 
cache on write is disabled
   [junit4]   2> 25612 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.HdfsDirectoryFactory 
creating directory factory for path 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node2/data
   [junit4]   2> 25624 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.HdfsDirectoryFactory 
creating directory factory for path 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node1/data
   [junit4]   2> 25719 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.HdfsDirectoryFactory 
creating directory factory for path 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node2/data/index
   [junit4]   2> 25758 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.HdfsDirectoryFactory 
creating directory factory for path 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node1/data/index
   [junit4]   2> 25761 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.HdfsDirectoryFactory 
Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 25761 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [4194304] will allocate [1] slabs and 
use ~[4194304] bytes
   [junit4]   2> 25761 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.HdfsDirectoryFactory 
Creating new single instance HDFS BlockCache
   [junit4]   2> 25782 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.HdfsDirectoryFactory 
Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 25783 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Block 
cache target memory usage, slab size of [4194304] will allocate [1] slabs and 
use ~[4194304] bytes
   [junit4]   2> 25783 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.HdfsDirectoryFactory 
Creating new single instance HDFS BlockCache
   [junit4]   2> 25823 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.s.b.BlockDirectory Block 
cache on write is disabled
   [junit4]   2> 25827 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.s.b.BlockDirectory Block 
cache on write is disabled
   [junit4]   2> 26465 INFO  (IPC Server handler 2 on 51007) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49158 is 
added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-b8064d89-1898-4366-a483-32e8cf956eed:NORMAL:127.0.0.1:49158|RBW]]}
 size 71
   [junit4]   2> 26469 INFO  (IPC Server handler 4 on 51007) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49158 is 
added to blk_1073741826_1002{UCState=COMMITTED, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-b8064d89-1898-4366-a483-32e8cf956eed:NORMAL:127.0.0.1:49158|RBW]]}
 size 71
   [junit4]   2> 28036 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.UpdateHandler Using 
UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 28036 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 28036 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 28042 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.UpdateHandler Using 
UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 28053 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.UpdateLog Initializing 
UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 28070 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 28123 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.CommitTracker Hard 
AutoCommit: disabled
   [junit4]   2> 28124 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.CommitTracker Soft 
AutoCommit: disabled
   [junit4]   2> 28130 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.CommitTracker Hard 
AutoCommit: disabled
   [junit4]   2> 28135 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.CommitTracker Soft 
AutoCommit: disabled
   [junit4]   2> 28325 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@6b6448df[acollectionafterbaddelete_shard1_replica1] main]
   [junit4]   2> 28330 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@267095ea[acollectionafterbaddelete_shard1_replica2] main]
   [junit4]   2> 28368 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 28375 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.r.ManagedResourceStorage 
Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 28424 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.r.ManagedResourceStorage 
Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 28425 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.r.ManagedResourceStorage 
Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 28451 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.h.ReplicationHandler Commits 
will be reserved for  10000
   [junit4]   2> 28454 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.h.ReplicationHandler Commits 
will be reserved for  10000
   [junit4]   2> 28614 INFO  
(searcherExecutor-27-thread-1-processing-n:127.0.0.1:50939_solr 
x:acollectionafterbaddelete_shard1_replica1 s:shard1 
c:acollectionafterbaddelete) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.SolrCore 
[acollectionafterbaddelete_shard1_replica1] Registered new searcher 
Searcher@6b6448df[acollectionafterbaddelete_shard1_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 28615 INFO  
(searcherExecutor-28-thread-1-processing-n:127.0.0.1:47037_solr 
x:acollectionafterbaddelete_shard1_replica2 s:shard1 
c:acollectionafterbaddelete) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.SolrCore 
[acollectionafterbaddelete_shard1_replica2] Registered new searcher 
Searcher@267095ea[acollectionafterbaddelete_shard1_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 28670 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.UpdateLog Could not find 
max version in index or recent updates, using new clock 1567532037921832960
   [junit4]   2> 28673 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.u.UpdateLog Could not find 
max version in index or recent updates, using new clock 1567532037926027264
   [junit4]   2> 28727 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.ShardLeaderElectionContext 
Enough replicas found to continue.
   [junit4]   2> 28728 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.ShardLeaderElectionContext 
I may be the new leader - try and sync
   [junit4]   2> 28730 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.SyncStrategy Sync replicas 
to https://127.0.0.1:47037/solr/acollectionafterbaddelete_shard1_replica2/
   [junit4]   2> 28735 INFO  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 28735 INFO  
(zkCallback-22-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 28741 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.PeerSync PeerSync: 
core=acollectionafterbaddelete_shard1_replica2 url=https://127.0.0.1:47037/solr 
START 
replicas=[https://127.0.0.1:50939/solr/acollectionafterbaddelete_shard1_replica1/]
 nUpdates=100
   [junit4]   2> 28742 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.u.PeerSync PeerSync: 
core=acollectionafterbaddelete_shard1_replica2 url=https://127.0.0.1:47037/solr 
DONE.  We have no versions.  sync failed.
   [junit4]   2> 28785 INFO  (qtp351731691-50) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1 r:core_node2 
x:acollectionafterbaddelete_shard1_replica1] o.a.s.c.S.Request 
[acollectionafterbaddelete_shard1_replica1]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=4
   [junit4]   2> 28788 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.SyncStrategy Leader's 
attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 28788 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.ShardLeaderElectionContext 
We failed sync, but we have no versions - we can't sync in that case - we were 
active before, so become leader anyway
   [junit4]   2> 28788 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.ShardLeaderElectionContext 
Found all replicas participating in election, clear LIR
   [junit4]   2> 28811 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.ShardLeaderElectionContext 
I am the new leader: 
https://127.0.0.1:47037/solr/acollectionafterbaddelete_shard1_replica2/ shard1
   [junit4]   2> 28917 INFO  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 28917 INFO  
(zkCallback-22-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 28966 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.c.ZkController I am the 
leader, no recovery necessary
   [junit4]   2> 28971 INFO  (qtp259325941-35) [n:127.0.0.1:47037_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica2] o.a.s.s.HttpSolrCall [admin] 
webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica2&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2}
 status=0 QTime=11093
   [junit4]   2> 29079 INFO  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 29080 INFO  
(zkCallback-22-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 29738 INFO  (qtp351731691-45) [n:127.0.0.1:50939_solr 
c:acollectionafterbaddelete s:shard1  
x:acollectionafterbaddelete_shard1_replica1] o.a.s.s.HttpSolrCall [admin] 
webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=acollectionafterbaddelete_shard1_replica1&action=CREATE&numShards=1&collection=acollectionafterbaddelete&shard=shard1&wt=javabin&version=2}
 status=0 QTime=11845
   [junit4]   2> 29746 INFO  (qtp748027989-28) [n:127.0.0.1:56481_solr    ] 
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 29846 INFO  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 29846 INFO  
(zkCallback-22-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/acollectionafterbaddelete/state.json] for collection 
[acollectionafterbaddelete] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 30748 INFO  (qtp748027989-28) [n:127.0.0.1:56481_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=2&collection.configName=conf&name=acollectionafterbaddelete&action=CREATE&numShards=1&wt=javabin&version=2}
 status=0 QTime=13133
   [junit4]   2> 30756 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testDeleteNonExistentCollection-seed#[1F377C89C3176F6])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testDeleteNonExistentCollection
   [junit4]   2> 30816 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[1F377C89C3176F6])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testCoresAreDistributedAcrossNodes
   [junit4]   2> 30818 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes-seed#[1F377C89C3176F6])
 [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 30823 INFO  (qtp748027989-44) [n:127.0.0.1:56481_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params 
name=acollectionafterbaddelete&action=DELETE&wt=javabin&version=2 and 
sendToOCPQueue=true
   [junit4]   2> 30827 INFO  
(OverseerThreadFactory-18-thread-3-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing 
Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 30827 INFO  
(OverseerCollectionConfigSetProcessor-97970750554832907-127.0.0.1:47037_solr-n_0000000000)
 [n:127.0.0.1:47037_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 30835 INFO  (qtp259325941-34) [n:127.0.0.1:47037_solr    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: 
solr.core.acollectionafterbaddelete.shard1.replica2
   [junit4]   2> 30836 INFO  (qtp259325941-34) [n:127.0.0.1:47037_solr    ] 
o.a.s.c.SolrCore [acollectionafterbaddelete_shard1_replica2]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@5af8d969
   [junit4]   2> 30840 INFO  (qtp351731691-48) [n:127.0.0.1:50939_solr    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: 
solr.core.acollectionafterbaddelete.shard1.replica1
   [junit4]   2> 30842 INFO  (qtp351731691-48) [n:127.0.0.1:50939_solr    ] 
o.a.s.c.SolrCore [acollectionafterbaddelete_shard1_replica1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@581ed413
   [junit4]   2> 31355 INFO  (qtp259325941-34) [n:127.0.0.1:47037_solr    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: 
solr.core.acollectionafterbaddelete.shard1.replica2
   [junit4]   2> 31397 INFO  (qtp351731691-48) [n:127.0.0.1:50939_solr    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: 
solr.core.acollectionafterbaddelete.shard1.replica1
   [junit4]   2> 31405 INFO  (qtp351731691-48) [n:127.0.0.1:50939_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node2/data/snapshot_metadata
   [junit4]   2> 31426 INFO  (qtp351731691-48) [n:127.0.0.1:50939_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node2/data/index
   [junit4]   2> 31427 INFO  (qtp351731691-48) [n:127.0.0.1:50939_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node2/data
   [junit4]   2> 31453 INFO  (IPC Server handler 9 on 51007) [    ] 
BlockStateChange BLOCK* addToInvalidates: blk_1073741826_1002 127.0.0.1:49158 
   [junit4]   2> 31458 INFO  (qtp259325941-34) [n:127.0.0.1:47037_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node1/data/snapshot_metadata
   [junit4]   2> 31462 INFO  (qtp259325941-34) [n:127.0.0.1:47037_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node1/data/index
   [junit4]   2> 31463 INFO  (qtp259325941-34) [n:127.0.0.1:47037_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:51007/data/acollectionafterbaddelete/core_node1/data
   [junit4]   2> 31483 INFO  (IPC Server handler 1 on 51007) [    ] 
BlockStateChange BLOCK* addToInvalidates: blk_1073741825_1001 127.0.0.1:49158 
   [junit4]   2> 31505 INFO  (qtp351731691-48) [n:127.0.0.1:50939_solr    ] 
o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader 
parent node, won't remove previous leader registration.
   [junit4]   2> 31508 WARN  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.LeaderElector Our node is no longer in 
line to be leader
   [junit4]   2> 31523 INFO  (qtp259325941-34) [n:127.0.0.1:47037_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={deleteInstanceDir=true&core=acollectionafterbaddelete_shard1_replica2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
 status=0 QTime=689
   [junit4]   2> 31524 INFO  (qtp351731691-48) [n:127.0.0.1:50939_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={deleteInstanceDir=true&core=acollectionafterbaddelete_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
 status=0 QTime=684
   [junit4]   2> 32234 INFO  (qtp748027989-44) [n:127.0.0.1:56481_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={name=acollectionafterbaddelete&action=DELETE&wt=javabin&version=2} 
status=0 QTime=1411
   [junit4]   2> 32241 INFO  (qtp748027989-47) [n:127.0.0.1:56481_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&collection.configName=conf&name=nodes_used_collection&action=CREATE&numShards=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 32254 INFO  
(OverseerThreadFactory-18-thread-4-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
nodes_used_collection
   [junit4]   2> 32255 INFO  
(OverseerCollectionConfigSetProcessor-97970750554832907-127.0.0.1:47037_solr-n_0000000000)
 [n:127.0.0.1:47037_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 32469 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr    ] 
o.a.s.u.TestInjection Inject random core creation delay of 5s
   [junit4]   2> 32481 INFO  (qtp351731691-49) [n:127.0.0.1:50939_solr    ] 
o.a.s.u.TestInjection Inject random core creation delay of 5s
   [junit4]   2> 32533 INFO  (qtp748027989-28) [n:127.0.0.1:56481_solr    ] 
o.a.s.u.TestInjection Inject random core creation delay of 5s
   [junit4]   2> 32540 INFO  (qtp1728100720-61) [n:127.0.0.1:38868_solr    ] 
o.a.s.u.TestInjection Inject random core creation delay of 5s
   [junit4]   2> 34151 INFO  
(org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@1e5ee701)
 [    ] BlockStateChange BLOCK* BlockManager: ask 127.0.0.1:49158 to delete 
[blk_1073741825_1001, blk_1073741826_1002]
   [junit4]   2> 37470 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 37481 INFO  (qtp351731691-49) [n:127.0.0.1:50939_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 37533 INFO  (qtp748027989-28) [n:127.0.0.1:56481_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard2_replica1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 37534 INFO  (qtp748027989-28) [n:127.0.0.1:56481_solr    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
   [junit4]   2> 37540 INFO  (qtp1728100720-61) [n:127.0.0.1:38868_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=nodes_used_collection_shard1_replica1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 37541 INFO  (qtp1728100720-61) [n:127.0.0.1:38868_solr    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
   [junit4]   2> 37579 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 37614 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.s.IndexSchema [nodes_used_collection_shard1_replica2] Schema name=minimal
   [junit4]   2> 37620 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 37620 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard1_replica2' 
using configuration from collection nodes_used_collection, trusted=true
   [junit4]   2> 37621 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:51007/data
   [junit4]   2> 37621 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 37621 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 37621 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.SolrCore [[nodes_used_collection_shard1_replica2] ] Opening new 
SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/tempDir-001/node4/nodes_used_collection_shard1_replica2],
 dataDir=[hdfs://localhost:51007/data/nodes_used_collection/core_node1/data/]
   [junit4]   2> 37623 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ac59ebc
   [junit4]   2> 37623 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:51007/data/nodes_used_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 37633 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 37634 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 37634 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 37641 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 37643 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:51007/data/nodes_used_collection/core_node1/data
   [junit4]   2> 37647 INFO  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 37647 INFO  
(zkCallback-20-thread-1-processing-n:127.0.0.1:38868_solr) 
[n:127.0.0.1:38868_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 37647 INFO  
(zkCallback-21-thread-1-processing-n:127.0.0.1:56481_solr) 
[n:127.0.0.1:56481_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 37648 INFO  
(zkCallback-22-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 37695 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:51007/data/nodes_used_collection/core_node1/data/index
   [junit4]   2> 37712 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 37712 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[4194304] will allocate [1] slabs and use ~[4194304] bytes
   [junit4]   2> 37712 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 37729 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 37850 INFO  (IPC Server handler 4 on 51007) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49158 is 
added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-a6d39e80-814f-429b-8781-9cecbe0c824f:NORMAL:127.0.0.1:49158|RBW]]}
 size 0
   [junit4]   2> 38047 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 38047 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 38047 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 38082 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 38083 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 38148 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@10c58559[nodes_used_collection_shard1_replica2] main]
   [junit4]   2> 38150 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 38150 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 38151 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 38153 INFO  
(searcherExecutor-37-thread-1-processing-n:127.0.0.1:47037_solr 
x:nodes_used_collection_shard1_replica2 s:shard1 c:nodes_used_collection) 
[n:127.0.0.1:47037_solr c:nodes_used_collection s:shard1  
x:nodes_used_collection_shard1_replica2] o.a.s.c.SolrCore 
[nodes_used_collection_shard1_replica2] Registered new searcher 
Searcher@10c58559[nodes_used_collection_shard1_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 38160 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1567532047873867776
   [junit4]   2> 38171 INFO  (qtp259325941-29) [n:127.0.0.1:47037_solr 
c:nodes_used_collection s:shard1  x:nodes_used_collection_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 38275 INFO  
(zkCallback-20-thread-1-processing-n:127.0.0.1:38868_solr) 
[n:127.0.0.1:38868_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 38276 INFO  
(zkCallback-21-thread-1-processing-n:127.0.0.1:56481_solr) 
[n:127.0.0.1:56481_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 38276 INFO  
(zkCallback-22-thread-1-processing-n:127.0.0.1:47037_solr) 
[n:127.0.0.1:47037_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 38275 INFO  
(zkCallback-23-thread-1-processing-n:127.0.0.1:50939_solr) 
[n:127.0.0.1:50939_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/nodes_used_collection/state.json] for collection 
[nodes_used_collection] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 38541 INFO  (qtp351731691-49) [n:127.0.0.1:50939_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 38565 INFO  (qtp351731691-49) [n:127.0.0.1:50939_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.s.IndexSchema [nodes_used_collection_shard2_replica2] Schema name=minimal
   [junit4]   2> 38573 INFO  (qtp351731691-49) [n:127.0.0.1:50939_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 38573 INFO  (qtp351731691-49) [n:127.0.0.1:50939_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.c.CoreContainer Creating SolrCore 'nodes_used_collection_shard2_replica2' 
using configuration from collection nodes_used_collection, trusted=true
   [junit4]   2> 38574 INFO  (qtp351731691-49) [n:127.0.0.1:50939_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:51007/data
   [junit4]   2> 38622 INFO  (qtp351731691-49) [n:127.0.0.1:50939_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 38622 INFO  (qtp351731691-49) [n:127.0.0.1:50939_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 38622 INFO  (qtp351731691-49) [n:127.0.0.1:50939_solr 
c:nodes_used_collection s:shard2  x:nodes_used_collection_shard2_replica2] 
o.a.s.c.SolrCore [[nodes_used_collection_shard2_replica2] ] Opening new 
SolrCore at [/x1/jenkins/

[...truncated too long message...]

ir-002/hdfsBaseDir/data/data1/, 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/tempDir-002/hdfsBaseDir/data/data2/]]
  heartbeating to localhost/127.0.0.1:51007) [    ] o.a.h.h.s.d.DataNode 
BPOfferService for Block pool BP-710556347-10.41.0.5-1494915024690 (Datanode 
Uuid b9f1bc5b-fca8-40aa-8f19-9a16cb00a028) service to localhost/127.0.0.1:51007 
interrupted
   [junit4]   2> 234858 WARN  (DataNode: 
[[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/tempDir-002/hdfsBaseDir/data/data1/,
 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001/tempDir-002/hdfsBaseDir/data/data2/]]
  heartbeating to localhost/127.0.0.1:51007) [    ] o.a.h.h.s.d.DataNode Ending 
block pool service for: Block pool BP-710556347-10.41.0.5-1494915024690 
(Datanode Uuid b9f1bc5b-fca8-40aa-8f19-9a16cb00a028) service to 
localhost/127.0.0.1:51007
   [junit4]   2> 234886 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 234888 ERROR 
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.h.m.l.MethodMetric Error invoking method getBlocksTotal
   [junit4]   2> java.lang.reflect.InvocationTargetException
   [junit4]   2>        at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown 
Source)
   [junit4]   2>        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:401)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:194)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2>        at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:203)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.teardownClass(HdfsCollectionsAPIDistributedZkTest.java:77)
   [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2>        ... 53 more
   [junit4]   2> 234891 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42404 42404
   [junit4]   2> 234892 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1F377C89C3176F6]-worker) [    
] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1F377C89C3176F6-001
   [junit4]   2> May 16, 2017 6:14:10 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 19 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{_version_=PostingsFormat(name=Memory doPackFST= true), 
id=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, 
maxPointsInLeafNode=1839, maxMBSortInHeap=5.499579560460535, 
sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=lt-LT, 
timezone=Africa/Khartoum
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_121 (64-bit)/cpus=4,threads=6,free=172563632,total=403177472
   [junit4]   2> NOTE: All tests run in this JVM: 
[TestJavabinTupleStreamParser, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [39/711 (1!)] on J1 in 244.26s, 21 tests, 1 failure <<< 
FAILURES!

[...truncated 9614 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/build.xml:783:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/build.xml:727:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid1694.hprof

Total time: 345 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to