Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1413/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.update.HdfsTransactionLog  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)  
at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)  at 
org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:161)  at 
org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:116)  at 
org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:108)
  at sun.reflect.GeneratedConstructorAccessor137.newInstance(Unknown Source)  
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)  at 
org.apache.solr.core.SolrCore.createInstance(SolrCore.java:773)  at 
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:835)  at 
org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)  at 
org.apache.solr.core.SolrCore.<init>(SolrCore.java:958)  at 
org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)  at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:969)  at 
org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)  at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
 at java.lang.Thread.run(Thread.java:748)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not 
released!!! [HdfsTransactionLog]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.update.HdfsTransactionLog
        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
        at 
org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)
        at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)
        at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:161)
        at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:116)
        at 
org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:108)
        at sun.reflect.GeneratedConstructorAccessor137.newInstance(Unknown 
Source)
        at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:773)
        at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:835)
        at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:958)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:969)
        at 
org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)
        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)


        at __randomizedtesting.SeedInfo.seed([F63395BBD76CFEF0]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:296)
        at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.handler.admin.MBeansHandlerTest.testDiff

Error Message:


Stack Trace:
java.lang.NullPointerException
        at 
__randomizedtesting.SeedInfo.seed([F63395BBD76CFEF0:33255120C7DAC690]:0)
        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.diffObject(SolrInfoMBeanHandler.java:240)
        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.diffNamedList(SolrInfoMBeanHandler.java:219)
        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.getDiff(SolrInfoMBeanHandler.java:187)
        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:87)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)
        at org.apache.solr.util.TestHarness.query(TestHarness.java:337)
        at org.apache.solr.util.TestHarness.query(TestHarness.java:319)
        at 
org.apache.solr.handler.admin.MBeansHandlerTest.testDiff(MBeansHandlerTest.java:57)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 11528 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.MBeansHandlerTest
   [junit4]   2> Creating dataDir: 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.MBeansHandlerTest_F63395BBD76CFEF0-001/init-core-data-001
   [junit4]   2> 926171 WARN  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=56 numCloses=56
   [junit4]   2> 926171 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 926173 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 926173 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 926173 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: 
[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib,
 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 926223 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 926241 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 926539 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 926578 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@63c22478
   [junit4]   2> 926581 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@63c22478
   [junit4]   2> 926581 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@63c22478
   [junit4]   2> 926582 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, 
from paths: 
[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib,
 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 926606 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 926623 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 926716 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 926721 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using 
configuration from instancedir 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1,
 trusted=true
   [junit4]   2> 926721 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@63c22478
   [junit4]   2> 926721 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 926721 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1],
 
dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.MBeansHandlerTest_F63395BBD76CFEF0-001/init-core-data-001/]
   [junit4]   2> 926722 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=16, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=35.7890625, 
floorSegmentMB=2.158203125, forceMergeDeletesPctAllowed=27.711768356853227, 
segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 926786 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 926786 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= 
defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 
numVersionBuckets=65536
   [junit4]   2> 926786 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 926786 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 926787 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=37, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=1.0]
   [junit4]   2> 926787 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@3153effe[collection1] main]
   [junit4]   2> 926787 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 926790 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.s.DirectSolrSpellChecker init: 
{name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 926802 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 926802 INFO  
(searcherExecutor-4890-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 926802 INFO  (coreLoadExecutor-4889-thread-1) [    
x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent 
updates, using new clock 1571677967085469696
   [junit4]   2> 926804 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 926806 INFO  
(TEST-MBeansHandlerTest.testDiff-seed#[F63395BBD76CFEF0]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testDiff
   [junit4]   2> 926808 INFO  
(searcherExecutor-4890-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 926808 INFO  
(searcherExecutor-4890-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 926808 INFO  
(searcherExecutor-4890-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: 
multipleFields
   [junit4]   2> 926809 INFO  
(searcherExecutor-4890-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 926809 INFO  
(searcherExecutor-4890-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 926809 INFO  
(searcherExecutor-4890-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 926810 INFO  
(searcherExecutor-4890-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 926810 INFO  
(searcherExecutor-4890-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 926811 INFO  
(searcherExecutor-4890-thread-1-processing-x:collection1) [    x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@3153effe[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 926812 INFO  
(TEST-MBeansHandlerTest.testDiff-seed#[F63395BBD76CFEF0]) [    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
   [junit4]   2> 926813 INFO  
(TEST-MBeansHandlerTest.testDiff-seed#[F63395BBD76CFEF0]) [    ] 
o.a.s.c.S.Request [collection1]  webapp=null path=null 
params={qt=/admin/mbeans&stats=true&wt=xml} status=0 QTime=6
   [junit4]   2> 926850 ERROR 
(TEST-MBeansHandlerTest.testDiff-seed#[F63395BBD76CFEF0]) [    ] 
o.a.s.h.RequestHandlerBase java.lang.NullPointerException
   [junit4]   2>        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.diffObject(SolrInfoMBeanHandler.java:240)
   [junit4]   2>        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.diffNamedList(SolrInfoMBeanHandler.java:219)
   [junit4]   2>        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.getDiff(SolrInfoMBeanHandler.java:187)
   [junit4]   2>        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:87)
   [junit4]   2>        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2>        at 
org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)
   [junit4]   2>        at 
org.apache.solr.util.TestHarness.query(TestHarness.java:337)
   [junit4]   2>        at 
org.apache.solr.util.TestHarness.query(TestHarness.java:319)
   [junit4]   2>        at 
org.apache.solr.handler.admin.MBeansHandlerTest.testDiff(MBeansHandlerTest.java:57)
   [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 926850 INFO  
(TEST-MBeansHandlerTest.testDiff-seed#[F63395BBD76CFEF0]) [    ] 
o.a.s.c.S.Request [collection1]  webapp=null path=null 
params={qt=/admin/mbeans&stats=true&diff=true&wt=xml} status=500 QTime=36
   [junit4]   2> 926850 INFO  
(TEST-MBeansHandlerTest.testDiff-seed#[F63395BBD76CFEF0]) [    ] 
o.a.s.SolrTestCaseJ4 ###Ending testDiff
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=MBeansHandlerTest 
-Dtests.method=testDiff -Dtests.seed=F63395BBD76CFEF0 -Dtests.slow=true 
-Dtests.locale=th-TH-u-nu-thai-x-lvariant-TH -Dtests.timezone=GB 
-Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.05s J1 | MBeansHandlerTest.testDiff <<<
   [junit4]    > Throwable #1: java.lang.NullPointerException
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([F63395BBD76CFEF0:33255120C7DAC690]:0)
   [junit4]    >        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.diffObject(SolrInfoMBeanHandler.java:240)
   [junit4]    >        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.diffNamedList(SolrInfoMBeanHandler.java:219)
   [junit4]    >        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.getDiff(SolrInfoMBeanHandler.java:187)
   [junit4]    >        at 
org.apache.solr.handler.admin.SolrInfoMBeanHandler.handleRequestBody(SolrInfoMBeanHandler.java:87)
   [junit4]    >        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]    >        at 
org.apache.solr.core.SolrCore.execute(SolrCore.java:2473)
   [junit4]    >        at 
org.apache.solr.util.TestHarness.query(TestHarness.java:337)
   [junit4]    >        at 
org.apache.solr.util.TestHarness.query(TestHarness.java:319)
   [junit4]    >        at 
org.apache.solr.handler.admin.MBeansHandlerTest.testDiff(MBeansHandlerTest.java:57)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 926854 INFO  
(TEST-MBeansHandlerTest.testXMLDiffWithExternalEntity-seed#[F63395BBD76CFEF0]) 
[    ] o.a.s.SolrTestCaseJ4 ###Starting testXMLDiffWithExternalEntity
   [junit4]   2> 926890 INFO  
(TEST-MBeansHandlerTest.testXMLDiffWithExternalEntity-seed#[F63395BBD76CFEF0]) 
[    ] o.a.s.SolrTestCaseJ4 ###Ending testXMLDiffWithExternalEntity
   [junit4]   2> 926890 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 926890 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=27050820
   [junit4]   2> 926890 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 926891 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 926892 INFO  
(SUITE-MBeansHandlerTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 926892 INFO  (coreCloseExecutor-4895-thread-1) [    
x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@74f47806
   [junit4]   2> 926892 INFO  (coreCloseExecutor-4895-thread-1) [    
x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.core.collection1, tag=1962178566
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.MBeansHandlerTest_F63395BBD76CFEF0-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, 
sim=RandomSimilarity(queryNorm=true): {}, locale=th-TH-u-nu-thai-x-lvariant-TH, 
timezone=GB
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_131 
(64-bit)/cpus=3,threads=1,free=69808416,total=383311872
   [junit4]   2> NOTE: All tests run in this JVM: 
[BigEndianAscendingWordSerializerTest, LoggingHandlerTest, 
PeerSyncWithIndexFingerprintCachingTest, TestStressCloudBlindAtomicUpdates, 
TestReloadDeadlock, TestPerFieldSimilarity, 
VMParamsZkACLAndCredentialsProvidersTest, UpdateParamsTest, TestRTGBase, 
TestSizeLimitedDistributedMap, TestDistribDocBasedVersion, 
TestMultiValuedNumericRangeQuery, ReplicationFactorTest, 
TestRandomRequestDistribution, TestClassNameShortening, CacheHeaderTest, 
SortSpecParsingTest, SchemaVersionSpecificBehaviorTest, 
OpenExchangeRatesOrgProviderTest, UUIDUpdateProcessorFallbackTest, 
BlockJoinFacetDistribTest, TestSubQueryTransformerCrossCore, TestRestManager, 
DeleteStatusTest, TestRuleBasedAuthorizationPlugin, TestMaxScoreQueryParser, 
ConnectionManagerTest, TestCollectionAPI, TestReplicaProperties, 
TestRequestStatusCollectionAPI, TestDistributedGrouping, TestSolrQueryParser, 
TestAuthorizationFramework, TestBlobHandler, BlockJoinFacetSimpleTest, 
DocExpirationUpdateProcessorFactoryTest, DocValuesMultiTest, 
RollingRestartTest, TestNRTOpen, SpellCheckCollatorTest, 
CoreAdminRequestStatusTest, TestPushWriter, HttpPartitionTest, 
BasicAuthStandaloneTest, TestSolrCloudWithKerberosAlt, TestStressUserVersions, 
DirectSolrSpellCheckerTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, 
TestJsonRequest, TestFreeTextSuggestions, TestManagedResource, 
TestRandomFaceting, TestSolrConfigHandlerCloud, TestBinaryResponseWriter, 
HdfsBasicDistributedZkTest, TestLMJelinekMercerSimilarityFactory, 
TestUseDocValuesAsStored, TestDocBasedVersionConstraints, 
FileBasedSpellCheckerTest, TestMaxTokenLenTokenizer, TestSegmentSorting, 
TestSuggestSpellingConverter, BooleanFieldTest, TestJettySolrRunner, 
TestPartialUpdateDeduplication, TestCoreDiscovery, 
TestPullReplicaErrorHandling, TestSimpleQParserPlugin, TestBadConfig, 
TestIndexSearcher, WrapperMergePolicyFactoryTest, 
TestWordDelimiterFilterFactory, TermVectorComponentTest, CurrencyFieldTypeTest, 
SynonymTokenizerTest, JSONWriterTest, TestSQLHandler, TestInfoStreamLogging, 
DistributedFacetPivotLongTailTest, TestPostingsSolrHighlighter, 
TestSolr4Spatial2, TestQueryTypes, MetricsConfigTest, TestSortByMinMaxFunction, 
DistributedSuggestComponentTest, TestInitQParser, SparseHLLTest, 
DocValuesMissingTest, SpellCheckComponentTest, EchoParamsTest, 
AnalyticsMergeStrategyTest, OverriddenZkACLAndCredentialsProvidersTest, 
ShowFileRequestHandlerTest, TestStressLucene, TestLegacyFieldCache, 
FullSolrCloudDistribCmdsTest, TestDefaultStatsCache, TestFastOutputStream, 
TestSchemaManager, TlogReplayBufferedWhileIndexingTest, 
DistributedQueryComponentCustomSortTest, OverseerStatusTest, CopyFieldTest, 
ExitableDirectoryReaderTest, CdcrReplicationDistributedZkTest, TestFastWriter, 
SolrPluginUtilsTest, DeleteLastCustomShardedReplicaTest, CustomCollectionTest, 
TestTolerantUpdateProcessorCloud, SystemInfoHandlerTest, TestQueryUtils, 
HdfsChaosMonkeySafeLeaderTest, TestLockTree, TestAnalyzedSuggestions, 
HdfsLockFactoryTest, TestReRankQParserPlugin, TestTrackingShardHandlerFactory, 
TestSolr4Spatial, TestSolrCloudWithHadoopAuthPlugin, SolrIndexConfigTest, 
IndexSchemaRuntimeFieldTest, SolrMetricManagerTest, TestLegacyNumericUtils, 
AssignTest, TestRawTransformer, MigrateRouteKeyTest, 
TestPerFieldSimilarityWithDefaultOverride, DeleteReplicaTest, 
PropertiesRequestHandlerTest, TestSearchPerf, ExplicitHLLTest, 
HdfsTlogReplayBufferedWhileIndexingTest, ChaosMonkeySafeLeaderTest, 
RemoteQueryErrorTest, TestSimpleTrackingShardHandler, TestSolrConfigHandler, 
MBeansHandlerTest]
   [junit4] Completed [222/724 (1!)] on J1 in 0.74s, 2 tests, 1 error <<< 
FAILURES!

[...truncated 1118 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest
   [junit4]   2> Creating dataDir: 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_F63395BBD76CFEF0-001/init-core-data-001
   [junit4]   2> 2882266 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=15 numCloses=15
   [junit4]   2> 2882266 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) 
w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2882267 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 2882268 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_F63395BBD76CFEF0-001/tempDir-001
   [junit4]   2> 2882268 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2882268 INFO  (Thread-6921) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2882268 INFO  (Thread-6921) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 2882270 ERROR (Thread-6921) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2882369 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:64446
   [junit4]   2> 2882378 INFO  (jetty-launcher-4049-thread-2) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2882378 INFO  (jetty-launcher-4049-thread-1) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2882380 INFO  (jetty-launcher-4049-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@2757f645{/solr,null,AVAILABLE}
   [junit4]   2> 2882380 INFO  (jetty-launcher-4049-thread-1) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@10b3e7d9{SSL,[ssl, 
http/1.1]}{127.0.0.1:60154}
   [junit4]   2> 2882380 INFO  (jetty-launcher-4049-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@707867e{/solr,null,AVAILABLE}
   [junit4]   2> 2882381 INFO  (jetty-launcher-4049-thread-1) [    ] 
o.e.j.s.Server Started @2885050ms
   [junit4]   2> 2882381 INFO  (jetty-launcher-4049-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=60154}
   [junit4]   2> 2882381 ERROR (jetty-launcher-4049-thread-1) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2882381 INFO  (jetty-launcher-4049-thread-2) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@2b562ec9{SSL,[ssl, 
http/1.1]}{127.0.0.1:47648}
   [junit4]   2> 2882381 INFO  (jetty-launcher-4049-thread-2) [    ] 
o.e.j.s.Server Started @2885051ms
   [junit4]   2> 2882381 INFO  (jetty-launcher-4049-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.0.0
   [junit4]   2> 2882381 INFO  (jetty-launcher-4049-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2882381 INFO  (jetty-launcher-4049-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config 
dir: null
   [junit4]   2> 2882381 INFO  (jetty-launcher-4049-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=47648}
   [junit4]   2> 2882381 INFO  (jetty-launcher-4049-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-07-01T01:01:06.447Z
   [junit4]   2> 2882382 ERROR (jetty-launcher-4049-thread-2) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 2882382 INFO  (jetty-launcher-4049-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.0.0
   [junit4]   2> 2882382 INFO  (jetty-launcher-4049-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2882382 INFO  (jetty-launcher-4049-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config 
dir: null
   [junit4]   2> 2882382 INFO  (jetty-launcher-4049-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-07-01T01:01:06.448Z
   [junit4]   2> 2882384 INFO  (jetty-launcher-4049-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2882384 INFO  (jetty-launcher-4049-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2882391 INFO  (jetty-launcher-4049-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64446/solr
   [junit4]   2> 2882391 INFO  (jetty-launcher-4049-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64446/solr
   [junit4]   2> 2882545 INFO  (jetty-launcher-4049-thread-2) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2882546 INFO  (jetty-launcher-4049-thread-2) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:47648_solr
   [junit4]   2> 2882551 INFO  (jetty-launcher-4049-thread-2) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.Overseer Overseer 
(id=98230001096130566-127.0.0.1:47648_solr-n_0000000000) starting
   [junit4]   2> 2882555 INFO  (jetty-launcher-4049-thread-1) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2882557 INFO  (jetty-launcher-4049-thread-1) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:60154_solr
   [junit4]   2> 2882559 INFO  
(zkCallback-4061-thread-1-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 2882559 INFO  
(zkCallback-4060-thread-1-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 2882562 INFO  (jetty-launcher-4049-thread-2) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:47648_solr
   [junit4]   2> 2882564 INFO  
(zkCallback-4061-thread-1-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 2882564 INFO  
(zkCallback-4060-thread-1-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 2882669 INFO  (jetty-launcher-4049-thread-1) 
[n:127.0.0.1:60154_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_60154.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4eaab97c
   [junit4]   2> 2882673 INFO  (jetty-launcher-4049-thread-1) 
[n:127.0.0.1:60154_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_60154.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4eaab97c
   [junit4]   2> 2882673 INFO  (jetty-launcher-4049-thread-1) 
[n:127.0.0.1:60154_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_60154.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4eaab97c
   [junit4]   2> 2882674 INFO  (jetty-launcher-4049-thread-1) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_F63395BBD76CFEF0-001/tempDir-001/node1/.
   [junit4]   2> 2882808 INFO  (jetty-launcher-4049-thread-2) 
[n:127.0.0.1:47648_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_47648.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4eaab97c
   [junit4]   2> 2882811 INFO  (jetty-launcher-4049-thread-2) 
[n:127.0.0.1:47648_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_47648.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4eaab97c
   [junit4]   2> 2882811 INFO  (jetty-launcher-4049-thread-2) 
[n:127.0.0.1:47648_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_47648.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4eaab97c
   [junit4]   2> 2882811 INFO  (jetty-launcher-4049-thread-2) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_F63395BBD76CFEF0-001/tempDir-001/node2/.
   [junit4]   2> 2882837 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2882839 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:64446/solr ready
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 2882871 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 2882878 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2882879 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 2882892 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] o.m.log 
Extract 
jar:file:/export/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs
 to ./temp/Jetty_serv1.vm3_58360_hdfs____d1uk8b/webapp
   [junit4]   2> 2883397 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] o.m.log 
Started HttpServer2$SelectChannelConnectorWithSafeStartup@serv1-vm3:58360
   [junit4]   2> 2883535 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2883536 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 2883549 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] o.m.log 
Extract 
jar:file:/export/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_61787_datanode____.ixkh4u/webapp
   [junit4]   2> 2884021 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] o.m.log 
Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:61787
   [junit4]   2> 2884086 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2884087 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 2884104 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] o.m.log 
Extract 
jar:file:/export/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_39836_datanode____.b02uqs/webapp
   [junit4]   2> 2884161 INFO  (IPC Server handler 3 on 57446) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-0926f9b8-ce32-461f-8eb5-8e7dd974fa9a node 
DatanodeRegistration(127.0.0.1:63536, 
datanodeUuid=dc02d9c0-1e2a-41f7-8907-33c11c662969, infoPort=40531, 
infoSecurePort=0, ipcPort=58818, 
storageInfo=lv=-56;cid=testClusterID;nsid=1049226872;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 2884162 INFO  (IPC Server handler 3 on 57446) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-05fa062c-2863-4135-bc67-573ab1cc13ee node 
DatanodeRegistration(127.0.0.1:63536, 
datanodeUuid=dc02d9c0-1e2a-41f7-8907-33c11c662969, infoPort=40531, 
infoSecurePort=0, ipcPort=58818, 
storageInfo=lv=-56;cid=testClusterID;nsid=1049226872;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2884610 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[F63395BBD76CFEF0]-worker) [    ] o.m.log 
Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39836
   [junit4]   2> 2884695 INFO  (IPC Server handler 2 on 57446) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-1d9e02eb-aaca-448c-a86c-5057ecb15d07 node 
DatanodeRegistration(127.0.0.1:32993, 
datanodeUuid=001faa7a-35ca-4c1b-b1ab-7d3980f732d5, infoPort=62583, 
infoSecurePort=0, ipcPort=37598, 
storageInfo=lv=-56;cid=testClusterID;nsid=1049226872;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 2884695 INFO  (IPC Server handler 2 on 57446) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-f1d2c2b2-ee0d-406c-883a-75ccecb333ec node 
DatanodeRegistration(127.0.0.1:32993, 
datanodeUuid=001faa7a-35ca-4c1b-b1ab-7d3980f732d5, infoPort=62583, 
infoSecurePort=0, ipcPort=37598, 
storageInfo=lv=-56;cid=testClusterID;nsid=1049226872;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 2884815 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[F63395BBD76CFEF0]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 2884835 INFO  (qtp1897270833-26503) [n:127.0.0.1:60154_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&nrtReplicas=2&action=CREATE&numShards=1&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 2884838 INFO  
(OverseerThreadFactory-10690-thread-1-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
recoverytest
   [junit4]   2> 2884947 INFO  
(OverseerStateUpdate-98230001096130566-127.0.0.1:47648_solr-n_0000000000) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"recoverytest",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"recoverytest_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:60154/solr";,
   [junit4]   2>   "type":"NRT"} 
   [junit4]   2> 2884952 INFO  
(OverseerStateUpdate-98230001096130566-127.0.0.1:47648_solr-n_0000000000) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"recoverytest",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"recoverytest_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:47648/solr";,
   [junit4]   2>   "type":"NRT"} 
   [junit4]   2> 2885162 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica_n1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2885170 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 2885199 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica_n2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 2885202 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 2885311 INFO  
(zkCallback-4061-thread-2-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2885311 INFO  
(zkCallback-4060-thread-2-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2885311 INFO  
(zkCallback-4060-thread-1-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2885311 INFO  
(zkCallback-4061-thread-1-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2886190 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2886198 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.s.IndexSchema [recoverytest_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 2886203 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2886203 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.CoreContainer Creating SolrCore 'recoverytest_shard1_replica_n1' using 
configuration from collection recoverytest, trusted=true
   [junit4]   2> 2886203 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_60154.solr.core.recoverytest.shard1.replica_n1' (registry 
'solr.core.recoverytest.shard1.replica_n1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4eaab97c
   [junit4]   2> 2886214 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://serv1-vm3:57446/data
   [junit4]   2> 2886214 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2886215 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2886215 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.SolrCore [[recoverytest_shard1_replica_n1] ] Opening new SolrCore at 
[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_F63395BBD76CFEF0-001/tempDir-001/node1/recoverytest_shard1_replica_n1],
 dataDir=[hdfs://serv1-vm3:57446/data/recoverytest/core_node1/data/]
   [junit4]   2> 2886216 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://serv1-vm3:57446/data/recoverytest/core_node1/data/snapshot_metadata
   [junit4]   2> 2886223 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2886226 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 2886226 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 2886226 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 2886234 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.s.IndexSchema [recoverytest_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 2886237 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 2886237 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.CoreContainer Creating SolrCore 'recoverytest_shard1_replica_n2' using 
configuration from collection recoverytest, trusted=true
   [junit4]   2> 2886237 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_47648.solr.core.recoverytest.shard1.replica_n2' (registry 
'solr.core.recoverytest.shard1.replica_n2') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4eaab97c
   [junit4]   2> 2886238 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://serv1-vm3:57446/data
   [junit4]   2> 2886238 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 2886238 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 2886238 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.SolrCore [[recoverytest_shard1_replica_n2] ] Opening new SolrCore at 
[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_F63395BBD76CFEF0-001/tempDir-001/node2/recoverytest_shard1_replica_n2],
 dataDir=[hdfs://serv1-vm3:57446/data/recoverytest/core_node2/data/]
   [junit4]   2> 2886241 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://serv1-vm3:57446/data/recoverytest/core_node2/data/snapshot_metadata
   [junit4]   2> 2886252 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 2886252 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 2886252 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 2886345 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2886347 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://serv1-vm3:57446/data/recoverytest/core_node1/data
   [junit4]   2> 2886355 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2886356 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://serv1-vm3:57446/data/recoverytest/core_node2/data
   [junit4]   2> 2887014 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://serv1-vm3:57446/data/recoverytest/core_node1/data/index
   [junit4]   2> 2887019 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://serv1-vm3:57446/data/recoverytest/core_node2/data/index
   [junit4]   2> 2887022 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 2887022 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 2887022 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 2887025 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 2887025 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 2887025 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 2887091 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2887098 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 2887262 INFO  (IPC Server handler 8 on 57446) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:63536 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-f1d2c2b2-ee0d-406c-883a-75ccecb333ec:NORMAL:127.0.0.1:32993|RBW],
 
ReplicaUC[[DISK]DS-05fa062c-2863-4135-bc67-573ab1cc13ee:NORMAL:127.0.0.1:63536|RBW]]}
 size 0
   [junit4]   2> 2887265 INFO  (IPC Server handler 4 on 57446) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:32993 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-f1d2c2b2-ee0d-406c-883a-75ccecb333ec:NORMAL:127.0.0.1:32993|RBW],
 
ReplicaUC[[DISK]DS-05fa062c-2863-4135-bc67-573ab1cc13ee:NORMAL:127.0.0.1:63536|RBW]]}
 size 0
   [junit4]   2> 2887266 INFO  (IPC Server handler 1 on 57446) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:63536 is 
added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-0926f9b8-ce32-461f-8eb5-8e7dd974fa9a:NORMAL:127.0.0.1:63536|RBW],
 
ReplicaUC[[DISK]DS-f1d2c2b2-ee0d-406c-883a-75ccecb333ec:NORMAL:127.0.0.1:32993|RBW]]}
 size 72
   [junit4]   2> 2887266 INFO  (IPC Server handler 7 on 57446) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:32993 is 
added to blk_1073741825_1001 size 72
   [junit4]   2> 2887320 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 2887320 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2887320 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 2887331 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2887331 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2887345 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@66893e9f[recoverytest_shard1_replica_n2] main]
   [junit4]   2> 2887347 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 2887348 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2887349 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2887349 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1571680022864855040
   [junit4]   2> 2887350 INFO  
(searcherExecutor-10696-thread-1-processing-n:127.0.0.1:47648_solr 
x:recoverytest_shard1_replica_n2 s:shard1 c:recoverytest r:core_node2) 
[n:127.0.0.1:47648_solr c:recoverytest s:shard1 r:core_node2 
x:recoverytest_shard1_replica_n2] o.a.s.c.SolrCore 
[recoverytest_shard1_replica_n2] Registered new searcher 
Searcher@66893e9f[recoverytest_shard1_replica_n2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2887359 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 2887361 INFO  
(zkCallback-4061-thread-1-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2887361 INFO  
(zkCallback-4060-thread-1-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2887361 INFO  
(zkCallback-4060-thread-2-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2887361 INFO  
(zkCallback-4061-thread-2-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2887738 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 2887738 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2887738 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 2887746 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2887746 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2887756 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@669ba94e[recoverytest_shard1_replica_n1] main]
   [junit4]   2> 2887757 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 2887757 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 2887758 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2887759 INFO  
(searcherExecutor-10695-thread-1-processing-n:127.0.0.1:60154_solr 
x:recoverytest_shard1_replica_n1 s:shard1 c:recoverytest r:core_node1) 
[n:127.0.0.1:60154_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica_n1] o.a.s.c.SolrCore 
[recoverytest_shard1_replica_n1] Registered new searcher 
Searcher@669ba94e[recoverytest_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2887759 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1571680023294771200
   [junit4]   2> 2887860 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2887860 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2887860 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/
   [junit4]   2> 2887861 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.PeerSync PeerSync: core=recoverytest_shard1_replica_n2 
url=https://127.0.0.1:47648/solr START 
replicas=[https://127.0.0.1:60154/solr/recoverytest_shard1_replica_n1/] 
nUpdates=100
   [junit4]   2> 2887861 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.PeerSync PeerSync: core=recoverytest_shard1_replica_n2 
url=https://127.0.0.1:47648/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 2887871 INFO  (qtp1897270833-26505) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.c.S.Request [recoverytest_shard1_replica_n1]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=1
   [junit4]   2> 2887872 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the 
next candidate
   [junit4]   2> 2887872 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 2887872 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 2887884 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/ shard1
   [junit4]   2> 2887991 INFO  
(zkCallback-4061-thread-1-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2887991 INFO  
(zkCallback-4060-thread-2-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2887991 INFO  
(zkCallback-4060-thread-1-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2887991 INFO  
(zkCallback-4061-thread-2-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2888041 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2888046 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica_n2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=2847
   [junit4]   2> 2888165 INFO  
(zkCallback-4061-thread-2-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2888165 INFO  
(zkCallback-4060-thread-1-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2888166 INFO  
(zkCallback-4061-thread-1-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2888166 INFO  
(zkCallback-4060-thread-2-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2888769 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node1&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica_n1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=3608
   [junit4]   2> 2888775 INFO  (qtp1897270833-26503) [n:127.0.0.1:60154_solr    
] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 2888874 INFO  
(zkCallback-4061-thread-1-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2888874 INFO  
(zkCallback-4060-thread-1-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2888874 INFO  
(zkCallback-4060-thread-2-processing-n:127.0.0.1:60154_solr) 
[n:127.0.0.1:60154_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2888874 INFO  
(zkCallback-4061-thread-2-processing-n:127.0.0.1:47648_solr) 
[n:127.0.0.1:47648_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 2889009 INFO  
(OverseerCollectionConfigSetProcessor-98230001096130566-127.0.0.1:47648_solr-n_0000000000)
 [n:127.0.0.1:47648_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 2889777 INFO  (qtp1897270833-26503) [n:127.0.0.1:60154_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&nrtReplicas=2&action=CREATE&numShards=1&wt=javabin&version=2}
 status=0 QTime=4943
   [junit4]   2> 2889778 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[F63395BBD76CFEF0]) [    ] 
o.a.s.c.RecoveryZkTest Indexing 1350 documents
   [junit4]   2> 2889792 DEBUG (qtp1403034289-26509) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.HdfsTransactionLog Opening new tlog hdfs 
tlog{file=hdfs://serv1-vm3:57446/data/recoverytest/core_node2/data/tlog/tlog.0000000000000000000
 refcount=1}
   [junit4]   2> 2889801 DEBUG (qtp1897270833-26498) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.HdfsTransactionLog Opening new tlog hdfs 
tlog{file=hdfs://serv1-vm3:57446/data/recoverytest/core_node1/data/tlog/tlog.0000000000000000000
 refcount=1}
   [junit4]   2> 2889809 INFO  (qtp1897270833-26501) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{add=[1-0
 (1571680025419186176)]} 0 13
   [junit4]   2> 2889810 INFO  (qtp1403034289-26497) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-0 
(1571680025419186176)]} 0 25
   [junit4]   2> 2889810 INFO  (qtp1897270833-26498) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{add=[2-0
 (1571680025421283328)]} 0 15
   [junit4]   2> 2889810 INFO  (qtp1403034289-26509) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-0 
(1571680025421283328)]} 0 23
   [junit4]   2> 2889812 INFO  (qtp1897270833-26502) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{add=[1-1
 (1571680025445400576)]} 0 1
   [junit4]   2> 2889813 INFO  (qtp1403034289-26511) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-1 
(1571680025445400576)]} 0 2
   [junit4]   2> 2889813 INFO  (qtp1897270833-26504) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{add=[2-1
 (1571680025446449152)]} 0 0
   [junit4]   2> 2889813 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-1 
(1571680025446449152)]} 0 2
   [junit4]   2> 2889816 INFO  (qtp1897270833-26505) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{add=[1-2
 (1571680025449594880)]} 0 1
   [junit4]   2> 2889816 INFO  (qtp1897270833-26506) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{add=[2-2
 (1571680025449594881)]} 0 1
   [junit4]   2> 2889816 INFO  (qtp1403034289-26513) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-2 
(1571680025449594880)]} 0 1
   [junit4]   2> 2889816 INFO  (qtp1403034289-26507) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-2 
(1571680025449594881)]} 0 2
   [junit4]   2> 2889818 INFO  (qtp1897270833-26501) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{delete=[1-0
 (-1571680025452740608)]} 0 0
   [junit4]   2> 2889818 INFO  (qtp1897270833-26503) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{delete=[2-0
 (-1571680025452740609)]} 0 1
   [junit4]   2> 2889819 INFO  (qtp1403034289-26497) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-0 
(-1571680025452740609)]} 0 1
   [junit4]   2> 2889819 INFO  (qtp1403034289-26510) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-0 
(-1571680025452740608)]} 0 1
   [junit4]   2> 2889821 INFO  (qtp1897270833-26498) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{add=[2-3
 (1571680025454837760)]} 0 0
   [junit4]   2> 2889821 INFO  (qtp1897270833-26504) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{add=[1-3
 (1571680025455886336)]} 0 0
   [junit4]   2> 2889821 INFO  (qtp1403034289-26509) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-3 
(1571680025454837760)]} 0 1
   [junit4]   2> 2889821 INFO  (qtp1403034289-26511) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-3 
(1571680025455886336)]} 0 1
   [junit4]   2> 2889823 INFO  (qtp1897270833-26505) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{add=[1-4
 (1571680025457983489)]} 0 0
   [junit4]   2> 2889823 INFO  (qtp1897270833-26502) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{add=[2-4
 (1571680025457983488)]} 0 0
   [junit4]   2> 2889823 INFO  (qtp1403034289-26513) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-4 
(1571680025457983489)]} 0 1
   [junit4]   2> 2889823 INFO  (qtp1403034289-26512) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-4 
(1571680025457983488)]} 0 1
   [junit4]   2> 2889825 INFO  (qtp1897270833-26503) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&version=2}{add=[2-5
 (1571680025460080641)]} 0 0
   [junit4]   2> 2889825 INFO  (qtp1403034289-26507) [n:127.0.0.1:47648_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica_n2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-5 
(1571680025460080641)]} 0 1
   [junit4]   2> 2889826 INFO  (qtp1897270833-26501) [n:127.0.0.1:60154_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica_n1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica_n1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:47648/solr/recoverytest_shard1_replica_n2/&wt=javabin&versio

[...truncated too long message...]

up, TestCustomDocTransformer, LeaderElectionTest, TestPKIAuthenticationPlugin, 
TestRequestForwarding, TestFunctionQuery, WordBreakSolrSpellCheckerTest, 
ChaosMonkeySafeLeaderWithPullReplicasTest, TestDFRSimilarityFactory, 
TestFieldCacheReopen, TestRawResponseWriter, TestHighlightDedupGrouping, 
SearchHandlerTest, TestMacros, ZkCLITest, TestSweetSpotSimilarityFactory, 
TestOmitPositions, TestInPlaceUpdatesDistrib, SuggesterTest, 
TestSolrCLIRunExample, TestRandomFlRTGCloud, 
TestHighFrequencyDictionaryFactory, TestUniqueKeyFieldResource, 
ConnectionReuseTest, URLClassifyProcessorTest, TestBackupRepositoryFactory, 
TestLocalFSCloudBackupRestore, TestStressReorder, RAMDirectoryFactoryTest, 
TestFilteredDocIdSet, TestReversedWildcardFilterFactory, JavabinLoaderTest, 
BadCopyFieldTest, PrimitiveFieldTypeTest, TestWriterPerf, 
FieldAnalysisRequestHandlerTest, TestConfigSets, 
DistributedExpandComponentTest, CheckHdfsIndexTest, TestSurroundQueryParser, 
TestPayloadCheckQParserPlugin, SolrCoreMetricManagerTest, GraphQueryTest, 
DistributedSpellCheckComponentTest, TestFoldingMultitermQuery, MetricUtilsTest, 
ChangedSchemaMergeTest, SpatialHeatmapFacetsTest, MinimalSchemaTest, 
TestPointFields, OverseerRolesTest, CoreAdminCreateDiscoverTest, 
SortByFunctionTest, DirectUpdateHandlerOptimizeTest, 
LeaderElectionContextKeyTest, TestExactStatsCache, SimplePostToolTest, 
TestManagedSchemaAPI, TestStressInPlaceUpdates, 
DistributedFacetPivotWhiteBoxTest, OverseerTest, 
TestManagedSynonymFilterFactory, RulesTest, CdcrVersionReplicationTest, 
TestObjectReleaseTracker, PrimUtilsTest, TestHdfsCloudBackupRestore, 
DeleteNodeTest, DeleteInactiveReplicaTest, TestSolrCloudWithDelegationTokens, 
CoreAdminHandlerTest, TestPseudoReturnFields, ZkControllerTest, 
TestHashPartitioner, HdfsDirectoryTest, TestNumericTerms32, TestSearcherReuse, 
TestUnifiedSolrHighlighter, TestStressVersions, HttpSolrCallGetCoreTest, 
TestExceedMaxTermLength, DocValuesTest, SolrMetricReporterTest, 
TestLegacyTerms, AsyncCallRequestStatusResponseTest, TestGeoJSONResponseWriter, 
MultiThreadedOCPTest, TestDistribIDF, DeleteShardTest, 
XmlUpdateRequestHandlerTest, TestStressRecovery, FullHLLTest, 
HdfsCollectionsAPIDistributedZkTest, CleanupOldIndexTest, RecoveryZkTest, 
TestRandomDVFaceting, TestInitParams, V2ApiIntegrationTest, 
TriLevelCompositeIdRoutingTest, HdfsUnloadDistributedZkTest, CircularListTest, 
ConjunctionSolrSpellCheckerTest, HdfsRecoverLeaseTest, 
TestSolrCloudWithSecureImpersonation, SolrCloudReportersTest, 
DateRangeFieldTest, TestFieldTypeResource, TestDistributedSearch, 
TestCloudSchemaless, MissingSegmentRecoveryTest, RegexBoostProcessorTest, 
EnumFieldTest, TestIntervalFaceting, TestPhraseSuggestions, 
TestNoOpRegenerator, TestCollectionAPIs, TestTrieFacet, TestFaceting, 
RequestHandlersTest, ShardRoutingTest, TestGroupingSearch, 
TestConfigSetImmutable, DistanceUnitsTest, PKIAuthenticationIntegrationTest, 
CollectionsAPISolrJTest, TestPivotHelperCode, BasicAuthIntegrationTest, 
TestUtils, SecurityConfHandlerTest, HLLUtilTest, SolrCLIZkUtilsTest, 
RecoveryAfterSoftCommitTest, OutputWriterTest, SolrGangliaReporterTest, 
TestManagedResourceStorage, TestSimpleTextCodec, SolrInfoBeanTest, 
ClusterStateUpdateTest, AutoCommitTest, TestRecovery, TestRealTimeGet, 
TestReload, HardAutoCommitTest, DistributedTermsComponentTest, TestRangeQuery, 
TestCoreContainer, SimpleFacetsTest, SolrCoreTest, StatsComponentTest, 
QueryElevationComponentTest, BadIndexSchemaTest, ConvertedLegacyTest, TestSort, 
TestFiltering, BasicFunctionalityTest, TestLazyCores, 
DistributedQueryElevationComponentTest, AnalysisAfterCoreReloadTest, 
TestExtendedDismaxParser, SolrRequestParserTest, SuggesterTSTTest, TestTrie, 
SpatialFilterTest, TestCSVLoader, PolyFieldTest, 
SolrCoreCheckLockOnStartupTest, TestUpdate, FieldMutatingUpdateProcessorTest, 
TestAtomicUpdateErrorCases, QueryEqualityTest, 
StatelessScriptUpdateProcessorFactoryTest, DistanceFunctionTest, 
TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, DebugComponentTest, 
IndexBasedSpellCheckerTest, LukeRequestHandlerTest, DisMaxRequestHandlerTest, 
DocumentAnalysisRequestHandlerTest, TermsComponentTest, DocumentBuilderTest, 
TestValueSourceCache, PathHierarchyTokenizerFactoryTest, 
TestIndexingPerformance, RequiredFieldsTest, FastVectorHighlighterTest, 
TestJmxIntegration, TestCollationField, ReturnFieldsTest, 
TestCSVResponseWriter, QueryParsingTest, JsonLoaderTest, 
BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, HighlighterConfigTest, 
SOLR749Test, TestQuerySenderListener, AlternateDirectoryTest, 
TestSolrIndexConfig, TestQuerySenderNoQuery, TestMergePolicyConfig, 
TestBinaryField, TestSolrCoreProperties, TestLuceneMatchVersion, 
TestCodecSupport, ResourceLoaderTest, PluginInfoTest, TestFastLRUCache, 
ChaosMonkeyNothingIsSafeTest, CdcrReplicationHandlerTest, 
CdcrRequestHandlerTest, CloudExitableDirectoryReaderTest, CollectionReloadTest, 
CollectionStateFormat2Test, CollectionTooManyReplicasTest, 
ConcurrentDeleteAndCreateCollectionTest, ConfigSetsAPITest, 
DistribDocExpirationUpdateProcessorTest, DistributedVersionInfoTest, 
LeaderFailoverAfterPartitionTest, LeaderFailureAfterFreshStartTest, 
LeaderInitiatedRecoveryOnShardRestartTest, MoveReplicaTest, SSLMigrationTest, 
SaslZkACLProviderTest, SharedFSAutoReplicaFailoverTest, 
SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, 
SolrCloudExampleTest, SolrXmlInZkTest, TestCloudDeleteByQuery, 
TestCloudInspectUtil, TestCloudPivotFacet, TestCloudPseudoReturnFields, 
TestCloudRecovery, TestDeleteCollectionOnDownNodes, 
TestExclusionRuleCollectionAccess, TestLeaderElectionZkExpiry, 
TestLeaderInitiatedRecoveryThread, TestShortCircuitedRequests, 
AutoScalingHandlerTest, TestPolicyCloud, HdfsRecoveryZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsRecoveryZkTest 
-Dtests.seed=F63395BBD76CFEF0 -Dtests.slow=true -Dtests.locale=en-GB 
-Dtests.timezone=America/Argentina/Cordoba -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | HdfsRecoveryZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 
object(s) that were not released!!! [HdfsTransactionLog]
   [junit4]    > 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.update.HdfsTransactionLog
   [junit4]    >        at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    >        at 
org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:132)
   [junit4]    >        at 
org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:203)
   [junit4]    >        at 
org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:161)
   [junit4]    >        at 
org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:116)
   [junit4]    >        at 
org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:108)
   [junit4]    >        at 
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   [junit4]    >        at 
org.apache.solr.core.SolrCore.createInstance(SolrCore.java:773)
   [junit4]    >        at 
org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:835)
   [junit4]    >        at 
org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1088)
   [junit4]    >        at 
org.apache.solr.core.SolrCore.<init>(SolrCore.java:958)
   [junit4]    >        at 
org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
   [junit4]    >        at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:969)
   [junit4]    >        at 
org.apache.solr.core.CoreContainer.lambda$load$7(CoreContainer.java:610)
   [junit4]    >        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
   [junit4]    >        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    >        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([F63395BBD76CFEF0]:0)
   [junit4]    >        at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:296)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [554/724 (2!)] on J0 in 38.55s, 1 test, 1 failure <<< 
FAILURES!

[...truncated 43582 lines...]
-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [echo] Checking for malformed docs...
     [exec] 
     [exec] 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/docs/solr-core/overview-summary.html
     [exec]   missing description: org.apache.solr.common.cloud
     [exec] 
     [exec] Missing javadocs were found!

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:810: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:669: 
The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:685: 
The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2506:
 exec returned: 1

Total time: 80 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to