Zack:

Ignore it for the time being. We're working on it, it's a complex bit
of code. Unfortunately I can't make it fail locally so we have have to
use Jenkins and its many environments to try to figure out what the
magic is.

NOTE: this code is _not_ in the 5x code line, I won't put it there
unless and until we get stable tests out of it. I suspect (but don't
quite know) that this something "interesting" in the test framework.
It's always fun to try to figure out whether it's a test artifact and
how to work around it or whether it's really a problem with the code
somewhere.

If you want to, you can just add the @Ignore annotation to any of the
CDCR tests locally, but do _not_ include that change with the patch.
There's really no need to though, the end of the test run prints out
all the failing tests and if any CDCR* tests are labeled you may
safely ignore them.

Best,
Erick

On Wed, Jul 22, 2015 at 9:22 PM, Liang, Hsuan Tzu <[email protected]> wrote:
> Hello everyone,
>
> I am currently working on a solr JIRA and I am encoutering this test
> failure as well.
> If I run the single test it can pass.
> This is my first time to get involved in solr so Iam not sure what should
> I do if the trunk branch itself might contain potential test failures. Am
> I suppoed to wait for the fix or just find a previous passed commit for
> my development? Or is it a badapples test?
> Thanks!
>
> Best,
> Zack Liang
>
> On 7/21/15, 10:05 PM, "Policeman JenkinServer" <[email protected]>
> wrote:
>
>>Build: http://jenkins.theaphi.de/job/Lucene-Solr-trunk-MacOSX/2534/
>>Java: 64bitjdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC
>>
>>1 tests failed.
>>FAILD:  org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTes
>>
>>Error Message:
>>Timeout while trying to assert update logs  collection=source_collection
>>
>>Stack Trace:
>>java.lang.AssertionError: Tieout while trying to assert update logs @
>>collection=source_collection
>>       at
>>__randomizedsting.SeedInfo.seed([FA6717C1ABDFB670:F20762EDA4D19E7B]:0)
>>       at 
>> >org.apache.solr.cloud.CdcrReplicationDistributedZkTest.assertNumberOfTlogF>iles(CdcrReplicationDistributedZkTest.java:644)
>>       at
>>orgapache.solr.cloud.CdcrReplicationDistributedZkTest.doTestUpdateLogSync
>>hronisation(CdcrReplicationDistributedZkTest.java:384)
>>       at
>>org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTests(CdcrReplica
>>tionDistributedZkTest.va:50)
>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native ethod)
>>       at
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccesorImpl.java:
>>62)
>>       at
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DlegatingMethodAccessorIm
>>pl.java:43)
>>       at java.lang.reflect.Method.invoke(Method.java:497)
> at
>>com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(andomizedRunne
>>r.java:1627)
>>       at
>>com.carrotsearch.randomizedtesting.RandmizedRunner$6.evaluate(RandomizedR
>>unner.java:836)
>>       at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedR
>>unner.java:872)
>>       at
>>com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedR
>>unner.java:886)
>>       at
>>org.apache.solr.BasestributedSearchTestCase$ShardsRepeatRule$ShardsFixed
>>Statement.allStatement(BaseDistributedSearchTestCase.java:963)
>>       at
>>org.apache.solrBaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStat
>>ment.evaluate(BaseDistributedSearchTestCase.java:938)
>>       at
>>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.eva
>>luate(SystemPropertiesRestoreRule.java:57)
>>       at
>>org.apache.lucene.util.TestRuleSetupTearwnChained$1.evaluate(TestRuleSet
>>upTeardownChained.java:50)
>>       a
>>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforef
>>terRule.java:46)
>>       at
>>org.apache.lucene.util.TesRuleThreadAndTestName$1.evaluate(TestRuleThread
>>AndTestName.java:49)
>>       at
>>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleI
>>gnoreAfterMaxFailures.java:65)
>>       at
>>org.apache.lucene.util.TestRuleMrkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>       at
>>com.carrotsearch.randomizedtesting.rules.StatemetAdapter.evaluate(Stateme
>>ntAdapter.java:36)
>>       at
>>com.carrotsearch.randoizedtesting.ThreadLeakControl$StatementRunner.run(T
>>hreadeakControl.java:365)
>>       at
>>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(Th
>>readLeakControl.java:798)
>>       at
>>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeak
>>Control.jav458)
>>       at
>>com.carrotsearch.randomizedtesting.RandomizedRunner.unSingleTest(Randomiz
>>edRunner.java:845)
>>       at
>>com.carrotsearch.randomizetesting.RandomizedRunner$3.evaluate(RandomizedR
>>unner.java:47)
>>       at
>>com.carrotsearch.randomizedtesting.RandomizedRunner$4evaluate(RandomizedR
>>unner.java:781)
>>       at
>>co.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(andomizedR
>>unner.java:792)
>>       at
>>com.carrotsearch.randomizedtesting.rles.StatementAdapter.evaluate(Stateme
>>ntAdapter.java:36)
>>       at
>>com.carrotsearch.randomidtesting.rules.SystemPropertiesRestoreRule$1.eva
>>luate(SystemPrpertiesRestoreRule.java:57)
>>       at
>>org.apache.lucene.util.AbstractBeforeAftrRule$1.evaluate(AbstractBeforeAf
>>terRule.java:46)
>>       at
>>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>>tAdapter.java:36)
>>       at
>>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClas
>>sName.java:42)
>>       at
>>com.carrotsear.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRu
>>le$1evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>       at
>>com.carrotsearh.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRu
>>le$1.evaluate(NoSdowingOrOverridesOnMethodsRule.java:39)
>>       at
>>com.carrotsearch.randomizetesting.rules.StatementAdapter.evaluate(Stateme
>>ntAdapter.java:36)
>>       at
>>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Sateme
>>ntAdapter.java:36)
>>       at
>>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>>ntAdapter.java:36)
>>       at
>>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAsser
>>tionsReqred.java:54)
>>       at
>>org.apache.lucene.util.TestRuleMarkFailure$1.evaluateTestRuleMarkFailure.
>>java:48)
>>       at
>>org.apache.luceneutil.TestRuleIgnoreAfterMaFailures$1.evaluate(TestRuleI
>>gnoreAfterMaxFailures.java:65)
>>       at
>>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreT
>>estSuites.java:55)
>>       at
>>com.carrotsearch.randomizedtesting.rules.StatemetAdapter.evaluate(Stateme
>>ntAdapter.java:36)
>>       at
>>com.carrotsearch.randomizedtestinghreadLeakControl$StatementRunner.run(T
>>hreadLeakControl.java:36)
>>       at java.lang.Thread.run(Thread.java:745)
>>Caused by: java.lang.AssertioError: Number of tlogs on leader: 70 is
>>superio to: 50.
>>       at org.junit.Assert.fail(Assert.java:93)
>>       at org.junit.Assert.assertTruAssert.java:43)
>>       at
>>org.apache.solr.cloud.CdcrReplicationDistibutedZkTest._assertNumberOfTlog
>>Files(CdcrReplicationDistributedZkTest.jaa:671)
>>       at
>>org.apache.solr.cloud.CdcrReplicationDistributedZkTest.ssertNumberOfTlogF
>>iles(CdcrReplicationDistributedZkTest.va:634)
>>       ... 43 more
>>
>>
>>
>>
>>Build Log:
>>[...truncated 10428 lnes...]
>>   [junit4] Suite: org.apache.solr.cloud.CdcrReplicationDistributeZkTest
>>   [junit4]   2> Creating dataDir:
>>/Users/jenkins/workspace/Lucene-Solrrunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrRplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/init-core-data-001
>>   [jnit4]   2> 1475810 INFO
>>(SUITE-CdcrReplicationDistributedZkTest-seed#[FA6717C1AFB670]-worker) [
>>   ] o.a.s.SolrTestCaseJ4 Randomized ssl (true and clientAuth (false)
>>   [junit4]   2> 1475810 INFO
>>(SUITE-CdcrReplicatonDistributedZkTest-seed#[FA6717C1ABDFB670]-worker) [
>>   ] o.a.s.BaseDistributeearchTestCase Setting hostContext system
>>property: /
>>   [junit]   2> 1475813 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[F6717C1ABDFB670]) [
>>   ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
>>   [junit   2> 1475814 INFO  (Thread-3290) [    ] o.a.s.c.ZkTestServer
>>cient port:0.0.0.0/0.0.0.0:0
>>   [junit4]   2> 1475814 INFO  (Thread-3290) [   ] o.a.s.c.ZkTestServer
>>Starting server
>    [junit4]   2> 1475915 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.ZkTestServer start zk server on port:58151
>>   [junit4]   2> 1475943 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.cbstractZkTestCase put
>>/Users/jenkins/workspace/Lucene-Solr-trun-MacOSX/solr/core/src/test-files
>>/solr/collection1/conf/solrconfig-cdcr.xm to
>>/configs/conf1/solrconfig.xml
>>   [junit4]   > 1475949 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1DFB670]) [
>>   ] o.a.s.c.AbstractZkTestCase put
>>/Users/jenkins/orkspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files
>>/solr/collectio1/conf/schema15.xml to /configs/conf1/schema.xml
>>   [jnit4]   2> 1475954 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTes-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.AbstractZkTestCase 
>> put>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files>/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
>>  to
>>/confis/conf1/solrconfig.snippet.randomindexconfig.xml
>>   [junit4]   2> 1475958 INFO
>>(TEST-CdcrReplicionDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o..s.c.AbstractZkTestCase put
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacSX/solr/core/src/test-files
>>/solr/collection1/conf/stopwords.txt to /confi/conf1/stopwords.txt
>>   [junit4]   2> 1475962 INFO
>>(TEST-CdcrRplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.cAbstractZkTestCase put
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/sr/core/src/test-files
>>/solr/collection1/conf/protwords.txt to /onfigs/conf1/protwords.txt
>>   [junit4]   2> 1475967 INFO
>>(TEST-CdcrReplictionDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>  ] o.a.s.c.AbstractZkTestCase put
>>/Users/jenkins/workspace/Lucen-Solr-trunk-MacOSX/solr/core/src/test-files
>>/solr/collection1/conf/curency.xml to /configs/conf1/currency.xml
>>   [junt4]   2> 1475971 INFO
>>(TEST-CdcrReplicationDistributedZTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.AbstractZkTestCase put
>>/Users/jenki/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files
>>/slr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
>>   [unit4]   2> 1475975 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.AbstractZkTestCase ut
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files
>>/solr/collection1/conf/open-exchange-rates.json to
>>/confs/conf1/open-exchange-rates.json
>>   [junit4]   2> 1475980 INFO
> (TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>  ] o.a.s.c.AbstractZkTestCase put
>>/Users/jenkins/workspace/Lucene-Solr-trunkacOSX/solr/core/src/test-files
>>/solr/collection1/conf/mapping-IOLatin1Accent.txt to
>>/configs/conf1/mapping-ISOLatin1Accent.txt
>>   [junit]   2> 1475984 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.s.c.AbstractZkTestCase put
>>/Users/jenkins/workspace/Lucene-Sol-trunk-MacOSX/solr/core/src/test-files
>>/solr/collection1/conf/old_synonymstxt to /configs/conf1/old_synonyms.txt
>>   [junit4]   2> 1475989 INFO
>>(TES-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] a.s.c.AbstractZkTestCase put
>>/Users/jenkins/workspace/Lucene-Sor-trunk-MacOSX/solr/core/src/test-files
>>/solr/collection1/conf/synonyms.tx to /configs/conf1/synonyms.txt
>>   [junit4]   2> 1476000 INFO
>>(TET-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1AFB670]) [
>>   ] o.a.s.SolrTestCaseJ4 ###Starting doTests
>>   [juit4]   2> 1476351 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed[FA6717C1ABDFB670]) [
>>   ] o.a.s.SolrTestCaseJ4 Writing core.properties file to
>>/Users/jenkinworkspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jety-001/cores/collection1
>>   [junit4]   2> 1476354 INFO
>>(TEST-CdrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] e.j.s.Server jetty-9.2.11.v20150529
>>   [junit4]   2> 1476358 INO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>   ] o.e.j.s.h.ContextHandler Started
>>o.e.j.s.ServletontextHandler@39ae247e{/,null,AVAILABLE}
>>   [junit4]   2> 1476375NFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA67171ABDFB670]) [
>>   ] o.e.j.s.ServerConnector Started
>>ServerConnector@cbbd36{SSL-http/1.1}{127.0.0.1:58155}
>>   [junit4]   2> 1476376 INFO
>>(TEST-CcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.e.j.Server Started @1482014ms
>>   [junit4]   2> 1476376 INFO
>>(TESTCdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] oa.s.c.s.e.JettySolrRunner Jetty properties: {shards=shard1,
>>hostContext=/, hostPort=58155,
>>coRootDirectory=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/olr/b
>>uild/solr-core/test/J1/temp/solr.cloud.CdcrReplicationDistributedZkTst_FA
>>6717C1ABDFB670-001/jetty-001/cores}
>>   [junit4]   2> 1476378 INFO
>>(TEST-CdcrReplicatioistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.ss.SolrDispatchFilter SolrDispatchFilter.init():
>>sun.misc.Launcher$AppClassoader@74a14482
>>   [junit4]   2> 1476378 INFO
>>(TEST-CdcrReplicationDistibutedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.SolrResorceLoader new SolrResourceLoader for directory:
>>'/Users/jenkins/workspace/Lucene-Solr-trunkacOSX/solr/build/solr-core/te
>>st/J1/temp/solr.cloud.CdcrReplicaionDistributedZkTest_FA6717C1ABDFB670-00
>>1/jetty-001/'
>>   [junit4]   2> 176419 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1BDFB670]) [
>>   ] o.a.s.c.SolrXmlConfig ading container configuration from
>>/Users/jenkins/workspace/Lucne-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrRepicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-001/solr.xml
>>  [junit4]   2> 1476435 INFO
>>(TEST-CdcrReplicationDistributZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.CoresLoctor Config-defined core root directory:
>>/Users/jenkins/workspace/Lucene-Sor-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrRepliationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-001/cores
>> [junit4]   2> 1476435 INFO
>>(TEST-CdcrReplicationDistributedZkTet.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.CoreContainer New CoreCotainer 1924536609
>>   [junit4]   2> 1476435 INFO
>>(TEST-CdcrReplicationDisributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.CoreCtainer Loading cores into CoreContainer
>>[instanceDir=/Users/jenins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/
>>solr-core/test/J1/temp/olr.cloud.CdcrReplicationDistributedZkTest_FA6717C
>>1ABDFB670-001/jetty-00/]
>>   [junit4]   2> 1476435 INFO
>>(TEST-CdcrReplicaonDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.as.c.CoreContainer loading shared library:
>>/Users/jenkins/workspace/Lucene-olr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDtributedZkTest_FA6717C1ABDFB670-001
>>/jetty-001/lib
>>   [junit4]  2> 1476436 WARN
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA617C1ABDFB670]) [
>>   ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to d to
>>classloader: lib (resolved as:
>>/Users/jenkins/workspace/Lucene-Solr-runk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jety-001/lib).
>>   [junit4]   2> 1476451 INFO
>>(TEST-CdcrRepicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.h.c.HttpShardHandlerFactory creed with socketTimeout :
>>90000,urlScheme : ,connTimeout : 15000,maxConnectinsPerHost :
>>20,maxConnections : 10000,corePoolSize : 0,maximmPoolSize :
>>2147483647,maxThreadIdleTime : 5,sizeOfQuee : -1,fairnessPolicy :
>>false,useRetries : false,
>>   [junit]   2> 1476457 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.u.UpdateShardHandler Creatg UpdateShardHandler HTTP client
>>with params: socketTimeout=340000&connTimout=45000&retry=true
>>   [junit4]   2> 1476459 INFO
>>(TEST-CdrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDB670]) [
>>   ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggeFactory
>>   [junit4]   2> 1476459 INFO
> TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA67171ABDFB670]) [
>>   ] o.a.s.l.LogWatcher Registering Log Listener [Log4j
> (org.slf4j.impl.Log4jLoggerFactory)]
>>   [junit4]   2> 1476459 INFO
>>(TEST-CdcrRepcationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.Zkontainer Zookeeper client=127.0.0.1:58151/solr
>>   [junit4]  2> 1476460 INFO
>>(TEST-CdcrReplicationDistributedZkTestdoTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.ZkController zkHost includes chroot
>>   [junit4]   2>476505 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ADFB670])
>>[n:127.0.0.1:58155_    ] o.a.s.c.ZkController Regiser node as live in
>>ZooKeeper:/live_nodes/127.0.0.1:5815_
>>   [junit4]   2> 1476525 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])[n:127.0.0.1:58155_
>>    ] o.a.s.c.Overseer Overseer (id=null) closing
>>   [jnit4]   2> 1476526 INFO
>>(TEST-CdcrReplicationDistributedZkTet.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58155_   ] o.a.s.c.LeaderElector Joined leadership election
>>with path:
>>/overseer_elect/election/942106022001372-127.0.0.1:58155_-n_0000000000
>>   [junit4]   2> 1476528 INFO
>>(TEST-CdcrRplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>n:127.0.0.1:58155_    ] o.a.s.c.OverseerElectionContextI am going to be
>>the leader 127.0.0.1:58155_
>>   [junit4]   2> 1476531 INFO
>>(TEST-CdcrReplationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58155    ] o.a.s.c.Overseer Overseer
>>(id=94210602200137732-127.0..1:58155_-n_0000000000) starting
>>   [junit4]   2> 147651 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58155_    ].a.s.c.OverseerAutoReplicaFailoverThread
>>Starting OverseerAutoReplicaFailoerThread
>>autoReplicaFailoverWorkLoopDelay=10000
>>autoReplicaailoverWaitAfterExpiration=30000
>>autoReplicaFailoverBaNodeExpiration=60000
>>   [junit4]   2> 1476552 INFO
>>(OveseerCollectionProcessor-94210602200137732-127.0.0.1:58155_-n_00000000
>>00) [127.0.0.1:58155_    ] o.a.s.c.OverseerCollectionProcessor Process
>>currnt queue of collection creations
>>   [junit4]   2> 1476553 INFO
>>(OverseeStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.Overseer Starting to work on the main
>>queue
>>   [junit4]   2> 1476561 INFO
>>(TEST-CdcrReplicationDistribedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:5855_    ] o.a.s.c.CoreContainer No authentication plugin
>>used.
>>   [junt4]   2> 1476563 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTestseed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58155_    ] o.a.s.c.oreContainer Security conf doesn't
>>exist. Skipping setup for authorizaion module.
>>   [junit4]   2> 1476567 INFO
>>(TEST-CdcrReplicationDistributedZkTestoTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58155_    ] o.a.s.c.Coresocator Looking for core
>>definitions underneath
>>/Users/jenkins/workspaceLucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-001/cores
>>   [junit4]   2> 1476568 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58155_    ] o.a.s.c.SolrCore Created CoreDescriptor:
>>{name=collection1, cong=solrconfig.xml, transient=false,
>>schema=schema.xml, loadOnStartup=true,
> configSetProperties=configsetprops.json,
>>instanceDir=/Users/enkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/s
> olr-core/test/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA677C1
>>ABDFB670-001/jetty-001/cores/collection1, collection=tmp_collection,
>>absoteInstDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/bui
>>ld/sor-core/test/J1/temp/solr.cloud.CdcrReplicationDistributedZkTet_FA67
>>17C1ABDFB670-001/jetty-001/cores/collection1/, oreNodeName=,
>>dataDir=data/, shard=}
>>   [junit4]   2> 1476569INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6711ABDFB670])
>>[n:127.0.0.1:58155_    ] o.a.s.c.CoresLocator Found core colletion1 in
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/slr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplictionDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-001/cores/collection1/
>>   [junit4]   2> 76569 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABFB670])
>>[n:127.0.0.1:58155_    ] o.a.s.c.CoresLocator Found  core definitions
>>   [junit4]   2> 1476570 INFO
>>(TESTCdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0:58155_    ] o.a.s.s.SolrDispatchFilter
>>user.dir=/Users/jenkins/workspace/ucene-Solr-trunk-MacOSX/solr/build/solr
>>-core/test/J1
>>   [jnit4]   2> 1476571 INFO
>>(TEST-CdcrReplicationDistributdZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58155_    ] o.a.s.sSolrDispatchFilter
>>SolrDispatchFilter.init() do
>>   [junit4]   2> 1476571 INFO
>>(coreLoadExecutor-4271-thead-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collecton   x:collection1]
>>o.a.s.c.ZkController publishing state=down
>>   [junit4]   2> 76571 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:5815_)
>>[n:127.0.0.1:58155_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkContoller numShards not found on descriptor - reading it from
>>system property
>>   [junit4]   2> 1476572 INFO
>>(zkCallback-1229-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_    ] o.a.s.c.DistributedQueue NodeChildrenChanged
>>fired on path /overseer/queue state SyncConnected
>>   [junit4]   2> 1476572 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkController look for our core node name
>>   [junit4]   2> 1476574NFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.Overseer processMessage: quueSize: 1,
>>message = {
>>   [junit4]   2>   "core":"collection1",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"https://127.0.0.1:58155";,
>>   [junit4]   2>   "node_name":"127.0.0.1:58155_",
>>   [junit4]   2>   "numShards":"2",
>>   [junit4]   2>   "state":"down",
>>   [junit4]   2>   "shard":null,
>>   [junit4]   2>   "collection":"tmp_collection",
>>   [junit4]   2>   "operation":"state"} current state version: 0
>>   [junit4]   2> 1476575 INFO
>>(OverseerStateUpdate-942106022137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.o.RplicaMutator Update state
>>numShards=2 message={
>>   [junit]   2>   "core":"collection1",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"https://127.0.0.1:58155";,
>>   [junit4]   2>   "node_name":"127.0.0.1:58155_",
>>   [junit4]   2>   "numShards":"2",
>> [junit4]   2>   "state":"down",
>>   [junit4]   2>   "shard":null,
>>   [junt4]   2>   "collection":"tmp_collection",
>>   [junit4]   2>   "operation":"ate"}
>>   [junit4]   2> 1476576 INFO
>>(OverseerStateUpdate-9421060220013773-127.0.0.1:58155_-n_0000000000)
>>[n:1270.0.1:58155_    ] o.a.s.c.o.ClusterStateMutator building a new
>>cName: tmp_collection
>>  junit4]   2> 1476576 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.158155_-n_0000000000)
>>[n:127.0.0.1:5815_    ] o.a.s.c.o.ReplicaMutator Assigning new node to
>>shard shard=shard2
>>   [junit4] 2> 1477574 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:5815_)
>>[n:127.0.0.1:58155_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkCtroller waiting to find shard id in clusterstate for
>>collection1
>>   [juni4]   2> 1477574 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:17.0.0.1:58155_)
>>[n:127.0.0.1:5855_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkController Check for collection zkNode:tmp_collection
>>   [junit4]   2> 1477576 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ cmp_collection   x:collection1]
>>o.a.s.c.ZkController Creating collection inZooKeeper:tmp_collection
>>   [junit4]   2> 1477576 INFO
>>(coeLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:555_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkController Looking for colection configName
>>   [junit4]   2> 1477577 INFO
>>(coreLoadExecutor-427-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkController Only one config set found in zk - using it:conf1
>>   [junit4]   2> 1477582 INFO
> coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.158155_ c:tmp_collection   x:collection1]
>>o.a.s.c.SolrResourceLoaer new SolrResourceLoader for directory:
>>'/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/te
>>st/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-00
>>1/jetty-00cores/collection1/'
>>   [junit4]   2> 1477599 INFO
>>(coreLoadExecutor-4271-hread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tp_collection   x:collection1] o.a.s.c.Config
>>loaded config solrconfig.xml with version 0
>>   [junit4]   2> 1477606 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:585_ c:tmp_collection   x:collection1] o.a.s.c.SolrConfig
>>current version ofrequestparams : -1
>>   [junit4]   2> 1477608 INFO
>>(coreLoadExecutor-4271-thread-1-proceing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection   x:collectio1] o.a.s.c.SolrConfig
>>Using Lucene MatchVersion: 6.0.0
>>   [unit4]   2> 1477612 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection   x:collection1] o.a.s.c.Config
>>Loaded SolrConfig: solrconfig.xml
>>   [junit4]   21477615 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_
>>[n:127.0.0.1:58155_ c:tmp_collection   x:collectio1] o.a.s.s.IndexSchema
>>Reading Solr Schema from /configs/conf1/schema.xml
>>   [junit4]   2> 1477617 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_coection   x:collection1] o.a.s.s.IndexSchema
>>[collection1] Schema name=test>   [junit4]   2> 1477713 INFO
>>(coreLoadExecutor-4271-thread-1-processingn:127.0.0.1:58155_)
>>[n:127.0.0.:58155_ c:tmp_collection   x:collection1] o.a.s.s.IndexSchema
>>default search field in schema is text
>>   [junit4]   2> 1477714 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:7.0.0.1:58155_ c:tmp_collection   x:collection1] o.a.s.s.IndexSchema
>>uniqu key field: id
>>   [junit4]   2> 1477714 INFO
>>(coreLoadExecutor-4271thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tp_collection   x:collection1]
>>o.a.s.s.FileExchangeRateProvide Reloading exchange rates from file
>>currency.xml
>>   [junit4]   2> 477719 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:7.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection   x:collection1]
>>o.as.s.FileExchangeRateProvider Reloading exchange rates from file
>>currencyxml
>>   [junit4]   2> 1477777 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:5815)
>>[n:127.0.0.1:58155_ c:tmp_collection   x:collection1]
>>o.a.s.c.ConfigSetroperties Did not find ConfigSet properties
>>   [junit4]   2> org.apache.solr.core.SolrResourceNotFoundEeption:
>>Can't find resource 'configsetprops.json' in classpath or
>>'/confis/conf1',
>>cwd=/Users/jenkins/workspace/Lucene-Solr-trun-MacOSX/solr/build/solr-core
>>/test/J1
>>   [junit4]   2>       at
>>org.apae.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoad
>>er.java:9)
>>   [junit4]   2>       at
>>org.apache.solr.core.ConfigSetProperties.readFromResourceLoader(ConfietP
>>roperties.java:49)
>>   [junit4]   2>       at
>>org.apache.solr.core.ConfigStService.createConfigSetProperties(ConfigSetS
>>ervice.java:114)
>>   [junit4]   2>at
>>org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java76)
>>   [junit4]   2>       at
>>org.apache.solr.core.CoreContainer.create(CorContainer.java:668)
>>   [junit4]   2>       at
>>org.apache.solr.core.CoreConiner$1.call(CoreContainer.java:397)
>>   [junit4]   2>       at
>>org.apache.sol.core.CoreContainer$1.call(CoreContainer.java:388)
>>   [junit4]   2>       at
>>java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>   [junit4]   2>       at
>>org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(
>>Executortil.java:156)
>>   [junit4]   2>       at
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>>1142)
>>   [junit4]  2>        at
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>>:617)
>>   [junit4]   2>       a java.lang.Thread.run(Thrad.java:745)
>>   [junit4]   2> 1477777 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection   x:collection1]
>>o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration
>>from collection tmp_collection
>>   [junit4]   2> 1477777 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
>>   [junit4]   2> 1477778 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at
>>[/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/te
>>st/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-00
>>1/jetty-001/cores/collection1/], dataDir=[null]
>>   [junit4]   2> 1477778 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to
>>JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a3bc8e8
>>   [junit4]   2> 1477778 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.CachingDirectoryFactory return new directory for
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-001/cores/collection1/data
>>   [junit4]   2> 1477778 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SolrCore New index directory detected: old=null
>>new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core
>>/test/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670
>>-001/jetty-001/cores/collection1/data/index/
>>   [junit4]   2> 1477778 WARN
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SolrCore [collection1] Solr index directory
>>'/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/te
>>st/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-00
>>1/jetty-001/cores/collection1/data/index' doesn't exist. Creating new
>>index...
>>   [junit4]   2> 1477779 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.CachingDirectoryFactory return new directory for
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-001/cores/colltion1/data/index
>>   [junit4]   2> 1477779 INFO
>>(coreLoadExecutor-471-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_ollection s:shard2 r:core_node1 x:collection1]
>>o.a.s.u.andomMergePolicy RandomMergePolicy wrapping class
>>org.apache.lucene.index.TieredMergePolicy: [TieredMrgePolicy:
>>maxMergeAtOnce=46, maxMergeAtOnceExplicit=47,
>>maxMergedSegmentMB83.1484375, floorSegmentMB=0.607421875,
>>forceMergeDeletesPctAllowed=3.220248038000804, segmentsPerTier=24.0,
>>maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>>   [junit4   2> 1477779 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:colection1]
>>o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
>>   [junit4]   2>       commit{dir=MockDirectoryWrapper(RAMDirectory@5fb00b23
>>lockFactry=org.apache.lucene.store.SingleInstanceLockFactory@2379c13f),se
>>gFN=segments_1,generation=1}
>>   [junit4]   2> 1477779 INFO
>>(creLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SolrCore newest comit generation = 1
>>   [junit4]   2> 1477780 INFO
>>(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1] o.a.s.c.SolrCore
>>Looking for old index directories to cleanup for core collection1 in
>>Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/uild/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-001/cores/collection1/data/
>>   [junit4]   2>1477781 WARN
>>(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.DirectoryFactory
>>/Users/jenkins/workace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.clod.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty001/cores/collection1/data/ does not point to a valid data
>>directory; skipping clean-up of old index directories.
>>   [junit4]   2>1477784 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.u.p.UpdateRequestProcessorChain creating
>>updateRequestProcessorChain "cdcr-processor-chain"
>>   [junit4]   2> 1477784 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SolrCore no updateRequestProcessorChain defined as default,
>>creating implicit default
>>   [junit4]   2> 1477789 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 1477790 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 1477791 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 1477792 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 1477797 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.RequestHandlers Registered paths:
>>/admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/se
>>gments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/
>>admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/cdcr
>>,/update,/admin/file
>>   [junit4]   2> 1477797 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SolrCore Using default statsCache cache:
>>org.apache.solr.search.stats.LocalStatsCache
>>   [junit4]   2> 1477798 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.u.UpdateHandler Using UpdateLog implementation:
>>org.apache.solr.update.CdcrUpdateLog
>>   [junit4]   2> 1477798 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH
>>numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
>>   [junit4]   2> 1477799 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SolrCore Hard AutoCommit: disabled
>>   [junit4]   2> 1477799 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SolrCore Soft AutoCommit: disabled
>>   [junit4]   2> 1477799 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
>>org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
>>minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807,
>>maxMergeSizeForForcedMerge=9223372036854775807,
>>calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
>>maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
>>   [junit4]   2> 1477799 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
>>   [junit4]   2>       commit{dir=MockDirectoryWrapper(RAMDirectory@5fb00b23
>>lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2379c13f),se
>>gFN=segments_1,generation=1}
>>   [junit4]   2> 1477799 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SolrCore newest commit generation = 1
>>   [junit4]   2> 1477799 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.s.SolrIndexSearcher Opening Searcher@7cf0c392[collection1] main
>>   [junit4]   2> 1477801 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the
>>RestManager with znodeBase: /configs/conf1
>>   [junit4]   2> 1477802 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
>>znodeBase: /configs/conf1
>>   [junit4]   2> 1477803 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.r.RestManager Initializing RestManager with initArgs: {}
>>   [junit4]   2> 1477803 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using
>>ZooKeeperStorageIO:path=/configs/conf1
>>   [junit4]   2> 1477804 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.r.ManagedResourceStorage No data found for znode
>>/configs/conf1/_rest_managed.json
>>   [junit4]   2> 1477804 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json
>>using ZooKeeperStorageIO:path=/configs/conf1
>>   [junit4]   2> 1477804 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.r.RestManager Initializing 0 registered ManagedResources
>>   [junit4]   2> 1477804 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.h.ReplicationHandler Commits will be reserved for  10000
>>   [junit4]   2> 1477812 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.h.CdcrBufferStateManager Created znode
>>/collections/tmp_collection/cdcr/state/buffer
>>   [junit4]   2> 1477816 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.h.CdcrProcessStateManager Created znode
>>/collections/tmp_collection/cdcr/state/process
>>   [junit4]   2> 1477819 INFO
>>(searcherExecutor-4272-thread-1-processing-n:127.0.0.1:58155_
>>x:collection1 s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1] o.a.s.c.SolrCore
>>[collection1] Registered new searcher Searcher@7cf0c392[collection1]
>>main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>   [junit4]   2> 1477819 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.u.UpdateLog Looking up max value of version field to seed version
>>buckets
>>   [junit4]   2> 1477819 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536
>>version buckets from index
>>   [junit4]   2> 1477819 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.u.VersionInfo No terms found for _version_, cannot seed version
>>bucket highest value from index
>>   [junit4]   2> 1477819 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.u.UpdateLog Could not find max version in index or recent updates,
>>using new clock 1507369637301452800
>>   [junit4]   2> 1477821 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.u.UpdateLog Took 2 ms to seed version buckets with highest version
>>1507369637301452800
>>   [junit4]   2> 1477821 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.ZkController watch zkdir /configs/conf1
>>   [junit4]   2> 1477822 INFO
>>(coreLoadExecutor-4271-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_ c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.CoreContainer registering core: collection1
>>   [junit4]   2> 1477823 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.ZkController Register replica - core:collection1
>>address:https://127.0.0.1:58155 collection:tmp_collection shard:shard2
>>   [junit4]   2> 1477832 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.LeaderElector Joined leadership election with path:
>>/collections/tmp_collection/leader_elect/shard2/election/94210602200137732
>>-core_node1-n_0000000000
>>   [junit4]   2> 1477834 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.ShardLeaderElectionContext Running the leader process for shard
>>shard2
>>   [junit4]   2> 1477837 INFO
>>(zkCallback-1229-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_    ] o.a.s.c.DistributedQueue NodeChildrenChanged
>>fired on path /overseer/queue state SyncConnected
>>   [junit4]   2> 1477838 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
>>   [junit4]   2> 1477838 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
>>   [junit4]   2> 1477839 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:58155/collection1/
>>   [junit4]   2> 1477839 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
>>   [junit4]   2> 1477839 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.SyncStrategy https://127.0.0.1:58155/collection1/ has no replicas
>>   [junit4]   2> 1477839 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.ShardLeaderElectionContext I am the new leader:
>>https://127.0.0.1:58155/collection1/ shard2
>>   [junit4]   2> 1477839 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.Overseer processMessage: queueSize: 1,
>>message = {
>>   [junit4]   2>   "operation":"leader",
>>   [junit4]   2>   "shard":"shard2",
>>   [junit4]   2>   "collection":"tmp_collection"} current state version: 1
>>   [junit4]   2> 1477845 INFO
>>(zkCallback-1229-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_    ] o.a.s.h.CdcrProcessStateManager Received new
>>leader state @ tmp_collection:shard2
>>   [junit4]   2> 1477849 INFO
>>(zkCallback-1229-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_    ] o.a.s.c.DistributedQueue NodeChildrenChanged
>>fired on path /overseer/queue state SyncConnected
>>   [junit4]   2> 1477851 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.Overseer processMessage: queueSize: 1,
>>message = {
>>   [junit4]   2>   "operation":"leader",
>>   [junit4]   2>   "shard":"shard2",
>>   [junit4]   2>   "collection":"tmp_collection",
>>   [junit4]   2>   "base_url":"https://127.0.0.1:58155";,
>>   [junit4]   2>   "core":"collection1",
>>   [junit4]   2>   "state":"active"} current state version: 1
>>   [junit4]   2> 1478004 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.ZkController We are https://127.0.0.1:58155/collection1/ and
>>leader is https://127.0.0.1:58155/collection1/
>>   [junit4]   2> 1478004 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.ZkController No LogReplay needed for core=collection1
>>baseURL=https://127.0.0.1:58155
>>   [junit4]   2> 1478004 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.ZkController I am the leader, no recovery necessary
>>   [junit4]   2> 1478005 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.ZkController publishing state=active
>>   [junit4]   2> 1478005 INFO
>>(coreZkRegister-4265-thread-1-processing-n:127.0.0.1:58155_ x:collection1
>>s:shard2 c:tmp_collection r:core_node1) [n:127.0.0.1:58155_
>>c:tmp_collection s:shard2 r:core_node1 x:collection1]
>>o.a.s.c.ZkController numShards not found on descriptor - reading it from
>>system property
>>   [junit4]   2> 1478007 INFO
>>(zkCallback-1229-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_    ] o.a.s.c.DistributedQueue NodeChildrenChanged
>>fired on path /overseer/queue state SyncConnected
>>   [junit4]   2> 1478008 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.Overseer processMessage: queueSize: 1,
>>message = {
>>   [junit4]   2>   "core":"collection1",
>>   [junit4]   2>   "core_node_name":"core_node1",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"https://127.0.0.1:58155";,
>>   [junit4]   2>   "node_name":"127.0.0.1:58155_",
>>   [junit4]   2>   "numShards":"2",
>>   [junit4]   2>   "state":"active",
>>   [junit4]   2>   "shard":"shard2",
>>   [junit4]   2>   "collection":"tmp_collection",
>>   [junit4]   2>   "operation":"state"} current state version: 2
>>   [junit4]   2> 1478010 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.o.ReplicaMutator Update state
>>numShards=2 message={
>>   [junit4]   2>   "core":"collection1",
>>   [junit4]   2>   "core_node_name":"core_node1",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"https://127.0.0.1:58155";,
>>   [junit4]   2>   "node_name":"127.0.0.1:58155_",
>>   [junit4]   2>   "numShards":"2",
>>   [junit4]   2>   "state":"active",
>>   [junit4]   2>   "shard":"shard2",
>>   [junit4]   2>   "collection":"tmp_collection",
>>   [junit4]   2>   "operation":"state"}
>>   [junit4]   2> 1478440 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.SolrTestCaseJ4 Writing core.properties file to
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-002/cores/collection1
>>   [junit4]   2> 1478443 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.e.j.s.Server jetty-9.2.11.v20150529
>>   [junit4]   2> 1478446 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.e.j.s.h.ContextHandler Started
>>o.e.j.s.ServletContextHandler@1584685b{/,null,AVAILABLE}
>>   [junit4]   2> 1478450 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.e.j.s.ServerConnector Started
>>ServerConnector@5f010186{SSL-http/1.1}{127.0.0.1:58162}
>>   [junit4]   2> 1478450 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.e.j.s.Server Started @1484087ms
>>   [junit4]   2> 1478450 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {shards=shard2,
>>hostContext=/, hostPort=58162,
>>coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/b
>>uild/solr-core/test/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA
>>6717C1ABDFB670-001/jetty-002/cores}
>>   [junit4]   2> 1478453 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init():
>>sun.misc.Launcher$AppClassLoader@74a14482
>>   [junit4]   2> 1478453 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory:
>>'/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/te
>>st/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-00
>>1/jetty-002/'
>>   [junit4]   2> 1478485 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.SolrXmlConfig Loading container configuration from
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-002/solr.xml
>>   [junit4]   2> 1478493 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.CoresLocator Config-defined core root directory:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-002/cores
>>   [junit4]   2> 1478494 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.CoreContainer New CoreContainer 1822857970
>>   [junit4]   2> 1478494 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.CoreContainer Loading cores into CoreContainer
>>[instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/
>>solr-core/test/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C
>>1ABDFB670-001/jetty-002/]
>>   [junit4]   2> 1478494 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.CoreContainer loading shared library:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-002/lib
>>   [junit4]   2> 1478494 WARN
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to
>>classloader: lib (resolved as:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-002/lib).
>>   [junit4]   2> 1478503 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout :
>>90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost :
>>20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize :
>>2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy :
>>false,useRetries : false,
>>   [junit4]   2> 1478508 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client
>>with params: socketTimeout=340000&connTimeout=45000&retry=true
>>   [junit4]   2> 1478510 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>   [junit4]   2> 1478510 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.l.LogWatcher Registering Log Listener [Log4j
>>(org.slf4j.impl.Log4jLoggerFactory)]
>>   [junit4]   2> 1478510 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58151/solr
>>   [junit4]   2> 1478510 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.ZkController zkHost includes chroot
>>   [junit4]   2> 1479542 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.c.ZkController Register node as live in
>>ZooKeeper:/live_nodes/127.0.0.1:58162_
>>   [junit4]   2> 1479548 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.c.Overseer Overseer (id=null) closing
>>   [junit4]   2> 1479549 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.c.LeaderElector Joined leadership election
>>with path:
>>/overseer_elect/election/94210602200137734-127.0.0.1:58162_-n_0000000001
>>   [junit4]   2> 1479550 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.c.LeaderElector Watching path
>>/overseer_elect/election/94210602200137732-127.0.0.1:58155_-n_0000000000
>>to know if I could be the leader
>>   [junit4]   2> 1479552 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.c.CoreContainer No authentication plugin
>>used.
>>   [junit4]   2> 1479553 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.c.CoreContainer Security conf doesn't
>>exist. Skipping setup for authorization module.
>>   [junit4]   2> 1479554 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.c.CoresLocator Looking for core
>>definitions underneath
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-002/cores
>>   [junit4]   2> 1479555 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.c.SolrCore Created CoreDescriptor:
>>{name=collection1, config=solrconfig.xml, transient=false,
>>schema=schema.xml, loadOnStartup=true,
>>configSetProperties=configsetprops.json,
>>instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/s
>>olr-core/test/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1
>>ABDFB670-001/jetty-002/cores/collection1, collection=tmp_collection,
>>absoluteInstDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/bui
>>ld/solr-core/test/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA67
>>17C1ABDFB670-001/jetty-002/cores/collection1/, coreNodeName=,
>>dataDir=data/, shard=}
>>   [junit4]   2> 1479555 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.c.CoresLocator Found core collection1 in
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-002/cores/collection1/
>>   [junit4]   2> 1479555 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.c.CoresLocator Found 1 core definitions
>>   [junit4]   2> 1479558 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.s.SolrDispatchFilter
>>user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr
>>-core/test/J1
>>   [junit4]   2> 1479558 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58162_    ] o.a.s.s.SolrDispatchFilter
>>SolrDispatchFilter.init() done
>>   [junit4]   2> 1479558 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkController publishing state=down
>>   [junit4]   2> 1479558 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkController numShards not found on descriptor - reading it from
>>system property
>>   [junit4]   2> 1479559 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkController look for our core node name
>>   [junit4]   2> 1479560 INFO
>>(zkCallback-1229-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_    ] o.a.s.c.DistributedQueue NodeChildrenChanged
>>fired on path /overseer/queue state SyncConnected
>>   [junit4]   2> 1479561 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.Overseer processMessage: queueSize: 1,
>>message = {
>>   [junit4]   2>   "core":"collection1",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"https://127.0.0.1:58162";,
>>   [junit4]   2>   "node_name":"127.0.0.1:58162_",
>>   [junit4]   2>   "numShards":"2",
>>   [junit4]   2>   "state":"down",
>>   [junit4]   2>   "shard":null,
>>   [junit4]   2>   "collection":"tmp_collection",
>>   [junit4]   2>   "operation":"state"} current state version: 3
>>   [junit4]   2> 1479563 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.o.ReplicaMutator Update state
>>numShards=2 message={
>>   [junit4]   2>   "core":"collection1",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"https://127.0.0.1:58162";,
>>   [junit4]   2>   "node_name":"127.0.0.1:58162_",
>>   [junit4]   2>   "numShards":"2",
>>   [junit4]   2>   "state":"down",
>>   [junit4]   2>   "shard":null,
>>   [junit4]   2>   "collection":"tmp_collection",
>>   [junit4]   2>   "operation":"state"}
>>   [junit4]   2> 1479563 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.o.ReplicaMutator Collection already
>>exists with numShards=2
>>   [junit4]   2> 1479563 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.o.ReplicaMutator Assigning new node to
>>shard shard=shard1
>>   [junit4]   2> 1480561 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkController waiting to find shard id in clusterstate for
>>collection1
>>   [junit4]   2> 1480562 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkController Check for collection zkNode:tmp_collection
>>   [junit4]   2> 1480563 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1]
>>o.a.s.c.ZkController Collection zkNode exists
>>   [junit4]   2> 1480565 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1]
>>o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory:
>>'/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/te
>>st/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-00
>>1/jetty-002/cores/collection1/'
>>   [junit4]   2> 1480589 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1] o.a.s.c.Config
>>loaded config solrconfig.xml with version 0
>>   [junit4]   2> 1480599 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1] o.a.s.c.SolrConfig
>>current version of requestparams : -1
>>   [junit4]   2> 1480602 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1] o.a.s.c.SolrConfig
>>Using Lucene MatchVersion: 6.0.0
>>   [junit4]   2> 1480623 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1] o.a.s.c.Config
>>Loaded SolrConfig: solrconfig.xml
>>   [junit4]   2> 1480626 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1] o.a.s.s.IndexSchema
>>Reading Solr Schema from /configs/conf1/schema.xml
>>   [junit4]   2> 1480630 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1] o.a.s.s.IndexSchema
>>[collection1] Schema name=test
>>   [junit4]   2> 1480698 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1] o.a.s.s.IndexSchema
>>default search field in schema is text
>>   [junit4]   2> 1480698 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1] o.a.s.s.IndexSchema
>>unique key field: id
>>   [junit4]   2> 1480699 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1]
>>o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file
>>currency.xml
>>   [junit4]   2> 1480702 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1]
>>o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file
>>currency.xml
>>   [junit4]   2> 1480755 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1]
>>o.a.s.c.ConfigSetProperties Did not find ConfigSet properties
>>   [junit4]   2> org.apache.solr.core.SolrResourceNotFoundException:
>>Can't find resource 'configsetprops.json' in classpath or
>>'/configs/conf1',
>>cwd=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core
>>/test/J1
>>   [junit4]   2>       at
>>org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoad
>>er.java:99)
>>   [junit4]   2>       at
>>org.apache.solr.core.ConfigSetProperties.readFromResourceLoader(ConfigSetP
>>roperties.java:49)
>>   [junit4]   2>       at
>>org.apache.solr.core.ConfigSetService.createConfigSetProperties(ConfigSetS
>>ervice.java:114)
>>   [junit4]   2>       at
>>org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:76)
>>   [junit4]   2>       at
>>org.apache.solr.core.CoreContainer.create(CoreContainer.java:668)
>>   [junit4]   2>       at
>>org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:397)
>>   [junit4]   2>       at
>>org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:388)
>>   [junit4]   2>       at
>>java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>   [junit4]   2>       at
>>org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(
>>ExecutorUtil.java:156)
>>   [junit4]   2>       at
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
>>1142)
>>   [junit4]   2>       at
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>>:617)
>>   [junit4]   2>       at java.lang.Thread.run(Thread.java:745)
>>   [junit4]   2> 1480756 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection   x:collection1]
>>o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration
>>from collection tmp_collection
>>   [junit4]   2> 1480756 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
>>   [junit4]   2> 1480756 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at
>>[/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/te
>>st/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-00
>>1/jetty-002/cores/collection1/], dataDir=[null]
>>   [junit4]   2> 1480756 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to
>>JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5a3bc8e8
>>   [junit4]   2> 1480757 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.CachingDirectoryFactory return new directory for
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-002/cores/collection1/data
>>   [junit4]   2> 1480757 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore New index directory detected: old=null
>>new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core
>>/test/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670
>>-001/jetty-002/cores/collection1/data/index/
>>   [junit4]   2> 1480757 WARN
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore [collection1] Solr index directory
>>'/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/te
>>st/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-00
>>1/jetty-002/cores/collection1/data/index' doesn't exist. Creating new
>>index...
>>   [junit4]   2> 1480757 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.CachingDirectoryFactory return new directory for
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-002/cores/collection1/data/index
>>   [junit4]   2> 1480758 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
>>org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
>>maxMergeAtOnce=46, maxMergeAtOnceExplicit=47,
>>maxMergedSegmentMB=83.1484375, floorSegmentMB=0.607421875,
>>forceMergeDeletesPctAllowed=3.2202480380008804, segmentsPerTier=24.0,
>>maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>>   [junit4]   2> 1480758 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
>>   [junit4]   2>       commit{dir=MockDirectoryWrapper(RAMDirectory@426c82f8
>>lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7b4e8f87),se
>>gFN=segments_1,generation=1}
>>   [junit4]   2> 1480758 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore newest commit generation = 1
>>   [junit4]   2> 1480760 INFO
>>(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore
>>Looking for old index directories to cleanup for core collection1 in
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-002/cores/collection1/data/
>>   [junit4]   2> 1480760 WARN
>>(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.DirectoryFactory
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-002/cores/collection1/data/ does not point to a valid data
>>directory; skipping clean-up of old index directories.
>>   [junit4]   2> 1480764 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.u.p.UpdateRequestProcessorChain creating
>>updateRequestProcessorChain "cdcr-processor-chain"
>>   [junit4]   2> 1480764 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore no updateRequestProcessorChain defined as default,
>>creating implicit default
>>   [junit4]   2> 1480771 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 1480773 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 1480774 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 1480776 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>>   [junit4]   2> 1480781 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.RequestHandlers Registered paths:
>>/admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/se
>>gments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/
>>admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/cdcr
>>,/update,/admin/file
>>   [junit4]   2> 1480782 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore Using default statsCache cache:
>>org.apache.solr.search.stats.LocalStatsCache
>>   [junit4]   2> 1480783 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.u.UpdateHandler Using UpdateLog implementation:
>>org.apache.solr.update.CdcrUpdateLog
>>   [junit4]   2> 1480783 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH
>>numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
>>   [junit4]   2> 1480784 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore Hard AutoCommit: disabled
>>   [junit4]   2> 1480784 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore Soft AutoCommit: disabled
>>   [junit4]   2> 1480785 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
>>org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
>>minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807,
>>maxMergeSizeForForcedMerge=9223372036854775807,
>>calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
>>maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
>>   [junit4]   2> 1480785 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
>>   [junit4]   2>       commit{dir=MockDirectoryWrapper(RAMDirectory@426c82f8
>>lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7b4e8f87),se
>>gFN=segments_1,generation=1}
>>   [junit4]   2> 1480786 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SolrCore newest commit generation = 1
>>   [junit4]   2> 1480786 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.s.SolrIndexSearcher Opening Searcher@6a060f72[collection1] main
>>   [junit4]   2> 1480788 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the
>>RestManager with znodeBase: /configs/conf1
>>   [junit4]   2> 1480790 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
>>znodeBase: /configs/conf1
>>   [junit4]   2> 1480790 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.r.RestManager Initializing RestManager with initArgs: {}
>>   [junit4]   2> 1480790 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using
>>ZooKeeperStorageIO:path=/configs/conf1
>>   [junit4]   2> 1480792 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.r.ManagedResourceStorage No data found for znode
>>/configs/conf1/_rest_managed.json
>>   [junit4]   2> 1480793 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json
>>using ZooKeeperStorageIO:path=/configs/conf1
>>   [junit4]   2> 1480793 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.r.RestManager Initializing 0 registered ManagedResources
>>   [junit4]   2> 1480793 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.h.ReplicationHandler Commits will be reserved for  10000
>>   [junit4]   2> 1480800 INFO
>>(searcherExecutor-4284-thread-1-processing-n:127.0.0.1:58162_
>>x:collection1 s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore
>>[collection1] Registered new searcher Searcher@6a060f72[collection1]
>>main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>>   [junit4]   2> 1480800 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.u.UpdateLog Looking up max value of version field to seed version
>>buckets
>>   [junit4]   2> 1480800 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536
>>version buckets from index
>>   [junit4]   2> 1480800 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.u.VersionInfo No terms found for _version_, cannot seed version
>>bucket highest value from index
>>   [junit4]   2> 1480801 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.u.UpdateLog Could not find max version in index or recent updates,
>>using new clock 1507369640428306432
>>   [junit4]   2> 1480802 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.u.UpdateLog Took 2 ms to seed version buckets with highest version
>>1507369640428306432
>>   [junit4]   2> 1480803 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.ZkController watch zkdir /configs/conf1
>>   [junit4]   2> 1480804 INFO
>>(coreLoadExecutor-4283-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_ c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.CoreContainer registering core: collection1
>>   [junit4]   2> 1480805 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.ZkController Register replica - core:collection1
>>address:https://127.0.0.1:58162 collection:tmp_collection shard:shard1
>>   [junit4]   2> 1480815 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.LeaderElector Joined leadership election with path:
>>/collections/tmp_collection/leader_elect/shard1/election/94210602200137734
>>-core_node2-n_0000000000
>>   [junit4]   2> 1480817 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.ShardLeaderElectionContext Running the leader process for shard
>>shard1
>>   [junit4]   2> 1480820 INFO
>>(zkCallback-1229-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_    ] o.a.s.c.DistributedQueue NodeChildrenChanged
>>fired on path /overseer/queue state SyncConnected
>>   [junit4]   2> 1480821 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
>>   [junit4]   2> 1480821 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
>>   [junit4]   2> 1480821 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:58162/collection1/
>>   [junit4]   2> 1480822 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
>>   [junit4]   2> 1480822 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.SyncStrategy https://127.0.0.1:58162/collection1/ has no replicas
>>   [junit4]   2> 1480822 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.ShardLeaderElectionContext I am the new leader:
>>https://127.0.0.1:58162/collection1/ shard1
>>   [junit4]   2> 1480822 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.Overseer processMessage: queueSize: 1,
>>message = {
>>   [junit4]   2>   "operation":"leader",
>>   [junit4]   2>   "shard":"shard1",
>>   [junit4]   2>   "collection":"tmp_collection"} current state version: 4
>>   [junit4]   2> 1480827 INFO
>>(zkCallback-1232-thread-1-processing-n:127.0.0.1:58162_)
>>[n:127.0.0.1:58162_    ] o.a.s.h.CdcrProcessStateManager Received new
>>leader state @ tmp_collection:shard1
>>   [junit4]   2> 1480832 INFO
>>(zkCallback-1229-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_    ] o.a.s.c.DistributedQueue NodeChildrenChanged
>>fired on path /overseer/queue state SyncConnected
>>   [junit4]   2> 1480834 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.Overseer processMessage: queueSize: 1,
>>message = {
>>   [junit4]   2>   "operation":"leader",
>>   [junit4]   2>   "shard":"shard1",
>>   [junit4]   2>   "collection":"tmp_collection",
>>   [junit4]   2>   "base_url":"https://127.0.0.1:58162";,
>>   [junit4]   2>   "core":"collection1",
>>   [junit4]   2>   "state":"active"} current state version: 4
>>   [junit4]   2> 1480988 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.ZkController We are https://127.0.0.1:58162/collection1/ and
>>leader is https://127.0.0.1:58162/collection1/
>>   [junit4]   2> 1480988 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.ZkController No LogReplay needed for core=collection1
>>baseURL=https://127.0.0.1:58162
>>   [junit4]   2> 1480988 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.ZkController I am the leader, no recovery necessary
>>   [junit4]   2> 1480988 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.ZkController publishing state=active
>>   [junit4]   2> 1480988 INFO
>>(coreZkRegister-4278-thread-1-processing-n:127.0.0.1:58162_ x:collection1
>>s:shard1 c:tmp_collection r:core_node2) [n:127.0.0.1:58162_
>>c:tmp_collection s:shard1 r:core_node2 x:collection1]
>>o.a.s.c.ZkController numShards not found on descriptor - reading it from
>>system property
>>   [junit4]   2> 1480990 INFO
>>(zkCallback-1229-thread-1-processing-n:127.0.0.1:58155_)
>>[n:127.0.0.1:58155_    ] o.a.s.c.DistributedQueue NodeChildrenChanged
>>fired on path /overseer/queue state SyncConnected
>>   [junit4]   2> 1480991 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.Overseer processMessage: queueSize: 1,
>>message = {
>>   [junit4]   2>   "core":"collection1",
>>   [junit4]   2>   "core_node_name":"core_node2",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"https://127.0.0.1:58162";,
>>   [junit4]   2>   "node_name":"127.0.0.1:58162_",
>>   [junit4]   2>   "numShards":"2",
>>   [junit4]   2>   "state":"active",
>>   [junit4]   2>   "shard":"shard1",
>>   [junit4]   2>   "collection":"tmp_collection",
>>   [junit4]   2>   "operation":"state"} current state version: 5
>>   [junit4]   2> 1480993 INFO
>>(OverseerStateUpdate-94210602200137732-127.0.0.1:58155_-n_0000000000)
>>[n:127.0.0.1:58155_    ] o.a.s.c.o.ReplicaMutator Update state
>>numShards=2 message={
>>   [junit4]   2>   "core":"collection1",
>>   [junit4]   2>   "core_node_name":"core_node2",
>>   [junit4]   2>   "roles":null,
>>   [junit4]   2>   "base_url":"https://127.0.0.1:58162";,
>>   [junit4]   2>   "node_name":"127.0.0.1:58162_",
>>   [junit4]   2>   "numShards":"2",
>>   [junit4]   2>   "state":"active",
>>   [junit4]   2>   "shard":"shard1",
>>   [junit4]   2>   "collection":"tmp_collection",
>>   [junit4]   2>   "operation":"state"}
>>   [junit4]   2> 1481445 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.SolrTestCaseJ4 Writing core.properties file to
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-003/cores/collection1
>>   [junit4]   2> 1481448 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.e.j.s.Server jetty-9.2.11.v20150529
>>   [junit4]   2> 1481452 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.e.j.s.h.ContextHandler Started
>>o.e.j.s.ServletContextHandler@14cc4072{/,null,AVAILABLE}
>>   [junit4]   2> 1481455 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.e.j.s.ServerConnector Started
>>ServerConnector@3d7e821d{SSL-http/1.1}{127.0.0.1:58167}
>>   [junit4]   2> 1481455 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.e.j.s.Server Started @1487091ms
>>   [junit4]   2> 1481455 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {shards=shard3,
>>hostContext=/, hostPort=58167,
>>coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/b
>>uild/solr-core/test/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA
>>6717C1ABDFB670-001/jetty-003/cores}
>>   [junit4]   2> 1481458 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init():
>>sun.misc.Launcher$AppClassLoader@74a14482
>>   [junit4]   2> 1481458 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory:
>>'/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/te
>>st/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-00
>>1/jetty-003/'
>>   [junit4]   2> 1481491 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.SolrXmlConfig Loading container configuration from
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-003/solr.xml
>>   [junit4]   2> 1481514 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.CoresLocator Config-defined core root directory:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-003/cores
>>   [junit4]   2> 1481514 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.CoreContainer New CoreContainer 1475657514
>>   [junit4]   2> 1481514 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.CoreContainer Loading cores into CoreContainer
>>[instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/
>>solr-core/test/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C
>>1ABDFB670-001/jetty-003/]
>>   [junit4]   2> 1481515 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.CoreContainer loading shared library:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-003/lib
>>   [junit4]   2> 1481515 WARN
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to
>>classloader: lib (resolved as:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>/jetty-003/lib).
>>   [junit4]   2> 1481526 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout :
>>90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost :
>>20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize :
>>2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy :
>>false,useRetries : false,
>>   [junit4]   2> 1481530 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client
>>with params: socketTimeout=340000&connTimeout=45000&retry=true
>>   [junit4]   2> 1481533 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>>   [junit4]   2> 1481533 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.l.LogWatcher Registering Log Listener [Log4j
>>(org.slf4j.impl.Log4jLoggerFactory)]
>>   [junit4]   2> 1481533 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58151/solr
>>   [junit4]   2> 1481533 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670]) [
>>   ] o.a.s.c.ZkController zkHost includes chroot
>>   [junit4]   2> 1482570 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58167_    ] o.a.s.c.ZkController Register node as live in
>>ZooKeeper:/live_nodes/127.0.0.1:58167_
>>   [junit4]   2> 1482579 INFO
>>(TEST-CdcrReplicationDistributedZkTest.doTests-seed#[FA6717C1ABDFB670])
>>[n:127.0.0.1:58167_    ] o.a.s.c.Overseer Overseer (id=null) closing
>>   [junit4]   2> 1482580 INFO  (TEST-CdcrReplication
>>
>>[...truncated too long message...]
>>
>>et_collection s:shard2 r:core_node4 x:target_collection_shard2_replica1]
>>o.a.s.c.ZkTestServer connecting to 127.0.0.1:58151 58151
>>   [junit4]   2> 1924299 INFO  (Thread-3290) [    ] o.a.s.c.ZkTestServer
>>connecting to 127.0.0.1:58151 58151
>>   [junit4]   2> 1925315 WARN  (Thread-3290) [    ] o.a.s.c.ZkTestServer
>>Watch limit violations:
>>   [junit4]   2> Maximum concurrent create/delete watches above limit:
>>   [junit4]   2>
>>   [junit4]   2>       292     /solr/aliases.json
>>   [junit4]   2>       292     /solr/clusterstate.json
>>   [junit4]   2>       80      /solr/configs/conf1
>>   [junit4]   2>       37      /solr/collections/source_collection/state.json
>>   [junit4]   2>       35      /solr/collections/target_collection/state.json
>>   [junit4]   2>       3       
>> /solr/collections/source_collection/leaders/shard1
>>   [junit4]   2>       2       
>> /solr/collections/source_collection/leaders/shard2
>>   [junit4]   2>       2       /solr/collections/tmp_collection/leaders/shard1
>>   [junit4]   2>       2       /solr/collections/tmp_collection/leaders/shard2
>>   [junit4]   2>       2       
>> /solr/collections/target_collection/leaders/shard1
>>   [junit4]   2>       2       
>> /solr/collections/target_collection/leaders/shard2
>>   [junit4]   2>
>>   [junit4]   2> Maximum concurrent data watches above limit:
>>   [junit4]   2>
>>   [junit4]   2>
>>       45      /solr/collections/source_collection/cdcr/state/process
>>   [junit4]   2>       36      
>> /solr/collections/source_collection/cdcr/state/buffer
>>   [junit4]   2>       35      
>> /solr/collections/target_collection/cdcr/state/buffer
>>   [junit4]   2>
>>       35      /solr/collections/target_collection/cdcr/state/process
>>   [junit4]   2>       8       
>> /solr/collections/tmp_collection/cdcr/state/buffer
>>   [junit4]   2>       8       
>> /solr/collections/tmp_collection/cdcr/state/process
>>   [junit4]   2>
>>       3       
>> /solr/overseer_elect/election/94210602200137748-127.0.0.1:58273_-n_0000
>>000005
>>   [junit4]   2>
>>       3       
>> /solr/overseer_elect/election/94210602200137736-127.0.0.1:58167_-n_0000
>>000002
>>   [junit4]   2>
>>       3       
>> /solr/overseer_elect/election/94210602200137732-127.0.0.1:58155_-n_0000
>>000000
>>   [junit4]   2>
>>       3       
>> /solr/overseer_elect/election/94210602200137787-127.0.0.1:58295_-n_0000
>>000009
>>   [junit4]   2>
>>       2       
>> /solr/collections/source_collection/leader_elect/shard1/election/942106
>>02200137774-core_node1-n_0000000000
>>   [junit4]   2>
>>       2       
>> /solr/collections/target_collection/leader_elect/shard2/election/942106
>>02200137787-core_node3-n_0000000000
>>   [junit4]   2>
>>       2       
>> /solr/overseer_elect/election/94210602200137752-127.0.0.1:58315_-n_0000
>>000007
>>   [junit4]   2>
>>       2       
>> /solr/collections/target_collection/leader_elect/shard1/election/942106
>>02200137752-core_node1-n_0000000000
>>   [junit4]   2>
>>       2       
>> /solr/collections/target_collection/leader_elect/shard2/election/942106
>>02200137746-core_node2-n_0000000000
>>   [junit4]   2>
>>       2       
>> /solr/collections/source_collection/leader_elect/shard2/election/942106
>>02200137736-core_node3-n_0000000000
>>   [junit4]   2>
>>       2       
>> /solr/overseer_elect/election/94210602200137895-127.0.0.1:58257_-n_0000
>>000013
>>   [junit4]   2>
>>   [junit4]   2> Maximum concurrent children watches above limit:
>>   [junit4]   2>
>>   [junit4]   2>       284     /solr/live_nodes
>>   [junit4]   2>       3       /solr/overseer/collection-queue-work
>>   [junit4]   2>       2       /solr/overseer/queue
>>   [junit4]   2>
>>   [junit4]   2> NOTE: reproduce with: ant test
>>-Dtestcase=CdcrReplicationDistributedZkTest -Dtests.method=doTests
>>-Dtests.seed=FA6717C1ABDFB670 -Dtests.slow=true -Dtests.locale=ar_SA
>>-Dtests.timezone=Australia/Sydney -Dtests.asserts=true
>>-Dtests.file.encoding=UTF-8
>>   [junit4] FAILURE  450s J1 | CdcrReplicationDistributedZkTest.doTests
>><<<
>>   [junit4]    > Throwable #1: java.lang.AssertionError: Timeout while
>>trying to assert update logs @ collection=source_collection
>>   [junit4]    >       at
>>__randomizedtesting.SeedInfo.seed([FA6717C1ABDFB670:F20762EDA4D19E7B]:0)
>>   [junit4]    >       at
>>org.apache.solr.cloud.CdcrReplicationDistributedZkTest.assertNumberOfTlogF
>>iles(CdcrReplicationDistributedZkTest.java:644)
>>   [junit4]    >       at
>>org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTestUpdateLogSync
>>hronisation(CdcrReplicationDistributedZkTest.java:384)
>>   [junit4]    >       at
>>org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTests(CdcrReplica
>>tionDistributedZkTest.java:50)
>>   [junit4]    >       at
>>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixed
>>Statement.callStatement(BaseDistributedSearchTestCase.java:963)
>>   [junit4]    >       at
>>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsState
>>ment.evaluate(BaseDistributedSearchTestCase.java:938)
>>   [junit4]    >       at java.lang.Thread.run(Thread.java:745)
>>   [junit4]    > Caused by: java.lang.AssertionError: Number of tlogs on
>>leader: 70 is superior to: 50.
>>   [junit4]    >       at
>>org.apache.solr.cloud.CdcrReplicationDistributedZkTest._assertNumberOfTlog
>>Files(CdcrReplicationDistributedZkTest.java:671)
>>   [junit4]    >       at
>>org.apache.solr.cloud.CdcrReplicationDistributedZkTest.assertNumberOfTlogF
>>iles(CdcrReplicationDistributedZkTest.java:634)
>>   [junit4]    >       ... 43 more
>>   [junit4]   2> 1925607 INFO
>>(SUITE-CdcrReplicationDistributedZkTest-seed#[FA6717C1ABDFB670]-worker)
>>[n:127.0.0.1:58315_ c:target_collection s:shard2 r:core_node4
>>x:target_collection_shard2_replica1] o.a.s.SolrTestCaseJ4 ###deleteCore
>>   [junit4]   2> NOTE: leaving temporary files on disk at:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/tes
>>t/J1/temp/solr.cloud.CdcrReplicationDistributedZkTest_FA6717C1ABDFB670-001
>>   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53):
>>{_version_=BlockTreeOrds(blocksize=128), id=Lucene50(blocksize=128)},
>>docValues:{}, sim=DefaultSimilarity, locale=ar_SA,
>>timezone=Australia/Sydney
>>   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.8.0_45
>>(64-bit)/cpus=3,threads=1,free=82748480,total=438304768
>>   [junit4]   2> NOTE: All tests run in this JVM:
>>[TestOverriddenPrefixQueryForCustomFieldType,
>>DistributedFacetPivotWhiteBoxTest, TestExceedMaxTermLength,
>>DirectUpdateHandlerOptimizeTest, UpdateParamsTest, TestSolrDynamicMBean,
>>SimpleMLTQParserTest, AutoCommitTest, TestReqParamsAPI,
>>CoreMergeIndexesAdminHandlerTest, CollectionsAPIDistributedZkTest,
>>BasicZkTest, TestJoin, TestManagedSchema, SolrIndexConfigTest,
>>TestCoreDiscovery, DistributedTermsComponentTest, TestMissingGroups,
>>TestInitParams, BigEndianAscendingWordSerializerTest,
>>CleanupOldIndexTest, TestSchemaManager, SmileWriterTest,
>>TestSolr4Spatial, SuggestComponentTest, SolrCmdDistributorTest,
>>CdcrRequestHandlerTest, TestCollapseQParserPlugin, AnalyticsQueryTest,
>>HighlighterConfigTest, ZkNodePropsTest, TestLRUStatsCache,
>>TestSimpleQParserPlugin, TestSchemaNameResource, HdfsLockFactoryTest,
>>ChaosMonkeySafeLeaderTest, ExitableDirectoryReaderTest,
>>TestSolrCoreProperties, DistribCursorPagingTest,
>>DistributedFacetPivotLongTailTest, CurrencyFieldOpenExchangeTest,
>>ParsingFieldUpdateProcessorsTest, HdfsBasicDistributedZk2Test,
>>SignatureUpdateProcessorFactoryTest, TestCopyFieldCollectionResource,
>>IndexSchemaTest, TestHighlightDedupGrouping, TestValueSourceCache,
>>BinaryUpdateRequestHandlerTest, SpellCheckComponentTest,
>>DebugComponentTest, HLLUtilTest, CdcrReplicationHandlerTest,
>>ResponseHeaderTest, TestDistributedStatsComponentCardinality,
>>TestConfigReload, PolyFieldTest, CollectionStateFormat2Test,
>>SimplePostToolTest, TestSmileRequest, WordBreakSolrSpellCheckerTest,
>>TermVectorComponentTest, LeaderElectionIntegrationTest,
>>StatelessScriptUpdateProcessorFactoryTest, HdfsUnloadDistributedZkTest,
>>TestBlendedInfixSuggestions, TestSolr4Spatial2, ReturnFieldsTest,
>>PrimitiveFieldTypeTest, DocValuesMissingTest, AddBlockUpdateTest,
>>CdcrVersionReplicationTest, TestDistributedGrouping,
>>TestClusterStateMutator, TestDefaultSimilarityFactory, FileUtilsTest,
>>TestSolrDeletionPolicy1, TestSearchPerf, RequestLoggingTest,
>>TestQuerySenderListener, RankQueryTest, ShardRoutingCustomTest,
>>MBeansHandlerTest, CSVRequestHandlerTest,
>>TestPHPSerializedResponseWriter, SpatialFilterTest,
>>DisMaxRequestHandlerTest, TestMiniSolrCloudCluster,
>>DistributedQueryComponentOptimizationTest, DirectSolrSpellCheckerTest,
>>TestFieldSortValues, DistributedMLTComponentTest,
>>SpellingQueryConverterTest, HdfsThreadLeakTest, TestInfoStreamLogging,
>>BadIndexSchemaTest, CollectionTooManyReplicasTest,
>>OverseerCollectionProcessorTest, QueryEqualityTest,
>>SpatialRPTFieldTypeTest, TestRTGBase, TestConfigSetImmutable,
>>AliasIntegrationTest, DateMathParserTest,
>>UpdateRequestProcessorFactoryTest, TestExactStatsCache,
>>AddSchemaFieldsUpdateProcessorFactoryTest, TestFoldingMultitermQuery,
>>SolrTestCaseJ4Test, DistributedFacetPivotSmallAdvancedTest,
>>TestFastWriter, SolrCloudExampleTest, TestQuerySenderNoQuery,
>>TestRangeQuery, TestNamedUpdateProcessors,
>>CdcrReplicationDistributedZkTest]
>>   [junit4] Completed [253/525] on J1 in 449.82s, 1 test, 1 failure <<<
>>FAILURES!
>>
>>[...truncated 831 lines...]
>>BUILD FAILED
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:526: The
>>following error occurred while executing this line:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:474: The
>>following error occurred while executing this line:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:61: The
>>following error occurred while executing this line:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:39:
>>The following error occurred while executing this line:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:230: The
>>following error occurred while executing this line:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:51
>>3: The following error occurred while executing this line:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:
>>1426: The following error occurred while executing this line:
>>/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:
>>981: There were test failures: 525 suites, 2071 tests, 1 failure, 65
>>ignored (25 assumptions)
>>
>>Total time: 77 minutes 47 seconds
>>Build step 'Invoke Ant' marked build as failure
>>Archiving artifacts
>>Recording test results
>>Email was triggered for: Failure - Any
>>Sending email for trigger: Failure - Any
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to