Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1448/
1 tests failed.
FAILED: org.apache.solr.cloud.RecoveryZkTest.test
Error Message:
Captured an uncaught exception in thread: Thread[id=7811,
name=updateExecutor-1220-thread-4, state=RUNNABLE, group=TGRP-RecoveryZkTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught
exception in thread: Thread[id=7811, name=updateExecutor-1220-thread-4,
state=RUNNABLE, group=TGRP-RecoveryZkTest]
at
__randomizedtesting.SeedInfo.seed([94BC77D554E0DAEA:1CE8480FFA1CB712]:0)
Caused by: org.apache.solr.common.SolrException: Replica:
https://127.0.0.1:38697/_pep/cj/collection1/ should have been marked under
leader initiated recovery in ZkController but wasn't.
at __randomizedtesting.SeedInfo.seed([94BC77D554E0DAEA]:0)
at
org.apache.solr.cloud.LeaderInitiatedRecoveryThread.run(LeaderInitiatedRecoveryThread.java:88)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11321 lines...]
[junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4] 2> Creating dataDir:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/init-core-data-001
[junit4] 2> 840136 INFO
(SUITE-RecoveryZkTest-seed#[94BC77D554E0DAEA]-worker) [ ]
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via:
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 840137 INFO
(SUITE-RecoveryZkTest-seed#[94BC77D554E0DAEA]-worker) [ ]
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property:
/_pep/cj
[junit4] 2> 840138 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.ZkTestServer
STARTING ZK TEST SERVER
[junit4] 2> 840145 INFO (Thread-2173) [ ] o.a.s.c.ZkTestServer client
port:0.0.0.0/0.0.0.0:0
[junit4] 2> 840145 INFO (Thread-2173) [ ] o.a.s.c.ZkTestServer
Starting server
[junit4] 2> 840239 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.ZkTestServer
start zk server on port:38997
[junit4] 2> 840249 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractZkTestCase put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
to /configs/conf1/solrconfig.xml
[junit4] 2> 840251 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractZkTestCase put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/schema15.xml
to /configs/conf1/schema.xml
[junit4] 2> 840253 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractZkTestCase put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 840254 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractZkTestCase put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
to /configs/conf1/stopwords.txt
[junit4] 2> 840255 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractZkTestCase put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt
to /configs/conf1/protwords.txt
[junit4] 2> 840256 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractZkTestCase put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml
to /configs/conf1/currency.xml
[junit4] 2> 840258 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractZkTestCase put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
to /configs/conf1/enumsConfig.xml
[junit4] 2> 840259 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractZkTestCase put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
to /configs/conf1/open-exchange-rates.json
[junit4] 2> 840260 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractZkTestCase put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 840262 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractZkTestCase put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
to /configs/conf1/old_synonyms.txt
[junit4] 2> 840263 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractZkTestCase put
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
to /configs/conf1/synonyms.txt
[junit4] 2> 840339 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.SolrTestCaseJ4
Writing core.properties file to
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/control-001/cores/collection1
[junit4] 2> 840342 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.e.j.s.Server
jetty-9.3.8.v20160314
[junit4] 2> 840343 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@d34aada{/_pep/cj,null,AVAILABLE}
[junit4] 2> 840347 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.e.j.s.ServerConnector Started ServerConnector@2d6a1ed{SSL,[ssl,
http/1.1]}{127.0.0.1:56086}
[junit4] 2> 840347 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.e.j.s.Server
Started @842841ms
[junit4] 2> 840347 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties:
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/tempDir-001/control/data,
hostContext=/_pep/cj, hostPort=56086,
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/control-001/cores}
[junit4] 2> 840347 ERROR
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 840347 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version
7.0.0
[junit4] 2> 840347 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 840347 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 840347 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2016-10-24T18:54:11.276Z
[junit4] 2> 840350 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 840350 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.SolrXmlConfig
Loading container configuration from
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/control-001/solr.xml
[junit4] 2> 840359 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.ZkContainer
Zookeeper client=127.0.0.1:38997/solr
[junit4] 2> 840380 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:56086__pep%2Fcj
] o.a.s.c.OverseerElectionContext I am going to be the leader
127.0.0.1:56086__pep%2Fcj
[junit4] 2> 840381 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:56086__pep%2Fcj
] o.a.s.c.Overseer Overseer
(id=96818643014451205-127.0.0.1:56086__pep%2Fcj-n_0000000000) starting
[junit4] 2> 840387 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:56086__pep%2Fcj
] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:56086__pep%2Fcj
[junit4] 2> 840388 INFO
(zkCallback-1214-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj ] o.a.s.c.c.ZkStateReader Updated live nodes
from ZooKeeper... (0) -> (1)
[junit4] 2> 840402 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:56086__pep%2Fcj
] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/control-001/cores
[junit4] 2> 840402 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:56086__pep%2Fcj
] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 840404 INFO
(OverseerStateUpdate-96818643014451205-127.0.0.1:56086__pep%2Fcj-n_0000000000)
[n:127.0.0.1:56086__pep%2Fcj ] o.a.s.c.o.ReplicaMutator Assigning new node
to shard shard=shard1
[junit4] 2> 841412 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection x:collection1]
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 841430 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection x:collection1]
o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 841538 WARN
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection x:collection1]
o.a.s.s.IndexSchema [collection1] default search field in schema is text.
WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 841539 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection x:collection1]
o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 841560 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection x:collection1]
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from
collection control_collection
[junit4] 2> 841560 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/control-001/cores/collection1],
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/control-001/cores/collection1/data/]
[junit4] 2> 841560 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr
mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@d87ba7c
[junit4] 2> 841561 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.5293731033140662]
[junit4] 2> 841567 WARN
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler
{type = requestHandler,name = /dump,class = DumpRequestHandler,args =
{defaults={a=A,b=B}}}
[junit4] 2> 841572 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation:
org.apache.solr.update.UpdateLog
[junit4] 2> 841572 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=
defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
numVersionBuckets=65536
[junit4] 2> 841573 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 841573 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 841573 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
maxMergeAtOnce=24, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=34.2314453125,
floorSegmentMB=1.052734375, forceMergeDeletesPctAllowed=24.029796342821786,
segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.27269275255163206
[junit4] 2> 841573 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.s.SolrIndexSearcher Opening
[Searcher@2420ec0f[collection1] main]
[junit4] 2> 841574 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO
with znodeBase: /configs/conf1
[junit4] 2> 841574 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 841574 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 841575 INFO
(coreLoadExecutor-3339-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent
updates, using new clock 1549098289729634304
[junit4] 2> 841577 INFO
(searcherExecutor-3340-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj
x:collection1 s:shard1 c:control_collection r:core_node1)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher
Searcher@2420ec0f[collection1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 841584 INFO
(coreZkRegister-3332-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj
x:collection1 s:shard1 c:control_collection r:core_node1)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to
continue.
[junit4] 2> 841584 INFO
(coreZkRegister-3332-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj
x:collection1 s:shard1 c:control_collection r:core_node1)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try
and sync
[junit4] 2> 841584 INFO
(coreZkRegister-3332-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj
x:collection1 s:shard1 c:control_collection r:core_node1)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.c.SyncStrategy Sync replicas to
https://127.0.0.1:56086/_pep/cj/collection1/
[junit4] 2> 841584 INFO
(coreZkRegister-3332-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj
x:collection1 s:shard1 c:control_collection r:core_node1)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 841584 INFO
(coreZkRegister-3332-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj
x:collection1 s:shard1 c:control_collection r:core_node1)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.c.SyncStrategy
https://127.0.0.1:56086/_pep/cj/collection1/ has no replicas
[junit4] 2> 841595 INFO
(coreZkRegister-3332-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj
x:collection1 s:shard1 c:control_collection r:core_node1)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader:
https://127.0.0.1:56086/_pep/cj/collection1/ shard1
[junit4] 2> 841745 INFO
(coreZkRegister-3332-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj
x:collection1 s:shard1 c:control_collection r:core_node1)
[n:127.0.0.1:56086__pep%2Fcj c:control_collection s:shard1 r:core_node1
x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 841911 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 841912 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.ChaosMonkey
monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 841988 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.SolrTestCaseJ4
Writing core.properties file to
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-1-001/cores/collection1
[junit4] 2> 841989 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-1-001
[junit4] 2> 841990 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.e.j.s.Server
jetty-9.3.8.v20160314
[junit4] 2> 841991 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@2e2cc9ca{/_pep/cj,null,AVAILABLE}
[junit4] 2> 841992 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.e.j.s.ServerConnector Started ServerConnector@d80c481{SSL,[ssl,
http/1.1]}{127.0.0.1:54776}
[junit4] 2> 841992 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.e.j.s.Server
Started @844487ms
[junit4] 2> 841992 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties:
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/tempDir-001/jetty1,
solrconfig=solrconfig.xml, hostContext=/_pep/cj, hostPort=54776,
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-1-001/cores}
[junit4] 2> 841993 ERROR
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 841993 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version
7.0.0
[junit4] 2> 841993 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 841993 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 841993 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2016-10-24T18:54:12.922Z
[junit4] 2> 841995 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 841995 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.SolrXmlConfig
Loading container configuration from
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-1-001/solr.xml
[junit4] 2> 842001 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.ZkContainer
Zookeeper client=127.0.0.1:38997/solr
[junit4] 2> 842019 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:54776__pep%2Fcj
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 842022 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:54776__pep%2Fcj
] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:54776__pep%2Fcj
[junit4] 2> 842022 INFO (zkCallback-1218-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 842023 INFO
(zkCallback-1214-thread-1-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj ] o.a.s.c.c.ZkStateReader Updated live nodes
from ZooKeeper... (1) -> (2)
[junit4] 2> 842026 INFO
(zkCallback-1223-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj ] o.a.s.c.c.ZkStateReader Updated live nodes
from ZooKeeper... (1) -> (2)
[junit4] 2> 842035 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:54776__pep%2Fcj
] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-1-001/cores
[junit4] 2> 842035 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:54776__pep%2Fcj
] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 842045 INFO
(OverseerStateUpdate-96818643014451205-127.0.0.1:56086__pep%2Fcj-n_0000000000)
[n:127.0.0.1:56086__pep%2Fcj ] o.a.s.c.o.ReplicaMutator Assigning new node
to shard shard=shard1
[junit4] 2> 843077 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 x:collection1] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 7.0.0
[junit4] 2> 843101 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 x:collection1] o.a.s.s.IndexSchema
[collection1] Schema name=test
[junit4] 2> 843157 WARN
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 x:collection1] o.a.s.s.IndexSchema
[collection1] default search field in schema is text. WARNING: Deprecated,
please use 'df' on request instead.
[junit4] 2> 843158 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 x:collection1] o.a.s.s.IndexSchema
Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 843168 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 x:collection1]
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from
collection collection1
[junit4] 2> 843168 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-1-001/cores/collection1],
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-1-001/cores/collection1/data/]
[junit4] 2> 843168 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@d87ba7c
[junit4] 2> 843169 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.5293731033140662]
[junit4] 2> 843172 WARN
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type =
requestHandler,name = /dump,class = DumpRequestHandler,args =
{defaults={a=A,b=B}}}
[junit4] 2> 843178 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.UpdateHandler Using UpdateLog implementation:
org.apache.solr.update.UpdateLog
[junit4] 2> 843178 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 843179 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 843179 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 843180 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
maxMergeAtOnce=24, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=34.2314453125,
floorSegmentMB=1.052734375, forceMergeDeletesPctAllowed=24.029796342821786,
segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.27269275255163206
[junit4] 2> 843180 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.s.SolrIndexSearcher Opening [Searcher@3186e55a[collection1] main]
[junit4] 2> 843180 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf1
[junit4] 2> 843181 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 843181 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 843181 INFO
(coreLoadExecutor-3350-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using
new clock 1549098291413647360
[junit4] 2> 843184 INFO
(searcherExecutor-3351-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore
[collection1] Registered new searcher Searcher@3186e55a[collection1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 843189 INFO
(coreZkRegister-3345-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 843190 INFO
(coreZkRegister-3345-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 843190 INFO
(coreZkRegister-3345-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync
replicas to https://127.0.0.1:54776/_pep/cj/collection1/
[junit4] 2> 843190 INFO
(coreZkRegister-3345-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync
Success - now sync replicas to me
[junit4] 2> 843190 INFO
(coreZkRegister-3345-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy
https://127.0.0.1:54776/_pep/cj/collection1/ has no replicas
[junit4] 2> 843194 INFO
(coreZkRegister-3345-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.c.ShardLeaderElectionContext I am the new leader:
https://127.0.0.1:54776/_pep/cj/collection1/ shard1
[junit4] 2> 843345 INFO
(coreZkRegister-3345-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am
the leader, no recovery necessary
[junit4] 2> 843621 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.SolrTestCaseJ4
Writing core.properties file to
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-2-001/cores/collection1
[junit4] 2> 843622 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-2-001
[junit4] 2> 843623 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.e.j.s.Server
jetty-9.3.8.v20160314
[junit4] 2> 843625 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@16a0332a{/_pep/cj,null,AVAILABLE}
[junit4] 2> 843626 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.e.j.s.ServerConnector Started ServerConnector@16b06341{SSL,[ssl,
http/1.1]}{127.0.0.1:38697}
[junit4] 2> 843626 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.e.j.s.Server
Started @846120ms
[junit4] 2> 843626 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties:
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/tempDir-001/jetty2,
solrconfig=solrconfig.xml, hostContext=/_pep/cj, hostPort=38697,
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-2-001/cores}
[junit4] 2> 843626 ERROR
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 843626 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version
7.0.0
[junit4] 2> 843627 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 843627 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 843627 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2016-10-24T18:54:14.556Z
[junit4] 2> 843629 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 843629 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.SolrXmlConfig
Loading container configuration from
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-2-001/solr.xml
[junit4] 2> 843638 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.ZkContainer
Zookeeper client=127.0.0.1:38997/solr
[junit4] 2> 843647 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:38697__pep%2Fcj
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 843659 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:38697__pep%2Fcj
] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:38697__pep%2Fcj
[junit4] 2> 843661 INFO
(zkCallback-1223-thread-1-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj ] o.a.s.c.c.ZkStateReader Updated live nodes
from ZooKeeper... (2) -> (3)
[junit4] 2> 843661 INFO (zkCallback-1218-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 843661 INFO
(zkCallback-1214-thread-4-processing-n:127.0.0.1:56086__pep%2Fcj)
[n:127.0.0.1:56086__pep%2Fcj ] o.a.s.c.c.ZkStateReader Updated live nodes
from ZooKeeper... (2) -> (3)
[junit4] 2> 843662 INFO
(zkCallback-1229-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj ] o.a.s.c.c.ZkStateReader Updated live nodes
from ZooKeeper... (2) -> (3)
[junit4] 2> 843672 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:38697__pep%2Fcj
] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-2-001/cores
[junit4] 2> 843673 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [n:127.0.0.1:38697__pep%2Fcj
] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 843674 INFO
(OverseerStateUpdate-96818643014451205-127.0.0.1:56086__pep%2Fcj-n_0000000000)
[n:127.0.0.1:56086__pep%2Fcj ] o.a.s.c.o.ReplicaMutator Assigning new node
to shard shard=shard1
[junit4] 2> 844685 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 x:collection1] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 7.0.0
[junit4] 2> 844708 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 x:collection1] o.a.s.s.IndexSchema
[collection1] Schema name=test
[junit4] 2> 844755 WARN
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 x:collection1] o.a.s.s.IndexSchema
[collection1] default search field in schema is text. WARNING: Deprecated,
please use 'df' on request instead.
[junit4] 2> 844756 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 x:collection1] o.a.s.s.IndexSchema
Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 844773 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 x:collection1]
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from
collection collection1
[junit4] 2> 844774 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-2-001/cores/collection1],
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J2/temp/solr.cloud.RecoveryZkTest_94BC77D554E0DAEA-001/shard-2-001/cores/collection1/data/]
[junit4] 2> 844774 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@d87ba7c
[junit4] 2> 844780 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.5293731033140662]
[junit4] 2> 844791 WARN
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type =
requestHandler,name = /dump,class = DumpRequestHandler,args =
{defaults={a=A,b=B}}}
[junit4] 2> 844808 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.UpdateHandler Using UpdateLog implementation:
org.apache.solr.update.UpdateLog
[junit4] 2> 844808 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 844809 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 844809 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 844817 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
maxMergeAtOnce=24, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=34.2314453125,
floorSegmentMB=1.052734375, forceMergeDeletesPctAllowed=24.029796342821786,
segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.27269275255163206
[junit4] 2> 844818 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.s.SolrIndexSearcher Opening [Searcher@45f24ca2[collection1] main]
[junit4] 2> 844818 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf1
[junit4] 2> 844818 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 844819 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 844819 INFO
(coreLoadExecutor-3361-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using
new clock 1549098293131214848
[junit4] 2> 844821 INFO
(searcherExecutor-3362-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore
[collection1] Registered new searcher Searcher@45f24ca2[collection1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 844834 INFO
(coreZkRegister-3356-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core
needs to recover:collection1
[junit4] 2> 844849 INFO
(updateExecutor-1226-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState
Running recovery
[junit4] 2> 844861 INFO
(recoveryExecutor-1227-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy
Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 844873 INFO
(recoveryExecutor-1227-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy
###### startupVersions=[[]]
[junit4] 2> 844873 INFO
(recoveryExecutor-1227-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy
Begin buffering updates. core=[collection1]
[junit4] 2> 844873 INFO
(recoveryExecutor-1227-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting
to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 844873 INFO
(recoveryExecutor-1227-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy
Publishing state of core [collection1] as recovering, leader is
[https://127.0.0.1:54776/_pep/cj/collection1/] and I am
[https://127.0.0.1:38697/_pep/cj/collection1/]
[junit4] 2> 844899 INFO
(recoveryExecutor-1227-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj
x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy
Sending prep recovery command to [https://127.0.0.1:54776/_pep/cj];
[WaitForState:
action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:38697__pep%252Fcj&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 844941 INFO (qtp2027390803-7746) [n:127.0.0.1:54776__pep%2Fcj
] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2,
state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 844941 INFO (qtp2027390803-7746) [n:127.0.0.1:54776__pep%2Fcj
] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1
(shard1 of collection1) have state: recovering
[junit4] 2> 844941 INFO (qtp2027390803-7746) [n:127.0.0.1:54776__pep%2Fcj
] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering):
collection=collection1, shard=shard1, thisCore=collection1,
leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true,
currentState=down, localState=active, nodeName=127.0.0.1:38697__pep%2Fcj,
coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps:
core_node2:{"core":"collection1","base_url":"https://127.0.0.1:38697/_pep/cj","node_name":"127.0.0.1:38697__pep%2Fcj","state":"down"}
[junit4] 2> 845175 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.SolrTestCaseJ4
###Starting test
[junit4] 2> 845257 INFO (qtp192340450-7702) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-0 (1549098293588393984)]} 0 1
[junit4] 2> 845276 INFO (qtp192340450-7706) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-0 (1549098293609365504)]} 0 2
[junit4] 2> 845396 INFO (qtp860070878-7769) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-0
(1549098293629288448)]} 0 0
[junit4] 2> 845397 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-0 (1549098293629288448)]} 0 103
[junit4] 2> 845400 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-1 (1549098293739388928)]} 0 0
[junit4] 2> 845404 INFO (qtp860070878-7768) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-1
(1549098293741486080)]} 0 2
[junit4] 2> 845404 INFO (qtp2027390803-7743) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-1 (1549098293741486080)]} 0 3
[junit4] 2> 845406 INFO (qtp192340450-7701) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-2 (1549098293745680384)]} 0 0
[junit4] 2> 845407 INFO (qtp860070878-7774) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-2
(1549098293746728960)]} 0 0
[junit4] 2> 845407 INFO (qtp2027390803-7740) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-2 (1549098293746728960)]} 0 1
[junit4] 2> 845408 INFO (qtp192340450-7705) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-3 (1549098293748826112)]} 0 0
[junit4] 2> 845410 INFO (qtp860070878-7773) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-0
(1549098293651308544)]} 0 0
[junit4] 2> 845411 INFO (qtp2027390803-7745) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-0 (1549098293651308544)]} 0 95
[junit4] 2> 845411 INFO (qtp192340450-7705) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-1 (1549098293751971840)]} 0 0
[junit4] 2> 845422 INFO (qtp860070878-7771) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-3
(1549098293749874688)]} 0 2
[junit4] 2> 845423 INFO (qtp2027390803-7744) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-3 (1549098293749874688)]} 0 14
[junit4] 2> 845424 INFO (qtp192340450-7707) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-1 (-1549098293764554752)]}
0 0
[junit4] 2> 845424 INFO (qtp860070878-7769) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-1
(1549098293758263296)]} 0 0
[junit4] 2> 845425 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-1 (1549098293758263296)]} 0 12
[junit4] 2> 845425 INFO (qtp192340450-7706) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-2 (1549098293766651904)]} 0 0
[junit4] 2> 845426 INFO (qtp860070878-7771) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[1-1
(-1549098293765603328)]} 0 0
[junit4] 2> 845426 INFO (qtp2027390803-7743) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[1-1 (-1549098293765603328)]} 0 2
[junit4] 2> 845427 INFO (qtp192340450-7706) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-4 (1549098293768749056)]} 0 0
[junit4] 2> 845427 INFO (qtp860070878-7771) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-2
(1549098293767700480)]} 0 0
[junit4] 2> 845428 INFO (qtp2027390803-7740) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-2 (1549098293767700480)]} 0 1
[junit4] 2> 845428 INFO (qtp860070878-7774) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-4
(1549098293768749056)]} 0 0
[junit4] 2> 845428 INFO (qtp192340450-7706) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-3 (1549098293769797632)]} 0 0
[junit4] 2> 845428 INFO (qtp2027390803-7745) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-4 (1549098293768749056)]} 0 1
[junit4] 2> 845429 INFO (qtp860070878-7773) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-3
(1549098293770846208)]} 0 0
[junit4] 2> 845429 INFO (qtp192340450-7706) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-5 (1549098293770846208)]} 0 0
[junit4] 2> 845429 INFO (qtp2027390803-7744) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-3 (1549098293770846208)]} 0 0
[junit4] 2> 845433 INFO (qtp192340450-7704) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-1 (-1549098293775040512)]}
0 0
[junit4] 2> 845434 INFO (qtp860070878-7767) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[2-1
(-1549098293775040512)]} 0 0
[junit4] 2> 845434 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[2-1 (-1549098293775040512)]} 0 0
[junit4] 2> 845435 INFO (qtp860070878-7769) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-5
(1549098293777137664)]} 0 0
[junit4] 2> 845436 INFO (qtp192340450-7707) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-4 (1549098293777137664)]} 0 0
[junit4] 2> 845436 INFO (qtp2027390803-7743) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-5 (1549098293777137664)]} 0 1
[junit4] 2> 845436 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-5 (-1549098293778186240)]}
0 0
[junit4] 2> 845437 INFO (qtp860070878-7769) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-4
(1549098293778186240)]} 0 0
[junit4] 2> 845437 INFO (qtp2027390803-7740) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-4 (1549098293778186240)]} 0 1
[junit4] 2> 845438 INFO (qtp860070878-7769) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[1-5
(-1549098293779234816)]} 0 0
[junit4] 2> 845438 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-5 (1549098293780283392)]} 0 0
[junit4] 2> 845438 INFO (qtp2027390803-7745) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[1-5 (-1549098293779234816)]} 0 0
[junit4] 2> 845439 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-6 (1549098293781331968)]} 0 0
[junit4] 2> 845439 INFO (qtp860070878-7774) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-5
(1549098293781331968)]} 0 0
[junit4] 2> 845440 INFO (qtp2027390803-7740) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-5 (1549098293781331968)]} 0 1
[junit4] 2> 845440 INFO (qtp860070878-7773) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-6
(1549098293782380544)]} 0 0
[junit4] 2> 845440 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-5 (-1549098293782380544)]}
0 0
[junit4] 2> 845441 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-6 (1549098293782380544)]} 0 0
[junit4] 2> 845445 INFO (qtp860070878-7773) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[2-5
(-1549098293786574848)]} 0 0
[junit4] 2> 845445 INFO (qtp2027390803-7743) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[2-5 (-1549098293786574848)]} 0 1
[junit4] 2> 845446 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-7 (1549098293784477696)]} 0 3
[junit4] 2> 845446 INFO (qtp192340450-7706) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-6 (1549098293788672000)]} 0 0
[junit4] 2> 845448 INFO (qtp860070878-7772) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-6
(1549098293789720576)]} 0 0
[junit4] 2> 845448 INFO (qtp2027390803-7744) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-6 (1549098293789720576)]} 0 1
[junit4] 2> 845448 INFO (qtp192340450-7701) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-7 (1549098293790769152)]} 0 0
[junit4] 2> 845464 INFO (qtp860070878-7768) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-7
(1549098293792866304)]} 0 10
[junit4] 2> 845469 INFO (qtp860070878-7772) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-7
(1549098293791817728)]} 0 19
[junit4] 2> 845469 INFO (qtp2027390803-7745) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-7 (1549098293791817728)]} 0 20
[junit4] 2> 845470 INFO (qtp192340450-7707) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-7 (-1549098293813837824)]}
0 0
[junit4] 2> 845471 INFO (qtp860070878-7769) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[2-7
(-1549098293814886400)]} 0 0
[junit4] 2> 845471 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[2-7 (-1549098293814886400)]} 0 0
[junit4] 2> 845472 INFO (qtp192340450-7705) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-8 (1549098293815934976)]} 0 0
[junit4] 2> 845474 INFO (qtp860070878-7769) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-8
(1549098293816983552)]} 0 0
[junit4] 2> 845474 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-8 (1549098293816983552)]} 0 0
[junit4] 2> 845474 INFO (qtp192340450-7705) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-9 (1549098293818032128)]} 0 0
[junit4] 2> 845475 INFO (qtp860070878-7769) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-9
(1549098293819080704)]} 0 0
[junit4] 2> 845475 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-9 (1549098293819080704)]} 0 0
[junit4] 2> 845476 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-10 (1549098293820129280)]} 0 0
[junit4] 2> 845477 INFO (qtp860070878-7767) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-10
(1549098293820129280)]} 0 0
[junit4] 2> 845478 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-10 (1549098293820129280)]} 0 1
[junit4] 2> 845478 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-8 (-1549098293822226432)]}
0 0
[junit4] 2> 845479 INFO (qtp860070878-7767) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[2-8
(-1549098293823275008)]} 0 0
[junit4] 2> 845479 INFO (qtp2027390803-7744) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[2-8 (-1549098293823275008)]} 0 0
[junit4] 2> 845480 INFO (qtp192340450-7702) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-11 (1549098293824323584)]} 0 0
[junit4] 2> 845481 INFO (qtp860070878-7767) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-11
(1549098293825372160)]} 0 0
[junit4] 2> 845481 INFO (qtp2027390803-7745) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-11 (1549098293825372160)]} 0 0
[junit4] 2> 845482 INFO (qtp192340450-7707) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-9 (-1549098293826420736)]}
0 0
[junit4] 2> 845482 INFO (qtp2027390803-7740) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-7 (1549098293792866304)]} 0 32
[junit4] 2> 845483 INFO (qtp192340450-7707) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-7 (-1549098293827469312)]}
0 0
[junit4] 2> 845484 INFO (qtp860070878-7774) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[2-9
(-1549098293826420736)]} 0 0
[junit4] 2> 845484 INFO (qtp2027390803-7743) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[2-9 (-1549098293826420736)]} 0 1
[junit4] 2> 845484 INFO (qtp860070878-7774) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[1-7
(-1549098293827469312)]} 0 0
[junit4] 2> 845485 INFO (qtp192340450-7707) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-12 (1549098293828517888)]} 0 0
[junit4] 2> 845485 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[1-7 (-1549098293827469312)]} 0 1
[junit4] 2> 845485 INFO (qtp192340450-7707) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-8 (1549098293829566464)]} 0 0
[junit4] 2> 845502 INFO (qtp860070878-7769) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-12
(1549098293829566464)]} 0 0
[junit4] 2> 845502 INFO (qtp2027390803-7744) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-12 (1549098293829566464)]} 0 17
[junit4] 2> 845503 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-10
(-1549098293848440832)]} 0 0
[junit4] 2> 845503 INFO (qtp860070878-7772) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-8
(1549098293848440832)]} 0 0
[junit4] 2> 845504 INFO (qtp2027390803-7745) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-8 (1549098293848440832)]} 0 1
[junit4] 2> 845504 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-9 (1549098293849489408)]} 0 0
[junit4] 2> 845505 INFO (qtp860070878-7772) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-9
(1549098293850537984)]} 0 0
[junit4] 2> 845505 INFO (qtp2027390803-7743) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-9 (1549098293850537984)]} 0 0
[junit4] 2> 845505 INFO (qtp860070878-7767) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[2-10
(-1549098293848440833)]} 0 0
[junit4] 2> 845506 INFO (qtp2027390803-7740) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[2-10 (-1549098293848440833)]} 0 2
[junit4] 2> 845506 INFO (qtp192340450-7704) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-10 (1549098293851586560)]} 0 0
[junit4] 2> 845507 INFO (qtp192340450-7704) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-13 (1549098293852635136)]} 0 0
[junit4] 2> 845507 INFO (qtp860070878-7772) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-10
(1549098293852635136)]} 0 0
[junit4] 2> 845507 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-10 (1549098293852635136)]} 0 0
[junit4] 2> 845508 INFO (qtp860070878-7774) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-13
(1549098293852635137)]} 0 0
[junit4] 2> 845508 INFO (qtp192340450-7704) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-8 (-1549098293853683712)]}
0 0
[junit4] 2> 845508 INFO (qtp2027390803-7744) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-13 (1549098293852635137)]} 0 0
[junit4] 2> 845508 INFO (qtp192340450-7704) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-11
(-1549098293853683713)]} 0 0
[junit4] 2> 845509 INFO (qtp860070878-7771) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[2-11
(-1549098293854732288)]} 0 0
[junit4] 2> 845510 INFO (qtp2027390803-7743) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[2-11 (-1549098293854732288)]} 0 0
[junit4] 2> 845510 INFO (qtp860070878-7774) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[1-8
(-1549098293853683712)]} 0 0
[junit4] 2> 845510 INFO (qtp2027390803-7745) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[1-8 (-1549098293853683712)]} 0 1
[junit4] 2> 845510 INFO (qtp192340450-7702) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-14 (1549098293855780864)]} 0 0
[junit4] 2> 845511 INFO (qtp192340450-7701) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-11 (1549098293856829440)]} 0 0
[junit4] 2> 845511 INFO (qtp860070878-7773) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-14
(1549098293856829440)]} 0 0
[junit4] 2> 845511 INFO (qtp2027390803-7740) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-14 (1549098293856829440)]} 0 0
[junit4] 2> 845512 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-15 (1549098293857878016)]} 0 0
[junit4] 2> 845514 INFO (qtp860070878-7773) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-15
(1549098293857878016)]} 0 0
[junit4] 2> 845514 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-15 (1549098293857878016)]} 0 2
[junit4] 2> 845515 INFO (qtp192340450-7708) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-13
(-1549098293861023744)]} 0 0
[junit4] 2> 845516 INFO (qtp860070878-7772) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[2-13
(-1549098293862072320)]} 0 0
[junit4] 2> 845517 INFO (qtp2027390803-7743) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[2-13 (-1549098293862072320)]} 0 0
[junit4] 2> 845517 INFO (qtp192340450-7706) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-16 (1549098293863120896)]} 0 0
[junit4] 2> 845519 INFO (qtp860070878-7769) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-16
(1549098293864169472)]} 0 0
[junit4] 2> 845519 INFO (qtp2027390803-7745) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-16 (1549098293864169472)]} 0 0
[junit4] 2> 845519 INFO (qtp192340450-7707) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-14
(-1549098293865218048)]} 0 0
[junit4] 2> 845533 INFO (qtp860070878-7769) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[2-14
(-1549098293866266624)]} 0 0
[junit4] 2> 845533 INFO (qtp2027390803-7740) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[2-14 (-1549098293866266624)]} 0 12
[junit4] 2> 845533 INFO (qtp860070878-7767) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[1-11
(1549098293861023744)]} 0 0
[junit4] 2> 845534 INFO (qtp2027390803-7744) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[1-11 (1549098293861023744)]} 0 18
[junit4] 2> 845534 INFO (qtp192340450-7707) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-17 (1549098293880946688)]} 0 0
[junit4] 2> 845534 INFO (qtp192340450-7705) [n:127.0.0.1:56086__pep%2Fcj
c:control_collection s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-9 (-1549098293880946689)]}
0 0
[junit4] 2> 845536 INFO (qtp860070878-7768) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{delete=[1-9
(-1549098293881995264)]} 0 0
[junit4] 2> 845536 INFO (qtp860070878-7774) [n:127.0.0.1:38697__pep%2Fcj
c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:54776/_pep/cj/collection1/&wt=javabin&version=2}{add=[2-17
(1549098293880946688)]} 0 0
[junit4] 2> 845536 INFO (qtp2027390803-7743) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{delete=[1-9 (-1549098293881995264)]} 0 1
[junit4] 2> 845536 INFO (qtp2027390803-7742) [n:127.0.0.1:54776__pep%2Fcj
c:collection1 s:shard1 r:core_node1 x:collection1]
o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/_pep/cj path=/update
params={wt=javabin&version=2}{add=[2-17 (1549098293880946688)]} 0 2
[junit4] 2> 845537 INFO (qtp192340450-7701)
[...truncated too long message...]
(2) -> (1)
[junit4] 2> 876158 INFO
(zkCallback-1236-thread-2-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj ] o.a.s.c.c.ZkStateReader Updated live nodes
from ZooKeeper... (2) -> (1)
[junit4] 2> 876160 INFO
(zkCallback-1236-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj ] o.a.s.c.OverseerElectionContext I am going to
be the leader 127.0.0.1:38697__pep%2Fcj
[junit4] 2> 876161 INFO
(zkCallback-1236-thread-3-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 876161 INFO
(zkCallback-1236-thread-1-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj ] o.a.s.c.Overseer Overseer
(id=96818643014451215-127.0.0.1:38697__pep%2Fcj-n_0000000003) starting
[junit4] 2> 877658 WARN
(zkCallback-1223-thread-2-processing-n:127.0.0.1:54776__pep%2Fcj)
[n:127.0.0.1:54776__pep%2Fcj ] o.a.s.c.c.ZkStateReader ZooKeeper watch
triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for
/live_nodes]
[junit4] 2> 877660 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.e.j.s.ServerConnector Stopped ServerConnector@d80c481{SSL,[ssl,
http/1.1]}{127.0.0.1:0}
[junit4] 2> 877660 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.e.j.s.h.ContextHandler Stopped
o.e.j.s.ServletContextHandler@2e2cc9ca{/_pep/cj,null,UNAVAILABLE}
[junit4] 2> 877661 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.ChaosMonkey
monkey: stop shard! 38697
[junit4] 2> 877661 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.CoreContainer
Shutting down CoreContainer instance=1397170922
[junit4] 2> 877664 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.Overseer
Overseer (id=96818643014451215-127.0.0.1:38697__pep%2Fcj-n_0000000003) closing
[junit4] 2> 877664 INFO
(OverseerStateUpdate-96818643014451215-127.0.0.1:38697__pep%2Fcj-n_0000000003)
[n:127.0.0.1:38697__pep%2Fcj ] o.a.s.c.Overseer Overseer Loop exiting :
127.0.0.1:38697__pep%2Fcj
[junit4] 2> 877665 INFO (zkCallback-1218-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 877666 WARN
(zkCallback-1236-thread-4-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj ] o.a.s.c.c.ZkStateReader ZooKeeper watch
triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for
/live_nodes]
[junit4] 2> 878661 WARN
(zkCallback-1236-thread-3-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.c.SyncStrategy Closed, skipping sync up.
[junit4] 2> 878661 INFO
(zkCallback-1236-thread-3-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we
can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 878661 INFO
(zkCallback-1236-thread-3-processing-n:127.0.0.1:38697__pep%2Fcj)
[n:127.0.0.1:38697__pep%2Fcj c:collection1 s:shard1 r:core_node2 x:collection1]
o.a.s.c.SolrCore [collection1] CLOSING SolrCore
org.apache.solr.core.SolrCore@269655bd
[junit4] 2> 878670 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.e.j.s.ServerConnector Stopped ServerConnector@2d8b42b{SSL,[ssl,
http/1.1]}{127.0.0.1:38697}
[junit4] 2> 878670 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ]
o.e.j.s.h.ContextHandler Stopped
o.e.j.s.ServletContextHandler@3c64d20b{/_pep/cj,null,UNAVAILABLE}
[junit4] 2> 878672 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.ChaosMonkey
monkey: stop shard! 56086
[junit4] 2> 878673 INFO
(TEST-RecoveryZkTest.test-seed#[94BC77D554E0DAEA]) [ ] o.a.s.c.ZkTestServer
connecting to 127.0.0.1:38997 38997
[junit4] 2> 878776 INFO (Thread-2173) [ ] o.a.s.c.ZkTestServer
connecting to 127.0.0.1:38997 38997
[junit4] 2> 878777 WARN (Thread-2173) [ ] o.a.s.c.ZkTestServer Watch
limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 4 /solr/security.json
[junit4] 2> 4 /solr/configs/conf1
[junit4] 2> 3 /solr/collections/collection1/state.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 5 /solr/clusterprops.json
[junit4] 2> 2
/solr/overseer_elect/election/96818643014451209-127.0.0.1:54776__pep%2Fcj-n_0000000001
[junit4] 2> 2
/solr/collections/collection1/leader_elect/shard1/election/96818643014451209-core_node1-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 60 /solr/overseer/queue
[junit4] 2> 37 /solr/overseer/collection-queue-work
[junit4] 2> 5 /solr/collections
[junit4] 2> 4 /solr/live_nodes
[junit4] 2> 4 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest
-Dtests.method=test -Dtests.seed=94BC77D554E0DAEA -Dtests.multiplier=2
-Dtests.slow=true -Dtests.locale=mt-MT -Dtests.timezone=America/Port-au-Prince
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] ERROR 38.6s J2 | RecoveryZkTest.test <<<
[junit4] > Throwable #1:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught
exception in thread: Thread[id=7811, name=updateExecutor-1220-thread-4,
state=RUNNABLE, group=TGRP-RecoveryZkTest]
[junit4] > at
__randomizedtesting.SeedInfo.seed([94BC77D554E0DAEA:1CE8480FFA1CB712]:0)
[junit4] > Caused by: org.apache.solr.common.SolrException: Replica:
https://127.0.0.1:38697/_pep/cj/collection1/ should have been marked under
leader initiated recovery in ZkController but wasn't.
[junit4] > at
__randomizedtesting.SeedInfo.seed([94BC77D554E0DAEA]:0)
[junit4] > at
org.apache.solr.cloud.LeaderInitiatedRecoveryThread.run(LeaderInitiatedRecoveryThread.java:88)
[junit4] > at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] > at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 878780 INFO
(SUITE-RecoveryZkTest-seed#[94BC77D554E0DAEA]-worker) [ ]
o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> Oct 24, 2016 6:54:49 PM
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked
thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70):
{rnd_b=Lucene50(blocksize=128), _version_=FST50,
a_t=PostingsFormat(name=LuceneVarGapFixedInterval),
a_i=Lucene50(blocksize=128), id=Lucene50(blocksize=128)}, docValues:{},
maxPointsInLeafNode=1131, maxMBSortInHeap=5.929308723359279,
sim=RandomSimilarity(queryNorm=true): {}, locale=mt-MT,
timezone=America/Port-au-Prince
[junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation
1.8.0_102 (64-bit)/cpus=4,threads=1,free=231094064,total=528482304
[junit4] 2> NOTE: All tests run in this JVM: [TestCloudPivotFacet,
PKIAuthenticationIntegrationTest, HdfsChaosMonkeyNothingIsSafeTest,
TestStressReorder, StatelessScriptUpdateProcessorFactoryTest, TestDocSet,
ReplicationFactorTest, CoreAdminRequestStatusTest, LeaderElectionTest,
CoreAdminHandlerTest, TestFastOutputStream, TestShardHandlerFactory,
TestFilteredDocIdSet, TestSolrQueryParser, TestExactStatsCache,
TestInitQParser, HdfsNNFailoverTest, PreAnalyzedFieldTest, UpdateParamsTest,
HttpPartitionTest, SpatialRPTFieldTypeTest, SolrTestCaseJ4Test,
CollectionStateFormat2Test, TestCSVResponseWriter,
PeerSyncWithIndexFingerprintCachingTest, RequestLoggingTest,
DebugComponentTest, RecoveryAfterSoftCommitTest, TestCollationField,
AutoCommitTest, SuggesterTSTTest, ExitableDirectoryReaderTest,
DistributedQueueTest, TestSchemaManager, AssignTest, AnalysisErrorHandlingTest,
TestTolerantSearch, ConnectionReuseTest, TestMinMaxOnMultiValuedField,
FieldMutatingUpdateProcessorTest, DistributedDebugComponentTest,
DistributedQueryComponentOptimizationTest, BinaryUpdateRequestHandlerTest,
DeleteInactiveReplicaTest, TestComponentsName, BlockCacheTest, TestRestManager,
DateRangeFieldTest, TestZkChroot, TestDistributedStatsComponentCardinality,
TestQueryWrapperFilter, TestMissingGroups, TestUtils, SpatialHeatmapFacetsTest,
SolrIndexConfigTest, UnloadDistributedZkTest, TestConfig, FullHLLTest,
OverseerCollectionConfigSetProcessorTest, TestNonDefinedSimilarityFactory,
DistributedFacetPivotLargeTest, TestCryptoKeys,
FieldAnalysisRequestHandlerTest, TestLeaderElectionZkExpiry, DeleteReplicaTest,
HdfsRestartWhileUpdatingTest, TestSlowCompositeReaderWrapper,
TestConfigSetProperties, RequestHandlersTest, SolrPluginUtilsTest,
HighlighterConfigTest, TestMacroExpander, TestRestoreCore, DocValuesMultiTest,
HdfsTlogReplayBufferedWhileIndexingTest, TestSearcherReuse, ZkStateWriterTest,
PreAnalyzedUpdateProcessorTest, TestExclusionRuleCollectionAccess,
DistributedFacetPivotSmallTest, TestRawTransformer, TestStressUserVersions,
DistribCursorPagingTest, CircularListTest, TestInitParams, TestCoreDiscovery,
StressHdfsTest, HdfsDirectoryTest, TestFieldCache, TestHdfsBackupRestoreCore,
OverseerTest, LukeRequestHandlerTest, DirectUpdateHandlerTest,
OverseerStatusTest, RecoveryZkTest]
[junit4] Completed [235/642 (1!)] on J2 in 38.98s, 1 test, 1 error <<<
FAILURES!
[...truncated 61637 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
