[
https://issues.apache.org/jira/browse/SOLR-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13809799#comment-13809799
]
Nathan Neulinger commented on SOLR-5407:
----------------------------------------
Digging further, it looks like it all keys around some sort of communications
problem with zookeeper - looks like it all started at the end of this log
snippet below (reverse time order) when it's reporting that 'Our previous
ZooKeeper session was expired. Attempting to reconnect to recover relationship
with ZooKeeper'.
2013-10-29T16:25:50.344Z Going to wait for coreNodeName: core_node2,
state: down, checkLive: null, onlyIfLeader: null
2013-10-29T16:25:50.329Z publishing
core=myappqa-master_v8_shard1_replica1 state=down
2013-10-29T16:25:50.329Z Creating new http client,
config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
2013-10-29T16:25:50.328Z Waited coreNodeName: core_node1, state: down,
checkLive: null, onlyIfLeader: null for: 1 seconds.
2013-10-29T16:25:49.884Z A cluster state change: WatchedEvent
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred
- updating... (live nodes size: 1)
2013-10-29T16:25:49.825Z Updating cloud state from ZooKeeper...
2013-10-29T16:25:49.825Z Update state numShards=1 message={
"operation":"state",
"state":"down",
"base_url":"http://10.170.2.54:8983/solr";,
"core":"hiv...
2013-10-29T16:25:49.324Z Going to wait for coreNodeName: core_node1,
state: down, checkLive: null, onlyIfLeader: null
2013-10-29T16:25:49.309Z Creating new http client,
config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
2013-10-29T16:25:49.308Z publishing
core=myappqa-master_v6_shard1_replica2 state=down
2013-10-29T16:25:49.308Z Waited coreNodeName: core_node1, state: down,
checkLive: null, onlyIfLeader: null for: 2 seconds.
2013-10-29T16:25:48.302Z A cluster state change: WatchedEvent
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred
- updating... (live nodes size: 1)
2013-10-29T16:25:48.239Z Updating cloud state from ZooKeeper...
2013-10-29T16:25:48.239Z Update state numShards=1 message={
"operation":"state",
"state":"down",
"base_url":"http://10.170.2.54:8983/solr";,
"core":"hiv...
2013-10-29T16:25:47.304Z Going to wait for coreNodeName: core_node1,
state: down, checkLive: null, onlyIfLeader: null
2013-10-29T16:25:47.289Z Creating new http client,
config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
2013-10-29T16:25:47.289Z publishing
core=myappstaging-profile_v7_shard1_replica1 state=down
2013-10-29T16:25:47.287Z Waited coreNodeName: core_node2, state: down,
checkLive: null, onlyIfLeader: null for: 2 seconds.
2013-10-29T16:25:46.469Z A cluster state change: WatchedEvent
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred
- updating... (live nodes size: 1)
2013-10-29T16:25:46.406Z Update state numShards=1 message={
"operation":"state",
"state":"down",
"base_url":"http://10.170.2.54:8983/solr";,
"core":"hiv...
2013-10-29T16:25:45.925Z Updating cloud state from ZooKeeper...
2013-10-29T16:25:45.286Z Going to wait for coreNodeName: core_node2,
state: down, checkLive: null, onlyIfLeader: null
2013-10-29T16:25:45.270Z publishing
core=myappstaging-profile_v8_shard1_replica1 state=down
2013-10-29T16:25:45.270Z Creating new http client,
config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
2013-10-29T16:25:45.269Z Waited coreNodeName: core_node2, state: down,
checkLive: null, onlyIfLeader: null for: 2 seconds.
2013-10-29T16:25:45.039Z A cluster state change: WatchedEvent
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred
- updating... (live nodes size: 1)
2013-10-29T16:25:44.994Z makePath:
/collections/myappproduction-production_v8/leaders/shard1
2013-10-29T16:25:44.994Z I am the new leader:
http://10.136.6.24:8983/solr/myappproduction-production_v8_shard1_replica1/
shard1
2013-10-29T16:25:44.994Z
http://10.136.6.24:8983/solr/myappproduction-production_v8_shard1_replica1/ has
no replicas
2013-10-29T16:25:44.991Z Sync replicas to
http://10.136.6.24:8983/solr/myappproduction-production_v8_shard1_replica1/
2013-10-29T16:25:44.991Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:44.991Z Running the leader process for shard shard1
2013-10-29T16:25:44.991Z I may be the new leader - try and sync
2013-10-29T16:25:44.991Z Sync Success - now sync replicas to me
2013-10-29T16:25:44.991Z Checking if I should try and be the leader.
2013-10-29T16:25:44.940Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-feature-completion_v9_shard1_replica1/
shard1
2013-10-29T16:25:44.940Z makePath:
/collections/myappstaging-feature-completion_v9/leaders/shard1
2013-10-29T16:25:44.939Z
http://10.136.6.24:8983/solr/myappstaging-feature-completion_v9_shard1_replica1/
has no replicas
2013-10-29T16:25:44.939Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:44.939Z Sync Success - now sync replicas to me
2013-10-29T16:25:44.939Z I may be the new leader - try and sync
2013-10-29T16:25:44.939Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-feature-completion_v9_shard1_replica1/
2013-10-29T16:25:44.938Z Running the leader process for shard shard1
2013-10-29T16:25:44.938Z Checking if I should try and be the leader.
2013-10-29T16:25:44.880Z makePath:
/collections/myappstaging-profile_v7/leaders/shard1
2013-10-29T16:25:44.873Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-profile_v7_shard1_replica2/ shard1
2013-10-29T16:25:44.872Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-profile_v7_shard1_replica2/
2013-10-29T16:25:44.872Z Stopping recovery for
zkNodeName=core_node2core=myappstaging-profile_v7_shard1_replica2
2013-10-29T16:25:44.872Z I may be the new leader - try and sync
2013-10-29T16:25:44.872Z Checking if I should try and be the leader.
2013-10-29T16:25:44.872Z
http://10.136.6.24:8983/solr/myappstaging-profile_v7_shard1_replica2/ has no
replicas
2013-10-29T16:25:44.872Z Sync Success - now sync replicas to me
2013-10-29T16:25:44.872Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:44.863Z Running the leader process for shard shard1
2013-10-29T16:25:44.809Z makePath:
/collections/myappqa-master_v6/leaders/shard1
2013-10-29T16:25:44.808Z Stopping recovery for
zkNodeName=core_node2core=myappqa-master_v6_shard1_replica1
2013-10-29T16:25:44.808Z
http://10.136.6.24:8983/solr/myappqa-master_v6_shard1_replica1/ has no replicas
2013-10-29T16:25:44.808Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:44.808Z I am the new leader:
http://10.136.6.24:8983/solr/myappqa-master_v6_shard1_replica1/ shard1
2013-10-29T16:25:44.808Z Sync Success - now sync replicas to me
2013-10-29T16:25:44.808Z I may be the new leader - try and sync
2013-10-29T16:25:44.808Z Sync replicas to
http://10.136.6.24:8983/solr/myappqa-master_v6_shard1_replica1/
2013-10-29T16:25:44.808Z Checking if I should try and be the leader.
2013-10-29T16:25:44.807Z Running the leader process for shard shard1
2013-10-29T16:25:44.749Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-feature-completion_v8_shard1_replica1/
shard1
2013-10-29T16:25:44.749Z makePath:
/collections/myappstaging-feature-completion_v8/leaders/shard1
2013-10-29T16:25:44.749Z
http://10.136.6.24:8983/solr/myappstaging-feature-completion_v8_shard1_replica1/
has no replicas
2013-10-29T16:25:44.747Z I may be the new leader - try and sync
2013-10-29T16:25:44.747Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:44.747Z Sync Success - now sync replicas to me
2013-10-29T16:25:44.747Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-feature-completion_v8_shard1_replica1/
2013-10-29T16:25:44.747Z Checking if I should try and be the leader.
2013-10-29T16:25:44.746Z Running the leader process for shard shard1
2013-10-29T16:25:44.694Z makePath:
/collections/myappproduction-production_v9/leaders/shard1
2013-10-29T16:25:44.693Z
http://10.136.6.24:8983/solr/myappproduction-production_v9_shard1_replica2/ has
no replicas
2013-10-29T16:25:44.693Z I am the new leader:
http://10.136.6.24:8983/solr/myappproduction-production_v9_shard1_replica2/
shard1
2013-10-29T16:25:44.692Z I may be the new leader - try and sync
2013-10-29T16:25:44.692Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:44.692Z Sync Success - now sync replicas to me
2013-10-29T16:25:44.692Z Sync replicas to
http://10.136.6.24:8983/solr/myappproduction-production_v9_shard1_replica2/
2013-10-29T16:25:44.692Z Checking if I should try and be the leader.
2013-10-29T16:25:44.691Z Running the leader process for shard shard1
2013-10-29T16:25:44.647Z makePath:
/collections/myappstaging-alpha4_v9/leaders/shard1
2013-10-29T16:25:44.647Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-alpha4_v9_shard1_replica1/ shard1
2013-10-29T16:25:44.647Z
http://10.136.6.24:8983/solr/myappstaging-alpha4_v9_shard1_replica1/ has no
replicas
2013-10-29T16:25:44.632Z Sync Success - now sync replicas to me
2013-10-29T16:25:44.632Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-alpha4_v9_shard1_replica1/
2013-10-29T16:25:44.632Z I may be the new leader - try and sync
2013-10-29T16:25:44.632Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:44.631Z Checking if I should try and be the leader.
2013-10-29T16:25:44.630Z Running the leader process for shard shard1
2013-10-29T16:25:44.572Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-master_v9_shard1_replica1/ shard1
2013-10-29T16:25:44.572Z makePath:
/collections/myappstaging-master_v9/leaders/shard1
2013-10-29T16:25:44.571Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:44.571Z
http://10.136.6.24:8983/solr/myappstaging-master_v9_shard1_replica1/ has no
replicas
2013-10-29T16:25:44.571Z Sync Success - now sync replicas to me
2013-10-29T16:25:44.571Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-master_v9_shard1_replica1/
2013-10-29T16:25:44.571Z I may be the new leader - try and sync
2013-10-29T16:25:44.571Z Checking if I should try and be the leader.
2013-10-29T16:25:44.568Z Running the leader process for shard shard1
2013-10-29T16:25:44.511Z makePath:
/collections/myappstaging-preetalpha4_v9/leaders/shard1
2013-10-29T16:25:44.511Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v9_shard1_replica2/ shard1
2013-10-29T16:25:44.511Z
http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v9_shard1_replica2/ has
no replicas
2013-10-29T16:25:44.510Z Sync Success - now sync replicas to me
2013-10-29T16:25:44.509Z Checking if I should try and be the leader.
2013-10-29T16:25:44.509Z Running the leader process for shard shard1
2013-10-29T16:25:44.509Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v9_shard1_replica2/
2013-10-29T16:25:44.509Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:44.509Z I may be the new leader - try and sync
2013-10-29T16:25:44.450Z makePath:
/collections/collection1/leaders/shard1
2013-10-29T16:25:44.450Z I am the new leader:
http://10.136.6.24:8983/solr/collection1/ shard1
2013-10-29T16:25:44.444Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:44.444Z http://10.136.6.24:8983/solr/collection1/ has
no replicas
2013-10-29T16:25:44.444Z Sync Success - now sync replicas to me
2013-10-29T16:25:44.444Z Sync replicas to
http://10.136.6.24:8983/solr/collection1/
2013-10-29T16:25:44.444Z I may be the new leader - try and sync
2013-10-29T16:25:44.443Z Checking if I should try and be the leader.
2013-10-29T16:25:44.434Z Running the leader process for shard shard1
2013-10-29T16:25:44.418Z A cluster state change: WatchedEvent
state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred -
updating... (live nodes size: 1)
2013-10-29T16:25:44.418Z Updating live nodes... (1)
2013-10-29T16:25:43.715Z makePath:
/collections/myappqa-master_v9/leaders/shard1
2013-10-29T16:25:43.714Z Checking if I should try and be the leader.
2013-10-29T16:25:43.714Z I may be the new leader - try and sync
2013-10-29T16:25:43.714Z I am the new leader:
http://10.136.6.24:8983/solr/myappqa-master_v9_shard1_replica1/ shard1
2013-10-29T16:25:43.714Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:43.714Z Sync replicas to
http://10.136.6.24:8983/solr/myappqa-master_v9_shard1_replica1/
2013-10-29T16:25:43.714Z
http://10.136.6.24:8983/solr/myappqa-master_v9_shard1_replica1/ has no replicas
2013-10-29T16:25:43.714Z Sync Success - now sync replicas to me
2013-10-29T16:25:43.713Z Running the leader process for shard shard1
2013-10-29T16:25:43.654Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-alpha3_v8_shard1_replica2/ shard1
2013-10-29T16:25:43.654Z makePath:
/collections/myappstaging-alpha3_v8/leaders/shard1
2013-10-29T16:25:43.654Z
http://10.136.6.24:8983/solr/myappstaging-alpha3_v8_shard1_replica2/ has no
replicas
2013-10-29T16:25:43.653Z I may be the new leader - try and sync
2013-10-29T16:25:43.653Z Checking if I should try and be the leader.
2013-10-29T16:25:43.653Z Sync Success - now sync replicas to me
2013-10-29T16:25:43.653Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-alpha3_v8_shard1_replica2/
2013-10-29T16:25:43.653Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:43.652Z Running the leader process for shard shard1
2013-10-29T16:25:43.591Z makePath:
/collections/myappqa-master_v8/leaders/shard1
2013-10-29T16:25:43.591Z I am the new leader:
http://10.136.6.24:8983/solr/myappqa-master_v8_shard1_replica2/ shard1
2013-10-29T16:25:43.591Z
http://10.136.6.24:8983/solr/myappqa-master_v8_shard1_replica2/ has no replicas
2013-10-29T16:25:43.575Z I may be the new leader - try and sync
2013-10-29T16:25:43.575Z Checking if I should try and be the leader.
2013-10-29T16:25:43.575Z Sync Success - now sync replicas to me
2013-10-29T16:25:43.575Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:43.575Z Sync replicas to
http://10.136.6.24:8983/solr/myappqa-master_v8_shard1_replica2/
2013-10-29T16:25:43.574Z Running the leader process for shard shard1
2013-10-29T16:25:43.505Z makePath:
/collections/myappstaging-profile_v8/leaders/shard1
2013-10-29T16:25:43.498Z
http://10.136.6.24:8983/solr/myappstaging-profile_v8_shard1_replica2/ has no
replicas
2013-10-29T16:25:43.498Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-profile_v8_shard1_replica2/
2013-10-29T16:25:43.498Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-profile_v8_shard1_replica2/ shard1
2013-10-29T16:25:43.498Z Sync Success - now sync replicas to me
2013-10-29T16:25:43.497Z Stopping recovery for
zkNodeName=core_node1core=myappstaging-profile_v8_shard1_replica2
2013-10-29T16:25:43.497Z I may be the new leader - try and sync
2013-10-29T16:25:43.497Z Checking if I should try and be the leader.
2013-10-29T16:25:43.497Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:43.489Z Running the leader process for shard shard1
2013-10-29T16:25:43.472Z Update state numShards=1 message={
"operation":"state",
"state":"down",
"base_url":"http://10.170.2.54:8983/solr";,
"core":"hiv...
2013-10-29T16:25:43.456Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v9_shard1_replica2/ shard1
2013-10-29T16:25:43.456Z makePath:
/collections/myappstaging-alpha4-adam_v9/leaders/shard1
2013-10-29T16:25:43.456Z [myappstaging-alpha4-adam_v9_shard1_replica1]
webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=...
2013-10-29T16:25:43.440Z Sync Success - now sync replicas to me
2013-10-29T16:25:43.440Z PeerSync:
core=myappstaging-alpha4-adam_v9_shard1_replica2
url=http://10.136.6.24:8983/solr DONE. sync succeeded
2013-10-29T16:25:43.440Z
http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v9_shard1_replica2/ has
no replicas
2013-10-29T16:25:43.440Z PeerSync:
core=myappstaging-alpha4-adam_v9_shard1_replica2
url=http://10.136.6.24:8983/solr Our versions are newer.
ourLowThreshold=14495694169781043...
2013-10-29T16:25:43.439Z PeerSync:
core=myappstaging-alpha4-adam_v9_shard1_replica2
url=http://10.136.6.24:8983/solr Received 17 versions from
10.170.2.54:8983/solr/myappstag...
2013-10-29T16:25:43.330Z Updating cloud state from ZooKeeper...
2013-10-29T16:25:43.322Z PeerSync:
core=myappstaging-alpha4-adam_v9_shard1_replica2
url=http://10.136.6.24:8983/solr START
replicas=[http://10.170.2.54:8983/solr/myappstaging-...
2013-10-29T16:25:43.321Z I may be the new leader - try and sync
2013-10-29T16:25:43.321Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v9_shard1_replica2/
2013-10-29T16:25:43.321Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:43.321Z Checking if I should try and be the leader.
2013-10-29T16:25:43.314Z Running the leader process for shard shard1
2013-10-29T16:25:43.314Z Process current queue of collection creations
2013-10-29T16:25:43.314Z Starting to work on the main queue
2013-10-29T16:25:43.264Z Going to wait for coreNodeName: core_node2,
state: down, checkLive: null, onlyIfLeader: null
2013-10-29T16:25:43.262Z Creating new http client,
config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
2013-10-29T16:25:43.247Z Overseer
(id=234546968080679008-10.136.6.24:8983_solr-n_0000000021) starting
2013-10-29T16:25:43.247Z makePath: /overseer_elect/leader
2013-10-29T16:25:43.203Z PeerSync:
core=myappstaging-preetalpha4_v8_shard1_replica1
url=http://10.170.2.54:8983/solr DONE. sync succeeded
2013-10-29T16:25:43.203Z [myappstaging-preetalpha4_v8_shard1_replica1]
webapp=/solr path=/get
params={sync=http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v8_shard1_repl...
2013-10-29T16:25:43.203Z makePath:
/collections/myappstaging-preetalpha4_v8/leaders/shard1
2013-10-29T16:25:43.189Z PeerSync:
core=myappstaging-preetalpha4_v8_shard1_replica1
url=http://10.170.2.54:8983/solr Our versions are newer.
ourLowThreshold=14490017764527308...
2013-10-29T16:25:43.189Z
http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v8_shard1_replica2/: sync
completed with http://10.170.2.54:8983/solr/myappstaging-preetalpha4_...
2013-10-29T16:25:43.189Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v8_shard1_replica2/ shard1
2013-10-29T16:25:43.187Z PeerSync:
core=myappstaging-preetalpha4_v8_shard1_replica1
url=http://10.170.2.54:8983/solr Received 10 versions from
10.136.6.24:8983/solr/myappstag...
2013-10-29T16:25:43.187Z [myappstaging-preetalpha4_v8_shard1_replica2]
webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=...
2013-10-29T16:25:43.187Z PeerSync:
core=myappstaging-preetalpha4_v8_shard1_replica1
url=http://10.170.2.54:8983/solr START
replicas=[http://10.136.6.24:8983/solr/myappstaging-...
2013-10-29T16:25:43.187Z [myappstaging-preetalpha4_v8_shard1_replica1]
webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=...
2013-10-29T16:25:43.187Z
http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v8_shard1_replica2/: try
and ask http://10.170.2.54:8983/solr/myappstaging-preetalpha4_v8_shard1...
2013-10-29T16:25:43.187Z Sync Success - now sync replicas to me
2013-10-29T16:25:43.170Z PeerSync:
core=myappstaging-preetalpha4_v8_shard1_replica2
url=http://10.136.6.24:8983/solr DONE. sync succeeded
2013-10-29T16:25:43.170Z PeerSync:
core=myappstaging-preetalpha4_v8_shard1_replica2
url=http://10.136.6.24:8983/solr Received 10 versions from
10.170.2.54:8983/solr/myappstag...
2013-10-29T16:25:43.170Z PeerSync:
core=myappstaging-preetalpha4_v8_shard1_replica2
url=http://10.136.6.24:8983/solr Our versions are newer.
ourLowThreshold=14490017764527308...
2013-10-29T16:25:43.100Z PeerSync:
core=myappstaging-preetalpha4_v8_shard1_replica2
url=http://10.136.6.24:8983/solr START
replicas=[http://10.170.2.54:8983/solr/myappstaging-...
2013-10-29T16:25:43.100Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v8_shard1_replica2/
2013-10-29T16:25:43.100Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:43.100Z I may be the new leader - try and sync
2013-10-29T16:25:43.100Z Checking if I should try and be the leader.
2013-10-29T16:25:43.099Z Running the leader process for shard shard1
2013-10-29T16:25:43.056Z makePath:
/collections/myappstaging-alpha3_v7/leaders/shard1
2013-10-29T16:25:43.044Z [myappstaging-alpha3_v7_shard1_replica2]
webapp=/solr path=/get
params={sync=http://10.136.6.24:8983/solr/myappstaging-alpha3_v7_shard1_replica1/&;...
2013-10-29T16:25:43.044Z PeerSync:
core=myappstaging-alpha3_v7_shard1_replica2 url=http://10.170.2.54:8983/solr
DONE. sync succeeded
2013-10-29T16:25:43.041Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-alpha3_v7_shard1_replica1/ shard1
2013-10-29T16:25:43.041Z
http://10.136.6.24:8983/solr/myappstaging-alpha3_v7_shard1_replica1/: sync
completed with http://10.170.2.54:8983/solr/myappstaging-alpha3_v7_shard1_...
2013-10-29T16:25:43.041Z [myappstaging-alpha3_v7_shard1_replica1]
webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} st...
2013-10-29T16:25:43.040Z PeerSync:
core=myappstaging-alpha3_v7_shard1_replica2 url=http://10.170.2.54:8983/solr
Our versions are newer. ourLowThreshold=1448183627488690176 ot...
2013-10-29T16:25:43.040Z PeerSync:
core=myappstaging-alpha3_v7_shard1_replica2 url=http://10.170.2.54:8983/solr
Received 98 versions from 10.136.6.24:8983/solr/myappstaging-a...
2013-10-29T16:25:43.027Z PeerSync:
core=myappstaging-alpha3_v7_shard1_replica2 url=http://10.170.2.54:8983/solr
START replicas=[http://10.136.6.24:8983/solr/myappstaging-alpha...
2013-10-29T16:25:43.011Z [myappstaging-alpha3_v7_shard1_replica2]
webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} st...
2013-10-29T16:25:42.989Z
http://10.136.6.24:8983/solr/myappstaging-alpha3_v7_shard1_replica1/: try and
ask http://10.170.2.54:8983/solr/myappstaging-alpha3_v7_shard1_replica2/...
2013-10-29T16:25:42.989Z Sync Success - now sync replicas to me
2013-10-29T16:25:42.988Z PeerSync:
core=myappstaging-alpha3_v7_shard1_replica1 url=http://10.136.6.24:8983/solr
DONE. sync succeeded
2013-10-29T16:25:42.982Z PeerSync:
core=myappstaging-alpha3_v7_shard1_replica1 url=http://10.136.6.24:8983/solr
Our versions are newer. ourLowThreshold=1448183627488690176 ot...
2013-10-29T16:25:42.982Z PeerSync:
core=myappstaging-alpha3_v7_shard1_replica1 url=http://10.136.6.24:8983/solr
Received 98 versions from 10.170.2.54:8983/solr/myappstaging-a...
2013-10-29T16:25:42.955Z PeerSync:
core=myappstaging-alpha3_v7_shard1_replica1 url=http://10.136.6.24:8983/solr
START replicas=[http://10.170.2.54:8983/solr/myappstaging-alpha...
2013-10-29T16:25:42.954Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-alpha3_v7_shard1_replica1/
2013-10-29T16:25:42.954Z Stopping recovery for
zkNodeName=core_node1core=myappstaging-alpha3_v7_shard1_replica1
2013-10-29T16:25:42.953Z I may be the new leader - try and sync
2013-10-29T16:25:42.953Z Running the leader process for shard shard1
2013-10-29T16:25:42.953Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:42.953Z Checking if I should try and be the leader.
2013-10-29T16:25:42.897Z [myappstaging-alpha4-adam_v8_shard1_replica1]
webapp=/solr path=/get
params={sync=http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v8_shard1_repl...
2013-10-29T16:25:42.896Z makePath:
/collections/myappstaging-alpha4-adam_v8/leaders/shard1
2013-10-29T16:25:42.882Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v8_shard1_replica2/ shard1
2013-10-29T16:25:42.881Z
http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v8_shard1_replica2/: sync
completed with http://10.170.2.54:8983/solr/myappstaging-alpha4-adam_...
2013-10-29T16:25:42.881Z [myappstaging-alpha4-adam_v8_shard1_replica2]
webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=...
2013-10-29T16:25:42.881Z PeerSync:
core=myappstaging-alpha4-adam_v8_shard1_replica1
url=http://10.170.2.54:8983/solr DONE. sync succeeded
2013-10-29T16:25:42.880Z PeerSync:
core=myappstaging-alpha4-adam_v8_shard1_replica1
url=http://10.170.2.54:8983/solr Received 10 versions from
10.136.6.24:8983/solr/myappstag...
2013-10-29T16:25:42.880Z PeerSync:
core=myappstaging-alpha4-adam_v8_shard1_replica1
url=http://10.170.2.54:8983/solr Our versions are newer.
ourLowThreshold=14489007407885189...
2013-10-29T16:25:42.798Z [myappstaging-alpha4-adam_v8_shard1_replica1]
webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=...
2013-10-29T16:25:42.798Z PeerSync:
core=myappstaging-alpha4-adam_v8_shard1_replica1
url=http://10.170.2.54:8983/solr START
replicas=[http://10.136.6.24:8983/solr/myappstaging-...
2013-10-29T16:25:42.782Z Sync Success - now sync replicas to me
2013-10-29T16:25:42.782Z PeerSync:
core=myappstaging-alpha4-adam_v8_shard1_replica2
url=http://10.136.6.24:8983/solr DONE. sync succeeded
2013-10-29T16:25:42.782Z
http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v8_shard1_replica2/: try
and ask http://10.170.2.54:8983/solr/myappstaging-alpha4-adam_v8_shard1...
2013-10-29T16:25:42.782Z PeerSync:
core=myappstaging-alpha4-adam_v8_shard1_replica2
url=http://10.136.6.24:8983/solr Our versions are newer.
ourLowThreshold=14489007407885189...
2013-10-29T16:25:42.781Z PeerSync:
core=myappstaging-alpha4-adam_v8_shard1_replica2
url=http://10.136.6.24:8983/solr Received 10 versions from
10.170.2.54:8983/solr/myappstag...
2013-10-29T16:25:42.765Z PeerSync:
core=myappstaging-alpha4-adam_v8_shard1_replica2
url=http://10.136.6.24:8983/solr START
replicas=[http://10.170.2.54:8983/solr/myappstaging-...
2013-10-29T16:25:42.754Z Checking if I should try and be the leader.
2013-10-29T16:25:42.754Z I may be the new leader - try and sync
2013-10-29T16:25:42.754Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v8_shard1_replica2/
2013-10-29T16:25:42.754Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:42.754Z Stopping recovery for
zkNodeName=core_node2core=myappstaging-alpha4-adam_v8_shard1_replica2
2013-10-29T16:25:42.749Z Running the leader process for shard shard1
2013-10-29T16:25:42.693Z makePath:
/collections/myappstaging-preetalpha5_v9/leaders/shard1
2013-10-29T16:25:42.693Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-preetalpha5_v9_shard1_replica2/ shard1
2013-10-29T16:25:42.692Z PeerSync:
core=myappstaging-preetalpha5_v9_shard1_replica1
url=http://10.170.2.54:8983/solr DONE. sync succeeded
2013-10-29T16:25:42.692Z
http://10.136.6.24:8983/solr/myappstaging-preetalpha5_v9_shard1_replica2/: sync
completed with http://10.170.2.54:8983/solr/myappstaging-preetalpha5_...
2013-10-29T16:25:42.692Z [myappstaging-preetalpha5_v9_shard1_replica1]
webapp=/solr path=/get
params={sync=http://10.136.6.24:8983/solr/myappstaging-preetalpha5_v9_shard1_repl...
2013-10-29T16:25:42.618Z [myappstaging-preetalpha5_v9_shard1_replica2]
webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=...
2013-10-29T16:25:42.603Z PeerSync:
core=myappstaging-preetalpha5_v9_shard1_replica1
url=http://10.170.2.54:8983/solr Received 10 versions from
10.136.6.24:8983/solr/myappstag...
2013-10-29T16:25:42.603Z PeerSync:
core=myappstaging-preetalpha5_v9_shard1_replica1
url=http://10.170.2.54:8983/solr Our versions are newer.
ourLowThreshold=14499085375829442...
2013-10-29T16:25:42.596Z PeerSync:
core=myappstaging-preetalpha5_v9_shard1_replica1
url=http://10.170.2.54:8983/solr START
replicas=[http://10.136.6.24:8983/solr/myappstaging-...
2013-10-29T16:25:42.579Z [myappstaging-preetalpha5_v9_shard1_replica1]
webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=...
2013-10-29T16:25:42.565Z
http://10.136.6.24:8983/solr/myappstaging-preetalpha5_v9_shard1_replica2/: try
and ask http://10.170.2.54:8983/solr/myappstaging-preetalpha5_v9_shard1...
2013-10-29T16:25:42.565Z Sync Success - now sync replicas to me
2013-10-29T16:25:42.565Z PeerSync:
core=myappstaging-preetalpha5_v9_shard1_replica2
url=http://10.136.6.24:8983/solr DONE. sync succeeded
2013-10-29T16:25:42.553Z PeerSync:
core=myappstaging-preetalpha5_v9_shard1_replica2
url=http://10.136.6.24:8983/solr Our versions are newer.
ourLowThreshold=14499085375829442...
2013-10-29T16:25:42.553Z PeerSync:
core=myappstaging-preetalpha5_v9_shard1_replica2
url=http://10.136.6.24:8983/solr Received 10 versions from
10.170.2.54:8983/solr/myappstag...
2013-10-29T16:25:42.549Z PeerSync:
core=myappstaging-preetalpha5_v9_shard1_replica2
url=http://10.136.6.24:8983/solr START
replicas=[http://10.170.2.54:8983/solr/myappstaging-...
2013-10-29T16:25:42.549Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-preetalpha5_v9_shard1_replica2/
2013-10-29T16:25:42.549Z I may be the new leader - try and sync
2013-10-29T16:25:42.548Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:42.547Z Running the leader process for shard shard1
2013-10-29T16:25:42.547Z Checking if I should try and be the leader.
2013-10-29T16:25:42.505Z [myappstaging-alpha4_v8_shard1_replica2]
webapp=/solr path=/get
params={sync=http://10.136.6.24:8983/solr/myappstaging-alpha4_v8_shard1_replica1/&;...
2013-10-29T16:25:42.502Z makePath:
/collections/myappstaging-alpha4_v8/leaders/shard1
2013-10-29T16:25:42.489Z [myappstaging-alpha4_v8_shard1_replica1]
webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} st...
2013-10-29T16:25:42.489Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-alpha4_v8_shard1_replica1/ shard1
2013-10-29T16:25:42.489Z
http://10.136.6.24:8983/solr/myappstaging-alpha4_v8_shard1_replica1/: sync
completed with http://10.170.2.54:8983/solr/myappstaging-alpha4_v8_shard1_...
2013-10-29T16:25:42.488Z PeerSync:
core=myappstaging-alpha4_v8_shard1_replica2 url=http://10.170.2.54:8983/solr
DONE. sync succeeded
2013-10-29T16:25:42.488Z PeerSync:
core=myappstaging-alpha4_v8_shard1_replica2 url=http://10.170.2.54:8983/solr
Our versions are newer. ourLowThreshold=1449121459772325888 ot...
2013-10-29T16:25:42.488Z PeerSync:
core=myappstaging-alpha4_v8_shard1_replica2 url=http://10.170.2.54:8983/solr
Received 1 versions from 10.136.6.24:8983/solr/myappstaging-al...
2013-10-29T16:25:42.411Z
http://10.136.6.24:8983/solr/myappstaging-alpha4_v8_shard1_replica1/: try and
ask http://10.170.2.54:8983/solr/myappstaging-alpha4_v8_shard1_replica2/...
2013-10-29T16:25:42.411Z Sync Success - now sync replicas to me
2013-10-29T16:25:42.410Z PeerSync:
core=myappstaging-alpha4_v8_shard1_replica2 url=http://10.170.2.54:8983/solr
START replicas=[http://10.136.6.24:8983/solr/myappstaging-alpha...
2013-10-29T16:25:42.410Z [myappstaging-alpha4_v8_shard1_replica2]
webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} st...
2013-10-29T16:25:42.394Z PeerSync:
core=myappstaging-alpha4_v8_shard1_replica1 url=http://10.136.6.24:8983/solr
DONE. sync succeeded
2013-10-29T16:25:42.394Z PeerSync:
core=myappstaging-alpha4_v8_shard1_replica1 url=http://10.136.6.24:8983/solr
Our versions are newer. ourLowThreshold=1449121459772325888 ot...
2013-10-29T16:25:42.394Z PeerSync:
core=myappstaging-alpha4_v8_shard1_replica1 url=http://10.136.6.24:8983/solr
Received 1 versions from 10.170.2.54:8983/solr/myappstaging-al...
2013-10-29T16:25:42.388Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
2013-10-29T16:25:42.388Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
2013-10-29T16:25:42.387Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252)
2013-10-29T16:25:42.387Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
2013-10-29T16:25:42.387Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249)
2013-10-29T16:25:42.387Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556)
2013-10-29T16:25:42.387Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249)
2013-10-29T16:25:42.383Z <13>Oct 29 16:25:49 solr-p1 solr:
[qtp739893596-11195] ERROR org.apache.solr.servlet.SolrDispatchFilter -
null:org.apache.zookeeper.KeeperExcept...
2013-10-29T16:25:42.371Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
2013-10-29T16:25:42.371Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
2013-10-29T16:25:42.371Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556)
2013-10-29T16:25:42.370Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252)
2013-10-29T16:25:42.370Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249)
2013-10-29T16:25:42.370Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
2013-10-29T16:25:42.370Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249)
2013-10-29T16:25:42.369Z <13>Oct 29 16:25:49 solr-p1 solr:
[qtp739893596-11186] ERROR org.apache.solr.servlet.SolrDispatchFilter -
null:org.apache.zookeeper.KeeperExcept...
2013-10-29T16:25:42.362Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
2013-10-29T16:25:42.362Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
2013-10-29T16:25:42.362Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
2013-10-29T16:25:42.362Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556)
2013-10-29T16:25:42.362Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249)
2013-10-29T16:25:42.361Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249)
2013-10-29T16:25:42.361Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252)
2013-10-29T16:25:42.360Z <13>Oct 29 16:25:49 solr-p1 solr:
[qtp739893596-11194] ERROR org.apache.solr.servlet.SolrDispatchFilter -
null:org.apache.zookeeper.KeeperExcept...
2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249)
2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249)
2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556)
2013-10-29T16:25:42.351Z <13>Oct 29 16:25:49 solr-p1 solr:
[qtp739893596-11196] ERROR org.apache.solr.servlet.SolrDispatchFilter -
null:org.apache.zookeeper.KeeperExcept...
2013-10-29T16:25:42.351Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252)
2013-10-29T16:25:42.347Z According to ZK I
(id=162489378735849528-10.170.2.54:8983_solr-n_0000000019) am no longer a
leader.
2013-10-29T16:25:42.346Z
null:org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /aliases.json
at org.apache.zookeeper.Ke...
2013-10-29T16:25:42.339Z
null:org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /aliases.json
at org.apache.zookeeper.Ke...
2013-10-29T16:25:42.327Z PeerSync:
core=myappstaging-alpha4_v8_shard1_replica1 url=http://10.136.6.24:8983/solr
START replicas=[http://10.170.2.54:8983/solr/myappstaging-alpha...
2013-10-29T16:25:42.326Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-alpha4_v8_shard1_replica1/
2013-10-29T16:25:42.324Z I may be the new leader - try and sync
2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr:
[qtp739893596-11192] ERROR org.apache.solr.servlet.SolrDispatchFilter -
null:org.apache.zookeeper.KeeperExcept...
2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249)
2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249)
2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252)
2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556)
2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
2013-10-29T16:25:42.316Z Checking if I should try and be the leader.
2013-10-29T16:25:42.316Z
null:org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /aliases.json
at org.apache.zookeeper.Ke...
2013-10-29T16:25:42.316Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:42.310Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556)
2013-10-29T16:25:42.310Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
2013-10-29T16:25:42.310Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
2013-10-29T16:25:42.309Z
null:org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /aliases.json
at org.apache.zookeeper.Ke...
2013-10-29T16:25:42.309Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249)
2013-10-29T16:25:42.309Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
2013-10-29T16:25:42.307Z Running the leader process for shard shard1
2013-10-29T16:25:42.306Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249)
2013-10-29T16:25:42.306Z <13>Oct 29 16:25:49 solr-p1 solr: #011at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252)
2013-10-29T16:25:42.300Z Overseer cannot talk to ZK
2013-10-29T16:25:42.300Z
null:org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /aliases.json
at org.apache.zookeeper.Ke...
2013-10-29T16:25:42.299Z <13>Oct 29 16:25:49 solr-p1 solr:
[qtp739893596-11191] ERROR org.apache.solr.servlet.SolrDispatchFilter -
null:org.apache.zookeeper.KeeperExcept...
2013-10-29T16:25:42.299Z
null:org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /aliases.json
at org.apache.zookeeper.Ke...
2013-10-29T16:25:42.298Z
2013-10-29T16:25:42.268Z makePath:
/collections/myappstaging-feature-rulecompletion_v9/leaders/shard1
2013-10-29T16:25:42.263Z I am the new leader:
http://10.136.6.24:8983/solr/myappstaging-feature-rulecompletion_v9_shard1_replica1/
shard1
2013-10-29T16:25:42.259Z
[myappstaging-feature-rulecompletion_v9_shard1_replica2] webapp=/solr path=/get
params={sync=http://10.136.6.24:8983/solr/myappstaging-feature-rulecom...
2013-10-29T16:25:42.259Z PeerSync:
core=myappstaging-feature-rulecompletion_v9_shard1_replica2
url=http://10.170.2.54:8983/solr DONE. sync succeeded
2013-10-29T16:25:42.251Z
[myappstaging-feature-rulecompletion_v9_shard1_replica1] webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&a...
2013-10-29T16:25:42.251Z PeerSync:
core=myappstaging-feature-rulecompletion_v9_shard1_replica2
url=http://10.170.2.54:8983/solr Our versions are newer.
ourLowThreshold=144962...
2013-10-29T16:25:42.251Z PeerSync:
core=myappstaging-feature-rulecompletion_v9_shard1_replica2
url=http://10.170.2.54:8983/solr Received 4 versions from
10.136.6.24:8983/solr...
2013-10-29T16:25:42.251Z
http://10.136.6.24:8983/solr/myappstaging-feature-rulecompletion_v9_shard1_replica1/:
sync completed with http://10.170.2.54:8983/solr/myappstaging-f...
2013-10-29T16:25:42.247Z
http://10.136.6.24:8983/solr/myappstaging-feature-rulecompletion_v9_shard1_replica1/:
try and ask http://10.170.2.54:8983/solr/myappstaging-feature-ru...
2013-10-29T16:25:42.246Z Sync Success - now sync replicas to me
2013-10-29T16:25:42.246Z PeerSync:
core=myappstaging-feature-rulecompletion_v9_shard1_replica1
url=http://10.136.6.24:8983/solr DONE. sync succeeded
2013-10-29T16:25:42.239Z PeerSync:
core=myappstaging-feature-rulecompletion_v9_shard1_replica2
url=http://10.170.2.54:8983/solr START
replicas=[http://10.136.6.24:8983/solr/hi...
2013-10-29T16:25:42.236Z
[myappstaging-feature-rulecompletion_v9_shard1_replica2] webapp=/solr path=/get
params={getVersions=100&distrib=false&wt=javabin&qt=/get&a...
2013-10-29T16:25:42.236Z Connection with ZooKeeper reestablished.
2013-10-29T16:25:42.236Z publishing
core=myappstaging-alpha3_v7_shard1_replica2 state=down
2013-10-29T16:25:42.232Z PeerSync:
core=myappstaging-feature-rulecompletion_v9_shard1_replica1
url=http://10.136.6.24:8983/solr Our versions are newer.
ourLowThreshold=144962...
2013-10-29T16:25:42.232Z PeerSync:
core=myappstaging-feature-rulecompletion_v9_shard1_replica1
url=http://10.136.6.24:8983/solr Received 4 versions from
10.170.2.54:8983/solr...
2013-10-29T16:25:42.231Z PeerSync:
core=myappstaging-feature-rulecompletion_v9_shard1_replica1
url=http://10.136.6.24:8983/solr START
replicas=[http://10.170.2.54:8983/solr/hi...
2013-10-29T16:25:42.230Z Client is connected to ZooKeeper
2013-10-29T16:25:42.230Z Sync replicas to
http://10.136.6.24:8983/solr/myappstaging-feature-rulecompletion_v9_shard1_replica1/
2013-10-29T16:25:42.230Z I may be the new leader - try and sync
2013-10-29T16:25:42.230Z My last published State was Active, it's okay
to be the leader.
2013-10-29T16:25:42.229Z Watcher
org.apache.solr.common.cloud.ConnectionManager@52e80740
name:ZooKeeperConnection
Watcher:solr-z1.domain.com:2181,solr-z2.domain.com:2181,sol...
2013-10-29T16:25:42.223Z Checking if I should try and be the leader.
2013-10-29T16:25:42.222Z Waiting for client to connect to ZooKeeper
2013-10-29T16:25:42.211Z Running the leader process for shard shard1
2013-10-29T16:25:42.186Z Connection expired - starting a new one...
2013-10-29T16:25:42.186Z Our previous ZooKeeper session was expired.
Attempting to reconnect to recover relationship with ZooKeeper...
2013-10-29T16:25:42.179Z Watcher fired on path: null state: Expired type
None
2013-10-29T16:25:42.179Z Watcher fired on path: null state: Expired type
None
2013-10-29T16:25:42.179Z Watcher fired on path: null state: Expired type
None
2013-10-29T16:25:42.179Z Watcher fired on path: null state: Expired type
None
2013-10-29T16:25:42.179Z Watcher
org.apache.solr.common.cloud.ConnectionManager@52e80740
name:ZooKeeperConnection
Watcher:solr-z1.domain.com:2181,solr-z2.domain.com:2181,sol...
2013-10-29T16:25:42.169Z
[myappstaging-feature-rulecompletion2_v9_shard1_replica2] webapp=/solr
path=/get
params={sync=http://10.136.6.24:8983/solr/myappstaging-feature-ruleco...
2013-10-29T16:25:42.169Z PeerSync:
core=myappstaging-feature-rulecompletion2_v9_shard1_replica2
url=http://10.170.2.54:8983/solr DONE. sync succeeded
> Strange error condition with cloud replication not working quite right
> ----------------------------------------------------------------------
>
> Key: SOLR-5407
> URL: https://issues.apache.org/jira/browse/SOLR-5407
> Project: Solr
> Issue Type: Bug
> Affects Versions: 4.5
> Reporter: Nathan Neulinger
> Labels: cloud, replication
>
> I have a clodu deployment of 4.5 on EC2. Architecture is 3 dedicated ZK
> nodes, and a pair of solr nodes. I'll apologize in advance that this error
> report is not going to have a lot of detail, I'm really hoping that the
> scenario/description will trigger some "likely" possible explanation.
> The situation I got into was that the server had decided to fail over, so my
> app servers were all taking to what should have been the primary for most of
> the shards/collections, but actually was the replica.
> Here's where it gets odd - no errors being returned to the client code for
> any of the searches or document updates - and the current primary server was
> definitely receiving all of the updates - even though they were being
> submitted to the inactive/replica node. (clients talking to solr-p1, which
> was not primary at the time, and writes were being passed through to solr-r1,
> which was primary at the time.)
> All sounds good so far right? Except - the replica server at the time,
> through which the writes were passing - never got any of those content
> updates. It had an old unmodified copy of the index.
> I restarted solr-p1 (was the replica at the time) - no change in behavior.
> Behavior did not change until I killed and restarted the current primary
> (solr-r1) to force it to fail over.
> At that point, everything was all happy again and working properly.
> Until this morning, when one of the developers provisioned a new collection,
> which happened to put it's primary on solr-r1. Again, clients all pointing at
> solr-p1. The developer reported that the documents were going into the index,
> but not visible on the replica server.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
