[
https://issues.apache.org/jira/browse/SOLR-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Darren Spehr updated SOLR-6995:
-------------------------------
Description:
This is related to a question I posted on
[stackoverflow|http://stackoverflow.com/questions/28004832/solr-4-10-3-is-not-proceeding-to-leader-election-on-new-cluster-startup-hangs]
a day ago.
When deploying a new cluster, new nodes never proceed to an 'active' state. I
have used both a custom (yet simple) deployment, as well as using the example
framework you get when downloading the Solr distribution. Further inspection of
the logs, and comparing them to the output of a 4.10.1 start up, it looks like
the nodes never move on to leader election.
For an apples-to-apples comparison I moved the solr/collection1 directory to
solr/ttPoiMDS, and modified the core.properties appropriately. I also removed
the 'conf' directory, assuming that it would pick this up from our ZooKeeper
cluster.
Here is the start up:
{code}
java
-Dcollection.configName=ttPoiMDS
-Djava.util.logging.config.file=etc/logging.properties
-DzkHost=our_zk_host_1:<zk_port>
-Djetty.port=8983
-jar start.jar
{code}
Here is the output in the logs:
{code}
0 [main] INFO org.eclipse.jetty.server.Server – jetty-8.1.10.v20130312
32 [main] INFO org.eclipse.jetty.deploy.providers.ScanningAppProvider –
Deployment monitor /site/pkgs/solr/solr-4.10.3/example/contexts at interval 0
38 [main] INFO org.eclipse.jetty.deploy.DeploymentManager – Deployable
added: /site/pkgs/solr/solr-4.10.3/example/contexts/solr-jetty-context.xml
114 [main] INFO org.eclipse.jetty.webapp.WebInfConfiguration – Extract
jar:file:/site/pkgs/solr/solr-4.10.3/example/webapps/solr.war!/ to
/site/pkgs/solr/solr-4.10.3/example/solr-webapp/webapp
1527 [main] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor – NO
JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
1597 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
SolrDispatchFilter.init()
1616 [main] INFO org.apache.solr.core.SolrResourceLoader – JNDI not
configured for solr (NoInitialContextEx)
1616 [main] INFO org.apache.solr.core.SolrResourceLoader – solr home
defaulted to 'solr/' (could not find system property or JNDI)
1617 [main] INFO org.apache.solr.core.SolrResourceLoader – new
SolrResourceLoader for directory: 'solr/'
1748 [main] INFO org.apache.solr.core.ConfigSolr – Loading container
configuration from /site/pkgs/solr/solr-4.10.3/example/solr/solr.xml
1884 [main] INFO org.apache.solr.core.CoresLocator – Config-defined core root
directory: /site/pkgs/solr/solr-4.10.3/example/solr
1890 [main] INFO org.apache.solr.core.CoreContainer – New CoreContainer
378496804
1890 [main] INFO org.apache.solr.core.CoreContainer – Loading cores into
CoreContainer [instanceDir=solr/]
1905 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting socketTimeout to: 0
1905 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting urlScheme to: null
1909 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting connTimeout to: 0
1911 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting maxConnectionsPerHost to: 20
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting corePoolSize to: 0
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting maximumPoolSize to: 2147483647
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting maxThreadIdleTime to: 5
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting sizeOfQueue to: -1
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting fairnessPolicy to: false
2054 [main] INFO org.apache.solr.update.UpdateShardHandler – Creating
UpdateShardHandler HTTP client with params:
socketTimeout=0&connTimeout=0&retry=false
2056 [main] INFO org.apache.solr.logging.LogWatcher – SLF4J impl is
org.slf4j.impl.Log4jLoggerFactory
2057 [main] INFO org.apache.solr.logging.LogWatcher – Registering Log
Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2058 [main] INFO org.apache.solr.core.CoreContainer – Host Name:
2059 [main] INFO org.apache.solr.core.ZkContainer – Zookeeper
client=our_zk_host_1:2195
2140 [main] INFO org.apache.solr.common.cloud.ConnectionManager – Waiting for
client to connect to ZooKeeper
2220 [main-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager –
Watcher org.apache.solr.common.cloud.ConnectionManager@36dadde6
name:ZooKeeperConnection Watcher:our_zk_host_1:2195 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
2220 [main] INFO org.apache.solr.common.cloud.ConnectionManager – Client is
connected to ZooKeeper
2293 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/queue
2447 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/collection-queue-work
2609 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/collection-map-running
2758 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/collection-map-completed
2919 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/collection-map-failure
3093 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/live_nodes
3173 [main] INFO org.apache.solr.cloud.ZkController – Register node as live
in ZooKeeper:/live_nodes/our_solr_host:8983_solr
3201 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/live_nodes/our_solr_host:8983_solr
3371 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer_elect
3477 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer_elect/election
3577 [main] INFO org.apache.solr.cloud.Overseer – Overseer (id=null) closing
3659 [main] INFO org.apache.solr.cloud.ElectionContext – I am going to be the
leader our_solr_host:8983_solr
3662 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer_elect/leader
3763 [main] INFO org.apache.solr.cloud.Overseer – Overseer
(id=93155855038349319-our_solr_host:8983_solr-n_0000000000) starting
3887 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/queue-work
4409 [main] INFO org.apache.solr.cloud.OverseerAutoReplicaFailoverThread –
Starting OverseerAutoReplicaFailoverThread
autoReplicaFailoverWorkLoopDelay=10000
autoReplicaFailoverWaitAfterExpiration=30000
autoReplicaFailoverBadNodeExpiration=60000
4438
[OverseerCollectionProcessor-93155855038349319-our_solr_host:8983_solr-n_0000000000]
INFO org.apache.solr.cloud.OverseerCollectionProcessor – Process current
queue of collection creations
4456 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/clusterstate.json
4559 [main] INFO org.apache.solr.common.cloud.ZkStateReader – Updating
cluster state from ZooKeeper...
4731
[OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
INFO org.apache.solr.cloud.Overseer – Starting to work on the main queue
4736 [main] INFO org.apache.solr.core.CoresLocator – Looking for core
definitions underneath /site/pkgs/solr/solr-4.10.3/example/solr
4744 [main] INFO org.apache.solr.core.CoresLocator – Found core ttPoiMDS in
/site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/
4744 [main] INFO org.apache.solr.core.CoresLocator – Found 1 core definitions
4767 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
publishing core=ttPoiMDS state=down collection=ttPoiMDS
4767 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
numShards not found on descriptor - reading it from system property
4796 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
look for our core node name
4797 [zkCallback-2-thread-1] INFO org.apache.solr.cloud.DistributedQueue –
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
4897
[OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
INFO org.apache.solr.cloud.Overseer – Update state numShards=null message={
"operation":"state",
"shard":null,
"roles":null,
"state":"down",
"core":"ttPoiMDS",
"collection":"ttPoiMDS",
"node_name":"our_solr_host:8983_solr",
"base_url":"http://our_solr_host:8983/solr"}
4899
[OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
INFO org.apache.solr.cloud.Overseer – Assigning new node to shard shard=shard1
5052 [zkCallback-2-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader
– A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
5800 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
waiting to find shard id in clusterstate for ttPoiMDS
5801 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
Check for collection zkNode:ttPoiMDS
5924 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
Collection zkNode exists
5925 [coreLoadExecutor-6-thread-1] INFO
org.apache.solr.common.cloud.ZkStateReader – Load collection config
from:/collections/ttPoiMDS
5970 [coreLoadExecutor-6-thread-1] INFO
org.apache.solr.common.cloud.ZkStateReader – path=/collections/ttPoiMDS
configName=ttPoiMDS specified config exists in ZooKeeper
5970 [coreLoadExecutor-6-thread-1] INFO
org.apache.solr.core.SolrResourceLoader – new SolrResourceLoader for
directory: '/site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/'
6054 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.SolrConfig –
Adding specified lib dirs to ClassLoader
6055 [coreLoadExecutor-6-thread-1] WARN
org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to
add to classloader: ../../lib/mq (resolved as:
/site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/../../lib/mq).
6139 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.SolrConfig –
Using Lucene MatchVersion: 4.10.3
6238 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.Config – Loaded
SolrConfig: solrconfig.xml
6320 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.schema.IndexSchema –
Reading Solr Schema from /configs/ttPoiMDS/schema.xml
6347 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.schema.IndexSchema –
[ttPoiMDS] Schema name=poiMDS
6582 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
user.dir=/site/pkgs/solr/solr-4.10.3/example
6582 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
SolrDispatchFilter.init() done
6614 [main] INFO org.eclipse.jetty.server.AbstractConnector – Started
[email protected]:8983
{code}
Startup then seems to hang. The Solr UI is available, but it shows that no
cores are available. In the cloud view the configuration from ZooKeeper is
present.
was:
This is related to a question I posted on
[stackoverflow|http://stackoverflow.com/questions/28004832/solr-4-10-3-is-not-proceeding-to-leader-election-on-new-cluster-startup-hangs]
a day ago.
When deploying a new cluster, new nodes never proceed to an 'active' state. I
have used both a custom (yet simple) deployment, as well as using the example
framework you get when downloading the Solr distribution. Further inspection of
the logs, and comparing them to the output of a 4.10.1 start up, it looks like
the nodes never move on to leader election.
For an apples-to-apples comparison I moved the solr/collection1 directory to
solr/ttPoiMDS, and modified the core.properties appropriately. I also removed
the 'conf' directory, assuming that it would pick this up from our ZooKeeper
cluster.
Here is the start up:
{code}
java
-Dcollection.configName=ttPoiMDS
-Djava.util.logging.config.file=etc/logging.properties
-DzkHost=our_zk_host_1:<zk_port>
-Djetty.port=8983
-jar start.jar
{code}
Here is the output in the logs:
{panel}
0 [main] INFO org.eclipse.jetty.server.Server – jetty-8.1.10.v20130312
32 [main] INFO org.eclipse.jetty.deploy.providers.ScanningAppProvider –
Deployment monitor /site/pkgs/solr/solr-4.10.3/example/contexts at interval 0
38 [main] INFO org.eclipse.jetty.deploy.DeploymentManager – Deployable
added: /site/pkgs/solr/solr-4.10.3/example/contexts/solr-jetty-context.xml
114 [main] INFO org.eclipse.jetty.webapp.WebInfConfiguration – Extract
jar:file:/site/pkgs/solr/solr-4.10.3/example/webapps/solr.war!/ to
/site/pkgs/solr/solr-4.10.3/example/solr-webapp/webapp
1527 [main] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor – NO
JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
1597 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
SolrDispatchFilter.init()
1616 [main] INFO org.apache.solr.core.SolrResourceLoader – JNDI not
configured for solr (NoInitialContextEx)
1616 [main] INFO org.apache.solr.core.SolrResourceLoader – solr home
defaulted to 'solr/' (could not find system property or JNDI)
1617 [main] INFO org.apache.solr.core.SolrResourceLoader – new
SolrResourceLoader for directory: 'solr/'
1748 [main] INFO org.apache.solr.core.ConfigSolr – Loading container
configuration from /site/pkgs/solr/solr-4.10.3/example/solr/solr.xml
1884 [main] INFO org.apache.solr.core.CoresLocator – Config-defined core root
directory: /site/pkgs/solr/solr-4.10.3/example/solr
1890 [main] INFO org.apache.solr.core.CoreContainer – New CoreContainer
378496804
1890 [main] INFO org.apache.solr.core.CoreContainer – Loading cores into
CoreContainer [instanceDir=solr/]
1905 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting socketTimeout to: 0
1905 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting urlScheme to: null
1909 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting connTimeout to: 0
1911 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting maxConnectionsPerHost to: 20
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting corePoolSize to: 0
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting maximumPoolSize to: 2147483647
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting maxThreadIdleTime to: 5
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting sizeOfQueue to: -1
1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory –
Setting fairnessPolicy to: false
2054 [main] INFO org.apache.solr.update.UpdateShardHandler – Creating
UpdateShardHandler HTTP client with params:
socketTimeout=0&connTimeout=0&retry=false
2056 [main] INFO org.apache.solr.logging.LogWatcher – SLF4J impl is
org.slf4j.impl.Log4jLoggerFactory
2057 [main] INFO org.apache.solr.logging.LogWatcher – Registering Log
Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
2058 [main] INFO org.apache.solr.core.CoreContainer – Host Name:
2059 [main] INFO org.apache.solr.core.ZkContainer – Zookeeper
client=our_zk_host_1:2195
2140 [main] INFO org.apache.solr.common.cloud.ConnectionManager – Waiting for
client to connect to ZooKeeper
2220 [main-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager –
Watcher org.apache.solr.common.cloud.ConnectionManager@36dadde6
name:ZooKeeperConnection Watcher:our_zk_host_1:2195 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
2220 [main] INFO org.apache.solr.common.cloud.ConnectionManager – Client is
connected to ZooKeeper
2293 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/queue
2447 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/collection-queue-work
2609 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/collection-map-running
2758 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/collection-map-completed
2919 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/collection-map-failure
3093 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/live_nodes
3173 [main] INFO org.apache.solr.cloud.ZkController – Register node as live
in ZooKeeper:/live_nodes/our_solr_host:8983_solr
3201 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/live_nodes/our_solr_host:8983_solr
3371 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer_elect
3477 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer_elect/election
3577 [main] INFO org.apache.solr.cloud.Overseer – Overseer (id=null) closing
3659 [main] INFO org.apache.solr.cloud.ElectionContext – I am going to be the
leader our_solr_host:8983_solr
3662 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer_elect/leader
3763 [main] INFO org.apache.solr.cloud.Overseer – Overseer
(id=93155855038349319-our_solr_host:8983_solr-n_0000000000) starting
3887 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/overseer/queue-work
4409 [main] INFO org.apache.solr.cloud.OverseerAutoReplicaFailoverThread –
Starting OverseerAutoReplicaFailoverThread
autoReplicaFailoverWorkLoopDelay=10000
autoReplicaFailoverWaitAfterExpiration=30000
autoReplicaFailoverBadNodeExpiration=60000
4438
[OverseerCollectionProcessor-93155855038349319-our_solr_host:8983_solr-n_0000000000]
INFO org.apache.solr.cloud.OverseerCollectionProcessor – Process current
queue of collection creations
4456 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
/clusterstate.json
4559 [main] INFO org.apache.solr.common.cloud.ZkStateReader – Updating
cluster state from ZooKeeper...
4731
[OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
INFO org.apache.solr.cloud.Overseer – Starting to work on the main queue
4736 [main] INFO org.apache.solr.core.CoresLocator – Looking for core
definitions underneath /site/pkgs/solr/solr-4.10.3/example/solr
4744 [main] INFO org.apache.solr.core.CoresLocator – Found core ttPoiMDS in
/site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/
4744 [main] INFO org.apache.solr.core.CoresLocator – Found 1 core definitions
4767 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
publishing core=ttPoiMDS state=down collection=ttPoiMDS
4767 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
numShards not found on descriptor - reading it from system property
4796 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
look for our core node name
4797 [zkCallback-2-thread-1] INFO org.apache.solr.cloud.DistributedQueue –
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
4897
[OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
INFO org.apache.solr.cloud.Overseer – Update state numShards=null message={
"operation":"state",
"shard":null,
"roles":null,
"state":"down",
"core":"ttPoiMDS",
"collection":"ttPoiMDS",
"node_name":"our_solr_host:8983_solr",
"base_url":"http://our_solr_host:8983/solr"}
4899
[OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
INFO org.apache.solr.cloud.Overseer – Assigning new node to shard shard=shard1
5052 [zkCallback-2-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader
– A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
5800 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
waiting to find shard id in clusterstate for ttPoiMDS
5801 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
Check for collection zkNode:ttPoiMDS
5924 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController –
Collection zkNode exists
5925 [coreLoadExecutor-6-thread-1] INFO
org.apache.solr.common.cloud.ZkStateReader – Load collection config
from:/collections/ttPoiMDS
5970 [coreLoadExecutor-6-thread-1] INFO
org.apache.solr.common.cloud.ZkStateReader – path=/collections/ttPoiMDS
configName=ttPoiMDS specified config exists in ZooKeeper
5970 [coreLoadExecutor-6-thread-1] INFO
org.apache.solr.core.SolrResourceLoader – new SolrResourceLoader for
directory: '/site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/'
6054 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.SolrConfig –
Adding specified lib dirs to ClassLoader
6055 [coreLoadExecutor-6-thread-1] WARN
org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to
add to classloader: ../../lib/mq (resolved as:
/site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/../../lib/mq).
6139 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.SolrConfig –
Using Lucene MatchVersion: 4.10.3
6238 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.Config – Loaded
SolrConfig: solrconfig.xml
6320 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.schema.IndexSchema –
Reading Solr Schema from /configs/ttPoiMDS/schema.xml
6347 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.schema.IndexSchema –
[ttPoiMDS] Schema name=poiMDS
6582 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
user.dir=/site/pkgs/solr/solr-4.10.3/example
6582 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
SolrDispatchFilter.init() done
6614 [main] INFO org.eclipse.jetty.server.AbstractConnector – Started
[email protected]:8983
{panel}
Startup then seems to hang. The Solr UI is available, but it shows that no
cores are available. In the cloud view the configuration from ZooKeeper is
present.
> On a new cluster setup, new nodes never move out of the 'down' state.
> ---------------------------------------------------------------------
>
> Key: SOLR-6995
> URL: https://issues.apache.org/jira/browse/SOLR-6995
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.10.3
> Reporter: Darren Spehr
> Priority: Blocker
>
> This is related to a question I posted on
> [stackoverflow|http://stackoverflow.com/questions/28004832/solr-4-10-3-is-not-proceeding-to-leader-election-on-new-cluster-startup-hangs]
> a day ago.
> When deploying a new cluster, new nodes never proceed to an 'active' state. I
> have used both a custom (yet simple) deployment, as well as using the example
> framework you get when downloading the Solr distribution. Further inspection
> of the logs, and comparing them to the output of a 4.10.1 start up, it looks
> like the nodes never move on to leader election.
> For an apples-to-apples comparison I moved the solr/collection1 directory to
> solr/ttPoiMDS, and modified the core.properties appropriately. I also removed
> the 'conf' directory, assuming that it would pick this up from our ZooKeeper
> cluster.
> Here is the start up:
> {code}
> java
> -Dcollection.configName=ttPoiMDS
> -Djava.util.logging.config.file=etc/logging.properties
> -DzkHost=our_zk_host_1:<zk_port>
> -Djetty.port=8983
> -jar start.jar
> {code}
> Here is the output in the logs:
> {code}
> 0 [main] INFO org.eclipse.jetty.server.Server – jetty-8.1.10.v20130312
> 32 [main] INFO org.eclipse.jetty.deploy.providers.ScanningAppProvider –
> Deployment monitor /site/pkgs/solr/solr-4.10.3/example/contexts at interval 0
> 38 [main] INFO org.eclipse.jetty.deploy.DeploymentManager – Deployable
> added: /site/pkgs/solr/solr-4.10.3/example/contexts/solr-jetty-context.xml
> 114 [main] INFO org.eclipse.jetty.webapp.WebInfConfiguration – Extract
> jar:file:/site/pkgs/solr/solr-4.10.3/example/webapps/solr.war!/ to
> /site/pkgs/solr/solr-4.10.3/example/solr-webapp/webapp
> 1527 [main] INFO org.eclipse.jetty.webapp.StandardDescriptorProcessor – NO
> JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
> 1597 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
> SolrDispatchFilter.init()
> 1616 [main] INFO org.apache.solr.core.SolrResourceLoader – JNDI not
> configured for solr (NoInitialContextEx)
> 1616 [main] INFO org.apache.solr.core.SolrResourceLoader – solr home
> defaulted to 'solr/' (could not find system property or JNDI)
> 1617 [main] INFO org.apache.solr.core.SolrResourceLoader – new
> SolrResourceLoader for directory: 'solr/'
> 1748 [main] INFO org.apache.solr.core.ConfigSolr – Loading container
> configuration from /site/pkgs/solr/solr-4.10.3/example/solr/solr.xml
> 1884 [main] INFO org.apache.solr.core.CoresLocator – Config-defined core
> root directory: /site/pkgs/solr/solr-4.10.3/example/solr
> 1890 [main] INFO org.apache.solr.core.CoreContainer – New CoreContainer
> 378496804
> 1890 [main] INFO org.apache.solr.core.CoreContainer – Loading cores into
> CoreContainer [instanceDir=solr/]
> 1905 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting socketTimeout to: 0
> 1905 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting urlScheme to: null
> 1909 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting connTimeout to: 0
> 1911 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting maxConnectionsPerHost to: 20
> 1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting corePoolSize to: 0
> 1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting maximumPoolSize to: 2147483647
> 1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting maxThreadIdleTime to: 5
> 1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting sizeOfQueue to: -1
> 1912 [main] INFO org.apache.solr.handler.component.HttpShardHandlerFactory
> – Setting fairnessPolicy to: false
> 2054 [main] INFO org.apache.solr.update.UpdateShardHandler – Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=0&connTimeout=0&retry=false
> 2056 [main] INFO org.apache.solr.logging.LogWatcher – SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
> 2057 [main] INFO org.apache.solr.logging.LogWatcher – Registering Log
> Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> 2058 [main] INFO org.apache.solr.core.CoreContainer – Host Name:
> 2059 [main] INFO org.apache.solr.core.ZkContainer – Zookeeper
> client=our_zk_host_1:2195
> 2140 [main] INFO org.apache.solr.common.cloud.ConnectionManager – Waiting
> for client to connect to ZooKeeper
> 2220 [main-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager
> – Watcher org.apache.solr.common.cloud.ConnectionManager@36dadde6
> name:ZooKeeperConnection Watcher:our_zk_host_1:2195 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> 2220 [main] INFO org.apache.solr.common.cloud.ConnectionManager – Client is
> connected to ZooKeeper
> 2293 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/queue
> 2447 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/collection-queue-work
> 2609 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/collection-map-running
> 2758 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/collection-map-completed
> 2919 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/collection-map-failure
> 3093 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /live_nodes
> 3173 [main] INFO org.apache.solr.cloud.ZkController – Register node as live
> in ZooKeeper:/live_nodes/our_solr_host:8983_solr
> 3201 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /live_nodes/our_solr_host:8983_solr
> 3371 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer_elect
> 3477 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer_elect/election
> 3577 [main] INFO org.apache.solr.cloud.Overseer – Overseer (id=null) closing
> 3659 [main] INFO org.apache.solr.cloud.ElectionContext – I am going to be
> the leader our_solr_host:8983_solr
> 3662 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer_elect/leader
> 3763 [main] INFO org.apache.solr.cloud.Overseer – Overseer
> (id=93155855038349319-our_solr_host:8983_solr-n_0000000000) starting
> 3887 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /overseer/queue-work
> 4409 [main] INFO org.apache.solr.cloud.OverseerAutoReplicaFailoverThread –
> Starting OverseerAutoReplicaFailoverThread
> autoReplicaFailoverWorkLoopDelay=10000
> autoReplicaFailoverWaitAfterExpiration=30000
> autoReplicaFailoverBadNodeExpiration=60000
> 4438
> [OverseerCollectionProcessor-93155855038349319-our_solr_host:8983_solr-n_0000000000]
> INFO org.apache.solr.cloud.OverseerCollectionProcessor – Process current
> queue of collection creations
> 4456 [main] INFO org.apache.solr.common.cloud.SolrZkClient – makePath:
> /clusterstate.json
> 4559 [main] INFO org.apache.solr.common.cloud.ZkStateReader – Updating
> cluster state from ZooKeeper...
> 4731
> [OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
> INFO org.apache.solr.cloud.Overseer – Starting to work on the main queue
> 4736 [main] INFO org.apache.solr.core.CoresLocator – Looking for core
> definitions underneath /site/pkgs/solr/solr-4.10.3/example/solr
> 4744 [main] INFO org.apache.solr.core.CoresLocator – Found core ttPoiMDS in
> /site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/
> 4744 [main] INFO org.apache.solr.core.CoresLocator – Found 1 core
> definitions
> 4767 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – publishing core=ttPoiMDS state=down collection=ttPoiMDS
> 4767 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – numShards not found on descriptor - reading it from system property
> 4796 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – look for our core node name
> 4797 [zkCallback-2-thread-1] INFO org.apache.solr.cloud.DistributedQueue –
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> 4897
> [OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
> INFO org.apache.solr.cloud.Overseer – Update state numShards=null message={
> "operation":"state",
> "shard":null,
> "roles":null,
> "state":"down",
> "core":"ttPoiMDS",
> "collection":"ttPoiMDS",
> "node_name":"our_solr_host:8983_solr",
> "base_url":"http://our_solr_host:8983/solr"}
> 4899
> [OverseerStateUpdate-93155855038349319-our_solr_host:8983_solr-n_0000000000]
> INFO org.apache.solr.cloud.Overseer – Assigning new node to shard
> shard=shard1
> 5052 [zkCallback-2-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader
> – A cluster state change: WatchedEvent state:SyncConnected
> type:NodeDataChanged path:/clusterstate.json, has occurred - updating...
> (live nodes size: 1)
> 5800 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – waiting to find shard id in clusterstate for ttPoiMDS
> 5801 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – Check for collection zkNode:ttPoiMDS
> 5924 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.cloud.ZkController
> – Collection zkNode exists
> 5925 [coreLoadExecutor-6-thread-1] INFO
> org.apache.solr.common.cloud.ZkStateReader – Load collection config
> from:/collections/ttPoiMDS
> 5970 [coreLoadExecutor-6-thread-1] INFO
> org.apache.solr.common.cloud.ZkStateReader – path=/collections/ttPoiMDS
> configName=ttPoiMDS specified config exists in ZooKeeper
> 5970 [coreLoadExecutor-6-thread-1] INFO
> org.apache.solr.core.SolrResourceLoader – new SolrResourceLoader for
> directory: '/site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/'
> 6054 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.SolrConfig –
> Adding specified lib dirs to ClassLoader
> 6055 [coreLoadExecutor-6-thread-1] WARN
> org.apache.solr.core.SolrResourceLoader – Can't find (or read) directory to
> add to classloader: ../../lib/mq (resolved as:
> /site/pkgs/solr/solr-4.10.3/example/solr/ttPoiMDS/../../lib/mq).
> 6139 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.SolrConfig –
> Using Lucene MatchVersion: 4.10.3
> 6238 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.core.Config –
> Loaded SolrConfig: solrconfig.xml
> 6320 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.schema.IndexSchema
> – Reading Solr Schema from /configs/ttPoiMDS/schema.xml
> 6347 [coreLoadExecutor-6-thread-1] INFO org.apache.solr.schema.IndexSchema
> – [ttPoiMDS] Schema name=poiMDS
> 6582 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
> user.dir=/site/pkgs/solr/solr-4.10.3/example
> 6582 [main] INFO org.apache.solr.servlet.SolrDispatchFilter –
> SolrDispatchFilter.init() done
> 6614 [main] INFO org.eclipse.jetty.server.AbstractConnector – Started
> [email protected]:8983
> {code}
> Startup then seems to hang. The Solr UI is available, but it shows that no
> cores are available. In the cloud view the configuration from ZooKeeper is
> present.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
