Hi,
I'm experiencing some odd behaviour with Solrcloud and Zookeeper. I am running
Solrcloud on one host and am running three Zookeepers on another three hosts.
The Zookeeper part of things works correctly, I can add/remove/etc nodes from
Zookeeper. I am running, or rather trying to run, Solrcloud on top of Hadoop.
Again, the Hadoop side of things works correctly, I can create/remove/etc
dirs/files under Hadoop.
Unfortunately, the solrctl utility bundled with Solrcloud doesn't appear to
work correctly. Depending on how or where I set the Zookeeper ensemble details
I get different results. My Zookeeper instances are used by other services, so
I am trying to force the Solrcloud configuration to be created under /solr -
from reading the documentation this appears to be the recommended appraoch.
I have set the Zookeeper ensemble and Hadoop configuration in /etc/default/solr:
SOLR_ZK_ENSEMBLE=zookeeper1:2181/solr,zookeeper2:2181/solr,zookeeper3:2181/solr
SOLR_HDFS_HOME=hdfs://zookeeper1:8020/solr
SOLR_HDFS_CONFIG=/etc/hadoop/conf
SOLR_HDFS_HOME=hdfs://3xNodeHA:8020/solr
If I do not specify any Zookeeper parameters for solrctl it creates it
Zookeeper configuration under '/solr,zookeeper2:2181' and under that is creates
'/solr,zookeeper3:2181/solr/configs/my-data'. This also occurs if I specify
--zk zookeeper1:2181/solr,zookeeper2:2181/solr,zookeeper3:2181/solr. I suspect
that something somewhere is not treating the SOLR_ZK_ENSEMBLE variable
correctly and believes it is a single connection (eg zookeeper1:2181) and the
path is /solr,zookeeper2:2181,zookeeper3:2181/solr.
If I run solrctl with --zk zookeeper1:2181, it creates its configuration under
/ (eg /solr.xml /configs).
If I run solrctl with --zk zookeeper1:2181/solr, it creates the configuration
under /solr
If I completely ignore the Zookeeper configuration Solr works correctly, but as
I'm using Lily I need Solr's configuration to exist under Zookeeper.
What am I missing? How can I specify a multi-node Zookeeper ensemble and have
all of the configuration nodes created under /solr? How do I point Tomcat
towards the Solr configuration under /solr?
If you would like more details, please look at the attachment as this explains
what I did at each step and the results of that step.
I'm using Cloudera's packages throughout.
thanks
Rob
Registered name: In Practice Systems Ltd.
Registered address: The Bread Factory, 1a Broughton Street, London, SW8 3QJ
Registered Number: 1788577
Registered in England
Visit our Internet Web site at www.inps.co.uk
The information in this internet email is confidential and is intended solely
for the addressee. Access, copying or re-use of information in it by anyone
else is not authorised. Any views or opinions presented are solely those of the
author and do not necessarily represent those of INPS or any of its affiliates.
If you are not the intended recipient please contact [email protected]
The following has been added to /etc/default/solr
SOLR_ZK_ENSEMBLE=zookeeper1:2181/solr,zookeeper2:2181/solr,zookeeper3:2181/solr
SOLR_HDFS_HOME=hdfs://zookeeper1:8020/solr
SOLR_HDFS_CONFIG=/etc/hadoop/conf
SOLR_HDFS_HOME=hdfs://3xNodeHA:8020/solr
-- Setup HDFS /solr directory
su hdfs -c 'hadoop fs -mkdir /solr'
su hdfs -c 'hadoop fs -chown solr:solr /solr'
-- Initialise SOLR
solrctl init --force
[zk: zookeeper1:2181(CONNECTED) 14] ls /
[zookeeper]
-- Start solr-server
service solr-server start
Tomcat logs the following error:
2014-11-19 11:47:39,486 | ERROR | null:org.apache.solr.common.SolrException:
solr.xml not found in ZooKeeper
This is expected as Zookeeper does not have a /solr node
[zk: zookeeper1:2181(CONNECTED) 15] ls /
[zookeeper]
-- Generate configuration
solrctl instancedir --generate /tmp/solr.njtO
-- Create instancedir collection
solrctl instancedir --create my-data /tmp/solr.njtO
[zk: zookeeper1:2181(CONNECTED) 16] ls /
[zookeeper, solr,zookeeper2:2181]
[zk: zookeeper1:2181(CONNECTED) 17] ls
/solr,zookeeper2:2181/solr,zookeeper3:2181/solr/configs/my-data
[admin-extra.menu-top.html, currency.xml, protwords.txt,
mapping-FoldToASCII.txt, solrconfig.xml.secure, solrconfig.xml, lang,
stopwords.txt, spellings.txt, README.1st, mapping-ISOLatin1Accent.txt,
admin-extra.html, xslt, synonyms.txt, scripts.conf, update-script.js, velocity,
elevate.xml, admin-extra.menu-bottom.html, schema.xml]
-- Create collection
solrctl collection --create my-data -s 1 -c my-data
Tomcat returns 503, probably due to the missing solr.xml
If I specify --zk zookeeper1:2181,zookeeper2:2181,zookeeper:2181
-- Setup HDFS /solr directory
su hdfs -c 'hadoop fs -mkdir /solr'
su hdfs -c 'hadoop fs -chown solr:solr /solr'
-- Initialise SOLR
solrctl --zk zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 init --force
[zk: zookeeper1:2181(CONNECTED) 1] ls /
[zookeeper, solr.xml]
-- Start solr-server
service solr-server start
ERROR | null:org.apache.solr.common.SolrException: solr.xml not found in
ZooKeeper
-- Generate configuration
solrctl --zk zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 instancedir
--generate /tmp/solr.QbhC
-- Create instancedir collection
solrctl --zk zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 instancedir
--create my-data /tmp/solr.QbhC
[zk: zookeeper1:2181(CONNECTED) 11] ls /
[configs, zookeeper, solr.xml]
-- Create collection
solrctl --zk zookeeper1:2181,zookeeper2:2181,zookeeper3:2181 collection
--create my-data -s 1 -c my-data
Tomcat returns 503, probably due to the missing solr.xml
If I specify --zk zookeeper1:2181/solr,zookeeper2:2181/solr,zookeeper:2181/solr
-- Setup HDFS /solr directory
su hdfs -c 'hadoop fs -mkdir /solr'
su hdfs -c 'hadoop fs -chown solr:solr /solr'
-- Initialise SOLR
solrctl --zk zookeeper1:2181/solr,zookeeper2:2181/solr,zookeeper3:2181/solr
init --force
[zk: zookeeper1:2181(CONNECTED) 0] ls /
[zookeeper]
-- Start solr-server
service solr-server start
ERROR | null:org.apache.solr.common.SolrException: solr.xml not found in
ZooKeeper
-- Generate configuration
solrctl --zk zookeeper1:2181/solr,zookeeper2:2181/solr,zookeeper3:2181/solr
instancedir --generate /tmp/solr.od0H
-- Create instance collection collection
solrctl --zk zookeeper1:2181/solr,zookeeper2:2181/solr,zookeeper3:2181/solr
instancedir --create my-data /tmp/solr.od0H
[zk: zookeeper1:2181(CONNECTED) 1] ls /
[zookeeper, solr,zookeeper2:2181]
-- Create collection
solrctl --zk zookeeper1:2181/solr,zookeeper2:2181/solr,zookeeper3:2181/solr
collection --create my-data -s 1 -c my-data
Tomcat returns 503, probably due to the missing solr.xml
If I specify --zk zookeeper1:2181
-- Setup HDFS /solr directory
su hdfs -c 'hadoop fs -mkdir /solr'
su hdfs -c 'hadoop fs -chown solr:solr /solr'
-- Initialise SOLR
solrctl --zk zookeeper1:2181 init --force
[zk: zookeeper1:2181(CONNECTED) 0] ls /
[zookeeper, solr.xml]
-- Start solr-server
service solr-server start
ERROR | null:org.apache.solr.common.SolrException: solr.xml not found in
ZooKeeper
-- Generate configuration
solrctl --zk zookeeper1:2181 instancedir --generate /tmp/solr.UHMZ
-- Create instance collection collection
solrctl --zk zookeeper1:2181 instancedir --create my-data /tmp/solr.UHMZ
[zk: zookeeper1:2181(CONNECTED) 1] ls /
[configs, zookeeper, solr.xml]
-- Create collection
solrctl --zk zookeeper1:2181 collection --create my-data -s 1 -c my-data
Tomcat returns 503, probably due to the missing solr.xml
If I specify --zk zookeeper1:2181/solr
-- Setup HDFS /solr directory
su hdfs -c 'hadoop fs -mkdir /solr'
su hdfs -c 'hadoop fs -chown solr:solr /solr'
-- Initialise SOLR
solrctl --zk hbase1:2181/solr init --force
[zk: zookeeper1:2181(CONNECTED) 0] ls /
[zookeeper, solr]
[zk: zookeeper1:2181(CONNECTED) 1] ls /solr
[solr.xml]
-- Start solr-server
service solr-server start
ERROR | null:org.apache.solr.common.SolrException: solr.xml not found in
ZooKeeper
-- Generate configuration
solrctl --zk hbase1:2181/solr instancedir --generate /tmp/solr.fHQD
-- Create instance collection collection
solrctl --zk hbase1:2181/solr instancedir --create my-data /tmp/solr.fHQD
[zk: zookeeper1:2181(CONNECTED) 2] ls /solr
[configs, solr.xml]
[zk: zookeeper1:2181(CONNECTED) 3] ls /solr/configs
[my-data]
[zk: zookeeper1:2181(CONNECTED) 4] ls /solr/configs/my-data
[admin-extra.menu-top.html, currency.xml, protwords.txt,
mapping-FoldToASCII.txt, solrconfig.xml.secure, solrconfig.xml, lang,
stopwords.txt, spellings.txt, README.1st, mapping-ISOLatin1Accent.txt,
admin-extra.html, xslt, synonyms.txt, scripts.conf, update-script.js, velocity,
elevate.xml, admin-extra.menu-bottom.html, schema.xml]
-- Create collection
solrctl --zk hbase1:2181/solr collection --create my-data -s 1 -c my-data
Tomcat returns 503, probably due to the missing solr.xml
