Re: documents are not commited distributively in solr cloud tomcat with core discovery, range is null for shards in clusterstate.json
I've solved this problem myself.
If you use core discovery, you must specify the numShards parameter in
core.properties.
or else solr won't be allocate range for each shards and then documents
won't be distributed properly.
Using core discovery to set up solr cloud in tomcat is much easier and
clean than coreAdmin described in the wiki:
http://wiki.apache.org/solr/SolrCloudTomcat.
It costs me some time to move from jetty to tomcat, but I think our IT team
will like this way. :)
On 6 October 2013 23:53, Liu Bo diabl...@gmail.com wrote:
Hi all
I've sent out this mail before, but I only subscribed to lucene-user but
not solr-user at that time. Sorry for repeating if any and your help will
be much of my appreciation.
I'm trying out the tutorial about solrcloud, and then I manage to write my
own plugin to import data from our set of databases, I use SolrWriter from
DataImporter package and the docs could be distributed commit to shards.
Every thing works fine using jetty from the solr example, but when I move
to tomcat, solrcloud seems not been configured right. As the documents are
just committed to the shard where update requested goes to.
The cause probably is the range is null for shards in clusterstate.json.
The router is implicit instead of compositeId as well.
Is there anything missed or configured wrong in the following steps? How
could I fix it. Your help will be much of my appreciation.
PS, solr cloud tomcat wiki page isn't up to 4.4 with core discovery, I'm
trying out after reading SoclrCloud, SolrCloudJboss, and CoreAdmin wiki
pages.
Here's what I've done and some useful logs:
1. start three zookeeper server.
2. upload configuration files to zookeeper, the collection name is
content_collection
3. start three tomcat instants on three server with core discovery
a) core file:
name=content
loadOnStartup=true
transient=false
shard=shard1 (differrent on servers)
collection=content_collection
b) solr.xml
solr
solrcloud
str name=host${host:}/str
str name=hostContext${hostContext:solr}/str
int name=hostPort8080/int
int name=zkClientTimeout${zkClientTimeout:15000}/int
str name=zkHost10.199.46.176:2181,10.199.46.165:2181,
10.199.46.158:2181/str
bool name=genericCoreNodeNames${genericCoreNodeNames:true}/bool
/solrcloud
shardHandlerFactory name=shardHandlerFactory
class=HttpShardHandlerFactory
int name=socketTimeout${socketTimeout:0}/int
int name=connTimeout${connTimeout:0}/int
/shardHandlerFactory
/solr
4. In the solr.log, I see the three shards are recognized, and the
solrcloud can see the content_collection has three shards as well.
5. write documents to content_collection using my update request, the
documents only commits to the shard the request goes to, in the log I can
see the DistributedUpdateProcessorFactory is in the processorChain and
disribute commit is triggered:
INFO - 2013-09-30 16:31:43.205;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
updata request processor factories:
INFO - 2013-09-30 16:31:43.206;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.LogUpdateProcessorFactory@4ae7b77
INFO - 2013-09-30 16:31:43.207;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.*DistributedUpdateProcessorFactory*
@5b2bc407
INFO - 2013-09-30 16:31:43.207;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.RunUpdateProcessorFactory@1652d654
INFO - 2013-09-30 16:31:43.283; org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onInit: commits: num=1
commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
INFO - 2013-09-30 16:31:43.284; org.apache.solr.core.SolrDeletionPolicy;
newest commit generation = 1
INFO - 2013-09-30 16:31:43.440; *org.apache.solr.update.SolrCmdDistributor;
Distrib commit to*:[StdNode: http://10.199.46.176:8080/solr/content/,
StdNode: http://10.199.46.165:8080/solr/content/]
params:commit_end_point=truecommit=truesoftCommit=falsewaitSearcher=trueexpungeDeletes=false
but the documents won't go to other shards, the other shards only has a
request with not documents:
INFO - 2013-09-30 16:31:43.841;
org.apache.solr.update.DirectUpdateHandler2; start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onInit: commits: num=1
commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
newest commit
documents are not commited distributively in solr cloud tomcat with core discovery, range is null for shards in clusterstate.json
Hi all
I've sent out this mail before, but I only subscribed to lucene-user but
not solr-user at that time. Sorry for repeating if any and your help will
be much of my appreciation.
I'm trying out the tutorial about solrcloud, and then I manage to write my
own plugin to import data from our set of databases, I use SolrWriter from
DataImporter package and the docs could be distributed commit to shards.
Every thing works fine using jetty from the solr example, but when I move
to tomcat, solrcloud seems not been configured right. As the documents are
just committed to the shard where update requested goes to.
The cause probably is the range is null for shards in clusterstate.json.
The router is implicit instead of compositeId as well.
Is there anything missed or configured wrong in the following steps? How
could I fix it. Your help will be much of my appreciation.
PS, solr cloud tomcat wiki page isn't up to 4.4 with core discovery, I'm
trying out after reading SoclrCloud, SolrCloudJboss, and CoreAdmin wiki
pages.
Here's what I've done and some useful logs:
1. start three zookeeper server.
2. upload configuration files to zookeeper, the collection name is
content_collection
3. start three tomcat instants on three server with core discovery
a) core file:
name=content
loadOnStartup=true
transient=false
shard=shard1 (differrent on servers)
collection=content_collection
b) solr.xml
solr
solrcloud
str name=host${host:}/str
str name=hostContext${hostContext:solr}/str
int name=hostPort8080/int
int name=zkClientTimeout${zkClientTimeout:15000}/int
str name=zkHost10.199.46.176:2181,10.199.46.165:2181,
10.199.46.158:2181/str
bool name=genericCoreNodeNames${genericCoreNodeNames:true}/bool
/solrcloud
shardHandlerFactory name=shardHandlerFactory
class=HttpShardHandlerFactory
int name=socketTimeout${socketTimeout:0}/int
int name=connTimeout${connTimeout:0}/int
/shardHandlerFactory
/solr
4. In the solr.log, I see the three shards are recognized, and the
solrcloud can see the content_collection has three shards as well.
5. write documents to content_collection using my update request, the
documents only commits to the shard the request goes to, in the log I can
see the DistributedUpdateProcessorFactory is in the processorChain and
disribute commit is triggered:
INFO - 2013-09-30 16:31:43.205;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
updata request processor factories:
INFO - 2013-09-30 16:31:43.206;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.LogUpdateProcessorFactory@4ae7b77
INFO - 2013-09-30 16:31:43.207;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.*DistributedUpdateProcessorFactory*
@5b2bc407
INFO - 2013-09-30 16:31:43.207;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.RunUpdateProcessorFactory@1652d654
INFO - 2013-09-30 16:31:43.283; org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onInit: commits: num=1
commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
INFO - 2013-09-30 16:31:43.284; org.apache.solr.core.SolrDeletionPolicy;
newest commit generation = 1
INFO - 2013-09-30 16:31:43.440; *org.apache.solr.update.SolrCmdDistributor;
Distrib commit to*:[StdNode: http://10.199.46.176:8080/solr/content/,
StdNode: http://10.199.46.165:8080/solr/content/]
params:commit_end_point=truecommit=truesoftCommit=falsewaitSearcher=trueexpungeDeletes=false
but the documents won't go to other shards, the other shards only has a
request with not documents:
INFO - 2013-09-30 16:31:43.841;
org.apache.solr.update.DirectUpdateHandler2; start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onInit: commits: num=1
commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
newest commit generation = 1
INFO - 2013-09-30 16:31:43.856;
org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes.
Skipping IW.commit.
INFO - 2013-09-30 16:31:43.865; org.apache.solr.search.SolrIndexSearcher;
Opening Searcher@3c74c144 main
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener;
QuerySenderListener sending requests to
Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)}
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener;
QuerySenderListener done.
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.SolrCore; [content]
Registered new searcher
documents are not commited distributively in solr cloud tomcat with core discovery, range is null for shards in clusterstate.json
Hi all
I'm trying out the tutorial about solrcloud, and then I manage to write my
own plugin to import data from our set of databases, I use SolrWriter from
DataImporter package and the docs could be distributed commit to shards.
Every thing works fine using jetty from the solr example, but when I move
to tomcat, solrcloud seems not been configured right. As the documents are
just committed to the shard where update requested goes to.
The cause probably is the range is null for shards in clusterstate.json.
The router is implicit instead of compositeId as well.
Is there anything missed or configured wrong in the following steps? How
could I fix it. Your help will be much of my appreciation.
PS, solr cloud tomcat wiki page isn't up to 4.4 with core discovery, I'm
trying out after reading SoclrCloud, SolrCloudJboss, and CoreAdmin wiki
pages.
Here's what I've done and some useful logs:
1. start three zookeeper server.
2. upload configuration files to zookeeper, the collection name is
content_collection
3. start three tomcat instants on three server with core discovery
a) core file:
name=content
loadOnStartup=true
transient=false
shard=shard1 (differrent on servers)
collection=content_collection
b) solr.xml
solr
solrcloud
str name=host${host:}/str
str name=hostContext${hostContext:solr}/str
int name=hostPort8080/int
int name=zkClientTimeout${zkClientTimeout:15000}/int
str name=zkHost10.199.46.176:2181,10.199.46.165:2181,
10.199.46.158:2181/str
bool name=genericCoreNodeNames${genericCoreNodeNames:true}/bool
/solrcloud
shardHandlerFactory name=shardHandlerFactory
class=HttpShardHandlerFactory
int name=socketTimeout${socketTimeout:0}/int
int name=connTimeout${connTimeout:0}/int
/shardHandlerFactory
/solr
4. In the solr.log, I see the three shards are recognized, and the
solrcloud can see the content_collection has three shards as well.
5. write documents to content_collection using my update request, the
documents only commits to the shard the request goes to, in the log I can
see the DistributedUpdateProcessorFactory is in the processorChain and
disribute commit is triggered:
INFO - 2013-09-30 16:31:43.205;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
updata request processor factories:
INFO - 2013-09-30 16:31:43.206;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.LogUpdateProcessorFactory@4ae7b77
INFO - 2013-09-30 16:31:43.207;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.*DistributedUpdateProcessorFactory*
@5b2bc407
INFO - 2013-09-30 16:31:43.207;
com.microstrategy.alert.search.solr.plugin.index.handler.IndexRequestHandler;
org.apache.solr.update.processor.RunUpdateProcessorFactory@1652d654
INFO - 2013-09-30 16:31:43.283; org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onInit: commits: num=1
commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
INFO - 2013-09-30 16:31:43.284; org.apache.solr.core.SolrDeletionPolicy;
newest commit generation = 1
INFO - 2013-09-30 16:31:43.440; *org.apache.solr.update.SolrCmdDistributor;
Distrib commit to*:[StdNode: http://10.199.46.176:8080/solr/content/,
StdNode: http://10.199.46.165:8080/solr/content/]
params:commit_end_point=truecommit=truesoftCommit=falsewaitSearcher=trueexpungeDeletes=false
but the documents won't go to other shards, the other shards only has a
request with not documents:
INFO - 2013-09-30 16:31:43.841;
org.apache.solr.update.DirectUpdateHandler2; start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onInit: commits: num=1
commit{dir=/home/bold/work/tomcat/solr/content/data/index,segFN=segments_1,generation=1}
INFO - 2013-09-30 16:31:43.855; org.apache.solr.core.SolrDeletionPolicy;
newest commit generation = 1
INFO - 2013-09-30 16:31:43.856;
org.apache.solr.update.DirectUpdateHandler2; No uncommitted changes.
Skipping IW.commit.
INFO - 2013-09-30 16:31:43.865; org.apache.solr.search.SolrIndexSearcher;
Opening Searcher@3c74c144 main
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener;
QuerySenderListener sending requests to
Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)}
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.QuerySenderListener;
QuerySenderListener done.
INFO - 2013-09-30 16:31:43.869; org.apache.solr.core.SolrCore; [content]
Registered new searcher
Searcher@3c74c144main{StandardDirectoryReader(segments_1:1:nrt)}
INFO - 2013-09-30 16:31:43.870;
org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
INFO - 2013-09-30
