Hi,
I was trying to configure a Solr instance with the near real-time search
and auto-complete capabilities. I stuck in the NRT feature. There are
15 new records per second that inserted into the database (mysql) and I
indexed them with DIH. First, I tried to manage autoCommits from
solrconfig.xml with the configuration below.
<autoCommit>
<maxDocs>10000</maxDocs>
<maxTime>100000</maxTime>
</autoCommit>
<autoSoftCommit>
<maxDocs>15</maxDocs>
<maxTime>1000</maxTime>
</autoSoftCommit>
And the bash script below responsible for getting delta's without
committing.
while [ 1 ]; do
wget -O /dev/null '
http://localhost:8080/solr-jak/dataimport?command=delta-import&commit=false'
2>/dev/null
sleep 1
done
Then I run my query from browser
http://localhost:8080/solr-jak/select?q=movie_name_prefix_full:"dogville"&defType=lucene&q.op=OR<http://localhost:8080/solr-sprongo/select?q=movie_name_prefix_full:%221398%22&defType=lucene&q.op=OR>
But I realized that, with this configuration index files are changing every
second and after a minute there are only 600 new records in Solr index
while 900 new records in the database.
After experienced that, I removed autoCommit and autoSoftCommit elements in
solrconfig.xml And updated my bashscript as follows. But still index files
are changing and solr can not syncronized with database.
while [ 1 ]; do
echo "Soft commit applied!"
wget -O /dev/null '
http://localhost:8080/solr-jak/dataimport?command=delta-import&commit=false'
2>/dev/null
curl http://localhost:8080/solr-jak/update -H "Content-Type: text/xml"
--data-binary '<commit softCommit="true" waitFlush="false"
waitSearcher="false"/>' 2>/dev/null
sleep 3
done
Even I decreased the pressure on Solr as 1 new record per sec. and soft
commits within 6 sec. still there is a gap between index and db. Is there
anything that I missed? I took a look to "/get" too, but it is working only
for pk. If there is an example configuration list (like 1 sec for soft
commit and 10 min for hard commit) as a best practice it would be great.
Finally, here is my configuration.
Ubuntu 11.04
JDK 1.6.0_27
Tomcat 7.0.21
Solr 4.0 2011-10-24_08-53-02
All advices are appreciated,
Best Regards,
Jak
