On 1/15/2013 10:42 AM, Tom Burton-West wrote:
We have been using Solr 4.0 for a while and suddenly we couldn't get
Solr to come up. As Solr was starting up it hung after opening a
Searcher. There wasn't anything else obvious in the logs. Eventually
we realized that the problem was that the updatelog was being read and
that the update log contained the entire text of all 800,000+ books that
we indexed (About 837GB).
Tom,
I ran into really huge update logs on my own foray into Solr 4.x. This
is probably not news to you, as you linked the email thread where I
mentioned it. I am now keeping the update log from getting huge during
full-import by configuring autoCommit to happen frequently. On my
production 3.5 servers autoCommit is turned off. This also seems to
have lessened the performance impact I discussed in that email thread.
The full extent of the performance impact will be unknown until I can
put the new version on the beefy production hardware. Tests suggest
that on my dev hardware, it's not much slower than an import without the
updateLog.
I am considering setting openSearcher to false in my autoCommit config
to speed things up further, but I need to be sure that this won't affect
what happens when I issue an explicit hard commit. For the moment, I do
want a new searcher on explicit hard commit. After I get everything
upgraded and stable, I will investigate using soft commits. Currently I
can use the exact same program (with solrj 4.1) to update both 3.5 and
4.x indexes. If I switch to soft commits before I have eliminated 3.5,
I will have to maintain two versions.
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
