On 2/23/07, Brian Whitman <[EMAIL PROTECTED]> wrote:
>
> Try not committing so often (perhaps until you are done).
> Don't use post.sh, or modify it to remove the commit.
>
OK, I modified it to not commit after and I also realized I had
SOLR-126 (autocommit) on, which I disabled. Is there a rule of thumb
on when to commit / optimize?
There is a map entry (UniqueKey->Integer) per document added/deleted,
and that's really the only in-memory state that is kept. So you
should be good with at least >100K docs.
If either autoCommit is on, or I commit after every add, the number
of open file descriptors from Lucene goes up way high and does not go
back down.
Do you have any warming configured?
Too many searchers trying to initialize fieldcache entries can blow
out the memory, causing most of the CPU to be consumed by the garbage
collector.
I just ran
# lsof | grep solr | wc -l
after adding 1000 docs and got 125265 open fdescs. If I stop adding
docs this count does not go down
Is there detectable activity going on (like CPU usage)?
Does the admin page list all of these open searchers (check the
statistics page under "CORE")
-- it does not go down until I
restart solr. This would be the cause of my too many files open
problem. Turning off autocommit / not commiting after every add keeps
this count steady at 100-200. The files are all of type:
[...]
Bug or feature?
If the searchers holding these index files open are still working,
then this is a problem, but not exactly a bug. If not, you may have
hit a new bug in searcher synchronization.
A workaround is to limit the number of warming searchers (see
maxWarmingSearchers in solrconfig.xml)
-Yonik
