With DocumentsWriterPerThread we can allow 2GB per thread, so that
should be a good step forward.
For realtime indexing on the RAM buffer I'm planning to remove even that
per-thread limit, because then you really want to make use of all the
RAM you have available on your machine.
Michael
On 4/26/10 7:24 AM, Shai Erera wrote:
Good point. I'll set the heap size larger.
It will be interesting to measure if there is a noticeable improvement
from 2GB RAM buffer size vs say 100MB or 512MB ... I'll report the
results here when I have them
But in the meantime, I think, unless it makes absolutely no sense and
does not improve anything, that we should not limit to 2GB just
because we use an int somewhere.
Shai
On Mon, Apr 26, 2010 at 5:18 PM, Mark Miller <markrmil...@gmail.com
<mailto:markrmil...@gmail.com>> wrote:
On 4/26/10 10:06 AM, Shai Erera wrote:
Hi
Today we limit the RAM buffer size to 2048. I was wondering
why, until
Mark referenced me to that issue
(https://issues.apache.org/jira/browse/LUCENE-1995) where someone
complained about it as well, but I didn't find a reason to why
it's
limited. I mean, I understand it's probably an int is used
somewhere,
but all the issue says here
(https://issues.apache.org/jira/browse/LUCENE-1995?focusedCommentId=12767757&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12767757
<https://issues.apache.org/jira/browse/LUCENE-1995?focusedCommentId=12767757&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12767757>
<https://issues.apache.org/jira/browse/LUCENE-1995?focusedCommentId=12767757&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12767757
<https://issues.apache.org/jira/browse/LUCENE-1995?focusedCommentId=12767757&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12767757>>)
is "Maybe someday Lucene will allow a larger RAM buffer than 2GB".
Why "maybe"? :)
I set up a benchmark run on a new machine I got, 4xQuad, 64GB
RAM, and
turns out I can't really take advantage of all that RAM ... :).
So I don't mind to look at the code, but I guess since Mike
and a bunch
of other smarter people were on that issue, there is probably
something
I need to know before I set to change it?
Appreciate your thoughts on that. Today, more and more
machines come w/
more than 2GB ...
Shai
Keep in mind that you need a lot more than 2GB of ram to use a
really large RAM buffer:
http://search.lucidimagination.com/search/document/b18fbeaaf53ead6c/io_exception_during_merge_optimize
64 GB might cover it though ;)
--
- Mark
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
<mailto:dev-unsubscr...@lucene.apache.org>
For additional commands, e-mail: dev-h...@lucene.apache.org
<mailto:dev-h...@lucene.apache.org>
