Coincidentally, I'm hacking at this very problem....

First, are you sure you're free memory calculation is OK? Why not
just use freeMemory? Perhaps also calling the gc if the avail isn't
enough. Although I confess I don't know the innards of the
interplay of getting the various memory amounts.....

The approach I've been using is to gather some data as I'm indexing
to decide whether to flush the indexwriter or not. That is, record the
size change that ramSizeInBytes() returns before I start to index
a document, record the amount after, and keep the worst
ratio around. This got easier when I subclassed IndexWriter and
overrode the add methods.

But it does require that you call into your writer before you start
adding fields to a document to record the start size......

Then I'm requiring that I have 2X the worst case I've seen for the
incoming document, and flushing (perhaps gc-ing) if I don't have
enough.

Mostly, this is to keep from having to experiment with each different
data set that we get to find the right MERGE & etc. factors to use,
I'm not actually entirely sure that this is giving me any measurable
performance gains.

And I think that this is "good enough". What it allows (as does your
approach) is letting the usual cases of much smaller than 20M+ files
to accumulate and flush reasonably efficiently, and not penalizing
my speed by, say, always keeping 250M free or some such. Again,
the critical thing is that you have to call into here *before* you index.

Curiously, I also got ratios of around 7X, so there's a lot going on.

Keep me posted if you come up with anything really cool!

Best
Erick


On 5/3/07, david m <[EMAIL PROTECTED]> wrote:

Our application includes an indexing server that writes to multiple
indexes in parallel (each thread writes to a single index). In order
to avoid an OutOfMemoryError, each request to index a document is
checked to see if the JVM has enough memory available to index the
document.

I know that IndexWriter.ramSizeInBytes() can be used to determine how
much memory was consumed at the conclusion of indexing a document, but
is there a way to know (or estimate) the peak memory consumed while
indexing a document?

For example, in a test set I have a 22 MB document where nearly every
"word" is unique. It has text like this:

'DestAddrType' bin: 00 0D
AttributeCustomerID 'Resources'
AttributeDNIS '7730'
AttributeUserData [295] 00 0E 00 00..
'DNIS_DATA' '323,000,TM,SDM1K5,AAR,,,'
'ENV_FLAG' 'P'
'T_APP_CODE' 'TM'
TelephoneLine' '8'
'C_CALL_DATE' '01/19/06'
'C_START_TIME' '145650'
'C_END_TIME' '145710'
AttributeCallType 2

and so on...

We are indexing a handful of fields for document meta-data - but they
are tiny compared to the body of the document. Eight of those fields are
stored (like a messageid, posteddate, typecode).

The body is indexed into a single field. Our Analyzer splits tokens
based on Character.isLetterOrDigit() and when in uppercase, indexes a
lowercase version of the term.

After indexing that single document ramSizeInBytes() returns 15.7 MB.
That seems ok to me.

But for this particular document I found (via trial and error) that
at -Xmx165m Lucene throws an OutOfMemoryError.

At -Xmx170m the it indexes successfully.

Just before calling addDoc() I see maximum available memory of: 160.5 MB

The 160.5 MB is from this calc:

  Runtime rt = Runtime.getRuntime();
  long maxAvail = rt.maxMemory() - (rt.totalMemory() - rt.freeMemory());

So it would appear that for this particular document, to avoid an
OutOfMemoryError I'd need to be certain of having available memory
approx 7x the doc size.

I could require 7x the doc size available memory for each doc (on
the assumption my test document is at the extreme), but for more
typical documents I'd be over-reserving memory with a result of
reduced throughput (as docs were forced to wait for sufficient
available memory that they likely don't need).

Instead I'm wondering if there is better way for the index server to
know (or guesstimate) what the memory requirement will be for each
document? - so that it doesn't start indexing in parallel more
documents than available memory can support.

Thanks,
david.

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


Reply via email to