I now think the main issue here is that a busy JVM gets into trouble
trying to find large free blocks of memory for large bitsets.
In my index of 64 million documents, ~8meg of contiguous free memory
must be found for each bitset allocated. The terms I was trying to cache
had 14 million entries so the new DocIdSet alternatives for bitsets
probably fare no better.
The JVM (Sun 1.5) doesn't seem to deal with these allocations well.
Perhaps there's an obscure JVM option I can set to reserve a section of
RAM for large allocations.
However, I wonder if we should help the JVM out a little here by having
pre-allocated pools of BitsSets/OpenBitSets that can be reserved and
reused by the application. This would imply a change to filter classes
so instead of constructing BitSets/OpenBitsets directly they get them
from a pool instead.
Thoughts?
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
