On Thu, May 31, 2012 at 5:16 PM, Dawid Weiss
<dawid.we...@cs.put.poznan.pl> wrote:
>> This test intentionally allocates ~256 MB packed ints ... the seed
>> doesn't fail in isolation, but I think the test fails if it's run with
>> other tests that leave too much uncollectible stuff allocated in the
>> heap ...
>
> It doesn't need to be hard refs. With parallel garbage collectors
> (with various staged memory pools) and fast allocation rate a thread
> may fail with an OOM even if there is theoretically enough space for a
> new allocated block. Running with SerialGC typically fixes the problem
> but then -- this isn't realistic :)
Got it.
>> Can we somehow mark that a test should be run in isolation (it's own
>> new JVM)...?
>
> Technically this is possible I think (can't tell how large refactoring
> it woudl be). But something in me objects to this idea. On the one
> hand this is ideal test isolation; on the other hand I bet with time
> all tests would just require a forked VM "because it's simpler this
> way". Good tests should clean up after themselves. I'm idealistic but
> I believe tests should be fixed if they don't follow this rule.
Yeah I hear you... hmm do we forcefully clear the FieldCache after
tests...? Though, in theory once the AtomicReader is collectible the
FC's entries should be too...
>> Another option ... would be to ignore the OOME ... but the risk there
>> is we suppress a "real" OOME from a sudden bug in the packed ints.
>> Though it's unlikely such a breakage would escape our usages of packed
>> ints... so maybe it's fine.
>
> How close are we to the memory limit if run in isolation (as a
> stand-alone test case)? We can probably measure this by allocating a
> byte[] before the test and doing binary search on its size depending
> on if it OOMs or not? Maybe it's just really close to the memory
> limit?
OK I did that: if I alloc 68 MB byte[] up front we OOME, but 67 MB
byte[] and the test passes (run in isolation).
That's closer than I expected: the max long[] we alloc in the test is
273 MB. So 512 - 273 - 68 = 171 MB unexplained.... hmm I think this
is because large arrays are alloc'd directly from the old generation:
http://stackoverflow.com/questions/9738911/javas-serial-garbage-collector-performing-far-better-than-other-garbage-collect
When I run with -XX:NewRatio=10 then I can pre-alloc 191 MB byte[] and
the test still passes ...
I think the best option is to ignore the OOME from this test case...?
Mike McCandless
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
