That's what it looked like to me, too. I wonder if it would be worth
improving the estimate for some very common Collections classes? I see
this comment eg in BaseIndexFileFormatTestCase:
// we have no way to estimate the size of these things in codecs although
// something like a Collections.newSetFromMap(new HashMap<>()) uses quite
// some memory... So for now the test ignores the overhead of such
// collections but can we do better?
This is in testRamBytesUsed and there is a kind of fudge factor in
there for handling mismeasurement errors of the sort we are talking
about. Actually the test seems to be more about validating the
RamUsageTester than about validating the accounting in SegmentReader!
There are lots of other usages in tests, but I suppose they don't
require very precise handling of Collections classes (since they
pass)? Anyway it is certainly possible to improve the estimate quite a
bit and pretty easily for HashMap by simply counting the size of the
Node that is used for each entry, although given the dynamic nature of
these data structures (HashMap eg can use TreeNodes sometimes
depending on data distribution) it would be almost impossible to be
100% accurate.
On Thu, Dec 6, 2018 at 7:14 AM Dawid Weiss <dawid.we...@gmail.com> wrote:
>
> > It's entirely possible it fails to dig into Maps correctly with newer Java
> > releases; maybe Dawid or Uwe would know?
>
> We have removed all reflection from that class a while ago exactly
> because of encapsulation issues introduced in newer Java versions.
>
> https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java
>
> I think you may be thinking of RamUsageTester which is in the test
> framework and indeed accumulates only keys and values from iterables.
> These methods are for tests only and are rough. You shouldn't rely on
> them for accurate memory consumption accounting (instead, use the
> Accountable interface).
>
> Dawid
>
>
> D.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
