In this second heap dump I drilled down only one of the SegmentReaders (of the
StandardDirectoryReader). Looks like StandardSegmentReader has 24 subReaders
(SegmentReaders)
Class Name | Shallow Heap |
Retained Heap | Percentage
---------------------------------------------------------------------------------------------------------
org.apache.lucene.index.StandardDirectoryReader @ 0x783932460| 72 |
59'255'872 | 3.04%
|- org.apache.lucene.index.SegmentReader[24] @ 0x794089ee0 | 112 |
59'190'960 | 3.03%
| |- org.apache.lucene.index.SegmentReader @ 0x788820f40 | 72 |
16'905'072 | 0.87% <- this is the one I "drilled down"
| |- org.apache.lucene.index.SegmentReader @ 0x783555a20 | 72 |
5'845'840 | 0.30%
| |- org.apache.lucene.index.SegmentReader @ 0x788cea030 | 72 |
5'399'656 | 0.28%
| |- org.apache.lucene.index.SegmentReader @ 0x7839e2bd8 | 72 |
4'256'552 | 0.22%
| |- org.apache.lucene.index.SegmentReader @ 0x77f281b98 | 72 |
4'116'944 | 0.21%
| |- org.apache.lucene.index.SegmentReader @ 0x7888212a0 | 72 |
3'887'368 | 0.20%
| |- org.apache.lucene.index.SegmentReader @ 0x7845ee638 | 72 |
3'818'192 | 0.20%
| |- org.apache.lucene.index.SegmentReader @ 0x794777c08 | 72 |
3'644'816 | 0.19%
| |- org.apache.lucene.index.SegmentReader @ 0x783956a90 | 72 |
3'583'312 | 0.18%
| |- org.apache.lucene.index.SegmentReader @ 0x783dd5790 | 72 |
3'153'568 | 0.16%
| |- org.apache.lucene.index.SegmentReader @ 0x783dd57d8 | 72 |
1'017'264 | 0.05%
| |- org.apache.lucene.index.SegmentReader @ 0x77f27b7e8 | 72 |
668'304 | 0.03%
| |- org.apache.lucene.index.SegmentReader @ 0x77d657100 | 72 |
635'344 | 0.03%
| |- org.apache.lucene.index.SegmentReader @ 0x77e35bb10 | 72 |
614'784 | 0.03%
| |- org.apache.lucene.index.SegmentReader @ 0x77d676e68 | 72 |
263'176 | 0.01%
| |- org.apache.lucene.index.SegmentReader @ 0x77d6579e8 | 72 |
244'328 | 0.01%
| |- org.apache.lucene.index.SegmentReader @ 0x77f281b08 | 72 |
211'152 | 0.01%
| |- org.apache.lucene.index.SegmentReader @ 0x77d6579a0 | 72 |
178'784 | 0.01%
| |- org.apache.lucene.index.SegmentReader @ 0x77f2761b8 | 72 |
175'384 | 0.01%
| |- org.apache.lucene.index.SegmentReader @ 0x77d657148 | 72 |
167'128 | 0.01%
| |- org.apache.lucene.index.SegmentReader @ 0x77d676e20 | 72 |
164'128 | 0.01%
| |- org.apache.lucene.index.SegmentReader @ 0x77d657a30 | 72 |
150'560 | 0.01%
| |- org.apache.lucene.index.SegmentReader @ 0x77d676eb0 | 72 |
53'992 | 0.00%
| |- org.apache.lucene.index.SegmentReader @ 0x77d676ef8 | 72 |
35'200 | 0.00%
| '- Total: 24 entries | |
|
---------------------------------------------------------------------------------------------------------
-----Ursprüngliche Nachricht-----
Von: Michael McCandless [mailto:[email protected]]
Gesendet: Montag, 16. Juni 2014 10:59
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged
Wait, in fst.ByteStore I see "only" 5'485'824 -- does this mean
5485824 bytes, or ~5.2 MB? This is probably "correct", meaning this is the RAM
to hold the terms index. But I can't see from your heap dump output where the
other ~51.3 MB is being used by StandardDirectoryReader.
Mike McCandless
http://blog.mikemccandless.com
On Sun, Jun 15, 2014 at 2:16 PM, Clemens Wyss DEV <[email protected]> wrote:
>> bytes (ie, ~56.5 MB) being used by the StandardDirectoryReader?
> Right, namely in org.apache.lucene.util.fst.BytesStore's
>
> Class Name
> | Shallow Heap | Retained Heap | Percentage
> ----------------------------------------------------------------------------------------------------------------------------------------------
> org.apache.lucene.index.StandardDirectoryReader @ 0x783932460
> | 72 | 59'255'872 | 3.04%
> |- org.apache.lucene.index.SegmentReader[24] @ 0x794089ee0
> | 112 | 59'190'960 | 3.03%
> | |- org.apache.lucene.index.SegmentReader @ 0x788820f40
> | 72 | 16'905'072 | 0.87%
> | | |- org.apache.lucene.index.SegmentCoreReaders @ 0x7910cacc8
> | 56 | 16'895'576 | 0.87%
> | | | |-
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader @
> 0x780661c50 | 24 | 16'864'864 | 0.86%
> | | | | |- org.apache.lucene.codecs.BlockTreeTermsReader @ 0x7910cae50
> | 56 | 16'864'240 | 0.86%
> | | | | | |- java.util.TreeMap @ 0x783902738
> | 48 | 16'858'472 | 0.86%
> | | | | | | '- java.util.TreeMap$Entry @ 0x77ec5f9f8
> | 40 | 16'858'424 | 0.86%
> | | | | | | |- java.util.TreeMap$Entry @ 0x77ec5fa20
> | 40 | 10'895'656 | 0.56%
> | | | | | | | |-
> org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader @ 0x78347fc08|
> 72 | 5'497'328 | 0.28%
> | | | | | | | | |- org.apache.lucene.util.fst.FST @ 0x788fe36d0
> | 104 | 5'497'208 | 0.28%
> | | | | | | | | | |- org.apache.lucene.util.fst.BytesStore @
> 0x77ec5fe80 | 40 | 5'485'824 | 0.28%
> | | | | | | | | | |- org.apache.lucene.util.fst.FST$Arc[128] @
> 0x786efec80 | 528 | 11'216 | 0.00%
> | | | | | | | | | |- org.apache.lucene.util.BytesRef @
> 0x780664410 | 24 | 48 | 0.00%
> | | | | | | | | | |- int[0] @ 0x7806643f8
> | 16 | 16 | 0.00%
> | | | | | | | | | '- Total: 4 entries
> | | |
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> --
> -----Ursprüngliche Nachricht-----
> Von: Michael McCandless [mailto:[email protected]]
> Gesendet: Freitag, 13. Juni 2014 15:48
> An: Lucene Users
> Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged
>
> On Fri, Jun 13, 2014 at 8:53 AM, Clemens Wyss DEV <[email protected]>
> wrote:
>> Thanks a lot!
>>>"large text fields"
>> What is a good limit (in characters) to switch from StringField to
>> TextField? Do <Langugae>Analyzers (e.g. GermanAnalyzer) help a lot in
>> reducing the size of an Index?
>
> It's more based on your app's requirements. StringField indexes everything
> as a single token.
>
>>> Add XXXDocValuesField instead of e.g. StringField.
>> Does this apply only for StringFields? Or for TextFields too?
>>
>>> Upgrade to the upcoming Lucene 4.9
>> we have not yet transitionen to Java 7/8 ... hopefully soon ;)
>>
>>> and take a heap dump and see what's using RAM
>> Find attached a snippet from MemoryAnalyzer
>
> Does this say 59255872 bytes (ie, ~56.5 MB) being used by the
> StandardDirectoryReader?
>
> I'm a little confused because I don't see which structures sum up to that
> total. And I would expect the FST (terms index) to take more RAM.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
