Hi,
(cc'ed hotspot-gc-use as it is the appropriate mailing list for such
quesstions).
On 13.08.19 17:01, shang xinli wrote:
There are two connected full GC but they are not able to reclaim bytes. Here
are gc logs.
[Times: user=0.35 sys=0.00, real=0.03 secs]
2019-08-01T15:12:14.594+0000: 755123.074: [Full GC (Allocation Failure)
132G->132G(180G), 37.4525886 secs]
[Eden: 0.0B(9216.0M)->0.0B(9216.0M) Survivors: 0.0B->0.0B Heap:
132.1G(180.0G)->132.1G(180.0G)], [Metaspace: 204550K->204459K(229376K)]
[Times: user=68.14 sys=0.07, real=37.45 secs]
2019-08-01T15:12:52.048+0000: 755160.527: [Full GC (Allocation Failure)
132G->132G(180G), 37.4584894 secs]
[Eden: 0.0B(9216.0M)->0.0B(9216.0M) Survivors: 0.0B->0.0B Heap:
132.1G(180.0G)->132.1G(180.0G)], [Metaspace: 204459K->204451K(229376K)]
[Times: user=68.38 sys=0.05, real=37.46 secs]
Here is the chart. We can see before and after GC, the heap is almost the same.
[cid:65dca3e8-3bc4-4b87-93dc-997ba0e601f2]
The service OOMs eventually, but we don't configure to generate heap dump. Is
there a way to investigate why?
The reason of the full gc might be humongous objects that use up lots of
additional space given that you get full gcs at 132 GB out of 180.
So does your application happen to use many - and I mean many - large
objects (~16M in size)?
You can add -XX:+UnlockExperimentalVMOptions -XX:G1LogLevel=finest to
your command line? (I assume you use some JDK8 given the log output, but
not sure)
Of particular interest are the following lines.
[Humongous Register: x.x ms]
[Humongous Total: y]
[Humongous Candidate: z]
Where y shows you the number of live humongous objects at the time of
the GC(s).
If that value is rather high, we found the issue.
The reason is that objects that have a total size that is just above the
humongous object threshold always use up full regions. This can waste a
lot of space.
That threshold is >= half a region size, i.e. 16M in your case, since
regions are 32M at that heap size by default. Such a humongous object
wastes the other half of the region. Remember that an object adds a
small header, so an object that has exactly 16M of usable data is
actually 16M + a little bit in size.
If possible, try to lower your object size a bit. Then the waste will be
much smaller in any case.
The only other option I can see is increasing the heap, or use a
different collector.
Thanks,
Thomas
_______________________________________________
hotspot-gc-use mailing list
hotspot-gc-use@openjdk.java.net
https://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use
