Actually the OOM error isn't b/c the JVM ran out of memory but is being
thrown directly from ByteArrayOutputStream:
private static int hugeCapacity(int minCapacity) {
if (minCapacity < 0) // overflow
*throw new OutOfMemoryError();*
return (minCapacity > MAX_ARRAY_SIZE) ?
Integer.MAX_VALUE :
MAX_ARRAY_SIZE;
}
Since the length field of an array is an int, it's constrained to
Integer.MAX_VALUE (~2G). To handle snapshots larger than 2G we'll have to
chuck it into multiple byte[].
On Tue, Jan 10, 2017 at 12:54 PM, Sela, Guy <guy.s...@hpe.com> wrote:
> From my understanding of the GC prints, there is no problem of free memory.
> I assume that this specific operation tried to create an object that is
> bigger than 15G-20G.
> Can you elaborate about "Snapshot has to be assembled as a byte[]"?
> Do you mean all the data in the data store (Or at least in the
> configuration data store) is serialized into a single byte[]?
>
> "Looks like a snapshot operation, i.e. roll-up of the journal."
> Can you elaborate about this as well?
>
> -----Original Message-----
> From: Robert Varga [mailto:n...@hq.sk]
> Sent: Tuesday, January 10, 2017 6:30 PM
> To: Sela, Guy <guy.s...@hpe.com>; controller-dev@lists.opendaylight.org
> Cc: odl netvirt dev <netvirt-...@lists.opendaylight.org>
> Subject: Re: [mdsal-dev] Netvirt Scale tests: OutOfMemory from datastore
>
> On 01/10/2017 03:40 PM, Sela, Guy wrote:
> > Hi,
>
> Hello,
>
> no MD-SAL in sight, -mdsal-dev, +controller-dev.
>
> >
> >
> >
> > I'm running scale tests for netvirt, using 24G heap, G1 collector, on
> > stable/boron.
> >
> > I'm trying to reach a mesh of tunnels between many computes.
> >
> > At about 100 computes, I get the exception below.
> >
> > Last GC before the exception got me to 3G occupied heap and 21G free
> heap.
>
> A heap dump is really needed to understand what is going on.
>
> > From what I understand, the MD-SAL tries to serialize an object, and
> > because it is too big, it gets OOM.
> >
> > 1) What is actually happening here? Is this a periodic flush to the
> > disk of the configuration data store?
>
> Looks like a snapshot operation, i.e. roll-up of the journal.
>
> > 2) Is there anything to do to avoid this scenario? Could the
> > objects be serialized in chunks?
>
> Depends on what the heap looks like. Snapshot has to be assembled as a
> byte[], there is not much we can do right now.
>
> Bye,
> Robert
>
> _______________________________________________
> controller-dev mailing list
> controller-dev@lists.opendaylight.org
> https://lists.opendaylight.org/mailman/listinfo/controller-dev
>
_______________________________________________
controller-dev mailing list
controller-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/controller-dev
