Some more test results (I definitely have better results with MINA 1.1.5, but
still not ideal under load). Seeing that I get different, and somewhat
better, result with an older version, I'm going to try even older versions
than 1.1.5 now.
In the meantime: howcome these different results though, and why will 1.1.5
perform better with Buffer (byte[]) handling/allocations/flushing than 1.1.6
and 1.1.7? Could be that I'm definitely doing something wrong with my MINA
integration, but then I should get consistent bad performances across all
versions?
MINA 1.1.5 results as follows:
==============>
MINA 1.1.5 - 30 bots
1. 30 bots - each sending one message every 2 seconds
1.1. after 10 minutes
1.1.1. Heap size - 20MB
1.1.2. byte[] allocation:
1.1.2.1. no. of objects - 12,104:
1.1.2.2. shallow size - 4,207,440
1.2. after killing 30 bots
1.2.1. Heap size - 1,5MB
1.2.2. byte[] allocation:
1.2.2.1. no. of objects - 3,794:
1.2.2.2. shallow size - 722,696
<==============
==============>
MINA 1.1.5 - 100 bots
2. 100 bots - each sending one message every 2 seconds
2.1. after 3 minutes
2.1.1. Heap size - 374MB
2.1.2. byte[] allocation:
2.1.2.1. no. of objects - 455,736:
2.1.2.2. shallow size - 216,055,400
2.2. after killing 30 bots
2.2.1. no effect - JVM unresponsive
Interesting part: after about 2 - 3 minutes, the server starts spewing out
logs again, seems to have some action on the buffers again, processing
messages still sent a while back, obviously printing out that the session are
closed already. Only after about 5 minutes it is starting to detect the
client disconnections. And then hangs again, after disconnecting only 14 of
the 100 bots. There it continues again, still trying to send chat messages
that was still in the buffers. Now detected 40 disconnections after another
2 minutes. Do note, Heap mem down to 40MB from 374MB. There we go: all
disconnected now, perform GC, 2MB heap used.
What could I be doing wrong that it seems that these buffers do not flush
quickly enough, getting bogged down with the data?
<=============
==============>
MINA 1.1.5 - 100 bots
2. 100 bots - each sending one message every 5 minutes, and flash client
sending message every 1 - 3 seconds
2.1. after 10minutes
2.1.1. Heap size - 165MB
2.1.2. byte[] allocation:
2.1.2.1. no. of objects - 180,352
2.1.2.2. shallow size - 103,458,904
1.2. after killing 100 bots
1.2.1. Heap size - 1,5MB
1.2.2. byte[] allocation:
1.2.2.1. no. of objects - 3,890:
1.2.2.2. shallow size - 792,536
<=============
> -----Original Message-----
> From: Gerrit Grobbelaar <[EMAIL PROTECTED]>
> Sent: Saturday 26 July 2008 10:32
> To: [email protected]
> CC:
> Subject: Re: byte[] usage - not GCd
>
>
> Yes, 1.1.7 but 1.1.6 as well.
>
> > -----Original Message-----
> > From: Emmanuel Lecharny <[EMAIL PROTECTED]>
> > Sent: Saturday 26 July 2008 10:20
> > To: [email protected]
> > CC:
> > Subject: Re: byte[] usage - not GCd
> >
> >
> > Hi,
> >
> > MINA 1.1.7 ?
> >
> > Gerrit Grobbelaar wrote:
> > > YourKit shows the Class list as follows:
> > > - java.nio.HeapByteBuffer
> > > - org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer
> > > - org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer
> > > - org.apache.mina.common.IoFilter$WriteRequest
> > > - java.util.concurrent.ConcurrentLinkedQueue$Node
> > >
> > > Note the above contributes 99%+ for the byte[] allocations, and the
> > > byte[] allocations are causing the OOM.
> > >
> > >> -----Original Message-----
> > >> From: Gerrit Grobbelaar <[EMAIL PROTECTED]>
> > >> Sent: Saturday 26 July 2008 09:48
> > >> To: [email protected]
> > >> CC:
> > >> Subject: Re: byte[] usage - not GCd
> > >>
> > >>
> > >> Another symptom:
> > >> - 100 Java client bots connected, sending NO messages.
> > >> - 1 Flash client, typing in a message every 2 seconds
> > >> - memory consumption (byte[] allocations) increases and a Mark Sweep
> > >> with JConsole doesn't do anything (stays on 100s of MBs).
> > >> - kill off all bots
> > >> - Mark sweep
> > >> - Heap mem usage drops to about 2 - 5MB
> > >>
> > >>> -----Original Message-----
> > >>> From: Gerrit Grobbelaar <[EMAIL PROTECTED]>
> > >>> Sent: Saturday 26 July 2008 09:33
> > >>> To: [email protected]
> > >>> CC:
> > >>> Subject: byte[] usage - not GCd
> > >>>
> > >>>
> > >>> Hi,
> > >>>
> > >>> I'm having an issue with OOM - here is my story:
> > >>>
> > >>> Criteria:
> > >>> --------
> > >>> - concurrent clients connecting to a chatserver, need to broadcast
> > >>> the messages
> > >>> - clients keep same connection (each one having its own IoSession
> > >>> during lifetime)
> > >>>
> > >>>
> > >>> Symptoms:
> > >>> --------
> > >>> - massive amounts of byte[] allocations that just grows and grows,
> > >>> until OOM. - still OOM even if i have 100 bots connected, sending one
> > >>> message each every 5 minutes
> > >>>
> > >>> When I have 1 - 5 clients connected (all simulated with bots) that
> > >>> send one message per bot every 2 - 5 seconds, heap space stays under
> > >>> control with GC.
> > >>>
> > >>>
> > >>> Configs/Tools:
> > >>> -------------
> > >>> - JVM: 1.6u7
> > >>> - VM arguments: -Xms128m -Xmx384m
> > >>> - Operating System: Linux 2.6.15.7
> > >>> - Architecture: i386
> > >>> - Number of processors: 2
> > >>> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz
> > >>> - Committed virtual memory: 681,564 kbytes
> > >>> - Heap buffer allocation (Direct is very insufficient)
> > >>> - SimpleByteBufferAllocator (do note I have same result with
> > >>> PooledByteBufferAllocator)
> > >>> - YourKit and JProfiler
> > >>>
> > >>> Both YourKit and JProfiler point fingers at:
> > >>> byte[] allocation due to org.apache.mina.common.IoSession.write()
> > >>> calls
> > >>>
> > >>>
> > >>> What could I possibly be doing wrong that I will have the above
> > >>> scenario where the byte[] allocations never GC properly?
> > >>>
> > >>> Thanks,
> > >>> Gerrit
