I have MUCH better results with MINA 2.0.0 M2, but after a short while all clients/bots disconnect with:
An error occurred : null org.apache.mina.core.write.WriteTimeoutException Any ideas anyone? Thanks, Gerrit > -----Original Message----- > From: Gerrit Grobbelaar <[EMAIL PROTECTED]> > Sent: Saturday 26 July 2008 14:14 > To: [email protected] > CC: > Subject: Re: byte[] usage - not GCd > > > MINA 1.1.4 performs even better than 1.1.5. > > I still believe I'm doing something wrong in my code though, as I read only > good things about MINA in the reviews, but for me it just seems to crumble > underneath heavy load, the buffers not clearing quickly enough to be GCd. > > ==============> > MINA 1.1.4 - 30 bots > 1. 30 bots - each sending one message every 2 seconds > 1.1. after 10 minutes > 1.1.1. Heap size - 7MB > 1.1.2. byte[] allocation: > 1.1.2.1. no. of objects - 3,791: > 1.1.2.2. shallow size - 996,320 > 1.2. after killing 30 bots > 1.2.1. Heap size - 2,7MB > 1.2.2. byte[] allocation: > 1.2.2.1. no. of objects - 3,051: > 1.2.2.2. shallow size - 659,344 > <============== > > ==============> > MINA 1.1.4 - 100 bots > 2. 100 bots - each sending one message every 2 seconds > 2.1. after 2 minutes > 2.1.1. Heap size - 329MB - JVM unresponsive > 2.1.2. byte[] allocation: > 2.1.2.1. no. of objects - 423,512: > 2.1.2.2. shallow size - 201,053,744 > > 2.2. after killing 100 bots > 2.2.1. no effect - JVM unresponsive > > Interesting part: after about 1 minute this time, the server starts > spewing out logs again, seems to have some action on the buffers again, > processing messages still sent a while back, obviously printing out that > the session are closed already. Very soon all bots disconnected, freeing > up all the heap mem again. > <============= > > > -----Original Message----- > > From: Gerrit Grobbelaar <[EMAIL PROTECTED]> > > Sent: Saturday 26 July 2008 12:03 > > To: [email protected] > > CC: > > Subject: Re: byte[] usage - not GCd > > > > > > Some more test results (I definitely have better results with MINA 1.1.5, > > but still not ideal under load). Seeing that I get different, and > > somewhat better, result with an older version, I'm going to try even > > older versions than 1.1.5 now. > > > > In the meantime: howcome these different results though, and why will > > 1.1.5 perform better with Buffer (byte[]) handling/allocations/flushing > > than 1.1.6 and 1.1.7? Could be that I'm definitely doing something wrong > > with my MINA integration, but then I should get consistent bad > > performances across all versions? > > > > MINA 1.1.5 results as follows: > > > > ==============> > > MINA 1.1.5 - 30 bots > > 1. 30 bots - each sending one message every 2 seconds > > 1.1. after 10 minutes > > 1.1.1. Heap size - 20MB > > 1.1.2. byte[] allocation: > > 1.1.2.1. no. of objects - 12,104: > > 1.1.2.2. shallow size - 4,207,440 > > 1.2. after killing 30 bots > > 1.2.1. Heap size - 1,5MB > > 1.2.2. byte[] allocation: > > 1.2.2.1. no. of objects - 3,794: > > 1.2.2.2. shallow size - 722,696 > > <============== > > > > ==============> > > MINA 1.1.5 - 100 bots > > 2. 100 bots - each sending one message every 2 seconds > > 2.1. after 3 minutes > > 2.1.1. Heap size - 374MB > > 2.1.2. byte[] allocation: > > 2.1.2.1. no. of objects - 455,736: > > 2.1.2.2. shallow size - 216,055,400 > > > > 2.2. after killing 30 bots > > 2.2.1. no effect - JVM unresponsive > > > > Interesting part: after about 2 - 3 minutes, the server starts spewing > > out logs again, seems to have some action on the buffers again, > > processing messages still sent a while back, obviously printing out that > > the session are closed already. Only after about 5 minutes it is > > starting to detect the client disconnections. And then hangs again, > > after disconnecting only 14 of the 100 bots. There it continues again, > > still trying to send chat messages that was still in the buffers. Now > > detected 40 disconnections after another 2 minutes. Do note, Heap mem > > down to 40MB from 374MB. There we go: all disconnected now, perform GC, > > 2MB heap used. > > > > What could I be doing wrong that it seems that these buffers do not flush > > quickly enough, getting bogged down with the data? > > <============= > > > > ==============> > > MINA 1.1.5 - 100 bots > > 2. 100 bots - each sending one message every 5 minutes, and flash client > > sending message every 1 - 3 seconds > > 2.1. after 10minutes > > 2.1.1. Heap size - 165MB > > 2.1.2. byte[] allocation: > > 2.1.2.1. no. of objects - 180,352 > > 2.1.2.2. shallow size - 103,458,904 > > > > 1.2. after killing 100 bots > > 1.2.1. Heap size - 1,5MB > > 1.2.2. byte[] allocation: > > 1.2.2.1. no. of objects - 3,890: > > 1.2.2.2. shallow size - 792,536 > > <============= > > > > > -----Original Message----- > > > From: Gerrit Grobbelaar <[EMAIL PROTECTED]> > > > Sent: Saturday 26 July 2008 10:32 > > > To: [email protected] > > > CC: > > > Subject: Re: byte[] usage - not GCd > > > > > > > > > Yes, 1.1.7 but 1.1.6 as well. > > > > > > > -----Original Message----- > > > > From: Emmanuel Lecharny <[EMAIL PROTECTED]> > > > > Sent: Saturday 26 July 2008 10:20 > > > > To: [email protected] > > > > CC: > > > > Subject: Re: byte[] usage - not GCd > > > > > > > > > > > > Hi, > > > > > > > > MINA 1.1.7 ? > > > > > > > > Gerrit Grobbelaar wrote: > > > > > YourKit shows the Class list as follows: > > > > > - java.nio.HeapByteBuffer > > > > > - > > > > > org.apache.mina.common.SimpleByteBufferAllocator$SimpleByteBuffer - > > > > > org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer - > > > > > org.apache.mina.common.IoFilter$WriteRequest > > > > > - java.util.concurrent.ConcurrentLinkedQueue$Node > > > > > > > > > > Note the above contributes 99%+ for the byte[] allocations, and the > > > > > byte[] allocations are causing the OOM. > > > > > > > > > >> -----Original Message----- > > > > >> From: Gerrit Grobbelaar <[EMAIL PROTECTED]> > > > > >> Sent: Saturday 26 July 2008 09:48 > > > > >> To: [email protected] > > > > >> CC: > > > > >> Subject: Re: byte[] usage - not GCd > > > > >> > > > > >> > > > > >> Another symptom: > > > > >> - 100 Java client bots connected, sending NO messages. > > > > >> - 1 Flash client, typing in a message every 2 seconds > > > > >> - memory consumption (byte[] allocations) increases and a Mark > > > > >> Sweep with JConsole doesn't do anything (stays on 100s of MBs). > > > > >> - kill off all bots > > > > >> - Mark sweep > > > > >> - Heap mem usage drops to about 2 - 5MB > > > > >> > > > > >>> -----Original Message----- > > > > >>> From: Gerrit Grobbelaar <[EMAIL PROTECTED]> > > > > >>> Sent: Saturday 26 July 2008 09:33 > > > > >>> To: [email protected] > > > > >>> CC: > > > > >>> Subject: byte[] usage - not GCd > > > > >>> > > > > >>> > > > > >>> Hi, > > > > >>> > > > > >>> I'm having an issue with OOM - here is my story: > > > > >>> > > > > >>> Criteria: > > > > >>> -------- > > > > >>> - concurrent clients connecting to a chatserver, need to > > > > >>> broadcast the messages > > > > >>> - clients keep same connection (each one having its own > > > > >>> IoSession during lifetime) > > > > >>> > > > > >>> > > > > >>> Symptoms: > > > > >>> -------- > > > > >>> - massive amounts of byte[] allocations that just grows and > > > > >>> grows, until OOM. - still OOM even if i have 100 bots connected, > > > > >>> sending one message each every 5 minutes > > > > >>> > > > > >>> When I have 1 - 5 clients connected (all simulated with bots) > > > > >>> that send one message per bot every 2 - 5 seconds, heap space > > > > >>> stays under control with GC. > > > > >>> > > > > >>> > > > > >>> Configs/Tools: > > > > >>> ------------- > > > > >>> - JVM: 1.6u7 > > > > >>> - VM arguments: -Xms128m -Xmx384m > > > > >>> - Operating System: Linux 2.6.15.7 > > > > >>> - Architecture: i386 > > > > >>> - Number of processors: 2 > > > > >>> - Model: Intel(R) Pentium(R) 4 CPU 2.80GHz > > > > >>> - Committed virtual memory: 681,564 kbytes > > > > >>> - Heap buffer allocation (Direct is very insufficient) > > > > >>> - SimpleByteBufferAllocator (do note I have same result with > > > > >>> PooledByteBufferAllocator) > > > > >>> - YourKit and JProfiler > > > > >>> > > > > >>> Both YourKit and JProfiler point fingers at: > > > > >>> byte[] allocation due to org.apache.mina.common.IoSession.write() > > > > >>> calls > > > > >>> > > > > >>> > > > > >>> What could I possibly be doing wrong that I will have the above > > > > >>> scenario where the byte[] allocations never GC properly? > > > > >>> > > > > >>> Thanks, > > > > >>> Gerrit
