Hey Paul, Nice analysis. I can think of a few things you may want to try:
1) Branch 1.1 (and I guess that trunk too) are using a ConcurrentLinkedQueue instead of an Array. So performance with big queues should be better. 2) Based on your description I infer that you are using a single connection that in turn uses a single thread (from the pool) to process incoming traffic. While the processing thread is busy events (e.g. incoming data) gets queued. Therefore: 2.1) verify that the time it takes to process the incoming data is as fast as possible. The faster is process incoming data then the smaller the queue will get 2.2) change your thread model (e.g. add another ExecutorFilter after your ExecutorFilter) so that you have a pool of threads processing the events queue. 2.3) add more connections between your 2 applications so that more threads from the ExecutorFilter get used. Note: Options 2.2 and 2.3 should be profiled to ensure that speed is really improving compared to current situation. I think that ThreadModel is also related to the number of CPUs that you have so that context switching is not an overhead and having several threads is a benefit. :) Regards, -- Gato -----Original Message----- From: Paul Mallach [mailto:[EMAIL PROTECTED] Sent: Thursday, January 11, 2007 9:05 AM To: [email protected] Subject: Huge SessionBuffer event queue empties too slowly Hi. We're using MINA as the network library for our quote (as in stock exchange market data) server. I'm currently debugging a problem that seems to occur when a lot of data is sent in a short amount of time: We have two quote servers that replicate data between them. So when one server starts up, it acts as a client to the other server and requests a complete data-snapshot. The active server then calls session.write(Quote q) for about a million quote objects. Every few thousand calls we use WriteFuture.join(), so that the client is not overloaded. The client received the data without a hitch. The problem is, that all the data seems to remain in memory on the server even after it has been sent to the client: The java heap grows by about 600 megabyte. After a few hours however the heap usage has dropped down again. I have tried to analyze heap dumps before and after the replication process: After the replication is completed (client has received all the data), all ByteBuffers etc. created during sending are still in memory. In particular I found that there is an instance of org.apache.mina.filter.executor.ExecutorFilter$SessionBuffer that has the status "not completed" and that contains more than 2 million elements in it's event queue. These seem to be split about halfway between org.apache.mina.filter.codec.ProtocolCodecFilter$MessageByteBuffer (first) and org.apache.mina.filter.codec.ProtocolCodecFilter$HiddenByteBuffer (later on). These entries seem to correspond to the objects that have previously been sent to the client. Heap usage return to normal after a few hours. Is this an expected behaviour or am I doing something wrong? Is there a way to make those entries disappear sooner? One of Mina's threads seems to be working on an event queue: "AnonymousIoService-1" daemon prio=10 tid=0x00002aab3dbcd400 nid=0x1334 runnable [0x0000000042d55000..0x0000000042d55dc0] java.lang.Thread.State: RUNNABLE at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageSent (AbstractIoFilterChain.java:531) at org.apache.mina.common.support.AbstractIoFilterChain.access$1100 (AbstractIoFilterChain.java:52) at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.message Sent(AbstractIoFilterChain.java:783) at org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilt er.java:248) at org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run (ExecutorFilter.java:305) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWor ker(ThreadPoolExecutor.java:987) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker .run(ThreadPoolExecutor.java:528) at java.lang.Thread.run(Thread.java:619) I digged around in the mina's source and found, that ExecuteFilter uses an ArrayList to queue events and calls ArrayList.remove(0) to get the first element. As a test I created an ArrayList with 2 million Integer objects. Just removing the first 10000 elements in this way takes between 45 and 95 seconds. Of course it will get faster when the list shrinks, but I would still estimate, that just removing the elements from the list might take more than an hour. Could a LinkedList be used instead of the ArrayList? As far as I can see the only operations used on the list are append at the end and remove at the beginning, all of which would be constant time operations for a LinkedList. We are using mina-core-1.0.0.jar. The used IoHandler is directly derived from IoHandlerAdapter. The only filter that is manually inserted is a ProtocolCodecFilter with my custom Decoder and Encoder implementation. bye, Paul. -- ARIVA.DE AG, Paul Mallach, Produktentwicklung Walkerdamm 17, 24103 Kiel Tel.: +(49) 0431/97108-23, Fax: +(49) 0431/97108-29 E-Mail: [EMAIL PROTECTED], Web: www.ariva.de
