[I tried to add this to Jira, but Jira throws me an error :) ]
Emmanuel, I remember a bug where ConcurrentLinkedQueue$Node was the main actor :) It is DIRMINA-709.
When I investigated it, I have seen that GC was busy all the time and there were tens of millions of ConcurrentLinkedQueue$Node objects, they were allocated and released frequently. I tried to profile our server with YourKit profiler... without success because of high load (it was in production). Then I prepared my own "profiling tool" for this concrete problem. It uses AspectJ - I have added an aspect for Queue.offer() and Collection.add() method executions and grabbed most popular stack-traces from where these methods were called. If necessary, I can share my "tool" here.
Emmanuel Lecharny (JIRA) wrote:
[ https://issues.apache.org/jira/browse/DIRMINA-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emmanuel Lecharny updated DIRMINA-762: -------------------------------------- Attachment: Screen shot 2010-02-02 at 7.49.18 PM.png Screen shot 2010-02-02 at 7.49.13 PM.png Here are 5 snapshots : 3 for the server 2 for the client They expose : - Memory + CPU over time - Threads at the end of the test (green = running) - and allocated objects on the server One thing is strange : after having sent around 350 000 messages, the client and the server suddenly slow down. CPU which topped at 85% on my system goes down to 30%, and this is what we can see on both graphs. Also we have an giant number of ConcurrentLinkedQueue$Node objects, which get garbage collected later. I guess that at some point, we are simply pushing too many messages in the queues.WARN org.apache.mina.core.service.IoProcessor - Create a new selector. Selected is 0, delta = 0 ------------------------------------------------------------------------------------------------ Key: DIRMINA-762 URL: https://issues.apache.org/jira/browse/DIRMINA-762 Project: MINA Issue Type: Bug Environment: Linux (2.6.26-2-amd64), java version 1.6.0_12 and also 1.6.0_18. Reporter: Omry Yadan Priority: Critical Fix For: 2.0.0-RC2 Attachments: BufferCodec.java, Screen shot 2010-02-02 at 7.48.39 PM.png, Screen shot 2010-02-02 at 7.48.46 PM.png, Screen shot 2010-02-02 at 7.48.59 PM.png, Screen shot 2010-02-02 at 7.49.13 PM.png, Screen shot 2010-02-02 at 7.49.18 PM.png, Server.java, StressClient.java Mina server gets into a bad state where it constantly prints : WARN org.apache.mina.core.service.IoProcessor - Create a new selector. Selected is 0, delta = 0 when this happens, server throughput drops significantly. to reproduce run the attached server and client for a short while (30 seconds on my box).
