Re: JAMES OOM, maybe the cause of MINA ?
Hi, I finally made some tests with mina-rc2 trying to reproduce the oom. I didn't succeed to reproduce it when stressing james with normal mails (well-formed,...). I would tend to follow the pointer Norman gave about the infinite loop in the exceptionCaught() method (writing on the session recause and exception,...) http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/mina/SMTPIoHandler.java?revision=930014view=markup I think that some sockets behaving badly may be the cause and tried to simulate this with long-time opened connection, with garbage being sent on the sockets,... without success (no exception). I will now roolback to the previous version of SMTPIoHandler with james not closing the mina session, and see what I can find. If you've got some idea of patterns, datas,... to send on the socket to make mina throw an exception, just post it. Tks, Eric On 04/13/2010 07:23 PM, Eric Charles wrote: Hi Ashish, James in running with netty since 2 days. I plan to let it run until this Friday (if no problem). Thereafter, I will go back to mina. The logs I've got are not helpfull and show nothing, except a sudden java OOM. I may try this weekend to find a reproducible way to have the oom. Tks, Eric On 04/13/2010 10:47 AM, Norman Maurer wrote: We need to let Eric test it again. Currently he run the netty version of james to see if the cause is really MINA .. Thx, Norman 2010/4/13 Ashishpaliwalash...@gmail.com: Is the OOM fixed after the change? thanks ashish On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I bet its the loop I mention before.. This would make sense when looking at the growing queued writes. I commited a change to trunk which close the session after write the data to the client. We will see if this helps.. Thx, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Is the OOM fixed after the change? thanks ashish On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I bet its the loop I mention before.. This would make sense when looking at the growing queued writes. I commited a change to trunk which close the session after write the data to the client. We will see if this helps.. Thx, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
We need to let Eric test it again. Currently he run the netty version of james to see if the cause is really MINA .. Thx, Norman 2010/4/13 Ashish paliwalash...@gmail.com: Is the OOM fixed after the change? thanks ashish On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I bet its the loop I mention before.. This would make sense when looking at the growing queued writes. I commited a change to trunk which close the session after write the data to the client. We will see if this helps.. Thx, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, I bet its the loop I mention before.. This would make sense when looking at the growing queued writes. I commited a change to trunk which close the session after write the data to the client. We will see if this helps.. Thx, Norman 2010/4/12 Ashish paliwalash...@gmail.com: On Sun, Apr 11, 2010 at 8:57 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, after thinkin more about the whole stuff I wonder if the exception handling in SMTP could sometimes be an endless loop. In the handler we write a message back to the client on exception. But what happens if this throw a new exception ? Could it so loop forever ? I am not sure, but it could be possible. http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/mina/SMTPIoHandler.java?revision=930014view=markup Maybe we should just close the connection.. hmm.. its brute force. I am not convinced that we should do this, but not sure as I don't have a complete picture as such. If the exception is coz of packet decoding, ignore the packet and move on.. Its also driven by the protocol as such. You may need to respond back to the client with some error code etc. Eric: Please mark a copy to mina dev ML as well, else I don't get your mails :) The analysis is a session is in hanging state, whose queue is building up. Heap dump are not going to help any more. Can you trace down what the hack is going on with that session. I can't get much out of heap dump and unfortunately not able to run it on laptop. jconsole simply refuse to detect the James process :( So the key lies in analyzing 1. Why is this session hanging, its simply not writing the data back to the client. There is a while before the session is closed and session.isConnected() returns false. Eric: can you upload the logs as well, if possible :) Anyone else has some idea's here? thanks ashish Bye Norman 2010/4/9 Eric Charles eric.char...@u-mangate.com: Hi Ashish, I am the user who has many OOM with the trunk and feed Norman with my issues :) 1. Indeed, you have the dump with CircularQueue from mina 2.0.0-RC1. This is the release we normally use in james and give the issue after a certain period of time. I mean, the server can stay working correctly 2 days or crash after 2 hours (see http://apache.u-mangate.com/james/oom/oom.png) due to a peak in memory usage. T To further investigate, we tried with mina trunk 2.0.0-RC2-SNAPSHOT : There, we see a direct crash. I've uploaded such a dump on http://apache.u-mangate.com/james/oom/index.html (click on shift-f5 to force refresh, I setup the web server in a hurry). You will see there recursive ConcurrentLinkedQueue. 2. As shown on http://apache.u-mangate.com/james/oom/oom.png, the system works while the line is flat (mail are received, spooled, delivered,...). On a few seconds, it peaks and of course, nothing works anymore. So, the messages don't remain in the spool. I made tests with quite huge messages, and they are delivered very fast as soon as they arrive in James. Don't hesitate to ask more questions or propose additional tests. Many tks in advance, Eric On 04/09/2010 12:40 PM, Ashish wrote: Norman, Couple of more queries 1. The heap dump uses circularqueue class, so seem to be taken for an earlier trunk snapshot. Is my take correct? 2. What's the state of the System? are the clients receiving the messages. The queue seems to be holding a very large number of objects. Essentially what I want to know is, if the clients are receiving the messages or the Server is holding them up. Will spend more time with the issue and see what I can figure out. thanks ashish On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer norman.mau...@googlemail.com wrote: Maybe Eric can do, cause he is the one who see it very freqently.. So Eric...;) ? Thx, Norman 2010/4/8 Ashishpaliwalash...@gmail.com: Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashishpaliwalash...@gmail.com wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail:
Re: JAMES OOM, maybe the cause of MINA ?
On Sun, Apr 11, 2010 at 8:57 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, after thinkin more about the whole stuff I wonder if the exception handling in SMTP could sometimes be an endless loop. In the handler we write a message back to the client on exception. But what happens if this throw a new exception ? Could it so loop forever ? I am not sure, but it could be possible. http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/mina/SMTPIoHandler.java?revision=930014view=markup Maybe we should just close the connection.. hmm.. its brute force. I am not convinced that we should do this, but not sure as I don't have a complete picture as such. If the exception is coz of packet decoding, ignore the packet and move on.. Its also driven by the protocol as such. You may need to respond back to the client with some error code etc. Eric: Please mark a copy to mina dev ML as well, else I don't get your mails :) The analysis is a session is in hanging state, whose queue is building up. Heap dump are not going to help any more. Can you trace down what the hack is going on with that session. I can't get much out of heap dump and unfortunately not able to run it on laptop. jconsole simply refuse to detect the James process :( So the key lies in analyzing 1. Why is this session hanging, its simply not writing the data back to the client. There is a while before the session is closed and session.isConnected() returns false. Eric: can you upload the logs as well, if possible :) Anyone else has some idea's here? thanks ashish Bye Norman 2010/4/9 Eric Charles eric.char...@u-mangate.com: Hi Ashish, I am the user who has many OOM with the trunk and feed Norman with my issues :) 1. Indeed, you have the dump with CircularQueue from mina 2.0.0-RC1. This is the release we normally use in james and give the issue after a certain period of time. I mean, the server can stay working correctly 2 days or crash after 2 hours (see http://apache.u-mangate.com/james/oom/oom.png) due to a peak in memory usage. T To further investigate, we tried with mina trunk 2.0.0-RC2-SNAPSHOT : There, we see a direct crash. I've uploaded such a dump on http://apache.u-mangate.com/james/oom/index.html (click on shift-f5 to force refresh, I setup the web server in a hurry). You will see there recursive ConcurrentLinkedQueue. 2. As shown on http://apache.u-mangate.com/james/oom/oom.png, the system works while the line is flat (mail are received, spooled, delivered,...). On a few seconds, it peaks and of course, nothing works anymore. So, the messages don't remain in the spool. I made tests with quite huge messages, and they are delivered very fast as soon as they arrive in James. Don't hesitate to ask more questions or propose additional tests. Many tks in advance, Eric On 04/09/2010 12:40 PM, Ashish wrote: Norman, Couple of more queries 1. The heap dump uses circularqueue class, so seem to be taken for an earlier trunk snapshot. Is my take correct? 2. What's the state of the System? are the clients receiving the messages. The queue seems to be holding a very large number of objects. Essentially what I want to know is, if the clients are receiving the messages or the Server is holding them up. Will spend more time with the issue and see what I can figure out. thanks ashish On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer norman.mau...@googlemail.com wrote: Maybe Eric can do, cause he is the one who see it very freqently.. So Eric...;) ? Thx, Norman 2010/4/8 Ashishpaliwalash...@gmail.com: Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashishpaliwalash...@gmail.com wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail:
Re: JAMES OOM, maybe the cause of MINA ?
On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I bet its the loop I mention before.. This would make sense when looking at the growing queued writes. I commited a change to trunk which close the session after write the data to the client. We will see if this helps.. I hope it solves the problem :) Closing the session means disconnecting the client. If the situation wants this its fine, else need to dig a bit more for solution :) If the problem is solved, I consider its a good start for the week :) Thx, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Forgot to include JAMES list.. 2010/4/12 Norman Maurer norman.mau...@googlemail.com: Hi Asish, yeah in our case this is fine. But I think even if this fix the problem MINA should at least try to break such loops. Bye, Norman 2010/4/12 Ashish paliwalash...@gmail.com: On Mon, Apr 12, 2010 at 1:15 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I bet its the loop I mention before.. This would make sense when looking at the growing queued writes. I commited a change to trunk which close the session after write the data to the client. We will see if this helps.. I hope it solves the problem :) Closing the session means disconnecting the client. If the situation wants this its fine, else need to dig a bit more for solution :) If the problem is solved, I consider its a good start for the week :) Thx, Norman - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, after thinkin more about the whole stuff I wonder if the exception handling in SMTP could sometimes be an endless loop. In the handler we write a message back to the client on exception. But what happens if this throw a new exception ? Could it so loop forever ? http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/mina/SMTPIoHandler.java?revision=930014view=markup Maybe we should just close the connection.. Bye Norman 2010/4/9 Eric Charles eric.char...@u-mangate.com: Hi Ashish, I am the user who has many OOM with the trunk and feed Norman with my issues :) 1. Indeed, you have the dump with CircularQueue from mina 2.0.0-RC1. This is the release we normally use in james and give the issue after a certain period of time. I mean, the server can stay working correctly 2 days or crash after 2 hours (see http://apache.u-mangate.com/james/oom/oom.png) due to a peak in memory usage. T To further investigate, we tried with mina trunk 2.0.0-RC2-SNAPSHOT : There, we see a direct crash. I've uploaded such a dump on http://apache.u-mangate.com/james/oom/index.html (click on shift-f5 to force refresh, I setup the web server in a hurry). You will see there recursive ConcurrentLinkedQueue. 2. As shown on http://apache.u-mangate.com/james/oom/oom.png, the system works while the line is flat (mail are received, spooled, delivered,...). On a few seconds, it peaks and of course, nothing works anymore. So, the messages don't remain in the spool. I made tests with quite huge messages, and they are delivered very fast as soon as they arrive in James. Don't hesitate to ask more questions or propose additional tests. Many tks in advance, Eric On 04/09/2010 12:40 PM, Ashish wrote: Norman, Couple of more queries 1. The heap dump uses circularqueue class, so seem to be taken for an earlier trunk snapshot. Is my take correct? 2. What's the state of the System? are the clients receiving the messages. The queue seems to be holding a very large number of objects. Essentially what I want to know is, if the clients are receiving the messages or the Server is holding them up. Will spend more time with the issue and see what I can figure out. thanks ashish On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer norman.mau...@googlemail.com wrote: Maybe Eric can do, cause he is the one who see it very freqently.. So Eric...;) ? Thx, Norman 2010/4/8 Ashishpaliwalash...@gmail.com: Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashishpaliwalash...@gmail.com wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Norman, Couple of more queries 1. The heap dump uses circularqueue class, so seem to be taken for an earlier trunk snapshot. Is my take correct? 2. What's the state of the System? are the clients receiving the messages. The queue seems to be holding a very large number of objects. Essentially what I want to know is, if the clients are receiving the messages or the Server is holding them up. Will spend more time with the issue and see what I can figure out. thanks ashish On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer norman.mau...@googlemail.com wrote: Maybe Eric can do, cause he is the one who see it very freqently.. So Eric...;) ? Thx, Norman 2010/4/8 Ashish paliwalash...@gmail.com: Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashish paliwalash...@gmail.com wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Did some more investigation. Here is one of the strange observations
Most of the hanging DefaultWriteRequest have this message 451 Unable
to process smtp request..
Can't analyze all of them, but saw content of close to 100 objects.
All of them had the same content
This error is coming from org.apache.james.smtpserver.mina.SMTPIoHandler Line:94
if (session.isConnected()) {
session.write(new SMTPResponse(SMTPRetCode.LOCAL_ERROR, Unable to
process smtp request));
}
Also, from the heapdump, all the WriteRequests are for a session
(id=13609), and they are hanging there :(
so seems like we have a hanging session.
Any thoughts?
Will try to reproduce this problem at my end. Some logs might be of help.
thanks
ashish
On Fri, Apr 9, 2010 at 4:10 PM, Ashish paliwalash...@gmail.com wrote:
Norman,
Couple of more queries
1. The heap dump uses circularqueue class, so seem to be taken for an
earlier trunk snapshot. Is my take correct?
2. What's the state of the System? are the clients receiving the
messages. The queue seems to be holding a very large number of
objects.
Essentially what I want to know is, if the clients are receiving the
messages or the Server is holding them up.
Will spend more time with the issue and see what I can figure out.
thanks
ashish
On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer
norman.mau...@googlemail.com wrote:
Maybe Eric can do, cause he is the one who see it very freqently..
So Eric...;) ?
Thx,
Norman
2010/4/8 Ashish paliwalash...@gmail.com:
Can you provide the heapdump for this OOM?
thanks
ashish
On Thu, Apr 8, 2010 at 1:40 PM, Ashish paliwalash...@gmail.com wrote:
On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer
norman.mau...@googlemail.com wrote:
Hi Ashish,
I think we tracked down the source of the problem a bit more.. The OOM
seems to be related to IMAP. Our IMAP server component is using the
StreamIoHandler (its the only one of our components who use this
handler). So I suspect there is the problem.
So there are two possible problems:
1) Bug in StreamIoHandler
2) Wrong usage of StreamIoHandler. Our implementations is here:
http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup
Thx,
Norman
Sorry, haven't been able to look at this so far :(
earliest I can give it a shot will be on Sunday.
thanks
ashish
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish,
yeah ofter some more investigation it seems that it's not the
StreamIoHandler which cause the problem. It must be something else...
Bye
Norman
2010/4/9 Ashish paliwalash...@gmail.com:
Did some more investigation. Here is one of the strange observations
Most of the hanging DefaultWriteRequest have this message 451 Unable
to process smtp request..
Can't analyze all of them, but saw content of close to 100 objects.
All of them had the same content
This error is coming from org.apache.james.smtpserver.mina.SMTPIoHandler
Line:94
if (session.isConnected()) {
session.write(new SMTPResponse(SMTPRetCode.LOCAL_ERROR, Unable to
process smtp request));
}
Also, from the heapdump, all the WriteRequests are for a session
(id=13609), and they are hanging there :(
so seems like we have a hanging session.
Any thoughts?
Will try to reproduce this problem at my end. Some logs might be of help.
thanks
ashish
On Fri, Apr 9, 2010 at 4:10 PM, Ashish paliwalash...@gmail.com wrote:
Norman,
Couple of more queries
1. The heap dump uses circularqueue class, so seem to be taken for an
earlier trunk snapshot. Is my take correct?
2. What's the state of the System? are the clients receiving the
messages. The queue seems to be holding a very large number of
objects.
Essentially what I want to know is, if the clients are receiving the
messages or the Server is holding them up.
Will spend more time with the issue and see what I can figure out.
thanks
ashish
On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer
norman.mau...@googlemail.com wrote:
Maybe Eric can do, cause he is the one who see it very freqently..
So Eric...;) ?
Thx,
Norman
2010/4/8 Ashish paliwalash...@gmail.com:
Can you provide the heapdump for this OOM?
thanks
ashish
On Thu, Apr 8, 2010 at 1:40 PM, Ashish paliwalash...@gmail.com wrote:
On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer
norman.mau...@googlemail.com wrote:
Hi Ashish,
I think we tracked down the source of the problem a bit more.. The OOM
seems to be related to IMAP. Our IMAP server component is using the
StreamIoHandler (its the only one of our components who use this
handler). So I suspect there is the problem.
So there are two possible problems:
1) Bug in StreamIoHandler
2) Wrong usage of StreamIoHandler. Our implementations is here:
http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup
Thx,
Norman
Sorry, haven't been able to look at this so far :(
earliest I can give it a shot will be on Sunday.
thanks
ashish
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Lets dig this further and finish this off.. :)
Will try to generate this OOM locally... will keep you all posted
On Fri, Apr 9, 2010 at 9:32 PM, Norman Maurer nor...@apache.org wrote:
Hi Ashish,
yeah ofter some more investigation it seems that it's not the
StreamIoHandler which cause the problem. It must be something else...
Bye
Norman
2010/4/9 Ashish paliwalash...@gmail.com:
Did some more investigation. Here is one of the strange observations
Most of the hanging DefaultWriteRequest have this message 451 Unable
to process smtp request..
Can't analyze all of them, but saw content of close to 100 objects.
All of them had the same content
This error is coming from org.apache.james.smtpserver.mina.SMTPIoHandler
Line:94
if (session.isConnected()) {
session.write(new SMTPResponse(SMTPRetCode.LOCAL_ERROR, Unable to
process smtp request));
}
Also, from the heapdump, all the WriteRequests are for a session
(id=13609), and they are hanging there :(
so seems like we have a hanging session.
Any thoughts?
Will try to reproduce this problem at my end. Some logs might be of help.
thanks
ashish
On Fri, Apr 9, 2010 at 4:10 PM, Ashish paliwalash...@gmail.com wrote:
Norman,
Couple of more queries
1. The heap dump uses circularqueue class, so seem to be taken for an
earlier trunk snapshot. Is my take correct?
2. What's the state of the System? are the clients receiving the
messages. The queue seems to be holding a very large number of
objects.
Essentially what I want to know is, if the clients are receiving the
messages or the Server is holding them up.
Will spend more time with the issue and see what I can figure out.
thanks
ashish
On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer
norman.mau...@googlemail.com wrote:
Maybe Eric can do, cause he is the one who see it very freqently..
So Eric...;) ?
Thx,
Norman
2010/4/8 Ashish paliwalash...@gmail.com:
Can you provide the heapdump for this OOM?
thanks
ashish
On Thu, Apr 8, 2010 at 1:40 PM, Ashish paliwalash...@gmail.com wrote:
On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer
norman.mau...@googlemail.com wrote:
Hi Ashish,
I think we tracked down the source of the problem a bit more.. The OOM
seems to be related to IMAP. Our IMAP server component is using the
StreamIoHandler (its the only one of our components who use this
handler). So I suspect there is the problem.
So there are two possible problems:
1) Bug in StreamIoHandler
2) Wrong usage of StreamIoHandler. Our implementations is here:
http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup
Thx,
Norman
Sorry, haven't been able to look at this so far :(
earliest I can give it a shot will be on Sunday.
thanks
ashish
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
-
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, I am the user who has many OOM with the trunk and feed Norman with my issues :) 1. Indeed, you have the dump with CircularQueue from mina 2.0.0-RC1. This is the release we normally use in james and give the issue after a certain period of time. I mean, the server can stay working correctly 2 days or crash after 2 hours (see http://apache.u-mangate.com/james/oom/oom.png) due to a peak in memory usage. T To further investigate, we tried with mina trunk 2.0.0-RC2-SNAPSHOT : There, we see a direct crash. I've uploaded such a dump on http://apache.u-mangate.com/james/oom/index.html (click on shift-f5 to force refresh, I setup the web server in a hurry). You will see there recursive ConcurrentLinkedQueue. 2. As shown on http://apache.u-mangate.com/james/oom/oom.png, the system works while the line is flat (mail are received, spooled, delivered,...). On a few seconds, it peaks and of course, nothing works anymore. So, the messages don't remain in the spool. I made tests with quite huge messages, and they are delivered very fast as soon as they arrive in James. Don't hesitate to ask more questions or propose additional tests. Many tks in advance, Eric On 04/09/2010 12:40 PM, Ashish wrote: Norman, Couple of more queries 1. The heap dump uses circularqueue class, so seem to be taken for an earlier trunk snapshot. Is my take correct? 2. What's the state of the System? are the clients receiving the messages. The queue seems to be holding a very large number of objects. Essentially what I want to know is, if the clients are receiving the messages or the Server is holding them up. Will spend more time with the issue and see what I can figure out. thanks ashish On Thu, Apr 8, 2010 at 2:03 PM, Norman Maurer norman.mau...@googlemail.com wrote: Maybe Eric can do, cause he is the one who see it very freqently.. So Eric...;) ? Thx, Norman 2010/4/8 Ashishpaliwalash...@gmail.com: Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashishpaliwalash...@gmail.com wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman 2010/4/5 Eric Charles eric.char...@u-mangate.com: Yes, One instance of org.apache.mina.core.session.DefaultIoSessionDataStructureFactory$DefaultWriteRequestQueue contains : - for 2.0.0-RC1 : one instance of org.apache.mina.util.CircularQueue with an array of many org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest - for 2.0.0-RC2-SNAPSHOT : many recursive instances of java.util.concurrent.ConcurrentLinkedQueue Tks, Eric On 04/05/2010 10:51 AM, Norman Maurer wrote: I think thats only because mina-trunk is not using CircularQueue anymore. It now seems to use ConcurrentLinkedQueue. Bye, Norman 2010/4/5 Eric Charleseric.char...@u-mangate.com: Hi Ashish, Please also note that we have 2 different exceptions: - http://people.apache.org/~norman/oom/Screenshot.png that occurs with mina 2.0.0-RC1 - http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png that occurs with 2.0.0-RC2-SNAPSHOT It happens on jdk 1.6 (and also 1.7) Tks, Eric On 04/05/2010 10:01 AM, Ashish wrote: Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashishpaliwalash...@gmail.com: On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurernor...@apache.org wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Maybe Eric can do, cause he is the one who see it very freqently.. So Eric...;) ? Thx, Norman 2010/4/8 Ashish paliwalash...@gmail.com: Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashish paliwalash...@gmail.com wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashish paliwalash...@gmail.com wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, You can find some information to download on http://apache.u-mangate.com/james/oom/index.html Tks, Eric On 04/08/2010 10:26 AM, Ashish wrote: Can you provide the heapdump for this OOM? thanks ashish On Thu, Apr 8, 2010 at 1:40 PM, Ashishpaliwalash...@gmail.com wrote: On Thu, Apr 8, 2010 at 1:28 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, I think we tracked down the source of the problem a bit more.. The OOM seems to be related to IMAP. Our IMAP server component is using the StreamIoHandler (its the only one of our components who use this handler). So I suspect there is the problem. So there are two possible problems: 1) Bug in StreamIoHandler 2) Wrong usage of StreamIoHandler. Our implementations is here: http://svn.apache.org/viewvc/james/server/trunk/imapserver/src/main/java/org/apache/james/imapserver/mina/ImapIoHandler.java?view=markup Thx, Norman Sorry, haven't been able to look at this so far :( earliest I can give it a shot will be on Sunday. thanks ashish - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurer nor...@apache.org wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashish paliwalash...@gmail.com: On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurer nor...@apache.org wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashish paliwalash...@gmail.com: On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurer nor...@apache.org wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Hi Ashish, Please also note that we have 2 different exceptions: - http://people.apache.org/~norman/oom/Screenshot.png that occurs with mina 2.0.0-RC1 - http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png that occurs with 2.0.0-RC2-SNAPSHOT It happens on jdk 1.6 (and also 1.7) Tks, Eric On 04/05/2010 10:01 AM, Ashish wrote: Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashishpaliwalash...@gmail.com: On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurernor...@apache.org wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
I think thats only because mina-trunk is not using CircularQueue anymore. It now seems to use ConcurrentLinkedQueue. Bye, Norman 2010/4/5 Eric Charles eric.char...@u-mangate.com: Hi Ashish, Please also note that we have 2 different exceptions: - http://people.apache.org/~norman/oom/Screenshot.png that occurs with mina 2.0.0-RC1 - http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png that occurs with 2.0.0-RC2-SNAPSHOT It happens on jdk 1.6 (and also 1.7) Tks, Eric On 04/05/2010 10:01 AM, Ashish wrote: Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashishpaliwalash...@gmail.com: On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurernor...@apache.org wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
Yes, One instance of org.apache.mina.core.session.DefaultIoSessionDataStructureFactory$DefaultWriteRequestQueue contains : - for 2.0.0-RC1 : one instance of org.apache.mina.util.CircularQueue with an array of many org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest - for 2.0.0-RC2-SNAPSHOT : many recursive instances of java.util.concurrent.ConcurrentLinkedQueue Tks, Eric On 04/05/2010 10:51 AM, Norman Maurer wrote: I think thats only because mina-trunk is not using CircularQueue anymore. It now seems to use ConcurrentLinkedQueue. Bye, Norman 2010/4/5 Eric Charleseric.char...@u-mangate.com: Hi Ashish, Please also note that we have 2 different exceptions: - http://people.apache.org/~norman/oom/Screenshot.png that occurs with mina 2.0.0-RC1 - http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png that occurs with 2.0.0-RC2-SNAPSHOT It happens on jdk 1.6 (and also 1.7) Tks, Eric On 04/05/2010 10:01 AM, Ashish wrote: Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer norman.mau...@googlemail.comwrote: Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashishpaliwalash...@gmail.com: On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurernor...@apache.org wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
On Mon, Apr 5, 2010 at 2:21 PM, Norman Maurer norman.mau...@googlemail.com wrote: I think thats only because mina-trunk is not using CircularQueue anymore. It now seems to use ConcurrentLinkedQueue. Bye, Norman I remember, Emm removed all references to CircularQueue, but this was to make it thread safe :) http://mina.markmail.org/message/xirxdepng3j65w4b?q=CircularQueue#query:CircularQueue+page:1+mid:xirxdepng3j65w4b+state:results Hmm, not sure why it would hold the references. Thanks for Heads up.. 2010/4/5 Eric Charles eric.char...@u-mangate.com: Hi Ashish, Please also note that we have 2 different exceptions: - http://people.apache.org/~norman/oom/Screenshot.png that occurs with mina 2.0.0-RC1 - http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png that occurs with 2.0.0-RC2-SNAPSHOT It happens on jdk 1.6 (and also 1.7) Tks, Eric On 04/05/2010 10:01 AM, Ashish wrote: Thanks Norman, Will try to see this as soon as I can get some time from paid job :) cheers ashish On Mon, Apr 5, 2010 at 1:27 PM, Norman Maurer norman.mau...@googlemail.com wrote: Hi Ashish, thx for your reply. The problem is sometimes it takes sometime to get the OOM, other times it happens within one minute. One of our users reported to me that it happened within 1 minute (10 seconds and 30 seconds), after he switched to MINA trunk. So it seems it is faster reproducable with MINA trunk. I uploaded the screenshot of Eclipse Memory Analyzer to : http://people.apache.org/~norman/oom/Screenshot-Eclipse_Memory_Analyzer.png This was the one we got with MINA trunk. So you it again shows all the memory allocated by MINA. To reproduce the OOM you must perform the following steps: 1) svn checkout http://svn.apache.org/repos/asf/server/trunk james-trunk 2) cd james-trunk 2) mvn clean package 3) tar xfvz spring-deployment/target/james-spring-deployment-3.0-M1-bin.tar.gz 4) cd james-string-deploymnet-3.0-M1/bin 5) sudo ./run.sh 6) telnet localhost 4555 (user root, pass root) 7) adduser test test 8) quit Now bomb the server with emails. You can use for example smtp-source for it (wich is included in postfix) 9) smtp-source -s 20 -l 10100 -m 20 -c -f t...@test.de -t test localhost:25 This will bomb the servers with 20 emails with 20 concurrent sessions. Hope this helps, Bye, Norman 2010/4/5 Ashishpaliwalash...@gmail.com: On Sun, Apr 4, 2010 at 12:59 AM, Norman Maurernor...@apache.org wrote: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc Norman, Is there a way to reproduce this? I just want to try it out myself and debug. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
JAMES OOM, maybe the cause of MINA ?
Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: JAMES OOM, maybe the cause of MINA ?
I should think of the attachment stripping on the ml.. You can find the screenshots here: http://people.apache.org/~norman/oom/ Bye, Norman 2010/4/3 Norman Maurer nor...@apache.org: Hi all, we at JAMES use MINA since a while now for our socket stuff. After doing some stress tests we encountered some OOM Exceptions. The same was seen by one of our users which use JAMES trunk. After debugging stuff it seems like the cause of the OOM is MINA. We took some heap which shows that MINA is takin the most memory. The class which shows the memory usage was: org.apache.mina.filter.codec.ProtocolCodecFilter$EncodedWriteRequest I attach the screnshots which shows the problem. Any idea ? We are using MINA-2.0.0-RC1. Maybe I'm wrong and the whole OOM is not related to MINA at all.. Thanks, Norman Ps: Please keep server-dev in the cc - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
