Could you upload the heap dump too ?
2010/4/24, Norman Maurer <norman.mau...@googlemail.com>: > Hi Eric, > > thx for the report. Comments are inline... > > > 2010/4/24 Eric Charles <eric.char...@u-mangate.com>: >> Hi All, >> >> I just got oom with netty impl. >> See http://apache.u-mangate.com/james/oom/20090424.png >> (http://apache.u-mangate.com/james/oom/index.html for history). >> >> Everything was fine with the trunk of last week that was really stable. >> I redeployed 3 days ago a new trunk version after a stop/start that >> showed >> ClassCastException of ImapMessage (ClassCastException was solved with the >> redeploy) > > The ClassCastException should be fixed since yesterday. This was > caused by the EntityManager got confused by entities.. > > >> >> Short infos about the today oom: >> - The logs had IOException "Bad file descriptor" (see copy of the >> exception >> after this mail) > > This error seems to get thrown when you write to a closed stream... > >> - The begining time of these exceptions in the log is 00:10, and there >> are >> numerous ones during 1 minute. >> - The heap dump automatically generated via >> -XX:+HeapDumpOnOutOfMemoryError >> -XX:HeapDumpPath=/ occured on 00:10, exactly on the same time the >> IOException "Bad file descriptor" began. > > Ok.. > >> - I forgot to lsof to see the open files... but when if the error came >> from >> too many open files, the received exception in the log would have been >> "java.io.IOException: too many open files". > > right.. > >> - Seems like an infinite loop as Norman anticipated, but now stacktraces >> from logs seem to confirm this. >> - Looking at line 150 of >> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup >> , out.write(line) seem to be the place that triggers the exception, out >> variable being = (OutputStream) >> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM); > > Good catch will see how this could happen.. > >> - The server filesystem seems sane (new hardware, running with enough >> privilege, other applications running on the same machine without any >> problem). I don't see why I have a "Bad file descriptor" >> > > See above.. > >> There are file in tmp with creation date that span 1 week (so not >> produced >> on 00:10): >> - A few messagestream*.ms >> - More rewindable*.tmp > > Will investigate .. > >> >> I restarted james. >> It still showed some issues (one account via pop3 gave errors in my >> client). >> oouch... >> >> I updated from trunk, redeployed, copied the var directory to migrate the >> account/domain/mail database, and restarted. >> Hopefully, now everything is working as before. >> >> So, we've got now a useful stacktrace that clearly shows where the oom >> "trigger" could be. >> I'm pretty sure that jconsole would have shown the same behaviour as >> http://apache.u-mangate.com/james/oom/oom.png >> >> Possible cause on the SMTP channel: >> - A weird behaviour of the smtp clients (hacking or respectfulness >> protocols >> clients) >> - A design in james causing infinite loop: thread clash (a thread >> accessing >> the same file as another thread?,...) >> - Additional James config (spamassassin or whatever) that leads to such >> behaviour >> - Hardware problem >> >> Currently, I run with default config + jdbcdomainlist. >> I except to receive another oom one of these days, unless the oom comes >> from >> an unactivated config (spamassassin or whatever). >> >> Tks, >> >> Eric >> >> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while >> processing DATA. >> java.io.IOException: Bad file descriptor >> at java.io.FileOutputStream.writeBytes(Native Method) >> at java.io.FileOutputStream.write(FileOutputStream.java:247) >> at >> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60) >> at >> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60) >> at >> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60) >> at >> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391) >> at >> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317) >> at >> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299) >> at >> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331) >> at >> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) >> at >> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502) >> at >> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583) >> at >> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119) >> at >> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) >> at >> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789) >> at >> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578) >> at org.jboss.netty.channel.Channels.close(Channels.java:1065) >> at >> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178) >> at >> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) >> at >> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619) >> at >> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) >> at >> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345) >> at >> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332) >> at >> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323) >> at >> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275) >> at >> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196) >> at >> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) >> at >> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while >> processing DATA. >> java.io.IOException: Bad file descriptor >> at java.io.FileOutputStream.writeBytes(Native Method) >> at java.io.FileOutputStream.write(FileOutputStream.java:247) >> at >> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60) >> at >> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60) >> at >> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75) >> at >> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60) >> at >> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391) >> at >> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317) >> at >> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299) >> at >> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331) >> at >> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) >> at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) >> at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796) >> at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191) >> ... >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> >> > > Bye, > Norman > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
