Hi Norman,
I launch with & (background process) so I can leave the terminal.
Would shutdown via remotemanager better ?
Tks,
Eric
On 04/24/2010 06:19 PM, Norman Maurer wrote:
Hi Eric,
doesn't CTRL-C work in the terminal you started james in ? And you
should not use kill-9<pid> because it will not let JAMES cleanup
stuff, if you want to use kill<pid> ..
Bye,
Norman
2010/4/24 Eric Charles<eric.char...@u-mangate.com>:
Hi Norman,
lsof -p<pid> | grep deleted | wc -l gives a stable number (around 20,
growing and after decreasing : cleanup of gc or os... whatever...).
No file in /tmp.
I wonder if the files I found were files left further to the few "hard-stop"
I made during last week.
Currently, to stop James, I use kill -9<pid>
Is there another cleaner way to do this ?
I saw shutdown command ("kills the current JVM (convenient when James is
run as a daemon)") in the remotemanager.
Is it different from kill -9 ? Should I use this ?
Tks,
Eric
On 04/24/2010 04:23 PM, Norman Maurer wrote:
Hi Eric,
just downloading it..
About the tread safety. I don't think so. I'm still wonder if the
exception was only thrown because of the OOM or if its the cause of
the OOM. Hopefully I will have a better view of the whole story once I
looked at the heap dump...
Bye,
Norman
2010/4/24 Eric Charles<eric.char...@u-mangate.com>:
Hi Norman,
You can download it from
http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip
"This error seems to get thrown when you write to a closed stream.."
Could it be a thread-safety problem ? (on thread using the session/state
of
another one ?)
I will also monitor with lsof -p<pid> | grep deleted | wc -l
As config change, I activated a few days ago the ValidRcptHandler
fastfail
handler.
Tks,
Eric
On 04/24/2010 03:13 PM, Norman Maurer wrote:
Could you upload the heap dump too ?
2010/4/24, Norman Maurer<norman.mau...@googlemail.com>:
Hi Eric,
thx for the report. Comments are inline...
2010/4/24 Eric Charles<eric.char...@u-mangate.com>:
Hi All,
I just got oom with netty impl.
See http://apache.u-mangate.com/james/oom/20090424.png
(http://apache.u-mangate.com/james/oom/index.html for history).
Everything was fine with the trunk of last week that was really
stable.
I redeployed 3 days ago a new trunk version after a stop/start that
showed
ClassCastException of ImapMessage (ClassCastException was solved with
the
redeploy)
The ClassCastException should be fixed since yesterday. This was
caused by the EntityManager got confused by entities..
Short infos about the today oom:
- The logs had IOException "Bad file descriptor" (see copy of the
exception
after this mail)
This error seems to get thrown when you write to a closed stream...
- The begining time of these exceptions in the log is 00:10, and there
are
numerous ones during 1 minute.
- The heap dump automatically generated via
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=ccured on 00:10, exactly on the same time the
IOException "Bad file descriptor" began.
Ok..
- I forgot to lsof to see the open files... but when if the error came
from
too many open files, the received exception in the log would have been
"java.io.IOException: too many open files".
right..
- Seems like an infinite loop as Norman anticipated, but now
stacktraces
from logs seem to confirm this.
- Looking at line 150 of
http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=up
, out.write(line) seem to be the place that triggers the exception,
out
variable being =tputStream)
session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
Good catch will see how this could happen..
- The server filesystem seems sane (new hardware, running with enough
privilege, other applications running on the same machine without any
problem). I don't see why I have a "Bad file descriptor"
See above..
There are file in tmp with creation date that span 1 week (so not
produced
on 00:10):
- A few messagestream*.ms
- More rewindable*.tmp
Will investigate ..
I restarted james.
It still showed some issues (one account via pop3 gave errors in my
client).
oouch...
I updated from trunk, redeployed, copied the var directory to migrate
the
account/domain/mail database, and restarted.
Hopefully, now everything is working as before.
So, we've got now a useful stacktrace that clearly shows where the oom
"trigger" could be.
I'm pretty sure that jconsole would have shown the same behaviour as
http://apache.u-mangate.com/james/oom/oom.png
Possible cause on the SMTP channel:
- A weird behaviour of the smtp clients (hacking or respectfulness
protocols
clients)
- A design in james causing infinite loop: thread clash (a thread
accessing
the same file as another thread?,...)
- Additional James config (spamassassin or whatever) that leads to
such
behaviour
- Hardware problem
Currently, I run with default config + jdbcdomainlist.
I except to receive another oom one of these days, unless the oom
comes
from
an unactivated config (spamassassin or whatever).
Tks,
Eric
ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
processing DATA.
java.io.IOException: Bad file descriptor
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:247)
at
org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
at
org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
at org.jboss.netty.channel.Channels.close(Channels.java:1065)
at
org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
at
org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
at
org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
at
org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
at
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
processing DATA.
java.io.IOException: Bad file descriptor
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:247)
at
org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
at
org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
at
org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
...
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
Bye,
Norman
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
