More detail, as requested. I am using JAMES 2.0a2, with the latest JDK 1.3
for HP-UX.
When I use 200+ threads to open connections to JAMES, the last few throw a
NoRouteToHostException during their connection attempt. The exact number
of connections required is not constant, but is usually less than 200 (and
sometimes as low as 100). JAMES continues to serve any requests that do
get through after that point. (If I use 1000 threads, about 600 of them
succeed.)
I suspect that the NoRouteToHostException being thrown when the server is
overloaded is a quirk of the HP JVM and/or TCP/IP stack. Other JVM's
and/or IP stacks may react differently. Essentially the trouble is a
connect timeout due to a backlog of connection requests.
Here is the stacktrace that I get:
javax.mail.SendFailedException: Sending failed;
nested exception is:
javax.mail.MessagingException: Could not connect to SMTP host: pluto,
port: 7625;
nested exception is:
java.net.NoRouteToHostException: Operation timed out: no further
information
at javax.mail.Transport.send0(Transport.java:219)
at javax.mail.Transport.send(Transport.java:81)
at
nz.co.vodafone.smtpsms.testharness.SmtpToSmsTest.sendMailToSms(SmtpToSmsTest.java:69)
at
nz.co.vodafone.smtpsms.testharness.SmtpToSmsTest.sendAndWait(SmtpToSmsTest.java:92)
at
nz.co.vodafone.smtpsms.testharness.SmtpToSmsTest.sendAndWait(SmtpToSmsTest.java:105)
at
nz.co.vodafone.smtpsms.testharness.SmtpToSmsTest.access$2(SmtpToSmsTest.java:10)
at
nz.co.vodafone.smtpsms.testharness.SmtpToSmsTest$5.run(SmtpToSmsTest.java:460)
at java.lang.Thread.run(Thread.java:484)
I have measured the rate at which JAMES is capable of accepting new
connections and receiving a single (short) mail message at around 10
messages/second. 10 messages/second and 200 simultaneous connections is
pretty low capacity (especially when my target performance requirement is
100 messages/second).
I have taken a quick look at some of the Avalon ConnectionManager source.
It seems to use a single thread to loop around calling accept, creating a
ConnectionHandler and running the ConnectionHandler in a new thread.
Perhaps this could be improved by increasing the backlog on the
ServerSocket, and/or by using several threads in the ServerSocket.accept()
loop.
I am considering writing my own ConnectionManager component to try to
improve performance. (Possibly based on the existing avalon component, or
perhaps just implementing the appropriate interfaces.) I there anywhere
that I can get the source for the avalon version used by JAMES, without
going through CVS? (I am blocked from using CVS by a firewall.)
Hope this helps,
Cheers
ADK
--------------------------------------------
There is no magic.
Serge
Knystautas To: James Users List
<[EMAIL PROTECTED]>
<sergek@lokite cc:
ch.com> Subject: Re: James stops responding after
many TCP connections to SMTP service -
DOS attack possible?
01/05/2002
07:12
Please respond
to "James
Users List"
Can you go into more detail about the "no route to host" exception?
Also, what version of James are you using, and are you using file system
or db repositories?
--
Serge Knystautas
Loki Technologies - Unstoppable Websites
http://www.lokitech.com/
[EMAIL PROTECTED] wrote:
> Further to this comment, I have been performing a little load testing on
> JAMES. (The load test is not really aimed at testing JAMES - just the
> throughput of my app, of which JAMES is only a part.) One thing I have
> discovered is that I can flood a JAMES box so that I get "no route to
host"
> without difficulty.
>
> I am running James on a quite modest HP 9000 (1 CPU, 1GB RAM). I have a
> single client machine (PIII 733, 384MB RAM) separated from JAMES by a
> broadband WAN link (not sure how fast). My client app opens a number of
> threads, each of which sends one short message to JAMES via the JavaMail
> API. I find that 200 threads are all that is necessary for "no route to
> host" to come back.
>
> Has anyone else struck this? Does anyone have any performance tuning
tips?
> I have tried adjusting the number of threads allocated to both the thread
> manager and the spool manager, but that seems to make no difference.
(Not
> surprising, as the spoolmanager operates asynchronously to the connection
> handler anyway.)
>
> Cheers
>
> ADK
>
> --------------------------------------------
>
> There is no magic.
>
>
>
> Brad Wallace
> <babafloolee@minds To:
[EMAIL PROTECTED]
> pring.com> cc:
> Subject: James stops
responding after many TCP connections to SMTP service - DOS
> 24/04/2002 11:37 attack possible?
> Please respond to
> "James Users List"
>
>
>
>
>
>
> Hi All - I'm running James in a load balanced environment (required by
> our policies... ;-) One side effect of this is that the load balancers
> establish repeated TCP connections to the SMTP service and then break
> them (TCP RST or FIN I expect). They do this to verify that the service
> is listening and accepting inbound connections. Right now I'm running
> in debug mode, so these connections are being logged - I've included
> samples below.
>
> Here's the problem - after james has been running for some period of
> time less than 1 week, I get the following on stdout:
>
>
> Logging Error: Unknown error writing event.
> java.lang.OutOfMemoryError
> <<no stack trace available>>
>
>
>
> and then james stops responding. So far I've had to restart james each
> time to get this fixed. I'll admit that the frequent TCP connection
> setups and breakdowns are a quirk of this type of environment, but it
> concerns me also b/c this seems like an easy DOS (Denial of Service)
> attack on a james server - simply establish and break TCP connections
> every 5 seconds or so for a while and it will hang until someone
> restarts it - it's not yet clear exactly how long you have to repeat
> these TCP connections for, but it's somewhere in the hours - days range,
> but not weeks. And every 5 seconds probably isn't frequent enough to
> trigger most firewall's DOS filters, so this is pretty likely to get
> through.
>
>
> I'm running:
>
> James 2.0a2 with just the SMTP listener (no pop3, nntp, remote admin,
> etc)
> java-1.3.1_02
> Solaris 7
>
>
> Thanks much for any input.
>
>
> -Brad
>
>
>
> Logs Excerpts:
>
>
> smtpserver.log:
>
> Tue Apr 23 22:02:58 GMT 2002 [INFO ] (smtpserver): Hello Name is:
> <local machine name>
> Tue Apr 23 22:02:58 GMT 2002 [DEBUG ] (smtpserver): Max message size
> is: 0
> Tue Apr 23 22:02:58 GMT 2002 [INFO ] (smtpserver): Connection from
> <load balancer IP> (<load balancer IP>)
> Tue Apr 23 22:02:58 GMT 2002 [DEBUG ] (smtpserver): Socket to <load
> balancer IP> closed remotely.
> java.net.SocketException: Connection reset by peer: Connection reset by
> peer
> at java.net.SocketInputStream.socketRead(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:90)
> at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:186)
> at
> java.io.BufferedInputStream.read(BufferedInputStream.java:204)
> at java.io.DataInputStream.readLine(DataInputStream.java:449)
> at
>
org.apache.james.smtpserver.SMTPHandler.handleConnection(SMTPHandler.java:163)
>
> at
>
org.apache.avalon.cornerstone.blocks.connection.ConnectionRunner.run(Connection.java:163)
>
> Tue Apr 23 22:02:58 GMT 2002 [ERROR ] (smtpserver): Connection timeout
> on socket
>
>
> connections.log:
>
> Tue Apr 23 22:11:34 GMT 2002 [DEBUG ] (connections): Starting
> connection on Socket[addr=<load balancer IP>/<load balancer
> IP>,port=<source port>,localport=<james smtp port>]
> Tue Apr 23 22:11:34 GMT 2002 [DEBUG ] (connections): Ending connection
> on Socket[addr=<load balancer IP>/<load balancer IP>,port=<source
> port>,localport=<james smtp port>]
> Tue Apr 23 22:11:36 GMT 2002 [ERROR ] (connections): Exception
> accepting connection
> java.net.SocketException: Software caused connection abort
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:468)
> at java.net.ServerSocket.implAccept(ServerSocket.java:243)
> at java.net.ServerSocket.accept(ServerSocket.java:222)
> at
>
org.apache.avalon.cornerstone.blocks.connection.Connection.run(Connection.java:93)
>
> at
>
org.apache.avalon.excalibur.thread.impl.ExecutableRunnable.execute(ExecutableRunnable.java:47)
>
> at
>
org.apache.avalon.excalibur.thread.impl.WorkerThread.run(WorkerThread.java:80)
>
> Tue Apr 23 22:11:38 GMT 2002 [DEBUG ] (connections): Starting
> connection on Socket[addr=<load balancer IP>/<load balancer
> IP>,port=<source port>,localport=<james smtp port>]
> Tue Apr 23 22:11:38 GMT 2002 [DEBUG ] (connections): Ending connection
> on Socket[addr=<load balancer IP>/<load balancer IP>,port=<source
> port>,localport=<james smtp port>]
> Tue Apr 23 22:11:40 GMT 2002 [ERROR ] (connections): Exception
> accepting connection
>
> --
> To unsubscribe, e-mail: <
mailto:[EMAIL PROTECTED]
>
> For additional commands, e-mail: <
mailto:[EMAIL PROTECTED]
>
>
>
>
>
>
>
>
-----------------------------------------------------------------------------------------------
> Have you seen our website?.... http://www.vodafone.co.nz
>
> CAUTION: This correspondence is confidential and intended for the named
recipient(s) only.
> If you are not the named recipient and receive this correspondence in
error, you must not copy,
> distribute or take any action in reliance on it and you should delete it
from your system and
> notify the sender immediately. Thank you.
>
> Unless otherwise stated, any views or opinions expressed are solely those
of the author and do
> not represent those of Vodafone New Zealand Limited.
>
> Vodafone New Zealand Limited
> 21 Pitt Street, Private Bag 92161, Auckland, 1020, New Zealand
> Telephone + 64 9 357 5100
> Facsimile + 64 9 377 0962
--
To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]
>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]
>
-----------------------------------------------------------------------------------------------
Have you seen our website?.... http://www.vodafone.co.nz
CAUTION: This correspondence is confidential and intended for the named recipient(s)
only.
If you are not the named recipient and receive this correspondence in error, you must
not copy,
distribute or take any action in reliance on it and you should delete it from your
system and
notify the sender immediately. Thank you.
Unless otherwise stated, any views or opinions expressed are solely those of the
author and do
not represent those of Vodafone New Zealand Limited.
Vodafone New Zealand Limited
21 Pitt Street, Private Bag 92161, Auckland, 1020, New Zealand
Telephone + 64 9 357 5100
Facsimile + 64 9 377 0962
--
To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]>
For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
