On Feb 16, 2006, at 7:45 AM, Denis Frolov wrote:
My configuration is: Mac OS X Server 10.3.9 with WebObjects
Deployment 5.2.3
and Apache 1.3.33
From time to time this system runs to a problem with growing number
of httpd
processes being in CLOSE_WAIT state which eventually leads to
denial of http
service due to the limit of maximum simultaneous connections of web
service.
The output of netstat command in such situations looks like this:
tcp4 0 0 server.80 remote_client1.high_port
CLOSE_WAIT
tcp4 0 0 server.80 remote_client2.high_port
CLOSE_WAIT
...
I've tried to track down the requests of one of the remote clients by
looking at the logs by it's ip and found out that this request was
made 15
hours before the problem. This request was made during high-load on
server
(backup which involved copying of large files) and it took about 3
minutes
for WO application instance to process this request. The processing
was
followed by "Broken pipe" exception as expected:
2006-02-12 10:13:02,001 WARN AppName[29116:2010 47] [WorkerThread79]
(ERXNSLogLog4jBridge.java:41) - <WOWorkerThread id=79
socket=Socket[addr=/127.0.0.1,port=57708,localport=2010]> Exception
while
sending response: java.net.SocketException: Broken pipe
Naturally, this time was greater than the timeout defined in "HTTP
Adaptor
Settings" in Monitor, so remote client got a redirect to
"Redirection URL"
defined in Monitor:
server_name server_ip - - [12/Feb/2006:10:03:31 +0300] "GET /
requestUrl
HTTP/1.0" 302 0 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)" -
The steps outlined above presumably lead to http process getting
stuck in
CLOSE_WAIT state for 15 hours. Later when httpd service became
unavailble it
was manually killed along with others by apachectl restart.
Right now I suspect that other requests from ip addresses in
netstat output
have similar history which means that long responses of application
during
high load lead to rapid growth of the number of httpd processes and
denial
of service.
Can this be a problem of WebObjects Adaptor? What are the possible
solutions?
Hi Denis,
The bug you describe is new to me, but 5.2.3 is known to have other
bugs where files are left in the CLOSE_WAIT state. The solution is
to upgrade to 5.2.4 or 5.3.
In the form of this bug that I know, wotaskd got choked up by
CLOSE_WAITs and had to be restarted nightly. If upgrading is not an
option, then you might look at automatically restarting Apache (and/
or wotaskd) every night.
sacha
--
Sacha Michel Mallais Senior Developer / President
Global Village Consulting Inc. http://www.global-village.net/
PGP Key ID: 7D757B65 AIM: smallais
Bottom posting is your friend:
http://www.river.com/users/share/etiquette/
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list ([email protected])
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/archive%40mail-archive.com
This email sent to [email protected]
