Re: WS-Addressing OutOfMemoryErrors under load
Hi Kyle,
I committed fixes for the oneway and server-side memory leaks yesterday
(r610027 r610060), so these issues should now be resolved.
I've also looked into the intermittent request timeout issue that you
encountered.
This appears to be a race condition in the handling of the decoupled
response (decoupled in the sense that its sent over a *separate*
server-client HTTP connection). The indeterminacy seems to be related
to the the extent that the decoupled HTTP connection is kept-alive and
the number of chunked responses sent over the connection.
Now I haven't tied it down as yet to a CXF bug or an issue in the Jetty
stack, but I have noticed that turning off chunking in the server-side
config is an effective work-around.
You can apply this work-around to your server.xml as follows below.
Cheers,
Eoghan
beans xmlns=http://www.springframework.org/schema/beans;
xmlns:cxf=http://cxf.apache.org/core;
xmlns:wsa=http://cxf.apache.org/ws/addressing;
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xmlns:http=http://cxf.apache.org/transports/http/configuration;
xsi:schemaLocation=
http://cxf.apache.org/core http://cxf.apache.org/schemas/core.xsd
http://cxf.apache.org/transports/http/configuration
http://cxf.apache.org/schemas/configuration/http-conf.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd;
cxf:bus
cxf:features
wsa:addressing/
/cxf:features
/cxf:bus
http:conduit name=*.http-conduit
http:client AllowChunking=false/
/http:conduit
/beans
Kyle Sampson wrote:
Actually, you're right about the test that used greetMeOneWay(). For some
reason I was thinking it would wait until the server responded. It would
make sense for one-way calls to not clean up by the time the client method
call returns, since it's working in the background. 760 attempts does seem
to be pretty fast for it to fail though. For the record, I'm using the
default heap size (64MB). Should it really use this much memory, or is
something else happening here?
The sayHi() calls on the other hand are not one-way. I would expect that
after the response comes back from the server, it doesn't need to hold
anything in memory any longer, but the server seems to be increasing the
heap size over time. The constant random halts between some calls seem
strange too. Often it would halt for 20-30s on after sending only 20
messages, then resume only to halt again at some later point. The amount of
time it halts and when they occur vary.
Thanks again for your help.
Glen Mazza-2 wrote:
Am Freitag, den 04.01.2008, 18:19 -0800 schrieb Kyle Sampson:
I'm currently using CXF 2.0.3. While playing around with WS-Addressing,
I've
noticed a number of errors that occur when sending lots of messages.
Here, lots of messages = messages being sent in a while(true) loop
I've
been able to reproduce them by making minor changes to the ws_addressing
sample provided with CXF.
During these tests, I turned off logging for anything less than WARNING.
In
Client.java, I changed the following lines:
implicitPropagation(port);
explicitPropagation(port);
implicitPropagation(port);
to the following:
int i = 0;
while(true) {
port.greetMeOneWay(Kyle);
i++;
if(i % 10 == 0) {
System.out.println(Sent + i + messages.);
}
}
Is there a SOAP client from any web service stack (Metro? Axis?
Spring-WS?) that *could* support nonstop web service calls at the rate
of a while(true) loop? Had CXF failed on your hardware at 7600 or
76,000 attempts instead of just 760, would you also be sending this
email noting a problem, or would that have been within acceptable bounds
of failure?
I'm not sure the problem you're reporting is:
1.) Given our hardware, and CXF's architecture, CXF failed far too soon
within a while(true) loop; i.e., tests with Axis or Metro gave a far
greater number of calls before failing. That would be a legitimate
concern for us to look into.
or
2.) CXF's client and server failed at *anytime* during a while(true)
loop of web service calls--i.e., you'd be sending this email if CXF
clients had failed at iteration 760,000. In this case, before I can be
convinced this is an error, I would first need to see Axis or Metro
clients be able to handle such while(true) rate calls indefinitely.
After calling greetMeOneWay() around 760 times in this way, I get an
OutOfMemoryError. I profiled the client and server using YourKit, and
both
the client and server seem to have a memory leak -- the client runs out
of
memory first. The problem seems to be MAPCodec.uncorrelatedExchanges map
--
it's using nearly all the heap memory available.
Is that necessarily *leaking*, or just the saving of state for each of
the hundreds of requests waiting for a response?
Regards,
Glen
IONA Technologies PLC
Re: WS-Addressing OutOfMemoryErrors under load
Actually, you're right about the test that used greetMeOneWay(). For some
reason I was thinking it would wait until the server responded. It would
make sense for one-way calls to not clean up by the time the client method
call returns, since it's working in the background. 760 attempts does seem
to be pretty fast for it to fail though. For the record, I'm using the
default heap size (64MB). Should it really use this much memory, or is
something else happening here?
The sayHi() calls on the other hand are not one-way. I would expect that
after the response comes back from the server, it doesn't need to hold
anything in memory any longer, but the server seems to be increasing the
heap size over time. The constant random halts between some calls seem
strange too. Often it would halt for 20-30s on after sending only 20
messages, then resume only to halt again at some later point. The amount of
time it halts and when they occur vary.
Thanks again for your help.
Glen Mazza-2 wrote:
Am Freitag, den 04.01.2008, 18:19 -0800 schrieb Kyle Sampson:
I'm currently using CXF 2.0.3. While playing around with WS-Addressing,
I've
noticed a number of errors that occur when sending lots of messages.
Here, lots of messages = messages being sent in a while(true) loop
I've
been able to reproduce them by making minor changes to the ws_addressing
sample provided with CXF.
During these tests, I turned off logging for anything less than WARNING.
In
Client.java, I changed the following lines:
implicitPropagation(port);
explicitPropagation(port);
implicitPropagation(port);
to the following:
int i = 0;
while(true) {
port.greetMeOneWay(Kyle);
i++;
if(i % 10 == 0) {
System.out.println(Sent + i + messages.);
}
}
Is there a SOAP client from any web service stack (Metro? Axis?
Spring-WS?) that *could* support nonstop web service calls at the rate
of a while(true) loop? Had CXF failed on your hardware at 7600 or
76,000 attempts instead of just 760, would you also be sending this
email noting a problem, or would that have been within acceptable bounds
of failure?
I'm not sure the problem you're reporting is:
1.) Given our hardware, and CXF's architecture, CXF failed far too soon
within a while(true) loop; i.e., tests with Axis or Metro gave a far
greater number of calls before failing. That would be a legitimate
concern for us to look into.
or
2.) CXF's client and server failed at *anytime* during a while(true)
loop of web service calls--i.e., you'd be sending this email if CXF
clients had failed at iteration 760,000. In this case, before I can be
convinced this is an error, I would first need to see Axis or Metro
clients be able to handle such while(true) rate calls indefinitely.
After calling greetMeOneWay() around 760 times in this way, I get an
OutOfMemoryError. I profiled the client and server using YourKit, and
both
the client and server seem to have a memory leak -- the client runs out
of
memory first. The problem seems to be MAPCodec.uncorrelatedExchanges map
--
it's using nearly all the heap memory available.
Is that necessarily *leaking*, or just the saving of state for each of
the hundreds of requests waiting for a response?
Regards,
Glen
--
View this message in context:
http://www.nabble.com/WS-Addressing-OutOfMemoryErrors-under-load-tp14628989p14673865.html
Sent from the cxf-user mailing list archive at Nabble.com.
Re: WS-Addressing OutOfMemoryErrors under load
Am Freitag, den 04.01.2008, 18:19 -0800 schrieb Kyle Sampson:
I'm currently using CXF 2.0.3. While playing around with WS-Addressing, I've
noticed a number of errors that occur when sending lots of messages.
Here, lots of messages = messages being sent in a while(true) loop
I've
been able to reproduce them by making minor changes to the ws_addressing
sample provided with CXF.
During these tests, I turned off logging for anything less than WARNING. In
Client.java, I changed the following lines:
implicitPropagation(port);
explicitPropagation(port);
implicitPropagation(port);
to the following:
int i = 0;
while(true) {
port.greetMeOneWay(Kyle);
i++;
if(i % 10 == 0) {
System.out.println(Sent + i + messages.);
}
}
Is there a SOAP client from any web service stack (Metro? Axis?
Spring-WS?) that *could* support nonstop web service calls at the rate
of a while(true) loop? Had CXF failed on your hardware at 7600 or
76,000 attempts instead of just 760, would you also be sending this
email noting a problem, or would that have been within acceptable bounds
of failure?
I'm not sure the problem you're reporting is:
1.) Given our hardware, and CXF's architecture, CXF failed far too soon
within a while(true) loop; i.e., tests with Axis or Metro gave a far
greater number of calls before failing. That would be a legitimate
concern for us to look into.
or
2.) CXF's client and server failed at *anytime* during a while(true)
loop of web service calls--i.e., you'd be sending this email if CXF
clients had failed at iteration 760,000. In this case, before I can be
convinced this is an error, I would first need to see Axis or Metro
clients be able to handle such while(true) rate calls indefinitely.
After calling greetMeOneWay() around 760 times in this way, I get an
OutOfMemoryError. I profiled the client and server using YourKit, and both
the client and server seem to have a memory leak -- the client runs out of
memory first. The problem seems to be MAPCodec.uncorrelatedExchanges map --
it's using nearly all the heap memory available.
Is that necessarily *leaking*, or just the saving of state for each of
the hundreds of requests waiting for a response?
Regards,
Glen
WS-Addressing OutOfMemoryErrors under load
I'm currently using CXF 2.0.3. While playing around with WS-Addressing, I've
noticed a number of errors that occur when sending lots of messages. I've
been able to reproduce them by making minor changes to the ws_addressing
sample provided with CXF.
During these tests, I turned off logging for anything less than WARNING. In
Client.java, I changed the following lines:
implicitPropagation(port);
explicitPropagation(port);
implicitPropagation(port);
to the following:
int i = 0;
while(true) {
port.greetMeOneWay(Kyle);
i++;
if(i % 10 == 0) {
System.out.println(Sent + i + messages.);
}
}
After calling greetMeOneWay() around 760 times in this way, I get an
OutOfMemoryError. I profiled the client and server using YourKit, and both
the client and server seem to have a memory leak -- the client runs out of
memory first. The problem seems to be MAPCodec.uncorrelatedExchanges map --
it's using nearly all the heap memory available.
I tried changing port.greetMeOneWay(Kyle) to port.sayHi() (which is not a
one-way operation). Every once in a while (around every 100 messages), it
will halt and eventually display the following:
Jan 4, 2008 6:12:00 PM org.apache.cxf.endpoint.ClientImpl waitResponse
WARNING: Timed out waiting for response to operation
{http://apache.org/hello_world_soap_http}sayHi.
YourKit indicates no memory leak on the client, but the server has the same
memory leak the previous example had.
Anyone have any ideas what's happening here? Thank you for your time.
--
View this message in context:
http://www.nabble.com/WS-Addressing-OutOfMemoryErrors-under-load-tp14628989p14628989.html
Sent from the cxf-user mailing list archive at Nabble.com.
