Hello, we are having an issue with Axis allocating multiple GB of memory
and crashing our servers.
It seems to be specific to HTTPS connections, and the server being under
load.
Using dtrace, we have been able to confirm that Axis is making
extremely large memory allocations (i.e. 2 GB in a single call).
We have been running successfully in a variety of dev, test,
performance, and pre-production environments for over a year without any
problems.
However, we recently changed to using HTTPS connections for this link,
and during a recent deployment, the memory usage on these processes
spiked from ~100MB each to 1-8 GB each, the 35 GB of swap space on the
box was all consumed, and everything came down. This happens all
happens within a few minutes of application startup.
We've been able to recreate the scenario in only one test environment so
far, but not in our dev environments.
The web service that is being invoked is a java process running in a
JBoss server on the same box. There are two APIs that are being
invoked, one of which returns a ~ 500k XML result. We've run a variety
of tests on the web service, and see no evidence of malformed XML or
other strangeness.
The server is a Solaris 10, SPARC chipset. We've tested on a variety of
other environments and not been able to reproduce this problem (Red hat
linux x86-64 and a Solaris 10 Sparc on a T1 'cool threads'
architecture). So, we're not sure if it's platform-specific, or just a
random memory bug.
Here are the results of several test scenarios:
One process using HTTP : OK
30 processes using HTTP : OK
One process using HTTPS: OK
30 processes using HTTPS: Problem
These processes don't talk to each other, so there should be no effect
of running 30, other than the box is busier, and the web service is
busier.
The problem occurs on some subset of the servers. Generally 1-5 servers
will show this huge memory growth before the box runs out of
memory/swap.
Looking at the size of the memory allocations inside of Axis, we see:
2 GB (2147483648 bytes) which is 0x80 00 00 00 or 1000 0000 0000 0000
0000 0000 0000 0000 binary
1 GB (1073741824 bytes) which is 0x40 00 00 00 or 0100 0000 0000 0000
0000 0000 0000 0000 binary
500 MB (536870912 bytes) which is 0x20 00 00 00 or 0010 0000 0000 0000
0000 0000 0000 0000 binary
256 MB (268435456 bytes) which is 0x10 00 00 00 or 0001 0000 0000 0000
0000 0000 0000 0000 binary
128 MB (134217728 bytes) which is 0x08 00 00 00 or 0000 1000 0000 0000
0000 0000 0000 0000 binary
Any suggestions?
Technical Details below.
==================================================
Using Axis 1.6.0.
64-bit application.
uname -a
SunOS xyz 5.10 Generic_138888-03 sun4u sparc SUNW,Sun-Fire-V490
psrinfo -v
Status of virtual processor 0 as of: 09/27/2012 13:22:35
on-line since 01/04/2012 14:00:37.
The sparcv9 processor operates at 2100 MHz,
and has a sparcv9 floating point processor.
... (omitted for brevity)
Excerpt from the dtrace output showing individual malloc calls, and the
associated call stack.
--------------------------------------
One allocation of 2 GB (2,147,483,648 bytes)
libc.so.1`malloc+0x78
libaxutil.so.0`0xffffffff77c0d19c
libguththila.so.0`0xffffffff74e0952c
libguththila.so.0`0xffffffff74e05fac
libaxis2_parser.so.0`0xffffffff752067bc
libaxis2_parser.so.0`0xffffffff75203d40
libaxis2_axiom.so.0`0xffffffff77e1cd8c
libaxis2_axiom.so.0`0xffffffff77e28de0
libaxis2_axiom.so.0`0xffffffff77e266ec
libaxis2_engine.so.0`0xffffffff77a7c030
libaxis2_http_sender.so`0xffffffff7360c794
libaxis2_http_sender.so`0xffffffff7360b340
libaxis2_engine.so.0`0xffffffff77a2e3b8
libaxis2_engine.so.0`0xffffffff77a68c3c
libaxis2_engine.so.0`0xffffffff77a674fc
libaxis2_engine.so.0`0xffffffff77a6b0c0
libCseSecuritys.so`0xffffffff7d524fd0
libCseSecuritys.so`0xffffffff7d51691c
libHvfXmlAppServers.so`0xffffffff7db1c4fc
libHvfXmlAppServers.so`0xffffffff7db20840
2147483648 1
--------------------------------------
One allocation of 1 GB (1,073,741,824 bytes):
libc.so.1`malloc+0x78
libaxutil.so.0`0xffffffff77c0d19c
libguththila.so.0`0xffffffff74e0952c
libguththila.so.0`0xffffffff74e06684
libaxis2_parser.so.0`0xffffffff752067bc
libaxis2_parser.so.0`0xffffffff75203d40
libaxis2_axiom.so.0`0xffffffff77e1cd8c
libaxis2_axiom.so.0`0xffffffff77e28de0
libaxis2_axiom.so.0`0xffffffff77e266ec
libaxis2_engine.so.0`0xffffffff77a7c030
libaxis2_http_sender.so`0xffffffff7360c794
libaxis2_http_sender.so`0xffffffff7360b340
libaxis2_engine.so.0`0xffffffff77a2e3b8
libaxis2_engine.so.0`0xffffffff77a68c3c
libaxis2_engine.so.0`0xffffffff77a674fc
libaxis2_engine.so.0`0xffffffff77a6b0c0
libCseSecuritys.so`0xffffffff7d524fd0
libCseSecuritys.so`0xffffffff7d51691c
libHvfXmlAppServers.so`0xffffffff7db1c4fc
libHvfXmlAppServers.so`0xffffffff7db20840
1073741824 1
--------------------------------------
One allocation of ~ 500 MB (536,870,912 bytes):
libc.so.1`malloc+0x78
libaxutil.so.0`0xffffffff77c0d19c
libguththila.so.0`0xffffffff74e0952c
libguththila.so.0`0xffffffff74e0624c
libaxis2_parser.so.0`0xffffffff752067bc
libaxis2_parser.so.0`0xffffffff75203d40
libaxis2_axiom.so.0`0xffffffff77e1cd8c
libaxis2_axiom.so.0`0xffffffff77e28de0
libaxis2_axiom.so.0`0xffffffff77e266ec
libaxis2_engine.so.0`0xffffffff77a7c030
libaxis2_http_sender.so`0xffffffff7360c794
libaxis2_http_sender.so`0xffffffff7360b340
libaxis2_engine.so.0`0xffffffff77a2e3b8
libaxis2_engine.so.0`0xffffffff77a68c3c
libaxis2_engine.so.0`0xffffffff77a674fc
libaxis2_engine.so.0`0xffffffff77a6b0c0
libCseSecuritys.so`0xffffffff7d524fd0
libCseSecuritys.so`0xffffffff7d51691c
libHvfXmlAppServers.so`0xffffffff7db1c4fc
libHvfXmlAppServers.so`0xffffffff7db20840
536870912 1
--------------------------------------
One allocation of 268 MB (268,435,456 bytes)
libc.so.1`malloc+0x78
libaxutil.so.0`0xffffffff77c0d19c
libguththila.so.0`0xffffffff74e0952c
libguththila.so.0`0xffffffff74e06b60
libaxis2_parser.so.0`0xffffffff752067bc
libaxis2_parser.so.0`0xffffffff75203d40
libaxis2_axiom.so.0`0xffffffff77e1cd8c
libaxis2_axiom.so.0`0xffffffff77e28de0
libaxis2_axiom.so.0`0xffffffff77e266ec
libaxis2_engine.so.0`0xffffffff77a7c030
libaxis2_http_sender.so`0xffffffff7360c794
libaxis2_http_sender.so`0xffffffff7360b340
libaxis2_engine.so.0`0xffffffff77a2e3b8
libaxis2_engine.so.0`0xffffffff77a68c3c
libaxis2_engine.so.0`0xffffffff77a674fc
libaxis2_engine.so.0`0xffffffff77a6b0c0
libCseSecuritys.so`0xffffffff7d524fd0
libCseSecuritys.so`0xffffffff7d51691c
libHvfXmlAppServers.so`0xffffffff7db1c4fc
libHvfXmlAppServers.so`0xffffffff7db20840
268435456 1
--------------------------------------
One allocation of 134 MB (134,217,728 bytes)
libc.so.1`malloc+0x78
libaxutil.so.0`0xffffffff77c0d19c
libguththila.so.0`0xffffffff74e0952c
libguththila.so.0`0xffffffff74e07330
libaxis2_parser.so.0`0xffffffff752067bc
libaxis2_parser.so.0`0xffffffff75203d40
libaxis2_axiom.so.0`0xffffffff77e1cd8c
libaxis2_axiom.so.0`0xffffffff77e28de0
libaxis2_axiom.so.0`0xffffffff77e266ec
libaxis2_engine.so.0`0xffffffff77a7c030
libaxis2_http_sender.so`0xffffffff7360c794
libaxis2_http_sender.so`0xffffffff7360b340
libaxis2_engine.so.0`0xffffffff77a2e3b8
libaxis2_engine.so.0`0xffffffff77a68c3c
libaxis2_engine.so.0`0xffffffff77a674fc
libaxis2_engine.so.0`0xffffffff77a6b0c0
libCseSecuritys.so`0xffffffff7d524fd0
libCseSecuritys.so`0xffffffff7d51691c
libHvfXmlAppServers.so`0xffffffff7db1c4fc
libHvfXmlAppServers.so`0xffffffff7db20840
134217728 1
