Yes, Each hung thread shows the exact same back trace. And each was spawned by
a request to the same CGI script but with differing arguments.
There is an LDAP login requirement for this tool. Not sure if that is
interesting as many other tools on this same server require LDAP authentication
as well.
MJ
On Thursday, June 18, 2015 7:56 AM, Jeff Trawick <[email protected]> wrote:
On Wed, Jun 17, 2015 at 8:51 PM, Mark Jacquet <[email protected]>
wrote:
Just another oddity to add to the issue.
Overnight several more hung threads appeared and the load on the system had
jumped into the mid 20's.After killing these the load did not drop. Looking at
the list of running processes I found httpd's running,spawned from the original
root httpd process that *were not even displayed* in the scoreboard!! After
killing these hidden zombies off the load dropped again.
What's common about the processes? Similar backtrace to the first one posted?
So now I have to catch and kill two types: Zombies on the scoreboard and hidden
zombies.
And this is cute. Some times the zombies hang around so long that when the
system gets back to creating a new process for slot #1, if the zombie was
originally in that slot it is displayed their along with it's brothers for the
new process:
"scoreboard squatting"
e.g. Note process 19597 below
1-0166310/33/1320_131.22202255280.01.6035.7910.172.91.217newyahoo.oak.sap.corp:80NULL1-0166310/18/1087_105.88340736980.00.6926.6510.172.240.113www-dse.oak.sap.corp:80GET
/cgi-bin/websql/websql.dir/QTS/bugsheetcont.hts?bugid=741331-0166310/11/1178_76.49589542980.00.5634.7810.172.91.92newyahoo.oak.sap.corp:80NULL1-0166310/32/1295_92.17425417130.04.0342.0710.172.240.113newyahoo.oak.sap.corp:80NULL1-0195970/26/1319W35.552441700.00.5437.1010.172.248.87www-rev.oak.sap.corp:80GET
/cgi-bin/rev.cgi?action=105;id=58037
HTTP/1.11-0166310/12/1427_18.41794100.00.14238.5210.172.240.113newyahoo.oak.sap.corp:80NULL1-0166310/27/1442_30.67719695430.00.7835.0710.172.85.9newyahoo.oak.sap.corp:80NULL1-0166310/19/784_10.70940630.00.4520.9510.172.246.203newyahoo.oak.sap.corp:80NULL1-0166310/8/1034_2.86103144630.00.0124.0410.172.90.155newyahoo.oak.sap.corp:80NULL2-0-0/0/99.58.943145013820.00.002.1510.136.66.135newyahoo.oak.sap.corp:80NULL2-0-0/0/82.2181.923144824390.00.001.4810.162.65.165www-dse.oak.sap.corp:80POST
/cgi-bin/websql/websql.dir/QTS/bugsescalated.pl?product=AN2-0-0/0/162.2027.12314509350.00.003.3610.50.3.99newyahoo.oak.sap.corp:80NULL2-0-0/0/576.1704.40314504100.00.0013.3810.172.240.113newyahoo.oak.sap.corp:80NULL2-0-0/0/928.1295.363145029750.00.0024.3810.50.17.221newyahoo.oak.sap.corp:80NULL2-0-0/0/852.1798.52314503810.00.0020.7210.162.65.165newyahoo.oak.sap.corp:80NULL2-0-0/0/1084.551.293145022210.00.0026.5210.176.138.162newyahoo.oak.sap.corp:80NULL2-0-0/0/1180.385.833145019630.00.0034.3110.162.65.197newyahoo.oak.sap.corp:80NULL2-0-0/0/50.50.713145000.00.001.6210.58.181.166www-rev.oak.sap.corp:80GET
/cgi-bin/rev.cgi?action=105;id=58051
HTTP/1.12-0137610/12/1078W58.803489600.00.1031.6710.172.107.38www-rev.oak.sap.corp:80POST
/cgi-bin/rev.cgi
HTTP/1.12-0-0/0/1075.1061.5331450790.00.0031.6510.172.90.88newyahoo.oak.sap.corp:80GET
/server-status
HTTP/1.12-0-0/0/1362.46.803145080.00.0039.7210.172.107.38www-rev.oak.sap.corp:80POST
/cgi-bin/rev.cgi
HTTP/1.12-0-0/0/1142.56.693145011490.00.0035.2210.172.240.113newyahoo.oak.sap.corp:80NUL
Slot #2 currently not being used (still has zombie)
MJ
Mj
On Tuesday, June 16, 2015 5:42 PM, Mark Jacquet
<[email protected]> wrote:
Upgrade as in Apache upgrade or Solaris 5.10 patch upgrad? :)
Apache is all new of course 2.4.12 with the latest add on sources (apr, pcre,
etc)The bad news is the OS is not at all up to date. And for reasons I have no
control over, I cannot patch.So if this is an OS issue then ......
I seem to be running with the Sun Native LDAP SDK. Would building against
different LDAP source help? (Open LDAP)?
Long term plan -> moving all Apache servers to Linux
Mj
On Tuesday, June 16, 2015 5:31 PM, Eric Covener <[email protected]> wrote:
On Tue, Jun 16, 2015 at 8:23 PM, Mark Jacquet
<[email protected]> wrote:
> So do you think this hang is related to the native LDAP lib code?
It is possible but IMO not very likely. It has to corrutp memory just
enough to put a looping structure in apr_rmm. What's your upgrade
history like?
--
Eric Covener
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
--
Born in Roswell... married an alien...
http://emptyhammock.com/
