This happened from 9 pm to 12 am on Friday night/Saturday morning on a holiday weekend last weekend-I think that safely rules out load. If anyone has any idea what to check next, it would be greatly appreciated.
Anne Ramey *********************************** E-mail correspondence to and from this address may be subject to the North Carolina Public Records Law and may be disclosed to third parties only by an authorized State Official. From: Ramey, Anne [mailto:anne.ra...@nc.gov] Sent: Thursday, September 02, 2010 1:03 PM To: arslist@ARSLIST.ORG Cc: remedy.its.support Subject: intermittent ERROR (91): RPC call failed; ONC/RPC call timed out I am at a loss about a problem we are seeing. We've been seeing intermittent issues with our mid tier servers. We have two mid tier servers and two server grouped ARS servers. Every so often we get errors: ERROR (91): RPC call failed; ONC/RPC call timed out for some users on one of the mid-tier servers for about 2 hours, then it will clear up by itself. It's not always the same users nor always the same of the two mid-tier servers, but it is always just one mid-tier server at a time. The mid-tier are talking to a load balancer in front of the ARS servers and go through a firewall. The users going through the same firewall and load balancer to get to the ARS servers via the user tool and never see the issue. Users on the other mid-tier don't have any issues at that time. So far we have: -Turned off all other processes running on the mid-tier box (including monitoring) -Turned off TCP SACK (selective acknowledgement) -Reviewed mid-tier logs with everything on the finest level (there is nothing in the tomcat or ARS logs at the time that appears to be related) -Looked at firewall logs and network traffic -Restarted network interface on the box I am running the following config: -Linux RHEL 4 OS -ARS 7.1 p 6 mid-tier -ARS 7.1 p 11 ARS servers -CMDB 2.1 p 4 -ITSM 7.0.03 p9 This started happening a couple of months ago but has become more and more frequent. This was happening back before we upgraded the ARS servers from p 6 to p 11. It doesn't seem to be on a certain day of week or a certain time of day. It does always seem to go on for between 2 and 3 hours and clear itself up. Any idea of where to look next will be *greatly* appreciated. Anne Ramey *********************************** E-mail correspondence to and from this address may be subject to the North Carolina Public Records Law and may be disclosed to third parties only by an authorized State Official. _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug10 www.wwrug.com ARSlist: "Where the Answers Are"
