Listers, We are having a strange problem with our mid-tier. We have a ticket (ISS03499205) open with BMC, but we would appreciate any help that we could get from the list.
Our current set up: App server 6.03.00 Patch 22 on Solaris 9 Mid-tier Two main production mid-tiers behind a load balancer Version 7.1.00 Patch 006 200812041310 Apache Tomcat/5.5.27 RHE Linux Java 1.60_06 We also have a test 7.1 mid-tier that that is not in the load balancer and a production 6.3 mid-tier and a test one that we have mainly for web services access. The production 6.3 mid-tier is on the same Solaris host as our productions app server. This set up has been stable for several months, but Wednesday we started having a problem where our version 7.1 mid-tier users could not log in. Their log in would fail with an "ARERROR [9388] Authentication failed" error event though they are using the correct username and password. They can log in via the Usertool. This started with one of our production mid-tiers and seemed to spread to the others as the night went on. Sometimes we will see errors like this in the arerror.log associated with the errors: Wed Sep 16 13:02:27 2009 HandleRPCs Wed Sep 16 13:02:27 2009 390635 : Error during symmetric key data decryption (ARERR 9004) We get the errors whether we are going through the load balancer or not. We have tried various things: - stopping and starting apache and tomcat - resetting the mid-tier password in the mid-tier and the app server - changing and resetting the mid-tier password in the mid-tier and the app server - flushing the cache on the mid-tier - stopping and starting the app server - added a ProxyFilter supplied by BMC to our mid-tier's web.xml file. - changed back and forth between allowing and requiring encryption at the AR server - enabling/disabling mid-tier side api logging None of these things really helped. However, sometimes when we stop and start tomcat (especially when adding or removing the mid-tier side ar api logging), the mid-tier will work correctly for a while, a few minutes to a few hours, then it will start generating the ARERR 9388 authentication errors again. It will start with one or two users getting the error and degrade from there with more and more of the users getting the error. As a side note, our version 6.3 mid-tier also started failing about the same time, but in a different way. After we restarted the app server, we restarted the 6.3 mid-tier and as soon as someone started to log in, the server applet quickly starts using more and more RAM over the course of a few minutes and then dies. Strangely enough, we repointed our test 6.3 mid-tier server at the production app server as a temporary work around while we are trying to figure our what is going on and it is still working, at least for now. Has anyone run into anything like this or have any suggestions? Thanks, Joe Kubasek The Ohio State University _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are"
