I had an issue with the standard memory manager on Solaris for 7.1 a while ago... I am re-posting for your info. Like Axton said, it may be worthwhile changing your memory manager to "libumem" and re-trying...
Terry
REPOST:
libumem, ITSM 7.0.2 and Solaris Load Testing
Hello everyone:
We are currently running ARS 7.1, ITSM 7.0.2 (latest patch) Incident/Change/CMDB/Problem on a high end Solaris box running Solaris 10 with a DB2 database. We had been conducting stress testing of the application utilizing Loadrunner (yes, we are aware of the limitations of this tool, but were able to get around them) and were previously experiencing exponential degradation of response times once a fixed number of users were logged into the system and performing certain functions (Note: These tests were with the WUT, not MidTier). These tests and results were both repeatable and consistent.
After working with Remedy engineers on this problem and analyzing various log files and pstack output on the server, they suggested that we replace the default Solaris memory manager with "libumem" as there was extreme memory heap contention by the arsystem process. Believe it or not, this has fixed our performance issue and the application now scales to the desired number of users without any known problems to date. With the previous memory manager, we could not get over 130 users logging in over an hour. With the new memory manager, we were able to get 300+ logged in without the previously experienced exponential degradation.
We will continue to do further load testing, however, I would be interested in hearing from anyone out there who has experience using "libumem" on your server to any capacity (development, testing, or production) Have you come across any issues or hints regarding it's use?
Thank you...
Terry
On Apr 6, 2010, Robert Halstead <[email protected]> wrote:
** Hey all,
We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted.
I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box?
I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing?
I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System.
Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists.
Our server specs are the following:
System Configuration: Sun Microsystems sun4u Sun Fire V210
System clock frequency: 167 MHZ
Memory size: 4GB
==================================== CPUs ====================================
E$ CPU CPU
CPU Freq Size Implementation Mask Status Location
--- -------- ---------- --------------------- ----- ------ --------
0 1002 MHz 1MB SUNW,UltraSPARC-IIIi 2.4 on-line MB/P0
1 1002 MHz 1MB SUNW,UltraSPARC-IIIi 2.4 on-line MB/P1
AR System 7.5 patch 004
Apache Tomcat 5.5.28 / Midtier 7.5 patch 004
If you guys need more server specs let me know. We are trying to replicate the issue but we are unsure how it happens and don't really know where to start.
Thanks for the help.
--
"A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows.
The ignoramus may be saved, but the fool knows that he is doomed."
Bob Halstead
_attend WWRUG10 www.wwrug.com ARSlist: "Where the Answers Are"_
