Re: Finding memory leaks in the AR System
We have apps where users query the system with webservices couple thousand times a day. It is through webservice calls. Any timedout webservice calls or webservice calls returning large data set etc...are causing memory leaks when executed over and over like several thousand times. On Apr 7, 3:28 pm, Robert Halstead badbee...@gmail.com wrote: Do you see the memory leak on just deneral usage of the midtier? Or is there a specific action to replicate? Is this through the mid-tier interface, or webservice call? On Wed, Apr 7, 2010 at 3:22 PM, patchsk vamsi...@gmail.com wrote: It is under limited availability and is going through UAT as per our support rep. We are expecting around end of this month, but she can not speak for sure as it depends on their test results. On Apr 7, 1:52 pm, Robert Halstead badbee...@gmail.com wrote: patchsk, We do utiize the midtier a lot with our custom applications and tools as that is the only cross-platform way to communicate with remedy without using their java api. That would explain a lot actually. I'll have do some testing with our other apps to see if I can replicate in patch 004. I know BMC isn't very forthcoming with their patch schedule, but did they say when patch 005 was going to go live? Thanks for the heads up though :) On Wed, Apr 7, 2010 at 2:47 PM, patchsk vamsi...@gmail.com wrote: We have seen memory issues too on Solaris5.10, ARSystem 7.5patch3. BMC has found three memory leaks related to be caused by webservice calls i.e., ARXMLGE api calls after a long investigation. Regular user tool we did not see much issues. They supposed to fix it with patch005. Also libumem worked better for us compared to default memory handler, even though they say libumem is only effective for older solaris os. After creating a ticket basically BMC has given us a script to log server env (prstat,vmstat etc..) at regualar intervals. We provided them those logs with remedy api logs and they did the investigation based on that. We had to fight a lot though to escalate it to server team. On Apr 7, 10:27 am, Robert Halstead badbee...@gmail.com wrote: Axton, Once I have this all setup with libumem and enable the UMEM_DEBUG and UMEM_LOGGING environment variables, do I just wait for the leak to occur to the point where the app crashes? Does the system produce a core file at that point? Do I then perform the MDB commands on that core file? Reading the article on dbx (RTC), it looks like I can connect to the running program without stopping it but I need to have the Sun Studio installed to get the dbx program correct? On Tue, Apr 6, 2010 at 8:14 PM, Axton axton.gr...@gmail.com wrote: ** Since you are on Solaris/sparc you have some really good options for seeing if there are memory leaks. Look into the slab memory allocator (libumem). http://blogs.sun.com/ahl/entry/solaris_10_top_11_20 http://blogs.sun.com/ahl/entry/solaris_10_top_11_20There are actually performance benefits to using this memory allocator to the standard libc (though it does make the memory footprint slightly larger), but it's going to hard stop your software (sigsegv, sigbus, etc.) in the event nasty things are going on that shouldn't be going on. Good news is that it tells you what/where if you tell your system to generate a core in the event. Once you have things running under libumem, you can put some checks on memory usage (see the following): UMEM_DEBUG UMEM_LOGGING Once you do that, it makes some handy options available in dbx (RTC): http://www.mail-archive.com/arsl...@arslist.org/msg33614.html http://www.fortran-2000.com/ArnaudRecipes/SunMemoryDB.html#SunDbx You can attach the debugger to the process in flight to check for memory leaks: http://technopark02.blogspot.com/2005/10/sun-studio-investigating-mem. .. See here for the high-level gory details. Pretty cool stuff: http://developers.sun.com/solaris/articles/libumem_library.html Axton Grams On Tue, Apr 6, 2010 at 4:38 PM, Robert Halstead badbee...@gmail.com wrote: ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory
Re: Finding memory leaks in the AR System
** Robert:I had an issue with the standard memory manager on Solaris for 7.1 a while ago... I am re-posting for your info. Like Axton said, it may be worthwhile changing your memory manager to "libumem" and re-trying...TerryREPOST: libumem, ITSM 7.0.2 and Solaris Load Testing Hello everyone: We are currently running ARS 7.1, ITSM 7.0.2 (latest patch) Incident/Change/CMDB/Problem on a high end Solaris box running Solaris 10 with a DB2 database. We had been conducting stress testing of the application utilizing Loadrunner (yes, we are aware of the limitations of this tool, but were able to get around them) and were previously experiencing exponential degradation of response times once a fixed number of users were logged into the system and performing certain functions (Note: These tests were with the WUT, not MidTier). These tests and results were both repeatable and consistent. After working with Remedy engineers on this problem and analyzing various log files and pstack output on the server, they suggested that we replace the default Solaris memory manager with "libumem" as there was extreme memory heap contention by the arsystem process. Believe it or not, this has fixed our performance issue and the application now scales to the desired number of users without any known problems to date. With the previous memory manager, we could not get over 130 users logging in over an hour. With the new memory manager, we were able to get 300+ logged in without the previously experienced exponential degradation. We will continue to do further load testing, however, I would be interested in hearing from anyone out there who has experience using "libumem" on your server to any capacity (development, testing, or production) Have you come across any issues or hints regarding it's use? Thank you... Terry On Apr 6, 2010, Robert Halstead badbee...@gmail.com wrote: ** Hey all,We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted.I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing?I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System.Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists.Our server specs are the following:System Configuration: Sun Microsystems sun4u Sun Fire V210System clock frequency: 167 MHZMemory size: 4GB CPUs E$ CPU CPUCPU Freq Size Implementation Mask Status Location--- -- - - -- 0 1002 MHz 1MB SUNW,UltraSPARC-IIIi 2.4 on-line MB/P01 1002 MHz 1MB SUNW,UltraSPARC-IIIi 2.4 on-line MB/P1AR System 7.5 patch 004Apache Tomcat 5.5.28 / Midtier 7.5 patch 004If you guys need more server specs let me know. We are trying to replicate the issue but we are unsure how it happens and don't really know where to start.Thanks for the help.-- "A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows.The ignoramus may be saved, but the fool knows that he is doomed."Bob Halstead_attend WWRUG10 www.wwrug.com ARSlist: "Where the Answers Are"_ _attend WWRUG10 www.wwrug.com ARSlist: "Where the Answers Are"_
Re: Finding memory leaks in the AR System
Thanks so much guys!! You have most likely put my project back on track here. Axton, I will read those articles right now and talk to our system engineers on getting the memory manager switched. Thanks for the tips NN, hopefully I will have everything I need when I open the ticket. On Wed, Apr 7, 2010 at 6:53 AM, Terry Bootsma tboot...@objectpath.comwrote: ** Robert: I had an issue with the standard memory manager on Solaris for 7.1 a while ago... I am re-posting for your info. Like Axton said, it may be worthwhile changing your memory manager to libumem and re-trying... Terry REPOST: libumem, ITSM 7.0.2 and Solaris Load Testinghttp://mail.objectpath.com/communities/message/84134#84134 Hello everyone: We are currently running ARS 7.1, ITSM 7.0.2 (latest patch) Incident/Change/CMDB/Problem on a high end Solaris box running Solaris 10 with a DB2 database. We had been conducting stress testing of the application utilizing Loadrunner (yes, we are aware of the limitations of this tool, but were able to get around them) and were previously experiencing exponential degradation of response times once a fixed number of users were logged into the system and performing certain functions (Note: These tests were with the WUT, not MidTier). These tests and results were both repeatable and consistent. After working with Remedy engineers on this problem and analyzing various log files and pstack output on the server, they suggested that we replace the default Solaris memory manager with libumem as there was extreme memory heap contention by the arsystem process. Believe it or not, this has fixed our performance issue and the application now scales to the desired number of users without any known problems to date. With the previous memory manager, we could not get over 130 users logging in over an hour. With the new memory manager, we were able to get 300+ logged in without the previously experienced exponential degradation. We will continue to do further load testing, however, I would be interested in hearing from anyone out there who has experience using libumem on your server to any capacity (development, testing, or production) Have you come across any issues or hints regarding it's use? Thank you... Terry On Apr 6, 2010, *Robert Halstead* badbee...@gmail.com wrote: ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System. Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists. Our server specs are the following: System Configuration: Sun Microsystems sun4u Sun Fire V210 System clock frequency: 167 MHZ Memory size: 4GB CPUs E$ CPUCPU CPU Freq SizeImplementation MaskStatus Location --- -- - - -- 01002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P0 11002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P1 AR System 7.5 patch 004 Apache Tomcat 5.5.28 / Midtier 7.5 patch 004 If you guys need more server specs let me know. We are trying to replicate the issue but we are unsure how it happens and don't really know where to start. Thanks for the help. -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus may be saved, but the fool knows that he is doomed. Bob Halstead _attend WWRUG10 www.wwrug.com ARSlist: Where the Answers Are_ _attend
Re: Finding memory leaks in the AR System
Axton, Once I have this all setup with libumem and enable the UMEM_DEBUG and UMEM_LOGGING environment variables, do I just wait for the leak to occur to the point where the app crashes? Does the system produce a core file at that point? Do I then perform the MDB commands on that core file? Reading the article on dbx (RTC), it looks like I can connect to the running program without stopping it but I need to have the Sun Studio installed to get the dbx program correct? On Tue, Apr 6, 2010 at 8:14 PM, Axton axton.gr...@gmail.com wrote: ** Since you are on Solaris/sparc you have some really good options for seeing if there are memory leaks. Look into the slab memory allocator (libumem). http://blogs.sun.com/ahl/entry/solaris_10_top_11_20 http://blogs.sun.com/ahl/entry/solaris_10_top_11_20There are actually performance benefits to using this memory allocator to the standard libc (though it does make the memory footprint slightly larger), but it's going to hard stop your software (sigsegv, sigbus, etc.) in the event nasty things are going on that shouldn't be going on. Good news is that it tells you what/where if you tell your system to generate a core in the event. Once you have things running under libumem, you can put some checks on memory usage (see the following): UMEM_DEBUG UMEM_LOGGING Once you do that, it makes some handy options available in dbx (RTC): http://www.mail-archive.com/arslist@arslist.org/msg33614.html http://www.fortran-2000.com/ArnaudRecipes/SunMemoryDB.html#SunDbx You can attach the debugger to the process in flight to check for memory leaks: http://technopark02.blogspot.com/2005/10/sun-studio-investigating-memory-leaks.html See here for the high-level gory details. Pretty cool stuff: http://developers.sun.com/solaris/articles/libumem_library.html Axton Grams On Tue, Apr 6, 2010 at 4:38 PM, Robert Halstead badbee...@gmail.comwrote: ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System. Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists. Our server specs are the following: System Configuration: Sun Microsystems sun4u Sun Fire V210 System clock frequency: 167 MHZ Memory size: 4GB CPUs E$ CPUCPU CPU Freq SizeImplementation MaskStatus Location --- -- - - -- 01002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P0 11002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P1 AR System 7.5 patch 004 Apache Tomcat 5.5.28 / Midtier 7.5 patch 004 If you guys need more server specs let me know. We are trying to replicate the issue but we are unsure how it happens and don't really know where to start. Thanks for the help. -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus may be saved, but the fool knows that he is doomed. Bob Halstead _attend WWRUG10 www.wwrug.com ARSlist: Where the Answers Are_ _attend WWRUG10 www.wwrug.com ARSlist: Where the Answers Are_ -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus may be saved, but the fool knows that he is doomed. Bob Halstead ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug10 www.wwrug.com ARSlist: Where
Re: Finding memory leaks in the AR System
We have seen memory issues too on Solaris5.10, ARSystem 7.5patch3. BMC has found three memory leaks related to be caused by webservice calls i.e., ARXMLGE api calls after a long investigation. Regular user tool we did not see much issues. They supposed to fix it with patch005. Also libumem worked better for us compared to default memory handler, even though they say libumem is only effective for older solaris os. After creating a ticket basically BMC has given us a script to log server env (prstat,vmstat etc..) at regualar intervals. We provided them those logs with remedy api logs and they did the investigation based on that. We had to fight a lot though to escalate it to server team. On Apr 7, 10:27 am, Robert Halstead badbee...@gmail.com wrote: Axton, Once I have this all setup with libumem and enable the UMEM_DEBUG and UMEM_LOGGING environment variables, do I just wait for the leak to occur to the point where the app crashes? Does the system produce a core file at that point? Do I then perform the MDB commands on that core file? Reading the article on dbx (RTC), it looks like I can connect to the running program without stopping it but I need to have the Sun Studio installed to get the dbx program correct? On Tue, Apr 6, 2010 at 8:14 PM, Axton axton.gr...@gmail.com wrote: ** Since you are on Solaris/sparc you have some really good options for seeing if there are memory leaks. Look into the slab memory allocator (libumem). http://blogs.sun.com/ahl/entry/solaris_10_top_11_20 http://blogs.sun.com/ahl/entry/solaris_10_top_11_20There are actually performance benefits to using this memory allocator to the standard libc (though it does make the memory footprint slightly larger), but it's going to hard stop your software (sigsegv, sigbus, etc.) in the event nasty things are going on that shouldn't be going on. Good news is that it tells you what/where if you tell your system to generate a core in the event. Once you have things running under libumem, you can put some checks on memory usage (see the following): UMEM_DEBUG UMEM_LOGGING Once you do that, it makes some handy options available in dbx (RTC): http://www.mail-archive.com/arsl...@arslist.org/msg33614.html http://www.fortran-2000.com/ArnaudRecipes/SunMemoryDB.html#SunDbx You can attach the debugger to the process in flight to check for memory leaks: http://technopark02.blogspot.com/2005/10/sun-studio-investigating-mem... See here for the high-level gory details. Pretty cool stuff: http://developers.sun.com/solaris/articles/libumem_library.html Axton Grams On Tue, Apr 6, 2010 at 4:38 PM, Robert Halstead badbee...@gmail.comwrote: ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System. Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists. Our server specs are the following: System Configuration: Sun Microsystems sun4u Sun Fire V210 System clock frequency: 167 MHZ Memory size: 4GB CPUs E$ CPU CPU CPU Freq Size Implementation Mask Status Location --- -- - - -- 0 1002 MHz 1MB SUNW,UltraSPARC-IIIi 2.4 on-line MB/P0 1 1002 MHz 1MB SUNW,UltraSPARC-IIIi 2.4 on-line MB/P1 AR System 7.5 patch 004 Apache Tomcat 5.5.28 / Midtier 7.5 patch 004 If you guys need more server specs let me know. We are trying to replicate the issue but
Re: Finding memory leaks in the AR System
patchsk, We do utiize the midtier a lot with our custom applications and tools as that is the only cross-platform way to communicate with remedy without using their java api. That would explain a lot actually. I'll have do some testing with our other apps to see if I can replicate in patch 004. I know BMC isn't very forthcoming with their patch schedule, but did they say when patch 005 was going to go live? Thanks for the heads up though :) On Wed, Apr 7, 2010 at 2:47 PM, patchsk vamsi...@gmail.com wrote: We have seen memory issues too on Solaris5.10, ARSystem 7.5patch3. BMC has found three memory leaks related to be caused by webservice calls i.e., ARXMLGE api calls after a long investigation. Regular user tool we did not see much issues. They supposed to fix it with patch005. Also libumem worked better for us compared to default memory handler, even though they say libumem is only effective for older solaris os. After creating a ticket basically BMC has given us a script to log server env (prstat,vmstat etc..) at regualar intervals. We provided them those logs with remedy api logs and they did the investigation based on that. We had to fight a lot though to escalate it to server team. On Apr 7, 10:27 am, Robert Halstead badbee...@gmail.com wrote: Axton, Once I have this all setup with libumem and enable the UMEM_DEBUG and UMEM_LOGGING environment variables, do I just wait for the leak to occur to the point where the app crashes? Does the system produce a core file at that point? Do I then perform the MDB commands on that core file? Reading the article on dbx (RTC), it looks like I can connect to the running program without stopping it but I need to have the Sun Studio installed to get the dbx program correct? On Tue, Apr 6, 2010 at 8:14 PM, Axton axton.gr...@gmail.com wrote: ** Since you are on Solaris/sparc you have some really good options for seeing if there are memory leaks. Look into the slab memory allocator (libumem). http://blogs.sun.com/ahl/entry/solaris_10_top_11_20 http://blogs.sun.com/ahl/entry/solaris_10_top_11_20There are actually performance benefits to using this memory allocator to the standard libc (though it does make the memory footprint slightly larger), but it's going to hard stop your software (sigsegv, sigbus, etc.) in the event nasty things are going on that shouldn't be going on. Good news is that it tells you what/where if you tell your system to generate a core in the event. Once you have things running under libumem, you can put some checks on memory usage (see the following): UMEM_DEBUG UMEM_LOGGING Once you do that, it makes some handy options available in dbx (RTC): http://www.mail-archive.com/arsl...@arslist.org/msg33614.html http://www.fortran-2000.com/ArnaudRecipes/SunMemoryDB.html#SunDbx You can attach the debugger to the process in flight to check for memory leaks: http://technopark02.blogspot.com/2005/10/sun-studio-investigating-mem. .. See here for the high-level gory details. Pretty cool stuff: http://developers.sun.com/solaris/articles/libumem_library.html Axton Grams On Tue, Apr 6, 2010 at 4:38 PM, Robert Halstead badbee...@gmail.com wrote: ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System. Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists. Our server specs are the following: System Configuration: Sun Microsystems sun4u Sun Fire V210 System clock frequency:
Re: Finding memory leaks in the AR System
It is under limited availability and is going through UAT as per our support rep. We are expecting around end of this month, but she can not speak for sure as it depends on their test results. On Apr 7, 1:52 pm, Robert Halstead badbee...@gmail.com wrote: patchsk, We do utiize the midtier a lot with our custom applications and tools as that is the only cross-platform way to communicate with remedy without using their java api. That would explain a lot actually. I'll have do some testing with our other apps to see if I can replicate in patch 004. I know BMC isn't very forthcoming with their patch schedule, but did they say when patch 005 was going to go live? Thanks for the heads up though :) On Wed, Apr 7, 2010 at 2:47 PM, patchsk vamsi...@gmail.com wrote: We have seen memory issues too on Solaris5.10, ARSystem 7.5patch3. BMC has found three memory leaks related to be caused by webservice calls i.e., ARXMLGE api calls after a long investigation. Regular user tool we did not see much issues. They supposed to fix it with patch005. Also libumem worked better for us compared to default memory handler, even though they say libumem is only effective for older solaris os. After creating a ticket basically BMC has given us a script to log server env (prstat,vmstat etc..) at regualar intervals. We provided them those logs with remedy api logs and they did the investigation based on that. We had to fight a lot though to escalate it to server team. On Apr 7, 10:27 am, Robert Halstead badbee...@gmail.com wrote: Axton, Once I have this all setup with libumem and enable the UMEM_DEBUG and UMEM_LOGGING environment variables, do I just wait for the leak to occur to the point where the app crashes? Does the system produce a core file at that point? Do I then perform the MDB commands on that core file? Reading the article on dbx (RTC), it looks like I can connect to the running program without stopping it but I need to have the Sun Studio installed to get the dbx program correct? On Tue, Apr 6, 2010 at 8:14 PM, Axton axton.gr...@gmail.com wrote: ** Since you are on Solaris/sparc you have some really good options for seeing if there are memory leaks. Look into the slab memory allocator (libumem). http://blogs.sun.com/ahl/entry/solaris_10_top_11_20 http://blogs.sun.com/ahl/entry/solaris_10_top_11_20There are actually performance benefits to using this memory allocator to the standard libc (though it does make the memory footprint slightly larger), but it's going to hard stop your software (sigsegv, sigbus, etc.) in the event nasty things are going on that shouldn't be going on. Good news is that it tells you what/where if you tell your system to generate a core in the event. Once you have things running under libumem, you can put some checks on memory usage (see the following): UMEM_DEBUG UMEM_LOGGING Once you do that, it makes some handy options available in dbx (RTC): http://www.mail-archive.com/arsl...@arslist.org/msg33614.html http://www.fortran-2000.com/ArnaudRecipes/SunMemoryDB.html#SunDbx You can attach the debugger to the process in flight to check for memory leaks: http://technopark02.blogspot.com/2005/10/sun-studio-investigating-mem. .. See here for the high-level gory details. Pretty cool stuff: http://developers.sun.com/solaris/articles/libumem_library.html Axton Grams On Tue, Apr 6, 2010 at 4:38 PM, Robert Halstead badbee...@gmail.com wrote: ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR
Re: Finding memory leaks in the AR System
Do you see the memory leak on just deneral usage of the midtier? Or is there a specific action to replicate? Is this through the mid-tier interface, or webservice call? On Wed, Apr 7, 2010 at 3:22 PM, patchsk vamsi...@gmail.com wrote: It is under limited availability and is going through UAT as per our support rep. We are expecting around end of this month, but she can not speak for sure as it depends on their test results. On Apr 7, 1:52 pm, Robert Halstead badbee...@gmail.com wrote: patchsk, We do utiize the midtier a lot with our custom applications and tools as that is the only cross-platform way to communicate with remedy without using their java api. That would explain a lot actually. I'll have do some testing with our other apps to see if I can replicate in patch 004. I know BMC isn't very forthcoming with their patch schedule, but did they say when patch 005 was going to go live? Thanks for the heads up though :) On Wed, Apr 7, 2010 at 2:47 PM, patchsk vamsi...@gmail.com wrote: We have seen memory issues too on Solaris5.10, ARSystem 7.5patch3. BMC has found three memory leaks related to be caused by webservice calls i.e., ARXMLGE api calls after a long investigation. Regular user tool we did not see much issues. They supposed to fix it with patch005. Also libumem worked better for us compared to default memory handler, even though they say libumem is only effective for older solaris os. After creating a ticket basically BMC has given us a script to log server env (prstat,vmstat etc..) at regualar intervals. We provided them those logs with remedy api logs and they did the investigation based on that. We had to fight a lot though to escalate it to server team. On Apr 7, 10:27 am, Robert Halstead badbee...@gmail.com wrote: Axton, Once I have this all setup with libumem and enable the UMEM_DEBUG and UMEM_LOGGING environment variables, do I just wait for the leak to occur to the point where the app crashes? Does the system produce a core file at that point? Do I then perform the MDB commands on that core file? Reading the article on dbx (RTC), it looks like I can connect to the running program without stopping it but I need to have the Sun Studio installed to get the dbx program correct? On Tue, Apr 6, 2010 at 8:14 PM, Axton axton.gr...@gmail.com wrote: ** Since you are on Solaris/sparc you have some really good options for seeing if there are memory leaks. Look into the slab memory allocator (libumem). http://blogs.sun.com/ahl/entry/solaris_10_top_11_20 http://blogs.sun.com/ahl/entry/solaris_10_top_11_20There are actually performance benefits to using this memory allocator to the standard libc (though it does make the memory footprint slightly larger), but it's going to hard stop your software (sigsegv, sigbus, etc.) in the event nasty things are going on that shouldn't be going on. Good news is that it tells you what/where if you tell your system to generate a core in the event. Once you have things running under libumem, you can put some checks on memory usage (see the following): UMEM_DEBUG UMEM_LOGGING Once you do that, it makes some handy options available in dbx (RTC): http://www.mail-archive.com/arsl...@arslist.org/msg33614.html http://www.fortran-2000.com/ArnaudRecipes/SunMemoryDB.html#SunDbx You can attach the debugger to the process in flight to check for memory leaks: http://technopark02.blogspot.com/2005/10/sun-studio-investigating-mem. .. See here for the high-level gory details. Pretty cool stuff: http://developers.sun.com/solaris/articles/libumem_library.html Axton Grams On Tue, Apr 6, 2010 at 4:38 PM, Robert Halstead badbee...@gmail.com wrote: ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a
Finding memory leaks in the AR System
Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System. Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists. Our server specs are the following: System Configuration: Sun Microsystems sun4u Sun Fire V210 System clock frequency: 167 MHZ Memory size: 4GB CPUs E$ CPUCPU CPU Freq SizeImplementation MaskStatus Location --- -- - - -- 01002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P0 11002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P1 AR System 7.5 patch 004 Apache Tomcat 5.5.28 / Midtier 7.5 patch 004 If you guys need more server specs let me know. We are trying to replicate the issue but we are unsure how it happens and don't really know where to start. Thanks for the help. -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus may be saved, but the fool knows that he is doomed. Bob Halstead ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug10 www.wwrug.com ARSlist: Where the Answers Are
Re: Finding memory leaks in the AR System
Two things: 1) The memory usage issue is definitely true at the DB level, especially if you're using SQL. 2) If this is an AR Server issue, it is one more issue with patch 4 that has and will keep me from installing it. Better the devil I know in patch 3... That being said, pre-allocating idle memory isn't an issue as long as the application is willing and able to give it back if something else needs it. SQL allegedly will. I haven't seen that behavior in AR System. Rick On Tue, Apr 6, 2010 at 2:38 PM, Robert Halstead badbee...@gmail.com wrote: ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System. Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists. Our server specs are the following: System Configuration: Sun Microsystems sun4u Sun Fire V210 System clock frequency: 167 MHZ Memory size: 4GB CPUs E$ CPUCPU CPU Freq SizeImplementation MaskStatus Location --- -- - - -- 01002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P0 11002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P1 AR System 7.5 patch 004 Apache Tomcat 5.5.28 / Midtier 7.5 patch 004 If you guys need more server specs let me know. We are trying to replicate the issue but we are unsure how it happens and don't really know where to start. Thanks for the help. -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus may be saved, but the fool knows that he is doomed. Bob Halstead _attend WWRUG10 www.wwrug.com ARSlist: Where the Answers Are_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug10 www.wwrug.com ARSlist: Where the Answers Are
Re: Finding memory leaks in the AR System
Are you making form or workflow changes on that server while users are using it? Are you creating new ITSM Support Groups while other are logged on to the system? Is Development Cache Mode turned off? If the answer to the last question is “yes”, then either of the first two actions will cause the server to create a new copy of the workflow/forms cache in memory until the users that were logged in during the change log out for a short period. For example, if you create a support group, the server thinks it needs to recache everything, so it creates a copy of the current cache for those that are currently logged in and then pulls the new one from the server to reflect the changes (or something along those lines). In any case, the result is that you then have multiple copies of the cache in memory, which can easily consume all of your memory depending on how much workflow you have loaded and how many times it pulls a new copy of the cache into memory. That said, my experience is based on ARS 7.1, but I wouldn’t be surprised if it’s essentially the same in 7.5… Lyle From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Robert Halstead Sent: Tuesday, April 06, 2010 3:39 PM To: arslist@ARSLIST.ORG Subject: Finding memory leaks in the AR System ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System. Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists. Our server specs are the following: System Configuration: Sun Microsystems sun4u Sun Fire V210 System clock frequency: 167 MHZ Memory size: 4GB CPUs E$ CPUCPU CPU Freq SizeImplementation MaskStatus Location --- -- - - -- 01002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P0 11002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P1 AR System 7.5 patch 004 Apache Tomcat 5.5.28 / Midtier 7.5 patch 004 If you guys need more server specs let me know. We are trying to replicate the issue but we are unsure how it happens and don't really know where to start. Thanks for the help. -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus may be saved, but the fool knows that he is doomed. Bob Halstead _attend WWRUG10 www.wwrug.com ARSlist: Where the Answers Are_ NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: Finding memory leaks in the AR System
Hi guys, during the time we had to restart it, I was off on vacation and I'm the only developer, so I don't believe we were making workflow modifications to the system, though i'm not too sure.. Development Cache mode is checked on the Configuration tab so it should not be making a copy of the cache. On Tue, Apr 6, 2010 at 3:55 PM, Lyle Taylor tayl...@ldschurch.org wrote: Are you making form or workflow changes on that server while users are using it? Are you creating new ITSM Support Groups while other are logged on to the system? Is Development Cache Mode turned off? If the answer to the last question is “yes”, then either of the first two actions will cause the server to create a new copy of the workflow/forms cache in memory until the users that were logged in during the change log out for a short period. For example, if you create a support group, the server thinks it needs to recache everything, so it creates a copy of the current cache for those that are currently logged in and then pulls the new one from the server to reflect the changes (or something along those lines). In any case, the result is that you then have multiple copies of the cache in memory, which can easily consume all of your memory depending on how much workflow you have loaded and how many times it pulls a new copy of the cache into memory. That said, my experience is based on ARS 7.1, but I wouldn’t be surprised if it’s essentially the same in 7.5… Lyle *From:* Action Request System discussion list(ARSList) [mailto: arsl...@arslist.org] *On Behalf Of *Robert Halstead *Sent:* Tuesday, April 06, 2010 3:39 PM *To:* arslist@ARSLIST.ORG *Subject:* Finding memory leaks in the AR System ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System. Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists. Our server specs are the following: System Configuration: Sun Microsystems sun4u Sun Fire V210 System clock frequency: 167 MHZ Memory size: 4GB CPUs E$ CPUCPU CPU Freq SizeImplementation MaskStatus Location --- -- - - -- 01002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P0 11002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P1 AR System 7.5 patch 004 Apache Tomcat 5.5.28 / Midtier 7.5 patch 004 If you guys need more server specs let me know. We are trying to replicate the issue but we are unsure how it happens and don't really know where to start. Thanks for the help. -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus may be saved, but the fool knows that he is doomed. Bob Halstead _attend WWRUG10 www.wwrug.com ARSlist: Where the Answers Are_ NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus may be saved, but the fool knows that he is doomed. Bob Halstead ___ UNSUBSCRIBE or access
Re: Finding memory leaks in the AR System
What about queries that have a large result set? How does the AR System handle those? On Tue, Apr 6, 2010 at 4:24 PM, Robert Halstead badbee...@gmail.com wrote: Hi guys, during the time we had to restart it, I was off on vacation and I'm the only developer, so I don't believe we were making workflow modifications to the system, though i'm not too sure.. Development Cache mode is checked on the Configuration tab so it should not be making a copy of the cache. On Tue, Apr 6, 2010 at 3:55 PM, Lyle Taylor tayl...@ldschurch.org wrote: Are you making form or workflow changes on that server while users are using it? Are you creating new ITSM Support Groups while other are logged on to the system? Is Development Cache Mode turned off? If the answer to the last question is “yes”, then either of the first two actions will cause the server to create a new copy of the workflow/forms cache in memory until the users that were logged in during the change log out for a short period. For example, if you create a support group, the server thinks it needs to recache everything, so it creates a copy of the current cache for those that are currently logged in and then pulls the new one from the server to reflect the changes (or something along those lines). In any case, the result is that you then have multiple copies of the cache in memory, which can easily consume all of your memory depending on how much workflow you have loaded and how many times it pulls a new copy of the cache into memory. That said, my experience is based on ARS 7.1, but I wouldn’t be surprised if it’s essentially the same in 7.5… Lyle *From:* Action Request System discussion list(ARSList) [mailto: arsl...@arslist.org] *On Behalf Of *Robert Halstead *Sent:* Tuesday, April 06, 2010 3:39 PM *To:* arslist@ARSLIST.ORG *Subject:* Finding memory leaks in the AR System ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System. Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists. Our server specs are the following: System Configuration: Sun Microsystems sun4u Sun Fire V210 System clock frequency: 167 MHZ Memory size: 4GB CPUs E$ CPUCPU CPU Freq SizeImplementation MaskStatus Location --- -- - - -- 01002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P0 11002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P1 AR System 7.5 patch 004 Apache Tomcat 5.5.28 / Midtier 7.5 patch 004 If you guys need more server specs let me know. We are trying to replicate the issue but we are unsure how it happens and don't really know where to start. Thanks for the help. -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus may be saved, but the fool knows that he is doomed. Bob Halstead _attend WWRUG10 www.wwrug.com ARSlist: Where the Answers Are_ NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus
Re: Finding memory leaks in the AR System
Open a ticket with BMC. Just make sure you have ALL the files they will want or you will wind up playing volleyball email. At a minimum: Do a trace on memory usage; You better make the case here. How long, how fast, etc. Monitor threads. Use prstat, truss, whatever debug tools at your disposal. Check the pfiles, give them filehandle usage. Capture the memory footprint right before it dies. Have your ar.conf file ready.. On Tue, Apr 6, 2010 at 4:38 PM, Robert Halstead badbee...@gmail.com wrote: ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System. Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists. Our server specs are the following: System Configuration: Sun Microsystems sun4u Sun Fire V210 System clock frequency: 167 MHZ Memory size: 4GB CPUs E$ CPUCPU CPU Freq SizeImplementation MaskStatus Location --- -- - - -- 01002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P0 11002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P1 AR System 7.5 patch 004 Apache Tomcat 5.5.28 / Midtier 7.5 patch 004 If you guys need more server specs let me know. We are trying to replicate the issue but we are unsure how it happens and don't really know where to start. Thanks for the help. -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus may be saved, but the fool knows that he is doomed. Bob Halstead _attend WWRUG10 www.wwrug.com ARSlist: Where the Answers Are_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug10 www.wwrug.com ARSlist: Where the Answers Are
Re: Finding memory leaks in the AR System
Since you are on Solaris/sparc you have some really good options for seeing if there are memory leaks. Look into the slab memory allocator (libumem). http://blogs.sun.com/ahl/entry/solaris_10_top_11_20 http://blogs.sun.com/ahl/entry/solaris_10_top_11_20There are actually performance benefits to using this memory allocator to the standard libc (though it does make the memory footprint slightly larger), but it's going to hard stop your software (sigsegv, sigbus, etc.) in the event nasty things are going on that shouldn't be going on. Good news is that it tells you what/where if you tell your system to generate a core in the event. Once you have things running under libumem, you can put some checks on memory usage (see the following): UMEM_DEBUG UMEM_LOGGING Once you do that, it makes some handy options available in dbx (RTC): http://www.mail-archive.com/arslist@arslist.org/msg33614.html http://www.fortran-2000.com/ArnaudRecipes/SunMemoryDB.html#SunDbx You can attach the debugger to the process in flight to check for memory leaks: http://technopark02.blogspot.com/2005/10/sun-studio-investigating-memory-leaks.html See here for the high-level gory details. Pretty cool stuff: http://developers.sun.com/solaris/articles/libumem_library.html Axton Grams On Tue, Apr 6, 2010 at 4:38 PM, Robert Halstead badbee...@gmail.com wrote: ** Hey all, We're running AR System 7.5 patch 004 and we are finding that our server is eating up memory and not releasing it. We are in the UAT process and have roughly 10 testers testing the system. During this time we've noticed a huge memory allocation and eventually the arserverd process would consume 2-3 gigs of memory and all the swap space, at which point the machine comes to it's knees and the process needs to be forcibly killed or the box hard restarted. I remember reading somewhere that the AR System doesn't release memory for large queries, but instead just reuses the memory address space. Is this still true for 7.5? Are there any type of performance configurations I can add to the ar.conf file to allow the AR System to release the memory it allocates? Or to prevent a query from taking all the available memory on the box? I thought the AR System used temporary file storage for storaging a large SQL result? Our 6.3 AR System stores temporary query result files in /var/tmp/ARpen* files, does 7.5 not do the same thing? I just thought I would ping the list before I open a ticket with BMC and see if anyone else is seeing a memory leak or has had this problem occur to them in the past. Though I'm not sure who all is running the latest 7.5 AR System. Any help would be appreciated as I'm not sure what BMC will want me to look for to determine a memory leak and I don't like to engage them without some sort of proof that one exists. Our server specs are the following: System Configuration: Sun Microsystems sun4u Sun Fire V210 System clock frequency: 167 MHZ Memory size: 4GB CPUs E$ CPUCPU CPU Freq SizeImplementation MaskStatus Location --- -- - - -- 01002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P0 11002 MHz 1MB SUNW,UltraSPARC-IIIi2.4on-line MB/P1 AR System 7.5 patch 004 Apache Tomcat 5.5.28 / Midtier 7.5 patch 004 If you guys need more server specs let me know. We are trying to replicate the issue but we are unsure how it happens and don't really know where to start. Thanks for the help. -- A fool acts, regardless; knowing well that he is wrong. The ignoramus acts on only what he knows, but all that he knows. The ignoramus may be saved, but the fool knows that he is doomed. Bob Halstead _attend WWRUG10 www.wwrug.com ARSlist: Where the Answers Are_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug10 www.wwrug.com ARSlist: Where the Answers Are
