Re: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
Hmmm. Are you saying 'Ogres' are like onions? On Fri, 2004-04-16 at 07:05, Scott Richardson wrote: > Performance of UV applications on various Operating Systems > is not rocket science. Perhaps better described as large, nasty > tight onions that need peeling, one layer at a time, and > understanding what each peeled layer is doing and why. > Once this knowledge is acquired and understood, a plan can > be built and executed to attack/resolve the problem. > > Are users logging out/off when they're done using the system, > or when they've completed some large tasks or operations? > How often is the system rebooted? > RAID 5 file systems can slow down IO. > We'll need specifics on file system setup and parameters. > How many users? What are these users doing? > Have you got everyone and their siblings all running SELECT and > SORT operations all the time? Data Entry out the wazoo? > > How big are the files, and how are they sized? How frequently > does data change in the files, (grow, shrink, etc...) > > How big is your /tmp file system, and what kind of file system, > and where is it physically located?? Provide it it's own file system, > on it's own disk or disk set, (i.e. not the same disks where other > activity is going on). > > 4GB of RAM, yet only 4 GB paging/swap space? > Where is this swap paging space, (i.e. what disks?) > > "topas" may be fine for quick and dirty analysis and understanding, > but using it extensively can help contribute to performance problems. > > You need to configure and tune the platform, the OS, the UV DB, > the IO sub-system, the applications, the users, and the > administration/operations, and thenensure they're all coordinated > with each other, to maximize platform performance. > > To find, (and therefore address & resolve), the root causes of what > is happening here, you need to profile the platform using something > such as the DPMonitor, (extremely low-overhead monitoring Agent) > and display/crunch the performance metrics on another platform, > (i..e. a Windows Performance Explorer Console). Using this method, > you'll be able completely profile the entire platform, (OS and > applications), > around the clock, and then easily dial into specific timeframes where > problems are occurring, and fully understand exactly what is happening > and learn why it is happening, so it can be addressed and resolved, > and measure the progress along the entire way. > > The DPMonitor is available with a free 10 day evaluation license where it > will track system-wide performance metrics. Fully licensed version will > track individual processes that you select, or all processes if you so > desire. When you monitor all of the processes, you can quickly and > easily identify processes deserving further analysis, and stop tracking > processes that are not casuing any problems. More information on the > DPMonitor can be found at http://www.deltek.us and the DPMonitor > can be downloaded right off the website. If you're short on memory, > DPMonitor will allow you to see how much memory you will need to > allow the system to run as fast as it can, given how you're running it. > If you need tuning of OS or UV parameters, or other things that ay be > playing contributing factor/roles, the DPMonitor will clearly point this > out, > grahically, so that anyone can plainly see what is happening. > > Once you make any changes, you'll be able to monitor, and measure, > any differences, consistently, and prove whether or not you have > improved, or detrimented, your cause. Best of all, you'll be able to > show, prove, and justify to management what you're doing, and > why, and show them what it will take to get the problems addressed > and resolved, positively, without question. > > Hope this helps. I know the DPMonitor can & will help. > I have used it personally, numerous times, to peel many a complex onion, > understand what is exactly going on, find out why, and then put together > and executed plans that have successfully addressed and resolved similar > problems and streamlined operations moving forward saving many a > business significant time, frustration, and money, and then ensured that > any & all operations moving forward were done from a pro-active, > knowing ahead of time manner, rather than fire-fighting problems on a > continual basis. If you want something done, why not do it right, once? > Stop beating your head against the onion wall! Work smarter! > Let the DPMonitor be your detailed, EKG-like instrument to cut to > the heart of your complex application server performance problems, > identify them, and help you to resolve them, quickly and easily. > > > Been there, done that. > Many times over. > > Sincere Regards, > Scott Richardson > Senior Systems Engineer / Consultant > Marlborough, MA 01752 > Email: [EMAIL PROTECTED] > Web: http://home.comcast.net/~CheetahFTL/CC/CheetahFTL_1.htm > eFax: 208-445-1259 > > - Original Message - > From: "Foo Chia Teck" <
RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
Why? He's got 4Gb of ram. If he overflows that by JUST ONE BYTE, if he was running linux his performance would *collapse*. If you do not have twice ram as swap, you only need to use ONE BYTE of swap space and the linux algorithm will shit itself trying to cope. The algorithm does not work very well if it doesn't have enough disk space, and that's where the "twice swap" rule came from. So it's a very old unix algorithm ... which is why I suspect AIX might suffer similarly. Cheers, Wol -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steve Ferries Sent: 16 April 2004 14:45 To: U2 Users Discussion List Subject: RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 HI All, Before doubling you swap space, check to see how much you are using at your busy times. We have an 8 Gig system, and a 6 Gig pool: Page Space Physical Volume Volume GroupSize %Used Active Auto Type paging00hdisk1rootvg6144MB 2 yes yes lv Everyone into the pool! Regards, Steve Ferries Vice President, Information Technologies Total Credit Recovery Limited -Original Message- From: Anthony Youngman [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] Sent: Friday, April 16, 2004 9:31 AM To: U2 Users Discussion List Subject: RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Okay, it's AIX not linux, but I've just noticed that RAM = swap. You are an ABSOLUTE FOOL if you do that on linux. Maybe (or maybe not) the same applies to AIX - quite likely since they are both nixen and probably manage memory similiarly. Double swap space to 8Gb and see if that improves matters. Oh - and if you don't believe me, a "swap = ram" configuration will CRASH the early vanilla 2.4 kernels and that's 2002 vintage. Cheers, Wol -Original Message- From: [EMAIL PROTECTED] [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] On Behalf Of Scott Richardson Sent: 16 April 2004 14:06 To: U2 Users Discussion List Subject: Re: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Performance of UV applications on various Operating Systems is not rocket science. Perhaps better described as large, nasty tight onions that need peeling, one layer at a time, and understanding what each peeled layer is doing and why. Once this knowledge is acquired and understood, a plan can be built and executed to attack/resolve the problem. Are users logging out/off when they're done using the system, or when they've completed some large tasks or operations? How often is the system rebooted? RAID 5 file systems can slow down IO. We'll need specifics on file system setup and parameters. How many users? What are these users doing? Have you got everyone and their siblings all running SELECT and SORT operations all the time? Data Entry out the wazoo? How big are the files, and how are they sized? How frequently does data change in the files, (grow, shrink, etc...) How big is your /tmp file system, and what kind of file system, and where is it physically located?? Provide it it's own file system, on it's own disk or disk set, (i.e. not the same disks where other activity is going on). 4GB of RAM, yet only 4 GB paging/swap space? Where is this swap paging space, (i.e. what disks?) "topas" may be fine for quick and dirty analysis and understanding, but using it extensively can help contribute to performance problems. You need to configure and tune the platform, the OS, the UV DB, the IO sub-system, the applications, the users, and the administration/operations, and thenensure they're all coordinated with each other, to maximize platform performance. To find, (and therefore address & resolve), the root causes of what is happening here, you need to profile the platform using something such as the DPMonitor, (extremely low-overhead monitoring Agent) and display/crunch the performance metrics on another platform, (i..e. a Windows Performance Explorer Console). Using this method, you'll be able completely profile the entire platform, (OS and applications), around the clock, and then easily dial into specific timeframes where problems are occurring, and fully understand exactly what is happening and learn why it is happening, so it can be addressed and resolved, and measure the progress along the entire way. The DPMonitor is available with a free 10 day evaluation license where it will track system-wide performance metrics. Fully licensed version will track individual processes that you select, or all processes if you so desire. When you monitor all of the processes, you can quickly and easily identify processes deserving further analysis, and stop tracking processes that are not casuing any problems. More information on the DPMonitor can be found at <http://www.deltek.us> http://www.deltek.us and the DPMonitor can be downloaded right o
RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
HI All, Before doubling you swap space, check to see how much you are using at your busy times. We have an 8 Gig system, and a 6 Gig pool: Page Space Physical Volume Volume GroupSize %Used Active Auto Type paging00hdisk1rootvg6144MB 2 yes yeslv Everyone into the pool! Regards, Steve Ferries Vice President, Information Technologies Total Credit Recovery Limited -Original Message- From: Anthony Youngman [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] Sent: Friday, April 16, 2004 9:31 AM To: U2 Users Discussion List Subject: RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Okay, it's AIX not linux, but I've just noticed that RAM = swap. You are an ABSOLUTE FOOL if you do that on linux. Maybe (or maybe not) the same applies to AIX - quite likely since they are both nixen and probably manage memory similiarly. Double swap space to 8Gb and see if that improves matters. Oh - and if you don't believe me, a "swap = ram" configuration will CRASH the early vanilla 2.4 kernels and that's 2002 vintage. Cheers, Wol -Original Message- From: [EMAIL PROTECTED] [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] On Behalf Of Scott Richardson Sent: 16 April 2004 14:06 To: U2 Users Discussion List Subject: Re: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Performance of UV applications on various Operating Systems is not rocket science. Perhaps better described as large, nasty tight onions that need peeling, one layer at a time, and understanding what each peeled layer is doing and why. Once this knowledge is acquired and understood, a plan can be built and executed to attack/resolve the problem. Are users logging out/off when they're done using the system, or when they've completed some large tasks or operations? How often is the system rebooted? RAID 5 file systems can slow down IO. We'll need specifics on file system setup and parameters. How many users? What are these users doing? Have you got everyone and their siblings all running SELECT and SORT operations all the time? Data Entry out the wazoo? How big are the files, and how are they sized? How frequently does data change in the files, (grow, shrink, etc...) How big is your /tmp file system, and what kind of file system, and where is it physically located?? Provide it it's own file system, on it's own disk or disk set, (i.e. not the same disks where other activity is going on). 4GB of RAM, yet only 4 GB paging/swap space? Where is this swap paging space, (i.e. what disks?) "topas" may be fine for quick and dirty analysis and understanding, but using it extensively can help contribute to performance problems. You need to configure and tune the platform, the OS, the UV DB, the IO sub-system, the applications, the users, and the administration/operations, and thenensure they're all coordinated with each other, to maximize platform performance. To find, (and therefore address & resolve), the root causes of what is happening here, you need to profile the platform using something such as the DPMonitor, (extremely low-overhead monitoring Agent) and display/crunch the performance metrics on another platform, (i..e. a Windows Performance Explorer Console). Using this method, you'll be able completely profile the entire platform, (OS and applications), around the clock, and then easily dial into specific timeframes where problems are occurring, and fully understand exactly what is happening and learn why it is happening, so it can be addressed and resolved, and measure the progress along the entire way. The DPMonitor is available with a free 10 day evaluation license where it will track system-wide performance metrics. Fully licensed version will track individual processes that you select, or all processes if you so desire. When you monitor all of the processes, you can quickly and easily identify processes deserving further analysis, and stop tracking processes that are not casuing any problems. More information on the DPMonitor can be found at <http://www.deltek.us> http://www.deltek.us and the DPMonitor can be downloaded right off the website. If you're short on memory, DPMonitor will allow you to see how much memory you will need to allow the system to run as fast as it can, given how you're running it. If you need tuning of OS or UV parameters, or other things that ay be playing contributing factor/roles, the DPMonitor will clearly point this out, grahically, so that anyone can plainly see what is happening. Once you make any changes, you'll be able to monitor, and measure, any differences, consistently, and prove whether or not you have improved, or detrimented, your cause. Best of all, you'll be able to show, prove, and justify to management what you're doing, and why, and show them what it will take to get the problems add
RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
Okay, it's AIX not linux, but I've just noticed that RAM = swap. You are an ABSOLUTE FOOL if you do that on linux. Maybe (or maybe not) the same applies to AIX - quite likely since they are both nixen and probably manage memory similiarly. Double swap space to 8Gb and see if that improves matters. Oh - and if you don't believe me, a "swap = ram" configuration will CRASH the early vanilla 2.4 kernels and that's 2002 vintage. Cheers, Wol -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Scott Richardson Sent: 16 April 2004 14:06 To: U2 Users Discussion List Subject: Re: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Performance of UV applications on various Operating Systems is not rocket science. Perhaps better described as large, nasty tight onions that need peeling, one layer at a time, and understanding what each peeled layer is doing and why. Once this knowledge is acquired and understood, a plan can be built and executed to attack/resolve the problem. Are users logging out/off when they're done using the system, or when they've completed some large tasks or operations? How often is the system rebooted? RAID 5 file systems can slow down IO. We'll need specifics on file system setup and parameters. How many users? What are these users doing? Have you got everyone and their siblings all running SELECT and SORT operations all the time? Data Entry out the wazoo? How big are the files, and how are they sized? How frequently does data change in the files, (grow, shrink, etc...) How big is your /tmp file system, and what kind of file system, and where is it physically located?? Provide it it's own file system, on it's own disk or disk set, (i.e. not the same disks where other activity is going on). 4GB of RAM, yet only 4 GB paging/swap space? Where is this swap paging space, (i.e. what disks?) "topas" may be fine for quick and dirty analysis and understanding, but using it extensively can help contribute to performance problems. You need to configure and tune the platform, the OS, the UV DB, the IO sub-system, the applications, the users, and the administration/operations, and thenensure they're all coordinated with each other, to maximize platform performance. To find, (and therefore address & resolve), the root causes of what is happening here, you need to profile the platform using something such as the DPMonitor, (extremely low-overhead monitoring Agent) and display/crunch the performance metrics on another platform, (i..e. a Windows Performance Explorer Console). Using this method, you'll be able completely profile the entire platform, (OS and applications), around the clock, and then easily dial into specific timeframes where problems are occurring, and fully understand exactly what is happening and learn why it is happening, so it can be addressed and resolved, and measure the progress along the entire way. The DPMonitor is available with a free 10 day evaluation license where it will track system-wide performance metrics. Fully licensed version will track individual processes that you select, or all processes if you so desire. When you monitor all of the processes, you can quickly and easily identify processes deserving further analysis, and stop tracking processes that are not casuing any problems. More information on the DPMonitor can be found at http://www.deltek.us and the DPMonitor can be downloaded right off the website. If you're short on memory, DPMonitor will allow you to see how much memory you will need to allow the system to run as fast as it can, given how you're running it. If you need tuning of OS or UV parameters, or other things that ay be playing contributing factor/roles, the DPMonitor will clearly point this out, grahically, so that anyone can plainly see what is happening. Once you make any changes, you'll be able to monitor, and measure, any differences, consistently, and prove whether or not you have improved, or detrimented, your cause. Best of all, you'll be able to show, prove, and justify to management what you're doing, and why, and show them what it will take to get the problems addressed and resolved, positively, without question. Hope this helps. I know the DPMonitor can & will help. I have used it personally, numerous times, to peel many a complex onion, understand what is exactly going on, find out why, and then put together and executed plans that have successfully addressed and resolved similar problems and streamlined operations moving forward saving many a business significant time, frustration, and money, and then ensured that any & all operations moving forward were done from a pro-active, knowing ahead of time manner, rather than fire-fighting problems on a continual basis. If you want something done, why not do it right, once? Stop beating your head against the onion wall! Work smarter
Re: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
Performance of UV applications on various Operating Systems is not rocket science. Perhaps better described as large, nasty tight onions that need peeling, one layer at a time, and understanding what each peeled layer is doing and why. Once this knowledge is acquired and understood, a plan can be built and executed to attack/resolve the problem. Are users logging out/off when they're done using the system, or when they've completed some large tasks or operations? How often is the system rebooted? RAID 5 file systems can slow down IO. We'll need specifics on file system setup and parameters. How many users? What are these users doing? Have you got everyone and their siblings all running SELECT and SORT operations all the time? Data Entry out the wazoo? How big are the files, and how are they sized? How frequently does data change in the files, (grow, shrink, etc...) How big is your /tmp file system, and what kind of file system, and where is it physically located?? Provide it it's own file system, on it's own disk or disk set, (i.e. not the same disks where other activity is going on). 4GB of RAM, yet only 4 GB paging/swap space? Where is this swap paging space, (i.e. what disks?) "topas" may be fine for quick and dirty analysis and understanding, but using it extensively can help contribute to performance problems. You need to configure and tune the platform, the OS, the UV DB, the IO sub-system, the applications, the users, and the administration/operations, and thenensure they're all coordinated with each other, to maximize platform performance. To find, (and therefore address & resolve), the root causes of what is happening here, you need to profile the platform using something such as the DPMonitor, (extremely low-overhead monitoring Agent) and display/crunch the performance metrics on another platform, (i..e. a Windows Performance Explorer Console). Using this method, you'll be able completely profile the entire platform, (OS and applications), around the clock, and then easily dial into specific timeframes where problems are occurring, and fully understand exactly what is happening and learn why it is happening, so it can be addressed and resolved, and measure the progress along the entire way. The DPMonitor is available with a free 10 day evaluation license where it will track system-wide performance metrics. Fully licensed version will track individual processes that you select, or all processes if you so desire. When you monitor all of the processes, you can quickly and easily identify processes deserving further analysis, and stop tracking processes that are not casuing any problems. More information on the DPMonitor can be found at http://www.deltek.us and the DPMonitor can be downloaded right off the website. If you're short on memory, DPMonitor will allow you to see how much memory you will need to allow the system to run as fast as it can, given how you're running it. If you need tuning of OS or UV parameters, or other things that ay be playing contributing factor/roles, the DPMonitor will clearly point this out, grahically, so that anyone can plainly see what is happening. Once you make any changes, you'll be able to monitor, and measure, any differences, consistently, and prove whether or not you have improved, or detrimented, your cause. Best of all, you'll be able to show, prove, and justify to management what you're doing, and why, and show them what it will take to get the problems addressed and resolved, positively, without question. Hope this helps. I know the DPMonitor can & will help. I have used it personally, numerous times, to peel many a complex onion, understand what is exactly going on, find out why, and then put together and executed plans that have successfully addressed and resolved similar problems and streamlined operations moving forward saving many a business significant time, frustration, and money, and then ensured that any & all operations moving forward were done from a pro-active, knowing ahead of time manner, rather than fire-fighting problems on a continual basis. If you want something done, why not do it right, once? Stop beating your head against the onion wall! Work smarter! Let the DPMonitor be your detailed, EKG-like instrument to cut to the heart of your complex application server performance problems, identify them, and help you to resolve them, quickly and easily. Been there, done that. Many times over. Sincere Regards, Scott Richardson Senior Systems Engineer / Consultant Marlborough, MA 01752 Email: [EMAIL PROTECTED] Web: http://home.comcast.net/~CheetahFTL/CC/CheetahFTL_1.htm eFax: 208-445-1259 - Original Message - From: "Foo Chia Teck" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, April 16, 2004 2:22 AM Subject: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Hi, We are facing performance degraded when running Universe 10.0.0 in AIX 5L 5.2. A bit intro on hardware specs. We are using pSeries 650 running o
RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
Hi Foo Could it be an application problem. It sounds like an application is adding data to a possible type 1 file as a log or a como file or even a print job and it is growing to an enormous size consuming resources. There might be some other application that has gone rogue. Can you identify the time this problem started and relate it to a program that was updated. Can you identify a user who is using a lot of the resources and find out what they are doing. Regards David Jordan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Foo Chia Teck Sent: Friday, 16 April 2004 4:22 PM To: [EMAIL PROTECTED] Subject: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Hi, We are facing performance degraded when running Universe 10.0.0 in AIX 5L 5.2. A bit intro on hardware specs. We are using pSeries 650 running on SMP 2 Power4 processor with 4GB of RAM, 4GB Paging size and RAID5 SSA Hard Disk. My Universe configuration as below: Current tunable parameter settings: MFILES = 300 T30FILE= 200 OPENCHK= 1 WIDE0 = 0x3dc0 UVSPOOL= /uvspool1 UVTEMP = /uvtmp1 SCRMIN = 3 SCRMAX = 5 SCRSIZE= 512 QDEPTH = 16 HISTSTK= 99 QSRUNSZ= 2000 QSBRNCH= 4 QSDEPTH= 8 QSMXKEY= 32 TXMODE = 0 LOGBLSZ= 512 LOGBLNUM = 8 LOGSYCNT = 0 LOGSYINT = 0 TXMEM = 32 OPTMEM = 64 SELBUF = 4 ULIMIT = 128000 FSEMNUM= 23 GSEMNUM= 97 PSEMNUM= 64 FLTABSZ= 11 GLTABSZ= 75 RLTABSZ= 75 RLOWNER= 300 PAKTIME= 5 NETTIME= 5 QBREAK = 1 VDIVDEF= 1 UVSYNC = 1 BLKMAX = 131072 PICKNULL = 0 SYNCALOC = 1 MAXRLOCK = 74 ISOMODE= 1 PKRJUST= 0 PROCACMD = 0 PROCRCMD = 0 PROCPRMT = 0 ALLOWNFS = 0 CSHDISPATCH= /bin/csh SHDISPATCH = /bin/sh DOSDISPATCH= NOT_SUPPORTED LAYERSEL = 0 OCVDATE= 0 MODFPTRS = 1 THDR512= 0 UDRMODE= 0 UDRBLKS= 0 MAXERRLOGENT = 100 JOINBUF= 4095 64BIT_FILES= 0 TSTIMEOUT = 60 PIOPENDEFAULT = 0 MAXKEYSIZE = 255 SMISDATA = 0 EXACTNUMERIC = 15 MALLOCTRACING = 0 CENTURYPIVOT = 1930 SPINTRIES = 0 SPINSLEEP = 1 CONVERT_EURO = 0 SYSTEM_EURO= 164 TERM_EURO = 164 SQLNULL= 128 When the uv restarted it run fine for a day before it used up all the CPU and memory resources. A fast check on 'topas' show CPU resources used up for Kernel and User. There are no free resources on Wait and Idle. Around 70% of the CPU resources used for User and 30% used for Kernel. On memory side, seem all the physical memory had been consumed up. Even Paging space also been used. A quick snapshot on the memory from 'topas' as below: MEMORY Real,MB4095 % Comp 22.4 % Noncomp 76.2 % Client 75.1 PAGING SPACE Size,MB4096 % Used 1.4 % Free 98.5 When all the physical resources are fully occupied, the Universe processing become slow. The only thing I can do now is to restart Universe when the performance degraded? Are there any performance tuning we need to do on the OS to prevent this issue? Or is there any known issue with this version of Universe? Please assist me to solve this problem. Regard's, Foo --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.657 / Virus Database: 422 - Release Date: 4/13/2004 -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
