Re: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
Hmmm. Are you saying 'Ogres' are like onions? On Fri, 2004-04-16 at 07:05, Scott Richardson wrote: > Performance of UV applications on various Operating Systems > is not rocket science. Perhaps better described as large, nasty > tight onions that need peeling, one layer at a time, and > understanding what each peeled layer is doing and why. > Once this knowledge is acquired and understood, a plan can > be built and executed to attack/resolve the problem. > > Are users logging out/off when they're done using the system, > or when they've completed some large tasks or operations? > How often is the system rebooted? > RAID 5 file systems can slow down IO. > We'll need specifics on file system setup and parameters. > How many users? What are these users doing? > Have you got everyone and their siblings all running SELECT and > SORT operations all the time? Data Entry out the wazoo? > > How big are the files, and how are they sized? How frequently > does data change in the files, (grow, shrink, etc...) > > How big is your /tmp file system, and what kind of file system, > and where is it physically located?? Provide it it's own file system, > on it's own disk or disk set, (i.e. not the same disks where other > activity is going on). > > 4GB of RAM, yet only 4 GB paging/swap space? > Where is this swap paging space, (i.e. what disks?) > > "topas" may be fine for quick and dirty analysis and understanding, > but using it extensively can help contribute to performance problems. > > You need to configure and tune the platform, the OS, the UV DB, > the IO sub-system, the applications, the users, and the > administration/operations, and thenensure they're all coordinated > with each other, to maximize platform performance. > > To find, (and therefore address & resolve), the root causes of what > is happening here, you need to profile the platform using something > such as the DPMonitor, (extremely low-overhead monitoring Agent) > and display/crunch the performance metrics on another platform, > (i..e. a Windows Performance Explorer Console). Using this method, > you'll be able completely profile the entire platform, (OS and > applications), > around the clock, and then easily dial into specific timeframes where > problems are occurring, and fully understand exactly what is happening > and learn why it is happening, so it can be addressed and resolved, > and measure the progress along the entire way. > > The DPMonitor is available with a free 10 day evaluation license where it > will track system-wide performance metrics. Fully licensed version will > track individual processes that you select, or all processes if you so > desire. When you monitor all of the processes, you can quickly and > easily identify processes deserving further analysis, and stop tracking > processes that are not casuing any problems. More information on the > DPMonitor can be found at http://www.deltek.us and the DPMonitor > can be downloaded right off the website. If you're short on memory, > DPMonitor will allow you to see how much memory you will need to > allow the system to run as fast as it can, given how you're running it. > If you need tuning of OS or UV parameters, or other things that ay be > playing contributing factor/roles, the DPMonitor will clearly point this > out, > grahically, so that anyone can plainly see what is happening. > > Once you make any changes, you'll be able to monitor, and measure, > any differences, consistently, and prove whether or not you have > improved, or detrimented, your cause. Best of all, you'll be able to > show, prove, and justify to management what you're doing, and > why, and show them what it will take to get the problems addressed > and resolved, positively, without question. > > Hope this helps. I know the DPMonitor can & will help. > I have used it personally, numerous times, to peel many a complex onion, > understand what is exactly going on, find out why, and then put together > and executed plans that have successfully addressed and resolved similar > problems and streamlined operations moving forward saving many a > business significant time, frustration, and money, and then ensured that > any & all operations moving forward were done from a pro-active, > knowing ahead of time manner, rather than fire-fighting problems on a > continual basis. If you want something done, why not do it right, once? > Stop beating your head against the onion wall! Work smarter! > Let the DPMonitor be your detailed, EKG-like instrument to cut to > the heart of your complex application server performance problems, > identify them, and help you to resolve them, quickly and easily. > > > Been there, done that. > Many times over. > > Sincere Regards, > Scott Richardson > Senior Systems Engineer / Consultant > Marlborough, MA 01752 > Email: [EMAIL PROTECTED] > Web: http://home.comcast.net/~CheetahFTL/CC/CheetahFTL_1.htm > eFax: 208-445-1259 > > - Original Message - > From: "Foo Chia Teck" <
RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
Why? He's got 4Gb of ram. If he overflows that by JUST ONE BYTE, if he was running linux his performance would *collapse*. If you do not have twice ram as swap, you only need to use ONE BYTE of swap space and the linux algorithm will shit itself trying to cope. The algorithm does not work very well if it doesn't have enough disk space, and that's where the "twice swap" rule came from. So it's a very old unix algorithm ... which is why I suspect AIX might suffer similarly. Cheers, Wol -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steve Ferries Sent: 16 April 2004 14:45 To: U2 Users Discussion List Subject: RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 HI All, Before doubling you swap space, check to see how much you are using at your busy times. We have an 8 Gig system, and a 6 Gig pool: Page Space Physical Volume Volume GroupSize %Used Active Auto Type paging00hdisk1rootvg6144MB 2 yes yes lv Everyone into the pool! Regards, Steve Ferries Vice President, Information Technologies Total Credit Recovery Limited -Original Message- From: Anthony Youngman [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] Sent: Friday, April 16, 2004 9:31 AM To: U2 Users Discussion List Subject: RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Okay, it's AIX not linux, but I've just noticed that RAM = swap. You are an ABSOLUTE FOOL if you do that on linux. Maybe (or maybe not) the same applies to AIX - quite likely since they are both nixen and probably manage memory similiarly. Double swap space to 8Gb and see if that improves matters. Oh - and if you don't believe me, a "swap = ram" configuration will CRASH the early vanilla 2.4 kernels and that's 2002 vintage. Cheers, Wol -Original Message- From: [EMAIL PROTECTED] [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] On Behalf Of Scott Richardson Sent: 16 April 2004 14:06 To: U2 Users Discussion List Subject: Re: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Performance of UV applications on various Operating Systems is not rocket science. Perhaps better described as large, nasty tight onions that need peeling, one layer at a time, and understanding what each peeled layer is doing and why. Once this knowledge is acquired and understood, a plan can be built and executed to attack/resolve the problem. Are users logging out/off when they're done using the system, or when they've completed some large tasks or operations? How often is the system rebooted? RAID 5 file systems can slow down IO. We'll need specifics on file system setup and parameters. How many users? What are these users doing? Have you got everyone and their siblings all running SELECT and SORT operations all the time? Data Entry out the wazoo? How big are the files, and how are they sized? How frequently does data change in the files, (grow, shrink, etc...) How big is your /tmp file system, and what kind of file system, and where is it physically located?? Provide it it's own file system, on it's own disk or disk set, (i.e. not the same disks where other activity is going on). 4GB of RAM, yet only 4 GB paging/swap space? Where is this swap paging space, (i.e. what disks?) "topas" may be fine for quick and dirty analysis and understanding, but using it extensively can help contribute to performance problems. You need to configure and tune the platform, the OS, the UV DB, the IO sub-system, the applications, the users, and the administration/operations, and thenensure they're all coordinated with each other, to maximize platform performance. To find, (and therefore address & resolve), the root causes of what is happening here, you need to profile the platform using something such as the DPMonitor, (extremely low-overhead monitoring Agent) and display/crunch the performance metrics on another platform, (i..e. a Windows Performance Explorer Console). Using this method, you'll be able completely profile the entire platform, (OS and applications), around the clock, and then easily dial into specific timeframes where problems are occurring, and fully understand exactly what is happening and learn why it is happening, so it can be addressed and resolved, and measure the progress along the entire way. The DPMonitor is available with a free 10 day evaluation license where it will track system-wide performance metrics. Fully licensed version will track individual processes that you select, or all processes if you so desire. When you monitor all of the processes, you can quickly and easily identify processes deserving further analysis, and stop tracking processes that are not casuing any problems. More information on the DPMonitor can be found at <http://www.deltek.us> http://www.deltek.us and the DPMonitor can be downloaded right o
RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
HI All, Before doubling you swap space, check to see how much you are using at your busy times. We have an 8 Gig system, and a 6 Gig pool: Page Space Physical Volume Volume GroupSize %Used Active Auto Type paging00hdisk1rootvg6144MB 2 yes yeslv Everyone into the pool! Regards, Steve Ferries Vice President, Information Technologies Total Credit Recovery Limited -Original Message- From: Anthony Youngman [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] Sent: Friday, April 16, 2004 9:31 AM To: U2 Users Discussion List Subject: RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Okay, it's AIX not linux, but I've just noticed that RAM = swap. You are an ABSOLUTE FOOL if you do that on linux. Maybe (or maybe not) the same applies to AIX - quite likely since they are both nixen and probably manage memory similiarly. Double swap space to 8Gb and see if that improves matters. Oh - and if you don't believe me, a "swap = ram" configuration will CRASH the early vanilla 2.4 kernels and that's 2002 vintage. Cheers, Wol -Original Message- From: [EMAIL PROTECTED] [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] On Behalf Of Scott Richardson Sent: 16 April 2004 14:06 To: U2 Users Discussion List Subject: Re: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Performance of UV applications on various Operating Systems is not rocket science. Perhaps better described as large, nasty tight onions that need peeling, one layer at a time, and understanding what each peeled layer is doing and why. Once this knowledge is acquired and understood, a plan can be built and executed to attack/resolve the problem. Are users logging out/off when they're done using the system, or when they've completed some large tasks or operations? How often is the system rebooted? RAID 5 file systems can slow down IO. We'll need specifics on file system setup and parameters. How many users? What are these users doing? Have you got everyone and their siblings all running SELECT and SORT operations all the time? Data Entry out the wazoo? How big are the files, and how are they sized? How frequently does data change in the files, (grow, shrink, etc...) How big is your /tmp file system, and what kind of file system, and where is it physically located?? Provide it it's own file system, on it's own disk or disk set, (i.e. not the same disks where other activity is going on). 4GB of RAM, yet only 4 GB paging/swap space? Where is this swap paging space, (i.e. what disks?) "topas" may be fine for quick and dirty analysis and understanding, but using it extensively can help contribute to performance problems. You need to configure and tune the platform, the OS, the UV DB, the IO sub-system, the applications, the users, and the administration/operations, and thenensure they're all coordinated with each other, to maximize platform performance. To find, (and therefore address & resolve), the root causes of what is happening here, you need to profile the platform using something such as the DPMonitor, (extremely low-overhead monitoring Agent) and display/crunch the performance metrics on another platform, (i..e. a Windows Performance Explorer Console). Using this method, you'll be able completely profile the entire platform, (OS and applications), around the clock, and then easily dial into specific timeframes where problems are occurring, and fully understand exactly what is happening and learn why it is happening, so it can be addressed and resolved, and measure the progress along the entire way. The DPMonitor is available with a free 10 day evaluation license where it will track system-wide performance metrics. Fully licensed version will track individual processes that you select, or all processes if you so desire. When you monitor all of the processes, you can quickly and easily identify processes deserving further analysis, and stop tracking processes that are not casuing any problems. More information on the DPMonitor can be found at <http://www.deltek.us> http://www.deltek.us and the DPMonitor can be downloaded right off the website. If you're short on memory, DPMonitor will allow you to see how much memory you will need to allow the system to run as fast as it can, given how you're running it. If you need tuning of OS or UV parameters, or other things that ay be playing contributing factor/roles, the DPMonitor will clearly point this out, grahically, so that anyone can plainly see what is happening. Once you make any changes, you'll be able to monitor, and measure, any differences, consistently, and prove whether or not you have improved, or detrimented, your cause. Best of all, you'll be able to show, prove, and justify to management what you're doing, and why, and show them what it will take to get the problems add
RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
Okay, it's AIX not linux, but I've just noticed that RAM = swap. You are an ABSOLUTE FOOL if you do that on linux. Maybe (or maybe not) the same applies to AIX - quite likely since they are both nixen and probably manage memory similiarly. Double swap space to 8Gb and see if that improves matters. Oh - and if you don't believe me, a "swap = ram" configuration will CRASH the early vanilla 2.4 kernels and that's 2002 vintage. Cheers, Wol -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Scott Richardson Sent: 16 April 2004 14:06 To: U2 Users Discussion List Subject: Re: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Performance of UV applications on various Operating Systems is not rocket science. Perhaps better described as large, nasty tight onions that need peeling, one layer at a time, and understanding what each peeled layer is doing and why. Once this knowledge is acquired and understood, a plan can be built and executed to attack/resolve the problem. Are users logging out/off when they're done using the system, or when they've completed some large tasks or operations? How often is the system rebooted? RAID 5 file systems can slow down IO. We'll need specifics on file system setup and parameters. How many users? What are these users doing? Have you got everyone and their siblings all running SELECT and SORT operations all the time? Data Entry out the wazoo? How big are the files, and how are they sized? How frequently does data change in the files, (grow, shrink, etc...) How big is your /tmp file system, and what kind of file system, and where is it physically located?? Provide it it's own file system, on it's own disk or disk set, (i.e. not the same disks where other activity is going on). 4GB of RAM, yet only 4 GB paging/swap space? Where is this swap paging space, (i.e. what disks?) "topas" may be fine for quick and dirty analysis and understanding, but using it extensively can help contribute to performance problems. You need to configure and tune the platform, the OS, the UV DB, the IO sub-system, the applications, the users, and the administration/operations, and thenensure they're all coordinated with each other, to maximize platform performance. To find, (and therefore address & resolve), the root causes of what is happening here, you need to profile the platform using something such as the DPMonitor, (extremely low-overhead monitoring Agent) and display/crunch the performance metrics on another platform, (i..e. a Windows Performance Explorer Console). Using this method, you'll be able completely profile the entire platform, (OS and applications), around the clock, and then easily dial into specific timeframes where problems are occurring, and fully understand exactly what is happening and learn why it is happening, so it can be addressed and resolved, and measure the progress along the entire way. The DPMonitor is available with a free 10 day evaluation license where it will track system-wide performance metrics. Fully licensed version will track individual processes that you select, or all processes if you so desire. When you monitor all of the processes, you can quickly and easily identify processes deserving further analysis, and stop tracking processes that are not casuing any problems. More information on the DPMonitor can be found at http://www.deltek.us and the DPMonitor can be downloaded right off the website. If you're short on memory, DPMonitor will allow you to see how much memory you will need to allow the system to run as fast as it can, given how you're running it. If you need tuning of OS or UV parameters, or other things that ay be playing contributing factor/roles, the DPMonitor will clearly point this out, grahically, so that anyone can plainly see what is happening. Once you make any changes, you'll be able to monitor, and measure, any differences, consistently, and prove whether or not you have improved, or detrimented, your cause. Best of all, you'll be able to show, prove, and justify to management what you're doing, and why, and show them what it will take to get the problems addressed and resolved, positively, without question. Hope this helps. I know the DPMonitor can & will help. I have used it personally, numerous times, to peel many a complex onion, understand what is exactly going on, find out why, and then put together and executed plans that have successfully addressed and resolved similar problems and streamlined operations moving forward saving many a business significant time, frustration, and money, and then ensured that any & all operations moving forward were done from a pro-active, knowing ahead of time manner, rather than fire-fighting problems on a continual basis. If you want something done, why not do it right, once? Stop beating your head against the onion wall! Work smarter
Re: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
Performance of UV applications on various Operating Systems is not rocket science. Perhaps better described as large, nasty tight onions that need peeling, one layer at a time, and understanding what each peeled layer is doing and why. Once this knowledge is acquired and understood, a plan can be built and executed to attack/resolve the problem. Are users logging out/off when they're done using the system, or when they've completed some large tasks or operations? How often is the system rebooted? RAID 5 file systems can slow down IO. We'll need specifics on file system setup and parameters. How many users? What are these users doing? Have you got everyone and their siblings all running SELECT and SORT operations all the time? Data Entry out the wazoo? How big are the files, and how are they sized? How frequently does data change in the files, (grow, shrink, etc...) How big is your /tmp file system, and what kind of file system, and where is it physically located?? Provide it it's own file system, on it's own disk or disk set, (i.e. not the same disks where other activity is going on). 4GB of RAM, yet only 4 GB paging/swap space? Where is this swap paging space, (i.e. what disks?) "topas" may be fine for quick and dirty analysis and understanding, but using it extensively can help contribute to performance problems. You need to configure and tune the platform, the OS, the UV DB, the IO sub-system, the applications, the users, and the administration/operations, and thenensure they're all coordinated with each other, to maximize platform performance. To find, (and therefore address & resolve), the root causes of what is happening here, you need to profile the platform using something such as the DPMonitor, (extremely low-overhead monitoring Agent) and display/crunch the performance metrics on another platform, (i..e. a Windows Performance Explorer Console). Using this method, you'll be able completely profile the entire platform, (OS and applications), around the clock, and then easily dial into specific timeframes where problems are occurring, and fully understand exactly what is happening and learn why it is happening, so it can be addressed and resolved, and measure the progress along the entire way. The DPMonitor is available with a free 10 day evaluation license where it will track system-wide performance metrics. Fully licensed version will track individual processes that you select, or all processes if you so desire. When you monitor all of the processes, you can quickly and easily identify processes deserving further analysis, and stop tracking processes that are not casuing any problems. More information on the DPMonitor can be found at http://www.deltek.us and the DPMonitor can be downloaded right off the website. If you're short on memory, DPMonitor will allow you to see how much memory you will need to allow the system to run as fast as it can, given how you're running it. If you need tuning of OS or UV parameters, or other things that ay be playing contributing factor/roles, the DPMonitor will clearly point this out, grahically, so that anyone can plainly see what is happening. Once you make any changes, you'll be able to monitor, and measure, any differences, consistently, and prove whether or not you have improved, or detrimented, your cause. Best of all, you'll be able to show, prove, and justify to management what you're doing, and why, and show them what it will take to get the problems addressed and resolved, positively, without question. Hope this helps. I know the DPMonitor can & will help. I have used it personally, numerous times, to peel many a complex onion, understand what is exactly going on, find out why, and then put together and executed plans that have successfully addressed and resolved similar problems and streamlined operations moving forward saving many a business significant time, frustration, and money, and then ensured that any & all operations moving forward were done from a pro-active, knowing ahead of time manner, rather than fire-fighting problems on a continual basis. If you want something done, why not do it right, once? Stop beating your head against the onion wall! Work smarter! Let the DPMonitor be your detailed, EKG-like instrument to cut to the heart of your complex application server performance problems, identify them, and help you to resolve them, quickly and easily. Been there, done that. Many times over. Sincere Regards, Scott Richardson Senior Systems Engineer / Consultant Marlborough, MA 01752 Email: [EMAIL PROTECTED] Web: http://home.comcast.net/~CheetahFTL/CC/CheetahFTL_1.htm eFax: 208-445-1259 - Original Message - From: "Foo Chia Teck" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, April 16, 2004 2:22 AM Subject: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Hi, We are facing performance degraded when running Universe 10.0.0 in AIX 5L 5.2. A bit intro on hardware specs. We are using pSeries 650 running o
RE: Performance Degraded running u10.0.0 in Aix 5.2 ML 2
Hi Foo Could it be an application problem. It sounds like an application is adding data to a possible type 1 file as a log or a como file or even a print job and it is growing to an enormous size consuming resources. There might be some other application that has gone rogue. Can you identify the time this problem started and relate it to a program that was updated. Can you identify a user who is using a lot of the resources and find out what they are doing. Regards David Jordan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Foo Chia Teck Sent: Friday, 16 April 2004 4:22 PM To: [EMAIL PROTECTED] Subject: Performance Degraded running u10.0.0 in Aix 5.2 ML 2 Hi, We are facing performance degraded when running Universe 10.0.0 in AIX 5L 5.2. A bit intro on hardware specs. We are using pSeries 650 running on SMP 2 Power4 processor with 4GB of RAM, 4GB Paging size and RAID5 SSA Hard Disk. My Universe configuration as below: Current tunable parameter settings: MFILES = 300 T30FILE= 200 OPENCHK= 1 WIDE0 = 0x3dc0 UVSPOOL= /uvspool1 UVTEMP = /uvtmp1 SCRMIN = 3 SCRMAX = 5 SCRSIZE= 512 QDEPTH = 16 HISTSTK= 99 QSRUNSZ= 2000 QSBRNCH= 4 QSDEPTH= 8 QSMXKEY= 32 TXMODE = 0 LOGBLSZ= 512 LOGBLNUM = 8 LOGSYCNT = 0 LOGSYINT = 0 TXMEM = 32 OPTMEM = 64 SELBUF = 4 ULIMIT = 128000 FSEMNUM= 23 GSEMNUM= 97 PSEMNUM= 64 FLTABSZ= 11 GLTABSZ= 75 RLTABSZ= 75 RLOWNER= 300 PAKTIME= 5 NETTIME= 5 QBREAK = 1 VDIVDEF= 1 UVSYNC = 1 BLKMAX = 131072 PICKNULL = 0 SYNCALOC = 1 MAXRLOCK = 74 ISOMODE= 1 PKRJUST= 0 PROCACMD = 0 PROCRCMD = 0 PROCPRMT = 0 ALLOWNFS = 0 CSHDISPATCH= /bin/csh SHDISPATCH = /bin/sh DOSDISPATCH= NOT_SUPPORTED LAYERSEL = 0 OCVDATE= 0 MODFPTRS = 1 THDR512= 0 UDRMODE= 0 UDRBLKS= 0 MAXERRLOGENT = 100 JOINBUF= 4095 64BIT_FILES= 0 TSTIMEOUT = 60 PIOPENDEFAULT = 0 MAXKEYSIZE = 255 SMISDATA = 0 EXACTNUMERIC = 15 MALLOCTRACING = 0 CENTURYPIVOT = 1930 SPINTRIES = 0 SPINSLEEP = 1 CONVERT_EURO = 0 SYSTEM_EURO= 164 TERM_EURO = 164 SQLNULL= 128 When the uv restarted it run fine for a day before it used up all the CPU and memory resources. A fast check on 'topas' show CPU resources used up for Kernel and User. There are no free resources on Wait and Idle. Around 70% of the CPU resources used for User and 30% used for Kernel. On memory side, seem all the physical memory had been consumed up. Even Paging space also been used. A quick snapshot on the memory from 'topas' as below: MEMORY Real,MB4095 % Comp 22.4 % Noncomp 76.2 % Client 75.1 PAGING SPACE Size,MB4096 % Used 1.4 % Free 98.5 When all the physical resources are fully occupied, the Universe processing become slow. The only thing I can do now is to restart Universe when the performance degraded? Are there any performance tuning we need to do on the OS to prevent this issue? Or is there any known issue with this version of Universe? Please assist me to solve this problem. Regard's, Foo --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.657 / Virus Database: 422 - Release Date: 4/13/2004 -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
Re: Performance
Pick used separation on R83. When they moved to Advanced Pick (AP), they removed the concept of separation and force it to 1. Eugene - Original Message - From: "Stevenson, Charles" <[EMAIL PROTECTED]> To: "U2 Users Discussion List" <[EMAIL PROTECTED]> Sent: Friday, April 09, 2004 2:24 PM Subject: RE: Performance Kevin, When you finally get this solved, let us know what the answer was. I am sure all responders would be interested. re. /tmp: I've seen marginal but not incredible inprovement moving UVTEMP onto our EMC storage rather than on the system's local disk (/tmp) re. file sizing: since you are porting from D3, I assume yoou made everything type 18, which is the standard Pick hashing algorithm? That ought to behave about the same as it did on D3. How about Separation? Does D3 have that concept? I don't think Jeff mentioned it. For most files you want to set separation such that you get integer number of groups for each OS disk read. If a sigle disk read grabs 8K, then separation 4 (512*4= 2K/group) means filescans will ask for a group, the OS will read in 4 groups, and the next 3 times the process asks for the next group, it's probably still sitting in memory. So if the OS does read 8K at a time, separations of 1,2,4,8,16,12 make sense, depending on the nature of the records. 4 is typical. re. locks: I notice the lock table is pretty small, and there are a lots of 'CLR.OM.LOCK" proceses. Is this one of those PICK aps where people developed their own record locking scheme because they didn't trust PICK's record lock handling? If so, maybe that is a source of ineffeciency. It's not clear how that would manifest itself with paging, though. What about loading programs into shared memory? Do you have an absolutely huge program that many users use? By default they each load their own copy of the object file into their private memory. But you can change that so only one copy is loaded. The same with small utility routines that get called by everyone throughout the day. Load them once in shared memory, then all users will run off that copy. Again, we're talking incremental, not incredible, performance improvements. I'm grasping here. I'm sure IBM's Hdwr, AIX, & U2 support has gone through all this already. You will post the answer once you know it, won't you? cds -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
Thanks to everyone for the performance suggestions...I will report to the board as soon as we resolve it. Kevin Kevin D. Vezertzis Project Manager Cypress Business Solutions, LLC. 678.494.9353 ext. 6576 Fax 678.494.9354 [EMAIL PROTECTED] Visit us at www.cypressesolutions.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Stevenson, Charles Sent: Friday, April 09, 2004 4:24 PM To: U2 Users Discussion List Subject: RE: Performance Kevin, When you finally get this solved, let us know what the answer was. I am sure all responders would be interested. re. /tmp: I've seen marginal but not incredible inprovement moving UVTEMP onto our EMC storage rather than on the system's local disk (/tmp) re. file sizing: since you are porting from D3, I assume yoou made everything type 18, which is the standard Pick hashing algorithm? That ought to behave about the same as it did on D3. How about Separation? Does D3 have that concept? I don't think Jeff mentioned it. For most files you want to set separation such that you get integer number of groups for each OS disk read. If a sigle disk read grabs 8K, then separation 4 (512*4= 2K/group) means filescans will ask for a group, the OS will read in 4 groups, and the next 3 times the process asks for the next group, it's probably still sitting in memory. So if the OS does read 8K at a time, separations of 1,2,4,8,16,12 make sense, depending on the nature of the records. 4 is typical. re. locks: I notice the lock table is pretty small, and there are a lots of 'CLR.OM.LOCK" proceses. Is this one of those PICK aps where people developed their own record locking scheme because they didn't trust PICK's record lock handling? If so, maybe that is a source of ineffeciency. It's not clear how that would manifest itself with paging, though. What about loading programs into shared memory? Do you have an absolutely huge program that many users use? By default they each load their own copy of the object file into their private memory. But you can change that so only one copy is loaded. The same with small utility routines that get called by everyone throughout the day. Load them once in shared memory, then all users will run off that copy. Again, we're talking incremental, not incredible, performance improvements. I'm grasping here. I'm sure IBM's Hdwr, AIX, & U2 support has gone through all this already. You will post the answer once you know it, won't you? cds -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
Kevin, When you finally get this solved, let us know what the answer was. I am sure all responders would be interested. re. /tmp: I've seen marginal but not incredible inprovement moving UVTEMP onto our EMC storage rather than on the system's local disk (/tmp) re. file sizing: since you are porting from D3, I assume yoou made everything type 18, which is the standard Pick hashing algorithm? That ought to behave about the same as it did on D3. How about Separation? Does D3 have that concept? I don't think Jeff mentioned it. For most files you want to set separation such that you get integer number of groups for each OS disk read. If a sigle disk read grabs 8K, then separation 4 (512*4= 2K/group) means filescans will ask for a group, the OS will read in 4 groups, and the next 3 times the process asks for the next group, it's probably still sitting in memory. So if the OS does read 8K at a time, separations of 1,2,4,8,16,12 make sense, depending on the nature of the records. 4 is typical. re. locks: I notice the lock table is pretty small, and there are a lots of 'CLR.OM.LOCK" proceses. Is this one of those PICK aps where people developed their own record locking scheme because they didn't trust PICK's record lock handling? If so, maybe that is a source of ineffeciency. It's not clear how that would manifest itself with paging, though. What about loading programs into shared memory? Do you have an absolutely huge program that many users use? By default they each load their own copy of the object file into their private memory. But you can change that so only one copy is loaded. The same with small utility routines that get called by everyone throughout the day. Load them once in shared memory, then all users will run off that copy. Again, we're talking incremental, not incredible, performance improvements. I'm grasping here. I'm sure IBM's Hdwr, AIX, & U2 support has gone through all this already. You will post the answer once you know it, won't you? cds -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
Re: Performance
In a message dated 4/8/2004 12:20:45 PM Pacific Daylight Time, [EMAIL PROTECTED] writes: > 1.) Application files have all been analyzed and sized correctly. > 2.) IBM U2 support analyzed Universe files, locking, swap space and all > have been adjusted accordingly or were 'ok'. > 3.) We are running RAID 5, with 8G allocated for Universe > 4.) We are already running nmon, which is how we identified the paging > faults and high disk I/O Indexes created without suppressing NULLs -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
You've gotten some good input, but I'd like to recommend dumping Raid 5 altogether. With several clients I've seen significant performance increases from switching from (recommended) Raid 5 arrays to what may be referred to as Raid 10 (Raid 0+1/1+0) arrays. This type of array mirrors pairs of striped disk arrays (or vice versa) and will definately perform better than a Raid 5 array. I don't recall specific numbers, but do remember clients smiling after making the change where frowns which your client apparently now have. Raid 5 uses a parity 'disk' which requires multiple reads and writes for each disk I/O operation. Removing that overhead alone will boost performance by as much as 8 to 20 percent, depending on the configuration and how many spindles are affected. Also, it might be worth the time to change the uvconfig parameters that affect SELECTs starting with the location of UVTEMP as previously recommended. I always do that for my clients as part of an install so if the tmp files need to be removed, /tmp doesn't have to be messed with. Another one is the SELBUF setting, which determines how much of a select is done in memory before going out to disk. Setting this higher can improve performance especially where disk I/O is suffering. FSEMNUM, GSEMNUM and PSEMNUM are other settings you may want to adjust to see if disk I/O will improve. There is one other issue with AIX you may be experiencing. For some reason, some sessions attached through TCP/IP links 'go away' and leave the session open. I have to use the TANDEM function to check those processes. They are usually at some prompt cycling through whatever they are doing as though someone is sitting at a keyboard with a book leaning on the ENTER key causing it to repeat over and over. AIX has the 'w' command which shows CPU time chunks each user has. There are 3 columns and if the 2nd or 3rd column is high and the first is 0 then that session is a candidate for having problems. One major clue something like this is happening is if the results of the uptime command show numbers unusually high. But all in all, you've gotten some good suggestions already and I may be up in the night, so to speak. Karl On Thu, 2004-04-08 at 13:18, Kevin Vezertzis wrote: > Thanks for all of the posts...here are some of our 'knowns'... > > 1.) Application files have all been analyzed and sized correctly. > 2.) IBM U2 support analyzed Universe files, locking, swap space and all > have been adjusted accordingly or were 'ok'. > 3.) We are running RAID 5, with 8G allocated for Universe > 4.) We are already running nmon, which is how we identified the paging > faults and high disk I/O > > 4.) Attached you will find the following: > smat -s > LIST.READU EVERY > PORT.STATUS > Uvconfig > Nmon (verbose and disk) > Vmtune > > I know this is a lot of data, but it is a mix of what each of you have > suggested. Thanks again for all of the help. > > Kevin > > > > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > On Behalf Of Kevin Vezertzis > Sent: Thursday, April 08, 2004 12:08 PM > To: [EMAIL PROTECTED] > Subject: Performance > > We are looking for some insight from anyone that has experienced > performance degradation in UV, as it relates to the OS. We are running > UV 10.0.14 on AIX 5.1.we are having terrible 'latency' within the > application. This is a recent conversion from D3 to UV and our client > is extremely disappointed with the performance. We've had IBM hardware > support and Universe support in on the box, but to no avail..we are > seeing high paging faults and very highly utilized disk space. Any > thoughts or suggestions? > > Thanks, > Kevin > > __ > -- > u2-users mailing list > [EMAIL PROTECTED] > http://www.oliver.com/mailman/listinfo/u2-users -- Karl L. Pearson Director of IT, ATS Industrial Supply Direct: 801-978-4429 Toll-free: 888-972-3182 x29 Fax: 801-972-3888 http://www.atsindustrial.com [EMAIL PROTECTED] -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
Kevin As this is AIX (and only on AIX please - as it has it's own memory flush) - set UVSYNC=0 in uvconfig and run "uvregen" (Universe stopped please) This stops UniVerse doing its own sync There is a small risk with this but not much. Please do *not* change SYNCALLOC. FSEMNUM and GSEMNUM look fine (assuming this was under load). UVTEMP could do with relocating off root onto another volume - ideally on a SAN. I am sure you know that RAID 5 costs disk throughput (parity disk writes - but this is a business call - just be aware). There are a lot of processes in CLR.OM.LOCK - what does this do and is it intensive? Looking at SELECTs in use - we don't see the file sizes information etc - but check out the following: SELECT UNPOSTED.MRE WITH TEMP.LOC EQ "BS2" BY BIN.SORT BY PROD SSELECT CATALOG.DETAIL BY-DSND DATE.TO (seen 3x) SSELECT INVENTORY WITH 67 = "PETSAFE" (seen 2x) SSELECT MAIL.MSG BY-DSND CREATE.DATE BY-DSND CREATE.TIME SSELECT ADJ.CODES SSELECT SMTP WITH 6 EQ "" SSELECT IMPORT.ORDERS Are these SELECTs performed a lot and are the files large? - if so look at indexes and watch the use of NO.NULLS where appropriate. Also watch that you do *not* build index on DICT itmes which translate to data out of the primary file (things like DATE() or a file translate). I have ignored a couple of SELECTS with SAMPLE statements. Would be interested in sar output for file opens - hopefully you are caching these file handles in COMMON (ideally named) - also the run queue (how many CPUs? - the run queue should not (other than very occasionally) exceed more than 2x the number of CPUs on the system (or in the LPAR if you use these). By "heavily utilized disk space" do you mean you use lots of disk ? - if so - and it seems disproportionate to yout data volumes - then are you using lots of dynamic files and if so have you tuned the file split/merge ratios?. Final points - I am sort-of-assuming that MFILES is appropriately set..? (see "PORT.STATUS .. MFILE.HIST"). You might also want to check out FILE.USAGE. Email me off-line if you want to go through any reports you already have on this - let's see what was and what was not covered (saves retreading old ground). Regards JayJay -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Vezertzis Sent: 08 April 2004 20:18 To: 'U2 Users Discussion List' Subject: RE: Performance Thanks for all of the posts...here are some of our 'knowns'... 1.) Application files have all been analyzed and sized correctly. 2.) IBM U2 support analyzed Universe files, locking, swap space and all have been adjusted accordingly or were 'ok'. 3.) We are running RAID 5, with 8G allocated for Universe 4.) We are already running nmon, which is how we identified the paging faults and high disk I/O 4.) Attached you will find the following: smat -s LIST.READU EVERY PORT.STATUS Uvconfig Nmon (verbose and disk) Vmtune I know this is a lot of data, but it is a mix of what each of you have suggested. Thanks again for all of the help. Kevin -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Vezertzis Sent: Thursday, April 08, 2004 12:08 PM To: [EMAIL PROTECTED] Subject: Performance We are looking for some insight from anyone that has experienced performance degradation in UV, as it relates to the OS. We are running UV 10.0.14 on AIX 5.1.we are having terrible 'latency' within the application. This is a recent conversion from D3 to UV and our client is extremely disappointed with the performance. We've had IBM hardware support and Universe support in on the box, but to no avail..we are seeing high paging faults and very highly utilized disk space. Any thoughts or suggestions? Thanks, Kevin -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
I saw /tmp mentioned in one of Scott Richardson's posts. UniVerse uses /tmp for building select lists and sorts. If it is undersized, it can cause page faults like you are seeing. How big is /tmp? It might pay to monitor it's usage. It can be as critical as having adequate swap / paging space. Double check Scott's recommendation on /tmp. Bob Gerrish - [EMAIL PROTECTED] Kingsgate Enterprises, Inc. At 12:18 PM 4/8/2004, you wrote: Thanks for all of the posts...here are some of our 'knowns'... 1.) Application files have all been analyzed and sized correctly. 2.) IBM U2 support analyzed Universe files, locking, swap space and all have been adjusted accordingly or were 'ok'. 3.) We are running RAID 5, with 8G allocated for Universe 4.) We are already running nmon, which is how we identified the paging faults and high disk I/O 4.) Attached you will find the following: smat -s LIST.READU EVERY PORT.STATUS Uvconfig Nmon (verbose and disk) Vmtune I know this is a lot of data, but it is a mix of what each of you have suggested. Thanks again for all of the help. Kevin -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Vezertzis Sent: Thursday, April 08, 2004 12:08 PM To: [EMAIL PROTECTED] Subject: Performance We are looking for some insight from anyone that has experienced performance degradation in UV, as it relates to the OS. We are running UV 10.0.14 on AIX 5.1.we are having terrible 'latency' within the application. This is a recent conversion from D3 to UV and our client is extremely disappointed with the performance. We've had IBM hardware support and Universe support in on the box, but to no avail..we are seeing high paging faults and very highly utilized disk space. Any thoughts or suggestions? Thanks, Kevin -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
Thanks for all of the posts...here are some of our 'knowns'... 1.) Application files have all been analyzed and sized correctly. 2.) IBM U2 support analyzed Universe files, locking, swap space and all have been adjusted accordingly or were 'ok'. 3.) We are running RAID 5, with 8G allocated for Universe 4.) We are already running nmon, which is how we identified the paging faults and high disk I/O 4.) Attached you will find the following: smat -s LIST.READU EVERY PORT.STATUS Uvconfig Nmon (verbose and disk) Vmtune I know this is a lot of data, but it is a mix of what each of you have suggested. Thanks again for all of the help. Kevin -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Vezertzis Sent: Thursday, April 08, 2004 12:08 PM To: [EMAIL PROTECTED] Subject: Performance We are looking for some insight from anyone that has experienced performance degradation in UV, as it relates to the OS. We are running UV 10.0.14 on AIX 5.1.we are having terrible 'latency' within the application. This is a recent conversion from D3 to UV and our client is extremely disappointed with the performance. We've had IBM hardware support and Universe support in on the box, but to no avail..we are seeing high paging faults and very highly utilized disk space. Any thoughts or suggestions? Thanks, Kevin File access State Netnode Owner Collisions Retries Semaphore # 1 00 0 0 0 Semaphore # 2 00 0 0 0 Semaphore # 3 00 0 0 0 Semaphore # 4 00 0 0 0 Semaphore # 5 00 0 0 0 Semaphore # 6 00 0 0 0 Semaphore # 7 00 0 0 0 Semaphore # 8 00 0 0 0 Semaphore # 9 00 0 0 0 Semaphore # 10 00 0 0 0 Semaphore # 11 00 0 0 0 Semaphore # 12 00 0 0 0 Semaphore # 13 00 0 0 0 Semaphore # 14 00 0 0 0 Semaphore # 15 00 0 0 0 Semaphore # 16 00 0 0 0 Semaphore # 17 00 0 0 0 Semaphore # 18 00 0 0 0 Semaphore # 19 00 0 0 0 Semaphore # 20 00 0 0 0 Semaphore # 21 00 0 0 0 Semaphore # 22 00 0 0 0 Semaphore # 23 00 0 0 0 Group accessState Netnode Owner Collisions Retries Semaphore # 1 00 0 34 34 Semaphore # 2 00 0 13 13 Semaphore # 3 00 0 6 6 Semaphore # 4 00 0 21 21 Semaphore # 5 00 0 10 10 Semaphore # 6 00 0 12 12 Semaphore # 7 00 0 12 12 Semaphore # 8 00 0 43 43 Semaphore # 9 00 0 7 7 Semaphore # 10 00 0 9 9 Semaphore # 11 00 0 11 11 Semaphore # 12 00 0 10 10 Semaphore # 13 00 0 11 11 Semaphore # 14 00 0 16 16 Semaphore # 15 00 0 10 10 Semaphore # 16 00 0 11 11 Semaphore # 17 00 0 17 17 Semaphore # 18 00 0 12 12 Semaphore # 19 00 0 19 19 Semaphore # 20 00 0 5 5 Semaphore # 21 00 0 22 22 Semaphore # 22 00 0 8 8 Semaphore # 23 00 0 34 34 Semaphore # 24 00 0 5 5 Semaphore # 25 00 0 10 10 Semaphore # 26 00 0 11 11 Semaphore # 27 00 0 15 15 Semaphore # 28 00 0 21 21 Semaphore # 29 00 0 12 12 Semaphore # 30 00 0 41 41 Semaphore # 31 00 0 7 7 Semaphore # 32 00 0 49 49 Semaphore # 33 00 0 9 9 Semaphore # 34 00 0 25 25 Semaphore # 35 00 0 13 13 Semaphore # 36 00 0 10 10 Semaphore # 37 00 0 6 6 Semaphore # 38 00 0 11 11 Semaph
Re: Performance
Another thing you could do is truss -p pid for a typical process to see every system call it makes. Capture the output, and see what's up in detail. Note especially where it hangs and waits for a resource while you're watching it on screen. Don't worry so much about the stuff that scrolls by fast (unless it's stuck in a loop, for example); when it stops scrolling might be an indication of a bottleneck. (also with exceptions, like waiting for input). Used with vlist, you can see what line of BASIC code is getting executed, for when you get into the repair phase. "Our greatest duty in this life is to help others. And please, if you can't help them, could you at least not hurt them?" - H.H. the Dalai Lama "When buying & selling are controlled by legislation, the first thing to be bought & sold are the legislators" - P.J. O'Rourke Dan Fitzgerald From: [EMAIL PROTECTED] Reply-To: U2 Users Discussion List <[EMAIL PROTECTED]> To: U2 Users Discussion List <[EMAIL PROTECTED]> Subject: Re: Performance Date: Thu, 8 Apr 2004 13:30:58 -0400 If you are running Topas to monitor the system performance, only have one instance of it open and have it update once every 5 seconds. I have heard of situations where 5 or 6 people would have topas running at the same time effectively bogging the server down. Topas can hog resources if not used in moderation. As was mentioned, nmon is an excellent replacement for topas. Set it's refresh/update setting at 4 seconds instead of the default of 2 seconds. Be careful if you are running performance gathering tools, they are a huge performance hit. filemon is extremely resource hungry. Only run it occasionally, not continiously. -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users _ MSN Toolbar provides one-click access to Hotmail from any Web page FREE download! http://toolbar.msn.com/go/onm00200413ave/direct/01/ -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
Re: Performance
All standard tools will significantly contribute to the problem, via overhead they inherently introduce into the scenario. DPMonitor has the lowest overhead - often measured at significantly less than 1% of system resources. - Original Message - From: <[EMAIL PROTECTED]> To: "U2 Users Discussion List" <[EMAIL PROTECTED]> Sent: Thursday, April 08, 2004 1:30 PM Subject: Re: Performance > > > > > If you are running Topas to monitor the system performance, only have one > instance of it open and have it update once every 5 seconds. I have heard of > situations where 5 or 6 people would have topas running at the same time > effectively bogging the server down. Topas can hog resources if not used in > moderation. > > As was mentioned, nmon is an excellent replacement for topas. Set it's > refresh/update setting at 4 seconds instead of the default of 2 seconds. > > Be careful if you are running performance gathering tools, they are a huge > performance hit. filemon is extremely resource hungry. Only run it > occasionally, not continiously. > > > > > -- > u2-users mailing list > [EMAIL PROTECTED] > http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
Re: Performance
Hello Kevin, I have seen many good posts in reply to your situation already. File-sizing, (and therefore disk IO) is a key/critical area. What kind of file systems do you have? How much memory & swap space do you have? What are the Virtual Memory AIX tuning parameters set to? IBM Hardware - AIX support and IBM U2 support - are all the same company, and they can't find it? Please give us the system configuration information so we can all develop a more clear picture of what you're running there. Is this system a recent OS upgrade from AIX 4.X? Any new or different hardare added or subtracted? Any other changes that may be noteworthy? The way you discuss memory, page faulting, and very high disk IO, I would make sure they verify each of your uvconfig parameters, and kernel system tunable parameters, and make sure you have more than ample swap space, and a large /tmp mounted file system space with fast striped disk sub-system underneath. One tool that will help map out exactly what is going on, and therefore provide a road map on how to address/resolve these issues, and then prove that these issues are indeed resolved, would be the DPMonitor. DPMonitor is available on the internet and has a free 10 day evaluation license available that will allow you to track system-wide parameters and performance metrics that will provide a very clear picture as to what is happening. Check it out at www.deltek.us. This tool has been used on AIX 5.1, on small single processor configurations, up through very large systems, 16 & 32 processor systems. Performance Agent runs on the AIX Application Server. Extremely low overhead Agent. Performance Explorer runs on a Windows Workstation. Well worth the free 10 day evaluation license. Regards, Scott Sr. Systems Engineer / Consultant Marlborough, MA - Original Message - From: "Kevin Vezertzis" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Thursday, April 08, 2004 12:08 PM Subject: Performance > We are looking for some insight from anyone that has experienced > performance degradation in UV, as it relates to the OS. We are running > UV 10.0.14 on AIX 5.1.we are having terrible 'latency' within the > application. This is a recent conversion from D3 to UV and our client > is extremely disappointed with the performance. We've had IBM hardware > support and Universe support in on the box, but to no avail..we are > seeing high paging faults and very highly utilized disk space. Any > thoughts or suggestions? > > Thanks, > Kevin > > > > Kevin D. Vezertzis > Project Manager > Cypress Business Solutions, LLC. > 678.494.9353 ext. 6576 Fax 678.494.9354 > > [EMAIL PROTECTED] > Visit us at www.cypressesolutions.com > > > > -- > u2-users mailing list > [EMAIL PROTECTED] > http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
Re: Performance
If you are running Topas to monitor the system performance, only have one instance of it open and have it update once every 5 seconds. I have heard of situations where 5 or 6 people would have topas running at the same time effectively bogging the server down. Topas can hog resources if not used in moderation. As was mentioned, nmon is an excellent replacement for topas. Set it's refresh/update setting at 4 seconds instead of the default of 2 seconds. Be careful if you are running performance gathering tools, they are a huge performance hit. filemon is extremely resource hungry. Only run it occasionally, not continiously. -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
The first thing I would look for are file sizing issues. Try to identify some of the critical and most used data files; don't forget code files, cross references and the like. For a quick check run GROUP.STAT against the files. Look for these conditions: (1) modulo dramatically too small, (2) bad record distribution, (3) non prime number modulos. For code files and other shared files, check for modulos of 1 or very small values possibly causing lock bottlenecks. Here are a couple small GROUP.STAT samples illustrating the three conditions mentioned: Too small modulo: (numbers in left column much larger than sep x 512) > GROUP.STAT TTT Type description= Hashed, entire keys are full range of ASCII characters. Bytes Records File= TTT Modulo= 7 Sep= 1 Type= 18 3944 103 + 4064 109 + 4356 123 + 4560 110 + 3844 106 + 4144 118 + 3904 115 + == 28816 784 Totals 4116 112 Averages per group 241 6 Standard deviation from average 5.9 5.4 Percent std dev from average Bad record distribution: (This one isn't too bad, but look for a few huge groups with lots of records amongst many small, sparse groups) Type description= Hashed, keys begin in full range of ASCII characters. Bytes Records File= TTT Modulo= 53 Sep= 1 Type= 9 51216 98828 92 3 >>> 32012 156027 >>> 77219 >>> 148 5 > 49614 >> 212049 > 45213 > 65217 > 260 9 > 42812 48814 >> 78023 >>> 220 8 Non-prime modulos: (Won't always be a problem, but can be, as shown here) Type description= Hashed, keys are letters. Bytes Records File= UNPOSTED.TRANSACTIONS Modulo= 20 Sep= 4 Type= 12 0 0 15008 108 + 0 0 0 0 0 0 1328488 + 0 0 0 0 0 0 1360897 + 0 0 0 0 0 0 13368 102 + 0 0 0 0 0 0 15428 105 + 0 0 0 0 == Hope this helps. You can find a number of tech papers about file sizing on our website at www.fitzlong.com. Jeff Fitzgerald Fitgerald & Long, Inc. mailto:[EMAIL PROTECTED] (303) 755-1102 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Vezertzis Sent: Thursday, April 08, 2004 9:02 AM To: [EMAIL PROTECTED] Subject: Performance We are looking for some insight from anyone that has experienced performance degradation in UV, as it relates to the OS. We are running UV 10.0.14 on AIX 5.1.we are having terrible 'latency' within the application. This is a recent conversion from D3 to UV and our client is extremely disappointed with the performance. We've had IBM hardware support and Universe support in on the box, but to no avail..we are seeing high paging faults and very highly utilized disk space. Any thoughts or suggestions? Thanks, Kevin Kevin D. Vezertzis Project Manager Cypress Business Solutions, LLC. 678.494.9353 ext. 6576 Fax 678.494.9354 [EMAIL PROTECTED] Visit us at www.cypressesolutions.com -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
Screen savers? Best performance to background processes? On AIX? -Original Message- From: Brian Leach [mailto:[EMAIL PROTECTED] Sent: Thursday, April 08, 2004 9:24 AM To: 'U2 Users Discussion List' Subject: RE: Performance First things, 1. turn off any screen savers 2. ensure your server is set to adjusted to give best performance to background processes 3. turn off any virus checkers 4. turn off veritas backup Brian Leach -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
Kevin, We had the same problem when migrating UV from DEC in one data center to HP managed in another data center, even though it was a honking big HP. The sys admins configured memory memory similar to how their 20 other unix machines were set up. The trick was to configure most memory for shared data, so that the large UV files used most of the day pretty much stay in memory all the time. I have no personal experience managing unix memory, but these guys did, for Informix, Oracle, etc, and they see UV as acting very differently (well, "weird" was their exact word) from all other DBMSs they have managed. Once the memory allocation was resolved, the new system screamed as expected. Give it a try, Chuck Stevenson > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Vezertzis > Sent: Thursday, April 08, 2004 9:08 AM > To: [EMAIL PROTECTED] > Subject: Performance > > > We are looking for some insight from anyone that has > experienced performance degradation in UV, as it relates to > the OS. We are running UV 10.0.14 on AIX 5.1.we are having > terrible 'latency' within the application. This is a recent > conversion from D3 to UV and our client is extremely > disappointed with the performance. We've had IBM hardware > support and Universe support in on the box, but to no > avail..we are seeing high paging faults and very highly > utilized disk space. Any thoughts or suggestions? > > Thanks, > Kevin > > > > Kevin D. Vezertzis > Project Manager > Cypress Business Solutions, LLC. > 678.494.9353 ext. 6576 Fax 678.494.9354 > > [EMAIL PROTECTED] > Visit us at www.cypressesolutions.com > > > > -- > u2-users mailing list > [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users > -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
Duh duh duh Please ignore my last post. I scanned it and missed 'AIX'. It's the end of the day here Brian -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Vezertzis Sent: 08 April 2004 17:08 To: [EMAIL PROTECTED] Subject: Performance We are looking for some insight from anyone that has experienced performance degradation in UV, as it relates to the OS. We are running UV 10.0.14 on AIX 5.1.we are having terrible 'latency' within the application. This is a recent conversion from D3 to UV and our client is extremely disappointed with the performance. We've had IBM hardware support and Universe support in on the box, but to no avail..we are seeing high paging faults and very highly utilized disk space. Any thoughts or suggestions? Thanks, Kevin Kevin D. Vezertzis Project Manager Cypress Business Solutions, LLC. 678.494.9353 ext. 6576 Fax 678.494.9354 [EMAIL PROTECTED] Visit us at www.cypressesolutions.com -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users This email was checked by MessageLabs SkyScan before entering Microgen. This email was checked on leaving Microgen for viruses, similar malicious code and inappropriate content by MessageLabs SkyScan. DISCLAIMER This email and any attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information. In the event of any technical difficulty with this email, please contact the sender or [EMAIL PROTECTED] Microgen Information Management Solutions http://www.microgen.co.uk -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
Re: Performance
There are many areas that could give you problems, file sizing, memory, swap space, etc. I would suggest using some performance monitoring tools to pin down where the performance issues lie and go from there. IBM AIX has a third party tool "nmon" available that has tools available to import the data into Excell and analyze it. I would start with that and then address the systems issues that it highlights as being bottlenecks. Other than that, right off the bat I would say it's probably a mixture of memory, swap space and file sizing which are some of the more common causes of the problems you describe. Bob Gerrish - [EMAIL PROTECTED] Kingsgate Enterprises, Inc. At 09:08 AM 4/8/2004, you wrote: We are looking for some insight from anyone that has experienced performance degradation in UV, as it relates to the OS. We are running UV 10.0.14 on AIX 5.1.we are having terrible 'latency' within the application. This is a recent conversion from D3 to UV and our client is extremely disappointed with the performance. We've had IBM hardware support and Universe support in on the box, but to no avail..we are seeing high paging faults and very highly utilized disk space. Any thoughts or suggestions? Thanks, Kevin Kevin D. Vezertzis Project Manager Cypress Business Solutions, LLC. 678.494.9353 ext. 6576 Fax 678.494.9354 [EMAIL PROTECTED] Visit us at www.cypressesolutions.com -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
First things, 1. turn off any screen savers 2. ensure your server is set to adjusted to give best performance to background processes 3. turn off any virus checkers 4. turn off veritas backup Brian Leach -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Vezertzis Sent: 08 April 2004 17:08 To: [EMAIL PROTECTED] Subject: Performance We are looking for some insight from anyone that has experienced performance degradation in UV, as it relates to the OS. We are running UV 10.0.14 on AIX 5.1.we are having terrible 'latency' within the application. This is a recent conversion from D3 to UV and our client is extremely disappointed with the performance. We've had IBM hardware support and Universe support in on the box, but to no avail..we are seeing high paging faults and very highly utilized disk space. Any thoughts or suggestions? Thanks, Kevin Kevin D. Vezertzis Project Manager Cypress Business Solutions, LLC. 678.494.9353 ext. 6576 Fax 678.494.9354 [EMAIL PROTECTED] Visit us at www.cypressesolutions.com -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users This email was checked by MessageLabs SkyScan before entering Microgen. This email was checked on leaving Microgen for viruses, similar malicious code and inappropriate content by MessageLabs SkyScan. DISCLAIMER This email and any attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information. In the event of any technical difficulty with this email, please contact the sender or [EMAIL PROTECTED] Microgen Information Management Solutions http://www.microgen.co.uk -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance
Kevin Please post output from: "smat -s" (executable in $UVBIN), "LIST.READU EVERY", "PORT.STATUS" and your uvconfig file Will take a look. Lease collect the stats when the system is under load. Regards JayJay -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Vezertzis Sent: 08 April 2004 17:08 To: [EMAIL PROTECTED] Subject: Performance We are looking for some insight from anyone that has experienced performance degradation in UV, as it relates to the OS. We are running UV 10.0.14 on AIX 5.1.we are having terrible 'latency' within the application. This is a recent conversion from D3 to UV and our client is extremely disappointed with the performance. We've had IBM hardware support and Universe support in on the box, but to no avail..we are seeing high paging faults and very highly utilized disk space. Any thoughts or suggestions? Thanks, Kevin Kevin D. Vezertzis Project Manager Cypress Business Solutions, LLC. 678.494.9353 ext. 6576 Fax 678.494.9354 [EMAIL PROTECTED] Visit us at www.cypressesolutions.com -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
RE: Performance Discussion - Unidata
If I perform tests via the system using both dd and mkfile, I see speeds of around 50MB/s for WRITES, 60MB/s for READS, however if a colleague loads a 100MB csv file using READSEQ into a Unidata file, not doing anything fancy, I see massive Average Service Times (asvc_t - using IOSTAT) and the device is usually always 100% busy, no real CPU overhead but with 15MB/s tops WRITE. There is only ONE person using this system (to test throughput). I don't claim to be an expert in performance monitoring so I'm probably setting myself up for a big fall but... Are you comparing like with like ? dd is simply throwing data out sequentially, Unidata is hashing the key for each record ( how many in 100 MB ?) and writing it to a possibly/ probably different area of the disk. What happens if your colleague loads 100MB using READSEQ and throws the same 100MB back out into a different csv file using WRITESEQ ? What sort of Unidata file is he writing to ? Static or Dynamic ? and what does it look like at the end ? Does your colleague's program first read to see whether the record exists ? is there any correlation between the order of records in the csv file and the groups they end up in the Unidata file ? Just a few thoughts Piers -- u2-users mailing list [EMAIL PROTECTED] http://www.oliver.com/mailman/listinfo/u2-users
