RE: [U2] UniData performance problems
Thanks for all the input and for the confirmation that overflow 2 generates more I/O. We tend to see normal or decreased I/O during slow periods, so maybe sizing isn't the root cause. I'll chew on things some more. Maybe I'll even break down and call Tim! Brad Schrag InfoLease Development 651-205-3074 RiverBank / EP-MN-BGF -- Electronic Privacy Notice. This e-mail, and any attachments, contains information that is, or may be, covered by electronic communications privacy laws, and is also confidential and proprietary in nature. If you are not the intended recipient, please be advised that you are legally prohibited from retaining, using, copying, distributing, or otherwise disclosing this information in any manner. Instead, please reply to the sender that you have received this communication in error, and then immediately delete it. Thank you in advance for your cooperation. == --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] UniData performance problems
Brad, Others will undoubtedly talk about file sizing issues. We had a problem where we had some jobs locking up to 30,000 records where the system was geared to handle, maybe, scores or hundreds of locks. Jobs that should have taken a minute, or less, were taking hours. I wrote a program to count the lines in a captured LIST.READU so that I would know how bad the problem was. In this case we had upgraded to a newer version of vendor supplied software which had a new locking strategy in inventory that affected job costing. This ran into a non-standard trick where we used inventory to track certain costs. We did make some helpful system changes to accommodate the larger number of locks but we solved the greatest part of the problem by making procedural changes. Karjala -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Wednesday, June 07, 2006 4:07 PM To: u2-users@listserver.u2ug.org Subject: [U2] UniData performance problems We have had ongoing intermittent performance issues with InfoLease for many months now. Recently the problems have become worse, especially at monthend. Users and customers are complaining and we're out of ideas. I'm hoping someone can point us in a direction that will get us to the root cause. I won't go into all the details, but here is some information on what we're seeing and what we've done. --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] UniData performance problems
[EMAIL PROTECTED] wrote on 06/07/2006 04:07:09 PM: * Resizing seems to have pretty much resolved our batch processing but hasn't helped online activity. Udtmon still shows a surprising amount over Overflow 2 activity. We're investigating, so maybe we have more work to do there. Does anyone know what a system in overflow 2 looks like to the O/S? Maybe a higher number of reads? It seems as if there is a bottleneck that noone can identify. Aside from periodic AIX read/write waits nothing shows any problems. Level 2 overflow will most likely manifest itself in terms of additional I/O requests at the O/S level. Look at the heavy hitters list in filemon. If there's a file at the top of the list, and the number of reads seems high, check sizing on that file. You say that batch processing is OK but interactive users are experiencing performance problems. Is this constant or intermittent? I wouldn't be surprised to hear that some users experience long delays while others fly through. Does the wait queue (see sar -q or topas) show high values? I would look at random write-behind and maybe high- and low-water marks for possible relief. There could be a complex combination of events - at the database and/or AIX levels - converging to create these problems. Shameless plug alert! You may want to have somebody perform a detailed health check of your system. I may just know of somebody who could help you out with this. ;-) Tim Snyder Consulting I/T Specialist , U2 Professional Services North American Lab Services DB2 Information Management, IBM Software Group 717-545-6403 [EMAIL PROTECTED] --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] UniData performance problems
What does sar say about your wait i/o? On Solaris, I use sar -u. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of [EMAIL PROTECTED] Sent: Wednesday, June 07, 2006 3:07 PM To: u2-users@listserver.u2ug.org Subject: [U2] UniData performance problems We have had ongoing intermittent performance issues with InfoLease for many months now. Recently the problems have become worse, especially at monthend. Users and customers are complaining and we're out of ideas. I'm hoping someone can point us in a direction that will get us to the root cause. I won't go into all the details, but here is some information on what we're seeing and what we've done. Application: InfoLease O/S: AIX 5.2.0.0 DB version: UniData 6.0.8 Disk: EMC HW: IBM P670 with two LPARs. One for production, the second cloned nightly for reporting. * Both online and batch processing/reporting have taken a dive at various times. Isssues used to be primarily a day or two before and after monthend, but we're creeping farther into the month as time goes on. * Slowness only affects InfoLease activity. Shell activity seems unaffected. * EMC (disk) tells us we are nowhere near taxing their system and read/write response times are in the 5-8 ms range * At the same time EMC says we're fine, AIX (filemon) indicate significant read/write waits * Nmon regularly indicates a high number of CPU waits, but I've never been able to map that back to user experience * When we experience slowdowns we may/may not be able to pinpoint a process or group of processes causing the problem. * Resizing seems to have pretty much resolved our batch processing but hasn't helped online activity. Udtmon still shows a surprising amount over Overflow 2 activity. We're investigating, so maybe we have more work to do there. Does anyone know what a system in overflow 2 looks like to the O/S? Maybe a higher number of reads? It seems as if there is a bottleneck that noone can identify. Aside from periodic AIX read/write waits nothing shows any problems. TIA Brad Schrag InfoLease Development -- Electronic Privacy Notice. This e-mail, and any attachments, contains information that is, or may be, covered by electronic communications privacy laws, and is also confidential and proprietary in nature. If you are not the intended recipient, please be advised that you are legally prohibited from retaining, using, copying, distributing, or otherwise disclosing this information in any manner. Instead, please reply to the sender that you have received this communication in error, and then immediately delete it. Thank you in advance for your cooperation. == --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] UniData performance problems
A few things come to mind: 1. Is the slowdown for the interactive users a general slowdown thought the business day, or only at specific points in time? 2. Do you experience any udt sessions that are run aways (that is, zombie sessions?) 3. How many users? What kind of locking activity? 4. What is the I/O activity that you are seeing across the adapters (through nmon)? These are items that indicate a need for an overall system assessment ... the issue could well be related to the environment ... and it could be related to the work load and timing of events throughout the day (most likely, a combination of both). You may want to consider an outside assessment of your overall system performance, both from the AIX and from the UniData on AIX perspectives. --- Regards, Robert -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Wednesday, June 07, 2006 15:07 To: u2-users@listserver.u2ug.org Subject: [U2] UniData performance problems We have had ongoing intermittent performance issues with InfoLease for many months now. Recently the problems have become worse, especially at monthend. Users and customers are complaining and we're out of ideas. I'm hoping someone can point us in a direction that will get us to the root cause. I won't go into all the details, but here is some information on what we're seeing and what we've done. Application: InfoLease O/S: AIX 5.2.0.0 DB version: UniData 6.0.8 Disk: EMC HW: IBM P670 with two LPARs. One for production, the second cloned nightly for reporting. * Both online and batch processing/reporting have taken a dive at various times. Isssues used to be primarily a day or two before and after monthend, but we're creeping farther into the month as time goes on. * Slowness only affects InfoLease activity. Shell activity seems unaffected. * EMC (disk) tells us we are nowhere near taxing their system and read/write response times are in the 5-8 ms range * At the same time EMC says we're fine, AIX (filemon) indicate significant read/write waits * Nmon regularly indicates a high number of CPU waits, but I've never been able to map that back to user experience * When we experience slowdowns we may/may not be able to pinpoint a process or group of processes causing the problem. * Resizing seems to have pretty much resolved our batch processing but hasn't helped online activity. Udtmon still shows a surprising amount over Overflow 2 activity. We're investigating, so maybe we have more work to do there. Does anyone know what a system in overflow 2 looks like to the O/S? Maybe a higher number of reads? It seems as if there is a bottleneck that noone can identify. Aside from periodic AIX read/write waits nothing shows any problems. TIA Brad Schrag InfoLease Development -- Electronic Privacy Notice. This e-mail, and any attachments, contains information that is, or may be, covered by electronic communications privacy laws, and is also confidential and proprietary in nature. If you are not the intended recipient, please be advised that you are legally prohibited from retaining, using, copying, distributing, or otherwise disclosing this information in any manner. Instead, please reply to the sender that you have received this communication in error, and then immediately delete it. Thank you in advance for your cooperation. == --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] UniData performance problems
Yes; you would see an increased number of disk accesses at the os level, both I and O, with significant level 2 overflow. This would also affect locks; if I set a lock, the longer it takes me to find it, process it, and write it back, the longer the lock is held. If it takes me a few dozen reads to get the whole group, that can be a very significant delay. But if a high percentage of the file is even just barely into level 2, that doubles the i/o latency. So you may be on to something there, but if that (resizing) doesn't do it, the variables are legion. I can tell you that I've worked with Tim Snyder in the past, and would not hesitate to call on him again in the future if the need arises. Our greatest duty in this life is to help others. And please, if you can't help them, could you at least not hurt them? - H.H. the Dalai Lama When buying selling are controlled by legislation, the first thing to be bought sold are the legislators - P.J. O'Rourke Dan Fitzgerald -Original Message- From: [EMAIL PROTECTED] [mailto:owner-u2- [EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Wednesday, June 07, 2006 15:07 To: u2-users@listserver.u2ug.org Subject: [U2] UniData performance problems We have had ongoing intermittent performance issues with InfoLease for many months now. Recently the problems have become worse, especially at monthend. Users and customers are complaining and we're out of ideas. I'm hoping someone can point us in a direction that will get us to the root cause. I won't go into all the details, but here is some information on what we're seeing and what we've done. Application: InfoLease O/S: AIX 5.2.0.0 DB version: UniData 6.0.8 Disk: EMC HW: IBM P670 with two LPARs. One for production, the second cloned nightly for reporting. * Both online and batch processing/reporting have taken a dive at various times. Isssues used to be primarily a day or two before and after monthend, but we're creeping farther into the month as time goes on. * Slowness only affects InfoLease activity. Shell activity seems unaffected. * EMC (disk) tells us we are nowhere near taxing their system and read/write response times are in the 5-8 ms range * At the same time EMC says we're fine, AIX (filemon) indicate significant read/write waits * Nmon regularly indicates a high number of CPU waits, but I've never been able to map that back to user experience * When we experience slowdowns we may/may not be able to pinpoint a process or group of processes causing the problem. * Resizing seems to have pretty much resolved our batch processing but hasn't helped online activity. Udtmon still shows a surprising amount over Overflow 2 activity. We're investigating, so maybe we have more work to do there. Does anyone know what a system in overflow 2 looks like to the O/S? Maybe a higher number of reads? It seems as if there is a bottleneck that noone can identify. Aside from periodic AIX read/write waits nothing shows any problems. TIA Brad Schrag InfoLease Development -- Electronic Privacy Notice. This e-mail, and any attachments, contains information that is, or may be, covered by electronic communications privacy laws, and is also confidential and proprietary in nature. If you are not the intended recipient, please be advised that you are legally prohibited from retaining, using, copying, distributing, or otherwise disclosing this information in any manner. Instead, please reply to the sender that you have received this communication in error, and then immediately delete it. Thank you in advance for your cooperation. == --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
