RE: [Fwd: UNIX Performance Issues]

[James Morle]

Title: Message

Hi Rahul.   Interesting, as ever! See below   James -- James Morle Scale Abilities, Ltd http://www.scaleabilities.co.uk Author of "Scaling Oracle8i - Building Highly Scalable OLTP System Architectures" -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Rahul Dandekar Sent: 19 February 2002 15:49 To: Multiple recipients of list ORACLE-L Subject: Re: [Fwd: UNIX Performance Issues] James,   Getting interesting, isn't it? I have added my response...   ----- Original Message ----- From: James Morle To: Multiple recipients of list ORACLE-L Sent: Tuesday, February 19, 2002 8:58 AM Subject: RE: [Fwd: UNIX Performance Issues] Rahul,   Here's what I would do. 1) I would use "mpstat" for the processor statistics. This breaks the usage up by processor in SMP configurations. This can be useful to see the relative loading of each CPU, in particular the breakdown of kernel and user time. 2) Memory: Concentrate on Page Outs and Free Memory more than anything else. That will give you plenty of clues about memory starvation, and the relevence of your VM tuning. 3) I/O: User "sar -d". It's a bit annoying on a system with a lot of disks, because it returns a row for every device, even if no I/O occurred in the sample period. However, it makes it easier to parse. ;-) Notably, keep an eye on the Service Times (avserv?), Wait times (avwait), and the queue depth. The utilisation is a function of these (queuing theory), but you can store that too as a shortcut. You can give sar any sample period, so your 5 minute averages are no problem. How can I get the current I/O load on the system? I don't know exactly what metric I am looking for. But I want to establish some baseline metric for each machine and then hunt for spikes from the gathered data. The metric can be "I/O load on system bus in Mb/sec" (like the netstat info packets input and output). I don't want individual disk statistics. I just want a overall number, which I can snapshot.    I know what you're after, but it's just not going to work that way! A network adapter is a single serial resource with a finite limit. An I/O subsystem is an arbitrarily complex *set* of resources with a finite capacity on each! For example, if you were to just measure the aggregate I/O rate across your SAN (or whatever), that may well return a good number. However, one disk in there could be assuming 50% or more of the load due to hotspots. This disk would probably be providing multi-SECOND response time, and because it's the hot disk, will be slowing nearly everything down. Your aggregate stats would not show this. You need per-disk, per-controller, and if you've got a very busy system you might want to start worrying about backplane capacity. There's no easy way to measure that one, however. 4) Network: "netstat 5" will report a row for every 5 seconds (for example), showing how many packets went in and out of each interface. Your question below is easily answered - you have two columns in your output; the first is for the named interface (hme0), the 100baseT network card. The second is a total of all cards - looks like you only have one. This total can also include the loopback interface (lo0), so look out for that. If I have only one card then why the total and hme0 data are different (by about 10%)?    I suspect it is reporting the lo0 interface in the total, but not showing it individually. Check out the options for netstat (I don't have Slowlaris in front of me right now). Good luck, you're doing the right thing. I've been working on some software to do just this for a couple of years. I'd love to hear how it goes! +Rahul Regards   James -- James Morle Scale Abilities, Ltd http://www.scaleabilities.co.uk Author of "Scaling Oracle8i - Building Highly Scalable OLTP System Architectures" -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Rahul Dandekar Sent: 19 February 2002 12:59 To: Multiple recipients of list ORACLE-L Subject: Re: [Fwd: UNIX Performance Issues] James,   Interleaved, please find my reply....   +Rahul ----- Original Message ----- From: James Morle To: Multiple recipients of list ORACLE-L Sent: Tuesday, February 19, 2002 6:03 AM Subject: RE: [Fwd: UNIX Performance Issues] Rahul,   Did you get a response on this? I'm not sure I fully understand the actual question - are you looking for specific commands you need to run to get the information, [Rahul] Yes. I would like to know which flags of the commonly used commands give good information. For general System stats, I use "sar -u" (same as default), for Memory / Virtual Memory I use "vmstat" and look for "r   b   w   swap   free   pi   po   us   sy   id" columns. I am looking for general monitoring. And once we have this general information giving a overall picture, we could know if there is a problem and we could investigate further. I am specifically looking for IO and Network statistics. Is there any command which would give me approx IO of the system, say in last 5 minutes or current? How to get network statistics? I was littlebit confused with netstat. There are two main categories in my output : hme0 and Total. What does that mean?     input   hme0      output           input  (Total)    output packets errs  packets errs  colls  packets errs  packets errs  colls 5757291 0     2447690 0     0      6071152 0     2761551 0     0 45      0     1       0     0      45      0     1       0     0 24      0     2       0     0      24      0     2       0     0   What I plan to do is to take snapshot of all these statistics at a certain frequency and put it in database. Later on I could generate reports based on this. Currently, I have a lot of "Camera"s like this taking snapshots of my system. Others involve Oracle stuff like DB Size Growth, Performance Ratios, UNIX File System usage, Replication Statistics, Growth of DB objects, a lot of monitors for application info (e.g. total # of clients, # of invoices generated per day). I generate trends based on this archival data for capacity planning and proactively anticipating chronic problems. or advice on how to interpret it? Don't forget that you will really need to correlate many of these statistics to the Oracle pathology at the same time. You said it! I want co-relation of Application Load, UNIX System Load and Database Statistics. And not just when the problem arises. So, that's what I am trying to develop.   This then causes a problem because your sample points will at the very least experience clock drift and become harder to compare over time. There are ways to solve it, though. Anyway, if you could elaborate a little, I can try to assist! Regards   James -- James Morle Scale Abilities, Ltd http://www.scaleabilities.co.uk Author of "Scaling Oracle8i - Building Highly Scalable OLTP System Architectures" -----Original Message----- From: Mogens N�rgaard [mailto:[EMAIL PROTECTED]] Sent: 18 February 2002 22:11 To: James Morle Subject: [Fwd: UNIX Performance Issues] Hi James, I've got no idea whether this is of interest or not to you, but you probably know a bit about this topic. Mogens -------- Original Message -------- Subject: UNIX Performance Issues Date: Thu, 14 Feb 2002 07:43:26 -0800 From: "Rahul Dandekar" <[EMAIL PROTECTED]> Reply-To: [EMAIL PROTECTED] Organization: Fat City Network Services, San Diego, California To: Multiple recipients of list ORACLE-L <[EMAIL PROTECTED]> DBAs, This might be littlebit (or completely!) UNIX related... But I am told to do the performance analysis of some 10-15 machines and generate some statistical data to find out bottlenecks and identify areas of tuning... Operating System : Solaris 2.6 I have been using sar, iostat, top... I actually plan to script these things and run these scripts at certain intervals and put the data in database (Oracle 8i) and then do the crunching... Inputs are appreciated... 1. I/O What is current I/O status. Is there a lot of I/O going on? 2. Paging Is there lot of swapping / paging happening? Which processes are getting swapped in/out continuously? Are the I/O waits due to swapping / paging or regular stuff like DB waiting to read from DB files? 3. CPU What is the CPU utulization? Which processes are using lot of CPU? 4. Memory What is the current picture of Real and Virtual Memory? What processes are using how much memory? Which processes are i n real memory and which are in virtual memory? Which processes are swapped in and out from/to real/virtual memory and how many times? 5. Network What is the percentage utilization of network pipe? What is the capacity (bandwidth) of the network device? What percentage of that bandwidth is getting used? Is the system waiting for data from outside network I/O? In short, is there any bandwidth problem with network device or network traffic. Thanks, ___ ___ ___ ___ ___ / /\ / /\ / /\ / /\ / /\ / /::\ / /::\ / /:/ / /:/ / /:/ / /:/\:\ / /:/:| / /:/ / /:/ / /:/ / /::\ \:\ / /:/|:| / /::\ __ / /:/ ___ / /:/ /__/:/\:\_\:\ /__/::\|:| /__/:/\:\/ /\ /__/:/ / /\ /__/:/ \__\/~|::\/:/ \__\/\:\:| \__\/ \:\/:/ \ \:\ / /:/ \ \:\ | |:|::/ \__\::| \__\::/ \ \:\ /:/ \ \:\ | |:|\/ | |:| / /:/ \ \:\/:/ \ \:\ |__|:| |__|:| /__/:/ \ \::/ \ \:\ \__\| \__\| \__\/ \__\/ \__\/ -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Rahul Dandekar INET: [EMAIL PROTECTED] Fat City Network Services -- (858) 538-5051 FAX: (858) 538-5051 San Diego, California -- Public Internet access / Mailing Lists -------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: [EMAIL PROTECTED] (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).