Hey Kevan,
Regarding monitoring...
I managed to run into xenmon.py.
It appears to log the system utilization for the whole box as well as each
VM to log files in 'your' home directory if you specify the '-n' flag.
Here is the help page for xenmon.py:
[EMAIL PROTECTED]:~$ sudo python /usr/sbin/xenmon.py -h
Usage: xenmon.py [options]
Options:
-h, --help show this help message and exit
-l, --live show the ncurses live monitoring frontend (default)
-n, --notlive write to file instead of live monitoring
-p PREFIX, --prefix=PREFIX
prefix to use for output files
-t DURATION, --time=DURATION
stop logging to file after this much time has elapsed
(in seconds). set to 0 to keep logging indefinitely
-i INTERVAL, --interval=INTERVAL
interval for logging (in ms)
--ms_per_sample=MSPERSAMPLE
determines how many ms worth of data goes in a sample
--cpu=CPU specifies which cpu to display data for
--allocated Display allocated time for each domain
--noallocated Don't display allocated time for each domain
--blocked Display blocked time for each domain
--noblocked Don't display blocked time for each domain
--waited Display waiting time for each domain
--nowaited Don't display waiting time for each domain
--excount Display execution count for each domain
--noexcount Don't display execution count for each domain
--iocount Display I/O count for each domain
--noiocount Don't display I/O count for each domain
And here is some sample output:
[EMAIL PROTECTED]:~$ cat log-dom0.log
# passed cpu dom cpu(tot) cpu(%) cpu/ex allocated/ex blocked(tot) blocked(%)
blocked/io waited(tot) waited(%) waited/ex ex/s io(tot) io/ex
0.000 0 0 2.086 0.000 38863.798 30000000.000 154.177 0.000 0.000 0.504 0.000
9383.278 0.000 0.000 0.000
2.750 1 0 2.512 0.000 53804.925 30000000.000 153.217 0.000 0.000 0.316 0.000
6774.813 0.000 0.000 0.000
4.063 2 0 2.625 0.000 59959.942 30000000.000 153.886 0.000 0.000 0.173 0.000
3939.987 0.000 0.000 0.000
5.203 3 0 3.020 0.000 47522.430 30000000.000 171.834 0.000 0.000 0.701 0.000
11031.759 0.000 0.000 0.000
6.403 4 0 2.130 0.000 39256.871 30000000.000 171.870 0.000 0.000 0.617 0.000
11378.014 0.000 0.000 0.000
9.230 6 0 0.836 0.000 53962.875 30000000.000 57.287 0.000 0.000 0.038 0.000
2450.488 0.000 0.000 0.000
10.305 7 0 2.171 0.000 46119.247 30000000.000 154.008 0.000 0.000 0.367 0.000
7804.444 0.000 0.000 0.000
11.518 0 0 15931680.822 1.593 54019.023 30000000.000 889706824.191 88.971 0.000
2630292.436 0.263 8918.446 294.927 0.000 0.000
1009.216 1 0 7687035.544 0.769 53822.548 30000000.000 473101345.004 47.310
0.000 864964.568 0.086 6056.248 142.822 0.000 0.000
1010.199 2 0 20502235.224 2.050 61655.293 30000000.000 979188763.754 97.919
0.000 4279443600.516 427.944 12869345.608 332.530 0.000 0.000
1011.239 3 0 13634865.766 1.363 45934.870 30000000.000 985479796.363 98.548
0.000 1593248.596 0.159 5367.538 296.830 0.000 0.000
1012.312 4 0 18228049.181 1.823 61242.790 30000000.000 979822521.396 97.982
0.000 2593364.560 0.259 8713.213 297.636 0.000 0.000
1013.338 5 0 9891757.872 0.989 65386.046 30000000.000 571275802.794 57.128
0.000 357431.539 0.036 2362.678 151.282 0.000 0.000
We could probably add a cron job to grab a single sample every X minutes
and append them together to build up a utilization history (rather than
simply running it all of the time).
I just tried to get a single sample and the smallest run I could get was
about three seconds with four samples taken.
Or, I also tried xentop in batch mode:
[EMAIL PROTECTED]:~$ sudo xentop -b -i 1
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%)
VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR SSID
Domain-0 -----r 430567 0.0 3939328 23.5 no limit n/a
8 4 0 0 0 0 0 0 2149631536
tck01 --b--- 750449 0.0 3145728 18.8 3145728 18.8
2 1 483054 1855493 1 15 655667 8445829 2149631536
tck02 --b--- 1101273 0.0 3145728 18.8 3145728 18.8
2 1 367792 1773407 1 83 1131709 9030663 2149631536
tck03 -----r 144552 0.0 3145728 18.8 3145728 18.8
2 1 188115 2370069 1 6 370431 1290683 2149631536
tck04 --b--- 103742 0.0 3145728 18.8 3145728 18.8
2 1 286936 2341941 1 7 381523 1484476 2149631536
It looks to me like having a cron job that periodically ran xentop and
build up a history would be the best option (without digging through
a ton of different specialized monitor packages).
Jay
Kevan Miller wrote:
>
> On Oct 10, 2008, at 11:29 AM, Kevan Miller wrote:
>
>>
>> On Oct 10, 2008, at 11:25 AM, Kevan Miller wrote:
>>
>>>
>>> On Oct 8, 2008, at 11:56 PM, Kevan Miller wrote:
>>>
>>>>
>>>> On Oct 8, 2008, at 4:31 PM, Jason Warner wrote:
>>>>
>>>>> We had some suggestions earlier for some alternate means of
>>>>> implementing this (Hudson, Conitnuum, etc...). Now that we've had
>>>>> Jason Dillon provide an overview of what we had in place before,
>>>>> does anyone have thoughts on what we should go with? I'm thinking
>>>>> we should stick with the AHP based solution. It will need to be
>>>>> updated most likely, but it's been tried and tested and shown to
>>>>> meet our needs. I'm wondering, though, why we stopped using it
>>>>> before. Was there a specific issue we're going to have to deal
>>>>> with again?
>>>>
>>>> IIRC, the overwhelming reason we stopped using it before was because
>>>> of hosting issues -- spotty networking, hardware failures, poor colo
>>>> support, etc. We shouldn't have any of these problems, now. If we do
>>>> run into problems, they should now be fixable. I have no reason to
>>>> favor Hudson/Continuum over AHP. So, if we can get AHP running
>>>> easily, I'm all for it. There's only one potential issue, that I'm
>>>> aware of.
>>>>
>>>> We previously had an Open Source License issued for our use of
>>>> Anthill. Here's some of the old discussion --
>>>> http://www.nabble.com/Geronimo-build-automation-status-(longish)-tt7649902.html#a7649902
>>>>
>>>>
>>>> Although the board was aware of our usage of AntHill, since we
>>>> weren't running AntHill on ASF hardware, I'm not sure the license
>>>> was fully vetted by Infra. I don't see any issues, but I'll want to
>>>> run this by Infra.
>>>>
>>>> Jason D, will the existing license cover the version of AntHill that
>>>> we'll want to use? I'll run the license by Infra and will also
>>>> describe the issue for review by the Board, in our quarterly report.
>
> Heh. Oops. Just noticed that I sent the following to myself and not the
> dev list. I hate when I do that...
>
>>
>> One more thing... from emails on [EMAIL PROTECTED] looks like
>> Infra is cool with us running Anthill on selene and phoebe.
>>
>> BTW, am planning on installing monitoring software over the weekend on
>> selene and phoebe. The board is interested in monitoring our usage...
>
>
> Also, we now have a new AntHill license for our use. I've placed the
> license in ~kevan/License2.txt on phoebe and selene. This license should
> only be used for Apache use. So, should not be placed in a public
> location (e.g. our public svn tree).
>
> Regarding monitoring software -- I haven't been able to get it to work
> yet. vmstat/iostat don't work, unless you run on every virtual machine.
> 'xm top' gathers data on all domains, however, doesn't make the data
> easy to tuck away in a log file/available to snmp... Advice welcome...
>
> --kevan
>
