Hi Martin,
I downloaded the source from the Monit website and compiled it on the
server.
I have started monit in verbose mode and this is the relevant information
it outputs when the event occurs.
cpu system usage of 50.0% matches resource limit [cpu system usage>30.0%]
-------------------------------------------------------------------------------
../tools/bin/monit() [0x41a533]
../tools/bin/monit(LogError+0x9f) [0x41ad2f]
../tools/bin/monit(Event_post+0x328) [0x417ba8]
..t/tools/bin/monit() [0x428071]
../tools/bin/monit(check_system+0x2b) [0x4285bb]
../tools/bin/monit(validate+0x226) [0x42ad16]
../tools/bin/monit() [0x41422d]
../tools/bin/monit(main+0x511) [0x4149e1]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x3592c1ecdd]
../tools/bin/monit() [0x40b179]
-------------------------------------------------------------------------------
Unfortunately remote access is not an option but I will happily run a debug
version to try and track down this problem as I really would like to use
Monit for my current build.
Regards
Wayne
On 7 December 2011 11:17, Martin Pala <[email protected]> wrote:
> Thanks for data.
>
> The /proc/stat format is this:
>
> cpu <user> <nice> <system> <idle> <wait> <irq> <softirq>
>
> The values count the cpu cycles, so if we subtract the corresponding
> values from your output, we get this:
>
> user nice system idle wait irq softirq |
> total
> 09:57:35 1 0 1 99 0 0 0
> | 101
> 09:57:36 1 0 0 98 0 0 0
> | 99
> 09:57:37 25 0 16 59 1 0 0
> | 101
> 09:57:38 1 0 2 98 0 0 0
> | 101
>
> => at 09:57:37 the cpu usage was:
>
> user = 24.75%
> system = 15.84%
> wait = 0.99%
>
> This corresponds to the previous vmstat output. Monit counts the cpu usage
> the same way as above and doesn't modify these values => your monit really
> reports strange cpu usage (reported 50% vs. real ~ 16%).
>
> What's the origin of your monit binary? Did you compile it from original
> source code or some 3rd party source code distibution? (such as RHEL or
> Fedora repository). Or do you use the pre-compiled binaries from
> www.mmonit.com? Or some 3rd party binary, patches or source code from
> other site?
>
> Please can you try to run monit in verbose mode and provide full output?:
>
> 1.) stop monit
> 2.) run monit in foreground with verbose mode enabled:
> ./monit -vI
> 3.) after the problem happens, stop monit with "^C" and send output
>
> I can also prepare debug version which will dump the cpu usage related
> informations or if you can provide remote access to the system, i can
> troubleshoot the problem remotely.
>
>
> Regards,
> Martin
>
>
>
> On Dec 7, 2011, at 11:07 AM, Lawrence, Wayne wrote:
>
> Hi Martin,
>
> this is the output of the commands you requested.
>
> 1.) uname -m
>
> x86_64
>
> 2.) file `which monit`
>
> ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically
> linked (uses shared libs), for GNU/Linux 2.6.18, not stripped
> I ran the command you supplied to get the cup usage directly as well while
> restarting the httpd service as i know this will generate an alert.
>
>
>
> Date: Wed, 07 Dec 2011 09:57:37
> Action: exec
> Host: <hostname removed>
> Description: cpu system usage of 50.0% matches resource limit [cpu
> system usage>30.0%]
>
> Wed Dec 7 09:57:34 GMT 2011
> cpu 207060 501 103542 49452254 25303 83 1569 0 0
> Wed Dec 7 09:57:35 GMT 2011
> cpu 207061 501 103543 49452353 25303 83 1569 0 0
> Wed Dec 7 09:57:36 GMT 2011
> cpu 207062 501 103543 49452451 25303 83 1569 0 0
> Wed Dec 7 09:57:37 GMT 2011
> cpu 207087 501 103559 49452510 25304 83 1569 0 0
> Wed Dec 7 09:57:38 GMT 2011
> cpu 207088 501 103561 49452608 25304 83 1569 0 0
> Wed Dec 7 09:57:40 GMT 2011
> If my understanding of /proc/stat is coreect this still doesnt make any
> sense but i may be wrong.
>
> Regards
>
> Wayne
>
>
>
> On 7 December 2011 09:37, Martin Pala <[email protected]> wrote:
>
>> Please can you check that your monit binary matches the system
>> architecture? (i.e. for example 64-bit monit binary on 64-bit system - not
>> 32-bit monit on 64-bit system)
>>
>> To verify provide please the output of following commands:
>> 1.) uname -m
>> 2.) file `which monit`
>>
>> Monit takes the statistics from the /proc/stat kernel interface. You can
>> collect the statistics manually like this - for example to fetch the state
>> in 1 second intervals (30 samples):
>>
>> $ for ((i=0; i<30; i++)); do date; grep "cpu " /proc/stat; sleep 1; done
>>
>> Note: monit takes the first /proc/stat line ("cpu") which contains the
>> overall cpu usage in the system (summary of all cpus). The /proc/stat also
>> contains per-cpu statistics if you want to collect all the statistics,
>> replace the "grep 'cpu '" simply with "cat".
>>
>> Regards,
>> Martin
>>
>>
>> On Dec 7, 2011, at 10:04 AM, Lawrence, Wayne wrote:
>>
>> Hi Martin,
>>
>> I have tried various methods to dientify the cause of this and took your
>> advice and used vmstat. I simply restarted the httpd process from the monit
>> web interface while the comand was running and got the following warning.
>>
>> Description: cpu system usage of 50.0% matches resource limit
>> [cpu system usage>30.0%]
>>
>> But vmstat doesnt show that level of usage at the point of alert. As you
>> can see there is some usage in the 3rd line of the output when i restarted
>> the httpd service but it doesnt seem enough to trigger an alert.
>>
>> vmstat 1 10
>> procs -----------memory---------- ---swap-- -----io---- --system--
>> -----cpu-----
>> r b swpd free buff cache si so bi bo in cs us sy
>> id wa st
>> 0 0 0 859596 114684 856908 0 0 4 6 81 77 0 0
>> 99 0 0
>> 0 0 0 859448 114684 856916 0 0 0 0 100 94 1 0
>> 99 0 0
>> 0 0 0 898352 114692 815600 0 0 0 168 555 605 23 15
>> 61 1 0
>>
>> Not sure if there are any other tests i can run to narrow this down a bit
>> further as it still isn't making sense.
>>
>> Regards
>>
>> Wayne
>>
>>
>>
>>
>>
>> On 7 December 2011 08:27, Martin Pala <[email protected]> wrote:
>>
>>> Hi Lawrence,
>>>
>>> the test which triggers the alert is "system" cpu => it's the time the
>>> system spend in kernel mode. The cpu usage could be triggered by some
>>> background kernel task, to verify the monit report matches the system cpu
>>> usage, you should use either "vmstat" or "top" instead of "ps".
>>>
>>> Best regards,
>>> Martin
>>>
>>>
>>> On Dec 6, 2011, at 1:19 PM, Lawrence, Wayne wrote:
>>>
>>> Hi Igor,
>>>
>>> the operating system is RHEL6 and monit version is 5.3.1
>>>
>>> this is what i have in my config
>>>
>>> if cpu usage (user) > 70% then alert
>>> if cpu usage (system) > 30% then alert
>>> if cpu usage (wait) > 20% then alert
>>>
>>> this is one of the errors
>>> Description: cpu system usage of 50.0% matches resource limit [cpu
>>> system usage>30.0%]
>>>
>>> this is what i get in /var/log/messages
>>> Dec 6 12:01:29 <hostname-removed> monit[864]: <hostname-removed> cpu
>>> system usage of 50.0% matches resource limit [cpu system usage>30.0%]
>>> Dec 6 12:02:29 <hostname-removed> monit[864]:
>>> <hostname-removed><hostname-removed>' cpu system usage check succeeded
>>> [current cpu system usage=0.9%]
>>>
>>> this is the output of ps --no-headers -A -o "%*cpu* sz ucomm" | sort
>>> -k1nr | head -20
>>>
>>> 12:01:29 up 4 days, 20:24, 2 users, load average: 0.04, 0.01, 0.00
>>> total used free shared buffers cached
>>> Mem: 2055108 1092176 962932 0 53156 811864
>>> -/+ buffers/cache: 227156 1827952
>>> Swap: 4128760 0 4128760
>>> 1.2 44308 perl
>>> 0.0 0 aio/0
>>> 0.0 0 async/mgr
>>> 0.0 0 ata/0
>>> 0.0 0 ata_aux
>>> 0.0 0 bdi-default
>>> 0.0 0 cpuset
>>> 0.0 0 crypto/0
>>> 0.0 0 events/0
>>> 0.0 0 ext4-dio-unwrit
>>> 0.0 0 flush-253:0
>>> 0.0 0 jbd2/dm-0-8
>>> 0.0 0 kacpi_hotplug
>>> 0.0 0 kacpi_notify
>>> 0.0 0 kacpid
>>> 0.0 0 kauditd
>>> 0.0 0 kblockd/0
>>> 0.0 0 kdmflush
>>> 0.0 0 khelper
>>> 0.0 0 khubd
>>>
>>> Have to say i am at a total loss as there is no way the usage figures
>>> are accurate.
>>> If there is any other info i can supply that will be useful please let
>>> me know.
>>>
>>> Regards
>>>
>>> Wayne
>>>
>>>
>>> On 6 December 2011 12:03, Igor Homyakov <
>>> [email protected]> wrote:
>>>
>>>> Hi Lawrence,
>>>>
>>>> Could you be a little bit more specific ? Please provide information
>>>> about you operation system, monit version on which the problem
>>>> occurred and so on.
>>>>
>>>> Regards
>>>> Igor Homyakov
>>>>
>>>> On Tue, Dec 6, 2011 at 15:35, Lawrence, Wayne
>>>> <[email protected]> wrote:
>>>> > Hi,
>>>> >
>>>> > I have a few CPU usage checks in my monitrc but it seems monit is
>>>> > misreporting the usage.
>>>> >
>>>> > I have run several tests and it seems that monit is multiplying the
>>>> actual
>>>> > usage by 10.
>>>> >
>>>> > I ran a process with top running in another shell and CPU usage for
>>>> the user
>>>> > was never above 10% yet monit informed me that there was 100% cpu
>>>> usage.
>>>> >
>>>> > I have tried various configurations including the one that came with
>>>> the
>>>> > default config for system cpu monitoring and all seem to demonstrate
>>>> the
>>>> > same issue.
>>>> >
>>>> > Any advice welcomed on this
>>>> >
>>>> > Regards
>>>> >
>>>> > Wayne Lawrence
>>>>
>>>
>
> --
> To unsubscribe:
> https://lists.nongnu.org/mailman/listinfo/monit-general
>
--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general
