Dear Khan,
as Vladimir said, "System CPU" is spent in the kernel on I/O, Interrupts,
memory management. Just out of curiosity: what Linux are you (is your
customer) running, which kernel version and what is the uptime?
I ask, because I recently was facing a similar issue on Servers running
SLES11/SP2 (kernel 3.0.58-0.6.2-default). Those were used for Tomcat (Java)
processes, not HPC. They started to really max out all CPUs 100% with 75%
solid "red". But that happened only after some days of uptime
It turned out that in our situation turning of the half-baked (at least in
that kernel) "Transparent Huge Pages" feature off (or to voluntary mode)
solved the problem:
# echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
# echo madvise > /sys/kernel/mm/transparent_hugepage/defrag
# cat /sys/kernel/mm/transparent_hugepage/{enabled,defrag}
always [madvise] never
always [madvise] never
Doing that is pretty much without risk and can be done/reverted at any
time. It may cost a bit of performance in systems with lots of memory, but
I personally think it is overrated for general usage.
As I said, not sure it applies to your situation, but comes from a real
world high throughput environment.
Cheers
Martin
On Tue, Oct 13, 2015 at 7:49 PM, Kamran Khan <kam...@pssclabs.com> wrote:
> Hi All,
>
>
> This isn't a problem with Ganglia, but I was hoping I might get a little
> advice on what I am seeing. I have a customer who is running ls-dyna
> applications, and he is noticing something odd. He is noticing his jobs
> being bogged down and not running at their full capacity. He looked at the
> Ganglia web interface and saw that "System CPU" was at 100%, while "User
> CPU" was at like 20%. What processes does the "System CPU" refer to? What
> tools can I use to track what might be pushing the "System CPU" to 100%?
> There are times when the "User CPU" goes up to 100%, which is what he
> wants, but then at times it spikes down to 20% ish and the "System CPU"
> stays up around 100%.
>
>
> Any advice is greatly appreciated. If you need me to send output, I
> certainly can. Just let me know what to run.
>
>
> Please let me know.
>
>
> Thanks.
> --
> Kamran Khan
> PSSC Labs
> HPC Software / Technical Engineer
>
>
> ------------------------------------------------------------------------------
>
> _______________________________________________
> Ganglia-general mailing list
> Ganglia-general@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ganglia-general
>
>
--
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de
------------------------------------------------------------------------------
_______________________________________________
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general
