Re: [Beowulf] Large Dell, odd IO delays

[Gus Correa]

On 02/15/2018 02:04 AM, John Hearns via Beowulf wrote:

Hmmm...  I will also chip in with my favourite tip
Look at the sysctl for min_free_kbytes    It is often set very low.
Increase this substantially. It will do no harm to your system (unless 
you set it ti an absurd value!)


You should be looking at the vm dirty ratios etc. also


+1
vm.dirty_background_bytes
vm.dirty_bytes
(or the corresponding _ratios)
vm.min_free_kbytes
Defaults are low.
Increasing them improved a lot our compute nodes IO.
https://www.kernel.org/doc/Documentation/sysctl/vm.txt
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-tunables



On 15 February 2018 at 00:44, Kilian Cavalotti 
> wrote:


On Wed, Feb 14, 2018 at 2:26 PM, David Mathog > wrote:
> Checked the hugepage settings and found a difference there.  The two 
systems
> that don't do this have  /sys/kernel/mm/redhat_transparent_hugepage/defrag
>
> always madvise [never]
>
> whereas the system with the issue has:
>
> [always] madvise never

THP defragmentation is definitely something that has bitten us in the
past, when under memory pressure, and we now default to [madvise]
pretty much everywhere (we're too timid to disable it entirely).

A good way to see if that's really the issue is to "echo never >
/sys/kernel/mm/redhat_transparent_hugepage/defrag" while the problem
is happening, while simultaneously monitoring the processes with htop,
for instance.
It's usually pretty instant:  if the issue is really with THP defrag,
then CPU usage for your stalling process should drop pretty much
immediately and things go back to normal.

Cheers,
--
Kilian
___
Beowulf mailing list, Beowulf@beowulf.org
 sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf





___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf



___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Large Dell, odd IO delays

[Michael Di Domenico]

On Wed, Feb 14, 2018 at 6:44 PM, Kilian Cavalotti
 wrote:
> On Wed, Feb 14, 2018 at 2:26 PM, David Mathog  wrote:
>> Checked the hugepage settings and found a difference there.  The two systems
>> that don't do this have  /sys/kernel/mm/redhat_transparent_hugepage/defrag
>>
>> always madvise [never]
>>
>> whereas the system with the issue has:
>>
>> [always] madvise never
>
> THP defragmentation is definitely something that has bitten us in the
> past, when under memory pressure, and we now default to [madvise]
> pretty much everywhere (we're too timid to disable it entirely).

i will second this stance as well.  i've seen huge issues with disk
performance when hugepage was enabled.  i disable it on all the
machines we have now.

the way i found it was when doing large IO with hugepages enabled, the
khugepage (sp?) process shoots right to the top of a top display.  and
the performance you describe was the same.
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Large Dell, odd IO delays

[John Hearns via Beowulf]

Hmmm...  I will also chip in with my favourite tip
Look at the sysctl for min_free_kbytesIt is often set very low.
Increase this substantially. It will do no harm to your system (unless you
set it ti an absurd value!)

You should be looking at the vm dirty ratios etc. also

On 15 February 2018 at 00:44, Kilian Cavalotti <
kilian.cavalotti.w...@gmail.com> wrote:

> On Wed, Feb 14, 2018 at 2:26 PM, David Mathog  wrote:
> > Checked the hugepage settings and found a difference there.  The two
> systems
> > that don't do this have  /sys/kernel/mm/redhat_
> transparent_hugepage/defrag
> >
> > always madvise [never]
> >
> > whereas the system with the issue has:
> >
> > [always] madvise never
>
> THP defragmentation is definitely something that has bitten us in the
> past, when under memory pressure, and we now default to [madvise]
> pretty much everywhere (we're too timid to disable it entirely).
>
> A good way to see if that's really the issue is to "echo never >
> /sys/kernel/mm/redhat_transparent_hugepage/defrag" while the problem
> is happening, while simultaneously monitoring the processes with htop,
> for instance.
> It's usually pretty instant:  if the issue is really with THP defrag,
> then CPU usage for your stalling process should drop pretty much
> immediately and things go back to normal.
>
> Cheers,
> --
> Kilian
> ___
> Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Large Dell, odd IO delays

[Kilian Cavalotti]

On Wed, Feb 14, 2018 at 2:26 PM, David Mathog  wrote:
> Checked the hugepage settings and found a difference there.  The two systems
> that don't do this have  /sys/kernel/mm/redhat_transparent_hugepage/defrag
>
> always madvise [never]
>
> whereas the system with the issue has:
>
> [always] madvise never

THP defragmentation is definitely something that has bitten us in the
past, when under memory pressure, and we now default to [madvise]
pretty much everywhere (we're too timid to disable it entirely).

A good way to see if that's really the issue is to "echo never >
/sys/kernel/mm/redhat_transparent_hugepage/defrag" while the problem
is happening, while simultaneously monitoring the processes with htop,
for instance.
It's usually pretty instant:  if the issue is really with THP defrag,
then CPU usage for your stalling process should drop pretty much
immediately and things go back to normal.

Cheers,
--
Kilian
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf

Re: [Beowulf] Large Dell, odd IO delays

[Christopher Samuel]

On 15/02/18 09:26, David Mathog wrote:

Sometimes for no reason that I can discern an IO operation on this 
machine will stall.  Things that should take seconds will run for 
minutes, or at least until I get tired of waiting and kill them.

Here is today's example:

gunzip -c largeFile.gz > largeFile


Does "perf top -p ${PID}" show anything useful about where the
processes is spending its time?

Good luck!
Chris
___
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit 
http://www.beowulf.org/mailman/listinfo/beowulf