Re: Linux "sleep" command not waking up under high CPU utilization

[Ted Rodriguez-Bell]

Sleep is actually a builtin in ksh, although not in bash (what you probably 
have as /bin/sh) or (don't program in this!) tcsh.  I'd be curious to see if 
changing the first line of your script to "#! /bin/ksh" helps---if you have ksh 
installed.  I second the suggestion of putting "set -x" in the script, and 
adding "time " to the beginning of each line that does anything substantial.

It strikes me that running this with cron might spin up more processes than 
running this as a daemon, but that's also worth trying.  Although you don't 
want to be the person who fills the logging disk by starting every line with "* 
* * * *".  It will cause people to curse many generations of your descendants.

And please:  let us know what you find out!

Ted Rodriguez-Bell
Wells Fargo, Mainframe and Midrange Services
[ Only slightly an opinion of my employer. ]

P.S.  I was looking at the man pages for RHEL 7 and SLES 11; I don't have 
access to older versions.

Company policy requires:  This message may contain confidential and/or 
privileged information.  If you are not the addressee or authorized to receive 
this for the addressee, you must not use, copy, disclose, or take any action 
based on this message or any information herein.  If you have received this 
message in error, please advise the sender immediately by reply e-mail and 
delete this message.  Thank you for your cooperation.


-Original Message-
From: John Campbell <soup...@gmail.com> 
Sent: Wednesday, April 18, 2018 7:10 AM
Subject: Re: Linux "sleep" command not waking up under high CPU utilization

Buy a clue?  That can be expensive depending upon how many vowels are 
necessary.  :-)

Look, the sleep call is awkward to begin with...  and that's within a C 
program!  (Since perl uses sleep() it can work more-or-less the same way.) A 
sleep(1) call will vary, for instance.  Depending upon the system using a
nap(ms) call has its own complications.

A "sleep 1m" (or, the way I code the command, "sleep 60" since the default 
granularity is in seconds) includes a fork()+exec() call and then passing, on 
completion, notifying the parent process it can now proceed.

In a very busy system, the fork()/exec() to LAUNCH the sleep command may be 
delayed. (Personally, when working within a system that is loaded down, I 
prefer to code a perl script, just to cut down on the fork()/exec() rate.) So, 
the need to execute fork() to get a child process launched-- which starts out 
as a copy of the existing shell-- can require resources that are in short 
supply EVEN WHEN the sleep command renders a teeny process, memory-wise, so, 
when the system is loaded down, forking can be impacted (RAM, CPU, etc) while 
the exec() call, itself, starts a process with a very minimal page set and lets 
page faults load the rest of it.  (And, just as a reminder, the fork() call 
means the impure "data" and "stack" segments have to be allocated and copied.)

So the system is loaded down.

If this is a virtual instance, however, you may want to investigate whether the 
instance is being impacted by OTHER instances where resources are 
over-committed.  Other instances can't be seen, howsomever, so finding out if 
some other instance is stealing your cycles (or putting your instance into 
paging space) can be harder to detect.  The usual performance monitoring tools 
don't work "well" when CPU and RAM are overcommitted.

YMMV.

If you are resident in an LPAR you can use internal performance tools: Look at 
the top command's output.  Run vmstat or even sar.  Consider what your system 
is doing.

If in a virtualized instance...
https://www.youtube.com/watch?v=__n5Bgxx-68

Finally, while a process is sleeping, it can be pushed out of RAM and then 
waking up means you'll have a whole buncha page faults as it tries to get 
running again.  A virtual instance can have DOUBLE page faults both within the 
instance which cause a page fault the hypervisor has to  address.

If you are running an "important" set of processes within a virtual instance 
(shakes head).

-soup

On Wed, Apr 18, 2018 at 9:36 AM, Dave Jones <djo...@itconline.com> wrote:

> Hello, gang.
>
> I have a very simple bash script that runs a trivial data collection 
> task, and then does a Linux "sleep im" to wait a minute before running 
> the data collection task again. Under very high CPU loads (> 90%) I 
> have noticed that the "sleep" command does not seem to wake up after 
> one minute but instead wakes up 15 to 20 minutes later. This is on a 
> Red Hat
> 6.9 guest running under z/VM 6.4 on a z12 box.
>
> I would like to buy a clue here if I could.
>
> Thanks.
>
> DJ
>
> --
>
> DAVID JONES | MANAGING DIRECTOR FOR ZSYSTEMS SERVICES | z/VM, Linux, 
> and Cloud
>  703.237.7370 (Office) | 281.578.7544 (CELL)
>

Re: Linux "sleep" command not waking up under high CPU utilization

[John Campbell]

Ah so it was "Test Plan Charlie"?   The "TPC" TLA was once used for "The
Phone Company".

Was it the 100Hz timer that caused most of the thrashing?

-soup

On Thu, Apr 19, 2018 at 1:44 PM, James Tison  wrote:

> And I have an old friend, former IBMer, who remembers Mr. Foxworthy when
> they worked in the same department at Eastern Airlines.
>
> Bonus points? Test Plan Charlie :-)
>
> Peace out.
>
> Jim Tison
> z/TPF Specialist
> Global Technology Services
> jti...@us.ibm.com
>
> IBM Services
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>



--
John R. Campbell Speaker to Machines  souperb at gmail dot
com
MacOS X proved it was easier to make Unix user-friendly than to fix Windows
"It doesn't matter how well-crafted a system is to eliminate errors;
Regardless
 of any and all checks and balances in place, all systems will fail because,
 somewhere, there is meat in the loop." - me

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[James Tison]

And I have an old friend, former IBMer, who remembers Mr. Foxworthy when
they worked in the same department at Eastern Airlines.

Bonus points? Test Plan Charlie :-)

Peace out.

Jim Tison
z/TPF Specialist
Global Technology Services
jti...@us.ibm.com

IBM Services

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[Scott Rohling]

Oh and PBR is a real thing...   I lived up in Nederland and Eldora and
Jamestown --  no flatlands for me -  Boulder was the cosmic center of the
universe for awhile (burp)

Scott Rohling

On Thu, Apr 19, 2018 at 10:20 AM, Scott Rohling 
wrote:

> Don't know - I worked on RETAIN and then HONE - wrote a thing called
> REGISTER that did DIRMAINT/RACF admin via panels - then z/VM tools and
> system support -then started working from home in '96 doing various and
> sundry things around z/VM and zLinux.   I just know the Boulder plant
> started to feel like a haunted house as manufacturing (especially printing)
> moved elsewhere.   Now it's a major DR site AFAIK ...
>
> Well we're very OT here but thanks for reminiscing ;-)
>
> Scott Rohling
>
> On Thu, Apr 19, 2018 at 9:55 AM, John Campbell  wrote:
>
>> Well, I spent a week at the People's Republic of Boulder to work on the
>> USF
>> Production Staging Environment though I thought the Tampa QA USF lab made
>> more sense as we had DNS service.
>>
>> It seemed to me, however, that Boulder is where IBM manufactures most of
>> their Blue Tape.
>>
>> -soup
>>
>> On Thu, Apr 19, 2018 at 12:34 PM, Scott Rohling 
>> wrote:
>>
>> > >
>> > >
>> > >
>> > >
>> > > I've also got some Foxworthy-ish routines;  When working for IBM in
>> Tampa
>> > > I've worked with folks who remembered when Jeff Foxworthy was
>> employed by
>> > > IBM.
>> > >
>> > >
>> > >
>> > ​I worked for his father, Jim Foxworthy at the Tampa IBM support center
>> > (1979-83) ...  met Jeff at a wedding when his dad remarried Annie who
>> > worked at the support center too.   I might have missed his working for
>> IBM
>> > ... headed for Boulder April '83 and stayed there for 30 years.
>>  Small
>> > world..  ;-)
>> >
>> > Scott Rohling​
>> >
>> > --
>> > For LINUX-390 subscribe / signoff / archive access instructions,
>> > send email to lists...@vm.marist.edu with the message: INFO LINUX-390
>> or
>> > visit
>> > http://www.marist.edu/htbin/wlvindex?LINUX-390
>> > --
>> > For more information on Linux on System z, visit
>> > http://wiki.linuxvm.org/
>> >
>>
>>
>>
>> --
>> John R. Campbell Speaker to Machines  souperb at gmail dot
>> com
>> MacOS X proved it was easier to make Unix user-friendly than to fix
>> Windows
>> "It doesn't matter how well-crafted a system is to eliminate errors;
>> Regardless
>>  of any and all checks and balances in place, all systems will fail
>> because,
>>  somewhere, there is meat in the loop." - me
>>
>> --
>> For LINUX-390 subscribe / signoff / archive access instructions,
>> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
>> visit
>> http://www.marist.edu/htbin/wlvindex?LINUX-390
>> --
>> For more information on Linux on System z, visit
>> http://wiki.linuxvm.org/
>>
>
>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[Scott Rohling]

shoot PRB .. peoples republic of boulder

On Thu, Apr 19, 2018 at 10:22 AM, Scott Rohling 
wrote:

> Oh and PBR is a real thing...   I lived up in Nederland and Eldora and
> Jamestown --  no flatlands for me -  Boulder was the cosmic center of the
> universe for awhile (burp)
>
> Scott Rohling
>
> On Thu, Apr 19, 2018 at 10:20 AM, Scott Rohling 
> wrote:
>
>> Don't know - I worked on RETAIN and then HONE - wrote a thing called
>> REGISTER that did DIRMAINT/RACF admin via panels - then z/VM tools and
>> system support -then started working from home in '96 doing various and
>> sundry things around z/VM and zLinux.   I just know the Boulder plant
>> started to feel like a haunted house as manufacturing (especially printing)
>> moved elsewhere.   Now it's a major DR site AFAIK ...
>>
>> Well we're very OT here but thanks for reminiscing ;-)
>>
>> Scott Rohling
>>
>> On Thu, Apr 19, 2018 at 9:55 AM, John Campbell  wrote:
>>
>>> Well, I spent a week at the People's Republic of Boulder to work on the
>>> USF
>>> Production Staging Environment though I thought the Tampa QA USF lab made
>>> more sense as we had DNS service.
>>>
>>> It seemed to me, however, that Boulder is where IBM manufactures most of
>>> their Blue Tape.
>>>
>>> -soup
>>>
>>> On Thu, Apr 19, 2018 at 12:34 PM, Scott Rohling >> >
>>> wrote:
>>>
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > I've also got some Foxworthy-ish routines;  When working for IBM in
>>> Tampa
>>> > > I've worked with folks who remembered when Jeff Foxworthy was
>>> employed by
>>> > > IBM.
>>> > >
>>> > >
>>> > >
>>> > ​I worked for his father, Jim Foxworthy at the Tampa IBM support center
>>> > (1979-83) ...  met Jeff at a wedding when his dad remarried Annie who
>>> > worked at the support center too.   I might have missed his working
>>> for IBM
>>> > ... headed for Boulder April '83 and stayed there for 30 years.
>>>  Small
>>> > world..  ;-)
>>> >
>>> > Scott Rohling​
>>> >
>>> > --
>>> > For LINUX-390 subscribe / signoff / archive access instructions,
>>> > send email to lists...@vm.marist.edu with the message: INFO LINUX-390
>>> or
>>> > visit
>>> > http://www.marist.edu/htbin/wlvindex?LINUX-390
>>> > --
>>> > For more information on Linux on System z, visit
>>> > http://wiki.linuxvm.org/
>>> >
>>>
>>>
>>>
>>> --
>>> John R. Campbell Speaker to Machines  souperb at gmail
>>> dot
>>> com
>>> MacOS X proved it was easier to make Unix user-friendly than to fix
>>> Windows
>>> "It doesn't matter how well-crafted a system is to eliminate errors;
>>> Regardless
>>>  of any and all checks and balances in place, all systems will fail
>>> because,
>>>  somewhere, there is meat in the loop." - me
>>>
>>> --
>>> For LINUX-390 subscribe / signoff / archive access instructions,
>>> send email to lists...@vm.marist.edu with the message: INFO LINUX-390
>>> or visit
>>> http://www.marist.edu/htbin/wlvindex?LINUX-390
>>> --
>>> For more information on Linux on System z, visit
>>> http://wiki.linuxvm.org/
>>>
>>
>>
>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[Scott Rohling]

Don't know - I worked on RETAIN and then HONE - wrote a thing called
REGISTER that did DIRMAINT/RACF admin via panels - then z/VM tools and
system support -then started working from home in '96 doing various and
sundry things around z/VM and zLinux.   I just know the Boulder plant
started to feel like a haunted house as manufacturing (especially printing)
moved elsewhere.   Now it's a major DR site AFAIK ...

Well we're very OT here but thanks for reminiscing ;-)

Scott Rohling

On Thu, Apr 19, 2018 at 9:55 AM, John Campbell  wrote:

> Well, I spent a week at the People's Republic of Boulder to work on the USF
> Production Staging Environment though I thought the Tampa QA USF lab made
> more sense as we had DNS service.
>
> It seemed to me, however, that Boulder is where IBM manufactures most of
> their Blue Tape.
>
> -soup
>
> On Thu, Apr 19, 2018 at 12:34 PM, Scott Rohling 
> wrote:
>
> > >
> > >
> > >
> > >
> > > I've also got some Foxworthy-ish routines;  When working for IBM in
> Tampa
> > > I've worked with folks who remembered when Jeff Foxworthy was employed
> by
> > > IBM.
> > >
> > >
> > >
> > ​I worked for his father, Jim Foxworthy at the Tampa IBM support center
> > (1979-83) ...  met Jeff at a wedding when his dad remarried Annie who
> > worked at the support center too.   I might have missed his working for
> IBM
> > ... headed for Boulder April '83 and stayed there for 30 years. Small
> > world..  ;-)
> >
> > Scott Rohling​
> >
> > --
> > For LINUX-390 subscribe / signoff / archive access instructions,
> > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> > visit
> > http://www.marist.edu/htbin/wlvindex?LINUX-390
> > --
> > For more information on Linux on System z, visit
> > http://wiki.linuxvm.org/
> >
>
>
>
> --
> John R. Campbell Speaker to Machines  souperb at gmail dot
> com
> MacOS X proved it was easier to make Unix user-friendly than to fix Windows
> "It doesn't matter how well-crafted a system is to eliminate errors;
> Regardless
>  of any and all checks and balances in place, all systems will fail
> because,
>  somewhere, there is meat in the loop." - me
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[John Campbell]

Well, I spent a week at the People's Republic of Boulder to work on the USF
Production Staging Environment though I thought the Tampa QA USF lab made
more sense as we had DNS service.

It seemed to me, however, that Boulder is where IBM manufactures most of
their Blue Tape.

-soup

On Thu, Apr 19, 2018 at 12:34 PM, Scott Rohling 
wrote:

> >
> >
> >
> >
> > I've also got some Foxworthy-ish routines;  When working for IBM in Tampa
> > I've worked with folks who remembered when Jeff Foxworthy was employed by
> > IBM.
> >
> >
> >
> ​I worked for his father, Jim Foxworthy at the Tampa IBM support center
> (1979-83) ...  met Jeff at a wedding when his dad remarried Annie who
> worked at the support center too.   I might have missed his working for IBM
> ... headed for Boulder April '83 and stayed there for 30 years. Small
> world..  ;-)
>
> Scott Rohling​
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>



-- 
John R. Campbell Speaker to Machines  souperb at gmail dot
com
MacOS X proved it was easier to make Unix user-friendly than to fix Windows
"It doesn't matter how well-crafted a system is to eliminate errors;
Regardless
 of any and all checks and balances in place, all systems will fail because,
 somewhere, there is meat in the loop." - me

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[Quay, Jonathan (IBM)]

Jeff used to be the S/34 CE at the small company where my wife (then 
girlfriend) and I worked.  When Jeff told her he was quitting IBM to do 
stand-up, she told him he was crazy to leave the security of a job with IBM.


From: Linux on 390 Port <LINUX-390@VM.MARIST.EDU> on behalf of Scott Rohling 
<scott.rohl...@gmail.com>
Sent: Thursday, April 19, 2018 12:34:49 PM
To: LINUX-390@VM.MARIST.EDU
Subject: Re: Linux "sleep" command not waking up under high CPU utilization

>
>
>
>
> I've also got some Foxworthy-ish routines;  When working for IBM in Tampa
> I've worked with folks who remembered when Jeff Foxworthy was employed by
> IBM.
>
>
>
​I worked for his father, Jim Foxworthy at the Tampa IBM support center
(1979-83) ...  met Jeff at a wedding when his dad remarried Annie who
worked at the support center too.   I might have missed his working for IBM
... headed for Boulder April '83 and stayed there for 30 years. Small
world..  ;-)

Scott Rohling​

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.marist.edu%2Fhtbin%2Fwlvindex%3FLINUX-390=02%7C01%7Cjonathan.quay%40IHG.COM%7C4cf6530e355a49069bcf08d5a613b414%7C2762c43f29c1448a89f67ac903cf8316%7C1%7C0%7C636597525910922744=Gh4z32xrRI2Tn41QuClYI0GQJobrUF3rSBaCN3BmAQg%3D=0
--
For more information on Linux on System z, visit
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.linuxvm.org%2F=02%7C01%7Cjonathan.quay%40IHG.COM%7C4cf6530e355a49069bcf08d5a613b414%7C2762c43f29c1448a89f67ac903cf8316%7C1%7C0%7C636597525910922744=YzsSo0SY2ZBkR%2FUN5OMtssNBiVVj77NZgwenKQG04q0%3D=0

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[Scott Rohling]

>
>
>
>
> I've also got some Foxworthy-ish routines;  When working for IBM in Tampa
> I've worked with folks who remembered when Jeff Foxworthy was employed by
> IBM.
>
>
>
​I worked for his father, Jim Foxworthy at the Tampa IBM support center
(1979-83) ...  met Jeff at a wedding when his dad remarried Annie who
worked at the support center too.   I might have missed his working for IBM
... headed for Boulder April '83 and stayed there for 30 years. Small
world..  ;-)

Scott Rohling​

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[John Campbell]

The dead giveaway (IMHO):

DAVID JONES
MANAGING DIRECTOR FOR ZSYSTEMS SERVICES
z/VM, Linux, and Cloud


All right, so take a bucket of salt as you read the below as it qualifies
as a WAG (and not a SWAG).

And, yes, my tongue is firmly planted in my cheek as I keyboarded the
below.  I cope with stress via levity and seldom hesitate to inflict it
upon all those around me, for I also suffer from ADD2 ("Audience Deficit
Disorder") and I seem to "share" the suffering.   :-) :-) :-)


The "z/VM" in his title makes me think "z/VM overcommitment".


What was the experiment's name when David Boyes did the 44K Linux
instances?  On an S/390 (31bit)?

As magical (after all, for some, IBM is a TLA for "I Bring Magic") as z/VM
may appear, it is merely "advanced technology" and the virtual instances
may _seem_ real...  but they're not.  There's always a trade-off.

"You get what you pay for...  and, if you really think you came out ahead,
you'll pay for what you got, though the currency may not be in dollars and
cents."  - me

Heinlein provided guidance, too:  "There Ain't No Such Thing As A Free
Lunch".  The uptake of Linux make some folks think there's a free lunch but
there are other costs;  some costs may be invisible.

[HUMOR]
Mind you, I've come to the conclusion, perhaps simplistically due to my
observational angle, many bean-counters seem to confuse brownian motion for
progress.
[/HUMOR]

I've also got some Foxworthy-ish routines;  When working for IBM in Tampa
I've worked with folks who remembered when Jeff Foxworthy was employed by
IBM.

(smirks)

-soup

On Thu, Apr 19, 2018 at 11:35 AM, Christian Borntraeger <
borntrae...@de.ibm.com> wrote:

> On 04/19/2018 04:56 PM, Dave Jones wrote:
> > Thanks to all who responded to my query. At least we now know why the
> > problem is occurring, and we will try some internal z/linux tuning.
> >
>
> Anything useful that can be shared with the list?
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>



--
John R. Campbell Speaker to Machines  souperb at gmail dot
com
MacOS X proved it was easier to make Unix user-friendly than to fix Windows
"It doesn't matter how well-crafted a system is to eliminate errors;
Regardless
 of any and all checks and balances in place, all systems will fail because,
 somewhere, there is meat in the loop." - me

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[Dave Jones]

We're just beginning this process, Christian. I'll let the group know
what we find out.

DJ

---

DAVID JONES | MANAGING DIRECTOR FOR ZSYSTEMS SERVICES | z/VM, Linux, and
Cloud
 703.237.7370 (Office) | 281.578.7544 (CELL)

INFORMATION TECHNOLOGY COMPANY [3]

On 04.19.2018 8:35 AM, Christian Borntraeger wrote:

> On 04/19/2018 04:56 PM, Dave Jones wrote:
>
>> Thanks to all who responded to my query. At least we now know why the 
>> problem is occurring, and we will try some internal z/linux tuning.
>
> Anything useful that can be shared with the list?
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390 [1]
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/ [2]


Links:
--
[1] http://www.marist.edu/htbin/wlvindex?LINUX-390
[2] http://wiki.linuxvm.org/
[3] http://www.itconline.com/

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[Christian Borntraeger]

On 04/19/2018 04:56 PM, Dave Jones wrote:
> Thanks to all who responded to my query. At least we now know why the
> problem is occurring, and we will try some internal z/linux tuning.
>

Anything useful that can be shared with the list?

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[Dave Jones]

Thanks to all who responded to my query. At least we now know why the
problem is occurring, and we will try some internal z/linux tuning.

DJ

---

DAVID JONES | MANAGING DIRECTOR FOR ZSYSTEMS SERVICES | z/VM, Linux, and
Cloud
 703.237.7370 (Office) | 281.578.7544 (CELL)

INFORMATION TECHNOLOGY COMPANY [3]

On 04.18.2018 1:16 PM, John Campbell wrote:

> "This should not happen"... Never confuse theory with practice.
>
> In Software development, theory and practice appear identical. (BT,DT)
>
> In Systems Integration and Application Production Support you quickly learn
> that theory and practice are seldom within the same galaxy much less the
> same solar system. (BT,DT)
>
> I've done enough problem determination within BOTH environments...
>
> (chuckles)
>
> (Draws on memory)
>
> Running an IBM labspace I had three AIX servers for supporting testing of
> an application that required SNA support. I also owned the delegated DNS
> subdomain so it was up to me to recommend names. One server handled
> functional testing, the second was for system testing and the final one for
> "user" testing. I named them flog, flail and flaunt. (The SNA gateway was
> named "phlegm", BTW, instead of "snaat".)
>
> In various places, beyond this, I have learned that "UAT" seems to stand
> for "User Avoided Testing".
>
> I've also worked w/ Tandem guys who throw around claims like 5 nines...
> and had the systems _I_ supported insulted as 9 fives servers.
>
> (shrugs)
>
> So I have a LOT of practice learning how systems DO NOT WORK AS EXPECTED.
>
> To paraphrase Patton's Law for this milieu... "No Plan Survives Contact
> With Reality".
>
> -soup
>
> On Wed, Apr 18, 2018 at 3:01 PM, Christian Borntraeger <
> borntrae...@de.ibm.com> wrote:
> On 04/18/2018 03:36 PM, Dave Jones wrote: Hello, gang. I have a very simple 
> bash script that runs a trivial data collection task, and then does a Linux 
> "sleep im" to wait a minute before running the data collection task again. 
> Under very high CPU loads (> 90%) I have noticed that the "sleep" command 
> does not seem to wake up after one minute but instead wakes up 15 to 20 
> minutes later. This is on a Red Hat 6.9 guest running under z/VM 6.4 on a z12 
> box. I would like to buy a clue here if I could. This should not happen, even 
> under load. Is this really the sleep that does not wakeup or is maybe the 
> following stuff not getting access to the data? Have you tried with "set -x" 
> in the bash script to see what commands bash is executing? 
> -- For 
> LINUX-390 subscribe / signoff / archive access instructions, send email to 
> lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390 [1] 
-- For more 
information on Linux on System z, visit http://wiki.linuxvm.org/ [2]

--
John R. Campbell Speaker to Machines souperb at gmail dot
com
MacOS X proved it was easier to make Unix user-friendly than to fix
Windows
"It doesn't matter how well-crafted a system is to eliminate errors;
Regardless
 of any and all checks and balances in place, all systems will fail
because,
 somewhere, there is meat in the loop." - me

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
visit
http://www.marist.edu/htbin/wlvindex?LINUX-390 [1]
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/ [2]



Links:
--
[1] http://www.marist.edu/htbin/wlvindex?LINUX-390
[2] http://wiki.linuxvm.org/
[3] http://www.itconline.com/

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[R P Herrold]

On Wed, 18 Apr 2018, John Campbell wrote:

> "This should not happen"...  Never confuse theory with practice.

The tell-tale give-away is the use of the verb form:
should

as it indicates the speaker is working from expectation rather
than observation

-- Russ herrold

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[John Campbell]

"This should not happen"...  Never confuse theory with practice.

In Software development, theory and practice appear identical.  (BT,DT)

In Systems Integration and Application Production Support you quickly learn
that theory and practice are seldom within the same galaxy much less the
same solar system.  (BT,DT)

I've done enough problem determination within BOTH environments...

(chuckles)

(Draws on memory)

Running an IBM labspace I had three AIX servers for supporting testing of
an application that required SNA support.  I also owned the delegated DNS
subdomain so it was up to me to recommend names.  One server handled
functional testing, the second was for system testing and the final one for
"user" testing.  I named them flog, flail and flaunt.  (The SNA gateway was
named "phlegm", BTW, instead of "snaat".)

In various places, beyond this, I have learned that "UAT" seems to stand
for "User Avoided Testing".

I've also worked w/ Tandem guys who throw around claims like 5 nines...
and had the systems _I_ supported insulted as 9 fives servers.

(shrugs)

So I have a LOT of practice learning how systems DO NOT WORK AS EXPECTED.

To paraphrase Patton's Law for this milieu...  "No Plan Survives Contact
With Reality".

-soup

On Wed, Apr 18, 2018 at 3:01 PM, Christian Borntraeger <
borntrae...@de.ibm.com> wrote:

> On 04/18/2018 03:36 PM, Dave Jones wrote:
> > Hello, gang.
> >
> > I have a very simple bash script that runs a trivial data collection
> > task, and then does a Linux "sleep im" to wait a minute before running
> > the data collection task again. Under very high CPU loads (> 90%) I have
> > noticed that the "sleep" command does not seem to wake up after one
> > minute but instead wakes up 15 to 20 minutes later. This is on a Red Hat
> > 6.9 guest running under z/VM 6.4 on a z12 box.
> >
> > I would like to buy a clue here if I could.
>
> This should not happen, even under load. Is this really the sleep that
> does not
> wakeup or is maybe the following stuff not getting access to the data?
> Have you
> tried with "set -x" in the bash script to see what commands bash is
> executing?
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>



--
John R. Campbell Speaker to Machines  souperb at gmail dot
com
MacOS X proved it was easier to make Unix user-friendly than to fix Windows
"It doesn't matter how well-crafted a system is to eliminate errors;
Regardless
 of any and all checks and balances in place, all systems will fail because,
 somewhere, there is meat in the loop." - me

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[Dave Jones]

> If in a virtualized instance...
> https://www.youtube.com/watch?v=__n5Bgxx-68

 [4]Very well played, Sir! remarkably funny and one of my favorites,
too.
DJ [4]

---

DAVID JONES | MANAGING DIRECTOR FOR ZSYSTEMS SERVICES | z/VM, Linux, and
Cloud
 703.237.7370 (Office) | 281.578.7544 (CELL)

INFORMATION TECHNOLOGY COMPANY [1]

On 04.18.2018 7:10 AM, John Campbell wrote:

> Buy a clue? That can be expensive depending upon how many vowels are
> necessary. :-)
>
> Look, the sleep call is awkward to begin with... and that's within a C
> program! (Since perl uses sleep() it can work more-or-less the same way.)
> A sleep(1) call will vary, for instance. Depending upon the system using a
> nap(ms) call has its own complications.
>
> A "sleep 1m" (or, the way I code the command, "sleep 60" since the default
> granularity is in seconds) includes a fork()+exec() call and then passing,
> on completion, notifying the parent process it can now proceed.
>
> In a very busy system, the fork()/exec() to LAUNCH the sleep command may be
> delayed. (Personally, when working within a system that is loaded down, I
> prefer to code a perl script, just to cut down on the fork()/exec() rate.)
> So, the need to execute fork() to get a child process launched-- which
> starts out as a copy of the existing shell-- can require resources that are
> in short supply EVEN WHEN the sleep command renders a teeny process,
> memory-wise, so, when the system is loaded down, forking can be impacted
> (RAM, CPU, etc) while the exec() call, itself, starts a process with a very
> minimal page set and lets page faults load the rest of it. (And, just as a
> reminder, the fork() call means the impure "data" and "stack" segments have
> to be allocated and copied.)
>
> So the system is loaded down.
>
> If this is a virtual instance, however, you may want to investigate whether
> the instance is being impacted by OTHER instances where resources are
> over-committed. Other instances can't be seen, howsomever, so finding out
> if some other instance is stealing your cycles (or putting your instance
> into paging space) can be harder to detect. The usual performance
> monitoring tools don't work "well" when CPU and RAM are overcommitted.
>
> YMMV.
>
> If you are resident in an LPAR you can use internal performance tools: Look
> at the top command's output. Run vmstat or even sar. Consider what your
> system is doing.
>
> If in a virtualized instance...
> https://www.youtube.com/watch?v=__n5Bgxx-68 [4]
>
> Finally, while a process is sleeping, it can be pushed out of RAM and then
> waking up means you'll have a whole buncha page faults as it tries to get
> running again. A virtual instance can have DOUBLE page faults both within
> the instance which cause a page fault the hypervisor has to address.
>
> If you are running an "important" set of processes within a virtual
> instance (shakes head).
>
> -soup
>
> On Wed, Apr 18, 2018 at 9:36 AM, Dave Jones  wrote:
>
>> Hello, gang. I have a very simple bash script that runs a trivial data 
>> collection task, and then does a Linux "sleep im" to wait a minute before 
>> running the data collection task again. Under very high CPU loads (> 90%) I 
>> have noticed that the "sleep" command does not seem to wake up after one 
>> minute but instead wakes up 15 to 20 minutes later. This is on a Red Hat 6.9 
>> guest running under z/VM 6.4 on a z12 box. I would like to buy a clue here 
>> if I could. Thanks. DJ -- DAVID JONES | MANAGING DIRECTOR FOR ZSYSTEMS 
>> SERVICES | z/VM, Linux, and Cloud 703.237.7370 (Office) | 281.578.7544 
>> (CELL) INFORMATION TECHNOLOGY COMPANY [1 [1]] Links: -- [1] 
>> http://www.itconline.com/ [1] 
>> -- For 
>> LINUX-390 subscribe / signoff / archive access instructions, send email to 
>> lists...@vm.marist.edu with the message: INFO LINUX-390 or visit 
>> http://www.marist.edu/htbin/wlvindex?LINUX-390 [2]
-- For more 
information on Linux on System z, visit http://wiki.linuxvm.org/ [3]
>
> --
> John R. Campbell Speaker to Machines souperb at gmail dot
> com
> MacOS X proved it was easier to make Unix user-friendly than to fix Windows
> "It doesn't matter how well-crafted a system is to eliminate errors;
> Regardless
> of any and all checks and balances in place, all systems will fail because,
> somewhere, there is meat in the loop." - me
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390 [2]
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/ [3]


Links:
--
[1] http://www.itconline.com/
[2] 

Re: Linux "sleep" command not waking up under high CPU utilization

[Christian Borntraeger]

On 04/18/2018 03:36 PM, Dave Jones wrote:
> Hello, gang.
>
> I have a very simple bash script that runs a trivial data collection
> task, and then does a Linux "sleep im" to wait a minute before running
> the data collection task again. Under very high CPU loads (> 90%) I have
> noticed that the "sleep" command does not seem to wake up after one
> minute but instead wakes up 15 to 20 minutes later. This is on a Red Hat
> 6.9 guest running under z/VM 6.4 on a z12 box.
>
> I would like to buy a clue here if I could.

This should not happen, even under load. Is this really the sleep that does not
wakeup or is maybe the following stuff not getting access to the data? Have you
tried with "set -x" in the bash script to see what commands bash is executing?

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Linux "sleep" command not waking up under high CPU utilization

[John Campbell]

Buy a clue?  That can be expensive depending upon how many vowels are
necessary.  :-)

Look, the sleep call is awkward to begin with...  and that's within a C
program!  (Since perl uses sleep() it can work more-or-less the same way.)
A sleep(1) call will vary, for instance.  Depending upon the system using a
nap(ms) call has its own complications.

A "sleep 1m" (or, the way I code the command, "sleep 60" since the default
granularity is in seconds) includes a fork()+exec() call and then passing,
on completion, notifying the parent process it can now proceed.

In a very busy system, the fork()/exec() to LAUNCH the sleep command may be
delayed. (Personally, when working within a system that is loaded down, I
prefer to code a perl script, just to cut down on the fork()/exec() rate.)
So, the need to execute fork() to get a child process launched-- which
starts out as a copy of the existing shell-- can require resources that are
in short supply EVEN WHEN the sleep command renders a teeny process,
memory-wise, so, when the system is loaded down, forking can be impacted
(RAM, CPU, etc) while the exec() call, itself, starts a process with a very
minimal page set and lets page faults load the rest of it.  (And, just as a
reminder, the fork() call means the impure "data" and "stack" segments have
to be allocated and copied.)

So the system is loaded down.

If this is a virtual instance, however, you may want to investigate whether
the instance is being impacted by OTHER instances where resources are
over-committed.  Other instances can't be seen, howsomever, so finding out
if some other instance is stealing your cycles (or putting your instance
into paging space) can be harder to detect.  The usual performance
monitoring tools don't work "well" when CPU and RAM are overcommitted.

YMMV.

If you are resident in an LPAR you can use internal performance tools: Look
at the top command's output.  Run vmstat or even sar.  Consider what your
system is doing.

If in a virtualized instance...
https://www.youtube.com/watch?v=__n5Bgxx-68

Finally, while a process is sleeping, it can be pushed out of RAM and then
waking up means you'll have a whole buncha page faults as it tries to get
running again.  A virtual instance can have DOUBLE page faults both within
the instance which cause a page fault the hypervisor has to  address.

If you are running an "important" set of processes within a virtual
instance (shakes head).

-soup

On Wed, Apr 18, 2018 at 9:36 AM, Dave Jones  wrote:

> Hello, gang.
>
> I have a very simple bash script that runs a trivial data collection
> task, and then does a Linux "sleep im" to wait a minute before running
> the data collection task again. Under very high CPU loads (> 90%) I have
> noticed that the "sleep" command does not seem to wake up after one
> minute but instead wakes up 15 to 20 minutes later. This is on a Red Hat
> 6.9 guest running under z/VM 6.4 on a z12 box.
>
> I would like to buy a clue here if I could.
>
> Thanks.
>
> DJ
>
> --
>
> DAVID JONES | MANAGING DIRECTOR FOR ZSYSTEMS SERVICES | z/VM, Linux, and
> Cloud
>  703.237.7370 (Office) | 281.578.7544 (CELL)
>
> INFORMATION TECHNOLOGY COMPANY [1]
>
>
> Links:
> --
> [1] http://www.itconline.com/
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> --
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>



--
John R. Campbell Speaker to Machines  souperb at gmail dot
com
MacOS X proved it was easier to make Unix user-friendly than to fix Windows
"It doesn't matter how well-crafted a system is to eliminate errors;
Regardless
 of any and all checks and balances in place, all systems will fail because,
 somewhere, there is meat in the loop." - me

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/