Re: nanosleep() for shorted than schedule slice

[Michael van Elst]

b...@softjar.se (Johnny Billquist) writes:

>Having the normal wall clock driven by a tick interrupt has its points. 

We usually avoid this and use what hardware timer the platform offers.


>It's just that for high resolution timers, ticks are not a good source. 
>For anything else, they are just fine. So why conflate the two? 

Because you don't need two clock interrupts. The regular interrupt is
just another event that happens to be scheduled in a regular interval.



-- 
-- 
Michael van Elst
Internet: mlel...@serpens.de
"A potential Snark may lurk in every tree."

Tickless kernel

[maya]

On Sun, Jul 02, 2017 at 11:10:19PM -, Michael van Elst wrote:
> it needs MD support on all platforms.

What more does an implementation need besides asking for an interrupt in
timo at cv_timedwait* (and increasing HZ)?

That could be a no-op for platforms that can't provide it.

Re: nanosleep() for shorted than schedule slice

[David Holland]

On Sun, Jul 02, 2017 at 05:04:56PM -0700, John Nemeth wrote:
 > } > > I wonder if it would make sense for nanosleep(2) to check
 > } > > that requested sleeping time is shorter than a schedule
 > } > > slice, and if it is, spin the CPU instead of scheduling
 > } > > another process. Any opinion on this?
 > } > 
 > } > No, that's wrong. It's also been discussed before.
 > } 
 > } How is that wrong? It was always more or less the point of nanosleep.
 > 
 >  If you start spinning right after the start of a timeslice,
 > you could spin for close to an entire timeslice.  On a modern
 > multi-GHz CPU that's a tremendous number of wasted cycles (also
 > doesn't help power consumption).

...huh?

What does "right after the start of a timeslice" have to do with it?

Anyway, if the requested sleep time is "close to an entire timeslice"
then you obviously don't spin. Please don't manufacture nonsense
counterexamples.

-- 
David A. Holland
dholl...@netbsd.org

Re: nanosleep() for shorted than schedule slice

[John Nemeth]

On Jul 2,  8:04pm, David Holland wrote:
} Subject: Re: nanosleep() for shorted than schedule slice
} On Sun, Jul 02, 2017 at 12:54:52PM +0200, Joerg Sonnenberger wrote:
}  > > I wonder if it would make sense for nanosleep(2) to check that requested
}  > > sleeping time is shorter than a schedule slice, and if it is, spin the
}  > > CPU instead of scheduling another process. Any opinion on this?
}  > 
}  > No, that's wrong. It's also been discussed before.
} 
} How is that wrong? It was always more or less the point of nanosleep.

 If you start spinning right after the start of a timeslice,
you could spin for close to an entire timeslice.  On a modern
multi-GHz CPU that's a tremendous number of wasted cycles (also
doesn't help power consumption).

}-- End of excerpt from David Holland

Re: nanosleep() for shorted than schedule slice

[Johnny Billquist]

On 2017-07-03 01:10, Michael van Elst wrote:

b...@softjar.se (Johnny Billquist) writes:


A tickless kernel wouldn't run callouts from the regular clock interrupt
but would use a hires timer to issue interrupts at arbitrary times.
The callout API could then be changed to either accept timespec values or
just fake a much higher HZ value.



Right. Not that I believe this have to be tied into tickless, but I
suspect it might be easier to do it if we go tickless.



Well, "not using a regular clock interrupt" is what "tickless" means.


One does not exclude the other.


We really should be able to deal with shorter times, even if we have
ticks.


That's a contradiction. "ticks" means that timed events are based
on a regular clock interrupt. Of course you can speed up the ticks
(e.g. Alpha uses HZ=1000), but that has other disadvantages.

N.B. going tickless isn't difficult, it's just lots of work as it needs
MD support on all platforms.


There is no conflict in having both. Some things can run based on a 
regular tick, while some things are based on hardware timer that 
interrupts when the specified time have passed.
That would, by pretty much all definitions I know, still classify as a 
tick-based system.


It's pretty much just that we have two clock sources. Which is nothing 
new, really. And having some things driven by interrupt events is 
already the norm for non-clock based events. I believe most hardware 
already supports such an idea as well. One low resolution clock which 
generates interrupt at something like 100Hz, and one high resolution 
clock, which can be programmed to whatever (short) time in the future to 
generate an interrupt.


Having the normal wall clock driven by a tick interrupt has its points. 
And having things like preemptive context switches based on ticks is 
pretty much what you want anyway. With a tickless kernel, you'll have to 
set up those ticks anyway.


It's just that for high resolution timers, ticks are not a good source. 
For anything else, they are just fine. So why conflate the two? 
(Rethorical question, and I won't object if someone just implements a 
tickless kernel as well, but it is not as trivial as hinted. You 
probably will end up with having such a system run a tick anyway, unless 
you want to start calculating wall clock time in all kind of places 
under various circumstances, where wall clock is used. But of course, 
it's all doable.)


Johnny

--
Johnny Billquist  || "I'm on a bus
  ||  on a psychedelic trip
email: b...@softjar.se ||  Reading murder books
pdp is alive! ||  tryin' to stay hip" - B. Idol

Re: nanosleep() for shorted than schedule slice

[Johnny Billquist]

On 2017-07-02 23:24, Michael van Elst wrote:

b...@softjar.se (Johnny Billquist) writes:


I don't get it. What was the problem with using nanosleep for short
usleep's?


usleep is just a wrapper around nanosleep. There is no difference
except that nanosleep accepts higher precision delays.


So, the comment previously that it was not ok to use nanosleep for short 
usleeps seems to not hold much water.



The kernel computes the number of ticks to sleep, schedules a callout
that will wake the thread up and calls the scheduler to run other
threads in the meantime. The callout is later dispatched by the
clock interrupt.

So you always have to wait for at least one tick, with HZ=100 that's 10ms.


Right. With the current implementation. And which means that "higher 
precision delays" are just imaginary. Our implementation does not 
actually give us any higher precision delays.



A tickless kernel wouldn't run callouts from the regular clock interrupt
but would use a hires timer to issue interrupts at arbitrary times.
The callout API could then be changed to either accept timespec values or
just fake a much higher HZ value.


Right. Not that I believe this have to be tied into tickless, but I 
suspect it might be easier to do it if we go tickless.


I think it's pretty bad that our timers only have the precision of 
ticks, even if people call something like nanosleep.
By the way, the fact that nanosleep also in the end uses ticks means 
that the OP suggestion of using nanosleep for short usleeps will not 
really improve the situation at all.


We really should be able to deal with shorter times, even if we have 
ticks. But I guess it all depends on the hardware (which it does if we 
go to tickless as well, though.)


Part of my confusion might also be slightly different uses of the term 
tickless. The kernels use of ticks for time keeping and some work does 
not necessarily force us to use them for callouts. It's just convenient. 
We could (maybe should?) use a different mechanism for the callout 
subsystem.


Johnny

--
Johnny Billquist  || "I'm on a bus
  ||  on a psychedelic trip
email: b...@softjar.se ||  Reading murder books
pdp is alive! ||  tryin' to stay hip" - B. Idol

Re: nanosleep() for shorted than schedule slice

[Joerg Sonnenberger]

On Sun, Jul 02, 2017 at 09:15:55PM +, m...@netbsd.org wrote:
> On Sun, Jul 02, 2017 at 02:10:38PM -0700, John Nemeth wrote:
> >  Saying, "we are being stupid," could most definitely be an
> > insult to the people that wrote the code in question.  I seriously
> > doubt that we are being stupid in this case.
> 
> Probably. I'm just upset, sorry.
> I like the idea of delay.

Wasting CPU time playing with your thumbs is my idea of stupid...

Joerg

Re: nanosleep() for shorted than schedule slice

[Michael van Elst]

m...@netbsd.org writes:

>On Sun, Jul 02, 2017 at 01:16:15PM +, Christos Zoulas wrote:
>> The solution is to implement "tickless kernel". It is not that difficult.

>It looks like we are always descheduling the thread, not jut because we
>got a clock tick. even a tickless kernel won't help stupid.

A tickless kernel would wake the thread in time and let it preempt
a running thread when necessary.

Of course, for this to work nicely, the sleeping thread needs to
be woken up at a high enough priority. But that's a different problem.

-- 
-- 
Michael van Elst
Internet: mlel...@serpens.de
"A potential Snark may lurk in every tree."

Re: nanosleep() for shorted than schedule slice

[Michael van Elst]

b...@softjar.se (Johnny Billquist) writes:

>I don't get it. What was the problem with using nanosleep for short 
>usleep's?

usleep is just a wrapper around nanosleep. There is no difference
except that nanosleep accepts higher precision delays.

The kernel computes the number of ticks to sleep, schedules a callout
that will wake the thread up and calls the scheduler to run other
threads in the meantime. The callout is later dispatched by the
clock interrupt.

So you always have to wait for at least one tick, with HZ=100 that's 10ms.

A tickless kernel wouldn't run callouts from the regular clock interrupt
but would use a hires timer to issue interrupts at arbitrary times.
The callout API could then be changed to either accept timespec values or
just fake a much higher HZ value.

-- 
-- 
Michael van Elst
Internet: mlel...@serpens.de
"A potential Snark may lurk in every tree."

Re: nanosleep() for shorted than schedule slice

[maya]

On Sun, Jul 02, 2017 at 02:10:38PM -0700, John Nemeth wrote:
>  Saying, "we are being stupid," could most definitely be an
> insult to the people that wrote the code in question.  I seriously
> doubt that we are being stupid in this case.

Probably. I'm just upset, sorry.
I like the idea of delay.

Re: nanosleep() for shorted than schedule slice

[John Nemeth]

On Jul 2,  8:41pm, m...@netbsd.org wrote:
} On Sun, Jul 02, 2017 at 08:38:24PM +, m...@netbsd.org wrote:
} > On Sun, Jul 02, 2017 at 01:16:15PM +, Christos Zoulas wrote:
} > > The solution is to implement "tickless kernel". It is not that difficult.
} > 
} > It looks like we are always descheduling the thread, not jut because we
} > got a clock tick. even a tickless kernel won't help stupid.
} 
} to clarify, I mean, "it won't help, because we are being stupid", not
} as an insult.

 Saying, "we are being stupid," could most definitely be an
insult to the people that wrote the code in question.  I seriously
doubt that we are being stupid in this case.

}-- End of excerpt from m...@netbsd.org

Re: nanosleep() for shorted than schedule slice

[maya]

On Sun, Jul 02, 2017 at 08:38:24PM +, m...@netbsd.org wrote:
> On Sun, Jul 02, 2017 at 01:16:15PM +, Christos Zoulas wrote:
> > The solution is to implement "tickless kernel". It is not that difficult.
> 
> It looks like we are always descheduling the thread, not jut because we
> got a clock tick. even a tickless kernel won't help stupid.

to clarify, I mean, "it won't help, because we are being stupid", not
as an insult.

Re: nanosleep() for shorted than schedule slice

[maya]

On Sun, Jul 02, 2017 at 01:16:15PM +, Christos Zoulas wrote:
> The solution is to implement "tickless kernel". It is not that difficult.

It looks like we are always descheduling the thread, not jut because we
got a clock tick. even a tickless kernel won't help stupid.

Re: Go on NetBSD needs love

[David Holland]

On Fri, Jun 30, 2017 at 07:36:41PM +0200, Joerg Sonnenberger wrote:
 > From the follow-up, it is far from clear that it is fixed completely.

Yes, I was wondering about that but people have been saying it works
on -8...

 > I'm running current with https://www.netbsd.org/~joerg/kern_event.c.diff
 > with the go 1.8.3 test suite in an infinite loop. Modulo the Perl 5.26.0
 > issue in vet_test and some occassional time outs from individual tests,
 > I don't see any issues.

any reason not to commit?

-- 
David A. Holland
dholl...@netbsd.org

Re: Go on NetBSD needs love

[David Holland]

On Fri, Jun 30, 2017 at 09:49:21AM -0700, Brad Fitzpatrick wrote:
 > > which leads to
 > >http://gnats.netbsd.org/50730
 > > which is fixed but the fix is apparently not in 7.1.
 > 
 > Where do you see that it's fixed?

That part requires some inside knowledge :-)  Christos committed a fix,
and the bug report state wasn't updated because Christos never does
that himself and occasionally nobody else gets to it.

Also since your original post there's been some discussion in chat.

 > I'm fumbling my way around these tools. Is there a link to the fix
 > somewhere?

Our tools suck I'm afraid :(

The commit referenced in the PR can be gotten here:

http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/kern_event.c.diff?r1=1.84=1.85=h
http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/sys/event.h.diff?r1=1.25=1.26=h

and also here:

https://mail-index.netbsd.org/source-changes/2016/01/31/msg072323.html

-- 
David A. Holland
dholl...@netbsd.org

Re: nanosleep() for shorted than schedule slice

[David Holland]

On Sun, Jul 02, 2017 at 12:54:52PM +0200, Joerg Sonnenberger wrote:
 > > I wonder if it would make sense for nanosleep(2) to check that requested
 > > sleeping time is shorter than a schedule slice, and if it is, spin the
 > > CPU instead of scheduling another process. Any opinion on this?
 > 
 > No, that's wrong. It's also been discussed before.

How is that wrong? It was always more or less the point of nanosleep.

-- 
David A. Holland
dholl...@netbsd.org

Re: nanosleep() for shorted than schedule slice

[Johnny Billquist]

On 2017-07-02 18:54, John Nemeth wrote:

On Jul 2,  1:16pm, Christos Zoulas wrote:
} In article <1n8j63y.1pcs0owrn6gcem%m...@netbsd.org>,
} Emmanuel Dreyfus  wrote:
} >
} >I just encountered a situation where PHP performance on NetBSD is rather
} >weak compared to Linux or MacOS X.
} >
} >The code calls PHP's uniqid() a lot of time. uniqid() creates an unique
} >id based on the clock. In order to avoid giving the same value for two
} >consecutive calls, PHP's uniqid() calls usleep(1) to skip to make sure
} >the current microsecond has changed.
} >
} >On NetBSD this turns into a 16 ms sleep, which is 16000 what was
} >requested. This happens because the kernel scheduled another process,
} >which is the behavior documented in the man page. However the result is
} >that a PHP script full of uniqid() is ridiculously slow.
} >
} >I worked around the problem by reimplementing PHP uniqid() using
} >uuidgen(), but that kind of performance problem could exist in many
} >other softwares.
} >
} >I wonder if it would make sense for nanosleep(2) to check that requested
} >sleeping time is shorter than a schedule slice, and if it is, spin the
} >CPU instead of scheduling another process. Any opinion on this?
}
} The solution is to implement "tickless kernel". It is not that difficult.

  The other option would be to tell PHP not to be so dumb.  What
happens on other OSes?  I find it hard to believe that we're the
only ones that aren't tickless.


One more reply in addition to my previous one. As I said, OSX for 
example, do use nanosleep.


But I just checked a little more in NetBSD, and it would seem that the 
man-page for nanosleep actually claims that our nanosleep is in fact 
using the normal clock ticks as well, giving nanosleep a resolution of 
(normally) 10ms. Which seems absurd. Are we really giving that poor 
resolution to nanosleep calls?
I would still say that this is not necessarily tied to a tickless 
implementation, even if I can see that there is a connection here. We 
should be able to deal with very high resolution clocks, even if we have 
a normal clock ticking at 100Hz.
But it might be easiest if we were to move to a tickless implementation 
in general, in order to deal with arbitrary clock times.


Johnny

--
Johnny Billquist  || "I'm on a bus
  ||  on a psychedelic trip
email: b...@softjar.se ||  Reading murder books
pdp is alive! ||  tryin' to stay hip" - B. Idol

Re: nanosleep() for shorted than schedule slice

[Johnny Billquist]

On 2017-07-02 18:54, John Nemeth wrote:

On Jul 2,  1:16pm, Christos Zoulas wrote:
} In article <1n8j63y.1pcs0owrn6gcem%m...@netbsd.org>,
} Emmanuel Dreyfus  wrote:
} >
} >I just encountered a situation where PHP performance on NetBSD is rather
} >weak compared to Linux or MacOS X.
} >
} >The code calls PHP's uniqid() a lot of time. uniqid() creates an unique
} >id based on the clock. In order to avoid giving the same value for two
} >consecutive calls, PHP's uniqid() calls usleep(1) to skip to make sure
} >the current microsecond has changed.
} >
} >On NetBSD this turns into a 16 ms sleep, which is 16000 what was
} >requested. This happens because the kernel scheduled another process,
} >which is the behavior documented in the man page. However the result is
} >that a PHP script full of uniqid() is ridiculously slow.
} >
} >I worked around the problem by reimplementing PHP uniqid() using
} >uuidgen(), but that kind of performance problem could exist in many
} >other softwares.
} >
} >I wonder if it would make sense for nanosleep(2) to check that requested
} >sleeping time is shorter than a schedule slice, and if it is, spin the
} >CPU instead of scheduling another process. Any opinion on this?
}
} The solution is to implement "tickless kernel". It is not that difficult.

  The other option would be to tell PHP not to be so dumb.  What
happens on other OSes?  I find it hard to believe that we're the
only ones that aren't tickless.


I don't get it. What was the problem with using nanosleep for short 
usleep's?


Also, how does tickless mode help?

This is all about the current implementation of usleep, which suspends 
the thread, and the time slice the scheduler uses between context 
switches. A tickless implementation will still split processing time 
into time slices between context switches, so you'd still get the same 
effect with a tickless kernel. You just wouldn't get the regular 
interrupt at each tick with tickless, since that is actually what 
tickless mode is about (unless I'm confused). With tickless mode, you 
instead keep track of time through other means than a regular clock 
interrupt, which might do nothing more than increment the time.


And I fail to see how it would be a problem implementing usleep using 
nanosleep for short time values. In fact, OSX as an example, uses 
nanosleep for all usleep calls.
You still have the caveat that suspension might be longer. The time 
argument actually just gives a lower bound. But OSX obviously uses 
nanosleep for this scenario.


(Yes, I might totally have missed some earlier discussion on this topic, 
and might be totally bonkers and clueless. In which case, feel free to 
educate me.)


Johnny

--
Johnny Billquist  || "I'm on a bus
  ||  on a psychedelic trip
email: b...@softjar.se ||  Reading murder books
pdp is alive! ||  tryin' to stay hip" - B. Idol

Re: nanosleep() for shorted than schedule slice

[John Nemeth]

On Jul 2,  1:16pm, Christos Zoulas wrote:
} In article <1n8j63y.1pcs0owrn6gcem%m...@netbsd.org>,
} Emmanuel Dreyfus  wrote:
} >
} >I just encountered a situation where PHP performance on NetBSD is rather
} >weak compared to Linux or MacOS X.
} >
} >The code calls PHP's uniqid() a lot of time. uniqid() creates an unique
} >id based on the clock. In order to avoid giving the same value for two
} >consecutive calls, PHP's uniqid() calls usleep(1) to skip to make sure
} >the current microsecond has changed.
} >
} >On NetBSD this turns into a 16 ms sleep, which is 16000 what was
} >requested. This happens because the kernel scheduled another process,
} >which is the behavior documented in the man page. However the result is
} >that a PHP script full of uniqid() is ridiculously slow. 
} >
} >I worked around the problem by reimplementing PHP uniqid() using
} >uuidgen(), but that kind of performance problem could exist in many
} >other softwares.
} >
} >I wonder if it would make sense for nanosleep(2) to check that requested
} >sleeping time is shorter than a schedule slice, and if it is, spin the
} >CPU instead of scheduling another process. Any opinion on this?
} 
} The solution is to implement "tickless kernel". It is not that difficult.

 The other option would be to tell PHP not to be so dumb.  What
happens on other OSes?  I find it hard to believe that we're the
only ones that aren't tickless.

}-- End of excerpt from Christos Zoulas

Re: nanosleep() for shorted than schedule slice

[Christos Zoulas]

On Jul 2,  3:31pm, jo...@bec.de (Joerg Sonnenberger) wrote:
-- Subject: Re: nanosleep() for shorted than schedule slice

| > The solution is to implement "tickless kernel". It is not that difficult.
| 
| Yes and no. The difficult hard is introducing it without creating
| noticable performance regressions. Just switching the call wheel to a RB
| tree for example gave a build.sh release regression of more than 1% when
| tried.

But it is a good start; there are workloads that will immediately benefit
from it, and it can be conditional until the performance issues are addressed.
The alternative is to bump HZ high enough and that can have worse performance.

christos

Re: nanosleep() for shorted than schedule slice

[Joerg Sonnenberger]

On Sun, Jul 02, 2017 at 01:16:15PM +, Christos Zoulas wrote:
> The solution is to implement "tickless kernel". It is not that difficult.

Yes and no. The difficult hard is introducing it without creating
noticable performance regressions. Just switching the call wheel to a RB
tree for example gave a build.sh release regression of more than 1% when
tried.

Joerg

Re: nanosleep() for shorted than schedule slice

[Christos Zoulas]

In article <1n8j63y.1pcs0owrn6gcem%m...@netbsd.org>,
Emmanuel Dreyfus  wrote:
>Hello
>
>I just encountered a situation where PHP performance on NetBSD is rather
>weak compared to Linux or MacOS X.
>
>The code calls PHP's uniqid() a lot of time. uniqid() creates an unique
>id based on the clock. In order to avoid giving the same value for two
>consecutive calls, PHP's uniqid() calls usleep(1) to skip to make sure
>the current microsecond has changed.
>
>On NetBSD this turns into a 16 ms sleep, which is 16000 what was
>requested. This happens because the kernel scheduled another process,
>which is the behavior documented in the man page. However the result is
>that a PHP script full of uniqid() is ridiculously slow. 
>
>I worked around the problem by reimplementing PHP uniqid() using
>uuidgen(), but that kind of performance problem could exist in many
>other softwares.
>
>I wonder if it would make sense for nanosleep(2) to check that requested
>sleeping time is shorter than a schedule slice, and if it is, spin the
>CPU instead of scheduling another process. Any opinion on this?

The solution is to implement "tickless kernel". It is not that difficult.

christos

Re: nanosleep() for shorted than schedule slice

[Joerg Sonnenberger]

On Sun, Jul 02, 2017 at 10:33:58AM +0200, Emmanuel Dreyfus wrote:
> I wonder if it would make sense for nanosleep(2) to check that requested
> sleeping time is shorter than a schedule slice, and if it is, spin the
> CPU instead of scheduling another process. Any opinion on this?

No, that's wrong. It's also been discussed before.

Joerg

nanosleep() for shorted than schedule slice

[Emmanuel Dreyfus]

Hello

I just encountered a situation where PHP performance on NetBSD is rather
weak compared to Linux or MacOS X.

The code calls PHP's uniqid() a lot of time. uniqid() creates an unique
id based on the clock. In order to avoid giving the same value for two
consecutive calls, PHP's uniqid() calls usleep(1) to skip to make sure
the current microsecond has changed.

On NetBSD this turns into a 16 ms sleep, which is 16000 what was
requested. This happens because the kernel scheduled another process,
which is the behavior documented in the man page. However the result is
that a PHP script full of uniqid() is ridiculously slow. 

I worked around the problem by reimplementing PHP uniqid() using
uuidgen(), but that kind of performance problem could exist in many
other softwares.

I wonder if it would make sense for nanosleep(2) to check that requested
sleeping time is shorter than a schedule slice, and if it is, spin the
CPU instead of scheduling another process. Any opinion on this?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
m...@netbsd.org