Re: TSC timekeeping and cpu states

[Ian Smith]

On Mon, 14 Aug 2017 09:48:07 -0700, Kevin Oberman wrote:
 > On Mon, Aug 14, 2017 at 8:38 AM, Ian Smith  wrote:
[..]
 > >  > > As far as possible TSC impact, I think older processors had TSC
 > >  > > issues when not all cores ran with the same clock speed. That said,
 > >  > > I am not remotely expert on such issues, so don't take this too
 > >  > > seriously.
 > >
 > > I wasn't aware that FreeBSD could yet do different freqs on different
 > > cores?  But I'm less expert than Kevin, and certainly behind the times.
[..]
 > I guess I need to clarify. No, FreeBSD does not have the ability to tun
 > different cores at different frequencies. I seem to recall that TCC on some
 > processors could adjust the frequency of a core exceeding a defined
 > temperature, skipping N of every 8 clock cycles to slow the processor and
 > reduce the temperature. This is what TCC was designed for. It is entirely
 > possible that I am not correctly remembering the details of the issue, but
 > it could only be resolved by switching from TCC to another clocking system.
 > 
 > If memory serves, and it may not, there was an issue a few years ago (jhb@
 > worked the issue) where TSC was varying with frequency and that caused
 > clock drift. I believe all "modern" processors do not have this issue and
 > it seems unlikely that any system running 24 cores is old enough that this
 > might be an issue.
 > 
 > Sorry for any confusion I may have caused.

Not at all.  It gave me an excuse to bug Alexander for some state-of-art 
details, to which he responded magnificently :)

Thankyou both, and Ari for entertaining such sport at his expense ..

cheers, Ian
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: TSC timekeeping and cpu states

[Kevin Oberman]

On Mon, Aug 14, 2017 at 8:38 AM, Ian Smith  wrote:

> On Mon, 14 Aug 2017 17:16:22 +1000, Aristedes Maniatis wrote:
>
>  > On 14/8/17 3:08PM, Kevin Oberman wrote:
>  > > Again, the documentation lags reality. The default was changed for
>  > > 11.0. It is still conservative. In ALMOST all cases, Cmax will yield
>  > > the bast results. However, on large systems with many cores, Cmax
>  > > will trigger very poor results, so the default is C2, just to be
>  > > safe.
>
> Given it's a server, anything beyond C2 is likely not worth trying.
> OTOH, C2 is perhaps not worth avoiding; it's probably low latency and
> should result in lower power consumption, so heat, and unlikely to hurt.
>
> Or at least, I suspect that's the case .. cc'ing Alexander, as the wiki
> article you referenced was his doing, so he's among those best placed.
>
>  > > As far as possible TSC impact, I think older processors had TSC
>  > > issues when not all cores ran with the same clock speed. That said,
>  > > I am not remotely expert on such issues, so don't take this too
>  > > seriously.
>
> I wasn't aware that FreeBSD could yet do different freqs on different
> cores?  But I'm less expert than Kevin, and certainly behind the times.
>
>  > Thanks Kevin
>  >
>  > What does 'large' and 'many cores' mean here? Is 24 cores large or
>  > small? For a server do we ever want the CPU to enter states other
>  > than C1?
>
> If C2 works well on your box, I don't see why you wouldn't want to use
> it .. but others might.  I have no personal experience beyond 2 cores,
> but I'm perennially curious about such issues, as Kevin knows :)
>
> Are you using powerd?  And what says, for example:
>
>  sysctl -a | egrep 'cx|available|_freq|freq_|choice' | grep -v net\.
>
> which should include your timecounter and eventtimer setup too.
>
> cheers, Ian
>

I guess I need to clarify. No, FreeBSD does not have the ability to tun
different cores at different frequencies. I seem to recall that TCC on some
processors could adjust the frequency of a core exceeding a defined
temperature, skipping N of every 8 clock cycles to slow the processor and
reduce the temperature. This is what TCC was designed for. It is entirely
possible that I am not correctly remembering the details of the issue, but
it could only be resolved by switching from TCC to another clocking system.

If memory serves, and it may not, there was an issue a few years ago (jhb@
worked the issue) where TSC was varying with frequency and that caused
clock drift. I believe all "modern" processors do not have this issue and
it seems unlikely that any system running 24 cores is old enough that this
might be an issue.

Sorry for any confusion I may have caused.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: TSC timekeeping and cpu states

[Alexander Motin]

On 14.08.2017 18:38, Ian Smith wrote:
> On Mon, 14 Aug 2017 17:16:22 +1000, Aristedes Maniatis wrote:
>  > On 14/8/17 3:08PM, Kevin Oberman wrote:
>  > > Again, the documentation lags reality. The default was changed for 
>  > > 11.0. It is still conservative. In ALMOST all cases, Cmax will yield 
>  > > the bast results. However, on large systems with many cores, Cmax 
>  > > will trigger very poor results, so the default is C2, just to be 
>  > > safe.
> 
> Given it's a server, anything beyond C2 is likely not worth trying. 
> OTOH, C2 is perhaps not worth avoiding; it's probably low latency and 
> should result in lower power consumption, so heat, and unlikely to hurt.
> 
> Or at least, I suspect that's the case .. cc'ing Alexander, as the wiki 
> article you referenced was his doing, so he's among those best placed.

C-states controlled here are ACPI C-states, which have limited relation
to real CPU C-states.  There are systems where they map exactly, but
there are also systems where ACPI C1/C2/C3 states map to CPU C1/C3/C6,
so it is difficult to make general recommendations.  Approximately the
map can be guessed looking on latency value (last of three) reported in
sysctl dev.cpu.0.cx_supported:  1 is usually CPU C1, 2+ is likely CPU
C2, 100+ can be C3, 500+ can be C6, but all that is very approximately
and I guess depends on BIOS writer mood.

What's about recommendations from me, I'd say that CPU C2 state should
not hurt in most cases, unless something is broken, but benefit is
rather small (often just covered by C1E enabled in BIOS);  CPU C3 state
gives significant power saving, but can either hurt performance due to
higher enter/exit latency or slightly improve it due to TurboBoost
activation (require CPU frequency to be set to max value); CPU C6 is
probably useful only for laptops, since it saves not so much power
power, while exit latency can be in milliseconds range.

>  > > As far as possible TSC impact, I think older processors had TSC
>  > > issues when not all cores ran with the same clock speed. That said,
>  > > I am not remotely expert on such issues, so don't take this too 
>  > > seriously.
> 
> I wasn't aware that FreeBSD could yet do different freqs on different 
> cores?  But I'm less expert than Kevin, and certainly behind the times.

On old CPUs TSC frequency was related to CPU frequency and so could
fluctuate with frequency change.  On modern CPUs it is always constant,
equal to base CPU frequency.  What's about different frequency for
different cores, IIRC ACPI allows that, but up to recent time neither
FreeBSD nor hardware could do that.  I have feeling I heard that some
very new CPUs may allow that, but to be efficient it would require very
tight interoperation between power manager and CPU scheduler, otherwise
performance may suffer.

-- 
Alexander Motin
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: TSC timekeeping and cpu states

[Ian Smith]

On Mon, 14 Aug 2017 17:16:22 +1000, Aristedes Maniatis wrote:

 > On 14/8/17 3:08PM, Kevin Oberman wrote:
 > > Again, the documentation lags reality. The default was changed for 
 > > 11.0. It is still conservative. In ALMOST all cases, Cmax will yield 
 > > the bast results. However, on large systems with many cores, Cmax 
 > > will trigger very poor results, so the default is C2, just to be 
 > > safe.

Given it's a server, anything beyond C2 is likely not worth trying. 
OTOH, C2 is perhaps not worth avoiding; it's probably low latency and 
should result in lower power consumption, so heat, and unlikely to hurt.

Or at least, I suspect that's the case .. cc'ing Alexander, as the wiki 
article you referenced was his doing, so he's among those best placed.

 > > As far as possible TSC impact, I think older processors had TSC
 > > issues when not all cores ran with the same clock speed. That said,
 > > I am not remotely expert on such issues, so don't take this too 
 > > seriously.

I wasn't aware that FreeBSD could yet do different freqs on different 
cores?  But I'm less expert than Kevin, and certainly behind the times.

 > Thanks Kevin
 > 
 > What does 'large' and 'many cores' mean here? Is 24 cores large or 
 > small? For a server do we ever want the CPU to enter states other 
 > than C1?

If C2 works well on your box, I don't see why you wouldn't want to use 
it .. but others might.  I have no personal experience beyond 2 cores, 
but I'm perennially curious about such issues, as Kevin knows :)

Are you using powerd?  And what says, for example:

 sysctl -a | egrep 'cx|available|_freq|freq_|choice' | grep -v net\.

which should include your timecounter and eventtimer setup too.

cheers, Ian
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: TSC timekeeping and cpu states

[Aristedes Maniatis]

On 14/8/17 3:08PM, Kevin Oberman wrote:
> Again, the documentation lags reality. The default was changed for 11.0. It 
> is still conservative. In ALMOST all cases, Cmax will yield the bast results. 
> However, on large systems with many cores, Cmax will trigger very poor 
> results, so the default is C2, just to be safe.
> 
> As far as possible TSC impact, I think older processors had TSC issues when 
> not all cores ran with the same clock speed. That said, I am not remotely 
> expert on such issues, so don't take this too seriously.

Thanks Kevin

What does 'large' and 'many cores' mean here? Is 24 cores large or small? For a 
server do we ever want the CPU to enter states other than C1?


Ari


-- 
-->
Aristedes Maniatis
CEO, ish
https://www.ish.com.au
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: TSC timekeeping and cpu states

[Kevin Oberman]

On Sun, Aug 13, 2017 at 8:25 PM, Aristedes Maniatis  wrote:

> I note that in FreeBSD 11, we now have this:
>
> # grep performance_cx_lowest /etc/defaults/rc.conf
> performance_cx_lowest="C2"  # Online CPU idle state
>
> However this wiki page suggests that C1 is the default
>
> https://wiki.freebsd.org/TuningPowerConsumption
>
>
> Are these inconsistent?
>
>
> I went looking for this because I've been having trouble with the TSC-low
> timecounter hardware choice and my system clock running at about 80% of
> normal speed. Moving to ACPI-fast solved this problem.
>
> Could the power saving CPU states be related to this problem, or should I
> look elsewhere for the TSC issue?
>
> This is a server, not a laptop.
>
>
> Thanks
> Ari
>
>
>
> --
> -->
> Aristedes Maniatis
> CEO, ish
> https://www.ish.com.au
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>

Again, the documentation lags reality. The default was changed for 11.0. It
is still conservative. In ALMOST all cases, Cmax will yield the bast
results. However, on large systems with many cores, Cmax will trigger very
poor results, so the default is C2, just to be safe.

As far as possible TSC impact, I think older processors had TSC issues when
not all cores ran with the same clock speed. That said, I am not remotely
expert on such issues, so don't take this too seriously.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

TSC timekeeping and cpu states

[Aristedes Maniatis]

I note that in FreeBSD 11, we now have this:

# grep performance_cx_lowest /etc/defaults/rc.conf
performance_cx_lowest="C2"  # Online CPU idle state

However this wiki page suggests that C1 is the default

https://wiki.freebsd.org/TuningPowerConsumption


Are these inconsistent?


I went looking for this because I've been having trouble with the TSC-low 
timecounter hardware choice and my system clock running at about 80% of normal 
speed. Moving to ACPI-fast solved this problem.

Could the power saving CPU states be related to this problem, or should I look 
elsewhere for the TSC issue?

This is a server, not a laptop.


Thanks
Ari



-- 
-->
Aristedes Maniatis
CEO, ish
https://www.ish.com.au
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"