Re: [v1 0/9] Early boot time stamps for x86
On Thu, 23 Mar 2017, Pasha Tatashin wrote: > I will add a condition to tsc_early_init() to check for TSC_ADJUST if it is > not 0, disable early TSC feature. Does this sound OK? Not really. I have strong objections against how this is crammed into the early boot process along with the code duplication and the extra magic which is caused by this. The early boot process is fragile enough, so we really only want to have code there which is absolutely required. That timestamp feature does not qualify for that at all. I need some quiet time to look into that, so please don't waste too much time on refactoring that patch set. Thanks, tglx
Re: [v1 0/9] Early boot time stamps for x86
On Thu, 23 Mar 2017, Pasha Tatashin wrote: > I will add a condition to tsc_early_init() to check for TSC_ADJUST if it is > not 0, disable early TSC feature. Does this sound OK? Not really. I have strong objections against how this is crammed into the early boot process along with the code duplication and the extra magic which is caused by this. The early boot process is fragile enough, so we really only want to have code there which is absolutely required. That timestamp feature does not qualify for that at all. I need some quiet time to look into that, so please don't waste too much time on refactoring that patch set. Thanks, tglx
Re: [v1 0/9] Early boot time stamps for x86
Hi Thomas, Thank you very much for looking at this patchset. Comments below: On 03/23/2017 06:56 AM, Thomas Gleixner wrote: On Wed, 22 Mar 2017, Pasha Tatashin wrote: Yes, I am certain it is 0 or near 0 on reset on this machine. Because, I Emphasis on "this machine' It's not guaranteed especially not on reboot and not with creative BIOSes fiddling with the TSC_ADJUST value. - It CANNOT be used to measure BIOS boot time reliably Yes, understood, I will remove comment about BIOS time from the next cover letter. However, I think the pr_info() with offset is still useful at least for those whose BIOS does not alter TSC_ADJUST, also it is consisten with every other clocksource in linux where offset is printed in pr_info(). From Intel PRM 2016/12: The time-stamp counter (as implemented in the P6 family, Pentium, Pentium M, Pentium 4, Intel Xeon, Intel Core Solo and Intel Core Duo processors and later processors) is a 64-bit counter that is set to 0 following a RESET of the processor Since early boot time stamps feature target processors that are later than "Pentium 4" because invariant TSC flag is checked, it is safe to assume that offset is going to be valid on power-on if TSC_ADJUST was not altered - If BIOS wreckaged TSC_ADJUST, then your whole time stamping goes out the window once the kernel sanitized it. I will add a condition to tsc_early_init() to check for TSC_ADJUST if it is not 0, disable early TSC feature. Does this sound OK? Thank you, Pasha
Re: [v1 0/9] Early boot time stamps for x86
Hi Thomas, Thank you very much for looking at this patchset. Comments below: On 03/23/2017 06:56 AM, Thomas Gleixner wrote: On Wed, 22 Mar 2017, Pasha Tatashin wrote: Yes, I am certain it is 0 or near 0 on reset on this machine. Because, I Emphasis on "this machine' It's not guaranteed especially not on reboot and not with creative BIOSes fiddling with the TSC_ADJUST value. - It CANNOT be used to measure BIOS boot time reliably Yes, understood, I will remove comment about BIOS time from the next cover letter. However, I think the pr_info() with offset is still useful at least for those whose BIOS does not alter TSC_ADJUST, also it is consisten with every other clocksource in linux where offset is printed in pr_info(). From Intel PRM 2016/12: The time-stamp counter (as implemented in the P6 family, Pentium, Pentium M, Pentium 4, Intel Xeon, Intel Core Solo and Intel Core Duo processors and later processors) is a 64-bit counter that is set to 0 following a RESET of the processor Since early boot time stamps feature target processors that are later than "Pentium 4" because invariant TSC flag is checked, it is safe to assume that offset is going to be valid on power-on if TSC_ADJUST was not altered - If BIOS wreckaged TSC_ADJUST, then your whole time stamping goes out the window once the kernel sanitized it. I will add a condition to tsc_early_init() to check for TSC_ADJUST if it is not 0, disable early TSC feature. Does this sound OK? Thank you, Pasha
Re: [v1 0/9] Early boot time stamps for x86
On Wed, 22 Mar 2017, Pasha Tatashin wrote: > Yes, I am certain it is 0 or near 0 on reset on this machine. Because, I Emphasis on "this machine' It's not guaranteed especially not on reboot and not with creative BIOSes fiddling with the TSC_ADJUST value. - It CANNOT be used to measure BIOS boot time reliably - If BIOS wreckaged TSC_ADJUST, then your whole time stamping goes out the window once the kernel sanitized it. See other mail. Thanks, tglx
Re: [v1 0/9] Early boot time stamps for x86
On Wed, 22 Mar 2017, Pasha Tatashin wrote: > Yes, I am certain it is 0 or near 0 on reset on this machine. Because, I Emphasis on "this machine' It's not guaranteed especially not on reboot and not with creative BIOSes fiddling with the TSC_ADJUST value. - It CANNOT be used to measure BIOS boot time reliably - If BIOS wreckaged TSC_ADJUST, then your whole time stamping goes out the window once the kernel sanitized it. See other mail. Thanks, tglx
Re: [v1 0/9] Early boot time stamps for x86
On 2017-03-22 16:27, Peter Zijlstra wrote: On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote: Last week I sent out patches to enable early boot time stamp on SPARC, that work is now under review: http://www.spinics.net/lists/sparclinux/msg17372.html This is continuation for that work, but adding early boot time stamps support for x86 machines. *groan*, so we get an even bigger trainwreck when TSC is fscked? Hi Peter, I am not sure what you mean. If tsc is not reliable, the worst case is that during early boot the timestamps won't be as accurate as the timestamps that are enabled once we initialize some other clock sources. In either way, if you look at the way I enabled early timestamps, it is really narrowed down to work only on modern hardware with stable tsc. (Nehalem and later) Thank you, Pasha
Re: [v1 0/9] Early boot time stamps for x86
On 2017-03-22 16:27, Peter Zijlstra wrote: On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote: Last week I sent out patches to enable early boot time stamp on SPARC, that work is now under review: http://www.spinics.net/lists/sparclinux/msg17372.html This is continuation for that work, but adding early boot time stamps support for x86 machines. *groan*, so we get an even bigger trainwreck when TSC is fscked? Hi Peter, I am not sure what you mean. If tsc is not reliable, the worst case is that during early boot the timestamps won't be as accurate as the timestamps that are enabled once we initialize some other clock sources. In either way, if you look at the way I enabled early timestamps, it is really narrowed down to work only on modern hardware with stable tsc. (Nehalem and later) Thank you, Pasha
Re: [v1 0/9] Early boot time stamps for x86
Hi Peter, Thank you for looking at this patchset. Yes, I am certain it is 0 or near 0 on reset on this machine. Because, I actually wondered about it, and used stop watch as an alternative way to verify the result, twice. While, I suspect it is often the case that on reset tsc is 0, it is not really import, as the offset value is not the important part of the project. The important part is that we can cover the whole Linux boot process with timestamps, and know where spend time, and avoid regressions in the future. Thank you, Pasha On 2017-03-22 16:28, Peter Zijlstra wrote: On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote: - early tsc offset is 549s, so it took over 9 minutes to get through POST, and GRUB before starting linux Lol, how cute. You assume TSC starts at 0 on reset.
Re: [v1 0/9] Early boot time stamps for x86
Hi Peter, Thank you for looking at this patchset. Yes, I am certain it is 0 or near 0 on reset on this machine. Because, I actually wondered about it, and used stop watch as an alternative way to verify the result, twice. While, I suspect it is often the case that on reset tsc is 0, it is not really import, as the offset value is not the important part of the project. The important part is that we can cover the whole Linux boot process with timestamps, and know where spend time, and avoid regressions in the future. Thank you, Pasha On 2017-03-22 16:28, Peter Zijlstra wrote: On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote: - early tsc offset is 549s, so it took over 9 minutes to get through POST, and GRUB before starting linux Lol, how cute. You assume TSC starts at 0 on reset.
Re: [v1 0/9] Early boot time stamps for x86
On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote: > Last week I sent out patches to enable early boot time stamp on SPARC, that > work is now under review: > http://www.spinics.net/lists/sparclinux/msg17372.html > > This is continuation for that work, but adding early boot time stamps > support for x86 machines. *groan*, so we get an even bigger trainwreck when TSC is fscked?
Re: [v1 0/9] Early boot time stamps for x86
On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote: > Last week I sent out patches to enable early boot time stamp on SPARC, that > work is now under review: > http://www.spinics.net/lists/sparclinux/msg17372.html > > This is continuation for that work, but adding early boot time stamps > support for x86 machines. *groan*, so we get an even bigger trainwreck when TSC is fscked?
Re: [v1 0/9] Early boot time stamps for x86
On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote: > - early tsc offset is 549s, so it took over 9 minutes to get through POST, > and GRUB before starting linux Lol, how cute. You assume TSC starts at 0 on reset.
Re: [v1 0/9] Early boot time stamps for x86
On Wed, Mar 22, 2017 at 04:24:16PM -0400, Pavel Tatashin wrote: > - early tsc offset is 549s, so it took over 9 minutes to get through POST, > and GRUB before starting linux Lol, how cute. You assume TSC starts at 0 on reset.
[v1 0/9] Early boot time stamps for x86
Last week I sent out patches to enable early boot time stamp on SPARC, that work is now under review: http://www.spinics.net/lists/sparclinux/msg17372.html This is continuation for that work, but adding early boot time stamps support for x86 machines. Here is example how this information is useful: Before: https://hastebin.com/zofiqazuze.scala After: https://hastebin.com/otayoliruc.scala If you take a look at the before case, it seems that this particular x86 machine took only 2 minutes to boot. Which, while can be improved, is not too bad considering that this machine has 192CPUs and 2.2T of memory. But, as it can be seen in the fix case, the time is much longer: - early boot time stamps account for another 80s! - early tsc offset is 549s, so it took over 9 minutes to get through POST, and GRUB before starting linux Now, the total boot time is 12m52s, which is really slow. Pavel Tatashin (9): sched/clock: broken stable to unstable transfer sched/clock: interface to allow timestamps early in boot x86/cpu: determining x86 vendor early x86/tsc: early MSR-based CPU/TSC frequency discovery x86/tsc: disable early messages from quick_pit_calibrate x86/tsc: use cpuid to determine TSC frequency x86/tsc: use cpuid to determine CPU frequency x86/tsc: tsc early x86/tsc: use tsc early arch/x86/include/asm/processor.h |1 + arch/x86/include/asm/tsc.h |5 + arch/x86/kernel/cpu/common.c | 36 arch/x86/kernel/head64.c |1 + arch/x86/kernel/time.c |1 + arch/x86/kernel/tsc.c| 166 +- arch/x86/kernel/tsc_msr.c| 38 ++--- include/linux/sched/clock.h |4 + kernel/sched/clock.c | 70 +++- 9 files changed, 284 insertions(+), 38 deletions(-)
[v1 0/9] Early boot time stamps for x86
Last week I sent out patches to enable early boot time stamp on SPARC, that work is now under review: http://www.spinics.net/lists/sparclinux/msg17372.html This is continuation for that work, but adding early boot time stamps support for x86 machines. Here is example how this information is useful: Before: https://hastebin.com/zofiqazuze.scala After: https://hastebin.com/otayoliruc.scala If you take a look at the before case, it seems that this particular x86 machine took only 2 minutes to boot. Which, while can be improved, is not too bad considering that this machine has 192CPUs and 2.2T of memory. But, as it can be seen in the fix case, the time is much longer: - early boot time stamps account for another 80s! - early tsc offset is 549s, so it took over 9 minutes to get through POST, and GRUB before starting linux Now, the total boot time is 12m52s, which is really slow. Pavel Tatashin (9): sched/clock: broken stable to unstable transfer sched/clock: interface to allow timestamps early in boot x86/cpu: determining x86 vendor early x86/tsc: early MSR-based CPU/TSC frequency discovery x86/tsc: disable early messages from quick_pit_calibrate x86/tsc: use cpuid to determine TSC frequency x86/tsc: use cpuid to determine CPU frequency x86/tsc: tsc early x86/tsc: use tsc early arch/x86/include/asm/processor.h |1 + arch/x86/include/asm/tsc.h |5 + arch/x86/kernel/cpu/common.c | 36 arch/x86/kernel/head64.c |1 + arch/x86/kernel/time.c |1 + arch/x86/kernel/tsc.c| 166 +- arch/x86/kernel/tsc_msr.c| 38 ++--- include/linux/sched/clock.h |4 + kernel/sched/clock.c | 70 +++- 9 files changed, 284 insertions(+), 38 deletions(-)
