Re: monotonic time going back by wrong skews

2021-04-30 Thread Scott Cheloha
> On Apr 30, 2021, at 15:53, Stuart Henderson wrote: > > On 2021/04/05 09:34, Scott Cheloha wrote: On Apr 5, 2021, at 09:07, Stuart Henderson wrote: >>> I've attached r620-E5_2630v2-2p6c2t.tgz, from Dell PE R620 with E5-2630v2. >>> This is a machine which has "disabling user TSC

Re: monotonic time going back by wrong skews

2021-04-30 Thread Stuart Henderson
On 2021/04/05 09:34, Scott Cheloha wrote: > > On Apr 5, 2021, at 09:07, Stuart Henderson wrote: > > > > I've attached r620-E5_2630v2-2p6c2t.tgz, from Dell PE R620 with E5-2630v2. > > This is a machine which has "disabling user TSC (skew=XXX)" reported for > > some cores: > > > > Nov 5

Re: monotonic time going back by wrong skews

2021-04-07 Thread Scott Cheloha
On Wed, Apr 07, 2021 at 06:18:40PM +0200, Mark Kettenis wrote: > > From: Scott Cheloha > > Date: Wed, 7 Apr 2021 10:25:04 -0500 > > > > > On Apr 6, 2021, at 07:49, Paul Irofti wrote: > > > > > The diff is obviously fine. But it is still a heuristic with no real > > motivation except

Re: monotonic time going back by wrong skews

2021-04-07 Thread Mark Kettenis
> From: Scott Cheloha > Date: Wed, 7 Apr 2021 10:25:04 -0500 > > > On Apr 6, 2021, at 07:49, Paul Irofti wrote: > > > The diff is obviously fine. But it is still a heuristic with no real > motivation except for this particular ESXi VM case. So my question > about why we choose

Re: monotonic time going back by wrong skews

2021-04-07 Thread Scott Cheloha
> On Apr 6, 2021, at 07:49, Paul Irofti wrote: > The diff is obviously fine. But it is still a heuristic with no real motivation except for this particular ESXi VM case. So my question about why we choose the minimum instead of the median or the mean has not been answered.

Re: monotonic time going back by wrong skews

2021-04-06 Thread Marcus MERIGHI
Good Evening! giova...@paclan.it (Giovanni Bechis), 2021.04.06 (Tue) 11:14 (CEST): > On 4/6/21 10:16 AM, YASUOKA Masahiko wrote: > > I'm sorry.. I send a wrong diff to the people. The result from > > giovanni@ and mcmer seems wrong. I suppose stu@ used the correct > > diff. > > giovanni and

Re: monotonic time going back by wrong skews

2021-04-06 Thread Paul Irofti
The diff is obviously fine. But it is still a heuristic with no real motivation except for this particular ESXi VM case. So my question about why we choose the minimum instead of the median or the mean has not been answered. Because median or mean is affected by outliers. We actually see some

Re: monotonic time going back by wrong skews

2021-04-06 Thread Giovanni Bechis
On 4/6/21 10:16 AM, YASUOKA Masahiko wrote: > Hi, > > I'm sorry.. I send a wrong diff to the people. The result from > giovanni@ and mcmer seems wrong. I suppose stu@ used the correct > diff. > > giovanni and mcmer, can you test with the correct diff again? > > I attached the correct diff at

Re: monotonic time going back by wrong skews

2021-04-06 Thread Stuart Henderson
On 2021/04/06 07:27, Marcus MERIGHI wrote: > Morning! > > scottchel...@gmail.com (Scott Cheloha), 2021.04.05 (Mon) 22:25 (CEST): > > To be clear, userland TSC is disabled on this machine, yes? And this > > is a physical server, not a VM? > > $ sysctl |grep tsc > kern.timecounter.hardware=tsc >

Re: monotonic time going back by wrong skews

2021-04-06 Thread YASUOKA Masahiko
Hi, I'm sorry.. I send a wrong diff to the people. The result from giovanni@ and mcmer seems wrong. I suppose stu@ used the correct diff. giovanni and mcmer, can you test with the correct diff again? I attached the correct diff at last of this mail. I'm sorry again. On Tue, 6 Apr 2021

Re: monotonic time going back by wrong skews

2021-04-06 Thread Giovanni Bechis
On Mon, Apr 05, 2021 at 07:14:49PM +0900, YASUOKA Masahiko wrote: > Hi, > > > Another issue that I see is that people have not reported, at least [...] > > publicly, that this runs fine on their normal OpenBSD machines. > > Some dmesgs posted on public lists seems to have the same problem. > >

Re: monotonic time going back by wrong skews

2021-04-05 Thread Marcus MERIGHI
Morning! scottchel...@gmail.com (Scott Cheloha), 2021.04.05 (Mon) 22:25 (CEST): > To be clear, userland TSC is disabled on this machine, yes? And this > is a physical server, not a VM? $ sysctl |grep tsc kern.timecounter.hardware=tsc Do you want me to disable this? yasuoka@ found a dmesg with

Re: monotonic time going back by wrong skews

2021-04-05 Thread Scott Cheloha
On Mon, Apr 05, 2021 at 09:29:10PM +0200, Marcus MERIGHI wrote: > Hello! > > scottchel...@gmail.com (Scott Cheloha), 2021.03.25 (Thu) 23:28 (CET): > > Feel free to share your raw data. > > I think I followed the instructions; is it expected to get a lot of > > pstat: cannot read 0x9704:

Re: monotonic time going back by wrong skews

2021-04-05 Thread Marcus MERIGHI
Hello! scottchel...@gmail.com (Scott Cheloha), 2021.03.25 (Thu) 23:28 (CET): > Feel free to share your raw data. I think I followed the instructions; is it expected to get a lot of pstat: cannot read 0x9704: invalid address (9704) and that the script takes about six minutes to run?

Re: monotonic time going back by wrong skews

2021-04-05 Thread Theo de Raadt
Mark Kettenis wrote: > Before you go further down this path, please realize that "fixing" the > TSC this way is racy in much the same ways as determining the skew is. > If an SMM or a VM exit happens between reading the TSC and the actual > write to the MSR, the new value of the TSC will be off

Re: monotonic time going back by wrong skews

2021-04-05 Thread Mark Kettenis
> Date: Sat, 3 Apr 2021 22:21:02 -0500 > From: Scott Cheloha > > On Fri, Apr 02, 2021 at 10:37:36AM -0700, Mike Larkin wrote: > > On Thu, Apr 01, 2021 at 06:43:30PM -0500, Scott Cheloha wrote: > > > > > > [...] > > > > > > Hmmm. Being able to work around this would be nice. > > > > > >

Re: monotonic time going back by wrong skews

2021-04-05 Thread Mike Larkin
On Sat, Apr 03, 2021 at 10:21:02PM -0500, Scott Cheloha wrote: > On Fri, Apr 02, 2021 at 10:37:36AM -0700, Mike Larkin wrote: > > On Thu, Apr 01, 2021 at 06:43:30PM -0500, Scott Cheloha wrote: > > > > > > [...] > > > > > > Hmmm. Being able to work around this would be nice. > > > > > > FreeBSD

Re: monotonic time going back by wrong skews

2021-04-05 Thread Scott Cheloha
> On Apr 5, 2021, at 09:07, Stuart Henderson wrote: > > I've attached r620-E5_2630v2-2p6c2t.tgz, from Dell PE R620 with E5-2630v2. > This is a machine which has "disabling user TSC (skew=XXX)" reported for > some cores: > > Nov 5 16:08:34 pweb /bsd: cpu11: disabling user TSC (skew=-107) > Nov

Re: monotonic time going back by wrong skews

2021-04-05 Thread Stuart Henderson
I've attached r620-E5_2630v2-2p6c2t.tgz, from Dell PE R620 with E5-2630v2. This is a machine which has "disabling user TSC (skew=XXX)" reported for some cores: Nov 5 16:08:34 pweb /bsd: cpu11: disabling user TSC (skew=-107) Nov 5 16:08:34 pweb /bsd: cpu13: disabling user TSC (skew=-101) 2 SMT

Re: monotonic time going back by wrong skews

2021-04-05 Thread YASUOKA Masahiko
On Mon, 5 Apr 2021 14:24:03 +0200 (CEST) Mark Kettenis wrote: >> Date: Mon, 05 Apr 2021 19:14:49 +0900 (JST) >> From: YASUOKA Masahiko >> >> Hi, >> >> On Mon, 5 Apr 2021 10:43:00 +0300 >> Paul Irofti wrote: >> > On 05.04.2021 06:13, Scott Cheloha wrote: >> >> On Mon, Mar 29, 2021 at

Re: monotonic time going back by wrong skews

2021-04-05 Thread Mark Kettenis
> Date: Mon, 05 Apr 2021 19:14:49 +0900 (JST) > From: YASUOKA Masahiko > > Hi, > > On Mon, 5 Apr 2021 10:43:00 +0300 > Paul Irofti wrote: > > On 05.04.2021 06:13, Scott Cheloha wrote: > >> On Mon, Mar 29, 2021 at 02:00:01PM +0900, YASUOKA Masahiko wrote: > >>> On Thu, 25 Mar 2021 19:41:35

Re: monotonic time going back by wrong skews

2021-04-05 Thread YASUOKA Masahiko
Hi, On Mon, 5 Apr 2021 10:43:00 +0300 Paul Irofti wrote: > On 05.04.2021 06:13, Scott Cheloha wrote: >> On Mon, Mar 29, 2021 at 02:00:01PM +0900, YASUOKA Masahiko wrote: >>> On Thu, 25 Mar 2021 19:41:35 +0100 (CET) >>> Mark Kettenis wrote: > From: Scott Cheloha > Date: Thu, 25 Mar 2021

Re: monotonic time going back by wrong skews

2021-04-05 Thread Paul Irofti
On 05.04.2021 06:13, Scott Cheloha wrote: On Mon, Mar 29, 2021 at 02:00:01PM +0900, YASUOKA Masahiko wrote: On Thu, 25 Mar 2021 19:41:35 +0100 (CET) Mark Kettenis wrote: From: Scott Cheloha Date: Thu, 25 Mar 2021 13:18:04 -0500 On Wed, Mar 24, 2021 at 05:40:21PM +0900, YASUOKA Masahiko

Re: monotonic time going back by wrong skews

2021-04-04 Thread Scott Cheloha
On Mon, Mar 29, 2021 at 02:00:01PM +0900, YASUOKA Masahiko wrote: > On Thu, 25 Mar 2021 19:41:35 +0100 (CET) > Mark Kettenis wrote: > >> From: Scott Cheloha > >> Date: Thu, 25 Mar 2021 13:18:04 -0500 > >> > On Wed, Mar 24, 2021 at 05:40:21PM +0900, YASUOKA Masahiko wrote: > >> Which diff did you

Re: monotonic time going back by wrong skews

2021-04-03 Thread Scott Cheloha
On Fri, Apr 02, 2021 at 10:37:36AM -0700, Mike Larkin wrote: > On Thu, Apr 01, 2021 at 06:43:30PM -0500, Scott Cheloha wrote: > > > > [...] > > > > Hmmm. Being able to work around this would be nice. > > > > FreeBSD has code that uses WRMSR to synchronize the TSC: > > > >

Re: monotonic time going back by wrong skews

2021-04-02 Thread Mike Larkin
On Thu, Apr 01, 2021 at 06:43:30PM -0500, Scott Cheloha wrote: > On Thu, Apr 01, 2021 at 03:41:24PM -0400, Josh Rickmar wrote: > > On Thu, Apr 01, 2021 at 03:22:00PM -0400, Josh Rickmar wrote: > > > On Thu, Apr 01, 2021 at 02:15:48PM -0500, Scott Cheloha wrote: > > > > On Sat, Mar 27, 2021 at

Re: monotonic time going back by wrong skews

2021-04-01 Thread Josh Rickmar
On Thu, Apr 01, 2021 at 06:43:30PM -0500, Scott Cheloha wrote: > On Thu, Apr 01, 2021 at 03:41:24PM -0400, Josh Rickmar wrote: > > On Thu, Apr 01, 2021 at 03:22:00PM -0400, Josh Rickmar wrote: > > > On Thu, Apr 01, 2021 at 02:15:48PM -0500, Scott Cheloha wrote: > > > > On Sat, Mar 27, 2021 at

Re: monotonic time going back by wrong skews

2021-04-01 Thread Scott Cheloha
On Thu, Apr 01, 2021 at 03:41:24PM -0400, Josh Rickmar wrote: > On Thu, Apr 01, 2021 at 03:22:00PM -0400, Josh Rickmar wrote: > > On Thu, Apr 01, 2021 at 02:15:48PM -0500, Scott Cheloha wrote: > > > On Sat, Mar 27, 2021 at 02:20:21AM +, Stefmorino wrote: > > > > > Feel free to share your raw

Re: monotonic time going back by wrong skews

2021-04-01 Thread Josh Rickmar
On Thu, Apr 01, 2021 at 03:22:00PM -0400, Josh Rickmar wrote: > On Thu, Apr 01, 2021 at 02:15:48PM -0500, Scott Cheloha wrote: > > On Sat, Mar 27, 2021 at 02:20:21AM +, Stefmorino wrote: > > > > Feel free to share your raw data. > > > > > > Also includes some standard sendbug dumps:

Re: monotonic time going back by wrong skews

2021-04-01 Thread Josh Rickmar
On Thu, Apr 01, 2021 at 02:15:48PM -0500, Scott Cheloha wrote: > On Sat, Mar 27, 2021 at 02:20:21AM +, Stefmorino wrote: > > > Feel free to share your raw data. > > > > Also includes some standard sendbug dumps: https://0x0.st/-qng.tgz > > Thanks! > > TL;DR: > > Two things: > > 1. Could

Re: monotonic time going back by wrong skews

2021-04-01 Thread Scott Cheloha
On Sat, Mar 27, 2021 at 02:20:21AM +, Stefmorino wrote: > > Feel free to share your raw data. > > Also includes some standard sendbug dumps: https://0x0.st/-qng.tgz Thanks! TL;DR: Two things: 1. Could you check whether Linux will use the TSC as a clocksource on this machine? The dmesg

Re: monotonic time going back by wrong skews

2021-03-28 Thread YASUOKA Masahiko
On Thu, 25 Mar 2021 19:41:35 +0100 (CET) Mark Kettenis wrote: >> From: Scott Cheloha >> Date: Thu, 25 Mar 2021 13:18:04 -0500 >> > On Wed, Mar 24, 2021 at 05:40:21PM +0900, YASUOKA Masahiko wrote: >> Which diff did you apply? Yasuoka provided two diffs. >> >> In any case, ignore this diff: >>

Re: monotonic time going back by wrong skews

2021-03-27 Thread Stefmorino
> Feel free to share your raw data. Also includes some standard sendbug dumps: https://0x0.st/-qng.tgz

Re: monotonic time going back by wrong skews

2021-03-25 Thread Scott Cheloha
On Thu, Mar 25, 2021 at 07:24:23PM -0400, Josh Rickmar wrote: > On Thu, Mar 25, 2021 at 05:28:54PM -0500, Scott Cheloha wrote: > > Feel free to share your raw data. > > Attached. Hmmm, interesting stuff. $ ministat -q cpu* x cpu1-skew + cpu2-skew * cpu3-skew % cpu4-skew #

Re: monotonic time going back by wrong skews

2021-03-25 Thread Josh Rickmar
On Thu, Mar 25, 2021 at 05:28:54PM -0500, Scott Cheloha wrote: > Feel free to share your raw data. Attached. e485skews.tgz Description: application/tar-gz

Re: monotonic time going back by wrong skews

2021-03-25 Thread Scott Cheloha
On Thu, Mar 25, 2021 at 02:33:43PM -0400, Josh Rickmar wrote: > On Thu, Mar 25, 2021 at 01:18:04PM -0500, Scott Cheloha wrote: > > > On Mar 24, 2021, at 8:29 AM, Josh Rickmar wrote: > > > > > > [...] > > > > Which diff did you apply? Yasuoka provided two diffs. > > > > In any case, ignore

Re: monotonic time going back by wrong skews

2021-03-25 Thread Mark Kettenis
> From: Scott Cheloha > Date: Thu, 25 Mar 2021 13:18:04 -0500 > > > On Mar 24, 2021, at 8:29 AM, Josh Rickmar wrote: > > > > On Wed, Mar 24, 2021 at 05:40:21PM +0900, YASUOKA Masahiko wrote: > >> Hi, > >> > >> I hit a problem which is caused by going back of monotonic time. It > >> happens

Re: monotonic time going back by wrong skews

2021-03-25 Thread Josh Rickmar
On Thu, Mar 25, 2021 at 01:18:04PM -0500, Scott Cheloha wrote: > > On Mar 24, 2021, at 8:29 AM, Josh Rickmar wrote: > > > > On Wed, Mar 24, 2021 at 05:40:21PM +0900, YASUOKA Masahiko wrote: > >> Hi, > >> > >> I hit a problem which is caused by going back of monotonic time. It > >> happens on

Re: monotonic time going back by wrong skews

2021-03-25 Thread Scott Cheloha
> On Mar 24, 2021, at 8:29 AM, Josh Rickmar wrote: > > On Wed, Mar 24, 2021 at 05:40:21PM +0900, YASUOKA Masahiko wrote: >> Hi, >> >> I hit a problem which is caused by going back of monotonic time. It >> happens on hosts on VMware ESXi. >> >> I wrote the program which repeats the problem. >>

Re: monotonic time going back by wrong skews

2021-03-24 Thread Josh Rickmar
On Wed, Mar 24, 2021 at 05:40:21PM +0900, YASUOKA Masahiko wrote: > Hi, > > I hit a problem which is caused by going back of monotonic time. It > happens on hosts on VMware ESXi. > > I wrote the program which repeats the problem. > > % cc -o monotime monotime.c -lpthread > % ./monotime >

Re: monotonic time going back by wrong skews

2021-03-24 Thread YASUOKA Masahiko
Hi, > Second, why is taking the minimum value the optimal choice? I would > assume an average would be better. Basically if you have a sequency > like 900, 900, 900, 900, 0, 900, 900, 900 you pick 0 which could lead > to some problems, right? Or am I missing something?" Skews on VMware >> -8445

Re: monotonic time going back by wrong skews

2021-03-24 Thread Paul Irofti
Hi, Thank you for taking this to tech@ as requested! I will reproduce here what I replied to Yasouka and Scott (which I think proposed taking the minimum skew value) in private. "First, thank you very much for the in-depth analysis. I would suggest you take this to a public forum like tech@