Re: svn commit: r335753 - head/sbin/devd [ broke the ci.freebsd.org FreeBSD-head-{mips,mip64,powerpcpowerpc64,powerpcspe,sparc64}-build ]

[Mark Millard]

[I used the wrong Email address the first time.]

On 2018-Jun-27, at 6:31 PM, Warner Losh  wrote:
> 
> 
> 
> On Wed, Jun 27, 2018, 7:27 PM Mark Millard  wrote:
> These are the gcc/g++ 4.2.1 based targets.
> 
> For example . . .
> 
> https://ci.freebsd.org/job/FreeBSD-head-powerpc64-build/6261/consoleText
> 
> --- all_subdir_sbin/devd ---
> /usr/src/sbin/devd/devd.cc: In member function 'std::string 
> config::shell_quote(const std::string&)':
> /usr/src/sbin/devd/devd.cc:652: error: a function-definition is not allowed 
> here before ':' token
> /usr/src/sbin/devd/devd.cc:1327: error: expected primary-expression at end of 
> input
> /usr/src/sbin/devd/devd.cc:1327: error: expected `;' at end of input
> /usr/src/sbin/devd/devd.cc:1327: error: expected primary-expression at end of 
> input
> /usr/src/sbin/devd/devd.cc:1327: error: expected `)' at end of input
> /usr/src/sbin/devd/devd.cc:1327: error: expected statement at end of input
> /usr/src/sbin/devd/devd.cc:1327: error: expected `}' at end of input
> --- all_subdir_kerberos5 ---
> 
> 
> The following is modern C++ syntax that is being rejected:
> 
> . . .
> for (const char &c : s) {
> . . .
> 
> (At least if I understand right.)
> 
> C++11 I thought was required. If not. That the issue. 

Looking at "C++11 Support in GCC" in:

https://gcc.gnu.org/projects/cxx-status.html

shows almost nothing is supported prior to
gcc 4.3 . Some things require gcc 4.8.1 .

As for "Range-based for": it requires gcc 4.6 .
The SD-6 feature test listed is:

__cpp_range_based_for >= 200907


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r335753 - head/sbin/devd [ broke the ci.freebsd.org FreeBSD-head-{mips,mip64,powerpcpowerpc64,powerpcspe,sparc64}-build ]

[Warner Losh]

On Wed, Jun 27, 2018, 7:27 PM Mark Millard  wrote:

> These are the gcc/g++ 4.2.1 based targets.
>
> For example . . .
>
> https://ci.freebsd.org/job/FreeBSD-head-powerpc64-build/6261/consoleText
>
> --- all_subdir_sbin/devd ---
> /usr/src/sbin/devd/devd.cc: In member function 'std::string
> config::shell_quote(const std::string&)':
> /usr/src/sbin/devd/devd.cc:652: error: a function-definition is not
> allowed here before ':' token
> /usr/src/sbin/devd/devd.cc:1327: error: expected primary-expression at end
> of input
> /usr/src/sbin/devd/devd.cc:1327: error: expected `;' at end of input
> /usr/src/sbin/devd/devd.cc:1327: error: expected primary-expression at end
> of input
> /usr/src/sbin/devd/devd.cc:1327: error: expected `)' at end of input
> /usr/src/sbin/devd/devd.cc:1327: error: expected statement at end of input
> /usr/src/sbin/devd/devd.cc:1327: error: expected `}' at end of input
> --- all_subdir_kerberos5 ---
>
>
> The following is modern C++ syntax that is being rejected:
>
> . . .
> for (const char &c : s) {
> . . .
>
> (At least if I understand right.)
>

C++11 I thought was required. If not. That the issue.

Warner


> ===
> Mark Millard
> marklmi at yahoo.com
> ( dsl-only.net went
> away in early 2018-Mar)
>
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Build updates [ ci.freebsd.org FreeBSD-head-{aarch64, armv7, armv6}-build failures as of, for example -r335711 and -r335713 ]

[Bryan Drewery]

On 6/27/2018 11:58 AM, Bryan Drewery wrote:
> On 6/27/2018 11:44 AM, Bryan Drewery wrote:
>> On 6/27/2018 10:53 AM, Mark Millard wrote:
>>>
 On 2018-Jun-27, at 10:01 AM, Bryan Drewery  wrote:

 As of r335704:

 - make tinderbox/universe will now build the bootstrap clang *once*.
 Each target clang is still build of course.  This support does not work
 with gcc.
 - make buildworld (kernel-toolchain and toolchain) will build the
 bootstrap clang (if needed per SYSTEM_COMPILER logic) with only the
 TARGET.TARGET_ARCH backend support. The installed clang has all still so
 SYSTEM_COMPILER logic works for cross-compling.

 This uses the feature dim@ added in r335558 to selectively disable LLVM
 targets. I've added a new option named WITH[OUT]_LLVM_TARGET_ALL which I
 suggest using rather than the per-arch options. It is default on (WITH).
 Set WITHOUT to only build the needed native arch on your system for both
 bootstrap and compiled clang. Setting WITHOUT disables SYSTEM_COMPILER
 support for cross-builds.

 Please CC me directly for any weird tinderbox/universe or clang failures
 for the next few weeks.
>>
>> Thanks!
>>
>>>
>>> https://ci.freebsd.org/job/FreeBSD-head-aarch64-build/8324/consoleText
>>>
>>> --- all_subdir_cloudabi32 ---
>>> clang (LLVM option parsing): Unknown command line argument 
>>> '-arm-add-build-attributes'.  Try: 'clang (LLVM option parsing) -help'
>>> clang (LLVM option parsing): Did you mean '-force-attribute'?
>>> *** [cloudabi32_vdso.o] Error code 1
>>>
>>
>> This was an aarch64 build. It looks like -arm-add-build-attributes is
>> from Target/ARM/AsmParser/ARMAsmParser.cpp which is only built for
>> LLVM_TARGET_ARM but not LLVM_TARGET_AARCH64.
>>
>> Looking in contrib/llvm/tools/clang/lib/Driver/ToolChains/Clang.cpp I
>> see the option is only added for:
>>
>> case llvm::Triple::arm:
>> case llvm::Triple::armeb:
>> case llvm::Triple::thumb:
>> case llvm::Triple::thumbeb:
>>
>> But not llvm::Triple::aarch64. So where is it coming from?
>>
> 
> cc -target aarch64-unknown-freebsd12.0
> --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp
> -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin -x assembler-with-cpp -m32
> -shared -nostdinc -nostdlib
> -Wl,-T/usr/src/sys/compat/cloudabi/cloudabi_vdso.lds
> /usr/src/sys/contrib/cloudabi/cloudabi_vdso_armv6_on_64bit.S -o
> cloudabi32_vdso.o
> 
> It must be the -m32 here making it build with llvm::Triple::arm.
> So we may need to include more of LLVM_TARGET_ARM in LLVM_TARGET_AARCH64.
> I'm testing locally to see how much is needed.
> 

r335747 should fix aarch64 but it's not necessarily the best fix. It may
be possible to reduce how much of MK_LLVM_TARGET_ARM is needed for -m32
support for arm64. I didn't look into that too much.

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: Build updates [ ci.freebsd.org FreeBSD-head-{aarch64, armv7, armv6}-build failures as of, for example -r335711 and -r335713 ]

[Bryan Drewery]

On 6/27/2018 11:44 AM, Bryan Drewery wrote:
> On 6/27/2018 10:53 AM, Mark Millard wrote:
>>
>>> On 2018-Jun-27, at 10:01 AM, Bryan Drewery  wrote:
>>>
>>> As of r335704:
>>>
>>> - make tinderbox/universe will now build the bootstrap clang *once*.
>>> Each target clang is still build of course.  This support does not work
>>> with gcc.
>>> - make buildworld (kernel-toolchain and toolchain) will build the
>>> bootstrap clang (if needed per SYSTEM_COMPILER logic) with only the
>>> TARGET.TARGET_ARCH backend support. The installed clang has all still so
>>> SYSTEM_COMPILER logic works for cross-compling.
>>>
>>> This uses the feature dim@ added in r335558 to selectively disable LLVM
>>> targets. I've added a new option named WITH[OUT]_LLVM_TARGET_ALL which I
>>> suggest using rather than the per-arch options. It is default on (WITH).
>>> Set WITHOUT to only build the needed native arch on your system for both
>>> bootstrap and compiled clang. Setting WITHOUT disables SYSTEM_COMPILER
>>> support for cross-builds.
>>>
>>> Please CC me directly for any weird tinderbox/universe or clang failures
>>> for the next few weeks.
> 
> Thanks!
> 
>>
>> https://ci.freebsd.org/job/FreeBSD-head-aarch64-build/8324/consoleText
>>
>> --- all_subdir_cloudabi32 ---
>> clang (LLVM option parsing): Unknown command line argument 
>> '-arm-add-build-attributes'.  Try: 'clang (LLVM option parsing) -help'
>> clang (LLVM option parsing): Did you mean '-force-attribute'?
>> *** [cloudabi32_vdso.o] Error code 1
>>
> 
> This was an aarch64 build. It looks like -arm-add-build-attributes is
> from Target/ARM/AsmParser/ARMAsmParser.cpp which is only built for
> LLVM_TARGET_ARM but not LLVM_TARGET_AARCH64.
> 
> Looking in contrib/llvm/tools/clang/lib/Driver/ToolChains/Clang.cpp I
> see the option is only added for:
> 
> case llvm::Triple::arm:
> case llvm::Triple::armeb:
> case llvm::Triple::thumb:
> case llvm::Triple::thumbeb:
> 
> But not llvm::Triple::aarch64. So where is it coming from?
> 

cc -target aarch64-unknown-freebsd12.0
--sysroot=/usr/obj/usr/src/arm64.aarch64/tmp
-B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin -x assembler-with-cpp -m32
-shared -nostdinc -nostdlib
-Wl,-T/usr/src/sys/compat/cloudabi/cloudabi_vdso.lds
/usr/src/sys/contrib/cloudabi/cloudabi_vdso_armv6_on_64bit.S -o
cloudabi32_vdso.o

It must be the -m32 here making it build with llvm::Triple::arm.
So we may need to include more of LLVM_TARGET_ARM in LLVM_TARGET_AARCH64.
I'm testing locally to see how much is needed.

>>
>> https://ci.freebsd.org/job/FreeBSD-head-armv7-build/460/consoleText
>> (armv6 is similar)
>>
>> --- all_subdir_lib/clang/libllvm ---
>> ===> lib/clang/libllvm (all)
>> [Creating objdir 
>> /usr/obj/usr/src/arm.armv7/tmp/obj-tools/lib/clang/libllvm...]
>> make[4]: "/usr/src/lib/clang/libllvm/Makefile" line 16: Please enable at 
>> least one of: MK_LLVM_TARGET_AARCH64, MK_LLVM_TARGET_ARM, 
>> MK_LLVM_TARGET_MIPS, MK_LLVM_TARGET_POWERPC, MK_LLVM_TARGET_SPARC, or 
>> MK_LLVM_TARGET_X86
>> *** [all_subdir_lib/clang/libllvm] Error code 1
>>
> 
> Arm failures fixed in r335718.
> 
> 
> 


-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: TSC calibration in virtual machines

[Ian Lepore]

On Wed, 2018-06-27 at 10:05 -0700, Rodney W. Grimes wrote:
> > 
> > On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim 
> > wrote:
> > 
> > > 
> > > On 06/27/2018 03:14, Andriy Gapon wrote:
> > > > 
> > > > 
> > > > It seems that TSC calibration in virtual machines sometimes can
> > > > do more
> > > harm
> > > > 
> > > > than good.  Should we default to trusting the information
> > > > provided by a
> > > hypervisor?
> > > > 
> > > > 
> > > > Specifically, I am observing a problem on GCE instances where
> > > > calibrated
> > > TSC
> > > > 
> > > > frequency is about 10% lower than advertised frequency.  And
> > > > apparently
> > > the
> > > > 
> > > > advertised frequency is the right one.
> > > > 
> > > > I found this thread with similar reports and a variety of
> > > > workarounds
> > > from
> > > > 
> > > > administratively disabling the calibration to switching to a
> > > > different
> > > timecounter:
> > > > 
> > > > https://lists.freebsd.org/pipermail/freebsd-cloud/2017-
> > > January/80.html
> > > 
> > > We already do that for VMware hosts since r221214.
> > > 
> > > https://svnweb.freebsd.org/changeset/base/221214
> > > 
> > > We should do the same for each hypervisor.
> > > 
> > > Jung-uk Kim
> > > 
> > > 
> > We probably should.  But why does calibration fail in the first
> > place?  If
> > it can fail in a VM, then it can probably fail on bare metal
> > too.  It would
> > be worth investigating.
> No, the failure in a VM is unique to a VM, it has to do with the fact
> your have the hypervisor timeslicing a CPU that you believe to be
> 100%
> dedicated to you.
> 
> There are several white papers, including one from VMWare about what
> they have done to help with the time keeping problems.
> 
> What is suggested above would be a correct thing to do.
> Bhyve creates these issues as well, and use of certain timers
> in a bhyve guest can cause you nightmares with ntp.

Iirc, bhyve's arithmetic when doing timer emulation leads to roundoff
errors that accumulate to effectively make the emulated timer run off-
frequency. The hpet timer was trivial to fix by just redefining it to
run at a power-of-2 frequency to eliminate rounding errors. The other
timers have to run at fixed frequencies, so better arithmetic will be
the way to fix them. I vaguely remember that being harder to do than to
say because of the way the code is currently structured, which is why I
just did the easy fix to the hpet so that people would have at least
one usable timer that didn't give ntpd fits in guest OSes.

-- Ian
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Build updates [ ci.freebsd.org FreeBSD-head-{aarch64, armv7, armv6}-build failures as of, for example -r335711 and -r335713 ]

[Bryan Drewery]

On 6/27/2018 10:53 AM, Mark Millard wrote:
> 
>> On 2018-Jun-27, at 10:01 AM, Bryan Drewery  wrote:
>>
>> As of r335704:
>>
>> - make tinderbox/universe will now build the bootstrap clang *once*.
>> Each target clang is still build of course.  This support does not work
>> with gcc.
>> - make buildworld (kernel-toolchain and toolchain) will build the
>> bootstrap clang (if needed per SYSTEM_COMPILER logic) with only the
>> TARGET.TARGET_ARCH backend support. The installed clang has all still so
>> SYSTEM_COMPILER logic works for cross-compling.
>>
>> This uses the feature dim@ added in r335558 to selectively disable LLVM
>> targets. I've added a new option named WITH[OUT]_LLVM_TARGET_ALL which I
>> suggest using rather than the per-arch options. It is default on (WITH).
>> Set WITHOUT to only build the needed native arch on your system for both
>> bootstrap and compiled clang. Setting WITHOUT disables SYSTEM_COMPILER
>> support for cross-builds.
>>
>> Please CC me directly for any weird tinderbox/universe or clang failures
>> for the next few weeks.

Thanks!

> 
> https://ci.freebsd.org/job/FreeBSD-head-aarch64-build/8324/consoleText
> 
> --- all_subdir_cloudabi32 ---
> clang (LLVM option parsing): Unknown command line argument 
> '-arm-add-build-attributes'.  Try: 'clang (LLVM option parsing) -help'
> clang (LLVM option parsing): Did you mean '-force-attribute'?
> *** [cloudabi32_vdso.o] Error code 1
> 

This was an aarch64 build. It looks like -arm-add-build-attributes is
from Target/ARM/AsmParser/ARMAsmParser.cpp which is only built for
LLVM_TARGET_ARM but not LLVM_TARGET_AARCH64.

Looking in contrib/llvm/tools/clang/lib/Driver/ToolChains/Clang.cpp I
see the option is only added for:

case llvm::Triple::arm:
case llvm::Triple::armeb:
case llvm::Triple::thumb:
case llvm::Triple::thumbeb:

But not llvm::Triple::aarch64. So where is it coming from?

> 
> https://ci.freebsd.org/job/FreeBSD-head-armv7-build/460/consoleText
> (armv6 is similar)
> 
> --- all_subdir_lib/clang/libllvm ---
> ===> lib/clang/libllvm (all)
> [Creating objdir 
> /usr/obj/usr/src/arm.armv7/tmp/obj-tools/lib/clang/libllvm...]
> make[4]: "/usr/src/lib/clang/libllvm/Makefile" line 16: Please enable at 
> least one of: MK_LLVM_TARGET_AARCH64, MK_LLVM_TARGET_ARM, 
> MK_LLVM_TARGET_MIPS, MK_LLVM_TARGET_POWERPC, MK_LLVM_TARGET_SPARC, or 
> MK_LLVM_TARGET_X86
> *** [all_subdir_lib/clang/libllvm] Error code 1
> 

Arm failures fixed in r335718.



-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: Build updates [ ci.freebsd.org FreeBSD-head-{aarch64, armv7, armv6}-build failures as of, for example -r335711 and -r335713 ]

[Mark Millard]

> On 2018-Jun-27, at 10:01 AM, Bryan Drewery  wrote:
> 
> As of r335704:
> 
> - make tinderbox/universe will now build the bootstrap clang *once*.
> Each target clang is still build of course.  This support does not work
> with gcc.
> - make buildworld (kernel-toolchain and toolchain) will build the
> bootstrap clang (if needed per SYSTEM_COMPILER logic) with only the
> TARGET.TARGET_ARCH backend support. The installed clang has all still so
> SYSTEM_COMPILER logic works for cross-compling.
> 
> This uses the feature dim@ added in r335558 to selectively disable LLVM
> targets. I've added a new option named WITH[OUT]_LLVM_TARGET_ALL which I
> suggest using rather than the per-arch options. It is default on (WITH).
> Set WITHOUT to only build the needed native arch on your system for both
> bootstrap and compiled clang. Setting WITHOUT disables SYSTEM_COMPILER
> support for cross-builds.
> 
> Please CC me directly for any weird tinderbox/universe or clang failures
> for the next few weeks.

https://ci.freebsd.org/job/FreeBSD-head-aarch64-build/8324/consoleText

--- all_subdir_cloudabi32 ---
clang (LLVM option parsing): Unknown command line argument 
'-arm-add-build-attributes'.  Try: 'clang (LLVM option parsing) -help'
clang (LLVM option parsing): Did you mean '-force-attribute'?
*** [cloudabi32_vdso.o] Error code 1


https://ci.freebsd.org/job/FreeBSD-head-armv7-build/460/consoleText
(armv6 is similar)

--- all_subdir_lib/clang/libllvm ---
===> lib/clang/libllvm (all)
[Creating objdir /usr/obj/usr/src/arm.armv7/tmp/obj-tools/lib/clang/libllvm...]
make[4]: "/usr/src/lib/clang/libllvm/Makefile" line 16: Please enable at least 
one of: MK_LLVM_TARGET_AARCH64, MK_LLVM_TARGET_ARM, MK_LLVM_TARGET_MIPS, 
MK_LLVM_TARGET_POWERPC, MK_LLVM_TARGET_SPARC, or MK_LLVM_TARGET_X86
*** [all_subdir_lib/clang/libllvm] Error code 1


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC calibration in virtual machines

[Stephen J. Kiernan]

On Wed, Jun 27, 2018, 12:48 PM Alan Somers  wrote:

> On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim  wrote:
>
> > On 06/27/2018 03:14, Andriy Gapon wrote:
> > >
> > > It seems that TSC calibration in virtual machines sometimes can do more
> > harm
> > > than good.  Should we default to trusting the information provided by a
> > hypervisor?
> > >
> > > Specifically, I am observing a problem on GCE instances where
> calibrated
> > TSC
> > > frequency is about 10% lower than advertised frequency.  And apparently
> > the
> > > advertised frequency is the right one.
> > >
> > > I found this thread with similar reports and a variety of workarounds
> > from
> > > administratively disabling the calibration to switching to a different
> > timecounter:
> > > https://lists.freebsd.org/pipermail/freebsd-cloud/2017-
> > January/80.html
> >
> > We already do that for VMware hosts since r221214.
> >
> > https://svnweb.freebsd.org/changeset/base/221214
> >
> > We should do the same for each hypervisor.
> >
> > Jung-uk Kim
> >
> >
> We probably should.  But why does calibration fail in the first place?  If
> it can fail in a VM, then it can probably fail on bare metal too.  It would
> be worth investigating.
>

The main problem is you can't be assured that the DELAY call will be
accurate in those cases. Also the way that some VMs implement the rdtsc
insrruction may not be as accurate as we would need. In some cases it can
compound the issue.

-Steve
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC calibration in virtual machines

[Jung-uk Kim]

On 06/27/2018 13:05, Rodney W. Grimes wrote:
> There are several white papers, including one from VMWare about what
> they have done to help with the time keeping problems.

https://www.vmware.com/files/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf

Jung-uk Kim



signature.asc
Description: OpenPGP digital signature

Re: TSC calibration in virtual machines

[Rodney W. Grimes]

> On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim  wrote:
> 
> > On 06/27/2018 03:14, Andriy Gapon wrote:
> > >
> > > It seems that TSC calibration in virtual machines sometimes can do more
> > harm
> > > than good.  Should we default to trusting the information provided by a
> > hypervisor?
> > >
> > > Specifically, I am observing a problem on GCE instances where calibrated
> > TSC
> > > frequency is about 10% lower than advertised frequency.  And apparently
> > the
> > > advertised frequency is the right one.
> > >
> > > I found this thread with similar reports and a variety of workarounds
> > from
> > > administratively disabling the calibration to switching to a different
> > timecounter:
> > > https://lists.freebsd.org/pipermail/freebsd-cloud/2017-
> > January/80.html
> >
> > We already do that for VMware hosts since r221214.
> >
> > https://svnweb.freebsd.org/changeset/base/221214
> >
> > We should do the same for each hypervisor.
> >
> > Jung-uk Kim
> >
> >
> We probably should.  But why does calibration fail in the first place?  If
> it can fail in a VM, then it can probably fail on bare metal too.  It would
> be worth investigating.

No, the failure in a VM is unique to a VM, it has to do with the fact
your have the hypervisor timeslicing a CPU that you believe to be 100%
dedicated to you.

There are several white papers, including one from VMWare about what
they have done to help with the time keeping problems.

What is suggested above would be a correct thing to do.
Bhyve creates these issues as well, and use of certain timers
in a bhyve guest can cause you nightmares with ntp.


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC calibration in virtual machines

[Jung-uk Kim]

On 06/27/2018 13:01, Ryan Stone wrote:
> I would guess that the calibration can fail because when running under
> the hypervisor, the FreeBSD guest code can be descheduled at the wrong
> time.  As I recall, the current algorithm looks like:
> 
> 1. Sample rdtsc
> 2. Use a fixed-frequency timer to busy-wait for exactly 1 second
> 3. Sample rdtsc again
> 4. tsc_freq = sample2 - sample1;
> 
> If we are descheduled between 2 and 3, the time we spend off-cpu will
> not be accounted for at step 4.  On bare-metal this is not possible as
> neither the scheduler nor interrupts are not running yet.
> 
> Although, come to think of it, I seem to recall something about SMI
> interrupts mucking this up long in the past, for exactly the same
> reason.

I think it was legacy USB device emulation for certain Intel
chipset-based motherboards.

Jung-uk Kim



signature.asc
Description: OpenPGP digital signature

Re: TSC calibration in virtual machines

[Ryan Stone]

I would guess that the calibration can fail because when running under
the hypervisor, the FreeBSD guest code can be descheduled at the wrong
time.  As I recall, the current algorithm looks like:

1. Sample rdtsc
2. Use a fixed-frequency timer to busy-wait for exactly 1 second
3. Sample rdtsc again
4. tsc_freq = sample2 - sample1;

If we are descheduled between 2 and 3, the time we spend off-cpu will
not be accounted for at step 4.  On bare-metal this is not possible as
neither the scheduler nor interrupts are not running yet.

Although, come to think of it, I seem to recall something about SMI
interrupts mucking this up long in the past, for exactly the same
reason.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC calibration in virtual machines

[Alan Somers]

On Wed, Jun 27, 2018 at 11:05 AM, Jung-uk Kim  wrote:

> On 06/27/2018 12:47, Alan Somers wrote:
> > On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim  > > wrote:
> >
> > On 06/27/2018 03:14, Andriy Gapon wrote:
> > >
> > > It seems that TSC calibration in virtual machines sometimes can do
> more harm
> > > than good.  Should we default to trusting the information provided
> by a hypervisor?
> > >
> > > Specifically, I am observing a problem on GCE instances where
> calibrated TSC
> > > frequency is about 10% lower than advertised frequency.  And
> apparently the
> > > advertised frequency is the right one.
> > >
> > > I found this thread with similar reports and a variety of
> workarounds from
> > > administratively disabling the calibration to switching to a
> different timecounter:
> > > https://lists.freebsd.org/pipermail/freebsd-cloud/2017-
> January/80.html
> >  January/80.html>
> >
> > We already do that for VMware hosts since r221214.
> >
> > https://svnweb.freebsd.org/changeset/base/221214
> > 
> >
> > We should do the same for each hypervisor.
> >
> > We probably should.  But why does calibration fail in the first place?
> Because multiple guests are sharing same physical CPUs and guest OS has
> no control, timing cannot be 100% accurate.
>
> > If it can fail in a VM, then it can probably fail on bare metal too.  It
> > would be worth investigating.
> It does not "fail" in bare metal because we have almost complete control.
>
> Jung-uk Kim
>
>
Makes sense.  I didn't realize that it ran before the scheduler or
interrupts were started.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC calibration in virtual machines

[Jung-uk Kim]

On 06/27/2018 12:47, Alan Somers wrote:
> On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim  > wrote:
> 
> On 06/27/2018 03:14, Andriy Gapon wrote:
> > 
> > It seems that TSC calibration in virtual machines sometimes can do more 
> harm
> > than good.  Should we default to trusting the information provided by a 
> hypervisor?
> > 
> > Specifically, I am observing a problem on GCE instances where 
> calibrated TSC
> > frequency is about 10% lower than advertised frequency.  And apparently 
> the
> > advertised frequency is the right one.
> > 
> > I found this thread with similar reports and a variety of workarounds 
> from
> > administratively disabling the calibration to switching to a different 
> timecounter:
> > 
> https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/80.html
> 
> 
> 
> We already do that for VMware hosts since r221214.
> 
> https://svnweb.freebsd.org/changeset/base/221214
> 
> 
> We should do the same for each hypervisor.
> 
> We probably should.  But why does calibration fail in the first place?
Because multiple guests are sharing same physical CPUs and guest OS has
no control, timing cannot be 100% accurate.

> If it can fail in a VM, then it can probably fail on bare metal too.  It
> would be worth investigating.
It does not "fail" in bare metal because we have almost complete control.

Jung-uk Kim



signature.asc
Description: OpenPGP digital signature

Build updates

[Bryan Drewery]

As of r335704:

- make tinderbox/universe will now build the bootstrap clang *once*.
Each target clang is still build of course.  This support does not work
with gcc.
- make buildworld (kernel-toolchain and toolchain) will build the
bootstrap clang (if needed per SYSTEM_COMPILER logic) with only the
TARGET.TARGET_ARCH backend support. The installed clang has all still so
SYSTEM_COMPILER logic works for cross-compling.

This uses the feature dim@ added in r335558 to selectively disable LLVM
targets. I've added a new option named WITH[OUT]_LLVM_TARGET_ALL which I
suggest using rather than the per-arch options. It is default on (WITH).
Set WITHOUT to only build the needed native arch on your system for both
bootstrap and compiled clang. Setting WITHOUT disables SYSTEM_COMPILER
support for cross-builds.

Please CC me directly for any weird tinderbox/universe or clang failures
for the next few weeks.

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: TSC calibration in virtual machines

[Alan Somers]

On Wed, Jun 27, 2018 at 10:36 AM, Jung-uk Kim  wrote:

> On 06/27/2018 03:14, Andriy Gapon wrote:
> >
> > It seems that TSC calibration in virtual machines sometimes can do more
> harm
> > than good.  Should we default to trusting the information provided by a
> hypervisor?
> >
> > Specifically, I am observing a problem on GCE instances where calibrated
> TSC
> > frequency is about 10% lower than advertised frequency.  And apparently
> the
> > advertised frequency is the right one.
> >
> > I found this thread with similar reports and a variety of workarounds
> from
> > administratively disabling the calibration to switching to a different
> timecounter:
> > https://lists.freebsd.org/pipermail/freebsd-cloud/2017-
> January/80.html
>
> We already do that for VMware hosts since r221214.
>
> https://svnweb.freebsd.org/changeset/base/221214
>
> We should do the same for each hypervisor.
>
> Jung-uk Kim
>
>
We probably should.  But why does calibration fail in the first place?  If
it can fail in a VM, then it can probably fail on bare metal too.  It would
be worth investigating.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: TSC calibration in virtual machines

[Jung-uk Kim]

On 06/27/2018 03:14, Andriy Gapon wrote:
> 
> It seems that TSC calibration in virtual machines sometimes can do more harm
> than good.  Should we default to trusting the information provided by a 
> hypervisor?
> 
> Specifically, I am observing a problem on GCE instances where calibrated TSC
> frequency is about 10% lower than advertised frequency.  And apparently the
> advertised frequency is the right one.
> 
> I found this thread with similar reports and a variety of workarounds from
> administratively disabling the calibration to switching to a different 
> timecounter:
> https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/80.html

We already do that for VMware hosts since r221214.

https://svnweb.freebsd.org/changeset/base/221214

We should do the same for each hypervisor.

Jung-uk Kim



signature.asc
Description: OpenPGP digital signature

Re: TSC calibration in virtual machines

[John Baldwin]

On 6/27/18 12:14 AM, Andriy Gapon wrote:
> 
> It seems that TSC calibration in virtual machines sometimes can do more harm
> than good.  Should we default to trusting the information provided by a 
> hypervisor?
> 
> Specifically, I am observing a problem on GCE instances where calibrated TSC
> frequency is about 10% lower than advertised frequency.  And apparently the
> advertised frequency is the right one.
> 
> I found this thread with similar reports and a variety of workarounds from
> administratively disabling the calibration to switching to a different 
> timecounter:
> https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/80.html

I suspect you are probably right that we should just "trust" TSC frequencies
provided by a hypervisor.  We could perhaps choose to whitelist hypervisors
known to provide accurate values if we wanted to be cautious.

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: swapping is completely broken in -CURRENT r334649?

[Mark Johnston]

On Wed, Jun 27, 2018 at 10:46:38AM +, Alexey Dokuchaev wrote:
> On Fri, Jun 15, 2018 at 04:40:22AM -0400, Mark Johnston wrote:
> > ...
> > The change was committed as r334752.  Are you seeing unexpected OOM
> > kills on or after that revision?
> 
> I've just discovered this thread.  I've updated my -CURRENT desktop ca.
> June 4th and my X.org session crashes within several minutes.  This is
> on i386 system with 4G RAM.
> 
> I cannot get SVN revision of that kernel because something got broken
> and it was not embedded in the image. :-(
> 
> Rebooting with kernel="kernel.old" from May 12th made this problem go
> away.  Is the root cause is known at this time?  Which revision shall
> I try to see whether it does not exhibit this bug any longer?  Thanks,

As noted earlier in the thread, r329882 introduced a problem where
out-of-memory kills were taking place despite an abundance of free pages
and swap space.  That bug was fixed in r334752.  If you are seeing a
problem that is fixed in a kernel from May 12th, then it is possibly
unrelated to this bug, but it is worth testing r334752 or later to
confirm.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: swapping is completely broken in -CURRENT r334649?

[Alexey Dokuchaev]

On Fri, Jun 15, 2018 at 04:40:22AM -0400, Mark Johnston wrote:
> ...
> The change was committed as r334752.  Are you seeing unexpected OOM
> kills on or after that revision?

I've just discovered this thread.  I've updated my -CURRENT desktop ca.
June 4th and my X.org session crashes within several minutes.  This is
on i386 system with 4G RAM.

I cannot get SVN revision of that kernel because something got broken
and it was not embedded in the image. :-(

Rebooting with kernel="kernel.old" from May 12th made this problem go
away.  Is the root cause is known at this time?  Which revision shall
I try to see whether it does not exhibit this bug any longer?  Thanks,

./danfe
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: error building clang in HEAD

[Gary Jennejohn]

On Tue, 26 Jun 2018 12:51:29 -0700
Bryan Drewery  wrote:

> On 6/26/2018 12:40 PM, Kevin Oberman wrote:
> > On Tue, Jun 26, 2018 at 11:00 AM, Gary Jennejohn  > > wrote:
> > 
> > On Tue, 26 Jun 2018 18:24:12 +0200
> > Gary Jennejohn mailto:gljennj...@gmail.com>>
> > wrote:
> >   
> > > On Mon, 25 Jun 2018 11:28:18 -0700
> > > Bryan Drewery  wrote:
> > >  
> > > > On 6/24/2018 12:57 AM, Gary Jennejohn wrote:__  
> > > > > On Sat, 23 Jun 2018 17:05:16 +0200
> > > > > Dimitry Andric  wrote:
> > > > >__ __ __  
> > > > >> On 23 Jun 2018, at 15:40, Gary Jennejohn  
> > mailto:gljennj...@gmail.com>> wrote:__ __  
> > > > >>>
> > > > >>> There is a strange error building clang with this use case:
> > > > >>>
> > > > >>> cd /usr/src
> > > > >>> make -j10 makeworld__ __ __  
> > > > >>
> > > > >> What's the "makeworld" target?__ I've not heard of this.
> > > > >>__ __  
> > > > >
> > > > > A typo.__ I meant buildowrld.
> > > > >__ __ __  
> > > > >>> which produces this error output:
> > > > >>>__ __ __ __  
> > > > >>> ===> lib/clang/libclang (all)__ __ __  
> > > > >>> error: unable to rename temporary  
> > 'Sema/SemaTemplate-12ad7e30.o.tmp' to output file
> > 'Sema/SemaTemplate.o': 'No such file or directory'  
> > > > >>> 1 error generated.
> > > > >>> --- Sema/SemaTemplate.o ---
> > > > >>> *** [Sema/SemaTemplate.o] Error code 1__ __ __  
> > > > >>
> > > > >> This typically happens if "make obj" was not run before the  
> > rest of the  
> > > > >> make targets.__ Normally, the order is: make obj, then make  
> > depend, then  
> > > > >> make (a.k.a. make all).
> > > > >>
> > > > >> Is there a directory /usr/obj/usr/src/lib/libclang/Sema ?
> > > > >>__ __  
> > > > >
> > > > > Well, I would hope/expect that make buildworld does make obj.
> > > > >
> > > > > Yes, the directory was there.
> > > > >__ __ __  
> > > >
> > > > Actually neither 'make obj' nor 'make depend' is done or needed  
> > anymore  
> > > > in buildworld.
> > > >
> > > > The directory above is incorrect, please check for
> > > >
> > > >__ __ __/usr/obj/usr/src/amd64.amd64/lib/clang/libclang/Sema
> > > >__ __  
> > >
> > > Well, now everything is there because I ran a buildworld without -j.
> > >  
> > > > Do you have another Makefile or script that is executing
> > > > buildworld for you?
> > > >__ __  
> > >
> > > No, I use a bash alias named mw:
> > > mw is aliased to `pushd /usr/src;time make -s -j$NCPU buildworld;popd'
> > >
> > > NCPU is defined as 10.
> > >  
> > > > What's in your src.conf and make.conf?
> > > >__ __  
> > >
> > > The only changes I made recently were to /etc/src.conf when I added:
> > >
> > > WITHOUT_LLVM_TARGET_AARCH64=yes
> > > WITHOUT_LLVM_TARGET_ARM=ys
> > > WITHOUT_LLVM_TARGET_MIPS=yes
> > > WITHOUT_LLVM_TARGET_POWERPC=yes
> > > WITHOUT_LLVM_TARGET_SPARC=yes
> > > WITH_LLVM_TARGET_X86=yes
> > >
> > > Otherwise, I haven't touched src.conf or make.conf in__ a long time.
> > >  
> > 
> > I removed some old cruft from src.conf and now make -j10 buildworld is
> > succeeding, even after rm -rf /usr/obj/usr.
> > 
> > Thanks for pointing me in the right direction.
> > 
> > I'd like to hear what triggered this as removing unneeded LLVM targets
> > seems like a good idea if you know that you won't need them. Building  
> 
> I don't think the options are related to the build error.
> 

Correct, these options were not the cause of the errors.  I had
some really old options from several years ago which were the cause.
Don't remeber now exactly which ones, but they were all related to
using CLANG instead of GCC.

-- 
Gary Jennejohn
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

TSC calibration in virtual machines

[Andriy Gapon]

It seems that TSC calibration in virtual machines sometimes can do more harm
than good.  Should we default to trusting the information provided by a 
hypervisor?

Specifically, I am observing a problem on GCE instances where calibrated TSC
frequency is about 10% lower than advertised frequency.  And apparently the
advertised frequency is the right one.

I found this thread with similar reports and a variety of workarounds from
administratively disabling the calibration to switching to a different 
timecounter:
https://lists.freebsd.org/pipermail/freebsd-cloud/2017-January/80.html

-- 
Andriy Gapon
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"