Re: aarch64 and armv6 vs. armv7 support: armv6 is not supported, despite what "man arch" reports

[Mark Millard]

On Dec 7, 2023, at 01:19, Dimitry Andric  wrote:

> On 7 Dec 2023, at 05:31, Mark Millard  wrote:
>> 
>> man arch reports:
>> 
>> QUOTE
>>Some machines support more than one FreeBSD ABI.  Typically these are
>>64-bit machines, where the “native” LP64 execution environment is
>>accompanied by the “legacy” ILP32 environment, which was the historical
>>32-bit predecessor for 64-bit evolution.  Examples are:
>> 
>>  LP64 ILP32 counterpart
>>  amd64i386
>>  powerpc64powerpc
>>  aarch64  armv6/armv7
> 
> So, this might be replaced with "armv6^armv7" or "armv6 xor armv7", then?

Only for folks that build from source. For those folks, a footnote
about updating MACHINE_ARCH32 in sys/arm64/include/param.h would be
appropriate. It is not exactly obvious or commonly known.

Hmm, thinking more about the old bugzilla information . . .

I'll also note that my information predated lib32 on aarch64: just
chroot/jail sorts of use back then, and I just tested chroot back
then. I've never tested a lib32 context for armv6 on aarch64 for
an adjusted MACHINE_ARCH32.


===
Mark Millard
marklmi at yahoo.com

Re: aarch64 and armv6 vs. armv7 support: armv6 is not supported, despite what "man arch" reports

[Warner Losh]

On Thu, Dec 7, 2023, 2:19 AM Dimitry Andric  wrote:

> On 7 Dec 2023, at 05:31, Mark Millard  wrote:
> >
> > man arch reports:
> >
> > QUOTE
> > Some machines support more than one FreeBSD ABI.  Typically these are
> > 64-bit machines, where the “native” LP64 execution environment is
> > accompanied by the “legacy” ILP32 environment, which was the
> historical
> > 32-bit predecessor for 64-bit evolution.  Examples are:
> >
> >   LP64 ILP32 counterpart
> >   amd64i386
> >   powerpc64powerpc
> >   aarch64  armv6/armv7
>
> So, this might be replaced with "armv6^armv7" or "armv6 xor armv7", then?
>

The binaries are basically the same. But you need one set of libraries or
the other since the calling conversations differ. I think you'll need to
enhance the current sysctl to be per jail.

Or you could migrate away from armv6. It's days are numbered in main.

Warner


> -Dimitry
>
>

Re: aarch64 and armv6 vs. armv7 support: armv6 is not supported, despite what "man arch" reports

[Robert Clausecker]

Hi;

Am Wed, Dec 06, 2023 at 08:31:13PM -0800 schrieb Mark Millard:
> man arch reports:
> 
> QUOTE
>  Some machines support more than one FreeBSD ABI.  Typically these are
>  64-bit machines, where the “native” LP64 execution environment is
>  accompanied by the “legacy” ILP32 environment, which was the historical
>  32-bit predecessor for 64-bit evolution.  Examples are:
> 
>LP64 ILP32 counterpart
>amd64i386
>powerpc64powerpc
>aarch64  armv6/armv7
> 
>  aarch64 will support execution of armv6 or armv7 binaries if the CPU
>  implements AArch32 execution state, however older armv4 and armv5
>  binaries aren't supported.
> END QUOTE
> 
> (I take "armv6 or armv7 binaries" as what was built targeting a FreeBSD
> architecture triple for one of those. FreeBSD keeps them distinct.)
> 
> However, the armv6 part of that is wrong: The infrastructure supports
> only one 32-bit alternative for a given kernel, not a family of them at
> once . . .

See also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256132

Yours,
Robert Clausecker

-- 
()  ascii ribbon campaign - for an encoding-agnostic world
/\  - against html email  - against proprietary attachments

Re: aarch64 and armv6 vs. armv7 support: armv6 is not supported, despite what "man arch" reports

[Dimitry Andric]

On 7 Dec 2023, at 05:31, Mark Millard  wrote:
> 
> man arch reports:
> 
> QUOTE
> Some machines support more than one FreeBSD ABI.  Typically these are
> 64-bit machines, where the “native” LP64 execution environment is
> accompanied by the “legacy” ILP32 environment, which was the historical
> 32-bit predecessor for 64-bit evolution.  Examples are:
> 
>   LP64 ILP32 counterpart
>   amd64i386
>   powerpc64powerpc
>   aarch64  armv6/armv7

So, this might be replaced with "armv6^armv7" or "armv6 xor armv7", then?

-Dimitry



signature.asc
Description: Message signed with OpenPGP

aarch64 and armv6 vs. armv7 support: armv6 is not supported, despite what "man arch" reports

[Mark Millard]

man arch reports:

QUOTE
 Some machines support more than one FreeBSD ABI.  Typically these are
 64-bit machines, where the “native” LP64 execution environment is
 accompanied by the “legacy” ILP32 environment, which was the historical
 32-bit predecessor for 64-bit evolution.  Examples are:

   LP64 ILP32 counterpart
   amd64i386
   powerpc64powerpc
   aarch64  armv6/armv7

 aarch64 will support execution of armv6 or armv7 binaries if the CPU
 implements AArch32 execution state, however older armv4 and armv5
 binaries aren't supported.
END QUOTE

(I take "armv6 or armv7 binaries" as what was built targeting a FreeBSD
architecture triple for one of those. FreeBSD keeps them distinct.)

However, the armv6 part of that is wrong: The infrastructure supports
only one 32-bit alternative for a given kernel, not a family of them at
once . . .

sys/kern/kern_mib.c :

static const char *
proc_machine_arch(struct proc *p)
{
  if (p->p_sysent->sv_machine_arch != NULL)
return (p->p_sysent->sv_machine_arch(p));
#ifdef COMPAT_FREEBSD32
if (SV_PROC_FLAG(p, SV_ILP32))
return (MACHINE_ARCH32);
#endif
return (MACHINE_ARCH);
}
. . .
static int
sysctl_kern_supported_archs(SYSCTL_HANDLER_ARGS)
{
const char *supported_archs;

supported_archs =
#ifdef COMPAT_FREEBSD32
compat_freebsd_32bit ? MACHINE_ARCH " " MACHINE_ARCH32 :
#endif
MACHINE_ARCH;
return (SYSCTL_OUT(req, supported_archs, strlen(supported_archs) + 1));
}


sys/arm64/include/param.h :

#define MACHINE_ARCHES MACHINE_ARCH " " MACHINE_ARCH32
. . .
#define MACHINE_ARCH32 "armv7"

(There is no "armv6" alternative present.)

But with something like:

#define MACHINE_ARCH32 "armv7 armv6"

MACHINE_ARCH32 is not interpreted as a list of alternatives, each
supported. There is code that would have to be reworked to allow
a list of alternatives to work.

One can build a custom kernel with:

#define MACHINE_ARCH32 "armv6"

and then, having booted that kernel, then run armv6 on aarch64
--but, then, not armv7.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256132 is
about this and has my messy notes as I explored and discovered
that multiple 32-bit alternatives did not work. I see that I
forgot various quote (") symbols.


This note was prompted by:

https://lists.freebsd.org/archives/freebsd-hackers/2023-December/002728.html

that mentions "the list of valid MACHINE_ARCH" that reminded me
of this old issue.

===
Mark Millard
marklmi at yahoo.com

Re: What is rc.d/opensm?

[Oleksandr Kryvulia]

24.11.23 12:20, Alexander Leidinger:

Hi,

for my work on service jails (https://reviews.freebsd.org/D40370) I 
try to find out what opensm is. On my amd64 system I don't have a man 
page nor the binary (and man.freebsd.org doesn't know either about 
opensm).


Bye,
Alexander.



See man /usr/src/contrib/ofed/opensm/man/opensm.8 and 
https://wiki.freebsd.org/InfiniBand

What is rc.d/opensm?

[Alexander Leidinger]

Hi,

for my work on service jails (https://reviews.freebsd.org/D40370) I try 
to find out what opensm is. On my amd64 system I don't have a man page 
nor the binary (and man.freebsd.org doesn't know either about opensm).


Bye,
Alexander.

--
http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.orgnetch...@freebsd.org  : PGP 0x8F31830F9F2772BF


signature.asc
Description: OpenPGP digital signature

Re: what do I do when git cherry-pick works, but results are bogus?

[Pat Maddox]

On Wed, May 17, 2023, at 9:44 AM, Rick Macklem wrote:
> So, the subject line basically says it.
> I do a git cherry-pick to MFC. It works, but the resultant file(s) are not
> correct. What do I do to fix this?
> (If the merge fails, then it's easy, but there doesn't seem to be an option
>  on cherry-pick that forces it into "merge failed", so you can edit/add the 
> file
>  and then "git cherry-pick --continue".)

git cherry-pick --no-commit

?

Re: what do I do when git cherry-pick works, but results are bogus?

[Matt Wheeler]

On Wed, 17 May 2023, 17:44 Rick Macklem,  wrote:

> So, the subject line basically says it.
> I do a git cherry-pick to MFC. It works, but the resultant file(s) are not
> correct. What do I do to fix this?
> (If the merge fails, then it's easy, but there doesn't seem to be an option
>  on cherry-pick that forces it into "merge failed", so you can edit/add
> the file
>  and then "git cherry-pick --continue".)
>

If you're cherry-picking multiple commits then you can turn the problem
into a rebase

After the cherry-pick commits are created, then run

  git rebase -i 

Then change the `i` at the start of the line for the broken commit to `e`
(edit) before saving the plan file

Re: what do I do when git cherry-pick works, but results are bogus?

[Alan Somers]

On Wed, May 17, 2023 at 9:44 AM Rick Macklem  wrote:
>
> So, the subject line basically says it.
> I do a git cherry-pick to MFC. It works, but the resultant file(s) are not
> correct. What do I do to fix this?
> (If the merge fails, then it's easy, but there doesn't seem to be an option
>  on cherry-pick that forces it into "merge failed", so you can edit/add the 
> file
>  and then "git cherry-pick --continue".)
>
> Thanks for any help with this, rick

After a cherry-pick that doesn't give you quite what you want, edit
the file until it has the correct contents, then do "git commit
--amend /path/to/file".

what do I do when git cherry-pick works, but results are bogus?

[Rick Macklem]

So, the subject line basically says it.
I do a git cherry-pick to MFC. It works, but the resultant file(s) are not
correct. What do I do to fix this?
(If the merge fails, then it's easy, but there doesn't seem to be an option
 on cherry-pick that forces it into "merge failed", so you can edit/add the file
 and then "git cherry-pick --continue".)

Thanks for any help with this, rick

Re: What llvm16 libc++ updates for -std=c++20 use [was: Re: Delay in 14.0-RELEASE cycle and blocking items]

[Jan Beich]

Mark Millard  writes:

> Alexey Dokuchaev  wrote on
> Date: Wed, 03 May 2023 07:53:09 UTC :
>
>> On Mon, May 01, 2023 at 06:14:49PM +, Glen Barber wrote:
>> > ...
>> > There is no feasible way we are going to make the branch point of
>> > stable/14 in time, with that scheduled for May 12, 2023 with the above
>> > points. That said, this is not an all-inclusive list, but the more
>> > major items on our radar at the moment.
>> 
>> Does this delay mean we might get Clang 16 in the base? Current 15.0.7
>> hits assertion on one of my ports which had allegedly been fixed in 16.
>> Also, AFAIU it comes with better support for modern C++, e.g. ranges.
>
> These notes are based on using -std=c++20 and llvm16 on
> opensuse tumblweed (in early April), which has libc++
> support configurable. They also presume that the FreeBSD
> llvm16 update fully adopts the libc++ from llvm16.
> (FreeBSD LLVM upgrades do not always do so at the initial
> upgrade time.)

FWIW, std::ranges in base libc++ 16 (via llvm-16-update branch) works
fine at least in emulators/yuzu (c++20) and x11-wm/hyprland (c++23).
More can take advantage but currently use a workaround e.g.,

$ rg -l :devel/range-v3 | sed s,/Makefile,,
biology/seqan3
devel/fbthrift
editors/imhex
lang/solidity
net-im/telegram-desktop

Re: What llvm16 libc++ updates for -std=c++20 use [was: Re: Delay in 14.0-RELEASE cycle and blocking items]

[Mark Millard]

On May 3, 2023, at 08:57, Mark Millard  wrote:

> Alexey Dokuchaev  wrote on
> Date: Wed, 03 May 2023 07:53:09 UTC :
> 
>> On Mon, May 01, 2023 at 06:14:49PM +, Glen Barber wrote:
>>> ...
>>> There is no feasible way we are going to make the branch point of
>>> stable/14 in time, with that scheduled for May 12, 2023 with the above
>>> points. That said, this is not an all-inclusive list, but the more
>>> major items on our radar at the moment.
>> 
>> Does this delay mean we might get Clang 16 in the base? Current 15.0.7
>> hits assertion on one of my ports which had allegedly been fixed in 16.
>> Also, AFAIU it comes with better support for modern C++, e.g. ranges.
> 
> These notes are based on using -std=c++20 and llvm16 on
> opensuse tumblweed (in early April), which has libc++
> support configurable. They also presume that the FreeBSD
> llvm16 update fully adopts the libc++ from llvm16.
> (FreeBSD LLVM upgrades do not always do so at the initial
> upgrade time.)
> 
> __cpp_lib_ranges would go from undefined to 202106 .
> C++20 also has a later 202110 . C++23 has 3 later values,
> the last being 202211 . (I'm generally omitting the L
> suffixes in my materials.)
> 
> A couple of the C++20 ranges versions are late,
> retroactive fixes ["(DR)"s] for things that
> could not wait for C++23:
> 
> __cpp_lib_ranges -- 202106 (C++20) (DR)
> __cpp_lib_ranges -- 202110 (C++20) (DR)
> 
> So only the 202106 one was in llvm16 when I tested
> llvm16. (If I remember right, using -std=c++23 got
> the 202110 fix as well.)
> 
> Other libc++ things going from undefined to a defined
> status are:
> 
> __cpp_lib_constexpr_complex
> __cpp_lib_constexpr_vector
> __cpp_lib_memory_resource
> __cpp_lib_polymorphic_allocator
> __cpp_lib_source_location
> 
> It does not appear that any other __cpp_lib_... macros
> would change values for -std=c+=20 use.

Typo fix to the above: -std=c++20

> As for the overall status for ranges . . .
> 
> C++23 has lots of changes/additions for ranges:
> (The --'s indicate being undefined in llvm15.)

I should have noted that the MM months
below are from/for the standard and are not
indications of llvm16 or later of having
implemented them for -std=c+=23 .

> __cpp_lib_ranges -- 202202 (C++23)
> __cpp_lib_ranges -- 202207 (C++23)
> __cpp_lib_ranges -- 202211 (C++23)
> __cpp_lib_ranges_as_const -- 202207 (C++23)
> __cpp_lib_ranges_as_rvalue -- 202207 (C++23)
> __cpp_lib_ranges_cartesian_product -- 202207 (C++23)
> __cpp_lib_ranges_chunk -- 202202 (C++23)
> __cpp_lib_ranges_chunk_by -- 202202 (C++23)
> __cpp_lib_ranges_contains -- 202207 (C++23)
> __cpp_lib_ranges_enumerate -- 202303 (C++23)
> __cpp_lib_ranges_fold -- 202207 (C++23)
> __cpp_lib_ranges_iota -- 202202 (C++23)
> __cpp_lib_ranges_join_with -- 202202 (C++23)
> __cpp_lib_ranges_repeat -- 202207 (C++23)
> __cpp_lib_ranges_slide -- 202202 (C++23)
> __cpp_lib_ranges_starts_ends_with -- 202106 (C++23)
> __cpp_lib_ranges_stride -- 202207 (C++23)
> __cpp_lib_ranges_to_container -- 202202 (C++23)
> __cpp_lib_ranges_zip -- 202110 (C++23)
> 
> Ranges seems to be an active area of development
> across multiple standard vintages.


===
Mark Millard
marklmi at yahoo.com

What llvm16 libc++ updates for -std=c++20 use [was: Re: Delay in 14.0-RELEASE cycle and blocking items]

[Mark Millard]

Alexey Dokuchaev  wrote on
Date: Wed, 03 May 2023 07:53:09 UTC :

> On Mon, May 01, 2023 at 06:14:49PM +, Glen Barber wrote:
> > ...
> > There is no feasible way we are going to make the branch point of
> > stable/14 in time, with that scheduled for May 12, 2023 with the above
> > points. That said, this is not an all-inclusive list, but the more
> > major items on our radar at the moment.
> 
> Does this delay mean we might get Clang 16 in the base? Current 15.0.7
> hits assertion on one of my ports which had allegedly been fixed in 16.
> Also, AFAIU it comes with better support for modern C++, e.g. ranges.

These notes are based on using -std=c++20 and llvm16 on
opensuse tumblweed (in early April), which has libc++
support configurable. They also presume that the FreeBSD
llvm16 update fully adopts the libc++ from llvm16.
(FreeBSD LLVM upgrades do not always do so at the initial
upgrade time.)

__cpp_lib_ranges would go from undefined to 202106 .
C++20 also has a later 202110 . C++23 has 3 later values,
the last being 202211 . (I'm generally omitting the L
suffixes in my materials.)

A couple of the C++20 ranges versions are late,
retroactive fixes ["(DR)"s] for things that
could not wait for C++23:

__cpp_lib_ranges -- 202106 (C++20) (DR)
__cpp_lib_ranges -- 202110 (C++20) (DR)

So only the 202106 one was in llvm16 when I tested
llvm16. (If I remember right, using -std=c++23 got
the 202110 fix as well.)

Other libc++ things going from undefined to a defined
status are:

__cpp_lib_constexpr_complex
__cpp_lib_constexpr_vector
__cpp_lib_memory_resource
__cpp_lib_polymorphic_allocator
__cpp_lib_source_location

It does not appear that any other __cpp_lib_... macros
would change values for -std=c+=20 use.


As for the overall status for ranges . . .

C++23 has lots of changes/additions for ranges:
(The --'s indicate being undefined in llvm15.)

__cpp_lib_ranges -- 202202 (C++23)
__cpp_lib_ranges -- 202207 (C++23)
__cpp_lib_ranges -- 202211 (C++23)
__cpp_lib_ranges_as_const -- 202207 (C++23)
__cpp_lib_ranges_as_rvalue -- 202207 (C++23)
__cpp_lib_ranges_cartesian_product -- 202207 (C++23)
__cpp_lib_ranges_chunk -- 202202 (C++23)
__cpp_lib_ranges_chunk_by -- 202202 (C++23)
__cpp_lib_ranges_contains -- 202207 (C++23)
__cpp_lib_ranges_enumerate -- 202303 (C++23)
__cpp_lib_ranges_fold -- 202207 (C++23)
__cpp_lib_ranges_iota -- 202202 (C++23)
__cpp_lib_ranges_join_with -- 202202 (C++23)
__cpp_lib_ranges_repeat -- 202207 (C++23)
__cpp_lib_ranges_slide -- 202202 (C++23)
__cpp_lib_ranges_starts_ends_with -- 202106 (C++23)
__cpp_lib_ranges_stride -- 202207 (C++23)
__cpp_lib_ranges_to_container -- 202202 (C++23)
__cpp_lib_ranges_zip -- 202110 (C++23)

Ranges seems to be an active area of development
across multiple standard vintages.

===
Mark Millard
marklmi at yahoo.com

Re: What to do about a few lines in vfs_domount() never executed?

[Rick Macklem]

Just to provide closure on this, I just committed a patch to
main (that will be MFC'd in 2 weeks) that sets MNT_EXPORTED
when the "export" option is specified to nmount(2).

This restores the "only root can do exports" semantics that
appear to have been the case pre-r158857.

Thanks everyone for your comments, rick


On Wed, Dec 14, 2022 at 9:15 AM Tomoaki AOKI 
wrote:

> Tracking the commits, it was originally introduced to
> sys/kern/vfs_syscalls.c at r22521 [1][2] (Mon Feb 10 02:22:35 1997 by
> dyson, submitted by h...@freebsd.org) and later centralized into
> sys/kern/vfs_mount.c at r99264 [2].
>
> But according to the comment above the codes, maybe it would be
> intended to block userland programs or ports FS modules setting
> MNT_EXPORTED.
>
> If I'm not mis-understanding, it can be the case when
>  *vfs.usermount sysctl is non-zero,
>  *underlying FS (to be exported) allows it, and
>  *non-root user tries to mount the FS via NFS.
>
>
> [1]
>
> https://svnweb.freebsd.org/base/head/sys/kern/vfs_syscalls.c?revision=22521=markup=99264
>
> [2]
>
> https://svnweb.freebsd.org/base/head/sys/kern/vfs_syscalls.c?r1=22520=22521=99264;
>
> [3]
>
> https://cgit.freebsd.org/src/commit/sys/kern/vfs_mount.c?id=2b4edb69f1ef62fc38b02ac22b0a3ac09e43fa77
>
>
> On Tue, 13 Dec 2022 14:19:39 -0800
> Rick Macklem  wrote:
>
> > Hi,
> >
> > While working on getting mountd/nfsd to run in a vnet
> > prison, I came across the following lines near the
> > beginning of vfs_domount() in sys/kern/vfs_mount.c:
> >
> > if (fsflags & MNT_EXPORTED) {
> >  error = priv_check(td, PRIV_VFS_MOUNT_EXPORTED);
> >  if (error)
> >return (error);
> > }
> >
> > #1 - Since MNT_EXPORTED is never set in fsflags, this code never
> >  gets executed.
> >  --> I am asking what to do with the above code, since that
> >  changes for the patch that allows mountd to run in a vnet
> >  prison.
> > #2 - priv_check(td, PRIV_VFS_MOUNT_EXPORTED) always returns 0
> >  because nothing in sys/kern/kern_priv.c checks
> >  PRIV_VFS_MOUNT_EXPORTED.
> >
> > I don't know what the original author's thinking was w.r.t. this.
> > Setting exports already checks that the mount operation can be
> > done by the requestor.
> >
> > So, what do you think should be done with the above code snippet?
> > - Consider it cruft and delete it.
> > - Try and figure out what PRIV_VFS_MOUNT_EXPORTED should check?
> > - Leave it as is. After the patch that allows mountd to run in
> >   a vnet prison, MNT_EXPORTED will be set in fsflags, but the
> >   priv_check() call will just return 0. (A little overhead,
> >   but otherwise no semantics change.)
> >
> > rick
>
>
> --
> Tomoaki AOKI
>
>

Re: What to do about a few lines in vfs_domount() never executed?

[Rick Macklem]

On Wed, Dec 14, 2022 at 2:04 PM Rick Macklem  wrote:

>
>
> On Wed, Dec 14, 2022 at 9:15 AM Tomoaki AOKI 
> wrote:
>
>> Tracking the commits, it was originally introduced to
>> sys/kern/vfs_syscalls.c at r22521 [1][2] (Mon Feb 10 02:22:35 1997 by
>> dyson, submitted by h...@freebsd.org) and later centralized into
>> sys/kern/vfs_mount.c at r99264 [2].
>>
>> But according to the comment above the codes, maybe it would be
>> intended to block userland programs or ports FS modules setting
>> MNT_EXPORTED.
>>
>> If I'm not mis-understanding, it can be the case when
>>  *vfs.usermount sysctl is non-zero,
>>  *underlying FS (to be exported) allows it, and
>>  *non-root user tries to mount the FS via NFS.
>>
> It does appear that ancient code restricted doing NFS exports
> to root only.
>
It looks like the semantics change was introduced when mountd
as converted to nmount() { r158857 about 17years ago }.
The code in vfs_mount.c assumed that MNT_EXPORTED was set in the
argument passed in from mountd.c when it called nmount(), but
that was not the case.
--> As such, the check was not performed.

The check was for suser() until r164033 when it was replaced
by priv_check(td, PRIV_VFS_MOUNT_EDPORTED).
--> This does the same thing, failing if cr_uid != 0.

So, I think the snippet was supposed to enforce "only root
can set exports", but the check has not worked post r158857
because MNT_EXPORTED was never set in fsflags.
(nmount() uses the "export" option.)

So, should I set MNT_EXPORTED in fsflags when nmount()
has specified the "export" option and restore the
"must be root to export" check?

rick
ps: Thanks Tomoaki AOKI for looking at the old code and
 spotting this.

> I don't think that restriction is exactly enforced now, since
> vfs_suser() allows a non-root owner to do the update (which
> would include updating exports).
> (As I noted, MNT_EXPORTED never gets set in fsflags. The variable
>  is local to one of the functions in vfs_mount.c and a search shows
>  it never gets set.)
>
> I suppose you could argue that priv_check(td,  PRIV_VFS_MOUNT_EXPORTED)
> should check for caller being root, since that is what ancient code did.
> Or, you could argue that, if a non-root user is allowed to mount a file
> system that they should also be allowed to export it, which is what I
> think the current code allows (and has for at least a decade).
>
> Admittedly, allowing a non-root user to do a mount and also add exports
> to it could cause confusion, since the system basically assumes that
> mountd will manage all exports.
>
> Do others think adding code to check that cr_uid == 0 for
> PRIV_VFS_MOUNT_EXPORTED to be allowed makes sense?
>
> rick
>
>
>
>>
>> [1]
>>
>> https://svnweb.freebsd.org/base/head/sys/kern/vfs_syscalls.c?revision=22521=markup=99264
>>
>> [2]
>>
>> https://svnweb.freebsd.org/base/head/sys/kern/vfs_syscalls.c?r1=22520=22521=99264;
>>
>> [3]
>>
>> https://cgit.freebsd.org/src/commit/sys/kern/vfs_mount.c?id=2b4edb69f1ef62fc38b02ac22b0a3ac09e43fa77
>>
>>
>> On Tue, 13 Dec 2022 14:19:39 -0800
>> Rick Macklem  wrote:
>>
>> > Hi,
>> >
>> > While working on getting mountd/nfsd to run in a vnet
>> > prison, I came across the following lines near the
>> > beginning of vfs_domount() in sys/kern/vfs_mount.c:
>> >
>> > if (fsflags & MNT_EXPORTED) {
>> >  error = priv_check(td, PRIV_VFS_MOUNT_EXPORTED);
>> >  if (error)
>> >return (error);
>> > }
>> >
>> > #1 - Since MNT_EXPORTED is never set in fsflags, this code never
>> >  gets executed.
>> >  --> I am asking what to do with the above code, since that
>> >  changes for the patch that allows mountd to run in a vnet
>> >      prison.
>> > #2 - priv_check(td, PRIV_VFS_MOUNT_EXPORTED) always returns 0
>> >  because nothing in sys/kern/kern_priv.c checks
>> >  PRIV_VFS_MOUNT_EXPORTED.
>> >
>> > I don't know what the original author's thinking was w.r.t. this.
>> > Setting exports already checks that the mount operation can be
>> > done by the requestor.
>> >
>> > So, what do you think should be done with the above code snippet?
>> > - Consider it cruft and delete it.
>> > - Try and figure out what PRIV_VFS_MOUNT_EXPORTED should check?
>> > - Leave it as is. After the patch that allows mountd to run in
>> >   a vnet prison, MNT_EXPORTED will be set in fsflags, but the
>> >   priv_check() call will just return 0. (A little overhead,
>> >   but otherwise no semantics change.)
>> >
>> > rick
>>
>>
>> --
>> Tomoaki AOKI
>>
>>

Re: What to do about a few lines in vfs_domount() never executed?

[Rick Macklem]

On Wed, Dec 14, 2022 at 9:15 AM Tomoaki AOKI 
wrote:

> Tracking the commits, it was originally introduced to
> sys/kern/vfs_syscalls.c at r22521 [1][2] (Mon Feb 10 02:22:35 1997 by
> dyson, submitted by h...@freebsd.org) and later centralized into
> sys/kern/vfs_mount.c at r99264 [2].
>
> But according to the comment above the codes, maybe it would be
> intended to block userland programs or ports FS modules setting
> MNT_EXPORTED.
>
> If I'm not mis-understanding, it can be the case when
>  *vfs.usermount sysctl is non-zero,
>  *underlying FS (to be exported) allows it, and
>  *non-root user tries to mount the FS via NFS.
>
It does appear that ancient code restricted doing NFS exports
to root only.
I don't think that restriction is exactly enforced now, since
vfs_suser() allows a non-root owner to do the update (which
would include updating exports).
(As I noted, MNT_EXPORTED never gets set in fsflags. The variable
 is local to one of the functions in vfs_mount.c and a search shows
 it never gets set.)

I suppose you could argue that priv_check(td,  PRIV_VFS_MOUNT_EXPORTED)
should check for caller being root, since that is what ancient code did.
Or, you could argue that, if a non-root user is allowed to mount a file
system that they should also be allowed to export it, which is what I
think the current code allows (and has for at least a decade).

Admittedly, allowing a non-root user to do a mount and also add exports
to it could cause confusion, since the system basically assumes that
mountd will manage all exports.

Do others think adding code to check that cr_uid == 0 for
PRIV_VFS_MOUNT_EXPORTED to be allowed makes sense?

rick



>
> [1]
>
> https://svnweb.freebsd.org/base/head/sys/kern/vfs_syscalls.c?revision=22521=markup=99264
>
> [2]
>
> https://svnweb.freebsd.org/base/head/sys/kern/vfs_syscalls.c?r1=22520=22521=99264;
>
> [3]
>
> https://cgit.freebsd.org/src/commit/sys/kern/vfs_mount.c?id=2b4edb69f1ef62fc38b02ac22b0a3ac09e43fa77
>
>
> On Tue, 13 Dec 2022 14:19:39 -0800
> Rick Macklem  wrote:
>
> > Hi,
> >
> > While working on getting mountd/nfsd to run in a vnet
> > prison, I came across the following lines near the
> > beginning of vfs_domount() in sys/kern/vfs_mount.c:
> >
> > if (fsflags & MNT_EXPORTED) {
> >  error = priv_check(td, PRIV_VFS_MOUNT_EXPORTED);
> >  if (error)
> >    return (error);
> > }
> >
> > #1 - Since MNT_EXPORTED is never set in fsflags, this code never
> >  gets executed.
> >  --> I am asking what to do with the above code, since that
> >  changes for the patch that allows mountd to run in a vnet
> >  prison.
> > #2 - priv_check(td, PRIV_VFS_MOUNT_EXPORTED) always returns 0
> >  because nothing in sys/kern/kern_priv.c checks
> >  PRIV_VFS_MOUNT_EXPORTED.
> >
> > I don't know what the original author's thinking was w.r.t. this.
> > Setting exports already checks that the mount operation can be
> > done by the requestor.
> >
> > So, what do you think should be done with the above code snippet?
> > - Consider it cruft and delete it.
> > - Try and figure out what PRIV_VFS_MOUNT_EXPORTED should check?
> > - Leave it as is. After the patch that allows mountd to run in
> >   a vnet prison, MNT_EXPORTED will be set in fsflags, but the
> >   priv_check() call will just return 0. (A little overhead,
> >   but otherwise no semantics change.)
> >
> > rick
>
>
> --
> Tomoaki AOKI
>
>

Re: What to do about a few lines in vfs_domount() never executed?

[Tomoaki AOKI]

Tracking the commits, it was originally introduced to
sys/kern/vfs_syscalls.c at r22521 [1][2] (Mon Feb 10 02:22:35 1997 by
dyson, submitted by h...@freebsd.org) and later centralized into
sys/kern/vfs_mount.c at r99264 [2].

But according to the comment above the codes, maybe it would be
intended to block userland programs or ports FS modules setting 
MNT_EXPORTED.

If I'm not mis-understanding, it can be the case when
 *vfs.usermount sysctl is non-zero,
 *underlying FS (to be exported) allows it, and
 *non-root user tries to mount the FS via NFS.


[1]
https://svnweb.freebsd.org/base/head/sys/kern/vfs_syscalls.c?revision=22521=markup=99264

[2]
https://svnweb.freebsd.org/base/head/sys/kern/vfs_syscalls.c?r1=22520=22521=99264;

[3]
https://cgit.freebsd.org/src/commit/sys/kern/vfs_mount.c?id=2b4edb69f1ef62fc38b02ac22b0a3ac09e43fa77


On Tue, 13 Dec 2022 14:19:39 -0800
Rick Macklem  wrote:

> Hi,
> 
> While working on getting mountd/nfsd to run in a vnet
> prison, I came across the following lines near the
> beginning of vfs_domount() in sys/kern/vfs_mount.c:
> 
> if (fsflags & MNT_EXPORTED) {
>  error = priv_check(td, PRIV_VFS_MOUNT_EXPORTED);
>  if (error)
>return (error);
> }
> 
> #1 - Since MNT_EXPORTED is never set in fsflags, this code never
>  gets executed.
>  --> I am asking what to do with the above code, since that
>  changes for the patch that allows mountd to run in a vnet
>  prison.
> #2 - priv_check(td, PRIV_VFS_MOUNT_EXPORTED) always returns 0
>  because nothing in sys/kern/kern_priv.c checks
>  PRIV_VFS_MOUNT_EXPORTED.
> 
> I don't know what the original author's thinking was w.r.t. this.
> Setting exports already checks that the mount operation can be
> done by the requestor.
> 
> So, what do you think should be done with the above code snippet?
> - Consider it cruft and delete it.
> - Try and figure out what PRIV_VFS_MOUNT_EXPORTED should check?
> - Leave it as is. After the patch that allows mountd to run in
>   a vnet prison, MNT_EXPORTED will be set in fsflags, but the
>   priv_check() call will just return 0. (A little overhead,
>   but otherwise no semantics change.)
> 
> rick


-- 
Tomoaki AOKI

Re: What to do about a few lines in vfs_domount() never executed?

[Alan Somers]

If you're 100.0% percent sure that it will never be run, you can
delete it.  But if you're only 99.9%, then I suggest converting it
into a KASSERT.

On Tue, Dec 13, 2022 at 3:20 PM Rick Macklem  wrote:
>
> Hi,
>
> While working on getting mountd/nfsd to run in a vnet
> prison, I came across the following lines near the
> beginning of vfs_domount() in sys/kern/vfs_mount.c:
>
> if (fsflags & MNT_EXPORTED) {
>  error = priv_check(td, PRIV_VFS_MOUNT_EXPORTED);
>  if (error)
>return (error);
> }
>
> #1 - Since MNT_EXPORTED is never set in fsflags, this code never
>  gets executed.
>  --> I am asking what to do with the above code, since that
>  changes for the patch that allows mountd to run in a vnet
>  prison.
> #2 - priv_check(td, PRIV_VFS_MOUNT_EXPORTED) always returns 0
>  because nothing in sys/kern/kern_priv.c checks
>  PRIV_VFS_MOUNT_EXPORTED.
>
> I don't know what the original author's thinking was w.r.t. this.
> Setting exports already checks that the mount operation can be
> done by the requestor.
>
> So, what do you think should be done with the above code snippet?
> - Consider it cruft and delete it.
> - Try and figure out what PRIV_VFS_MOUNT_EXPORTED should check?
> - Leave it as is. After the patch that allows mountd to run in
>   a vnet prison, MNT_EXPORTED will be set in fsflags, but the
>   priv_check() call will just return 0. (A little overhead,
>   but otherwise no semantics change.)
>
> rick
>

What to do about a few lines in vfs_domount() never executed?

[Rick Macklem]

Hi,

While working on getting mountd/nfsd to run in a vnet
prison, I came across the following lines near the
beginning of vfs_domount() in sys/kern/vfs_mount.c:

if (fsflags & MNT_EXPORTED) {
 error = priv_check(td, PRIV_VFS_MOUNT_EXPORTED);
 if (error)
   return (error);
}

#1 - Since MNT_EXPORTED is never set in fsflags, this code never
 gets executed.
 --> I am asking what to do with the above code, since that
 changes for the patch that allows mountd to run in a vnet
 prison.
#2 - priv_check(td, PRIV_VFS_MOUNT_EXPORTED) always returns 0
 because nothing in sys/kern/kern_priv.c checks
 PRIV_VFS_MOUNT_EXPORTED.

I don't know what the original author's thinking was w.r.t. this.
Setting exports already checks that the mount operation can be
done by the requestor.

So, what do you think should be done with the above code snippet?
- Consider it cruft and delete it.
- Try and figure out what PRIV_VFS_MOUNT_EXPORTED should check?
- Leave it as is. After the patch that allows mountd to run in
  a vnet prison, MNT_EXPORTED will be set in fsflags, but the
  priv_check() call will just return 0. (A little overhead,
  but otherwise no semantics change.)

rick

Re: 13-STABLE: can not cross build 13.0-RELENG and 13.1-RELENG, what on CURRENT works fine

[FreeBSD User]

On Tue, 19 Apr 2022 06:48:04 -0600
Warner Losh  wrote:

Hello again,

I'm on 13-STABLE, version FreeBSD 13.1-STABLE #30
stable/13-n250552-364a69a529b: Tue Apr 26 07:32:42 CEST 2022 amd64 (builder
host). Sources for a NanoBSD of 13.0-RELENG (latest as of today) doesn't build,
still the reported [vdso] error (see below).

> The vdso error is fixed by cherry-picking
> b3b462229f972e2ed24d450d7d2f8855cdd58a87.

Still does not build.

> I'm not sure about the second error, though if it's a general profile
> error, you make need WITHOUT_PROFILE=t in both the build and install phases.

This is now working on the above reported builder host version. The key was,
indeed, setting WITHOUT_PROFILE=t. Thank you for the hint.

On CURRENT I can build both, 13.0-RELENG and 13.1-RELENG without issues.

Kind regards,

Oliver Hartmann

> 
> 
> On Tue, Apr 19, 2022 at 5:46 AM FreeBSD User  wrote:
> 
> > I regular build for a NanoBSD appliance 13-STABLE, 13.0-RELENG and
> > 13.1-RELENG
> > on either 14-CURRENT and 13-STABLE. Several days ago, some changes has been
> > made to /usr/src/Makefile.inc1, first on CURRENT and shortly after on 13.
> >
> > As with today, building from sources either 13.1-RELENG and 13.0-RELENG on
> > a
> > CURRENT host (recent 14-CURRENT!) works without problems.
> >
> > On 13-STABLE (FreeBSD 13.1-STABLE #26 stable/13-n250478-bb8e1dfbff3: Thu
> > Apr 14
> > 10:47:51 CEST 2022 amd64) building either 13.0-RELENG or 13.1-RELENG fails!
> >
> > On building from sources 13.0-RELENG I receive while in installworld:
> >
> > [...]
> > --  
> > >>> Install check world  
> > --
> > mkdir -p /tmp/install.6R4Ifq8o
> > ...
> > cp: [vdso]: No such file or directory
> > *** Error code 1
> > [...]
> >
> > and on building from sources 13.1-RELENG while in installworld we get:
> >
> > [...]  
> > ===> usr.bin/lex/lib (install)  
> > install -l h -o root -g wheel -m 444
> >
> > /home/ohartmann/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/lib
> > ln_p.a
> >
> > /home/ohartmann/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/libl_p.a
> > install: link
> > /home/ohartmann/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/libln_p.a
> > -> /home/ohartman  
> >
> > n/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/libl_p.a:
> > No such file or directory
> > *** Error code 71
> >
> >
> > Thanks in advance,
> >
> > O. Hartmann
> >
> >  

Re: 13-STABLE: can not cross build 13.0-RELENG and 13.1-RELENG, what on CURRENT works fine

[Warner Losh]

The vdso error is fixed by cherry-picking
b3b462229f972e2ed24d450d7d2f8855cdd58a87.
I'm not sure about the second error, though if it's a general profile
error, you make need WITHOUT_PROFILE=t in both the build and install phases.


On Tue, Apr 19, 2022 at 5:46 AM FreeBSD User  wrote:

> I regular build for a NanoBSD appliance 13-STABLE, 13.0-RELENG and
> 13.1-RELENG
> on either 14-CURRENT and 13-STABLE. Several days ago, some changes has been
> made to /usr/src/Makefile.inc1, first on CURRENT and shortly after on 13.
>
> As with today, building from sources either 13.1-RELENG and 13.0-RELENG on
> a
> CURRENT host (recent 14-CURRENT!) works without problems.
>
> On 13-STABLE (FreeBSD 13.1-STABLE #26 stable/13-n250478-bb8e1dfbff3: Thu
> Apr 14
> 10:47:51 CEST 2022 amd64) building either 13.0-RELENG or 13.1-RELENG fails!
>
> On building from sources 13.0-RELENG I receive while in installworld:
>
> [...]
> --
> >>> Install check world
> --
> mkdir -p /tmp/install.6R4Ifq8o
> ...
> cp: [vdso]: No such file or directory
> *** Error code 1
> [...]
>
> and on building from sources 13.1-RELENG while in installworld we get:
>
> [...]
> ===> usr.bin/lex/lib (install)
> install -l h -o root -g wheel -m 444
>
> /home/ohartmann/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/lib
> ln_p.a
>
> /home/ohartmann/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/libl_p.a
> install: link
> /home/ohartmann/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/libln_p.a
> -> /home/ohartman
>
> n/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/libl_p.a:
> No such file or directory
> *** Error code 71
>
>
> Thanks in advance,
>
> O. Hartmann
>
>

13-STABLE: can not cross build 13.0-RELENG and 13.1-RELENG, what on CURRENT works fine

[FreeBSD User]

I regular build for a NanoBSD appliance 13-STABLE, 13.0-RELENG and 13.1-RELENG
on either 14-CURRENT and 13-STABLE. Several days ago, some changes has been
made to /usr/src/Makefile.inc1, first on CURRENT and shortly after on 13.

As with today, building from sources either 13.1-RELENG and 13.0-RELENG on a
CURRENT host (recent 14-CURRENT!) works without problems.

On 13-STABLE (FreeBSD 13.1-STABLE #26 stable/13-n250478-bb8e1dfbff3: Thu Apr 14
10:47:51 CEST 2022 amd64) building either 13.0-RELENG or 13.1-RELENG fails!

On building from sources 13.0-RELENG I receive while in installworld:

[...]
--
>>> Install check world
--
mkdir -p /tmp/install.6R4Ifq8o
...
cp: [vdso]: No such file or directory
*** Error code 1
[...]

and on building from sources 13.1-RELENG while in installworld we get:

[...]
===> usr.bin/lex/lib (install)
install -l h -o root -g wheel -m 444
/home/ohartmann/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/lib
ln_p.a
/home/ohartmann/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/libl_p.a
install: link
/home/ohartmann/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/libln_p.a
 -> /home/ohartman
n/Projects/TEST/fbsd13.1-dev/os-base/systemconf/../../world/amd64/TEST-DEV-amd64-13.1-RELENG/_.w/usr/lib/libl_p.a:
No such file or directory
*** Error code 71


Thanks in advance,

O. Hartmann

Re: What are the in-kernel functions to print human readable timestamps (bintime)?

[Alexander Leidinger]

Quoting Warner Losh  (from Fri, 11 Mar 2022 08:57:33 -0700):


On Fri, Mar 11, 2022 at 2:52 AM Alexander Leidinger 
wrote:


Hi,

I'm looking for a function to convert bintime to a human readable
format in the kernel... and what is the usual format we use?



Yes. We don't generally log it.


Would it be acceptable in this particular case (or should I keep this  
change for me)?


I have to check out the kern.msgbuf_show_timestamp part in this  
thread, which looks interesting. It may fit my needs here.



The use case for this is: if something throws a log from the kernel
about a signal, I want to know when it happened, or in terms of code
see below (tabs are most probably messed up).

Do we have some kind of policy in terms of kernel messages and
timestamps? Like "do not commit logging with timestamps"?



Correct. The kernel doesn't know enough to properly render timestamps,
nor should it. It's a lot more complicated than you'd expect, and the
simple,
naive implementation has too many flaws...


I'm aware that it is complicated. IMO too complicated for a full  
implementation which will satisfy all needs.



I have the
code below because I needed it at least once and think something like
this (in a human readably shape) would be beneficial to have in the
tree.



I really don't want to see code like that in the tree. Having it per-message
in an ad-hoc manner strikes me as quite unwise, since how do you interpret
it after the fact, especially in the face of adjustments to boottime for any
time adjustments that happen after the event.


Sorry, I was not verbose enough in my mail it seems. I do _not_ want  
to commit this code like it is. I was looking for stuff which could  
print something human readable, like "2022-12-03 14:23:22.45  
'kernel-time'" (just as an example). I don't want to push the TZ into  
the kernel, or the knowledge if the cmos clock is set to localtime or  
UTC. I want to provide an admin with a way to determine when such a  
message may have printed. Currently there is no way to know at which  
time it was printed. The admin needs to know if the clock is set in  
UTC or localtime, and how to calculate the wall-clock time from this,  
which is less heavy on the implementation and provides a mean to  
correlate the mesage with some action on the machine.



Now, having said that, having good timestamps in the dmesg buffer is
a desirable feature. 'Good' is subjective here, and there are times early
in boot where it's simply not possible to get a time better than '0' until
the timehands are ticking...  But the dmesg buffer has more than what
dmesg prints: it has syslog'd things (at least some of them) as well.
There's
also a priority that some lines have. <3>Foo, for example. It would be
better,
imho, to add a timestamp to the start of the lines (perhaps optionally
since that might be expensive in $HUGE systems, or at times of
extreme dmesg spew and the could be omitted in those cases).
If you are interested in just the log messages, it wouldn't be terrible
since we already add stuff to what's printed for the priority. We could say
<3,seconds-since-boot.fracsec> instead of just <3> and hack dmesg
to print the right thing.


From the other message in the thread it looks like  
kern.msgbuf_show_timestamp is what you describe here?
And it looks like kern.msgbuf_show_timestamp is not the same as  
printing a timestamp to the console... (it looks like the timestamp is  
printed in the dmesg, but not to the console, which is what I have in  
mind for this particular message... respectively could be added log()  
with a sysctl to activate it or not).


Bye,
Alexander.


Warner



Code:
---snip---
diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c
index 4a15bd45355..a83eebe0736 100644
--- a/sys/kern/kern_sig.c
+++ b/sys/kern/kern_sig.c
@@ -80,6 +80,7 @@ __FBSDID("$FreeBSD$");
  #include 
  #include 
  #include 
+#include 
  #include 
  #include 
  #include 
@@ -3440,14 +3441,18 @@ sigexit(struct thread *td, int sig)
  */
 if (coredump(td) == 0)
 sig |= WCOREFLAG;
-   if (kern_logsigexit)
+   if (kern_logsigexit) {
+   struct bintime now;
+
+   getbintime();
 log(LOG_INFO,
-   "pid %d (%s), jid %d, uid %d: exited on "
-   "signal %d%s\n", p->p_pid, p->p_comm,
+   "%zd: pid %d (%s), jid %d, uid %d: exited on "
+   "signal %d%s\n", now.sec, p->p_pid, p->p_comm,
 p->p_ucred->cr_prison->pr_id,
 td->td_ucred->cr_uid,
 sig &~ WCOREFLAG,
 sig & WCOREFLAG ? " (core dumped)" : "");
+   }
 } els

Re: What are the in-kernel functions to print human readable timestamps (bintime)?

[Jamie Landeg-Jones]

Warner Losh  wrote:

> since we already add stuff to what's printed for the priority. We could say
> <3,seconds-since-boot.fracsec> instead of just <3> and hack dmesg
> to print the right thing.

Isn't that what kern.msgbuf_show_timestamp does already?

I use that, along with this script:

17:15 (40.0°C 400) [2] (49) "completions" root@thompson# cat 
/usr/common/bin/dmesg-uptime-to-date
#!/bin/sh -efu
set -efu

boottime="$(sysctl -n kern.boottime | gawk '{printf "%d", gensub ("^.* sec = 
([1-9][0-9]*), .*$", "\\1", 1)}')"

[ -z "$(printf '%s' "$boottime" | egrep '^0$|^[1-9][0-9]*$')" ] && { printf 
'Invalid boottime retrieved.\n' >& 2; exit 1; }

dmesg "$@" | gawk -v boottime="$boottime" '

{
  uptime = gensub ("^\\[([1-9][0-9]*)\\] .*$", "\\1", 1)
  if (uptime == $0) realtime = "??? ?? ??:??;??"
   else realtime = strftime ("%b %d %T", uptime + boottime)

  print realtime " " $0
}'
 

Mar 11 00:41:51 [3568757] Limiting closed port RST response from 313 to 200 
packets/sec
Mar 11 00:41:54 [3568760] Limiting closed port RST response from 308 to 200 
packets/sec
Mar 11 06:23:28 [3589254] icmp redirect from 183.196.23.176: 192.168.2.104 => 
183.196.23.161

etc.

Cheers, Jamie

Re: What are the in-kernel functions to format time?

[Hans Petter Selasky]

On 3/11/22 12:20, Gary Jennejohn wrote:

Do you mean the %zd?  kvprintf() checks for a zflag and handles the
argument as size_t or ssize_t, depending on whether the sign is
positive or negative.


Hi,

Given that time is a 64-bit value, then probably "%llu", and (unsigned 
long long)bintime would do it, but then you need that cast.


./_types.h:typedef  __int64_t   __sbintime_t;

I was tinking of a %XXX that divides the value somehow into something 
more readable, maybe suffixing "ns" or "ms" or "us" or something.


--HPS

Re: What are the in-kernel functions to print human readable timestamps (bintime)?

[Warner Losh]

On Fri, Mar 11, 2022 at 2:52 AM Alexander Leidinger 
wrote:

> Hi,
>
> I'm looking for a function to convert bintime to a human readable
> format in the kernel... and what is the usual format we use?
>

Yes. We don't generally log it.


>
> The use case for this is: if something throws a log from the kernel
> about a signal, I want to know when it happened, or in terms of code
> see below (tabs are most probably messed up).
>
> Do we have some kind of policy in terms of kernel messages and
> timestamps? Like "do not commit logging with timestamps"?


Correct. The kernel doesn't know enough to properly render timestamps,
nor should it. It's a lot more complicated than you'd expect, and the
simple,
naive implementation has too many flaws...


> I have the
> code below because I needed it at least once and think something like
> this (in a human readably shape) would be beneficial to have in the
> tree.
>

I really don't want to see code like that in the tree. Having it per-message
in an ad-hoc manner strikes me as quite unwise, since how do you interpret
it after the fact, especially in the face of adjustments to boottime for any
time adjustments that happen after the event.

Now, having said that, having good timestamps in the dmesg buffer is
a desirable feature. 'Good' is subjective here, and there are times early
in boot where it's simply not possible to get a time better than '0' until
the timehands are ticking...  But the dmesg buffer has more than what
dmesg prints: it has syslog'd things (at least some of them) as well.
There's
also a priority that some lines have. <3>Foo, for example. It would be
better,
imho, to add a timestamp to the start of the lines (perhaps optionally
since that might be expensive in $HUGE systems, or at times of
extreme dmesg spew and the could be omitted in those cases).
If you are interested in just the log messages, it wouldn't be terrible
since we already add stuff to what's printed for the priority. We could say
<3,seconds-since-boot.fracsec> instead of just <3> and hack dmesg
to print the right thing.

Warner


> Code:
> ---snip---
> diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c
> index 4a15bd45355..a83eebe0736 100644
> --- a/sys/kern/kern_sig.c
> +++ b/sys/kern/kern_sig.c
> @@ -80,6 +80,7 @@ __FBSDID("$FreeBSD$");
>   #include 
>   #include 
>   #include 
> +#include 
>   #include 
>   #include 
>   #include 
> @@ -3440,14 +3441,18 @@ sigexit(struct thread *td, int sig)
>   */
>  if (coredump(td) == 0)
>  sig |= WCOREFLAG;
> -   if (kern_logsigexit)
> +   if (kern_logsigexit) {
> +   struct bintime now;
> +
> +   getbintime();
>  log(LOG_INFO,
> -   "pid %d (%s), jid %d, uid %d: exited on "
> -   "signal %d%s\n", p->p_pid, p->p_comm,
> +   "%zd: pid %d (%s), jid %d, uid %d: exited on "
> +   "signal %d%s\n", now.sec, p->p_pid, p->p_comm,
>  p->p_ucred->cr_prison->pr_id,
>  td->td_ucred->cr_uid,
>  sig &~ WCOREFLAG,
>  sig & WCOREFLAG ? " (core dumped)" : "");
> +   }
>  } else
>  PROC_UNLOCK(p);
>  exit1(td, 0, sig);
> ---snip---
>
> Bye,
> Alexander.
>
> --
> http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
> http://www.FreeBSD.orgnetch...@freebsd.org  : PGP 0x8F31830F9F2772BF
>

Re: What are the in-kernel functions to format time?

[Gary Jennejohn]

On Fri, 11 Mar 2022 11:01:03 +0100
Hans Petter Selasky  wrote:

> On 3/11/22 10:49, Alexander Leidinger wrote:
> > Hi,
> > 
> > I'm looking for a function to convert bintime to a human readable format > 
> > in the kernel... and what is the usual format we use?
> > 
> > 
> > The use case for this is: if something throws a log from the kernel > about 
> > a signal, I want to know when it happened, or in terms of code see > below 
> > (tabs are most probably messed up).
> > 
> > Do we have some kind of policy in terms of kernel messages and > 
> > timestamps? Like "do not commit logging with timestamps"? I have the > code 
> > below because I needed it at least once and think something like > this (in 
> > a human readably shape) would be beneficial to have in the tree.
> >   
> 
> Hi,
> 
> I think our kernel printer doesn't support this:
> 
> sys/kern/subr_prf.c
> 

Do you mean the %zd?  kvprintf() checks for a zflag and handles the
argument as size_t or ssize_t, depending on whether the sign is
positive or negative.

However, %n isn't supported.

> If you need to extend the format, please check other OS'es too, like
> OpenBSD, NetBSD and Linux, what they support, so there won't be any
> obvious conflicts when moving code cross platforms!
> 


-- 
Gary Jennejohn

Re: What are the in-kernel functions to format time?

[Hans Petter Selasky]

On 3/11/22 10:49, Alexander Leidinger wrote:

Hi,

I'm looking for a function to convert bintime to a human readable format 
in the kernel... and what is the usual format we use?



The use case for this is: if something throws a log from the kernel 
about a signal, I want to know when it happened, or in terms of code see 
below (tabs are most probably messed up).


Do we have some kind of policy in terms of kernel messages and 
timestamps? Like "do not commit logging with timestamps"? I have the 
code below because I needed it at least once and think something like 
this (in a human readably shape) would be beneficial to have in the tree.




Hi,

I think our kernel printer doesn't support this:

sys/kern/subr_prf.c

If you need to extend the format, please check other OS'es too, like 
OpenBSD, NetBSD and Linux, what they support, so there won't be any 
obvious conflicts when moving code cross platforms!


--HPS

What are the in-kernel functions to print human readable timestamps (bintime)?

[Alexander Leidinger]

Hi,

I'm looking for a function to convert bintime to a human readable  
format in the kernel... and what is the usual format we use?



The use case for this is: if something throws a log from the kernel  
about a signal, I want to know when it happened, or in terms of code  
see below (tabs are most probably messed up).


Do we have some kind of policy in terms of kernel messages and  
timestamps? Like "do not commit logging with timestamps"? I have the  
code below because I needed it at least once and think something like  
this (in a human readably shape) would be beneficial to have in the  
tree.


Code:
---snip---
diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c
index 4a15bd45355..a83eebe0736 100644
--- a/sys/kern/kern_sig.c
+++ b/sys/kern/kern_sig.c
@@ -80,6 +80,7 @@ __FBSDID("$FreeBSD$");
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -3440,14 +3441,18 @@ sigexit(struct thread *td, int sig)
 */
if (coredump(td) == 0)
sig |= WCOREFLAG;
-   if (kern_logsigexit)
+   if (kern_logsigexit) {
+   struct bintime now;
+
+   getbintime();
log(LOG_INFO,
-   "pid %d (%s), jid %d, uid %d: exited on "
-   "signal %d%s\n", p->p_pid, p->p_comm,
+   "%zd: pid %d (%s), jid %d, uid %d: exited on "
+   "signal %d%s\n", now.sec, p->p_pid, p->p_comm,
p->p_ucred->cr_prison->pr_id,
td->td_ucred->cr_uid,
sig &~ WCOREFLAG,
sig & WCOREFLAG ? " (core dumped)" : "");
+   }
} else
PROC_UNLOCK(p);
exit1(td, 0, sig);
---snip---

Bye,
Alexander.

--
http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.orgnetch...@freebsd.org  : PGP 0x8F31830F9F2772BF


pgpG9ftB1Te63.pgp
Description: Digitale PGP-Signatur

What are the in-kernel functions to format time?

[Alexander Leidinger]

Hi,

I'm looking for a function to convert bintime to a human readable  
format in the kernel... and what is the usual format we use?



The use case for this is: if something throws a log from the kernel  
about a signal, I want to know when it happened, or in terms of code  
see below (tabs are most probably messed up).


Do we have some kind of policy in terms of kernel messages and  
timestamps? Like "do not commit logging with timestamps"? I have the  
code below because I needed it at least once and think something like  
this (in a human readably shape) would be beneficial to have in the  
tree.


Code:
---snip---
diff --git a/sys/kern/kern_sig.c b/sys/kern/kern_sig.c
index 4a15bd45355..a83eebe0736 100644
--- a/sys/kern/kern_sig.c
+++ b/sys/kern/kern_sig.c
@@ -80,6 +80,7 @@ __FBSDID("$FreeBSD$");
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -3440,14 +3441,18 @@ sigexit(struct thread *td, int sig)
 */
if (coredump(td) == 0)
sig |= WCOREFLAG;
-   if (kern_logsigexit)
+   if (kern_logsigexit) {
+   struct bintime now;
+
+   getbintime();
log(LOG_INFO,
-   "pid %d (%s), jid %d, uid %d: exited on "
-   "signal %d%s\n", p->p_pid, p->p_comm,
+   "%zd: pid %d (%s), jid %d, uid %d: exited on "
+   "signal %d%s\n", now.sec, p->p_pid, p->p_comm,
p->p_ucred->cr_prison->pr_id,
td->td_ucred->cr_uid,
sig &~ WCOREFLAG,
sig & WCOREFLAG ? " (core dumped)" : "");
+   }
} else
PROC_UNLOCK(p);
exit1(td, 0, sig);
---snip---

Bye,
Alexander.

--
http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.orgnetch...@freebsd.org  : PGP 0x8F31830F9F2772BF


pgpiH30aWZMth.pgp
Description: Digitale PGP-Signatur

Re: WITH_ASAN= vs. vfork: ASAN can not deal with vfork in a reliable manor, so what to do to avoid vfork fairly generally, including for kyua?

[Mark Millard]

On 2022-Jan-15, at 15:25, Mark Millard  wrote:

> ASAN is documented to not be able to deal reliably with vfork use
> (inaccurate results, result mixing interpretations of the old
> and new contexts, and so on). [There may be other routines
> sufficiently analogous to vfork to have the same sorts of issues.
> I'll write only of vfork explicitly.]
> 
> It turns out that using SH_DISABLE_VFORK= with kyua runs
> helps avoid a bunch of junk failure ASAN reports that involve
> /bin/sh using vfork. So I've been recently using the likes of:
> 
> env SH_DISABLE_VFORK= ASAN_OPTIONS=detect_container_overflow=0 \
> kyua test -k /usr/tests/Kyuafile
> 
> in my kyua runs. But that need not cover all the vfork use in
> the kyua runs --or in general system operation.
> 
> 
> 
> There is more that could be done. For example . . .
> 
> contrib/llvm-project/compiler-rt/lib/asan/asan_interceptors.h
> currently has:
> 
> #if SANITIZER_LINUX &&\
>(defined(__arm__) || defined(__aarch64__) || defined(__i386__) || \
> defined(__x86_64__) || SANITIZER_RISCV64)
> # define ASAN_INTERCEPT_VFORK 1
> #else
> # define ASAN_INTERCEPT_VFORK 0
> #endif
> 
> The "# define ASAN_INTERCEPT_VFORK 1" causes interception to
> use fork for the vfork implementation, instead of an actual
> vfork.
> 
> It looks to me like asan_interceptors.h for FreeBSD should
> also be using:
> 
> # define ASAN_INTERCEPT_VFORK 1
> 
> but it currently is not.

Well, looking around there does not seem to be a FreeBSD-supporting
implementation to go with use of "# define ASAN_INTERCEPT_VFORK 1".
So there would be more to it than just causing the #define to happen.

Thus, there does not seem to be a quick way for me to see what would
result in the kyua run from more avoiding of vfork use (beyond what
use of SH_DISABLE_VFORK= did).

> There is even the possibility that a WITH_ASAN= buildworld could
> build a world that uses fork behavior for vfork and never does an
> actual vfork (by behavior). Basically restricting itself to things
> that fit with ASAN use.
> 
> 
> 
> I tracked this issue down via a report in a kyua run that
> referenced:
> 
> QUOTE
> ERROR: AddressSanitizer: stack-buffer-underflow on address 0x7fffa580 at 
> pc 0x000801e0f8ed bp 0x7fff9bf0 sp 0x7fff9be8
> WRITE of size 8 at 0x7fffa580 thread T0
> . . .
> Address 0x7fffa580 is located in stack of thread T0 at offset 0 in frame
>#0 0x7fffa5ef  ()
> 
>  This frame has 1 object(s):
>[32, 288) 'buf' (line 180)
> HINT: this may be a false positive if your program uses some custom stack 
> unwind mechanism, swapcontext or vfork
>  (longjmp and C++ exceptions *are* supported)
> END QUOTE
> 
> The "'buf' (line 180)" turned out to be from the declaration in
> bin/sh/exec.c 's tryexec :
> 
> static void
> tryexec(char *cmd, char **argv, char **envp)
> {
>int e, in;
>ssize_t n;
>char buf[256];
> 
>execve(cmd, argv, envp);
>e = errno;
>if (e == ENOEXEC) {
>INTOFF;
>in = open(cmd, O_RDONLY | O_NONBLOCK);
>if (in != -1) {
>n = pread(in, buf, sizeof buf, 0);
>close(in);
>if (n > 0 && isbinary(buf, n)) {
>errno = ENOEXEC;
>return;
>}
>}
>*argv = cmd;
>*--argv = __DECONST(char *, _PATH_BSHELL);
>execve(_PATH_BSHELL, argv, envp);
>}
>errno = e;
> }
> 
> So I could finally see/validate that an example
> stack-buffer-* report was tied to vfork handling
> (analyzing other code related to the tryexec use).
> 
> The report happens well after tryexec did its
> execve. The information about buf is just old,
> no longer accurate information, leading to a
> false report:
> 
> 
> ==87961==ERROR: AddressSanitizer: stack-buffer-underflow on address 
> 0x7fffa580 at pc 0x000801e0f8ed bp 0x7fff9bf0 sp 0x7fff9be8
> WRITE of size 8 at 0x7fffa580 thread T0
>#0 0x801e0f8ec in bintime2timespec 
> /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/main-src/amd64.amd64/tmp/usr/include/sys/time.h:285:14
>#1 0x801e0f8ec in __vdso_clock_gettime 
> /usr/main-src/lib/libc/sys/__vdso_gettimeofday.c:195:2
>#2 0x801e0e1a0 in clock_gettime 
> /usr/main-src/lib/libc/sys/clock_gettime.c:48:11
>#3 0x10d54da in clock_gettime 
> /usr/main-src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common

WITH_ASAN= vs. vfork: ASAN can not deal with vfork in a reliable manor, so what to do to avoid vfork fairly generally, including for kyua?

[Mark Millard]

ASAN is documented to not be able to deal reliably with vfork use
(inaccurate results, result mixing interpretations of the old
and new contexts, and so on). [There may be other routines
sufficiently analogous to vfork to have the same sorts of issues.
I'll write only of vfork explicitly.]

It turns out that using SH_DISABLE_VFORK= with kyua runs
helps avoid a bunch of junk failure ASAN reports that involve
/bin/sh using vfork. So I've been recently using the likes of:

env SH_DISABLE_VFORK= ASAN_OPTIONS=detect_container_overflow=0 \
kyua test -k /usr/tests/Kyuafile

in my kyua runs. But that need not cover all the vfork use in
the kyua runs --or in general system operation.



There is more that could be done. For example . . .

contrib/llvm-project/compiler-rt/lib/asan/asan_interceptors.h
currently has:

#if SANITIZER_LINUX &&\
(defined(__arm__) || defined(__aarch64__) || defined(__i386__) || \
 defined(__x86_64__) || SANITIZER_RISCV64)
# define ASAN_INTERCEPT_VFORK 1
#else
# define ASAN_INTERCEPT_VFORK 0
#endif

The "# define ASAN_INTERCEPT_VFORK 1" causes interception to
use fork for the vfork implementation, instead of an actual
vfork.

It looks to me like asan_interceptors.h for FreeBSD should
also be using:

# define ASAN_INTERCEPT_VFORK 1

but it currently is not.



There is even the possibility that a WITH_ASAN= buildworld could
build a world that uses fork behavior for vfork and never does an
actual vfork (by behavior). Basically restricting itself to things
that fit with ASAN use.



I tracked this issue down via a report in a kyua run that
referenced:

QUOTE
ERROR: AddressSanitizer: stack-buffer-underflow on address 0x7fffa580 at pc 
0x000801e0f8ed bp 0x7fff9bf0 sp 0x7fff9be8
WRITE of size 8 at 0x7fffa580 thread T0
. . .
Address 0x7fffa580 is located in stack of thread T0 at offset 0 in frame
#0 0x7fffa5ef  ()

  This frame has 1 object(s):
[32, 288) 'buf' (line 180)
HINT: this may be a false positive if your program uses some custom stack 
unwind mechanism, swapcontext or vfork
  (longjmp and C++ exceptions *are* supported)
END QUOTE

The "'buf' (line 180)" turned out to be from the declaration in
bin/sh/exec.c 's tryexec :

static void
tryexec(char *cmd, char **argv, char **envp)
{
int e, in;
ssize_t n;
char buf[256];

execve(cmd, argv, envp);
e = errno;
if (e == ENOEXEC) {
INTOFF;
in = open(cmd, O_RDONLY | O_NONBLOCK);
if (in != -1) {
n = pread(in, buf, sizeof buf, 0);
close(in);
if (n > 0 && isbinary(buf, n)) {
errno = ENOEXEC;
return;
}
}
*argv = cmd;
*--argv = __DECONST(char *, _PATH_BSHELL);
execve(_PATH_BSHELL, argv, envp);
}
errno = e;
}

So I could finally see/validate that an example
stack-buffer-* report was tied to vfork handling
(analyzing other code related to the tryexec use).

The report happens well after tryexec did its
execve. The information about buf is just old,
no longer accurate information, leading to a
false report:


==87961==ERROR: AddressSanitizer: stack-buffer-underflow on address 
0x7fffa580 at pc 0x000801e0f8ed bp 0x7fff9bf0 sp 0x7fff9be8
WRITE of size 8 at 0x7fffa580 thread T0
#0 0x801e0f8ec in bintime2timespec 
/usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/main-src/amd64.amd64/tmp/usr/include/sys/time.h:285:14
#1 0x801e0f8ec in __vdso_clock_gettime 
/usr/main-src/lib/libc/sys/__vdso_gettimeofday.c:195:2
#2 0x801e0e1a0 in clock_gettime 
/usr/main-src/lib/libc/sys/clock_gettime.c:48:11
#3 0x10d54da in clock_gettime 
/usr/main-src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc:2189:13
#4 0x11234f5 in __sanitizer::MonotonicNanoTime() 
/usr/main-src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:860:3
#5 0x10ba02c in 
__sanitizer::SizeClassAllocator64<__asan::AP64<__sanitizer::LocalAddressSpaceView>
 >::PopulateFreeArray(__sanitizer::AllocatorStats*, unsigned long, 
__sanitizer::SizeClassAllocator
64<__asan::AP64<__sanitizer::LocalAddressSpaceView> >::RegionInfo*, unsigned 
long) 
/usr/main-src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h:790:45
#6 0x10b9c4b in 
__sanitizer::SizeClassAllocator64<__asan::AP64<__sanitizer::LocalAddressSpaceView>
 >::GetFromAllocator(__sanitizer::AllocatorStats*, unsigned long, unsigned 
int*, unsigned long) /u
sr/main-src/contrib/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h:220:11
#7 0x10b9955 in 

Re: What to do about tgammal?

[Marc Fonvieille]

Le 21.12.2021 08:47, Steve Kargl a écrit :
> Not at the moment.  tlibm only handles math functions with
> a single real agrument [1].  I've been thinking about adding
> the 2 argument functions such as atan2 and complex argument
> function, but lack the time.  I also need to improve the man
> page to decoment the default domain for each function and its
> complete domain.
> 
> [1] acos, acosh, asin, asinh, atan, atanh, cbrt, cos, cosh, erf,
> erfc, exp, exp2, expm1, j0, j1, lgamma, log, log10, log1p,
> log2, sin, sinh, sqrt, tan, tanh, tgamma, y0, y1.
>

No problem.
It's a great project.  Keep up posting about your progress.

-- 
Marc

Re: What to do about tgammal?

[Steve Kargl]

On Tue, Dec 21, 2021 at 03:46:40PM +0100, Marc Fonvieille wrote:
> Le 20.12.2021 16:48, Steve Kargl a écrit :
> > On Mon, Dec 20, 2021 at 11:15:53AM +0100, Marc Fonvieille wrote:
> > >
> > > I assume what Steve is talking about is the corresponding value in
> > > decimal of the number of ULP.
> > > 
> > 
> > Bad assumption.  Please read Goldberg's paper.  I am talking
> > about ULP in the underlying floating point format.
> >
> 
> Thanks, it's more clear now: it's the ULP value.
> Is your tlibm_libm program available for testing ?  If I find time I'd like to
> do some tests.
> 

Not at the moment.  tlibm only handles math functions with
a single real agrument [1].  I've been thinking about adding
the 2 argument functions such as atan2 and complex argument
function, but lack the time.  I also need to improve the man
page to decoment the default domain for each function and its
complete domain.

[1] acos, acosh, asin, asinh, atan, atanh, cbrt, cos, cosh, erf,
erfc, exp, exp2, expm1, j0, j1, lgamma, log, log10, log1p,
log2, sin, sinh, sqrt, tan, tanh, tgamma, y0, y1.
-- 
Steve

Re: What to do about tgammal?

[Marc Fonvieille]

Le 20.12.2021 16:48, Steve Kargl a écrit :
> On Mon, Dec 20, 2021 at 11:15:53AM +0100, Marc Fonvieille wrote:
> >
> > I assume what Steve is talking about is the corresponding value in
> > decimal of the number of ULP.
> > 
> 
> Bad assumption.  Please read Goldberg's paper.  I am talking
> about ULP in the underlying floating point format.
>

Thanks, it's more clear now: it's the ULP value.
Is your tlibm_libm program available for testing ?  If I find time I'd like to
do some tests.

-- 
Marc

Re: What to do about tgammal?

[Steve Kargl]

On Mon, Dec 20, 2021 at 11:15:53AM +0100, Marc Fonvieille wrote:
>
> I assume what Steve is talking about is the corresponding value in
> decimal of the number of ULP.
> 

Bad assumption.  Please read Goldberg's paper.  I am talking
about ULP in the underlying floating point format.

-- 
Steve

Re: What to do about tgammal?

[Marc Fonvieille]

Le 18.12.2021 17:59, Mark Murray a écrit :
> 
> 
> > On 18 Dec 2021, at 17:51, Steve Kargl  
> > wrote:
> > 
> > On Sat, Dec 18, 2021 at 10:41:14AM +, Mark Murray wrote:
> >> 
> >> Hmm. I think my understanding of ULP is missing something?
> >> 
> >> I thought that ULP could not be greater than the mantissa size
> >> in bits?
> >> 
> >> I.e., I thought it represents average rounding error (compared with
> >> "perfect rounding"), not truncation error, as the above very large
> >> ULPs suggest.
> >> 
> > 
> > The definition of ULP differs according which expert you
> > choose to follow. :-)  For me (a non-expert), ULP is measured
> > in the system of the "accurate answer", which is assumed to
> > have many more bits of precision than the "approximate answer".
> > From a very old das@ email and for long double I have
> 
> 
> 
> Thank you!
> 
> I checked the definition that I was used to, and it is roughly
> "how many bits of the mantissa are inaccurate (because of
> rounding error)".
>

Hi,

ULP (Unit in the last place) is at first the weight of the least
significant bit of the mantissa.  E.g., in IEEE 754 single precision =
2^-23.
It can also be seen as the distance between 2 consecutive significands
(which is not the distance between 2 consecutive floating numbers).
Some people, use ULP (or number of ULP) as Units (plural) in the Last
Place to show the number of bits in error in the least significant bits
of the significand.
I assume what Steve is talking about is the corresponding value in
decimal of the number of ULP.

This thread is really interesting (even if I'm loosely following it).

> I can see how both work. For utterly massive numbers like
> from Gamma(), I can see how accounting for a much larger
> range works.
> 
> It still feels slightly tricky, as e.g. how many digits after the
> floating point do you account for?
> 
> > I don't print out the hex representation in ld128, but you see
> > the number of correct decimal digits is 33 digits compared to
> > 36.
> 
> Looking good!
> 
> M
> --
> Mark R V Murray
> 



-- 
Marc


signature.asc
Description: PGP signature

Re: What to do about tgammal?

[Mark Murray]

> On 18 Dec 2021, at 17:51, Steve Kargl  
> wrote:
>
> On Sat, Dec 18, 2021 at 10:41:14AM +, Mark Murray wrote:
>>
>> Hmm. I think my understanding of ULP is missing something?
>>
>> I thought that ULP could not be greater than the mantissa size
>> in bits?
>>
>> I.e., I thought it represents average rounding error (compared with
>> "perfect rounding"), not truncation error, as the above very large
>> ULPs suggest.
>>
>
> The definition of ULP differs according which expert you
> choose to follow. :-)  For me (a non-expert), ULP is measured
> in the system of the "accurate answer", which is assumed to
> have many more bits of precision than the "approximate answer".
> From a very old das@ email and for long double I have



Thank you!

I checked the definition that I was used to, and it is roughly
"how many bits of the mantissa are inaccurate (because of
rounding error)".

I can see how both work. For utterly massive numbers like
from Gamma(), I can see how accounting for a much larger
range works.

It still feels slightly tricky, as e.g. how many digits after the
floating point do you account for?

> I don't print out the hex representation in ld128, but you see
> the number of correct decimal digits is 33 digits compared to
> 36.

Looking good!

M
--
Mark R V Murray



signature.asc
Description: Message signed with OpenPGP

Re: What to do about tgammal?

[Steve Kargl]

On Sat, Dec 18, 2021 at 10:41:14AM +, Mark Murray wrote:
> 
> Hmm. I think my understanding of ULP is missing something?
> 
> I thought that ULP could not be greater than the mantissa size
> in bits?
> 
> I.e., I thought it represents average rounding error (compared with
> "perfect rounding"), not truncation error, as the above very large
> ULPs suggest.
> 

The definition of ULP differs according which expert you
choose to follow. :-)  For me (a non-expert), ULP is measured
in the system of the "accurate answer", which is assumed to
have many more bits of precision than the "approximate answer".
>From a very old das@ email and for long double I have 

/* From a das email:
 *
 *   ulps = err * (2**(LDBL_MANT_DIG - e)), where e = ilogb(accurate).
 *
 * I use:
 *
 *   mpfr_sub(err, accurate, approximate, RNDN);
 *   mpfr_abs(err, err, RNDN);
 *   mpfr_mul_2si(ulpx, err, -mpfr_get_exp(accurate) + LDBL_MANT_DIG, RNDN);
 *   ulp = mpfr_get_d(ulpx, RNDN);
 *   if (ulp  0.506 && mpfr_cmpabs(err, ldbl_minx)  0) {
 * print warning...;
 *   }
 */

Here, "err = |accurate - approximate|" is done in the precision
of the "accurate answer", and RNDN is round-to-nearest.  The
line 'mpfr_mul_2si(...)' is doing the scaling to manipulate the
exponent of the radix 2.

This is agreement with Goldberg.  IIRC, Jean-Michel Muller et al
have much more detailed discussion about error and ULPs in their
book "Handbook of Floating-Point Arithmetic".   

https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

So, on ld80 and tgamma, I have 

% tlibm -l -x 8 -X 1755 -n 10 -P tgamma
Interval tested for tgammal: [8,1755]
   ulp <= 0.5: 90.207% 90207 |  90.207% 90207
0.5 <  ulp <  0.6:  7.085%  7085 |  97.292% 97292
0.6 <  ulp <  0.7:  2.383%  2383 |  99.675% 99675
0.7 <  ulp <  0.8:  0.314%   314 |  99.989% 99989
0.8 <  ulp <  0.9:  0.011%11 | 100.000%10
Max ulp: 0.853190 at 9.236118561185611856579e+02

The "ulp <= 0.5" line indicates the number of correctly
rounded case.  If the value of ULP exceeds 1, then you are
starting to lose more than 1 bit in the significant.  Consider
individual points.  Here's a correctly rounded case:

% tlibm -l -a 9.23611856118500 tgamma
   xu = LD80C(0x93c72441a44a57a9, 3,  9.2361185611849994e+00L),
libmu = LD80C(0x82f85b3fe4b36f9b,16,  6.70567128873706962722e+04L),
mpfru = LD80C(0x82f85b3fe4b36f9b,16,  6.70567128873706962722e+04L),
 ULP = 0.42318

Notice, ULP < 0.5 and the bits in libm and mpfr answer rounded to 
long double are all the same as is the decimal representation. Now
consider the max ULP case:

% tlibm -l -a  9.236118561185611856579e+02 tgamma
   xu = LD80C(0xe6e728a690c4fa87, 9,  9.23611856118561185658e+02L),
libmu = LD80C(0xba6bea661a7eda3c,  7762,  5.7294439856327202e+2336L),
mpfru = LD80C(0xba6bea661a7eda3d,  7762,  5.7294439856327245e+2336L),
 ULP = 0.85319

ULP is < 1, which is desireable.  The bits agree except for the
last place where we're off by 1, ie, 0xc vs 0xd.  The decimal
representation is of course off.

So, now on grimoirei without my patch, I have 

% ./tlibm_libm -l -a 1.59255925592559255925592559255925594e+01 tgamma
   x =  1.59255925592559255925592559255925594e+01L
libm =  1.06660136733166064453125e+12L
mpfr =  1.06660136733166211967839038834033459e+12L,
 ULP = 13932370826141802496.0

You have 16 correct decimal digits out of 36, which is the best
you can expect from mapping tgammal to tgamma.  ULP is significant
larger than 1.  The ulp is almost guaranteed to be larger than
2**(113-53).

Now with my patch and worse case for 1 x in [10:16]

% ./tlibm_lmath -l -a 1.59255925592559255925592559255925594e+01 tgamma
  x =  1.59255925592559255925592559255925594e+01L
libm =  1.06660136733166211967839038834033897e+12
mpfr =  1.06660136733166211967839038834033459e+12L,
 ULP = 41.40877

I don't print out the hex representation in ld128, but you see
the number of correct decimal digits is 33 digits compared to
36.

-- 
Steve

Re: What to do about tgammal?

[Mark Murray]

> On 18 Dec 2021, at 03:52, Steve Kargl  
> wrote:
>
> On Tue, Dec 14, 2021 at 10:21:55PM +, Mark Murray wrote:
>> On 14 Dec 2021, at 21:51, Steve Kargl  
>> wrote:
>>> Interval  max ULP  x at Max ULP
>>> [6,1755.1]0.873414 at 1.480588145237629047468e+03
>>> [1.0662,6]0.861508 at 1.999467927053585410537e+00
>>> [1.01e-17,1.0661] 0.938041 at 1.023286481537296307856e+00
>>> [-1.,-1.0001] 3.157770 at -1.246957268051453610329e+00
>>> [-2.,-2.0001] 2.987659 at -2.220949465449893090070e+00
>>>
>>> Note, 1.01e-17 can be reduced to soemthing like 1.01e-19 or
>>
>> Extra diffs most welcome!
>>
>
> Hi Mark,
>
> Don't know if you noticed, but I borroewed a few cpu cycles
> from grimoire.

Didn't notice a thing! :-)

>  My WIP is already better than the imprecise.c
> kludge from theraven@.  I need to work out the details of
> computing logl(x) in extra precision or see if I can leverage
> what Bruce did a few years ago.  Anywho, current results:
>
> Interval tested for tgammal: [128,1755]
> count: 100
>  xm =  1.71195767195767195767195767195767183e+03L
> libm =  7.79438030237108165017007606176403036e+4790L
> mpfr =  7.79438030237108165017007606175285456e+4790L
> ULP = 14869.19517
>
> Interval tested for tgammal: [16,128]
> count: 100
>  xm =  1.27687183687183687183687183687183690e+02L
> libm =  6.61421998891483212224382625339007663e+212L
> mpfr =  6.61421998891483212224382625338960267e+212L
> ULP = 731.00958
>
> Interval tested for tgammal: [10,16]
> count: 100
>  xm =  1.54261654261654261654261654261654251e+01L
> libm =  2.74203137295418912508367515208072654e+11L
> mpfr =  2.74203137295418912508367515208073861e+11L
> ULP = 45.61161
>
> Interval tested for tgammal: [1.2446e-60,10]
> count: 100
>  xm =  6.26200626138006138006138006138006065e-02L
> libm =  1.54507103764516989381203274093299079e+01L
> mpfr =  1.54507103764516989381203274093299091e+01L
> ULP = 0.76751

Hmm. I think my understanding of ULP is missing something?

I thought that ULP could not be greater than the mantissa size
in bits?

I.e., I thought it represents average rounding error (compared with
"perfect rounding"), not truncation error, as the above very large
ULPs suggest.

M
--
Mark R V Murray



signature.asc
Description: Message signed with OpenPGP

Re: What to do about tgammal?

[Steve Kargl]

On Tue, Dec 14, 2021 at 10:21:55PM +, Mark Murray wrote:
> On 14 Dec 2021, at 21:51, Steve Kargl  
> wrote:
> > Interval  max ULP  x at Max ULP
> > [6,1755.1]0.873414 at 1.480588145237629047468e+03
> > [1.0662,6]0.861508 at 1.999467927053585410537e+00
> > [1.01e-17,1.0661] 0.938041 at 1.023286481537296307856e+00
> > [-1.,-1.0001] 3.157770 at -1.246957268051453610329e+00
> > [-2.,-2.0001] 2.987659 at -2.220949465449893090070e+00
> > 
> > Note, 1.01e-17 can be reduced to soemthing like 1.01e-19 or
> 
> Extra diffs most welcome!
> 

Hi Mark,

Don't know if you noticed, but I borroewed a few cpu cycles
from grimoire.  My WIP is already better than the imprecise.c
kludge from theraven@.  I need to work out the details of 
computing logl(x) in extra precision or see if I can leverage
what Bruce did a few years ago.  Anywho, current results:

Interval tested for tgammal: [128,1755]
count: 100
  xm =  1.71195767195767195767195767195767183e+03L
libm =  7.79438030237108165017007606176403036e+4790L
mpfr =  7.79438030237108165017007606175285456e+4790L
 ULP = 14869.19517

Interval tested for tgammal: [16,128]
count: 100
  xm =  1.27687183687183687183687183687183690e+02L
libm =  6.61421998891483212224382625339007663e+212L
mpfr =  6.61421998891483212224382625338960267e+212L
 ULP = 731.00958

Interval tested for tgammal: [10,16]
count: 100
  xm =  1.54261654261654261654261654261654251e+01L
libm =  2.74203137295418912508367515208072654e+11L
mpfr =  2.74203137295418912508367515208073861e+11L
 ULP = 45.61161

Interval tested for tgammal: [1.2446e-60,10]
count: 100
  xm =  6.26200626138006138006138006138006065e-02L
libm =  1.54507103764516989381203274093299079e+01L
mpfr =  1.54507103764516989381203274093299091e+01L
 ULP = 0.76751




-- 
Steve

Re: What to do about tgammal?

[Steve Kargl]

On Tue, Dec 14, 2021 at 03:47:17PM -0700, Warner Losh wrote:
> On Tue, Dec 14, 2021 at 3:23 PM Mark Murray  wrote:
> 
> > On 14 Dec 2021, at 21:51, Steve Kargl 
> > wrote:
> > > Interval  max ULP  x at Max ULP
> > > [6,1755.1]0.873414 at 1.480588145237629047468e+03
> > > [1.0662,6]0.861508 at 1.999467927053585410537e+00
> > > [1.01e-17,1.0661] 0.938041 at 1.023286481537296307856e+00
> > > [-1.,-1.0001] 3.157770 at -1.246957268051453610329e+00
> > > [-2.,-2.0001] 2.987659 at -2.220949465449893090070e+00
> > >
> > > Note, 1.01e-17 can be reduced to soemthing like 1.01e-19 or
> >
> > Extra diffs most welcome!
> >
> 
> Those results have allayed my fears. Is 1e-17 sufficient to commit these
> changes, or do
> we need to work to get it down to 1e-19 for it to be acceptable?
> 

Well, egg-on-the-face.  I updated the sloppy limit to isolate
x = 0 from 0x1p-56 to use 0x1p-116 and forgot to update my test
scripts.  The code in tgammal in the interval [-iota:iota] is

if (x > iota)
RETURNI(smaller_gam(x));

if (x > -iota) {
if (x != 0)
u.a = 1 - tiny; /* raise inexact */
RETURNI(1 / x);
}

so tgammal(x) -> 1/x as x -> 0.


-- 
Steve

Re: What to do about tgammal?

[Warner Losh]

On Tue, Dec 14, 2021 at 3:23 PM Mark Murray  wrote:

> On 14 Dec 2021, at 21:51, Steve Kargl 
> wrote:
> > Interval  max ULP  x at Max ULP
> > [6,1755.1]0.873414 at 1.480588145237629047468e+03
> > [1.0662,6]0.861508 at 1.999467927053585410537e+00
> > [1.01e-17,1.0661] 0.938041 at 1.023286481537296307856e+00
> > [-1.,-1.0001] 3.157770 at -1.246957268051453610329e+00
> > [-2.,-2.0001] 2.987659 at -2.220949465449893090070e+00
> >
> > Note, 1.01e-17 can be reduced to soemthing like 1.01e-19 or
>
> Extra diffs most welcome!
>

Those results have allayed my fears. Is 1e-17 sufficient to commit these
changes, or do
we need to work to get it down to 1e-19 for it to be acceptable?

Warner

Re: What to do about tgammal?

[Mark Murray]

On 14 Dec 2021, at 21:51, Steve Kargl  wrote:
> Interval  max ULP  x at Max ULP
> [6,1755.1]0.873414 at 1.480588145237629047468e+03
> [1.0662,6]0.861508 at 1.999467927053585410537e+00
> [1.01e-17,1.0661] 0.938041 at 1.023286481537296307856e+00
> [-1.,-1.0001] 3.157770 at -1.246957268051453610329e+00
> [-2.,-2.0001] 2.987659 at -2.220949465449893090070e+00
>
> Note, 1.01e-17 can be reduced to soemthing like 1.01e-19 or

Extra diffs most welcome!

M
--
Mark R V Murray



signature.asc
Description: Message signed with OpenPGP

Re: What to do about tgammal?

[Mark Murray]

> On 14 Dec 2021, at 20:41, Steve Kargl  
> wrote:
>
> On Tue, Dec 14, 2021 at 06:26:13PM +, Mark Murray wrote:
>>
>> This is now visible for review at
>> https://reviews.freebsd.org/D33444 <https://reviews.freebsd.org/D33444>
>>
>
> Just looked at this again.  I cannot tell from the
> diff whether you've 'git mv' src/imprecise.c to
> ld128/b_tgammal.c.  This is need to still get the
> mapping of tgammal -> tgamma.

Fixed, thanks! Thats what I get from checking only on an ld80 machine!

M
--
Mark R V Murray



signature.asc
Description: Message signed with OpenPGP

Re: What to do about tgammal?

[Steve Kargl]

On Tue, Dec 14, 2021 at 06:26:13PM +, Mark Murray wrote:
> 
> This is now visible for review at
> https://reviews.freebsd.org/D33444 
> 

I see imp lamented that that fact that he is not sufficiently
versed in the numerical methods used (neither am I!).  bde
use to be my go-to reviewer, but he's no longer with us.  To
allay fears, I've tested 5 million values distributed in the
intervals of the various approximations.  Here's the result

 Interval  max ULP  x at Max ULP
[6,1755.1]0.873414 at 1.480588145237629047468e+03
[1.0662,6]0.861508 at 1.999467927053585410537e+00
[1.01e-17,1.0661] 0.938041 at 1.023286481537296307856e+00
[-1.,-1.0001] 3.157770 at -1.246957268051453610329e+00
[-2.,-2.0001] 2.987659 at -2.220949465449893090070e+00 

Note, 1.01e-17 can be reduced to soemthing like 1.01e-19 or
1.01e-20.

-- 
Steve

Re: What to do about tgammal?

[Steve Kargl]

On Tue, Dec 14, 2021 at 06:26:13PM +, Mark Murray wrote:
> 
> This is now visible for review at
> https://reviews.freebsd.org/D33444 
> 

Just looked at this again.  I cannot tell from the
diff whether you've 'git mv' src/imprecise.c to 
ld128/b_tgammal.c.  This is need to still get the
mapping of tgammal -> tgamma.

-- 
Steve

Re: What to do about tgammal?

[Steve Kargl]

On Tue, Dec 14, 2021 at 06:26:13PM +, Mark Murray wrote:
> 
> This is now visible for review at
> https://reviews.freebsd.org/D33444 
> 

Thanks!  I looks okay to me (but, of course, I wrote the patch ;-)

BTW, I'll probably take a shot at ld128 tgammal(x) this
weekend as I still have my account on your aarch64 system
for testing.

-- 
Steve

Re: What to do about tgammal?

[Mark Murray]

> On 13 Dec 2021, at 02:22, Steve Kargl  
> wrote:
> 
> On Sat, Dec 04, 2021 at 11:48:13PM +, Mark Murray wrote:
>> 
>> 
>>> On 4 Dec 2021, at 18:53, Steve Kargl  
>>> wrote:
>>> 
>>> 
>>> So, is anyone interested in seeing a massive patch?
>> 
>> Me, please!
>> 
> 
> So, I have the ld80 case working.  I'll open a PR if
> on does not exist.  The pr will contain

This is now visible for review at
https://reviews.freebsd.org/D33444 

Thanks!

M
--
Mark R V Murray



signature.asc
Description: Message signed with OpenPGP

Re: RFC: What to do about Allocate in the NFS server for FreeBSD13?

[John Baldwin]

On 12/13/21 8:30 AM, Konstantin Belousov wrote:

On Mon, Dec 13, 2021 at 04:26:42PM +, Rick Macklem wrote:

Hi,

There are two problems with Allocate in the NFSv4.2 server in FreeBSD13:
1 - It uses the thread credentials instead of the ones for the RPC.
2 - It does not ensure that file changes are committed to stable storage.
These problems are fixed by commit f0c9847a6c47 in main, which added
ioflag and cred arguments to VOP_ALLOCATE().

I can think of 3 ways to fix Allocate in FreeBSD13:
1 - Apply a *hackish* patch like this:
+   savcred = p->td_ucred;
+   td->td_ucred = cred;
do {
olen = len;
error = VOP_ALLOCATE(vp, , );
if (error == 0 && len > 0 && olen > len)
maybe_yield();
} while (error == 0 && len > 0 && olen > len);
+   p->td_ucred = savcred;
if (error == 0 && len > 0)
error = NFSERR_IO;
+   if (error == 0)
+   error = VOP_FSYNC(vp, MNT_WAIT, p);
The worst part of it is temporarily setting td_ucred to cred.

2 - MFC'ng commit f0c9847a6c47. Normally changes to the
  VOP/VFS are not MFC'd. However, in this case, it might be
  ok to do so, since it is unlikely there is an out of source tree
  file system with a custom VOP_ALLOCATE() method?

I do not see much wrong with #2, this is what I would do myself.


I also think this is fine.

--
John Baldwin

Re: RFC: What to do about Allocate in the NFS server for FreeBSD13?

[Konstantin Belousov]

On Mon, Dec 13, 2021 at 04:26:42PM +, Rick Macklem wrote:
> Hi,
> 
> There are two problems with Allocate in the NFSv4.2 server in FreeBSD13:
> 1 - It uses the thread credentials instead of the ones for the RPC.
> 2 - It does not ensure that file changes are committed to stable storage.
> These problems are fixed by commit f0c9847a6c47 in main, which added
> ioflag and cred arguments to VOP_ALLOCATE().
> 
> I can think of 3 ways to fix Allocate in FreeBSD13:
> 1 - Apply a *hackish* patch like this:
> + savcred = p->td_ucred;
> + td->td_ucred = cred;
>   do {
>   olen = len;
>   error = VOP_ALLOCATE(vp, , );
>   if (error == 0 && len > 0 && olen > len)
>   maybe_yield();
>   } while (error == 0 && len > 0 && olen > len);
> + p->td_ucred = savcred;
>   if (error == 0 && len > 0)
>   error = NFSERR_IO;
> + if (error == 0)
> + error = VOP_FSYNC(vp, MNT_WAIT, p);
> The worst part of it is temporarily setting td_ucred to cred.
> 
> 2 - MFC'ng commit f0c9847a6c47. Normally changes to the
>  VOP/VFS are not MFC'd. However, in this case, it might be
>  ok to do so, since it is unlikely there is an out of source tree
>  file system with a custom VOP_ALLOCATE() method?
I do not see much wrong with #2, this is what I would do myself.

> 
> 3 - Just disable it. Currently it is disabled by default and it
>  could just be wired down disabled.
>  Allocate is not that useful, since ZFS does not support it.
> 
> As an aside to this, the IETF NFSv4 working group seems to
> have agreed that NFS4ERR_NOTSUPP can be returned by a
> NFSv4.2 server on a 'per file system basis" instead of globally,
> since the Linux knfsd already does this.
> --> As such, Allocate can be enabled by default in main and
>could be enabled by default in FreeBSD13, if #1 or #2 was
>done.
>--> It still would not work for ZFS exports.
> 
> So, what do you think is the preferred alternative?
> 
> rick
> 

RFC: What to do about Allocate in the NFS server for FreeBSD13?

[Rick Macklem]

Hi,

There are two problems with Allocate in the NFSv4.2 server in FreeBSD13:
1 - It uses the thread credentials instead of the ones for the RPC.
2 - It does not ensure that file changes are committed to stable storage.
These problems are fixed by commit f0c9847a6c47 in main, which added
ioflag and cred arguments to VOP_ALLOCATE().

I can think of 3 ways to fix Allocate in FreeBSD13:
1 - Apply a *hackish* patch like this:
+   savcred = p->td_ucred;
+   td->td_ucred = cred;
do {
olen = len;
error = VOP_ALLOCATE(vp, , );
if (error == 0 && len > 0 && olen > len)
maybe_yield();
} while (error == 0 && len > 0 && olen > len);
+   p->td_ucred = savcred;
if (error == 0 && len > 0)
error = NFSERR_IO;
+   if (error == 0)
+   error = VOP_FSYNC(vp, MNT_WAIT, p);
The worst part of it is temporarily setting td_ucred to cred.

2 - MFC'ng commit f0c9847a6c47. Normally changes to the
 VOP/VFS are not MFC'd. However, in this case, it might be
 ok to do so, since it is unlikely there is an out of source tree
 file system with a custom VOP_ALLOCATE() method?

3 - Just disable it. Currently it is disabled by default and it
 could just be wired down disabled.
 Allocate is not that useful, since ZFS does not support it.

As an aside to this, the IETF NFSv4 working group seems to
have agreed that NFS4ERR_NOTSUPP can be returned by a
NFSv4.2 server on a 'per file system basis" instead of globally,
since the Linux knfsd already does this.
--> As such, Allocate can be enabled by default in main and
   could be enabled by default in FreeBSD13, if #1 or #2 was
   done.
   --> It still would not work for ZFS exports.

So, what do you think is the preferred alternative?

rick

Re: What to do about tgammal?

[Steve Kargl]

On Sat, Dec 04, 2021 at 11:48:13PM +, Mark Murray wrote:
> 
> 
> > On 4 Dec 2021, at 18:53, Steve Kargl  
> > wrote:
> > 
> > 
> > So, is anyone interested in seeing a massive patch?
> 
> Me, please!
> 

So, I have the ld80 case working.  I'll open a PR if 
on does not exist.  The pr will contain

1) Diff for nonfunctional changes to code for tgamma(x).
This is a re-organization of the code so I could reverse
it.  Some cleanup of comments and documentations.  A
move towards style(9)

2) Advice on what to do about tgammal(x) on ld128 hardware.
   Essentially, need to do the git equivalent of 'svn mv
   imprecise.c ld128/b_tgammal.c.

3) Diff with the new code for ld80 tgamma.


Interval tested for tgammal: [6,1755.1]
500 calls, 2.056068 secs, 0.41121 usecs/call
   ulp <= 0.5: 90.233%   4511666 |  90.233%   4511666
0.5 <  ulp <  0.6:  7.061%353033 |  97.294%   4864699
0.6 <  ulp <  0.7:  2.393%119644 |  99.687%   4984343
0.7 <  ulp <  0.8:  0.300% 14981 |  99.986%   4999324
0.8 <  ulp <  0.9:  0.014%   676 | 100.000%   500
Max ulp: 0.873414 at 1.480588145237629047468e+03

Interval tested for tgammal: [1.0662,6]
500 calls, 0.748966 secs, 0.14979 usecs/call
   ulp <= 0.5: 96.355%   4817737 |  96.355%   4817737
0.5 <  ulp <  0.6:  3.291%164534 |  99.645%   4982271
0.6 <  ulp <  0.7:  0.328% 16403 |  99.973%   4998674
0.7 <  ulp <  0.8:  0.026%  1297 |  99.999%   471
0.8 <  ulp <  0.9:  0.001%29 | 100.000%   500
Max ulp: 0.861508 at 1.999467927053585410537e+00

Interval tested for tgammal: [1.01e-17,1.0661]
500 calls, 0.904246 secs, 0.18085 usecs/call
   ulp <= 0.5: 96.201%   4810043 |  96.201%   4810043
0.5 <  ulp <  0.6:  3.297%164875 |  99.498%   4974918
0.6 <  ulp <  0.7:  0.412% 20581 |  99.910%   4995499
0.7 <  ulp <  0.8:  0.082%  4108 |  99.992%   4999607
0.8 <  ulp <  0.9:  0.008%   389 | 100.000%   496
0.9 <  ulp <  1.0:  0.000% 4 | 100.000%   500
Max ulp: 0.938041 at 1.023286481537296307856e+00

Interval tested for tgammal: [-1.,-1.0001]
500 calls, 1.740820 secs, 0.34816 usecs/call
   ulp <= 0.5: 56.596%   2829776 |  56.596%   2829776
0.5 <  ulp <  0.6:  8.616%430793 |  65.211%   3260569
0.6 <  ulp <  0.7:  7.452%372621 |  72.664%   3633190
0.7 <  ulp <  0.8:  6.250%312511 |  78.914%   3945701
0.8 <  ulp <  0.9:  5.147%257372 |  84.061%   4203073
0.9 <  ulp <  1.0:  4.100%205010 |  88.162%   4408083
1.0 <  ulp <  1.5:  9.846%492324 |  98.008%   4900407
1.5 <  ulp <  2.0:  1.790% 89514 |  99.798%   4989921
2.0 <  ulp <  3.0:  0.201% 10074 | 100.000%   495
3.0 <  ulp <  0.0:  0.000% 5 | 100.000%   500

-- 
Steve

Re: What to do about tgammal?

[Steve Kargl]

On Sat, Dec 04, 2021 at 11:48:13PM +, Mark Murray wrote:
> 
> 
> > On 4 Dec 2021, at 18:53, Steve Kargl  
> > wrote:
> > 
> > 
> > So, is anyone interested in seeing a massive patch?
> 
> Me, please!
> 

It is ld80 only.  ld128 is still broken.  I'll clean things
up and create a diff over the next few days.  The patch
will disconnect imprecise.c from the build.  So, if you like
what you see and pursue committing the patch(es),
one pre-requisite will be to copy src/imprecise.c to 
ld128/b_tgammal.c.  In the good old day, one would use

% svn copy src/imprecise.c ld128/b_tgammal.c
% svn delete src/imprecise.c

or 

% svn move src/imprecise.c ld128/b_tgammal.c

Don't know (or care) how to do this in a git world.

-- 
Steve

Re: What to do about tgammal?

[Mark Murray]

> On 4 Dec 2021, at 18:53, Steve Kargl  
> wrote:
>
>
> So, is anyone interested in seeing a massive patch?

Me, please!

M
--
Mark R V Murray



signature.asc
Description: Message signed with OpenPGP

Re: What to do about tgammal?

[Steve Kargl]

On Sat, Dec 04, 2021 at 08:40:56PM +0100, Hans Petter Selasky wrote:
> On 12/4/21 19:53, Steve Kargl wrote:
> > What to do about tgammal?

(trim some history)

> > 
> >Interval | Max ULP
> > ---+
> >   [6,171]   |  1340542.2
> >   [1.0662,6]|14293.3
> >   [1.01e-17,1.0661] | 3116.1
> >   [-1.,-1.0001] | 15330369.3
> > ---+
> > 
> > Well, I finally have gotten around to removing theraven@'s last kludge
> > for FreeBSD on systems that support ld80.  This is done with a straight
> > forward modification of the msun/bsdsrc code.  The limitation on
> > domain is removed and the accuracy substantially improved.
> > 
> >Interval | Max ULP
> > ---+--
> >   [6,1755]  |8.457
> >   [1.0662,6]|   11.710
> >   [1.01e-17,1.0661] |   11.689
> >   [-1.,-1.0001] |   11.871
> > ---+--
> > 
> > My modifications leverage the fact that tgamma(x) (ie., double function)
> > uses extend arithmetic to do the computations (approximately 85 bits of
> > precision).  To get the Max ULP below 1 (the desired upper limit), a few
> > minimax polynomials need to be determined and the mystery around a few
> > magic numbers need to be unraveled.
> > 
> > Extending what I have done to an ld128 implementation requires much
> > more effort than I have time and energy to pursue.  Someone with
> > interest in floating point math on ld128 system can provide an
> > implementation.
> > 
> > So, is anyone interested in seeing a massive patch?
> > 
> 
> Hi,
> 
> Do you need a implementation of tgamma() which is 100% correct, or a
> so-called speed-hack version of tgamma() which is almost correct?
> 
> I've looked a bit into libm in FreeBSD and I see some functions are
> implemented so that they execute quickly, instead of producing exact
> results. Is this true?
> 

I'm afraid that I don't fully understand your questions.

The ULP, listed above, were computed by comparing the libm tgammal(x)
against a tgammal(x) computed with MPFR.  The MPFR result was configured
to have 256 bits of precision.  In other words, MPFR is assumed to be
exact for the comparison between a 64-bit tgammal(x) and a 256-bit
mpfr_gamma() function.

There is no speed hack with mpfr_gamma().

% time ./tlibm_lmath -l -s 0 -x 6 -X 1755 -n 10 tgamma
Interval tested for tgammal: [6,1755]
10 calls, 0.042575 secs, 0.42575 usecs/call
count: 10
  xmu = LD80C(0xae3587b6f275c42c, 4,  2.17761377613776137760e+01L),
libmu = LD80C(0xb296591784078768,64,  2.57371418855839536160e+19L),
mpfru = LD80C(0xb296591784078760,64,  2.57371418855839536000e+19L),
 ULP = 8.28349
6.04 real 6.02 user 0.01 sys

My test program shows 10 libm tgammal(x) calls took about 0.04
seconds while the program takes 6 seconds to finish.  Most of that
time is dominated by MPFR.

In general, floating point arithmetic, where a finite number is
the result, is inexact.  The basic binary operators, +x-/*, are
specified by IEEE 754 to have an error no larger that 0.5 ULP.

The mantra that I follow (and know bde followed) is to try
to optimize libm functions to give the most accurate result
as fast as possible.

-- 
Steve

Re: What to do about tgammal?

[Hans Petter Selasky]

On 12/4/21 19:53, Steve Kargl wrote:

What to do about tgammal?

A long time ago (2013-09-06), theraven@ committed a kludge that mapped
several missing long double math functions to double math functions
(e.g., tanhl(x) was mapped to tanh(x)).  Over the next few years, I
(along with bde and das reviews) provided Intel 80-bit (ld80) and IEEE
128-bit (ld128) implementations for some of these functions; namely,
coshl(x), sinhl(x), tanhl(x), erfl(x), erfcl(x), and lgamma(x).  The
last remaining function is tgammal(x).  If one links a program that uses
tgammal(x) with libm, one sees

   /usr/local/bin/ld: fcn_list.o: in function `build_fcn_list':
   fcn_list.c:(.text+0x7c4): warning: tgammal has lower than advertised
   precision

The warning is actually misleading.  Not only does tgammal(x) have a
*MUCH* lower precision, it has a reduced domain.  That is, tgammal(x)
produces +inf for x > 172 whereas tgammal(x) should produce a finite
result for values of x up to 1755 (or so).  On amd64-*-freebsd,
testing 100 in the below intervals demonstrates pathetic accuracy.

Current implmentation via imprecise.c

   Interval | Max ULP
---+
  [6,171]   |  1340542.2
  [1.0662,6]|14293.3
  [1.01e-17,1.0661] | 3116.1
  [-1.,-1.0001] | 15330369.3
---+

Well, I finally have gotten around to removing theraven@'s last kludge
for FreeBSD on systems that support ld80.  This is done with a straight
forward modification of the msun/bsdsrc code.  The limitation on
domain is removed and the accuracy substantially improved.

   Interval | Max ULP
---+--
  [6,1755]  |8.457
  [1.0662,6]|   11.710
  [1.01e-17,1.0661] |   11.689
  [-1.,-1.0001] |   11.871
---+--

My modifications leverage the fact that tgamma(x) (ie., double function)
uses extend arithmetic to do the computations (approximately 85 bits of
precision).  To get the Max ULP below 1 (the desired upper limit), a few
minimax polynomials need to be determined and the mystery around a few
magic numbers need to be unraveled.

Extending what I have done to an ld128 implementation requires much
more effort than I have time and energy to pursue.  Someone with
interest in floating point math on ld128 system can provide an
implementation.

So, is anyone interested in seeing a massive patch?



Hi,

Do you need a implementation of tgamma() which is 100% correct, or a 
so-called speed-hack version of tgamma() which is almost correct?


I've looked a bit into libm in FreeBSD and I see some functions are 
implemented so that they execute quickly, instead of producing exact 
results. Is this true?


--HPS

What to do about tgammal?

[Steve Kargl]

What to do about tgammal?

A long time ago (2013-09-06), theraven@ committed a kludge that mapped
several missing long double math functions to double math functions
(e.g., tanhl(x) was mapped to tanh(x)).  Over the next few years, I
(along with bde and das reviews) provided Intel 80-bit (ld80) and IEEE
128-bit (ld128) implementations for some of these functions; namely,
coshl(x), sinhl(x), tanhl(x), erfl(x), erfcl(x), and lgamma(x).  The
last remaining function is tgammal(x).  If one links a program that uses
tgammal(x) with libm, one sees

  /usr/local/bin/ld: fcn_list.o: in function `build_fcn_list':
  fcn_list.c:(.text+0x7c4): warning: tgammal has lower than advertised
  precision

The warning is actually misleading.  Not only does tgammal(x) have a
*MUCH* lower precision, it has a reduced domain.  That is, tgammal(x)
produces +inf for x > 172 whereas tgammal(x) should produce a finite
result for values of x up to 1755 (or so).  On amd64-*-freebsd,
testing 100 in the below intervals demonstrates pathetic accuracy.

Current implmentation via imprecise.c

  Interval | Max ULP
---+
 [6,171]   |  1340542.2
 [1.0662,6]|14293.3
 [1.01e-17,1.0661] | 3116.1
 [-1.,-1.0001] | 15330369.3
---+

Well, I finally have gotten around to removing theraven@'s last kludge
for FreeBSD on systems that support ld80.  This is done with a straight
forward modification of the msun/bsdsrc code.  The limitation on
domain is removed and the accuracy substantially improved. 

  Interval | Max ULP
---+--
 [6,1755]  |8.457
 [1.0662,6]|   11.710
 [1.01e-17,1.0661] |   11.689
 [-1.,-1.0001] |   11.871
---+--

My modifications leverage the fact that tgamma(x) (ie., double function)
uses extend arithmetic to do the computations (approximately 85 bits of
precision).  To get the Max ULP below 1 (the desired upper limit), a few
minimax polynomials need to be determined and the mystery around a few
magic numbers need to be unraveled.

Extending what I have done to an ld128 implementation requires much
more effort than I have time and energy to pursue.  Someone with 
interest in floating point math on ld128 system can provide an 
implementation.

So, is anyone interested in seeing a massive patch?

-- 
Steve

Re: what does "failed to read progbits" mean?

[Mark Johnston]

On Mon, Oct 25, 2021 at 11:46:49AM -0400, Michael Butler via freebsd-current 
wrote:
> This seems to have gone away after 
> https://freshbsd.org/freebsd/src/commit/70f51f0e474ffe1fb74cb427423a2fba3637544d
> 
> Not sure if the bug that commit fixes was the underlying cause,

Yes, that commit fixes the underlying problem, which manifested for me
as kldxref errors among other things.

> On 10/25/21 11:40, Nuno Teixeira wrote:
> > Same here:
> > kldxref /boot/kernel
> > failed to read progbits
> > 
> > But kernel failed to install. I will include log tomorrow, I'm doing a 
> > clean build with /usr/obj/.. deleted.
> > 
> > Michael Butler via freebsd-current  > > escreveu no dia quinta, 21/10/2021 
> > à(s) 20:14:
> > 
> > Well this is different .. I did a full rebuild (after "rm -rf
> > /usr/obj/*") this morning and now see ..
> > 
> > ===> linux_common (install)
> > install -T release -o root -g wheel -m 555   linux_common.ko
> > /boot/kernel/
> > install -T dbg -o root -g wheel -m 555   linux_common.ko.debug
> > /usr/lib/debug/boot/kernel/
> > ===> linuxkpi (install)
> > install -T release -o root -g wheel -m 555   linuxkpi.ko /boot/kernel/
> > install -T dbg -o root -g wheel -m 555   linuxkpi.ko.debug
> > /usr/lib/debug/boot/kernel/
> > kldxref /boot/kernel
> > failed to read progbits
> > failed to read progbits
> > failed to read progbits
> > failed to read progbits
> > failed to read progbits
> > failed to read progbits
> > failed to read progbits
> > kldxref: /boot/kernel/kernel: cannot load DT_RELA section
> > --
> >   >>> Installing kernel TOSHI completed on Thu Oct 21 15:05:00 EDT 2021
> > --
> > 
> > Is something broken or just cosmetic noise?
> > 
> >          imb
> > 
> 
> 

Re: what does "failed to read progbits" mean?

[Michael Butler via freebsd-current]

This seems to have gone away after 
https://freshbsd.org/freebsd/src/commit/70f51f0e474ffe1fb74cb427423a2fba3637544d


Not sure if the bug that commit fixes was the underlying cause,

Michael


On 10/25/21 11:40, Nuno Teixeira wrote:

Same here:
kldxref /boot/kernel
failed to read progbits

But kernel failed to install. I will include log tomorrow, I'm doing a 
clean build with /usr/obj/.. deleted.


Michael Butler via freebsd-current > escreveu no dia quinta, 21/10/2021 
à(s) 20:14:


Well this is different .. I did a full rebuild (after "rm -rf
/usr/obj/*") this morning and now see ..

===> linux_common (install)
install -T release -o root -g wheel -m 555   linux_common.ko
/boot/kernel/
install -T dbg -o root -g wheel -m 555   linux_common.ko.debug
/usr/lib/debug/boot/kernel/
===> linuxkpi (install)
install -T release -o root -g wheel -m 555   linuxkpi.ko /boot/kernel/
install -T dbg -o root -g wheel -m 555   linuxkpi.ko.debug
/usr/lib/debug/boot/kernel/
kldxref /boot/kernel
failed to read progbits
failed to read progbits
failed to read progbits
failed to read progbits
failed to read progbits
failed to read progbits
failed to read progbits
kldxref: /boot/kernel/kernel: cannot load DT_RELA section
--
  >>> Installing kernel TOSHI completed on Thu Oct 21 15:05:00 EDT 2021
--

Is something broken or just cosmetic noise?

         imb

Re: what does "failed to read progbits" mean?

[Nuno Teixeira]

Same here:
kldxref /boot/kernel
failed to read progbits

But kernel failed to install. I will include log tomorrow, I'm doing a
clean build with /usr/obj/.. deleted.

Michael Butler via freebsd-current  escreveu
no dia quinta, 21/10/2021 à(s) 20:14:

> Well this is different .. I did a full rebuild (after "rm -rf
> /usr/obj/*") this morning and now see ..
>
> ===> linux_common (install)
> install -T release -o root -g wheel -m 555   linux_common.ko /boot/kernel/
> install -T dbg -o root -g wheel -m 555   linux_common.ko.debug
> /usr/lib/debug/boot/kernel/
> ===> linuxkpi (install)
> install -T release -o root -g wheel -m 555   linuxkpi.ko /boot/kernel/
> install -T dbg -o root -g wheel -m 555   linuxkpi.ko.debug
> /usr/lib/debug/boot/kernel/
> kldxref /boot/kernel
> failed to read progbits
> failed to read progbits
> failed to read progbits
> failed to read progbits
> failed to read progbits
> failed to read progbits
> failed to read progbits
> kldxref: /boot/kernel/kernel: cannot load DT_RELA section
> --
>  >>> Installing kernel TOSHI completed on Thu Oct 21 15:05:00 EDT 2021
> --
>
> Is something broken or just cosmetic noise?
>
> imb
>
>

what does "failed to read progbits" mean?

[Michael Butler via freebsd-current]

Well this is different .. I did a full rebuild (after "rm -rf 
/usr/obj/*") this morning and now see ..


===> linux_common (install)
install -T release -o root -g wheel -m 555   linux_common.ko /boot/kernel/
install -T dbg -o root -g wheel -m 555   linux_common.ko.debug 
/usr/lib/debug/boot/kernel/

===> linuxkpi (install)
install -T release -o root -g wheel -m 555   linuxkpi.ko /boot/kernel/
install -T dbg -o root -g wheel -m 555   linuxkpi.ko.debug 
/usr/lib/debug/boot/kernel/

kldxref /boot/kernel
failed to read progbits
failed to read progbits
failed to read progbits
failed to read progbits
failed to read progbits
failed to read progbits
failed to read progbits
kldxref: /boot/kernel/kernel: cannot load DT_RELA section
--
>>> Installing kernel TOSHI completed on Thu Oct 21 15:05:00 EDT 2021
--

Is something broken or just cosmetic noise?

imb

Re: How do I find out what changed for an freebsd release update?

[Ronald Klop]

See https://cgit.freebsd.org/src/log/?h=releng/12.2
Everything on 2021-06-29.


Regards,
Ronald.


Van: Rick Macklem 
Datum: 16 oktober 2021 01:49
Aan: FreeBSD Current 
Onderwerp: How do I find out what changed for an freebsd release update?




Hi,

PR#259163 reports a problem w.r.t. the NFSv3 server that surfaced when
the server was upgraded from 12.2-p8 to 12.2-p9.
I know that nothing changed in NFS server code, but how can I find out
what did get changed by that update?

Thanks in advance for any help, rick

Re: How do I find out what changed for an freebsd release update?

[Herbert J. Skuhra]

On Fri, Oct 15, 2021 at 11:48:01PM +, Rick Macklem wrote:
> Hi,
> 
> PR#259163 reports a problem w.r.t. the NFSv3 server that surfaced when
> the server was upgraded from 12.2-p8 to 12.2-p9.
> I know that nothing changed in NFS server code, but how can I find out
> what did get changed by that update?
> 
> Thanks in advance for any help, rick

1. Check UPDATING https://cgit.freebsd.org/src/tree/UPDATING?h=releng/12.2
2. git log and git diff e.g.: git diff 6e927d10c587..2430d82b40ea

-- 
Herbert

How do I find out what changed for an freebsd release update?

[Rick Macklem]

Hi,

PR#259163 reports a problem w.r.t. the NFSv3 server that surfaced when
the server was upgraded from 12.2-p8 to 12.2-p9.
I know that nothing changed in NFS server code, but how can I find out
what did get changed by that update?

Thanks in advance for any help, rick

Re: What happen to mailing list archives?

[Baptiste Daroussin]

8 juin 2021 13:15:50 Michael Gmelin :

>
>
> On Mon, 7 Jun 2021 22:35:20 +0200
> Baptiste Daroussin  wrote:
>
>> On Mon, Jun 07, 2021 at 12:30:46AM -0700, Mark Millard wrote:
>>> On 2021-Jun-6, at 13:25, Mark Millard  wrote:
>>>  
>>>> Baptiste Daroussin  wrote on
>>>> Date: Sun, 6 Jun 2021 10:53:49 +0200 :
>>>>  
>>>>> What has happended:
>>>>> plan A: we migrated everything off mailman/pipermail seamlessly
>>>>> with redirection and so on. We patched the new archiver to
>>>>> produce the same file names has pipermail
>>>>>
>>>>> Plan A worked fine up to a limit, there was plenty of hand
>>>>> edition in the past, we we decided to move to plan B which is
>>>>> what is happening now.
>>>>>
>>>>> Plan B: We keep a frozen version of the archives up to the
>>>>> migration date under the pipermail directory and have the new
>>>>> archives created in the archives directory.
>>>>>
>>>>> All the pipermail archives have been restored as they were. The
>>>>> new archives receives in their index a new link to point people
>>>>> to the pipermails archive if looking for older archives.
>>>>>
>>>>> this has been done a couple of hours ago (before Steve emails)
>>>>> during a window, of ~ 10 hours, the mailing lists which slow
>>>>> traffic aka the one which didn't received any email since the
>>>>> migration ended up with an empty "archives" directory (aka a
>>>>> 404), a file with explanation and redirection to pipermail has
>>>>> been installed there.
>>>>>
>>>>> Some work is still needed for the mailing lists which has been
>>>>> transformed as readonly, this will be done in the next couple of
>>>>> days 
>>>>
>>>> It is too bad that a reference to a "no examples yet"
>>>> month, such as, (at the time I write this):
>>>>
>>>> https://lists.freebsd.org/archives/freebsd-toolchain/2021-June/date.html
>>>>
>>>> does not show at least (Date view specific example):
>>>>
>>>>   • Other periods:[ Previous, Date view ] [ List of Folders
>>>> ]
>>>>   • Other: [ Actives mailing lists ] [ Archives from
>>>> mailman's time ]
>>>>
>>>> when there are prior months available in
>>>> https://lists.freebsd.org/archives/ or show at least just:
>>>>
>>>>   • Other: [ Actives mailing lists ] [ Archives from
>>>> mailman's time ]
>>>>
>>>> when no prior months are available there.
>>>>  
>>>
>>> Looks like there are missing months.
>>>
>>> Using freebsd-hackers as an example:
>>>
>>> https://lists.freebsd.org/archives/freebsd-hackers/index.html
>>> shows the oldest month being "May 2021".
>>>
>>> But . . .
>>>
>>> https://lists.freebsd.org/pipermail/freebsd-hackers/
>>> shows the most recent month being "September 2020".
>>>
>>> So: 2020-Oct through 2021-Apr are completely missing.
>>>
>>> Then, going for more detail for 2020-Sep and 2021-May . . .
>>>
>>> https://lists.freebsd.org/archives/freebsd-hackers/2021-May/date.html
>>> shows "Starting Tue May 18 2021 - 21:07:44 UTC".
>>>
>>> https://lists.freebsd.org/pipermail/freebsd-hackers/2020-September/date.html
>>> shows "Ending: Tue Sep 15 14:12:20 UTC 2020".
>>>
>>> So there are about 2 more half-months missing.
>>>
>>> Some other lists have other date ranges, some similar,
>>> some not. 
>>
>> This missing month are being populated right now
>
> Will historic links still work though (so that old references work and
> current google results are ok)?
>
> Just stumbled over this one:
> https://lists.freebsd.org/archives/freebsd-jail/2017-March/003360.html
>
Old reference yes, not the one that lived a couple of weeks like this one

Bapt

Re: What happen to mailing list archives?

[Michael Gmelin]

On Mon, 7 Jun 2021 22:35:20 +0200
Baptiste Daroussin  wrote:

> On Mon, Jun 07, 2021 at 12:30:46AM -0700, Mark Millard wrote:
> > On 2021-Jun-6, at 13:25, Mark Millard  wrote:
> >   
> > > Baptiste Daroussin  wrote on
> > > Date: Sun, 6 Jun 2021 10:53:49 +0200 :
> > >   
> > >> What has happended:
> > >> plan A: we migrated everything off mailman/pipermail seamlessly
> > >> with redirection and so on. We patched the new archiver to
> > >> produce the same file names has pipermail
> > >> 
> > >> Plan A worked fine up to a limit, there was plenty of hand
> > >> edition in the past, we we decided to move to plan B which is
> > >> what is happening now.
> > >> 
> > >> Plan B: We keep a frozen version of the archives up to the
> > >> migration date under the pipermail directory and have the new
> > >> archives created in the archives directory.
> > >> 
> > >> All the pipermail archives have been restored as they were. The
> > >> new archives receives in their index a new link to point people
> > >> to the pipermails archive if looking for older archives.
> > >> 
> > >> this has been done a couple of hours ago (before Steve emails)
> > >> during a window, of ~ 10 hours, the mailing lists which slow
> > >> traffic aka the one which didn't received any email since the
> > >> migration ended up with an empty "archives" directory (aka a
> > >> 404), a file with explanation and redirection to pipermail has
> > >> been installed there.
> > >> 
> > >> Some work is still needed for the mailing lists which has been
> > >> transformed as readonly, this will be done in the next couple of
> > >> days  
> > > 
> > > It is too bad that a reference to a "no examples yet"
> > > month, such as, (at the time I write this):
> > > 
> > > https://lists.freebsd.org/archives/freebsd-toolchain/2021-June/date.html
> > > 
> > > does not show at least (Date view specific example):
> > > 
> > >   • Other periods:[ Previous, Date view ] [ List of Folders
> > > ]
> > >   • Other: [ Actives mailing lists ] [ Archives from
> > > mailman's time ]
> > > 
> > > when there are prior months available in
> > > https://lists.freebsd.org/archives/ or show at least just:
> > > 
> > >   • Other: [ Actives mailing lists ] [ Archives from
> > > mailman's time ]
> > > 
> > > when no prior months are available there.
> > >   
> > 
> > Looks like there are missing months.
> > 
> > Using freebsd-hackers as an example:
> > 
> > https://lists.freebsd.org/archives/freebsd-hackers/index.html
> > shows the oldest month being "May 2021".
> > 
> > But . . .
> > 
> > https://lists.freebsd.org/pipermail/freebsd-hackers/
> > shows the most recent month being "September 2020".
> > 
> > So: 2020-Oct through 2021-Apr are completely missing.
> > 
> > Then, going for more detail for 2020-Sep and 2021-May . . .
> > 
> > https://lists.freebsd.org/archives/freebsd-hackers/2021-May/date.html
> > shows "Starting Tue May 18 2021 - 21:07:44 UTC".
> > 
> > https://lists.freebsd.org/pipermail/freebsd-hackers/2020-September/date.html
> > shows "Ending: Tue Sep 15 14:12:20 UTC 2020".
> > 
> > So there are about 2 more half-months missing.
> > 
> > Some other lists have other date ranges, some similar,
> > some not.  
> 
> This missing month are being populated right now

Will historic links still work though (so that old references work and
current google results are ok)?

Just stumbled over this one:
https://lists.freebsd.org/archives/freebsd-jail/2017-March/003360.html

Cheers
Michael

> 
> Best regards,
> Bapt



-- 
Michael Gmelin

Re: What happen to mailing list archives?

[Baptiste Daroussin]

On Mon, Jun 07, 2021 at 12:30:46AM -0700, Mark Millard wrote:
> On 2021-Jun-6, at 13:25, Mark Millard  wrote:
> 
> > Baptiste Daroussin  wrote on
> > Date: Sun, 6 Jun 2021 10:53:49 +0200 :
> > 
> >> What has happended:
> >> plan A: we migrated everything off mailman/pipermail seamlessly with 
> >> redirection
> >> and so on. We patched the new archiver to produce the same file names has
> >> pipermail
> >> 
> >> Plan A worked fine up to a limit, there was plenty of hand edition in the 
> >> past,
> >> we we decided to move to plan B which is what is happening now.
> >> 
> >> Plan B: We keep a frozen version of the archives up to the migration date 
> >> under
> >> the pipermail directory and have the new archives created in the archives
> >> directory.
> >> 
> >> All the pipermail archives have been restored as they were. The new 
> >> archives
> >> receives in their index a new link to point people to the pipermails 
> >> archive if
> >> looking for older archives.
> >> 
> >> this has been done a couple of hours ago (before Steve emails) during a 
> >> window,
> >> of ~ 10 hours, the mailing lists which slow traffic aka the one which 
> >> didn't
> >> received any email since the migration ended up with an empty "archives"
> >> directory (aka a 404), a file with explanation and redirection to 
> >> pipermail has
> >> been installed there.
> >> 
> >> Some work is still needed for the mailing lists which has been transformed 
> >> as
> >> readonly, this will be done in the next couple of days
> > 
> > It is too bad that a reference to a "no examples yet"
> > month, such as, (at the time I write this):
> > 
> > https://lists.freebsd.org/archives/freebsd-toolchain/2021-June/date.html
> > 
> > does not show at least (Date view specific example):
> > 
> > • Other periods:[ Previous, Date view ] [ List of Folders ]
> > • Other: [ Actives mailing lists ] [ Archives from mailman's time ]
> > 
> > when there are prior months available in
> > https://lists.freebsd.org/archives/ or show at least just:
> > 
> > • Other: [ Actives mailing lists ] [ Archives from mailman's time ]
> > 
> > when no prior months are available there.
> > 
> 
> Looks like there are missing months.
> 
> Using freebsd-hackers as an example:
> 
> https://lists.freebsd.org/archives/freebsd-hackers/index.html
> shows the oldest month being "May 2021".
> 
> But . . .
> 
> https://lists.freebsd.org/pipermail/freebsd-hackers/
> shows the most recent month being "September 2020".
> 
> So: 2020-Oct through 2021-Apr are completely missing.
> 
> Then, going for more detail for 2020-Sep and 2021-May . . .
> 
> https://lists.freebsd.org/archives/freebsd-hackers/2021-May/date.html
> shows "Starting Tue May 18 2021 - 21:07:44 UTC".
> 
> https://lists.freebsd.org/pipermail/freebsd-hackers/2020-September/date.html
> shows "Ending: Tue Sep 15 14:12:20 UTC 2020".
> 
> So there are about 2 more half-months missing.
> 
> Some other lists have other date ranges, some similar,
> some not.

This missing month are being populated right now

Best regards,
Bapt

Re: What happen to mailing list archives?

[Mark Millard via freebsd-current]

On 2021-Jun-6, at 13:25, Mark Millard  wrote:

> Baptiste Daroussin  wrote on
> Date: Sun, 6 Jun 2021 10:53:49 +0200 :
> 
>> What has happended:
>> plan A: we migrated everything off mailman/pipermail seamlessly with 
>> redirection
>> and so on. We patched the new archiver to produce the same file names has
>> pipermail
>> 
>> Plan A worked fine up to a limit, there was plenty of hand edition in the 
>> past,
>> we we decided to move to plan B which is what is happening now.
>> 
>> Plan B: We keep a frozen version of the archives up to the migration date 
>> under
>> the pipermail directory and have the new archives created in the archives
>> directory.
>> 
>> All the pipermail archives have been restored as they were. The new archives
>> receives in their index a new link to point people to the pipermails archive 
>> if
>> looking for older archives.
>> 
>> this has been done a couple of hours ago (before Steve emails) during a 
>> window,
>> of ~ 10 hours, the mailing lists which slow traffic aka the one which didn't
>> received any email since the migration ended up with an empty "archives"
>> directory (aka a 404), a file with explanation and redirection to pipermail 
>> has
>> been installed there.
>> 
>> Some work is still needed for the mailing lists which has been transformed as
>> readonly, this will be done in the next couple of days
> 
> It is too bad that a reference to a "no examples yet"
> month, such as, (at the time I write this):
> 
> https://lists.freebsd.org/archives/freebsd-toolchain/2021-June/date.html
> 
> does not show at least (Date view specific example):
> 
>   • Other periods:[ Previous, Date view ] [ List of Folders ]
>   • Other: [ Actives mailing lists ] [ Archives from mailman's time ]
> 
> when there are prior months available in
> https://lists.freebsd.org/archives/ or show at least just:
> 
>   • Other: [ Actives mailing lists ] [ Archives from mailman's time ]
> 
> when no prior months are available there.
> 

Looks like there are missing months.

Using freebsd-hackers as an example:

https://lists.freebsd.org/archives/freebsd-hackers/index.html
shows the oldest month being "May 2021".

But . . .

https://lists.freebsd.org/pipermail/freebsd-hackers/
shows the most recent month being "September 2020".

So: 2020-Oct through 2021-Apr are completely missing.

Then, going for more detail for 2020-Sep and 2021-May . . .

https://lists.freebsd.org/archives/freebsd-hackers/2021-May/date.html
shows "Starting Tue May 18 2021 - 21:07:44 UTC".

https://lists.freebsd.org/pipermail/freebsd-hackers/2020-September/date.html
shows "Ending: Tue Sep 15 14:12:20 UTC 2020".

So there are about 2 more half-months missing.

Some other lists have other date ranges, some similar,
some not.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Re: What happen to mailing list archives?

[Mark Millard via freebsd-current]

Baptiste Daroussin  wrote on
Date: Sun, 6 Jun 2021 10:53:49 +0200 :

> What has happended:
> plan A: we migrated everything off mailman/pipermail seamlessly with 
> redirection
> and so on. We patched the new archiver to produce the same file names has
> pipermail
> 
> Plan A worked fine up to a limit, there was plenty of hand edition in the 
> past,
> we we decided to move to plan B which is what is happening now.
> 
> Plan B: We keep a frozen version of the archives up to the migration date 
> under
> the pipermail directory and have the new archives created in the archives
> directory.
> 
> All the pipermail archives have been restored as they were. The new archives
> receives in their index a new link to point people to the pipermails archive 
> if
> looking for older archives.
> 
> this has been done a couple of hours ago (before Steve emails) during a 
> window,
> of ~ 10 hours, the mailing lists which slow traffic aka the one which didn't
> received any email since the migration ended up with an empty "archives"
> directory (aka a 404), a file with explanation and redirection to pipermail 
> has
> been installed there.
> 
> Some work is still needed for the mailing lists which has been transformed as
> readonly, this will be done in the next couple of days

It is too bad that a reference to a "no examples yet"
month, such as, (at the time I write this):

https://lists.freebsd.org/archives/freebsd-toolchain/2021-June/date.html

does not show at least (Date view specific example):

• Other periods:[ Previous, Date view ] [ List of Folders ]
• Other: [ Actives mailing lists ] [ Archives from mailman's time ]

when there are prior months available in
https://lists.freebsd.org/archives/ or show at least just:

• Other: [ Actives mailing lists ] [ Archives from mailman's time ]

when no prior months are available there.


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Re: What happen to mailing list archives?

[Rainer Hurling]

Hi Steve,

Am 06.06.21 um 01:13 schrieb Steve Kargl:
> On Sun, Jun 06, 2021 at 01:00:40AM +0200, Michael Gmelin wrote:
>>
>>
>>> On 6. Jun 2021, at 00:53, Steve Kargl  
>>> wrote:
>>>
>>> It seems someone has tried to migrate the mailing list archives
>>> from mailman to some new fangle code.  This has broken the archives
>>> for at least freebsd-numerics@, freebsd-office@, freebsd-net@
>>>
>>> As a comparison, simply go to
>>>
>>> https://lists.freebsd.org/mailman/listinfo
>>>
>>> and follow the links to freebsd-numerics and freebsd-toolchain.
>>>
>>> Can this be fixed?
>>>
>>
>> New archives are here:
>> https://lists.freebsd.org/archives/
>>
> 
> If I go to https://www.freebsd.org/community/mailinglists/
> 
> I see the lines
> 
>   Mailing list archives
> 
>   You can search or browse the mailing list archives at www.FreeBSD.org.
>   It is also possible to browse the mailing lists via the Mailman Web
>   interface.
> 
> Both instances of the word "browse" are hyperlinks.  If I click of the
> second "browse" link, then I end up at
> 
> https://lists.freebsd.org/mailman/listinfo
> 
> which presents a list of archives, you'll see freebsd-numerics
> is a hyperlink (as well as many others).  If I click on freebsd-numerics,
> this takes me to
> 
> https://lists.freebsd.org/subscription/freebsd-numerics
> 
> which displays
> 
>   Subscription for freebsd-numerics
> 
>  Browse the archives: here
>  (and 6 hyperlinks for subscribing and unsubscribing).
> 
> If I click on "here", I get

When I click on "here" I also get a "404 Not Found" at first, but only
for about 5 seconds. This is followed by a redirect to the archive list.
Maybe try it again?

If that doesn't work for you: I have saved some threads with Bruce Evans
locally. I could provide them, hoping that there is something useful?

Regards,
Rainer


> 
> 404 Not Found
> 
> Ergo, the freebsd-numerics archive is unreachable from the "here" link.
> Shouldn't this redirect to the new fangle way?
> 

Re: What happen to mailing list archives?

[Baptiste Daroussin]

On Sat, Jun 05, 2021 at 05:43:01PM -0600, Warner Losh wrote:
> On Sat, Jun 5, 2021, 5:29 PM Steve Kargl 
> wrote:
> 
> > On Sun, Jun 06, 2021 at 01:04:46AM +0200, Michael Gmelin wrote:
> > >
> > > p.s. If you go to https://lists.freebsd.org/mailman/listinfo, click
> > FreeBSD-net and then "Archives from mailman’s time", you get to a list that
> > brings you to https://lists.freebsd.org/pipermail/freebsd-net/. So the
> > old archives are still reachable that way. (I still find the
> > docs.FreeBSD.org/mail page to be confusing).
> > >
> > >
> >
> > There isn't an "Archives from mailman's time" link on freebsd-numerics.
> >
> > Hundreds of emails from me are now gone or sufficiently hidden that
> > I cannot find them.  More importantly, Bruce Evans often replied
> > with reviews of libm patch's I sent the list.  Those reviews and
> > his detailed analysis of the libm code are now gone or sufficiently
> > hidded that I cannot find them.
> >
> 
> 
> Bapt was working on this stuff today. He said something about the archive
> regenerating with the new software on IRC..
> 
> Warner
> 
What has happended:
plan A: we migrated everything off mailman/pipermail seamlessly with redirection
and so on. We patched the new archiver to produce the same file names has
pipermail

Plan A worked fine up to a limit, there was plenty of hand edition in the past,
we we decided to move to plan B which is what is happening now.

Plan B: We keep a frozen version of the archives up to the migration date under
the pipermail directory and have the new archives created in the archives
directory.

All the pipermail archives have been restored as they were. The new archives
receives in their index a new link to point people to the pipermails archive if
looking for older archives.

this has been done a couple of hours ago (before Steve emails) during a window,
of ~ 10 hours, the mailing lists which slow traffic aka the one which didn't
received any email since the migration ended up with an empty "archives"
directory (aka a 404), a file with explanation and redirection to pipermail has
been installed there.

Some work is still needed for the mailing lists which has been transformed as
readonly, this will be done in the next couple of days

Bapt

FreeBSD service bug 256182 (was: What happen to mailing list archives?)

[Graham Perrin]

On 06/06/2021 00:28, Steve Kargl wrote:

Hundreds of emails from me are now gone or sufficiently hidden that
I cannot find them.  More importantly, Bruce Evans often replied
with reviews of libm patch's I sent the list.  Those reviews and
his detailed analysis of the libm code are now gone or sufficiently
hidded that I cannot find them.


256182 – lists.freebsd.org/pipermail/… redirections to 
lists.freebsd.org/archives/…: sometimes to the wrong content, sometimes 
not found (404)

Re: What happen to mailing list archives?

[grarpamp]

There is also this useful and efficient form of archive/mirror to include
in the update so that it does not remain broken for too long...

https://lists.freebsd.org/pipermail/freebsd-questions/2021-June/294104.html
https://lists.freebsd.org/archives/freebsd-hubs/2021-June/00.html

Re: What happen to mailing list archives?

[Michael Gmelin]

> On 6. Jun 2021, at 01:44, Warner Losh  wrote:
> 
> 
> 
> 
>> On Sat, Jun 5, 2021, 5:29 PM Steve Kargl  
>> wrote:
>> On Sun, Jun 06, 2021 at 01:04:46AM +0200, Michael Gmelin wrote:
>> > 
>> > p.s. If you go to https://lists.freebsd.org/mailman/listinfo, click 
>> > FreeBSD-net and then "Archives from mailman’s time", you get to a list 
>> > that brings you to https://lists.freebsd.org/pipermail/freebsd-net/. So 
>> > the old archives are still reachable that way. (I still find the 
>> > docs.FreeBSD.org/mail page to be confusing).
>> > 
>> > 
>> 
>> There isn't an "Archives from mailman's time" link on freebsd-numerics.
>> 
>> Hundreds of emails from me are now gone or sufficiently hidden that
>> I cannot find them.  More importantly, Bruce Evans often replied
>> with reviews of libm patch's I sent the list.  Those reviews and
>> his detailed analysis of the libm code are now gone or sufficiently
>> hidded that I cannot find them.
> 
> 
> 
> Bapt was working on this stuff today. He said something about the archive 
> regenerating with the new software on IRC.. 
> 

In the meantime, maybe this helps:

https://lists.freebsd.org/pipermail/freebsd-numerics/


> Warner 

Re: What happen to mailing list archives?

[Warner Losh]

On Sat, Jun 5, 2021, 5:29 PM Steve Kargl 
wrote:

> On Sun, Jun 06, 2021 at 01:04:46AM +0200, Michael Gmelin wrote:
> >
> > p.s. If you go to https://lists.freebsd.org/mailman/listinfo, click
> FreeBSD-net and then "Archives from mailman’s time", you get to a list that
> brings you to https://lists.freebsd.org/pipermail/freebsd-net/. So the
> old archives are still reachable that way. (I still find the
> docs.FreeBSD.org/mail page to be confusing).
> >
> >
>
> There isn't an "Archives from mailman's time" link on freebsd-numerics.
>
> Hundreds of emails from me are now gone or sufficiently hidden that
> I cannot find them.  More importantly, Bruce Evans often replied
> with reviews of libm patch's I sent the list.  Those reviews and
> his detailed analysis of the libm code are now gone or sufficiently
> hidded that I cannot find them.
>


Bapt was working on this stuff today. He said something about the archive
regenerating with the new software on IRC..

Warner

>

Re: What happen to mailing list archives?

[Steve Kargl]

On Sun, Jun 06, 2021 at 01:04:46AM +0200, Michael Gmelin wrote:
> 
> p.s. If you go to https://lists.freebsd.org/mailman/listinfo, click 
> FreeBSD-net and then "Archives from mailman’s time", you get to a list that 
> brings you to https://lists.freebsd.org/pipermail/freebsd-net/. So the old 
> archives are still reachable that way. (I still find the 
> docs.FreeBSD.org/mail page to be confusing).
> 
> 

There isn't an "Archives from mailman's time" link on freebsd-numerics.

Hundreds of emails from me are now gone or sufficiently hidden that
I cannot find them.  More importantly, Bruce Evans often replied
with reviews of libm patch's I sent the list.  Those reviews and
his detailed analysis of the libm code are now gone or sufficiently
hidded that I cannot find them.

-- 
Steve

Re: What happen to mailing list archives?

[Steve Kargl]

On Sun, Jun 06, 2021 at 01:00:40AM +0200, Michael Gmelin wrote:
> 
> 
> > On 6. Jun 2021, at 00:53, Steve Kargl  
> > wrote:
> > 
> > It seems someone has tried to migrate the mailing list archives
> > from mailman to some new fangle code.  This has broken the archives
> > for at least freebsd-numerics@, freebsd-office@, freebsd-net@
> > 
> > As a comparison, simply go to
> > 
> > https://lists.freebsd.org/mailman/listinfo
> > 
> > and follow the links to freebsd-numerics and freebsd-toolchain.
> > 
> > Can this be fixed?
> > 
> 
> New archives are here:
> https://lists.freebsd.org/archives/
> 

If I go to https://www.freebsd.org/community/mailinglists/

I see the lines

  Mailing list archives

  You can search or browse the mailing list archives at www.FreeBSD.org.
  It is also possible to browse the mailing lists via the Mailman Web
  interface.

Both instances of the word "browse" are hyperlinks.  If I click of the
second "browse" link, then I end up at

https://lists.freebsd.org/mailman/listinfo

which presents a list of archives, you'll see freebsd-numerics
is a hyperlink (as well as many others).  If I click on freebsd-numerics,
this takes me to

https://lists.freebsd.org/subscription/freebsd-numerics

which displays

  Subscription for freebsd-numerics

 Browse the archives:   here
 (and 6 hyperlinks for subscribing and unsubscribing).

If I click on "here", I get

404 Not Found

Ergo, the freebsd-numerics archive is unreachable from the "here" link.
Shouldn't this redirect to the new fangle way?

-- 
Steve

Re: What happen to mailing list archives?

[Michael Gmelin]

> On 6. Jun 2021, at 01:00, Michael Gmelin  wrote:
> 
> 
> 
> 
>>> On 6. Jun 2021, at 00:53, Steve Kargl  
>>> wrote:
>>> 
>> It seems someone has tried to migrate the mailing list archives
>> from mailman to some new fangle code.  This has broken the archives
>> for at least freebsd-numerics@, freebsd-office@, freebsd-net@
>> 
>> As a comparison, simply go to
>> 
>> https://lists.freebsd.org/mailman/listinfo
>> 
>> and follow the links to freebsd-numerics and freebsd-toolchain.
>> 
>> Can this be fixed?
>> 
> 
> New archives are here:
> https://lists.freebsd.org/archives/
> 
> e.g., https://lists.freebsd.org/archives/freebsd-net/2021-June/index.html
> 
> As we still have mailman archives and https://docs.freebsd.org/mail/, merging 
> or cross linking would certainly be useful.
> 
> Michael
> 

p.s. If you go to https://lists.freebsd.org/mailman/listinfo, click FreeBSD-net 
and then "Archives from mailman’s time", you get to a list that brings you to 
https://lists.freebsd.org/pipermail/freebsd-net/. So the old archives are still 
reachable that way. (I still find the docs.FreeBSD.org/mail page to be 
confusing).




> 
>> -- 
>> Steve

Re: What happen to mailing list archives?

[Michael Gmelin]

> On 6. Jun 2021, at 00:53, Steve Kargl  
> wrote:
> 
> It seems someone has tried to migrate the mailing list archives
> from mailman to some new fangle code.  This has broken the archives
> for at least freebsd-numerics@, freebsd-office@, freebsd-net@
> 
> As a comparison, simply go to
> 
> https://lists.freebsd.org/mailman/listinfo
> 
> and follow the links to freebsd-numerics and freebsd-toolchain.
> 
> Can this be fixed?
> 

New archives are here:
https://lists.freebsd.org/archives/

e.g., https://lists.freebsd.org/archives/freebsd-net/2021-June/index.html

As we still have mailman archives and https://docs.freebsd.org/mail/, merging 
or cross linking would certainly be useful.

Michael


> -- 
> Steve

What happen to mailing list archives?

[Steve Kargl]

It seems someone has tried to migrate the mailing list archives
from mailman to some new fangle code.  This has broken the archives
for at least freebsd-numerics@, freebsd-office@, freebsd-net@

As a comparison, simply go to

https://lists.freebsd.org/mailman/listinfo

and follow the links to freebsd-numerics and freebsd-toolchain.

Can this be fixed?

-- 
Steve

Re: What is OpenZFS doing during boot?

[Alan Somers]

On Fri, Apr 30, 2021 at 7:21 AM Ulrich Spörlein  wrote:

> Hi folks, this is a stable/13 question but I figured it's still close
> enough to -CURRENT to count.
>
> So I wanted to update my (remote) system with freebsd-update, but that
> installed half a kernel and bricked the machine upon reboot. Lucky me I
> fixed OOB access just the day before.
>
> Did the usual world/kernel build and ran etcupdate, merging in my
> local changes. This bricked the system again, as it removed the -x bit on
> /etc/rc.d/netif, I filed
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255514 for that though
> (I
> never had such trouble with mergemaster, just even understanding what
> etcupdate is trying to do and how to bootstrap it is a mystery to me).
>
> Anyway, I have a data zpool on 2x encrypted GELI providers that I can only
> unlock (and zpool import) with 2 passphrases after the system has booted.
>
> Color me surprised when some RC script thought otherwise and tried to
> import the pool during boot. Why does it do that, that's not supposed to
> work and it should not even touch the encrypted bits (yet).
>
> mountroot: waiting for device /dev/mirror/gm0a...
> Dual Console: Serial Primary, Video Secondary
> GEOM_ELI: Device gpt/swap0.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI: Crypto: accelerated software
> GEOM_ELI: Device gpt/swap1.eli created.
> GEOM_ELI: Encryption: AES-XTS 128
> GEOM_ELI: Crypto: accelerated software
> Setting hostuuid: d7902500-4c7c-0706-0025-90d77c4c0e0f.
> Setting hostid: 0x8a2b4277.
> cannot import 'data': no such pool or dataset
> Destroy and re-create the pool fipmi0: Unknown IOCTL 40086481
> ipmi0: Unknown IOCTL 40086481
> rom
> a backup source.
> cachefile import failed, retrying
> nvpair_value_nvlpid 69 (zpool), jid 0, uid ist(nvp, ) == 0 (0x16 == 0)
> ASSERT at /usr/src/sys/contrib/openzfs/module/nv0: exited on signal 6
> pair/fnvpair.c:586:fnvpair_value_nvlist()Abort trap
> cannot import 'data': no such pool or dataset
> ipmi0: Unknown IOCTL 40086481
> ipmi0: Unknown IOCTL 40086481
> Destroy and re-cpid 74 (zpool), jid 0, uid 0: exited on signal 6
> reate the pool from
> a backup source.
> cachefile import failed, retrying
> nvpair_value_nvlist(nvp, ) == 0 (0x16 == 0)
> ASSERT at
>
> /usr/src/sys/contrib/openzfs/module/nvpair/fnvpair.c:586:fnvpair_value_nvlist()Abort
> trap
> Starting file system checks:
> /dev/mirror/gm0a: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/mirror/gm0a: clean, 370582 free (814 frags, 46221 blocks, 0.2%
> fragmentation)
> /dev/mirror/gm0d: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/mirror/gm0d: clean, 867640 free (1160 frags, 108310 blocks, 0.1%
> fragmentation)
> /dev/mirror/gm0e: FILE SYSTEM CLEAN; SKIPPING CHECKS
> /dev/mirror/gm0e: clean, 1267948 free (17228 frags, 156340 blocks, 0.7%
> fragmentation)
> Mounting local filesystems:.
>
>
> What do I need to do to _not_ have any zpool operations be attempted during
> startup? How does it even know of the existence of that pool?
>
> I guess it's zfs_enable=NO to stop /etc/rc.d/zpool from messing about. But
> more importantly, the GELI providers don't exist yet, why does it then
> segfault? Shouldn't it be a bit more robust on that front?
>
> Thanks all
> Uli
>

Your problem is the zpool cache file.  As soon as ZFS loads, it tries to
import all pools mentioned in /boot/zfs/zpool.cache.  If you're using ZFS
on top of GELI, then obviously you don't want that.  What you should is
move the cachefile somewhere else.  Do it like this:
$ zpool set cachefile=/some/where/else my-data-pool

And on every boot, import it like this:
$ service geli start
$ zpool import -a -c /some/where/else -o cachefile=/some/where/else

Hope this helps.
-Alan
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

What is OpenZFS doing during boot?

[Ulrich Spörlein]

Hi folks, this is a stable/13 question but I figured it's still close
enough to -CURRENT to count.

So I wanted to update my (remote) system with freebsd-update, but that
installed half a kernel and bricked the machine upon reboot. Lucky me I
fixed OOB access just the day before.

Did the usual world/kernel build and ran etcupdate, merging in my
local changes. This bricked the system again, as it removed the -x bit on
/etc/rc.d/netif, I filed
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255514 for that though (I
never had such trouble with mergemaster, just even understanding what
etcupdate is trying to do and how to bootstrap it is a mystery to me).

Anyway, I have a data zpool on 2x encrypted GELI providers that I can only
unlock (and zpool import) with 2 passphrases after the system has booted.

Color me surprised when some RC script thought otherwise and tried to
import the pool during boot. Why does it do that, that's not supposed to
work and it should not even touch the encrypted bits (yet).

mountroot: waiting for device /dev/mirror/gm0a...
Dual Console: Serial Primary, Video Secondary
GEOM_ELI: Device gpt/swap0.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: accelerated software
GEOM_ELI: Device gpt/swap1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: accelerated software
Setting hostuuid: d7902500-4c7c-0706-0025-90d77c4c0e0f.
Setting hostid: 0x8a2b4277.
cannot import 'data': no such pool or dataset
Destroy and re-create the pool fipmi0: Unknown IOCTL 40086481
ipmi0: Unknown IOCTL 40086481
rom
a backup source.
cachefile import failed, retrying
nvpair_value_nvlpid 69 (zpool), jid 0, uid ist(nvp, ) == 0 (0x16 == 0)
ASSERT at /usr/src/sys/contrib/openzfs/module/nv0: exited on signal 6
pair/fnvpair.c:586:fnvpair_value_nvlist()Abort trap
cannot import 'data': no such pool or dataset
ipmi0: Unknown IOCTL 40086481
ipmi0: Unknown IOCTL 40086481
Destroy and re-cpid 74 (zpool), jid 0, uid 0: exited on signal 6
reate the pool from
a backup source.
cachefile import failed, retrying
nvpair_value_nvlist(nvp, ) == 0 (0x16 == 0)
ASSERT at
/usr/src/sys/contrib/openzfs/module/nvpair/fnvpair.c:586:fnvpair_value_nvlist()Abort
trap
Starting file system checks:
/dev/mirror/gm0a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mirror/gm0a: clean, 370582 free (814 frags, 46221 blocks, 0.2%
fragmentation)
/dev/mirror/gm0d: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mirror/gm0d: clean, 867640 free (1160 frags, 108310 blocks, 0.1%
fragmentation)
/dev/mirror/gm0e: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/mirror/gm0e: clean, 1267948 free (17228 frags, 156340 blocks, 0.7%
fragmentation)
Mounting local filesystems:.


What do I need to do to _not_ have any zpool operations be attempted during
startup? How does it even know of the existence of that pool?

I guess it's zfs_enable=NO to stop /etc/rc.d/zpool from messing about. But
more importantly, the GELI providers don't exist yet, why does it then
segfault? Shouldn't it be a bit more robust on that front?

Thanks all
Uli
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Old PowerMacs, Giant, and stable/13 branch and other futures: What is the intent?

[Mark Millard]

On 2021-Jan-12, at 13:11, Mark Millard  wrote:

> PowerMac G5's and G4's with not much in them or connected to them report:
> 
> WARNING: Device "openfirm" is Giant locked and may be deleted before FreeBSD 
> 13.0.
> WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 13.0.
> 
> There is also, using and example G4 that I have access to,
> 
> WARNING: Device "agp" is Giant locked and may be deleted before FreeBSD 13.0.
> WARNING: Device "consolectl" is Giant locked and may be deleted before 
> FreeBSD 13.0.
> 
> There could be more than I'm reporting. Such warnings are from
> post-git-conversion builds that I've done (given the lack of
> artifacts.ci.freebsd.org materials to test with).
> 
> The 13 "Code slush" started on 2021-Jan-08. stable/13 is scheduled
> to branch on 2021-Jan-22.
> 
> What is the intended status for these old powerc64 and 32-bit
> powerpc machines, given that the Giant use still exists? Intended
> status for 13? For 14?
> 
> (There may be other types of issues that I'm not thinking of
> as well.)
> 



It turns out that the below was apparently dealt with in
13-CURRENT's -r362406 ( 37f530582dc5 ).

> Another that used to happen on both G5's and G4's that I have/had
> access to:
> 
> WARNING: Device "powermac_nvram" is Giant locked and may be deleted before 
> FreeBSD 13.0.
> 
> I'm unsure if this one is modal in some way and it I happen to
> be using a mode that avoided the usage. (I do not normally run an
> unpatched FreeBSD on old PowerMacs as well.)
> 

powermac_nvram had its Giant lock use removed/replaced, no modal
issue is involved.


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Old PowerMacs, Giant, and stable/13 branch and other futures: What is the intent?

[Mark Millard]

PowerMac G5's and G4's with not much in them or connected to them report:

WARNING: Device "openfirm" is Giant locked and may be deleted before FreeBSD 
13.0.
WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 13.0.

There is also, using and example G4 that I have access to,

WARNING: Device "agp" is Giant locked and may be deleted before FreeBSD 13.0.
WARNING: Device "consolectl" is Giant locked and may be deleted before FreeBSD 
13.0.

There could be more than I'm reporting. Such warnings are from
post-git-conversion builds that I've done (given the lack of
artifacts.ci.freebsd.org materials to test with).

The 13 "Code slush" started on 2021-Jan-08. stable/13 is scheduled
to branch on 2021-Jan-22.

What is the intended status for these old powerc64 and 32-bit
powerpc machines, given that the Giant use still exists? Intended
status for 13? For 14?

(There may be other types of issues that I'm not thinking of
as well.)

Another that used to happen on both G5's and G4's that I have/had
access to:

WARNING: Device "powermac_nvram" is Giant locked and may be deleted before 
FreeBSD 13.0.

I'm unsure if this one is modal in some way and it I happen to
be using a mode that avoided the usage. (I do not normally run an
unpatched FreeBSD on old PowerMacs as well.)

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: What is the zstreamdump command in OpenZFS?

[Ryan Moeller]

On 9/1/20 2:13 PM, Ryan Moeller wrote:


On 9/1/20 1:32 PM, Thomas Laus wrote:

Group:

Whenever I perform a major upgrade to any of my ZFS pools, I make a
habit of sending the most recent snapshot to a file and then to a DLT
tape or DVD.  I run the zstreamdump utility on the file and record the
checksum.  I also run zstreamdump on the archive media and compare
checksums.  With the recent ZFS upgrade in CURRENT, I could not locate
this utility.  With a major upgrade such as this, I was really concerned
about not being able to boot and loosing everything on a production
system.  Fortunately this upgrade went smoothly and I did not have to
restore from alternate media.  The next time, I may not be lucky.

Tom

zstreamdump is supposed to be a script wrapper around a new, more 
general, command: zstream




#!/bin/sh

zstream dump "$@"



It doesn't appear to be installing itself properly, though. I'll have 
to look into that.



https://svnweb.freebsd.org/changeset/base/365057




-Ryan

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: What is the zstreamdump command in OpenZFS?

[Ryan Moeller]

On 9/1/20 1:32 PM, Thomas Laus wrote:

Group:

Whenever I perform a major upgrade to any of my ZFS pools, I make a
habit of sending the most recent snapshot to a file and then to a DLT
tape or DVD.  I run the zstreamdump utility on the file and record the
checksum.  I also run zstreamdump on the archive media and compare
checksums.  With the recent ZFS upgrade in CURRENT, I could not locate
this utility.  With a major upgrade such as this, I was really concerned
about not being able to boot and loosing everything on a production
system.  Fortunately this upgrade went smoothly and I did not have to
restore from alternate media.  The next time, I may not be lucky.

Tom

zstreamdump is supposed to be a script wrapper around a new, more 
general, command: zstream




#!/bin/sh

zstream dump "$@"



It doesn't appear to be installing itself properly, though. I'll have to 
look into that.


-Ryan

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

What is the zstreamdump command in OpenZFS?

[Thomas Laus]

Group:

Whenever I perform a major upgrade to any of my ZFS pools, I make a
habit of sending the most recent snapshot to a file and then to a DLT
tape or DVD.  I run the zstreamdump utility on the file and record the
checksum.  I also run zstreamdump on the archive media and compare
checksums.  With the recent ZFS upgrade in CURRENT, I could not locate
this utility.  With a major upgrade such as this, I was really concerned
about not being able to boot and loosing everything on a production
system.  Fortunately this upgrade went smoothly and I did not have to
restore from alternate media.  The next time, I may not be lucky.

Tom

-- 
Public Keys:
PGP KeyID = 0x5F22FDC1
GnuPG KeyID = 0x620836CF
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: what 3rd party boot mgr is required to boot multiple freebsd versions?

[Gary Palmer]

On Tue, Mar 17, 2020 at 09:02:14AM -0700, Chris wrote:
> On Tue, 17 Mar 2020 16:17:31 +0200 Toomas Soome tso...@me.com said
> 
> > > On 17. Mar 2020, at 15:51, Matthew Seaman  wrote:
> > > > On 17/03/2020 12:58, Florian Limberger wrote:
> > >> On 16.03.20 23:33, Chris wrote:
> > >> >>> For the record. I'm *only* using FreeBSD in this situation. I
> > >>> only mentioned Windows above, for the use of it's boot manager.
> > >> >> If you only use FreeBSD, and also use ZFS, you might find beadm[1]
> > >> interesting.
> > >> >> [1]: https://www.freshports.org/sysutils/beadm
> > >> > > Did you know that the system now comes with bectl(8) which is
> > very
> > > similar to beadm?  As far as I can tell, the biggest difference is that
> > > if you have more than one ZFS in your boot environment then:
> > > >beadm create FOO
> > > > is actually equivalent to
> > > >bectl create -r FOO
> > > > ie. turning on the recursive functionality in bectl.
> > > > However, this is not really what the OP was asking about.  As I
> > > understand it, they were looking for something that would allow choosing
> > > between several independent installations of different versions of
> > > FreeBSD, rather than having multiple environments in the same
> > > installation.  You can achieve pretty much the same effect though --
> > > there is a boot menu option to switch between BEs.  You would have to
> > > manage any ported software between the different OS versions, perhaps by
> > > including /usr/local and /var/db/pkg are both parts of your BEs.
> > > > Cheers,
> > > > Matthew
> > >
> > 
> > BE???s can solve some scenarios. However, it is easy to add support for few
> > more. The current BE menu is populated automatically from the zfs. However,
> > it is also simple task to add a file parser to it and read menu file with
> > entries with different pool (we only need to activate such entries same way
> > as it is currently done for ???normal??? BE, or entries with chain load).
> > Read this menu file first and zfs BE list after, and you have BE menu with
> > manual and automatic entries. Can be implemented within few hours.
> > 
> A *huge* thanks for all the thoughtful replies!
> In detail. I maintain *many* ports, and it's not always enough to ensure
> that they build properly. In some cases I need to ensure they actually
> operate on FreeBSD/some-version. To test for building; it's been enough for
> me to spin up any number of jails using whatever (fbsd) revision I'm testing
> against. I had/am hoping that I can create a similar environment. That allows
> for pouring a (fbsd) revision on a slice, and actually booting into it *and*
> a DE. This requires (in my mind) the necessity to dedicate a box for that
> task. This is (currently) all on GPT/UFS scheme/slices. I use the pre-created
> revision(s) I already use in my jails, which are all tarballs named after
> the version-revision.
> disk0:
> gpart destroy -F ada0
> gpart create -s GPT ada0
> gpart add -t freebsd-boot -l  ... ada0
> gpart add -t freebsd-ufs -l workdrive -s  ada0
> ...
> gpart add -t freebsd-ufs -l 11R-rXX -s  ada0
> gpart add -t freebsd-usf -l 12S-rXX -s  ada0
> gpart add -t freebsd-usf -l 12C-rXX -s  ada0
> ...
> disk1:
> one *giant* unbootable slice containing all the tarred up revisions I work
> with.
> Boot into workdrive; which also mounts disk1; newfs a slice && unpack
> the appropriate archive onto the newly formatted slice. Then *attempt* to
> boot into it after bouncing the box. The last part is the one I'm asking this
> question for. It seems to me that /boot on any one of the slices should have
> enough in it to be a legal candidate to boot into. It seems that it *should*
> be possible to get there with whats already available on the beginning of the
> drive. I'd use any one of the ZFS suggestions, except the spare I'm working
> with is only a SandyBridge. So probably not powerful enough to manage the
> overhead required.
> 
> I hope this clears things up, and isn't *too* verbose. Thanks for all the
> valuable input I received, and any additional enlightenment that might 
> follow! :)

Hi,

Why not use VirtualBox and different VMs with different versions?  That
would seem to be easier than messing with boot environments, assuming
the hardware has enough RAM to support VBOX.

Gary

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: what 3rd party boot mgr is required to boot multiple freebsd versions?

[Chris]

On Tue, 17 Mar 2020 16:17:31 +0200 Toomas Soome tso...@me.com said


> On 17. Mar 2020, at 15:51, Matthew Seaman  wrote:
> 
> On 17/03/2020 12:58, Florian Limberger wrote:

>> On 16.03.20 23:33, Chris wrote:
>> 
>>> For the record. I'm *only* using FreeBSD in this situation. I

>>> only mentioned Windows above, for the use of it's boot manager.
>> 
>> If you only use FreeBSD, and also use ZFS, you might find beadm[1]

>> interesting.
>> 
>> [1]: https://www.freshports.org/sysutils/beadm
>> 
> 
> Did you know that the system now comes with bectl(8) which is very

> similar to beadm?  As far as I can tell, the biggest difference is that
> if you have more than one ZFS in your boot environment then:
> 
>beadm create FOO
> 
> is actually equivalent to
> 
>bectl create -r FOO
> 
> ie. turning on the recursive functionality in bectl.
> 
> However, this is not really what the OP was asking about.  As I

> understand it, they were looking for something that would allow choosing
> between several independent installations of different versions of
> FreeBSD, rather than having multiple environments in the same
> installation.  You can achieve pretty much the same effect though --
> there is a boot menu option to switch between BEs.  You would have to
> manage any ported software between the different OS versions, perhaps by
> including /usr/local and /var/db/pkg are both parts of your BEs.
> 
> 	Cheers,
> 
> 	Matthew
> 


BE’s can solve some scenarios. However, it is easy to add support for few
more. The current BE menu is populated automatically from the zfs. However,
it is also simple task to add a file parser to it and read menu file with
entries with different pool (we only need to activate such entries same way
as it is currently done for “normal” BE, or entries with chain load).
Read this menu file first and zfs BE list after, and you have BE menu with
manual and automatic entries. Can be implemented within few hours. 


A *huge* thanks for all the thoughtful replies!
In detail. I maintain *many* ports, and it's not always enough to ensure
that they build properly. In some cases I need to ensure they actually
operate on FreeBSD/some-version. To test for building; it's been enough for
me to spin up any number of jails using whatever (fbsd) revision I'm testing
against. I had/am hoping that I can create a similar environment. That allows
for pouring a (fbsd) revision on a slice, and actually booting into it *and*
a DE. This requires (in my mind) the necessity to dedicate a box for that
task. This is (currently) all on GPT/UFS scheme/slices. I use the pre-created
revision(s) I already use in my jails, which are all tarballs named after
the version-revision.
disk0:
gpart destroy -F ada0
gpart create -s GPT ada0
gpart add -t freebsd-boot -l  ... ada0
gpart add -t freebsd-ufs -l workdrive -s  ada0
...
gpart add -t freebsd-ufs -l 11R-rXX -s  ada0
gpart add -t freebsd-usf -l 12S-rXX -s  ada0
gpart add -t freebsd-usf -l 12C-rXX -s  ada0
...
disk1:
one *giant* unbootable slice containing all the tarred up revisions I work
with.
Boot into workdrive; which also mounts disk1; newfs a slice && unpack
the appropriate archive onto the newly formatted slice. Then *attempt* to
boot into it after bouncing the box. The last part is the one I'm asking this
question for. It seems to me that /boot on any one of the slices should have
enough in it to be a legal candidate to boot into. It seems that it *should*
be possible to get there with whats already available on the beginning of the
drive. I'd use any one of the ZFS suggestions, except the spare I'm working
with is only a SandyBridge. So probably not powerful enough to manage the
overhead required.

I hope this clears things up, and isn't *too* verbose. Thanks for all the
valuable input I received, and any additional enlightenment that might follow! 
:)

--Chris


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: what 3rd party boot mgr is required to boot multiple freebsd versions?

[Toomas Soome]

> On 17. Mar 2020, at 15:51, Matthew Seaman  wrote:
> 
> On 17/03/2020 12:58, Florian Limberger wrote:
>> On 16.03.20 23:33, Chris wrote:
>> 
>>> For the record. I'm *only* using FreeBSD in this situation. I
>>> only mentioned Windows above, for the use of it's boot manager.
>> 
>> If you only use FreeBSD, and also use ZFS, you might find beadm[1]
>> interesting.
>> 
>> [1]: https://www.freshports.org/sysutils/beadm
>> 
> 
> Did you know that the system now comes with bectl(8) which is very
> similar to beadm?  As far as I can tell, the biggest difference is that
> if you have more than one ZFS in your boot environment then:
> 
>beadm create FOO
> 
> is actually equivalent to
> 
>bectl create -r FOO
> 
> ie. turning on the recursive functionality in bectl.
> 
> However, this is not really what the OP was asking about.  As I
> understand it, they were looking for something that would allow choosing
> between several independent installations of different versions of
> FreeBSD, rather than having multiple environments in the same
> installation.  You can achieve pretty much the same effect though --
> there is a boot menu option to switch between BEs.  You would have to
> manage any ported software between the different OS versions, perhaps by
> including /usr/local and /var/db/pkg are both parts of your BEs.
> 
>   Cheers,
> 
>   Matthew
> 

BE’s can solve some scenarios. However, it is easy to add support for few more. 
The current BE menu is populated automatically from the zfs. However, it is 
also simple task to add a file parser to it and read menu file with entries 
with different pool (we only need to activate such entries same way as it is 
currently done for “normal” BE, or entries with chain load). Read this menu 
file first and zfs BE list after, and you have BE menu with manual and 
automatic entries. Can be implemented within few hours. 

rgds,
Toomas



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: what 3rd party boot mgr is required to boot multiple freebsd versions?

[Chris]

On Mon, 16 Mar 2020 22:03:02 -0500 Karl Denninger k...@denninger.net said


On 3/16/2020 17:33, Chris wrote:
> On Mon, 16 Mar 2020 17:24:24 -0500 Karl Denninger k...@denninger.net said
>
>> On 3/16/2020 17:23, Chris wrote:
>> > I'm attempting to boot multiple versions of FreeBSD.
>> > I started with an install of older 11 with a (u)efi
>> > boot partition installed. I then grabbed an current 11
>> > usbstick, and installed that. Which stated it needed to
>> > install a (u)efi boot partition. I let it do it. But the
>> > new (additional) install doesn't show up at boot. Altho
>> > my UEFI BIOS sees it.
>> > I guess there are just too many uefi bios versions,
>> > and too many changes in the FreeBSD uefi boot code
>> > to expect consistent results over the long haul.
>> > Should I just convert the 1st efi (GPT) boot partition
>> > to a PMBR, and delete the second efi partition. Or is
>> > there a recommended bootmanager I can use to boot multiple
>> > versions of FreeBSD? Windows?
>> >
>> > Thank you!
>> >
>> > --Chris
>> >
>> Refind perhaps?
> Thanks for the reply, Karl! :)
> I've used that before for FreeBSD/MacOS combos. I'll look at it again.
>
> For the record. I'm *only* using FreeBSD in this situation. I
> only mentioned Windows above, for the use of it's boot manager.
>
> Thanks again!
>
Refind will find all the bootable EFI "elements" in the EFI partition
and menu them.  The question then becomes whether multiple efi loaders
can be told to each use a *different* partition to load the kernel from
(and thus the loader file, which in turn can tell it where the root
filesystem is.)

Reading through the man page it appears they may not be.  You could of
course interrupt it and manually select that, but I suspect that's not
what you want to have to do.

I use refind on a dual-boot (win10/FreeBSD) system but not with multiple
independent FreeBSD versions.

Thanks for all the detail, Karl. I was also looking at those details. It
occurs to me that an *ideal* arrangement might be the ability to name each
efi loader according to it's target, or probably better; (name) the directories
that hold the loaders. It just seems wasteful/inefficient to create x number
of efi partitions for x number of OSs. Maybe boot0 could be coerced info
something similar? I'm currently looking at Clover (popular with the
"hackintosh" users) that seems to do something similar.

Thanks again!

--Chris


--
Karl Denninger
k...@denninger.net <mailto:k...@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: what 3rd party boot mgr is required to boot multiple freebsd versions?

[Matthew Seaman]

On 17/03/2020 12:58, Florian Limberger wrote:
> On 16.03.20 23:33, Chris wrote:
> 
>> For the record. I'm *only* using FreeBSD in this situation. I
>> only mentioned Windows above, for the use of it's boot manager.
> 
> If you only use FreeBSD, and also use ZFS, you might find beadm[1]
> interesting.
> 
> [1]: https://www.freshports.org/sysutils/beadm
> 

Did you know that the system now comes with bectl(8) which is very
similar to beadm?  As far as I can tell, the biggest difference is that
if you have more than one ZFS in your boot environment then:

beadm create FOO

is actually equivalent to

bectl create -r FOO

ie. turning on the recursive functionality in bectl.

However, this is not really what the OP was asking about.  As I
understand it, they were looking for something that would allow choosing
between several independent installations of different versions of
FreeBSD, rather than having multiple environments in the same
installation.  You can achieve pretty much the same effect though --
there is a boot menu option to switch between BEs.  You would have to
manage any ported software between the different OS versions, perhaps by
including /usr/local and /var/db/pkg are both parts of your BEs.

Cheers,

Matthew



signature.asc
Description: OpenPGP digital signature

Re: what 3rd party boot mgr is required to boot multiple freebsd versions?

[Tomoaki AOKI]

On Tue, 17 Mar 2020 01:31:59 +0300
Andrey Fesenko  wrote:

> On Tue, Mar 17, 2020 at 1:24 AM Chris  wrote:
> >
> > I'm attempting to boot multiple versions of FreeBSD.
> > I started with an install of older 11 with a (u)efi
> > boot partition installed. I then grabbed an current 11
> > usbstick, and installed that. Which stated it needed to
> > install a (u)efi boot partition. I let it do it. But the
> > new (additional) install doesn't show up at boot. Altho
> > my UEFI BIOS sees it.
> > I guess there are just too many uefi bios versions,
> > and too many changes in the FreeBSD uefi boot code
> > to expect consistent results over the long haul.
> > Should I just convert the 1st efi (GPT) boot partition
> > to a PMBR, and delete the second efi partition. Or is
> > there a recommended bootmanager I can use to boot multiple
> > versions of FreeBSD? Windows?
> >
> 
> upgrade system and use
> https://www.freebsd.org/cgi/man.cgi?query=efibootmgr=8=freebsd-release-ports
> ;)
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Or if you want boot-time selection, try the patch proposed on Bug 207940
[1]. efibootmgr can be used only for "next boot and later", IIUC.

The latest patch is for head and stable/12. Applicable on top
of /usr/src.

Patch for stable/11 is NOT tested / maintained for more than one year,
so possibly doesn't apply / work anymore.

 [1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207940

-- 
Tomoaki AOKI
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: what 3rd party boot mgr is required to boot multiple freebsd versions?

[Florian Limberger]

On 16.03.20 23:33, Chris wrote:

> For the record. I'm *only* using FreeBSD in this situation. I
> only mentioned Windows above, for the use of it's boot manager.

If you only use FreeBSD, and also use ZFS, you might find beadm[1]
interesting.

[1]: https://www.freshports.org/sysutils/beadm

Regards

-flo
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: what 3rd party boot mgr is required to boot multiple freebsd versions?

[Karl Denninger]

On 3/16/2020 17:33, Chris wrote:
> On Mon, 16 Mar 2020 17:24:24 -0500 Karl Denninger k...@denninger.net said
>
>> On 3/16/2020 17:23, Chris wrote:
>> > I'm attempting to boot multiple versions of FreeBSD.
>> > I started with an install of older 11 with a (u)efi
>> > boot partition installed. I then grabbed an current 11
>> > usbstick, and installed that. Which stated it needed to
>> > install a (u)efi boot partition. I let it do it. But the
>> > new (additional) install doesn't show up at boot. Altho
>> > my UEFI BIOS sees it.
>> > I guess there are just too many uefi bios versions,
>> > and too many changes in the FreeBSD uefi boot code
>> > to expect consistent results over the long haul.
>> > Should I just convert the 1st efi (GPT) boot partition
>> > to a PMBR, and delete the second efi partition. Or is
>> > there a recommended bootmanager I can use to boot multiple
>> > versions of FreeBSD? Windows?
>> >
>> > Thank you!
>> >
>> > --Chris
>> >
>> Refind perhaps?
> Thanks for the reply, Karl! :)
> I've used that before for FreeBSD/MacOS combos. I'll look at it again.
>
> For the record. I'm *only* using FreeBSD in this situation. I
> only mentioned Windows above, for the use of it's boot manager.
>
> Thanks again!
>
Refind will find all the bootable EFI "elements" in the EFI partition
and menu them.  The question then becomes whether multiple efi loaders
can be told to each use a *different* partition to load the kernel from
(and thus the loader file, which in turn can tell it where the root
filesystem is.)

Reading through the man page it appears they may not be.  You could of
course interrupt it and manually select that, but I suspect that's not
what you want to have to do.

I use refind on a dual-boot (win10/FreeBSD) system but not with multiple
independent FreeBSD versions.

-- 
Karl Denninger
k...@denninger.net <mailto:k...@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/


smime.p7s
Description: S/MIME Cryptographic Signature

Re: what 3rd party boot mgr is required to boot multiple freebsd versions?

[Chris]

On Mon, 16 Mar 2020 17:24:24 -0500 Karl Denninger k...@denninger.net said


On 3/16/2020 17:23, Chris wrote:
> I'm attempting to boot multiple versions of FreeBSD.
> I started with an install of older 11 with a (u)efi
> boot partition installed. I then grabbed an current 11
> usbstick, and installed that. Which stated it needed to
> install a (u)efi boot partition. I let it do it. But the
> new (additional) install doesn't show up at boot. Altho
> my UEFI BIOS sees it.
> I guess there are just too many uefi bios versions,
> and too many changes in the FreeBSD uefi boot code
> to expect consistent results over the long haul.
> Should I just convert the 1st efi (GPT) boot partition
> to a PMBR, and delete the second efi partition. Or is
> there a recommended bootmanager I can use to boot multiple
> versions of FreeBSD? Windows?
>
> Thank you!
>
> --Chris
>
Refind perhaps?

Thanks for the reply, Karl! :)
I've used that before for FreeBSD/MacOS combos. I'll look at it again.

For the record. I'm *only* using FreeBSD in this situation. I
only mentioned Windows above, for the use of it's boot manager.

Thanks again!

--Chris



--
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: what 3rd party boot mgr is required to boot multiple freebsd versions?

[Andrey Fesenko]

On Tue, Mar 17, 2020 at 1:24 AM Chris  wrote:
>
> I'm attempting to boot multiple versions of FreeBSD.
> I started with an install of older 11 with a (u)efi
> boot partition installed. I then grabbed an current 11
> usbstick, and installed that. Which stated it needed to
> install a (u)efi boot partition. I let it do it. But the
> new (additional) install doesn't show up at boot. Altho
> my UEFI BIOS sees it.
> I guess there are just too many uefi bios versions,
> and too many changes in the FreeBSD uefi boot code
> to expect consistent results over the long haul.
> Should I just convert the 1st efi (GPT) boot partition
> to a PMBR, and delete the second efi partition. Or is
> there a recommended bootmanager I can use to boot multiple
> versions of FreeBSD? Windows?
>

upgrade system and use
https://www.freebsd.org/cgi/man.cgi?query=efibootmgr=8=freebsd-release-ports
;)
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: what 3rd party boot mgr is required to boot multiple freebsd versions?

[Karl Denninger]

On 3/16/2020 17:23, Chris wrote:
> I'm attempting to boot multiple versions of FreeBSD.
> I started with an install of older 11 with a (u)efi
> boot partition installed. I then grabbed an current 11
> usbstick, and installed that. Which stated it needed to
> install a (u)efi boot partition. I let it do it. But the
> new (additional) install doesn't show up at boot. Altho
> my UEFI BIOS sees it.
> I guess there are just too many uefi bios versions,
> and too many changes in the FreeBSD uefi boot code
> to expect consistent results over the long haul.
> Should I just convert the 1st efi (GPT) boot partition
> to a PMBR, and delete the second efi partition. Or is
> there a recommended bootmanager I can use to boot multiple
> versions of FreeBSD? Windows?
>
> Thank you!
>
> --Chris
>
Refind perhaps?


-- 
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/


smime.p7s
Description: S/MIME Cryptographic Signature