Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Mark Millard]

On Apr 13, 2023, at 21:44, Charlie Li  wrote:

> Mark Millard wrote:
>> FYI: in my original report for a context that has never had
>> block_cloning enabled, I reported BOTH missing files and
>> file content corruption in the poudriere-devel bulk build
>> testing. This predates:
>> https://people.freebsd.org/~pjd/patches/brt_revert.patch
>> but had the changes from:
>> https://github.com/openzfs/zfs/pull/14739/files
>> The files were missing from packages installed to be used
>> during a port's build. No other types of examples of missing
>> files happened. (But only 11 ports failed.)
> I also don't have block_cloning enabled. "Missing files" prior to brt_revert 
> may actually be present, but as the corruption also messes with the file(1) 
> signature, some tools like ldconfig report them as missing.

For reference, the specific messages that were not explicit
null-byte complaints were (some shown with a little context):


===>   py39-lxml-4.9.2 depends on shared library: libxml2.so - not found
===>   Installing existing package /packages/All/libxml2-2.10.3_1.pkg
[CA72_ZFS] Installing libxml2-2.10.3_1...
[CA72_ZFS] Extracting libxml2-2.10.3_1: .. done
===>   py39-lxml-4.9.2 depends on shared library: libxml2.so - found 
(/usr/local/lib/libxml2.so)
. . .
[CA72_ZFS] Extracting libxslt-1.1.37: .. done
===>   py39-lxml-4.9.2 depends on shared library: libxslt.so - found 
(/usr/local/lib/libxslt.so)
===>   Returning to build of py39-lxml-4.9.2
. . .
===>  Configuring for py39-lxml-4.9.2
Building lxml version 4.9.2.
Building with Cython 0.29.33.
Error: Please make sure the libxml2 and libxslt development packages are 
installed.


[CA72_ZFS] Extracting libunistring-1.1: .. done
===>   libidn2-2.3.4 depends on shared library: libunistring.so - not found


[CA72_ZFS] Extracting gmp-6.2.1: .. done
===>   mpfr-4.2.0,1 depends on shared library: libgmp.so - not found


===>   nettle-3.8.1 depends on shared library: libgmp.so - not found
===>   Installing existing package /packages/All/gmp-6.2.1.pkg
[CA72_ZFS] Installing gmp-6.2.1...
the most recent version of gmp-6.2.1 is already installed
===>   nettle-3.8.1 depends on shared library: libgmp.so - not found
*** Error code 1


autom4te: error: need GNU m4 1.4 or later: /usr/local/bin/gm4


checking for GNU 
M4 that supports accurate traces... configure: error: no acceptable m4 could be 
found in $PATH.
GNU M4 1.4.6 or later is required; 1.4.16 or newer is recommended.
GNU M4 1.4.15 uses a buggy replacement strstr on some systems.
Glibc 2.9 - 2.12 and GNU M4 1.4.11 - 1.4.15 have another strstr bug.


ld: error: /usr/local/lib/libblkid.a: unknown file type


===
Mark Millard
marklmi at yahoo.com

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Charlie Li]

Mark Millard wrote:

FYI: in my original report for a context that has never had
block_cloning enabled, I reported BOTH missing files and
file content corruption in the poudriere-devel bulk build
testing. This predates:

https://people.freebsd.org/~pjd/patches/brt_revert.patch

but had the changes from:

https://github.com/openzfs/zfs/pull/14739/files

The files were missing from packages installed to be used
during a port's build. No other types of examples of missing
files happened. (But only 11 ports failed.)

I also don't have block_cloning enabled. "Missing files" prior to 
brt_revert may actually be present, but as the corruption also messes 
with the file(1) signature, some tools like ldconfig report them as missing.


--
Charlie Li
…nope, still don't have an exit line.



OpenPGP_signature
Description: OpenPGP digital signature

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Mark Millard]

On Apr 13, 2023, at 21:27, Charlie Li  wrote:
> 
> Pawel Jakub Dawidek wrote:
>> On 4/14/23 09:23, Charlie Li wrote:
>>> Pawel Jakub Dawidek wrote:
 Here is the change that reverts most of the modifications and disables 
 cloning new blocks. It does retain ability to free existing cloned blocks 
 and keeps block_cloning feature around, so upgraded pools can be imported 
 and existing cloned blocks freed.
 
 It does not handle replaying ZIL with block-cloning logs, so make sure you 
 import pools that were cleanly exported.
 
 I'd appreciate if someone who can reproduce those corruptions could try it.
 
 https://github.com/pjd/openzfs/commit/f2cfbcf76a733c44e25cba8c649162ef68047103
 
>>> Does not apply to sys/contrib/openzfs tip, conflicts in 
>>> module/os/freebsd/zfs/zfs_vnops_os.c and module/zfs/dmu.c.
>> This should work:
>> https://people.freebsd.org/~pjd/patches/brt_revert.patch
> This results in missing files rather than corruption.

FYI: in my original report for a context that has never had
block_cloning enabled, I reported BOTH missing files and
file content corruption in the poudriere-devel bulk build
testing. This predates:

https://people.freebsd.org/~pjd/patches/brt_revert.patch

but had the changes from:

https://github.com/openzfs/zfs/pull/14739/files

The files were missing from packages installed to be used
during a port's build. No other types of examples of missing
files happened. (But only 11 ports failed.)

===
Mark Millard
marklmi at yahoo.com

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Charlie Li]

Pawel Jakub Dawidek wrote:

On 4/14/23 09:23, Charlie Li wrote:

Pawel Jakub Dawidek wrote:
Here is the change that reverts most of the modifications and 
disables cloning new blocks. It does retain ability to free existing 
cloned blocks and keeps block_cloning feature around, so upgraded 
pools can be imported and existing cloned blocks freed.


It does not handle replaying ZIL with block-cloning logs, so make 
sure you import pools that were cleanly exported.


I'd appreciate if someone who can reproduce those corruptions could 
try it.


https://github.com/pjd/openzfs/commit/f2cfbcf76a733c44e25cba8c649162ef68047103

Does not apply to sys/contrib/openzfs tip, conflicts in 
module/os/freebsd/zfs/zfs_vnops_os.c and module/zfs/dmu.c.


This should work:

https://people.freebsd.org/~pjd/patches/brt_revert.patch


This results in missing files rather than corruption.

--
Charlie Li
…nope, still don't have an exit line.



OpenPGP_signature
Description: OpenPGP digital signature

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Pawel Jakub Dawidek]

On 4/14/23 09:23, Charlie Li wrote:

Pawel Jakub Dawidek wrote:
Here is the change that reverts most of the modifications and disables 
cloning new blocks. It does retain ability to free existing cloned 
blocks and keeps block_cloning feature around, so upgraded pools can 
be imported and existing cloned blocks freed.


It does not handle replaying ZIL with block-cloning logs, so make sure 
you import pools that were cleanly exported.


I'd appreciate if someone who can reproduce those corruptions could 
try it.


https://github.com/pjd/openzfs/commit/f2cfbcf76a733c44e25cba8c649162ef68047103

Does not apply to sys/contrib/openzfs tip, conflicts in 
module/os/freebsd/zfs/zfs_vnops_os.c and module/zfs/dmu.c.


This should work:

https://people.freebsd.org/~pjd/patches/brt_revert.patch

--
Pawel Jakub Dawidek

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Mateusz Guzik]

On 4/14/23, Charlie Li  wrote:
> Pawel Jakub Dawidek wrote:
>> On 4/14/23 07:52, Charlie Li wrote:
>>> Pawel Jakub Dawidek wrote:
 thank you for your testing and patience so far. I'm working on a
 patch to revert block cloning without affecting people who already
 upgraded their pools.

>>> Testing with mjg@ earlier today revealed that block_cloning was not
>>> the cause of poudriere bulk build (and similar cp(1)/install(1)-based)
>>> corruption, although may have exacerbated it.
>>
>> Can you please elaborate how were you testing and what exactly did you
>> exclude?
>>
> mjg@ prepared
> https://gitlab.com/vishwin/freebsd-src/-/commit/b41f187ba329621cda1e8e67a0786f07b1221a3c
>
> which only removes block_cloning, rebuilding kernel only (buildworld
> fails) for me to test poudriere bulk -c builds with. I used a world from
> https://gitlab.com/vishwin/freebsd-src/-/tree/zfs-revert which consists
> of reverting the merge commit plus a few other conflicts, but keeping
> vop_fplookup_vexec.
>

I'm going to narrow down the non-blockcopy corruption after my testjig
gets off the ground.

Basically I expect to have it sorted out on Friday.


-- 
Mateusz Guzik 

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Charlie Li]

Pawel Jakub Dawidek wrote:
Here is the change that reverts most of the modifications and disables 
cloning new blocks. It does retain ability to free existing cloned 
blocks and keeps block_cloning feature around, so upgraded pools can be 
imported and existing cloned blocks freed.


It does not handle replaying ZIL with block-cloning logs, so make sure 
you import pools that were cleanly exported.


I'd appreciate if someone who can reproduce those corruptions could try it.

https://github.com/pjd/openzfs/commit/f2cfbcf76a733c44e25cba8c649162ef68047103

Does not apply to sys/contrib/openzfs tip, conflicts in 
module/os/freebsd/zfs/zfs_vnops_os.c and module/zfs/dmu.c.


--
Charlie Li
…nope, still don't have an exit line.



OpenPGP_signature
Description: OpenPGP digital signature

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Charlie Li]

Pawel Jakub Dawidek wrote:

On 4/14/23 07:52, Charlie Li wrote:

Pawel Jakub Dawidek wrote:
thank you for your testing and patience so far. I'm working on a 
patch to revert block cloning without affecting people who already 
upgraded their pools.


Testing with mjg@ earlier today revealed that block_cloning was not 
the cause of poudriere bulk build (and similar cp(1)/install(1)-based) 
corruption, although may have exacerbated it.


Can you please elaborate how were you testing and what exactly did you 
exclude?


mjg@ prepared 
https://gitlab.com/vishwin/freebsd-src/-/commit/b41f187ba329621cda1e8e67a0786f07b1221a3c 
which only removes block_cloning, rebuilding kernel only (buildworld 
fails) for me to test poudriere bulk -c builds with. I used a world from 
https://gitlab.com/vishwin/freebsd-src/-/tree/zfs-revert which consists 
of reverting the merge commit plus a few other conflicts, but keeping 
vop_fplookup_vexec.


--
Charlie Li
…nope, still don't have an exit line.



OpenPGP_signature
Description: OpenPGP digital signature

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Pawel Jakub Dawidek]

On 4/14/23 07:52, Charlie Li wrote:

Pawel Jakub Dawidek wrote:
thank you for your testing and patience so far. I'm working on a patch 
to revert block cloning without affecting people who already upgraded 
their pools.


Testing with mjg@ earlier today revealed that block_cloning was not the 
cause of poudriere bulk build (and similar cp(1)/install(1)-based) 
corruption, although may have exacerbated it.


Can you please elaborate how were you testing and what exactly did you 
exclude?


Thanks.

--
Pawel Jakub Dawidek

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Pawel Jakub Dawidek]

On 4/14/23 07:40, Pawel Jakub Dawidek wrote:

On 4/13/23 22:56, Cy Schubert wrote:
I'm in the process of building a branch reverting the merge altogether 
and

will test it on my sandbox machine later today.


Cy,

thank you for your testing and patience so far. I'm working on a patch 
to revert block cloning without affecting people who already upgraded 
their pools.


I'd also greatly appreciate if you could provide a procedure for me to 
reproduce the corruption, ideally without the internet access, as I'll 
be on the plane(s) for the next ~24h.


Here is the change that reverts most of the modifications and disables 
cloning new blocks. It does retain ability to free existing cloned 
blocks and keeps block_cloning feature around, so upgraded pools can be 
imported and existing cloned blocks freed.


It does not handle replaying ZIL with block-cloning logs, so make sure 
you import pools that were cleanly exported.


I'd appreciate if someone who can reproduce those corruptions could try it.

https://github.com/pjd/openzfs/commit/f2cfbcf76a733c44e25cba8c649162ef68047103

Thank you guys for your help!

--
Pawel Jakub Dawidek

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Charlie Li]

Pawel Jakub Dawidek wrote:
thank you for your testing and patience so far. I'm working on a patch 
to revert block cloning without affecting people who already upgraded 
their pools.


Testing with mjg@ earlier today revealed that block_cloning was not the 
cause of poudriere bulk build (and similar cp(1)/install(1)-based) 
corruption, although may have exacerbated it.
I'd also greatly appreciate if you could provide a procedure for me to 
reproduce the corruption, ideally without the internet access, as I'll 
be on the plane(s) for the next ~24h.


Due to non-deterministic conditions, there...kind of isn't one. Best is 
probably just poudriere bulk builds.


--
Charlie Li
…nope, still don't have an exit line.



OpenPGP_signature
Description: OpenPGP digital signature

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Charlie Li]

Shawn Webb wrote:

Does the ZFS project have some sort of automated testing to catch
data-gobbling, pool killing bugs? It seems like this would have been
caught with some CI/CD stress testing automation scripts.

I can't speak about how the OpenZFS project does things, but this 
particular corruption does not have any deterministic characteristics 
both pre- and post-condition, so would be hard to automate testing.


--
Charlie Li
…nope, still don't have an exit line.



OpenPGP_signature
Description: OpenPGP digital signature

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Pawel Jakub Dawidek]

On 4/13/23 22:56, Cy Schubert wrote:

I'm in the process of building a branch reverting the merge altogether and
will test it on my sandbox machine later today.


Cy,

thank you for your testing and patience so far. I'm working on a patch 
to revert block cloning without affecting people who already upgraded 
their pools.


I'd also greatly appreciate if you could provide a procedure for me to 
reproduce the corruption, ideally without the internet access, as I'll 
be on the plane(s) for the next ~24h.


--
Pawel Jakub Dawidek

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Pawel Jakub Dawidek]

On 4/13/23 23:05, Shawn Webb wrote:

I've learned over the years downstream that it's not really my place
to tell upstream what to do or how to do it. However, I think given
the seriousness of this, upstream might do well to revert the commit
until a solid fix is in place. Upstream might want to consider the
impacts this is having not just with downstream projects, but also
regular users.

Really bad timing to have a lot of new tax documentation that I really
don't want to lose. I'd really like to have an up-to-date, security
patched OS, but I guess I'll stay behind so that I don't risk losing
critical financial documentation.


Shawn,

I'm working on a patch to safely revert this that would also work for 
people who already upgraded their pools.


I'm sorry for this mess.

--
Pawel Jakub Dawidek

Re: aarch64: lang/gcc1* build regression between Mar-28 and Apr-8

[Dimitry Andric]

On 13 Apr 2023, at 23:42, Dima Panov  wrote:
> Somethings changed in main branch between Mar, 28 and Apr, 8 which causes 
> permanent build error for lang/gcc1[012] on aarch64 (exactly VMWare container 
> on M1Pro mac)
> 
> during RTL pass: sched1
> gimple-match.cc: In function 'bool gimple_nop_atomic_bit_test_and_p(tree, 
> tree_node**, tree_node* (*)(tree))':
> gimple-match.cc:39215:1: internal compiler error: Segmentation fault
> 39215 | }
>  | ^
> mmap: Invalid argument
> Please submit a full bug report, with preprocessed source (by using 
> -freport-bug).
> See  for instructions.
> gmake[4]: *** [Makefile:1145: gimple-match.o] Error 1
> gmake[4]: *** Waiting for unfinished jobs
> rm gcc.pod gfortran.pod

Are you running this build on zfs? There are apparently a bunch of
filesystem corruption problems which have surfaced due to recent openzfs
imports.

-Dimitry



signature.asc
Description: Message signed with OpenPGP

aarch64: lang/gcc1* build regression between Mar-28 and Apr-8

[Dima Panov]

Moin-moin!

Somethings changed in main branch between Mar, 28 and Apr, 8 which causes 
permanent build error for lang/gcc1[012] on aarch64 (exactly VMWare container 
on M1Pro mac)

during RTL pass: sched1
gimple-match.cc: In function 'bool gimple_nop_atomic_bit_test_and_p(tree, 
tree_node**, tree_node* (*)(tree))':
gimple-match.cc:39215:1: internal compiler error: Segmentation fault
39215 | }
  | ^
mmap: Invalid argument
Please submit a full bug report, with preprocessed source (by using 
-freport-bug).
See  for instructions.
gmake[4]: *** [Makefile:1145: gimple-match.o] Error 1
gmake[4]: *** Waiting for unfinished jobs
rm gcc.pod gfortran.pod


--
Sincerely,
Dima (flu...@freebsd.org, https://t.me/FluffyBSD)
(desktop, kde, x11, office, ports-secteam)@FreeBSD team


OpenPGP_signature
Description: OpenPGP digital signature

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Shawn Webb]

On Thu, Apr 13, 2023 at 06:56:35AM -0700, Cy Schubert wrote:
> In message  om>
> , Mateusz Guzik writes:
> > On 4/13/23, Cy Schubert  wrote:
> > > On Thu, 13 Apr 2023 19:54:42 +0900
> > > Pawe=C5=82 Jakub Dawidek  wrote:
> > >
> > >> On Apr 13, 2023, at 16:10, Cy Schubert  wrote=
> > :
> > >> >
> > >> > =EF=BB=BFIn message <20230413070426.8a54f...@slippy.cwsent.com>, Cy Sc=
> > hubert
> > >> > writes:
> > >> > In message <20230413064252.1e5c1...@slippy.cwsent.com>, Cy Schubert
> > >> > writes:
> > >> >> In message , Mark
> > >> >> Millard
> > >> >>> write
> > >> >>> s:
> > >> >>> [This just puts my prior reply's material into Cy's
> > >>  adjusted resend of the original. The To/Cc should
> > >>  be coomplete this time.]
> > >> 
> > >>  On Apr 12, 2023, at 22:52, Cy Schubert  =
> > =3D
> > >>  wrote:
> > >> 
> > >>  In message , Mark =
> > =3D
> > >> > Millard=3D20
> > >>  write
> > >> > s:
> > >> > From: Charlie Li  wrote on
> > >> >> Date: Wed, 12 Apr 2023 20:11:16 UTC :
> > >> >> =3D20
> > >> >> Charlie Li wrote:
> > >> >>> Mateusz Guzik wrote:
> > >>  can you please test poudriere with
> > >> > https://github.com/openzfs/zfs/pull/14739/files
> > >> > =3D20
> > >> > After applying, on the md(4)-backed pool regardless of =3D3D
> > >>  block_cloning,=3D3D20
> > >> >> the cy@ `cp -R` test reports no differing (ie corrupted) files. =
> > =3D
> > >>  Will=3D3D20=3D3D
> > >>  =3D20
> > >> >> report back on poudriere results (no block_cloning).
> > >>  =3D3D20
> > >>  As for poudriere, build failures are still rolling in. These ar=
> > e
> > >>  =3D
> > >> >>> (and=3D3D20=3D3D
> > >>  =3D20
> > >> >> have been) entirely random on every run. Some examples from this =
> > =3D
> > >> >>> run:
> > >>  =3D3D20
> > >> >>> lang/php81:
> > >> >>> - post-install: @${INSTALL_DATA}
> > >> >>> ${WRKSRC}/php.ini-development=3D3D20
> > >> >>> ${WRKSRC}/php.ini-production ${WRKDIR}/php.conf =3D3D
> > >> >>> ${STAGEDIR}/${PREFIX}/etc
> > >> >> - consumers fail to build due to corrupted php.conf packaged
> > >> >>> =3D3D20
> > >> >>> devel/ninja:
> > >> >>> - phase: stage
> > >> >>> - install -s -m 555=3D3D20
> > >> >>> /wrkdirs/usr/ports/devel/ninja/work/ninja-1.11.1/ninja=3D3D20
> > >> >>> /wrkdirs/usr/ports/devel/ninja/work/stage/usr/local/bin
> > >> >>> - consumers fail to build due to corrupted bin/ninja packaged
> > >> >>> =3D3D20
> > >> >>> devel/netsurf-buildsystem:
> > >> >>> - phase: stage
> > >> >>> - mkdir -p=3D3D20
> > >> >>> =3D3D
> > >> >>> =3D
> > >> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local=
> > /share/n
> > >>  e=3D
> > >> >> =3D3D
> > >>  tsurf-buildsystem/makefiles=3D3D20
> > >> >> =3D3D
> > >> >>> =3D
> > >> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local=
> > /share/n
> > >>  e=3D
> > >> >> =3D3D
> > >>  tsurf-buildsystem/testtools
> > >> >> for M in Makefile.top Makefile.tools Makefile.subdir =3D3D
> > >> >>> Makefile.pkgconfig=3D3D20
> > >> >> Makefile.clang Makefile.gcc Makefile.norcroft Makefile.open64; do
> > >> >> \
> > >> >>> cp makefiles/$M=3D3D20
> > >> >>> =3D3D
> > >> >>> =3D
> > >> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local=
> > /share/n
> > >>  e=3D
> > >> >> =3D3D
> > >>  tsurf-buildsystem/makefiles/;=3D3D20
> > >> >> \
> > >> >>> done
> > >> >>> - graphics/libnsgif fails to build due to NUL characters in=3D3D=
> > 20
> > >> >>> Makefile.{clang,subdir}, causing nothing to link
> > >> >>> =3D20
> > >> >> Summary: I have problems building ports into packages
> > >> >> via poudriere-devel use despite being fully updated/patched
> > >> >> (as of when I started the experiment), never having enabled
> > >> >> block_cloning ( still using openzfs-2.1-freebsd ).
> > >> >> =3D20
> > >> >> In other words, I can confirm other reports that have
> > >> >> been made.
> > >> >> =3D20
> > >> >> The details follow.
> > >> >> =3D20
> > >> >> =3D20
> > >> >> [Written as I was working on setting up for the experiments
> > >> >> and then executing those experiments, adjusting as I went
> > >> >> along.]
> > >> >> =3D20
> > >> >> I've run my own tests in a context that has never had the
> > >> >> zpool upgrade and that jump from before the openzfs import to
> > >> >> after the existing commits for trying to fix openzfs on
> > >> >> FreeBSD. I report on the sequence of activities getting to
> > >> >> the point of testing as well.
> > >> >> =3D20
> > >> >> By personal policy I keep my (non-temporary) pool's compatible
> > >> >> with what the most recent ??.?-RELEASE supports, using
> > >> >> openzfs-2.1-freebsd for now. The 

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Cy Schubert]

In message 
, Mateusz Guzik writes:
> On 4/13/23, Cy Schubert  wrote:
> > On Thu, 13 Apr 2023 19:54:42 +0900
> > Pawe=C5=82 Jakub Dawidek  wrote:
> >
> >> On Apr 13, 2023, at 16:10, Cy Schubert  wrote=
> :
> >> >
> >> > =EF=BB=BFIn message <20230413070426.8a54f...@slippy.cwsent.com>, Cy Sc=
> hubert
> >> > writes:
> >> > In message <20230413064252.1e5c1...@slippy.cwsent.com>, Cy Schubert
> >> > writes:
> >> >> In message , Mark
> >> >> Millard
> >> >>> write
> >> >>> s:
> >> >>> [This just puts my prior reply's material into Cy's
> >>  adjusted resend of the original. The To/Cc should
> >>  be coomplete this time.]
> >> 
> >>  On Apr 12, 2023, at 22:52, Cy Schubert  =
> =3D
> >>  wrote:
> >> 
> >>  In message , Mark =
> =3D
> >> > Millard=3D20
> >>  write
> >> > s:
> >> > From: Charlie Li  wrote on
> >> >> Date: Wed, 12 Apr 2023 20:11:16 UTC :
> >> >> =3D20
> >> >> Charlie Li wrote:
> >> >>> Mateusz Guzik wrote:
> >>  can you please test poudriere with
> >> > https://github.com/openzfs/zfs/pull/14739/files
> >> > =3D20
> >> > After applying, on the md(4)-backed pool regardless of =3D3D
> >>  block_cloning,=3D3D20
> >> >> the cy@ `cp -R` test reports no differing (ie corrupted) files. =
> =3D
> >>  Will=3D3D20=3D3D
> >>  =3D20
> >> >> report back on poudriere results (no block_cloning).
> >>  =3D3D20
> >>  As for poudriere, build failures are still rolling in. These ar=
> e
> >>  =3D
> >> >>> (and=3D3D20=3D3D
> >>  =3D20
> >> >> have been) entirely random on every run. Some examples from this =
> =3D
> >> >>> run:
> >>  =3D3D20
> >> >>> lang/php81:
> >> >>> - post-install: @${INSTALL_DATA}
> >> >>> ${WRKSRC}/php.ini-development=3D3D20
> >> >>> ${WRKSRC}/php.ini-production ${WRKDIR}/php.conf =3D3D
> >> >>> ${STAGEDIR}/${PREFIX}/etc
> >> >> - consumers fail to build due to corrupted php.conf packaged
> >> >>> =3D3D20
> >> >>> devel/ninja:
> >> >>> - phase: stage
> >> >>> - install -s -m 555=3D3D20
> >> >>> /wrkdirs/usr/ports/devel/ninja/work/ninja-1.11.1/ninja=3D3D20
> >> >>> /wrkdirs/usr/ports/devel/ninja/work/stage/usr/local/bin
> >> >>> - consumers fail to build due to corrupted bin/ninja packaged
> >> >>> =3D3D20
> >> >>> devel/netsurf-buildsystem:
> >> >>> - phase: stage
> >> >>> - mkdir -p=3D3D20
> >> >>> =3D3D
> >> >>> =3D
> >> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local=
> /share/n
> >>  e=3D
> >> >> =3D3D
> >>  tsurf-buildsystem/makefiles=3D3D20
> >> >> =3D3D
> >> >>> =3D
> >> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local=
> /share/n
> >>  e=3D
> >> >> =3D3D
> >>  tsurf-buildsystem/testtools
> >> >> for M in Makefile.top Makefile.tools Makefile.subdir =3D3D
> >> >>> Makefile.pkgconfig=3D3D20
> >> >> Makefile.clang Makefile.gcc Makefile.norcroft Makefile.open64; do
> >> >> \
> >> >>> cp makefiles/$M=3D3D20
> >> >>> =3D3D
> >> >>> =3D
> >> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local=
> /share/n
> >>  e=3D
> >> >> =3D3D
> >>  tsurf-buildsystem/makefiles/;=3D3D20
> >> >> \
> >> >>> done
> >> >>> - graphics/libnsgif fails to build due to NUL characters in=3D3D=
> 20
> >> >>> Makefile.{clang,subdir}, causing nothing to link
> >> >>> =3D20
> >> >> Summary: I have problems building ports into packages
> >> >> via poudriere-devel use despite being fully updated/patched
> >> >> (as of when I started the experiment), never having enabled
> >> >> block_cloning ( still using openzfs-2.1-freebsd ).
> >> >> =3D20
> >> >> In other words, I can confirm other reports that have
> >> >> been made.
> >> >> =3D20
> >> >> The details follow.
> >> >> =3D20
> >> >> =3D20
> >> >> [Written as I was working on setting up for the experiments
> >> >> and then executing those experiments, adjusting as I went
> >> >> along.]
> >> >> =3D20
> >> >> I've run my own tests in a context that has never had the
> >> >> zpool upgrade and that jump from before the openzfs import to
> >> >> after the existing commits for trying to fix openzfs on
> >> >> FreeBSD. I report on the sequence of activities getting to
> >> >> the point of testing as well.
> >> >> =3D20
> >> >> By personal policy I keep my (non-temporary) pool's compatible
> >> >> with what the most recent ??.?-RELEASE supports, using
> >> >> openzfs-2.1-freebsd for now. The pools involved below have
> >> >> never had a zpool upgrade from where they started. (I've no
> >> >> pools that have ever had a zpool upgrade.)
> >> >> =3D20
> >> >> (Temporary pools are rare for me, such as this investigation.
> >> >> But I'm not testing block_cloning or anything new this time.)
> 

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Mateusz Guzik]

On 4/13/23, Cy Schubert  wrote:
> On Thu, 13 Apr 2023 19:54:42 +0900
> Paweł Jakub Dawidek  wrote:
>
>> On Apr 13, 2023, at 16:10, Cy Schubert  wrote:
>> >
>> > In message <20230413070426.8a54f...@slippy.cwsent.com>, Cy Schubert
>> > writes:
>> > In message <20230413064252.1e5c1...@slippy.cwsent.com>, Cy Schubert
>> > writes:
>> >> In message , Mark
>> >> Millard
>> >>> write
>> >>> s:
>> >>> [This just puts my prior reply's material into Cy's
>>  adjusted resend of the original. The To/Cc should
>>  be coomplete this time.]
>> 
>>  On Apr 12, 2023, at 22:52, Cy Schubert  =
>>  wrote:
>> 
>>  In message , Mark =
>> > Millard=20
>>  write
>> > s:
>> > From: Charlie Li  wrote on
>> >> Date: Wed, 12 Apr 2023 20:11:16 UTC :
>> >> =20
>> >> Charlie Li wrote:
>> >>> Mateusz Guzik wrote:
>>  can you please test poudriere with
>> > https://github.com/openzfs/zfs/pull/14739/files
>> > =20
>> > After applying, on the md(4)-backed pool regardless of =3D
>>  block_cloning,=3D20
>> >> the cy@ `cp -R` test reports no differing (ie corrupted) files. =
>>  Will=3D20=3D
>>  =20
>> >> report back on poudriere results (no block_cloning).
>>  =3D20
>>  As for poudriere, build failures are still rolling in. These are
>>  =
>> >>> (and=3D20=3D
>>  =20
>> >> have been) entirely random on every run. Some examples from this =
>> >>> run:
>>  =3D20
>> >>> lang/php81:
>> >>> - post-install: @${INSTALL_DATA}
>> >>> ${WRKSRC}/php.ini-development=3D20
>> >>> ${WRKSRC}/php.ini-production ${WRKDIR}/php.conf =3D
>> >>> ${STAGEDIR}/${PREFIX}/etc
>> >> - consumers fail to build due to corrupted php.conf packaged
>> >>> =3D20
>> >>> devel/ninja:
>> >>> - phase: stage
>> >>> - install -s -m 555=3D20
>> >>> /wrkdirs/usr/ports/devel/ninja/work/ninja-1.11.1/ninja=3D20
>> >>> /wrkdirs/usr/ports/devel/ninja/work/stage/usr/local/bin
>> >>> - consumers fail to build due to corrupted bin/ninja packaged
>> >>> =3D20
>> >>> devel/netsurf-buildsystem:
>> >>> - phase: stage
>> >>> - mkdir -p=3D20
>> >>> =3D
>> >>> =
>> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/n
>>  e=
>> >> =3D
>>  tsurf-buildsystem/makefiles=3D20
>> >> =3D
>> >>> =
>> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/n
>>  e=
>> >> =3D
>>  tsurf-buildsystem/testtools
>> >> for M in Makefile.top Makefile.tools Makefile.subdir =3D
>> >>> Makefile.pkgconfig=3D20
>> >> Makefile.clang Makefile.gcc Makefile.norcroft Makefile.open64; do
>> >> \
>> >>> cp makefiles/$M=3D20
>> >>> =3D
>> >>> =
>> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/n
>>  e=
>> >> =3D
>>  tsurf-buildsystem/makefiles/;=3D20
>> >> \
>> >>> done
>> >>> - graphics/libnsgif fails to build due to NUL characters in=3D20
>> >>> Makefile.{clang,subdir}, causing nothing to link
>> >>> =20
>> >> Summary: I have problems building ports into packages
>> >> via poudriere-devel use despite being fully updated/patched
>> >> (as of when I started the experiment), never having enabled
>> >> block_cloning ( still using openzfs-2.1-freebsd ).
>> >> =20
>> >> In other words, I can confirm other reports that have
>> >> been made.
>> >> =20
>> >> The details follow.
>> >> =20
>> >> =20
>> >> [Written as I was working on setting up for the experiments
>> >> and then executing those experiments, adjusting as I went
>> >> along.]
>> >> =20
>> >> I've run my own tests in a context that has never had the
>> >> zpool upgrade and that jump from before the openzfs import to
>> >> after the existing commits for trying to fix openzfs on
>> >> FreeBSD. I report on the sequence of activities getting to
>> >> the point of testing as well.
>> >> =20
>> >> By personal policy I keep my (non-temporary) pool's compatible
>> >> with what the most recent ??.?-RELEASE supports, using
>> >> openzfs-2.1-freebsd for now. The pools involved below have
>> >> never had a zpool upgrade from where they started. (I've no
>> >> pools that have ever had a zpool upgrade.)
>> >> =20
>> >> (Temporary pools are rare for me, such as this investigation.
>> >> But I'm not testing block_cloning or anything new this time.)
>> >> =20
>> >> I'll note that I use zfs for bectl, not for redundancy. So
>> >> my evidence is more limited in that respect.
>> >> =20
>> >> The activities were done on a HoneyComb (16 Cortex-A72 cores).
>> >> The system has and supports ECC RAM, 64 GiBytes of RAM are
>> >> present.
>> >> =20
>> >> I started by duplicating my normal zfs environment to an
>> >> external USB3 NVMe drive and 

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Cy Schubert]

On Thu, 13 Apr 2023 19:54:42 +0900
Paweł Jakub Dawidek  wrote:

> On Apr 13, 2023, at 16:10, Cy Schubert  wrote:
> > 
> > In message <20230413070426.8a54f...@slippy.cwsent.com>, Cy Schubert writes:
> > In message <20230413064252.1e5c1...@slippy.cwsent.com>, Cy Schubert writes:
> >> In message , Mark Millard
> >>> write
> >>> s:
> >>> [This just puts my prior reply's material into Cy's
>  adjusted resend of the original. The To/Cc should
>  be coomplete this time.]
>  
>  On Apr 12, 2023, at 22:52, Cy Schubert  =
>  wrote:
>  
>  In message , Mark =
> > Millard=20
>  write
> > s:
> > From: Charlie Li  wrote on
> >> Date: Wed, 12 Apr 2023 20:11:16 UTC :
> >> =20
> >> Charlie Li wrote:
> >>> Mateusz Guzik wrote:
>  can you please test poudriere with
> > https://github.com/openzfs/zfs/pull/14739/files
> > =20
> > After applying, on the md(4)-backed pool regardless of =3D
>  block_cloning,=3D20
> >> the cy@ `cp -R` test reports no differing (ie corrupted) files. =
>  Will=3D20=3D
>  =20
> >> report back on poudriere results (no block_cloning).
>  =3D20
>  As for poudriere, build failures are still rolling in. These are =
> >>> (and=3D20=3D
>  =20
> >> have been) entirely random on every run. Some examples from this =
> >>> run:
>  =3D20
> >>> lang/php81:
> >>> - post-install: @${INSTALL_DATA} ${WRKSRC}/php.ini-development=3D20
> >>> ${WRKSRC}/php.ini-production ${WRKDIR}/php.conf =3D
> >>> ${STAGEDIR}/${PREFIX}/etc
> >> - consumers fail to build due to corrupted php.conf packaged
> >>> =3D20
> >>> devel/ninja:
> >>> - phase: stage
> >>> - install -s -m 555=3D20
> >>> /wrkdirs/usr/ports/devel/ninja/work/ninja-1.11.1/ninja=3D20
> >>> /wrkdirs/usr/ports/devel/ninja/work/stage/usr/local/bin
> >>> - consumers fail to build due to corrupted bin/ninja packaged
> >>> =3D20
> >>> devel/netsurf-buildsystem:
> >>> - phase: stage
> >>> - mkdir -p=3D20
> >>> =3D
> >>> =
> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/n
>  e=
> >> =3D
>  tsurf-buildsystem/makefiles=3D20
> >> =3D
> >>> =
> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/n
>  e=
> >> =3D
>  tsurf-buildsystem/testtools
> >> for M in Makefile.top Makefile.tools Makefile.subdir =3D
> >>> Makefile.pkgconfig=3D20
> >> Makefile.clang Makefile.gcc Makefile.norcroft Makefile.open64; do \
> >>> cp makefiles/$M=3D20
> >>> =3D
> >>> =
> >> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/n
>  e=
> >> =3D
>  tsurf-buildsystem/makefiles/;=3D20
> >> \
> >>> done
> >>> - graphics/libnsgif fails to build due to NUL characters in=3D20
> >>> Makefile.{clang,subdir}, causing nothing to link
> >>> =20
> >> Summary: I have problems building ports into packages
> >> via poudriere-devel use despite being fully updated/patched
> >> (as of when I started the experiment), never having enabled
> >> block_cloning ( still using openzfs-2.1-freebsd ).
> >> =20
> >> In other words, I can confirm other reports that have
> >> been made.
> >> =20
> >> The details follow.
> >> =20
> >> =20
> >> [Written as I was working on setting up for the experiments
> >> and then executing those experiments, adjusting as I went
> >> along.]
> >> =20
> >> I've run my own tests in a context that has never had the
> >> zpool upgrade and that jump from before the openzfs import to
> >> after the existing commits for trying to fix openzfs on
> >> FreeBSD. I report on the sequence of activities getting to
> >> the point of testing as well.
> >> =20
> >> By personal policy I keep my (non-temporary) pool's compatible
> >> with what the most recent ??.?-RELEASE supports, using
> >> openzfs-2.1-freebsd for now. The pools involved below have
> >> never had a zpool upgrade from where they started. (I've no
> >> pools that have ever had a zpool upgrade.)
> >> =20
> >> (Temporary pools are rare for me, such as this investigation.
> >> But I'm not testing block_cloning or anything new this time.)
> >> =20
> >> I'll note that I use zfs for bectl, not for redundancy. So
> >> my evidence is more limited in that respect.
> >> =20
> >> The activities were done on a HoneyComb (16 Cortex-A72 cores).
> >> The system has and supports ECC RAM, 64 GiBytes of RAM are
> >> present.
> >> =20
> >> I started by duplicating my normal zfs environment to an
> >> external USB3 NVMe drive and adjusting the host name and such
> >> to produce the below. (Non-debug, although I do not strip
> >> symbols.) :
> >> =20
> >> # uname -apKU
> >> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Mark Millard]

On Apr 13, 2023, at 04:04, Charlie Li  wrote:

> Paweł Jakub Dawidek wrote:
>> Can you please try this patch:
>> 
>> Unfortunately I don’t see how this can happen with block cloning disabled.
> This patch made no difference in poudriere; corruption still rolled in.
> 

Same "made no difference" here.

https://lists.freebsd.org/archives/dev-commits-src-main/2023-April/014512.html

indicated that I'd applied the patch already: "I then also applied the patch
from: https://github.com/openzfs/zfs/pull/14739/files;

Cy reported the same in:

https://lists.freebsd.org/archives/dev-commits-src-main/2023-April/014519.html

"The EXDEV patch is applied. Block_cloning is disabled."

===
Mark Millard
marklmi at yahoo.com

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Charlie Li]

Danilo Egea Gondolfo wrote:

I'm having a funny issue here and I'm wondering if it is related.

When building one of my ports I will, eventually, not always, get a file 
full of zeros as a result.


The build will create copies of crispy-setup and, every once in a while, 
one of them will be a blob of zeros:


I'm running the recent ZFS update but I never upgraded my pool:

This is exactly it. The copy operation within the same dataset will 
sometimes turn up corruption and randomly, so not the same file(s) get hit.


--
Charlie Li
…nope, still don't have an exit line.



OpenPGP_signature
Description: OpenPGP digital signature

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Charlie Li]

Paweł Jakub Dawidek wrote:

Can you please try this patch:


Unfortunately I don’t see how this can happen with block cloning disabled.


This patch made no difference in poudriere; corruption still rolled in.

--
Charlie Li
…nope, still don't have an exit line.



OpenPGP_signature
Description: OpenPGP digital signature

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Danilo Egea Gondolfo]

On 13/04/2023 06:28, Mark Millard wrote:

From: Charlie Li  wrote on
Date: Wed, 12 Apr 2023 20:11:16 UTC :


Charlie Li wrote:

Mateusz Guzik wrote:

can you please test poudriere with
https://github.com/openzfs/zfs/pull/14739/files


After applying, on the md(4)-backed pool regardless of block_cloning,
the cy@ `cp -R` test reports no differing (ie corrupted) files. Will
report back on poudriere results (no block_cloning).


As for poudriere, build failures are still rolling in. These are (and
have been) entirely random on every run. Some examples from this run:

lang/php81:
- post-install: @${INSTALL_DATA} ${WRKSRC}/php.ini-development
${WRKSRC}/php.ini-production ${WRKDIR}/php.conf ${STAGEDIR}/${PREFIX}/etc
- consumers fail to build due to corrupted php.conf packaged

devel/ninja:
- phase: stage
- install -s -m 555
/wrkdirs/usr/ports/devel/ninja/work/ninja-1.11.1/ninja
/wrkdirs/usr/ports/devel/ninja/work/stage/usr/local/bin
- consumers fail to build due to corrupted bin/ninja packaged

devel/netsurf-buildsystem:
- phase: stage
- mkdir -p
/wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/netsurf-buildsystem/makefiles
/wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/netsurf-buildsystem/testtools
for M in Makefile.top Makefile.tools Makefile.subdir Makefile.pkgconfig
Makefile.clang Makefile.gcc Makefile.norcroft Makefile.open64; do \
cp makefiles/$M
/wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/netsurf-buildsystem/makefiles/;
\
done
- graphics/libnsgif fails to build due to NUL characters in
Makefile.{clang,subdir}, causing nothing to link

Summary: I have problems building ports into packages
via poudriere-devel use despite being fully updated/patched
(as of when I started the experiment), never having enabled
block_cloning ( still using openzfs-2.1-freebsd ).

In other words, I can confirm other reports that have
been made.

The details follow.


[Written as I was working on setting up for the experiments
and then executing those experiments, adjusting as I went
along.]

I've run my own tests in a context that has never had the
zpool upgrade and that jump from before the openzfs import to
after the existing commits for trying to fix openzfs on
FreeBSD. I report on the sequence of activities getting to
the point of testing as well.

By personal policy I keep my (non-temporary) pool's compatible
with what the most recent ??.?-RELEASE supports, using
openzfs-2.1-freebsd for now. The pools involved below have
never had a zpool upgrade from where they started. (I've no
pools that have ever had a zpool upgrade.)

(Temporary pools are rare for me, such as this investigation.
But I'm not testing block_cloning or anything new this time.)

I'll note that I use zfs for bectl, not for redundancy. So
my evidence is more limited in that respect.

The activities were done on a HoneyComb (16 Cortex-A72 cores).
The system has and supports ECC RAM, 64 GiBytes of RAM are
present.

I started by duplicating my normal zfs environment to an
external USB3 NVMe drive and adjusting the host name and such
to produce the below. (Non-debug, although I do not strip
symbols.) :

# uname -apKU
FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #90 
main-n261544-cee09bda03c8-dirty: Wed Mar 15 20:25:49 PDT 2023 
root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72
 arm64 aarch64 1400082 1400082

I then did: git fetch, stash push ., merge --ff-only, stash apply . :
my normal procedure. I then also applied the patch from:

https://github.com/openzfs/zfs/pull/14739/files

Then I did: buildworld buildkernel, install them, and rebooted.

The result was:

# uname -apKU
FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #91 
main-n262122-2ef2c26f3f13-dirty: Wed Apr 12 19:23:35 PDT 2023 
root@CA72_4c8G_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72
 arm64 aarch64 1400086 1400086

The later poudriere-devel based build of packages from ports is
based on:

# ~/fbsd-based-on-what-commit.sh -C /usr/ports
4e94ac9eb97f (HEAD -> main, freebsd/main, freebsd/HEAD) devel/freebsd-gcc12: 
Bump to 12.2.0.
Author: John Baldwin 
Commit: John Baldwin 
CommitDate: 2023-03-25 00:06:40 +
branch: main
merge-base: 4e94ac9eb97fab16510b74ebcaa9316613182a72
merge-base: CommitDate: 2023-03-25 00:06:40 +
n613214 (--first-parent --count for merge-base)

poudriere attempted to build 476 packages, starting
with pkg (in order to build the 56 that I explicitly
indicate that I want). It is my normal set of ports.
The form of building is biased to allowing a high
load average compared to the number of hardware
threads (same as cores here): each builder is allowed
to use the full count of hardware threads. The build
used USE_TMPFS="data" instead of the USE_TMPFS=all I
normally use on the build machine involved.

And it produced some random errors during the attempted
builds. A 

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Mark Millard]

On Apr 12, 2023, at 23:52, Alexander Leidinger  wrote:

> Quoting Mark Millard  (from Wed, 12 Apr 2023 22:28:13 
> -0700):
> 
>> A fair number of errors are of the form: the build
>> installing a previously built package for use in the
>> builder but later the builder can not find some file
>> from the package's installation.
> 
> As a data point, last year I had such issues with one particular package. It 
> was consistent no matter how often I was updating the ports tree. Poudriere 
> always failed on port X which was depending on port Y (don't remember the 
> names). The problem was, that port Y was build successfully but an extract of 
> it was not having a file it was supposed to have. IIRC I fixed the issue by 
> building the port Y manually, as re-building port Y with poudriere didn't 
> change the outcome.
> 
> So it seems this may not be specific to the most recent ZFS version, but 
> could be an older issue. It may be the case that the more recent ZFS version 
> amplifies the problem. It can also be that it is related to a specific use 
> case in poudriere.

In my procedure I'm building the same versions of the same ports
that I'd built in the pre-ZFS-import context, just in my
jail for experiments instead of in the jail for normal use.
(So I still have the original package files available.) I
am working a distinct media that started as a copy of my
good context.

In other words, I was reporting differences with the known-status
as shown by prior builds of the same /usr/ports/ tree. The
difference is just my progressing FreeBSD's version.

I'm even using the exact same machine to do the builds, but with
distinct media. (My good environment's FreeBSD still predates
the zfs import.)

> I remember a recent mail which talks about poudriere failing to copy files in 
> resource-limited environments, see 
> https://lists.freebsd.org/archives/dev-commits-src-all/2023-April/025153.html
> While the issue you are trying to pin-point may not be related to this 
> discussion, I mention it because it smells to me like we could be in a 
> situation where a similar combination of unrelated to each other FreeBSD 
> features could form a combination which triggers the issue at hand.

My procedure eliminates this alternative.

===
Mark Millard
marklmi at yahoo.com

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Cy Schubert]

In message <20230413070426.8a54f...@slippy.cwsent.com>, Cy Schubert writes:
> In message <20230413064252.1e5c1...@slippy.cwsent.com>, Cy Schubert writes:
> > In message , Mark Millard 
> > write
> > s:
> > > [This just puts my prior reply's material into Cy's
> > > adjusted resend of the original. The To/Cc should
> > > be coomplete this time.]
> > >
> > > On Apr 12, 2023, at 22:52, Cy Schubert  =
> > > wrote:
> > >
> > > > In message , Mark =
> > > Millard=20
> > > > write
> > > > s:
> > > >> From: Charlie Li  wrote on
> > > >> Date: Wed, 12 Apr 2023 20:11:16 UTC :
> > > >>=20
> > > >>> Charlie Li wrote:
> > >  Mateusz Guzik wrote:
> > > > can you please test poudriere with
> > > > https://github.com/openzfs/zfs/pull/14739/files
> > > >=20
> > >  After applying, on the md(4)-backed pool regardless of =3D
> > > >> block_cloning,=3D20
> > >  the cy@ `cp -R` test reports no differing (ie corrupted) files. =
> > > Will=3D20=3D
> > > >>=20
> > >  report back on poudriere results (no block_cloning).
> > >  =3D20
> > > >>> As for poudriere, build failures are still rolling in. These are =
> > > (and=3D20=3D
> > > >>=20
> > > >>> have been) entirely random on every run. Some examples from this =
> > > run:
> > > >>> =3D20
> > > >>> lang/php81:
> > > >>> - post-install: @${INSTALL_DATA} ${WRKSRC}/php.ini-development=3D20
> > > >>> ${WRKSRC}/php.ini-production ${WRKDIR}/php.conf =3D
> > > >> ${STAGEDIR}/${PREFIX}/etc
> > > >>> - consumers fail to build due to corrupted php.conf packaged
> > > >>> =3D20
> > > >>> devel/ninja:
> > > >>> - phase: stage
> > > >>> - install -s -m 555=3D20
> > > >>> /wrkdirs/usr/ports/devel/ninja/work/ninja-1.11.1/ninja=3D20
> > > >>> /wrkdirs/usr/ports/devel/ninja/work/stage/usr/local/bin
> > > >>> - consumers fail to build due to corrupted bin/ninja packaged
> > > >>> =3D20
> > > >>> devel/netsurf-buildsystem:
> > > >>> - phase: stage
> > > >>> - mkdir -p=3D20
> > > >>> =3D
> > > >> =
> > > /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/n
> e=
> > > =3D
> > > >> tsurf-buildsystem/makefiles=3D20
> > > >>> =3D
> > > >> =
> > > /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/n
> e=
> > > =3D
> > > >> tsurf-buildsystem/testtools
> > > >>> for M in Makefile.top Makefile.tools Makefile.subdir =3D
> > > >> Makefile.pkgconfig=3D20
> > > >>> Makefile.clang Makefile.gcc Makefile.norcroft Makefile.open64; do \
> > > >>> cp makefiles/$M=3D20
> > > >>> =3D
> > > >> =
> > > /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/n
> e=
> > > =3D
> > > >> tsurf-buildsystem/makefiles/;=3D20
> > > >>> \
> > > >>> done
> > > >>> - graphics/libnsgif fails to build due to NUL characters in=3D20
> > > >>> Makefile.{clang,subdir}, causing nothing to link
> > > >>=20
> > > >> Summary: I have problems building ports into packages
> > > >> via poudriere-devel use despite being fully updated/patched
> > > >> (as of when I started the experiment), never having enabled
> > > >> block_cloning ( still using openzfs-2.1-freebsd ).
> > > >>=20
> > > >> In other words, I can confirm other reports that have
> > > >> been made.
> > > >>=20
> > > >> The details follow.
> > > >>=20
> > > >>=20
> > > >> [Written as I was working on setting up for the experiments
> > > >> and then executing those experiments, adjusting as I went
> > > >> along.]
> > > >>=20
> > > >> I've run my own tests in a context that has never had the
> > > >> zpool upgrade and that jump from before the openzfs import to
> > > >> after the existing commits for trying to fix openzfs on
> > > >> FreeBSD. I report on the sequence of activities getting to
> > > >> the point of testing as well.
> > > >>=20
> > > >> By personal policy I keep my (non-temporary) pool's compatible
> > > >> with what the most recent ??.?-RELEASE supports, using
> > > >> openzfs-2.1-freebsd for now. The pools involved below have
> > > >> never had a zpool upgrade from where they started. (I've no
> > > >> pools that have ever had a zpool upgrade.)
> > > >>=20
> > > >> (Temporary pools are rare for me, such as this investigation.
> > > >> But I'm not testing block_cloning or anything new this time.)
> > > >>=20
> > > >> I'll note that I use zfs for bectl, not for redundancy. So
> > > >> my evidence is more limited in that respect.
> > > >>=20
> > > >> The activities were done on a HoneyComb (16 Cortex-A72 cores).
> > > >> The system has and supports ECC RAM, 64 GiBytes of RAM are
> > > >> present.
> > > >>=20
> > > >> I started by duplicating my normal zfs environment to an
> > > >> external USB3 NVMe drive and adjusting the host name and such
> > > >> to produce the below. (Non-debug, although I do not strip
> > > >> symbols.) :
> > > >>=20
> > > >> # uname -apKU
> > > >> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #90 =3D
> > > >> main-n261544-cee09bda03c8-dirty: Wed Mar 15 20:25:49 PDT 2023 =3D
> > > >> =
> > > 

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Cy Schubert]

In message <20230413064252.1e5c1...@slippy.cwsent.com>, Cy Schubert writes:
> In message , Mark Millard 
> write
> s:
> > [This just puts my prior reply's material into Cy's
> > adjusted resend of the original. The To/Cc should
> > be coomplete this time.]
> >
> > On Apr 12, 2023, at 22:52, Cy Schubert  =
> > wrote:
> >
> > > In message , Mark =
> > Millard=20
> > > write
> > > s:
> > >> From: Charlie Li  wrote on
> > >> Date: Wed, 12 Apr 2023 20:11:16 UTC :
> > >>=20
> > >>> Charlie Li wrote:
> >  Mateusz Guzik wrote:
> > > can you please test poudriere with
> > > https://github.com/openzfs/zfs/pull/14739/files
> > >=20
> >  After applying, on the md(4)-backed pool regardless of =3D
> > >> block_cloning,=3D20
> >  the cy@ `cp -R` test reports no differing (ie corrupted) files. =
> > Will=3D20=3D
> > >>=20
> >  report back on poudriere results (no block_cloning).
> >  =3D20
> > >>> As for poudriere, build failures are still rolling in. These are =
> > (and=3D20=3D
> > >>=20
> > >>> have been) entirely random on every run. Some examples from this =
> > run:
> > >>> =3D20
> > >>> lang/php81:
> > >>> - post-install: @${INSTALL_DATA} ${WRKSRC}/php.ini-development=3D20
> > >>> ${WRKSRC}/php.ini-production ${WRKDIR}/php.conf =3D
> > >> ${STAGEDIR}/${PREFIX}/etc
> > >>> - consumers fail to build due to corrupted php.conf packaged
> > >>> =3D20
> > >>> devel/ninja:
> > >>> - phase: stage
> > >>> - install -s -m 555=3D20
> > >>> /wrkdirs/usr/ports/devel/ninja/work/ninja-1.11.1/ninja=3D20
> > >>> /wrkdirs/usr/ports/devel/ninja/work/stage/usr/local/bin
> > >>> - consumers fail to build due to corrupted bin/ninja packaged
> > >>> =3D20
> > >>> devel/netsurf-buildsystem:
> > >>> - phase: stage
> > >>> - mkdir -p=3D20
> > >>> =3D
> > >> =
> > /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne=
> > =3D
> > >> tsurf-buildsystem/makefiles=3D20
> > >>> =3D
> > >> =
> > /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne=
> > =3D
> > >> tsurf-buildsystem/testtools
> > >>> for M in Makefile.top Makefile.tools Makefile.subdir =3D
> > >> Makefile.pkgconfig=3D20
> > >>> Makefile.clang Makefile.gcc Makefile.norcroft Makefile.open64; do \
> > >>> cp makefiles/$M=3D20
> > >>> =3D
> > >> =
> > /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne=
> > =3D
> > >> tsurf-buildsystem/makefiles/;=3D20
> > >>> \
> > >>> done
> > >>> - graphics/libnsgif fails to build due to NUL characters in=3D20
> > >>> Makefile.{clang,subdir}, causing nothing to link
> > >>=20
> > >> Summary: I have problems building ports into packages
> > >> via poudriere-devel use despite being fully updated/patched
> > >> (as of when I started the experiment), never having enabled
> > >> block_cloning ( still using openzfs-2.1-freebsd ).
> > >>=20
> > >> In other words, I can confirm other reports that have
> > >> been made.
> > >>=20
> > >> The details follow.
> > >>=20
> > >>=20
> > >> [Written as I was working on setting up for the experiments
> > >> and then executing those experiments, adjusting as I went
> > >> along.]
> > >>=20
> > >> I've run my own tests in a context that has never had the
> > >> zpool upgrade and that jump from before the openzfs import to
> > >> after the existing commits for trying to fix openzfs on
> > >> FreeBSD. I report on the sequence of activities getting to
> > >> the point of testing as well.
> > >>=20
> > >> By personal policy I keep my (non-temporary) pool's compatible
> > >> with what the most recent ??.?-RELEASE supports, using
> > >> openzfs-2.1-freebsd for now. The pools involved below have
> > >> never had a zpool upgrade from where they started. (I've no
> > >> pools that have ever had a zpool upgrade.)
> > >>=20
> > >> (Temporary pools are rare for me, such as this investigation.
> > >> But I'm not testing block_cloning or anything new this time.)
> > >>=20
> > >> I'll note that I use zfs for bectl, not for redundancy. So
> > >> my evidence is more limited in that respect.
> > >>=20
> > >> The activities were done on a HoneyComb (16 Cortex-A72 cores).
> > >> The system has and supports ECC RAM, 64 GiBytes of RAM are
> > >> present.
> > >>=20
> > >> I started by duplicating my normal zfs environment to an
> > >> external USB3 NVMe drive and adjusting the host name and such
> > >> to produce the below. (Non-debug, although I do not strip
> > >> symbols.) :
> > >>=20
> > >> # uname -apKU
> > >> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #90 =3D
> > >> main-n261544-cee09bda03c8-dirty: Wed Mar 15 20:25:49 PDT 2023 =3D
> > >> =
> > root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6=
> > =3D
> > >> 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400082 1400082
> > >>=20
> > >> I then did: git fetch, stash push ., merge --ff-only, stash apply . :
> > >> my normal procedure. I then also applied the patch from:
> > >>=20
> > >> https://github.com/openzfs/zfs/pull/14739/files
> > >>=20
> > >> 

Re: Status of Alder and Raptor lake on FreeBSD Current

[Stephane Rochoy]

Dries Michiels  writes:

I was wondering what the status of Alder/Raptor lake support is 
on FreeBSD?
Does it boot? Integrated graphics are supported from what I 
recall with the

newest drm drivers.
Are there any plans for changes to our scheduler to account for 
efficiency

and performance cores?
Are there real world downsides of not having such a scheduler 
when running

an Alder or Raptor lake CPU?

Interested to hear from users using these CPU's right now in 
there system!
The Reason I ask is that I'm interested in upgrading my home 
server

hardware :-).


Hi,

AFAIK, last status was given by Mike Karels (karels@) on Feb 7,
2023 on freebsd-stable@ [1]. I'm also interested in any news on
the topic :)

[1] 
https://lists.freebsd.org/archives/freebsd-stable/2023-February/001103.html


Regards,
--
Stéphane Rochoy
O: Stormshield

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Alexander Leidinger]

Quoting Mark Millard  (from Wed, 12 Apr 2023  
22:28:13 -0700):



A fair number of errors are of the form: the build
installing a previously built package for use in the
builder but later the builder can not find some file
from the package's installation.


As a data point, last year I had such issues with one particular  
package. It was consistent no matter how often I was updating the  
ports tree. Poudriere always failed on port X which was depending on  
port Y (don't remember the names). The problem was, that port Y was  
build successfully but an extract of it was not having a file it was  
supposed to have. IIRC I fixed the issue by building the port Y  
manually, as re-building port Y with poudriere didn't change the  
outcome.


So it seems this may not be specific to the most recent ZFS version,  
but could be an older issue. It may be the case that the more recent  
ZFS version amplifies the problem. It can also be that it is related  
to a specific use case in poudriere.


I remember a recent mail which talks about poudriere failing to copy  
files in resource-limited environments, see  
https://lists.freebsd.org/archives/dev-commits-src-all/2023-April/025153.html
While the issue you are trying to pin-point may not be related to this  
discussion, I mention it because it smells to me like we could be in a  
situation where a similar combination of unrelated to each other  
FreeBSD features could form a combination which triggers the issue at  
hand.


Bye,
Alexander.

--
http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.orgnetch...@freebsd.org  : PGP 0x8F31830F9F2772BF


pgpjoaPNf5aAM.pgp
Description: Digitale PGP-Signatur

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Cy Schubert]

In message , Mark Millard 
write
s:
> [This just puts my prior reply's material into Cy's
> adjusted resend of the original. The To/Cc should
> be coomplete this time.]
>
> On Apr 12, 2023, at 22:52, Cy Schubert  =
> wrote:
>
> > In message , Mark =
> Millard=20
> > write
> > s:
> >> From: Charlie Li  wrote on
> >> Date: Wed, 12 Apr 2023 20:11:16 UTC :
> >>=20
> >>> Charlie Li wrote:
>  Mateusz Guzik wrote:
> > can you please test poudriere with
> > https://github.com/openzfs/zfs/pull/14739/files
> >=20
>  After applying, on the md(4)-backed pool regardless of =3D
> >> block_cloning,=3D20
>  the cy@ `cp -R` test reports no differing (ie corrupted) files. =
> Will=3D20=3D
> >>=20
>  report back on poudriere results (no block_cloning).
>  =3D20
> >>> As for poudriere, build failures are still rolling in. These are =
> (and=3D20=3D
> >>=20
> >>> have been) entirely random on every run. Some examples from this =
> run:
> >>> =3D20
> >>> lang/php81:
> >>> - post-install: @${INSTALL_DATA} ${WRKSRC}/php.ini-development=3D20
> >>> ${WRKSRC}/php.ini-production ${WRKDIR}/php.conf =3D
> >> ${STAGEDIR}/${PREFIX}/etc
> >>> - consumers fail to build due to corrupted php.conf packaged
> >>> =3D20
> >>> devel/ninja:
> >>> - phase: stage
> >>> - install -s -m 555=3D20
> >>> /wrkdirs/usr/ports/devel/ninja/work/ninja-1.11.1/ninja=3D20
> >>> /wrkdirs/usr/ports/devel/ninja/work/stage/usr/local/bin
> >>> - consumers fail to build due to corrupted bin/ninja packaged
> >>> =3D20
> >>> devel/netsurf-buildsystem:
> >>> - phase: stage
> >>> - mkdir -p=3D20
> >>> =3D
> >> =
> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne=
> =3D
> >> tsurf-buildsystem/makefiles=3D20
> >>> =3D
> >> =
> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne=
> =3D
> >> tsurf-buildsystem/testtools
> >>> for M in Makefile.top Makefile.tools Makefile.subdir =3D
> >> Makefile.pkgconfig=3D20
> >>> Makefile.clang Makefile.gcc Makefile.norcroft Makefile.open64; do \
> >>> cp makefiles/$M=3D20
> >>> =3D
> >> =
> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne=
> =3D
> >> tsurf-buildsystem/makefiles/;=3D20
> >>> \
> >>> done
> >>> - graphics/libnsgif fails to build due to NUL characters in=3D20
> >>> Makefile.{clang,subdir}, causing nothing to link
> >>=20
> >> Summary: I have problems building ports into packages
> >> via poudriere-devel use despite being fully updated/patched
> >> (as of when I started the experiment), never having enabled
> >> block_cloning ( still using openzfs-2.1-freebsd ).
> >>=20
> >> In other words, I can confirm other reports that have
> >> been made.
> >>=20
> >> The details follow.
> >>=20
> >>=20
> >> [Written as I was working on setting up for the experiments
> >> and then executing those experiments, adjusting as I went
> >> along.]
> >>=20
> >> I've run my own tests in a context that has never had the
> >> zpool upgrade and that jump from before the openzfs import to
> >> after the existing commits for trying to fix openzfs on
> >> FreeBSD. I report on the sequence of activities getting to
> >> the point of testing as well.
> >>=20
> >> By personal policy I keep my (non-temporary) pool's compatible
> >> with what the most recent ??.?-RELEASE supports, using
> >> openzfs-2.1-freebsd for now. The pools involved below have
> >> never had a zpool upgrade from where they started. (I've no
> >> pools that have ever had a zpool upgrade.)
> >>=20
> >> (Temporary pools are rare for me, such as this investigation.
> >> But I'm not testing block_cloning or anything new this time.)
> >>=20
> >> I'll note that I use zfs for bectl, not for redundancy. So
> >> my evidence is more limited in that respect.
> >>=20
> >> The activities were done on a HoneyComb (16 Cortex-A72 cores).
> >> The system has and supports ECC RAM, 64 GiBytes of RAM are
> >> present.
> >>=20
> >> I started by duplicating my normal zfs environment to an
> >> external USB3 NVMe drive and adjusting the host name and such
> >> to produce the below. (Non-debug, although I do not strip
> >> symbols.) :
> >>=20
> >> # uname -apKU
> >> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #90 =3D
> >> main-n261544-cee09bda03c8-dirty: Wed Mar 15 20:25:49 PDT 2023 =3D
> >> =
> root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6=
> =3D
> >> 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400082 1400082
> >>=20
> >> I then did: git fetch, stash push ., merge --ff-only, stash apply . :
> >> my normal procedure. I then also applied the patch from:
> >>=20
> >> https://github.com/openzfs/zfs/pull/14739/files
> >>=20
> >> Then I did: buildworld buildkernel, install them, and rebooted.
> >>=20
> >> The result was:
> >>=20
> >> # uname -apKU
> >> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #91 =3D
> >> main-n262122-2ef2c26f3f13-dirty: Wed Apr 12 19:23:35 PDT 2023 =3D
> >> =
> 

Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75

[Mark Millard]

[This just puts my prior reply's material into Cy's
adjusted resend of the original. The To/Cc should
be coomplete this time.]

On Apr 12, 2023, at 22:52, Cy Schubert  wrote:

> In message , Mark Millard 
> write
> s:
>> From: Charlie Li  wrote on
>> Date: Wed, 12 Apr 2023 20:11:16 UTC :
>> 
>>> Charlie Li wrote:
 Mateusz Guzik wrote:
> can you please test poudriere with
> https://github.com/openzfs/zfs/pull/14739/files
> 
 After applying, on the md(4)-backed pool regardless of =
>> block_cloning,=20
 the cy@ `cp -R` test reports no differing (ie corrupted) files. Will=20=
>> 
 report back on poudriere results (no block_cloning).
 =20
>>> As for poudriere, build failures are still rolling in. These are (and=20=
>> 
>>> have been) entirely random on every run. Some examples from this run:
>>> =20
>>> lang/php81:
>>> - post-install: @${INSTALL_DATA} ${WRKSRC}/php.ini-development=20
>>> ${WRKSRC}/php.ini-production ${WRKDIR}/php.conf =
>> ${STAGEDIR}/${PREFIX}/etc
>>> - consumers fail to build due to corrupted php.conf packaged
>>> =20
>>> devel/ninja:
>>> - phase: stage
>>> - install -s -m 555=20
>>> /wrkdirs/usr/ports/devel/ninja/work/ninja-1.11.1/ninja=20
>>> /wrkdirs/usr/ports/devel/ninja/work/stage/usr/local/bin
>>> - consumers fail to build due to corrupted bin/ninja packaged
>>> =20
>>> devel/netsurf-buildsystem:
>>> - phase: stage
>>> - mkdir -p=20
>>> =
>> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne=
>> tsurf-buildsystem/makefiles=20
>>> =
>> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne=
>> tsurf-buildsystem/testtools
>>> for M in Makefile.top Makefile.tools Makefile.subdir =
>> Makefile.pkgconfig=20
>>> Makefile.clang Makefile.gcc Makefile.norcroft Makefile.open64; do \
>>> cp makefiles/$M=20
>>> =
>> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/local/share/ne=
>> tsurf-buildsystem/makefiles/;=20
>>> \
>>> done
>>> - graphics/libnsgif fails to build due to NUL characters in=20
>>> Makefile.{clang,subdir}, causing nothing to link
>> 
>> Summary: I have problems building ports into packages
>> via poudriere-devel use despite being fully updated/patched
>> (as of when I started the experiment), never having enabled
>> block_cloning ( still using openzfs-2.1-freebsd ).
>> 
>> In other words, I can confirm other reports that have
>> been made.
>> 
>> The details follow.
>> 
>> 
>> [Written as I was working on setting up for the experiments
>> and then executing those experiments, adjusting as I went
>> along.]
>> 
>> I've run my own tests in a context that has never had the
>> zpool upgrade and that jump from before the openzfs import to
>> after the existing commits for trying to fix openzfs on
>> FreeBSD. I report on the sequence of activities getting to
>> the point of testing as well.
>> 
>> By personal policy I keep my (non-temporary) pool's compatible
>> with what the most recent ??.?-RELEASE supports, using
>> openzfs-2.1-freebsd for now. The pools involved below have
>> never had a zpool upgrade from where they started. (I've no
>> pools that have ever had a zpool upgrade.)
>> 
>> (Temporary pools are rare for me, such as this investigation.
>> But I'm not testing block_cloning or anything new this time.)
>> 
>> I'll note that I use zfs for bectl, not for redundancy. So
>> my evidence is more limited in that respect.
>> 
>> The activities were done on a HoneyComb (16 Cortex-A72 cores).
>> The system has and supports ECC RAM, 64 GiBytes of RAM are
>> present.
>> 
>> I started by duplicating my normal zfs environment to an
>> external USB3 NVMe drive and adjusting the host name and such
>> to produce the below. (Non-debug, although I do not strip
>> symbols.) :
>> 
>> # uname -apKU
>> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #90 =
>> main-n261544-cee09bda03c8-dirty: Wed Mar 15 20:25:49 PDT 2023 =
>> root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6=
>> 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400082 1400082
>> 
>> I then did: git fetch, stash push ., merge --ff-only, stash apply . :
>> my normal procedure. I then also applied the patch from:
>> 
>> https://github.com/openzfs/zfs/pull/14739/files
>> 
>> Then I did: buildworld buildkernel, install them, and rebooted.
>> 
>> The result was:
>> 
>> # uname -apKU
>> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #91 =
>> main-n262122-2ef2c26f3f13-dirty: Wed Apr 12 19:23:35 PDT 2023 =
>> root@CA72_4c8G_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6=
>> 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400086 1400086
>> 
>> The later poudriere-devel based build of packages from ports is
>> based on:
>> 
>> # ~/fbsd-based-on-what-commit.sh -C /usr/ports
>> 4e94ac9eb97f (HEAD -> main, freebsd/main, freebsd/HEAD) =
>> devel/freebsd-gcc12: Bump to 12.2.0.
>> Author: John Baldwin 
>> Commit: John Baldwin 
>> CommitDate: 2023-03-25 00:06:40 +
>> branch: main
>>