Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Mark Millard via freebsd-stable]

On 2021-May-23, at 01:27, Mark Millard  wrote:

> On 2021-May-23, at 00:44, Mark Millard  wrote:
> 
>> On 2021-May-21, at 17:56, Rick Macklem  wrote:
>> 
>>> Mark Millard wrote:
>>> [stuff snipped]
 Well, why is it that ls -R, find, and diff -r all get file
 name problems via genet0 but diff -r gets no problems
 comparing the content of files that it does match up (the
 vast majority)? Any clue how could the problems possibly
 be unique to the handling of file names/paths? Does it
 suggest anything else to look into for getting some more
 potentially useful evidence?
>>> Well, all I can do is describe the most common TSO related
>>> failure:
>>> - When a read RPC reply (including NFS/RPC/TCP/IP headers)
>>> is slightly less than 64K bytes (many TSO implementations are
>>> limited to 64K or 32 discontiguous segments, think 32 2K
>>> mbuf clusters), the driver decides it is ok, but when the MAC
>>> header is added it exceeds what the hardware can handle correctly...
>>> --> This will happen when reading a regular file that is slightly less
>>> than a multiple of 64K in size.
>>> or
>>> --> This will happen when reading just about any large directory,
>>>since the directory reply for a 64K request is converted to Sun XDR
>>>format and clipped at the last full directory entry that will fit within 
>>> 64K.
>>> For ports, where most files are small, I think you can tell which is more
>>> likely to happen.
>>> --> If TSO is disabled, I have no idea how this might matter, but??
>>> 
 I'll note that netstat -I ue0 -d and netstat -I genet0 -d
 do not report changes in Ierrs or Idrop in a before vs.
 after failures comparison. (There may be better figures
 to look at for all I know.)
 
 I tried "ifconfig genet0 -rxcsum -rxcsum -rxcsum6 -txcsum6"
 and got no obvious change in behavior.
>>> All we know is that the data is getting corrupted somehow.
>>> 
>>> NFS traffic looks very different than typical TCP traffic. It is
>>> mostly small messages travelling in both directions concurrently,
>>> with some large messages thrown in the mix.
>>> All I'm saying is that, testing a net interface with something like
>>> bulk data transfer in one direction doesn't verify it works for NFS
>>> traffic.
>>> 
>>> Also, the large RPC messages are a chain of about 33 mbufs of
>>> various lengths, including a mix of partial clusters and regular
>>> data mbufs, whereas a bulk send on a socket will typically
>>> result in an mbuf chain of a lot of full 2K clusters.
>>> --> As such, NFS can be good at tickling subtle bugs it the
>>>net driver related to mbuf handling.
>>> 
>>> rick
>>> 
> W.r.t. reverting r367492...the patch to replace r367492 was just
> committed to "main" by rscheff@ with a two week MFC, so it
> should be in stable/13 soon. Not sure if an errata can be done
> for it for releng13.0?
 
 That update is reported to be causing "rack" related panics:
 
 https://lists.freebsd.org/pipermail/dev-commits-src-main/2021-May/004440.html
 
 reports (via links):
 
 panic: _mtx_lock_sleep: recursed on non-recursive mutex so_snd @ 
 /syzkaller/managers/i386/kernel/sys/modules/tcp/rack/../../../netinet/tcp_stacks/rack.c:10632
 
 Still, I have a non-debug update to main building and will
 likely do a debug build as well. llvm is rebuilding, so
 the builds will take a notable time.
>> 
>> I got the following built and installed on the two
>> machines:
>> 
>> # uname -apKU
>> FreeBSD CA72_16Gp_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #1 
>> main-n246854-03b0505b8fe8-dirty: Sat May 22 16:25:04 PDT 2021 
>> root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-dbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-DBG-CA72
>>   arm64 aarch64 1400013 1400013
>> 
>> # uname -apKU
>> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #1 
>> main-n246854-03b0505b8fe8-dirty: Sat May 22 16:25:04 PDT 2021 
>> root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-dbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-DBG-CA72
>>   arm64 aarch64 1400013 1400013
>> 
>> Note that both are booted with debug builds of main.
>> 
>> Using the context with the alternate EtherNet device that has not
>> had an associated diff -r, find, pr ls -R failure yet
>> yet got a panic that looks likely to be unrelated:
>> 
>> # mount -onoatime 192.168.1.187:/usr/ports/ /mnt/
>> # diff -r /usr/ports/ /mnt/ | more
>> nvme0: cpl does not map to outstanding cmd
>> cdw0: sqhd:0020 sqid:0003 cid:007e p:1 sc:00 sct:0 m:0 dnr:0
>> panic: received completion for unknown cmd
>> cpuid = 3
>> time = 1621743752
>> KDB: stack backtrace:
>> db_trace_self() at db_trace_self
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
>> vpanic() at vpanic+0x188
>> panic() at panic+0x44
>> nvme_qpair_process_completions() at nvme_qpair_process_completions+0x1fc
>> nvme_timeout() at nvme_timeout+0x3c
>> softclock_call_cc() at softclock_call_cc+0x124
>> softclock() 

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Mark Millard via freebsd-stable]

On 2021-May-23, at 00:44, Mark Millard  wrote:

> On 2021-May-21, at 17:56, Rick Macklem  wrote:
> 
>> Mark Millard wrote:
>> [stuff snipped]
>>> Well, why is it that ls -R, find, and diff -r all get file
>>> name problems via genet0 but diff -r gets no problems
>>> comparing the content of files that it does match up (the
>>> vast majority)? Any clue how could the problems possibly
>>> be unique to the handling of file names/paths? Does it
>>> suggest anything else to look into for getting some more
>>> potentially useful evidence?
>> Well, all I can do is describe the most common TSO related
>> failure:
>> - When a read RPC reply (including NFS/RPC/TCP/IP headers)
>> is slightly less than 64K bytes (many TSO implementations are
>> limited to 64K or 32 discontiguous segments, think 32 2K
>> mbuf clusters), the driver decides it is ok, but when the MAC
>> header is added it exceeds what the hardware can handle correctly...
>> --> This will happen when reading a regular file that is slightly less
>>  than a multiple of 64K in size.
>> or
>> --> This will happen when reading just about any large directory,
>> since the directory reply for a 64K request is converted to Sun XDR
>> format and clipped at the last full directory entry that will fit within 
>> 64K.
>> For ports, where most files are small, I think you can tell which is more
>> likely to happen.
>> --> If TSO is disabled, I have no idea how this might matter, but??
>> 
>>> I'll note that netstat -I ue0 -d and netstat -I genet0 -d
>>> do not report changes in Ierrs or Idrop in a before vs.
>>> after failures comparison. (There may be better figures
>>> to look at for all I know.)
>>> 
>>> I tried "ifconfig genet0 -rxcsum -rxcsum -rxcsum6 -txcsum6"
>>> and got no obvious change in behavior.
>> All we know is that the data is getting corrupted somehow.
>> 
>> NFS traffic looks very different than typical TCP traffic. It is
>> mostly small messages travelling in both directions concurrently,
>> with some large messages thrown in the mix.
>> All I'm saying is that, testing a net interface with something like
>> bulk data transfer in one direction doesn't verify it works for NFS
>> traffic.
>> 
>> Also, the large RPC messages are a chain of about 33 mbufs of
>> various lengths, including a mix of partial clusters and regular
>> data mbufs, whereas a bulk send on a socket will typically
>> result in an mbuf chain of a lot of full 2K clusters.
>> --> As such, NFS can be good at tickling subtle bugs it the
>> net driver related to mbuf handling.
>> 
>> rick
>> 
 W.r.t. reverting r367492...the patch to replace r367492 was just
 committed to "main" by rscheff@ with a two week MFC, so it
 should be in stable/13 soon. Not sure if an errata can be done
 for it for releng13.0?
>>> 
>>> That update is reported to be causing "rack" related panics:
>>> 
>>> https://lists.freebsd.org/pipermail/dev-commits-src-main/2021-May/004440.html
>>> 
>>> reports (via links):
>>> 
>>> panic: _mtx_lock_sleep: recursed on non-recursive mutex so_snd @ 
>>> /syzkaller/managers/i386/kernel/sys/modules/tcp/rack/../../../netinet/tcp_stacks/rack.c:10632
>>> 
>>> Still, I have a non-debug update to main building and will
>>> likely do a debug build as well. llvm is rebuilding, so
>>> the builds will take a notable time.
> 
> I got the following built and installed on the two
> machines:
> 
> # uname -apKU
> FreeBSD CA72_16Gp_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #1 
> main-n246854-03b0505b8fe8-dirty: Sat May 22 16:25:04 PDT 2021 
> root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-dbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-DBG-CA72
>   arm64 aarch64 1400013 1400013
> 
> # uname -apKU
> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #1 
> main-n246854-03b0505b8fe8-dirty: Sat May 22 16:25:04 PDT 2021 
> root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-dbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-DBG-CA72
>   arm64 aarch64 1400013 1400013
> 
> Note that both are booted with debug builds of main.
> 
> Using the context with the alternate EtherNet device that has not
> had an associated diff -r, find, pr ls -R failure yet
> yet got a panic that looks likely to be unrelated:
> 
> # mount -onoatime 192.168.1.187:/usr/ports/ /mnt/
> # diff -r /usr/ports/ /mnt/ | more
> nvme0: cpl does not map to outstanding cmd
> cdw0: sqhd:0020 sqid:0003 cid:007e p:1 sc:00 sct:0 m:0 dnr:0
> panic: received completion for unknown cmd
> cpuid = 3
> time = 1621743752
> KDB: stack backtrace:
> db_trace_self() at db_trace_self
> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
> vpanic() at vpanic+0x188
> panic() at panic+0x44
> nvme_qpair_process_completions() at nvme_qpair_process_completions+0x1fc
> nvme_timeout() at nvme_timeout+0x3c
> softclock_call_cc() at softclock_call_cc+0x124
> softclock() at softclock+0x60
> ithread_loop() at ithread_loop+0x2a8
> fork_exit() at fork_exit+0x74
> fork_trampoline() at fork_trampoline+0x14
> KDB: enter: panic
> [ 

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Mark Millard via freebsd-stable]

On 2021-May-21, at 17:56, Rick Macklem  wrote:

> Mark Millard wrote:
> [stuff snipped]
>> Well, why is it that ls -R, find, and diff -r all get file
>> name problems via genet0 but diff -r gets no problems
>> comparing the content of files that it does match up (the
>> vast majority)? Any clue how could the problems possibly
>> be unique to the handling of file names/paths? Does it
>> suggest anything else to look into for getting some more
>> potentially useful evidence?
> Well, all I can do is describe the most common TSO related
> failure:
> - When a read RPC reply (including NFS/RPC/TCP/IP headers)
>  is slightly less than 64K bytes (many TSO implementations are
>  limited to 64K or 32 discontiguous segments, think 32 2K
>  mbuf clusters), the driver decides it is ok, but when the MAC
>  header is added it exceeds what the hardware can handle correctly...
> --> This will happen when reading a regular file that is slightly less
>   than a multiple of 64K in size.
> or
> --> This will happen when reading just about any large directory,
>  since the directory reply for a 64K request is converted to Sun XDR
>  format and clipped at the last full directory entry that will fit within 
> 64K.
> For ports, where most files are small, I think you can tell which is more
> likely to happen.
> --> If TSO is disabled, I have no idea how this might matter, but??
> 
>> I'll note that netstat -I ue0 -d and netstat -I genet0 -d
>> do not report changes in Ierrs or Idrop in a before vs.
>> after failures comparison. (There may be better figures
>> to look at for all I know.)
>> 
>> I tried "ifconfig genet0 -rxcsum -rxcsum -rxcsum6 -txcsum6"
>> and got no obvious change in behavior.
> All we know is that the data is getting corrupted somehow.
> 
> NFS traffic looks very different than typical TCP traffic. It is
> mostly small messages travelling in both directions concurrently,
> with some large messages thrown in the mix.
> All I'm saying is that, testing a net interface with something like
> bulk data transfer in one direction doesn't verify it works for NFS
> traffic.
> 
> Also, the large RPC messages are a chain of about 33 mbufs of
> various lengths, including a mix of partial clusters and regular
> data mbufs, whereas a bulk send on a socket will typically
> result in an mbuf chain of a lot of full 2K clusters.
> --> As such, NFS can be good at tickling subtle bugs it the
>  net driver related to mbuf handling.
> 
> rick
> 
>>> W.r.t. reverting r367492...the patch to replace r367492 was just
>>> committed to "main" by rscheff@ with a two week MFC, so it
>>> should be in stable/13 soon. Not sure if an errata can be done
>>> for it for releng13.0?
>> 
>> That update is reported to be causing "rack" related panics:
>> 
>> https://lists.freebsd.org/pipermail/dev-commits-src-main/2021-May/004440.html
>> 
>> reports (via links):
>> 
>> panic: _mtx_lock_sleep: recursed on non-recursive mutex so_snd @ 
>> /syzkaller/managers/i386/kernel/sys/modules/tcp/rack/../../../netinet/tcp_stacks/rack.c:10632
>> 
>> Still, I have a non-debug update to main building and will
>> likely do a debug build as well. llvm is rebuilding, so
>> the builds will take a notable time.

I got the following built and installed on the two
machines:

# uname -apKU
FreeBSD CA72_16Gp_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #1 
main-n246854-03b0505b8fe8-dirty: Sat May 22 16:25:04 PDT 2021 
root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-dbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-DBG-CA72
  arm64 aarch64 1400013 1400013

# uname -apKU
FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #1 
main-n246854-03b0505b8fe8-dirty: Sat May 22 16:25:04 PDT 2021 
root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-dbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-DBG-CA72
  arm64 aarch64 1400013 1400013

Note that both are booted with debug builds of main.

Using the context with the alternate EtherNet device that has not
had an associated diff -r, find, pr ls -R failure yet
yet got a panic that looks likely to be unrelated:

# mount -onoatime 192.168.1.187:/usr/ports/ /mnt/
# diff -r /usr/ports/ /mnt/ | more
nvme0: cpl does not map to outstanding cmd
cdw0: sqhd:0020 sqid:0003 cid:007e p:1 sc:00 sct:0 m:0 dnr:0
panic: received completion for unknown cmd
cpuid = 3
time = 1621743752
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x30
vpanic() at vpanic+0x188
panic() at panic+0x44
nvme_qpair_process_completions() at nvme_qpair_process_completions+0x1fc
nvme_timeout() at nvme_timeout+0x3c
softclock_call_cc() at softclock_call_cc+0x124
softclock() at softclock+0x60
ithread_loop() at ithread_loop+0x2a8
fork_exit() at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x14
KDB: enter: panic
[ thread pid 12 tid 100028 ]
Stopped at  kdb_enter+0x48: undefined   f904411f
db> 

Based on the "nvme" references, I expect this is tied to
handling the Optane 480 GiByte that is in the PCIe 

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Rick Macklem]

Mark Millard wrote:
[stuff snipped]
>Well, why is it that ls -R, find, and diff -r all get file
>name problems via genet0 but diff -r gets no problems
>comparing the content of files that it does match up (the
>vast majority)? Any clue how could the problems possibly
>be unique to the handling of file names/paths? Does it
>suggest anything else to look into for getting some more
>potentially useful evidence?
Well, all I can do is describe the most common TSO related
failure:
- When a read RPC reply (including NFS/RPC/TCP/IP headers)
  is slightly less than 64K bytes (many TSO implementations are
  limited to 64K or 32 discontiguous segments, think 32 2K
  mbuf clusters), the driver decides it is ok, but when the MAC
  header is added it exceeds what the hardware can handle correctly...
--> This will happen when reading a regular file that is slightly less
   than a multiple of 64K in size.
or
--> This will happen when reading just about any large directory,
  since the directory reply for a 64K request is converted to Sun XDR
  format and clipped at the last full directory entry that will fit within 
64K.
For ports, where most files are small, I think you can tell which is more
likely to happen.
--> If TSO is disabled, I have no idea how this might matter, but??

>I'll note that netstat -I ue0 -d and netstat -I genet0 -d
>do not report changes in Ierrs or Idrop in a before vs.
>after failures comparison. (There may be better figures
>to look at for all I know.)
>
>I tried "ifconfig genet0 -rxcsum -rxcsum -rxcsum6 -txcsum6"
>and got no obvious change in behavior.
All we know is that the data is getting corrupted somehow.

NFS traffic looks very different than typical TCP traffic. It is
mostly small messages travelling in both directions concurrently,
with some large messages thrown in the mix.
All I'm saying is that, testing a net interface with something like
bulk data transfer in one direction doesn't verify it works for NFS
traffic.

Also, the large RPC messages are a chain of about 33 mbufs of
various lengths, including a mix of partial clusters and regular
data mbufs, whereas a bulk send on a socket will typically
result in an mbuf chain of a lot of full 2K clusters.
--> As such, NFS can be good at tickling subtle bugs it the
  net driver related to mbuf handling.

rick

> W.r.t. reverting r367492...the patch to replace r367492 was just
> committed to "main" by rscheff@ with a two week MFC, so it
> should be in stable/13 soon. Not sure if an errata can be done
> for it for releng13.0?

That update is reported to be causing "rack" related panics:

https://lists.freebsd.org/pipermail/dev-commits-src-main/2021-May/004440.html

reports (via links):

panic: _mtx_lock_sleep: recursed on non-recursive mutex so_snd @ 
/syzkaller/managers/i386/kernel/sys/modules/tcp/rack/../../../netinet/tcp_stacks/rack.c:10632

Still, I have a non-debug update to main building and will
likely do a debug build as well. llvm is rebuilding, so
the builds will take a notable time.

> Thanks for isolating this, rick
> ps: Co-incidentally, I've been thinking of buying an RBPi4 as a toy.

I'll warn that the primary "small arm" development/support
folk(s) do not work on the RPi*'s these days, beyond
committing what others provide and the like.




===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Mark Millard via freebsd-stable]

On 2021-May-21, at 09:00, Rick Macklem  wrote:

> Mark Millard wrote:
>> On 2021-May-20, at 22:19, Rick Macklem  wrote:
> [stuff snipped]
>>> ps: I do not think that r367492 could cause this, but it would be
>>>nice if you try a kernel with the r367492 patch reverted.
>>>It is currently in all of releng13, stable13 and main, although
>>>the patch to fix this is was just reviewed and may hit main soon.
>> 
>> Do you want a debug kernel to be used? Do you have a preference
>> for main vs. stable/13 vs. release/13.0.0 based? Is it okay to
>> stick to the base version things are now based on --or do you
>> want me to update to more recent? (That last only applies if
>> main or stable/13 is to be put to use.)
> Well, it sounds like you've isolated it to the genet interface.
> Good sluething.
> Unfortunately, NFS is only as good as the network fabric under it.
> However, it's usually hangs or poor performance. Except maybe
> for the readdir issue that Jason Bacon reported and resolved via
> an upgrade, this is a first.
> --> In the old days, I would have expected IP checksums to catch
>   this, but I'm guessing the hardware/net driver are doing them
>   these days?

Well, why is it that ls -R, find, and diff -r all get file
name problems via genet0 but diff -r gets no problems
comparing the content of files that it does match up (the
vast majority)? Any clue how could the problems possibly
be unique to the handling of file names/paths? Does it
suggest anything else to look into for getting some more
potentially useful evidence?

I'll note that netstat -I ue0 -d and netstat -I genet0 -d
do not report changes in Ierrs or Idrop in a before vs.
after failures comparison. (There may be better figures
to look at for all I know.)

I tried "ifconfig genet0 -rxcsum -rxcsum -rxcsum6 -txcsum6"
and got no obvious change in behavior.

> W.r.t. reverting r367492...the patch to replace r367492 was just
> committed to "main" by rscheff@ with a two week MFC, so it
> should be in stable/13 soon. Not sure if an errata can be done
> for it for releng13.0?

That update is reported to be causing "rack" related panics:

https://lists.freebsd.org/pipermail/dev-commits-src-main/2021-May/004440.html

reports (via links):

panic: _mtx_lock_sleep: recursed on non-recursive mutex so_snd @ 
/syzkaller/managers/i386/kernel/sys/modules/tcp/rack/../../../netinet/tcp_stacks/rack.c:10632

Still, I have a non-debug update to main building and will
likely do a debug build as well. llvm is rebuilding, so
the builds will take a notable time.

> Thanks for isolating this, rick
> ps: Co-incidentally, I've been thinking of buying an RBPi4 as a toy.

I'll warn that the primary "small arm" development/support
folk(s) do not work on the RPi*'s these days, beyond
committing what others provide and the like.




===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Rick Macklem]

Mark Millard wrote:
>On 2021-May-20, at 22:19, Rick Macklem  wrote:
[stuff snipped]
>> ps: I do not think that r367492 could cause this, but it would be
>> nice if you try a kernel with the r367492 patch reverted.
>> It is currently in all of releng13, stable13 and main, although
>> the patch to fix this is was just reviewed and may hit main soon.
>
>Do you want a debug kernel to be used? Do you have a preference
>for main vs. stable/13 vs. release/13.0.0 based? Is it okay to
>stick to the base version things are now based on --or do you
>want me to update to more recent? (That last only applies if
>main or stable/13 is to be put to use.)
Well, it sounds like you've isolated it to the genet interface.
Good sluething.
Unfortunately, NFS is only as good as the network fabric under it.
However, it's usually hangs or poor performance. Except maybe
for the readdir issue that Jason Bacon reported and resolved via
an upgrade, this is a first.
--> In the old days, I would have expected IP checksums to catch
   this, but I'm guessing the hardware/net driver are doing them
   these days?

W.r.t. reverting r367492...the patch to replace r367492 was just
committed to "main" by rscheff@ with a two week MFC, so it
should be in stable/13 soon. Not sure if an errata can be done
for it for releng13.0?

Thanks for isolating this, rick
ps: Co-incidentally, I've been thinking of buying an RBPi4 as a toy.
> . . . old history deleted . . .

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context) [RPi4B genet0 involved in problem]

[Mark Millard via freebsd-stable]

[Looks like the RPi4B genet0 handling is involved.]

On 2021-May-20, at 22:56, Mark Millard  wrote:
> 
> On 2021-May-20, at 22:19, Rick Macklem  wrote:
> 
>> Ok, so it isn't related to "soft".
>> I am wondering if it is something specific to what
>> "diff -r" does?
>> 
>> Could you try:
>> # cd /usr/ports
>> # ls -R > /tmp/x
>> # cd /mnt
>> # ls -R > /tmp/y
>> # cd /tmp
>> # diff -u -p x y
>> --> To see if "ls -R" finds any difference?
>> 
> 
> # diff -u -p x y 
> --- x   2021-05-20 22:35:48.021663000 -0700
> +++ y   2021-05-20 22:39:03.691936000 -0700
> @@ -227209,10 +227209,10 @@ 
> patch-chrome_browser_background_background__mode__mana
> patch-chrome_browser_background_background__mode__optimizer.cc
> patch-chrome_browser_browser__resources.grd
> patch-chrome_browser_browsing__data_chrome__browsing__data__remover__delegate.cc
> +patch-chrome_browser_chrome__browser
> patch-chrome_browser_chrome__browser__interface__binders.cc
> patch-chrome_browser_chrome__browser__main.cc
> patch-chrome_browser_chrome__browser__main__linux.cc
> -patch-chrome_browser_chrome__browser__main__posix.cc
> patch-chrome_browser_chrome__content__browser__client.cc
> patch-chrome_browser_chrome__content__browser__client.h
> patch-chrome_browser_crash__upload__list_crash__upload__list.cc
> 
> # find /usr/ports/ -name 'patch-chrome_browser_chrome__browser*' -print | more
> /usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main__linux.cc
> /usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main.cc
> /usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main__posix.cc
> /usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__interface__binders.cc
> /usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main__posix.cc
> /usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main.cc
> /usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main__linux.cc
> /usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__interface__binders.cc
> 
> find /mnt/ -name 'patch-chrome_browser_chrome__browser*' -print | more
> /mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main__linux.cc
> /mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main.cc
> /mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main__posix.cc
> /mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__interface__binders.cc
> /mnt/www/chromium/files/patch-chrome_browser_chrome__browser
> /mnt/www/chromium/files/patch-chrome_browser_chrome__browser__main.cc
> /mnt/www/chromium/files/patch-chrome_browser_chrome__browser__main__linux.cc
> /mnt/www/chromium/files/patch-chrome_browser_chrome__browser__interface__binders.cc
> 
> So: patch-chrome_browser_chrome__browser appears to be a
> truncated: patch-chrome_browser_chrome__browser__main__posix.cc
> file name and find also gets the same oddity.
> 
> (Note: This had /usr/ports in a main context and /mnt/
> referring to a release/13.0.0 context.)
> 
>> ps: I do not think that r367492 could cause this, but it would be
>>nice if you try a kernel with the r367492 patch reverted.
>>It is currently in all of releng13, stable13 and main, although
>>the patch to fix this is was just reviewed and may hit main soon.
> 
> Do you want a debug kernel to be used? Do you have a preference
> for main vs. stable/13 vs. release/13.0.0 based? Is it okay to
> stick to the base version things are now based on --or do you
> want me to update to more recent? (That last only applies if
> main or stable/13 is to be put to use.)
> 
>> . . . old history deleted . . .

I reversed the roles of the faster vs. somewhat slower
machine and so far my diff -r attempts for this found
no differences. The machines were using different types
of EtherNet devices.

So I've substituted a different EtherNet device onto
the slower machine: the same type of USB3 EtherNet
device in use on the faster machine (instead of
using the RPi4B's builtin EtherNet). So the below
testing is with both machines having a:

ugen0.6:  at usbus0
ure0 on uhub0
ure0:  on usbus0
miibus1:  on ure0
rgephy0:  PHY 0 on miibus1
rgephy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 
1000baseT-FDX-master, auto

in use.

I rebooted with this connected instead of the genet0
interface.

Mounting the slower machine's /usr/ports/ as /mnt/ from the faster machine:
No differences found by diff -r this way (expected result).

Mounting the faster machine's /usr/ports/ as /mnt/ from the slower machine:
No differences found by diff -r this way (expected result).

Doing diff -r's from both sides at the same time:
No differences found by diff -r this way (expected result).


So it looks like genet0 or its supporting software
is contributing to the problems that I had reported.

It is interesting that there were no examples of the
content of files reporting a mismatch, just some file
names/paths not 

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Mark Millard via freebsd-stable]

On 2021-May-20, at 22:19, Rick Macklem  wrote:

> Ok, so it isn't related to "soft".
> I am wondering if it is something specific to what
> "diff -r" does?
> 
> Could you try:
> # cd /usr/ports
> # ls -R > /tmp/x
> # cd /mnt
> # ls -R > /tmp/y
> # cd /tmp
> # diff -u -p x y
> --> To see if "ls -R" finds any difference?
> 

# diff -u -p x y 
--- x   2021-05-20 22:35:48.021663000 -0700
+++ y   2021-05-20 22:39:03.691936000 -0700
@@ -227209,10 +227209,10 @@ 
patch-chrome_browser_background_background__mode__mana
 patch-chrome_browser_background_background__mode__optimizer.cc
 patch-chrome_browser_browser__resources.grd
 
patch-chrome_browser_browsing__data_chrome__browsing__data__remover__delegate.cc
+patch-chrome_browser_chrome__browser
 patch-chrome_browser_chrome__browser__interface__binders.cc
 patch-chrome_browser_chrome__browser__main.cc
 patch-chrome_browser_chrome__browser__main__linux.cc
-patch-chrome_browser_chrome__browser__main__posix.cc
 patch-chrome_browser_chrome__content__browser__client.cc
 patch-chrome_browser_chrome__content__browser__client.h
 patch-chrome_browser_crash__upload__list_crash__upload__list.cc

# find /usr/ports/ -name 'patch-chrome_browser_chrome__browser*' -print | more
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main__linux.cc
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main.cc
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__main__posix.cc
/usr/ports/devel/electron12/files/patch-chrome_browser_chrome__browser__interface__binders.cc
/usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main__posix.cc
/usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main.cc
/usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__main__linux.cc
/usr/ports/www/chromium/files/patch-chrome_browser_chrome__browser__interface__binders.cc

 find /mnt/ -name 'patch-chrome_browser_chrome__browser*' -print | more
/mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main__linux.cc
/mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main.cc
/mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__main__posix.cc
/mnt/devel/electron12/files/patch-chrome_browser_chrome__browser__interface__binders.cc
/mnt/www/chromium/files/patch-chrome_browser_chrome__browser
/mnt/www/chromium/files/patch-chrome_browser_chrome__browser__main.cc
/mnt/www/chromium/files/patch-chrome_browser_chrome__browser__main__linux.cc
/mnt/www/chromium/files/patch-chrome_browser_chrome__browser__interface__binders.cc

So: patch-chrome_browser_chrome__browser appears to be a
truncated: patch-chrome_browser_chrome__browser__main__posix.cc
file name and find also gets the same oddity.

(Note: This had /usr/ports in a main context and /mnt/
referring to a release/13.0.0 context.)

> ps: I do not think that r367492 could cause this, but it would be
> nice if you try a kernel with the r367492 patch reverted.
> It is currently in all of releng13, stable13 and main, although
> the patch to fix this is was just reviewed and may hit main soon.

Do you want a debug kernel to be used? Do you have a preference
for main vs. stable/13 vs. release/13.0.0 based? Is it okay to
stick to the base version things are now based on --or do you
want me to update to more recent? (That last only applies if
main or stable/13 is to be put to use.)

> . . . old history deleted . . .

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Rick Macklem]

Ok, so it isn't related to "soft".
I am wondering if it is something specific to what
"diff -r" does?

Could you try:
# cd /usr/ports
# ls -R > /tmp/x
# cd /mnt
# ls -R > /tmp/y
# cd /tmp
# diff -u -p x y
--> To see if "ls -R" finds any difference?

rick
ps: I do not think that r367492 could cause this, but it would be
 nice if you try a kernel with the r367492 patch reverted.
 It is currently in all of releng13, stable13 and main, although
 the patch to fix this is was just reviewed and may hit main soon.



From: Mark Millard 
Sent: Friday, May 21, 2021 12:40 AM
To: Rick Macklem
Cc: FreeBSD-STABLE Mailing List
Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in 
a zfs file systems context)

CAUTION: This email originated from outside of the University of Guelph. Do not 
click links or open attachments unless you recognize the sender and know the 
content is safe. If in doubt, forward suspicious emails to ith...@uoguelph.ca


[main test example and main/releng/13 mixed example]

On 2021-May-20, at 20:36, Mark Millard  wrote:

> [stable/13 test: example ends up being odder. That might
> allow eliminating some potential alternatives.]
>
> On 2021-May-20, at 19:38, Mark Millard  wrote:
>>
>> On 2021-May-20, at 18:09, Rick Macklem  wrote:
>>>
>>> Oh, one additional thing that I'll dare to top post...
>>> r367492 broke the TCP upcalls that the NFS server uses, such
>>> that intermittent hangs of NFS mounts to FreeBSD13 servers can occur.
>>> This has not yet been resolved in "main" etc and could explain
>>> why an RPC could time out for a soft mount.
>>
>> See later notes that I added: soft mount is not required
>> to see the problem.
>>
>>> You can revert the patch in r367492 to avoid the problem.
>>
>> If I understand right, you are indicating that this would
>> not apply to the non-soft mount case that I got.
>>
>>> Disabling TSO, LRO are also de-facto standard things to do when
>>> you observe weird NFS  behaviour, because they are often broken
>>> in various network device drivers.
>>
>> I'll have to figure out how to experiment with such. Things
>> are at defaults rather generally on the systems. I'm not
>> literate in the subject areas.
>>
>> I'm the only user of the machines and network. It is not
>> outward facing. It is a rather small EtherNet network.
>>
>>> rick
>>>
>>> ____
>>> From: owner-freebsd-sta...@freebsd.org  
>>> on behalf of Rick Macklem 
>>> Sent: Thursday, May 20, 2021 8:55 PM
>>> To: FreeBSD-STABLE Mailing List; Mark Millard
>>> Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs 
>>> (in a zfs file systems context)
>>>
>>> Mark Millard wrote:
>>>> [I warn that I'm a fairly minimal user of NFS
>>>> mounts, not knowing all that much. I'm mostly
>>>> reporting this in case it ends up as evidence
>>>> via eventually matching up with others observing
>>>> possibly related oddities.]
>>>>
>>>> I got the following odd sequence (that I've
>>>> mixed notes into). It involved a diff -r over NFS
>>>> showing differences (files missing) and then a
>>>> later diff finding matches for the same files,
>>>> no file system changes made on either machine.
>>>> I'm unable to reproduce the oddity on demand.
>>>>
>>>> Note: A larger scope diff -r originally returned the
>>>> below as well, but doing the narrower diff -r did
>>>> repeat the result and that is what I show. (I
>>>> make no use of devel/ice .)
>>>>
>>>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>>>> Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD
>> . . .
>>>> Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py
>>>>
>>>> Note: The above was not expected. So I tried:
>>>>
>>>> # ls -Tld /mnt/devel/ice/files/*
>>>> -rw-r--r--  1 root  wheel   755 Apr 21 21:07:54 2021 
>>>> /mnt/devel/ice/files/Make.rules.FreeBSD
>> . . .
>>>> -rw-r--r--  1 root  wheel  2588 Apr 21 21:07:54 2021 
>>>> /mnt/devel/ice/files/patch-scripts-TestUtil.py
>>>>
>>>> Note: So that indicated that the files were there on the
>>>> machine that /mnt references. So attempting the original
>>>> diff -r again:
>>>>
>>>> # di

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Mark Millard via freebsd-stable]

[Direct drive connection to machine: no problem.]

On 2021-May-20, at 21:40, Mark Millard  wrote:

> [main test example and main/releng/13 mixed example]
> 
> On 2021-May-20, at 20:36, Mark Millard  wrote:
> 
>> [stable/13 test: example ends up being odder. That might
>> allow eliminating some potential alternatives.]
>> 
>> On 2021-May-20, at 19:38, Mark Millard  wrote:
>>> 
>>> On 2021-May-20, at 18:09, Rick Macklem  wrote:
>>>> 
>>>> Oh, one additional thing that I'll dare to top post...
>>>> r367492 broke the TCP upcalls that the NFS server uses, such
>>>> that intermittent hangs of NFS mounts to FreeBSD13 servers can occur.
>>>> This has not yet been resolved in "main" etc and could explain
>>>> why an RPC could time out for a soft mount.
>>> 
>>> See later notes that I added: soft mount is not required
>>> to see the problem.
>>> 
>>>> You can revert the patch in r367492 to avoid the problem.
>>> 
>>> If I understand right, you are indicating that this would
>>> not apply to the non-soft mount case that I got.
>>> 
>>>> Disabling TSO, LRO are also de-facto standard things to do when
>>>> you observe weird NFS  behaviour, because they are often broken
>>>> in various network device drivers.
>>> 
>>> I'll have to figure out how to experiment with such. Things
>>> are at defaults rather generally on the systems. I'm not
>>> literate in the subject areas.
>>> 
>>> I'm the only user of the machines and network. It is not
>>> outward facing. It is a rather small EtherNet network.
>>> 
>>>> rick
>>>> 
>>>> 
>>>> From: owner-freebsd-sta...@freebsd.org  
>>>> on behalf of Rick Macklem 
>>>> Sent: Thursday, May 20, 2021 8:55 PM
>>>> To: FreeBSD-STABLE Mailing List; Mark Millard
>>>> Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs 
>>>> (in a zfs file systems context)
>>>> 
>>>> Mark Millard wrote:
>>>>> [I warn that I'm a fairly minimal user of NFS
>>>>> mounts, not knowing all that much. I'm mostly
>>>>> reporting this in case it ends up as evidence
>>>>> via eventually matching up with others observing
>>>>> possibly related oddities.]
>>>>> 
>>>>> I got the following odd sequence (that I've
>>>>> mixed notes into). It involved a diff -r over NFS
>>>>> showing differences (files missing) and then a
>>>>> later diff finding matches for the same files,
>>>>> no file system changes made on either machine.
>>>>> I'm unable to reproduce the oddity on demand.
>>>>> 
>>>>> Note: A larger scope diff -r originally returned the
>>>>> below as well, but doing the narrower diff -r did
>>>>> repeat the result and that is what I show. (I
>>>>> make no use of devel/ice .)
>>>>> 
>>>>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>>>>> Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD
>>> . . .
>>>>> Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py
>>>>> 
>>>>> Note: The above was not expected. So I tried:
>>>>> 
>>>>> # ls -Tld /mnt/devel/ice/files/*
>>>>> -rw-r--r--  1 root  wheel   755 Apr 21 21:07:54 2021 
>>>>> /mnt/devel/ice/files/Make.rules.FreeBSD
>>> . . .
>>>>> -rw-r--r--  1 root  wheel  2588 Apr 21 21:07:54 2021 
>>>>> /mnt/devel/ice/files/patch-scripts-TestUtil.py
>>>>> 
>>>>> Note: So that indicated that the files were there on the
>>>>> machine that /mnt references. So attempting the original
>>>>> diff -r again:
>>>>> 
>>>>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>>>>> #
>>>>> 
>>>>> (Empty difference.)
>>>>> 
>>>>> Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*"
>>>>> the odd result of the diff -r no longer happened: no
>>>>> differences reported.
>>>>> 
>>>>> 
>>>>> 
>>>>> For reference (both machines reported):
>>>>> 
>>>>> . . .
>>>>> The original mount command w

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Mark Millard via freebsd-stable]

[main test example and main/releng/13 mixed example]

On 2021-May-20, at 20:36, Mark Millard  wrote:

> [stable/13 test: example ends up being odder. That might
> allow eliminating some potential alternatives.]
> 
> On 2021-May-20, at 19:38, Mark Millard  wrote:
>> 
>> On 2021-May-20, at 18:09, Rick Macklem  wrote:
>>> 
>>> Oh, one additional thing that I'll dare to top post...
>>> r367492 broke the TCP upcalls that the NFS server uses, such
>>> that intermittent hangs of NFS mounts to FreeBSD13 servers can occur.
>>> This has not yet been resolved in "main" etc and could explain
>>> why an RPC could time out for a soft mount.
>> 
>> See later notes that I added: soft mount is not required
>> to see the problem.
>> 
>>> You can revert the patch in r367492 to avoid the problem.
>> 
>> If I understand right, you are indicating that this would
>> not apply to the non-soft mount case that I got.
>> 
>>> Disabling TSO, LRO are also de-facto standard things to do when
>>> you observe weird NFS  behaviour, because they are often broken
>>> in various network device drivers.
>> 
>> I'll have to figure out how to experiment with such. Things
>> are at defaults rather generally on the systems. I'm not
>> literate in the subject areas.
>> 
>> I'm the only user of the machines and network. It is not
>> outward facing. It is a rather small EtherNet network.
>> 
>>> rick
>>> 
>>> 
>>> From: owner-freebsd-sta...@freebsd.org  
>>> on behalf of Rick Macklem 
>>> Sent: Thursday, May 20, 2021 8:55 PM
>>> To: FreeBSD-STABLE Mailing List; Mark Millard
>>> Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs 
>>> (in a zfs file systems context)
>>> 
>>> Mark Millard wrote:
>>>> [I warn that I'm a fairly minimal user of NFS
>>>> mounts, not knowing all that much. I'm mostly
>>>> reporting this in case it ends up as evidence
>>>> via eventually matching up with others observing
>>>> possibly related oddities.]
>>>> 
>>>> I got the following odd sequence (that I've
>>>> mixed notes into). It involved a diff -r over NFS
>>>> showing differences (files missing) and then a
>>>> later diff finding matches for the same files,
>>>> no file system changes made on either machine.
>>>> I'm unable to reproduce the oddity on demand.
>>>> 
>>>> Note: A larger scope diff -r originally returned the
>>>> below as well, but doing the narrower diff -r did
>>>> repeat the result and that is what I show. (I
>>>> make no use of devel/ice .)
>>>> 
>>>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>>>> Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD
>> . . .
>>>> Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py
>>>> 
>>>> Note: The above was not expected. So I tried:
>>>> 
>>>> # ls -Tld /mnt/devel/ice/files/*
>>>> -rw-r--r--  1 root  wheel   755 Apr 21 21:07:54 2021 
>>>> /mnt/devel/ice/files/Make.rules.FreeBSD
>> . . .
>>>> -rw-r--r--  1 root  wheel  2588 Apr 21 21:07:54 2021 
>>>> /mnt/devel/ice/files/patch-scripts-TestUtil.py
>>>> 
>>>> Note: So that indicated that the files were there on the
>>>> machine that /mnt references. So attempting the original
>>>> diff -r again:
>>>> 
>>>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>>>> #
>>>> 
>>>> (Empty difference.)
>>>> 
>>>> Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*"
>>>> the odd result of the diff -r no longer happened: no
>>>> differences reported.
>>>> 
>>>> 
>>>> 
>>>> For reference (both machines reported):
>>>> 
>>>> . . .
>>>> The original mount command was on CA72_16Gp_ZFS:
>>>> 
>>>> # mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/
>>> The likely explanation for this is your use of a "soft" mount.
>>> - If the NFS server is slow to respond or there is a temporary network 
>>> issue,
>>> the RPC request can time out and then the
>>> syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including the
>>>  readdir(3) libc functions ex

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Mark Millard via freebsd-stable]

[stable/13 test: example ends up being odder. That might
allow eliminating some potential alternatives.]

On 2021-May-20, at 19:38, Mark Millard  wrote:
> 
> On 2021-May-20, at 18:09, Rick Macklem  wrote:
>> 
>> Oh, one additional thing that I'll dare to top post...
>> r367492 broke the TCP upcalls that the NFS server uses, such
>> that intermittent hangs of NFS mounts to FreeBSD13 servers can occur.
>> This has not yet been resolved in "main" etc and could explain
>> why an RPC could time out for a soft mount.
> 
> See later notes that I added: soft mount is not required
> to see the problem.
> 
>> You can revert the patch in r367492 to avoid the problem.
> 
> If I understand right, you are indicating that this would
> not apply to the non-soft mount case that I got.
> 
>> Disabling TSO, LRO are also de-facto standard things to do when
>> you observe weird NFS  behaviour, because they are often broken
>> in various network device drivers.
> 
> I'll have to figure out how to experiment with such. Things
> are at defaults rather generally on the systems. I'm not
> literate in the subject areas.
> 
> I'm the only user of the machines and network. It is not
> outward facing. It is a rather small EtherNet network.
> 
>> rick
>> 
>> 
>> From: owner-freebsd-sta...@freebsd.org  on 
>> behalf of Rick Macklem 
>> Sent: Thursday, May 20, 2021 8:55 PM
>> To: FreeBSD-STABLE Mailing List; Mark Millard
>> Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs 
>> (in a zfs file systems context)
>> 
>> Mark Millard wrote:
>>> [I warn that I'm a fairly minimal user of NFS
>>> mounts, not knowing all that much. I'm mostly
>>> reporting this in case it ends up as evidence
>>> via eventually matching up with others observing
>>> possibly related oddities.]
>>> 
>>> I got the following odd sequence (that I've
>>> mixed notes into). It involved a diff -r over NFS
>>> showing differences (files missing) and then a
>>> later diff finding matches for the same files,
>>> no file system changes made on either machine.
>>> I'm unable to reproduce the oddity on demand.
>>> 
>>> Note: A larger scope diff -r originally returned the
>>> below as well, but doing the narrower diff -r did
>>> repeat the result and that is what I show. (I
>>> make no use of devel/ice .)
>>> 
>>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>>> Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD
> . . .
>>> Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py
>>> 
>>> Note: The above was not expected. So I tried:
>>> 
>>> # ls -Tld /mnt/devel/ice/files/*
>>> -rw-r--r--  1 root  wheel   755 Apr 21 21:07:54 2021 
>>> /mnt/devel/ice/files/Make.rules.FreeBSD
> . . .
>>> -rw-r--r--  1 root  wheel  2588 Apr 21 21:07:54 2021 
>>> /mnt/devel/ice/files/patch-scripts-TestUtil.py
>>> 
>>> Note: So that indicated that the files were there on the
>>> machine that /mnt references. So attempting the original
>>> diff -r again:
>>> 
>>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>>> #
>>> 
>>> (Empty difference.)
>>> 
>>> Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*"
>>> the odd result of the diff -r no longer happened: no
>>> differences reported.
>>> 
>>> 
>>> 
>>> For reference (both machines reported):
>>> 
>>> . . .
>>> The original mount command was on CA72_16Gp_ZFS:
>>> 
>>> # mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/
>> The likely explanation for this is your use of a "soft" mount.
>> - If the NFS server is slow to respond or there is a temporary network issue,
>>  the RPC request can time out and then the
>>  syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including the
>>   readdir(3) libc functions expect syscalls to fail this way...
>>   Then the cached directory is messed up.
>>   Doing the "ls" read the directory again and fixed the problem.
>> 
>> Try to reproduce it for a mount without the "soft" option.
>> (If a mount point is hung, due to an unresponsive server "umount -N /mnt"
>> can usually get rid of it.)
>> Personally, I thought "soft" was a bad idea when Sun introduced it in NFS in 
>> 1985
>> and I still feel that 

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Mark Millard via freebsd-stable]

> On 2021-May-20, at 18:09, Rick Macklem  wrote:
> 
> Oh, one additional thing that I'll dare to top post...
> r367492 broke the TCP upcalls that the NFS server uses, such
> that intermittent hangs of NFS mounts to FreeBSD13 servers can occur.
> This has not yet been resolved in "main" etc and could explain
> why an RPC could time out for a soft mount.

See later notes that I added: soft mount is not required
to see the problem.

> You can revert the patch in r367492 to avoid the problem.

If I understand right, you are indicating that this would
not apply to the non-soft mount case that I got.

> Disabling TSO, LRO are also de-facto standard things to do when
> you observe weird NFS  behaviour, because they are often broken
> in various network device drivers.

I'll have to figure out how to experiment with such. Things
are at defaults rather generally on the systems. I'm not
literate in the subject areas.

I'm the only user of the machines and network. It is not
outward facing. It is a rather small EtherNet network.

> rick
> 
> 
> From: owner-freebsd-sta...@freebsd.org  on 
> behalf of Rick Macklem 
> Sent: Thursday, May 20, 2021 8:55 PM
> To: FreeBSD-STABLE Mailing List; Mark Millard
> Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs 
> (in a zfs file systems context)
> 
> Mark Millard wrote:
>> [I warn that I'm a fairly minimal user of NFS
>> mounts, not knowing all that much. I'm mostly
>> reporting this in case it ends up as evidence
>> via eventually matching up with others observing
>> possibly related oddities.]
>> 
>> I got the following odd sequence (that I've
>> mixed notes into). It involved a diff -r over NFS
>> showing differences (files missing) and then a
>> later diff finding matches for the same files,
>> no file system changes made on either machine.
>> I'm unable to reproduce the oddity on demand.
>> 
>> Note: A larger scope diff -r originally returned the
>> below as well, but doing the narrower diff -r did
>> repeat the result and that is what I show. (I
>> make no use of devel/ice .)
>> 
>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>> Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD
. . .
>> Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py
>> 
>> Note: The above was not expected. So I tried:
>> 
>> # ls -Tld /mnt/devel/ice/files/*
>> -rw-r--r--  1 root  wheel   755 Apr 21 21:07:54 2021 
>> /mnt/devel/ice/files/Make.rules.FreeBSD
. . .
>> -rw-r--r--  1 root  wheel  2588 Apr 21 21:07:54 2021 
>> /mnt/devel/ice/files/patch-scripts-TestUtil.py
>> 
>> Note: So that indicated that the files were there on the
>> machine that /mnt references. So attempting the original
>> diff -r again:
>> 
>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>> #
>> 
>> (Empty difference.)
>> 
>> Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*"
>> the odd result of the diff -r no longer happened: no
>> differences reported.
>> 
>> 
>> 
>> For reference (both machines reported):
>> 
>> . . .
>> The original mount command was on CA72_16Gp_ZFS:
>> 
>> # mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/
> The likely explanation for this is your use of a "soft" mount.
> - If the NFS server is slow to respond or there is a temporary network issue,
>   the RPC request can time out and then the
>   syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including the
>readdir(3) libc functions expect syscalls to fail this way...
>Then the cached directory is messed up.
>Doing the "ls" read the directory again and fixed the problem.
> 
> Try to reproduce it for a mount without the "soft" option.
> (If a mount point is hung, due to an unresponsive server "umount -N /mnt"
> can usually get rid of it.)
> Personally, I thought "soft" was a bad idea when Sun introduced it in NFS in 
> 1985
> and I still feel that way.
> --> If you can reproduce it without "soft" then I can't explain it.
>  To be honest, the directory reading/caching code in the NFSv3 client
>  hasn't changed significantly in literally decades, as far as I can 
> remember.

Well . . . trying an even wider scope diff than
the original . . .

# umount /mnt/
# mount -onoatime 192.168.1.170:/usr/ports/ /mnt/
# diff -r /usr/ports/ /mnt/ | more
Only in /mnt/databases/mongodb42/files/aarch64: patch-src_third__party_mozjs-60_
Only in /usr/ports/databases/mongodb42/files/aarch64: 
patch-src_th

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Rick Macklem]

Oh, one additional thing that I'll dare to top post...
r367492 broke the TCP upcalls that the NFS server uses, such
that intermittent hangs of NFS mounts to FreeBSD13 servers can occur.
This has not yet been resolved in "main" etc and could explain
why an RPC could time out for a soft mount.

You can revert the patch in r367492 to avoid the problem.

Disabling TSO, LRO are also de-facto standard things to do when
you observe weird NFS  behaviour, because they are often broken
in various network device drivers.

rick


From: owner-freebsd-sta...@freebsd.org  on 
behalf of Rick Macklem 
Sent: Thursday, May 20, 2021 8:55 PM
To: FreeBSD-STABLE Mailing List; Mark Millard
Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in 
a zfs file systems context)

Mark Millard wrote:
>[I warn that I'm a fairly minimal user of NFS
>mounts, not knowing all that much. I'm mostly
>reporting this in case it ends up as evidence
>via eventually matching up with others observing
>possibly related oddities.]
>
>I got the following odd sequence (that I've
>mixed notes into). It involved a diff -r over NFS
>showing differences (files missing) and then a
>later diff finding matches for the same files,
>no file system changes made on either machine.
>I'm unable to reproduce the oddity on demand.
>
>Note: A larger scope diff -r originally returned the
>below as well, but doing the narrower diff -r did
>repeat the result and that is what I show. (I
>make no use of devel/ice .)
>
># diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD
>Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules
>Only in /usr/ports/devel/ice/files: patch-cpp-Makefile
>Only in /usr/ports/devel/ice/files: patch-cpp-allTests.py
>Only in /usr/ports/devel/ice/files: patch-cpp-config-Make.rules
>Only in /usr/ports/devel/ice/files: patch-cpp-include-Ice-FactoryTableInit.h
>Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-Config.h
>Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-ScannerConfig.h
>Only in /usr/ports/devel/ice/files: 
>patch-cpp-src-Glacier2CryptPermissionsVerifier-CryptPermissionsVerifierI.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-src-Ice-ProxyFactory.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-PluginFacadeI.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-RegistryI.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceSSL-Makefile
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Glacier2-ssl-Server.cpp
>Only in /usr/ports/devel/ice/files: 
>patch-cpp-test-Glacier2-staticFiltering-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-info-AllTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-metrics-AllTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-objects-Makefile
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-properties-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-admin-run.py
>Only in /usr/ports/devel/ice/files: 
>patch-cpp-test-IceGrid-deployer-AllTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-Makefile
>Only in /usr/ports/devel/ice/files: 
>patch-cpp-test-IceGrid-deployer-application.xml
>Only in /usr/ports/devel/ice/files: 
>patch-cpp-test-IceGrid-distribution-AllTests.cpp
>Only in /usr/ports/devel/ice/files: 
>patch-cpp-test-IceGrid-distribution-application.xml
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-session-run.py
>Only in /usr/ports/devel/ice/files: 
>patch-cpp-test-IceSSL-configuration-AllTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceSSL-configuration-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-headers-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-unicodePaths-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-include-TestCommon.h
>Only in /usr/ports/devel/ice/files: patch-php-Makefile
>Only in /usr/ports/devel/ice/files: patch-php-config-Make.rules.php
>Only in /usr/ports/devel/ice/files: patch-php-lib-Makefile
>Only in /usr/ports/devel/ice/files: patch-python-Makefile
>Only in /usr/ports/devel/ice/files: patch-python-config-Make.rules
>Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.cpp
>Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.h
>Only in /usr/ports/devel/ice/files: patch-python-python-Makefile
>Only in /usr/ports/devel/ice/files: patch-python-test-Ice-info-AllTests.py
>Only in /usr/ports/devel/ice/files: patch-python-test-Ice-properties-run.py
>Only in /usr/ports/devel/ice/files: patch-python-test

Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Rick Macklem]

t;Note: So that indicated that the files were there on the
>machine that /mnt references. So attempting the original
>diff -r again:
>
># diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>#
>
>(Empty difference.)
>
>Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*"
>the odd result of the diff -r no longer happened: no
>differences reported.
>
>
>
>For reference (both machines reported):
>
># uname -apKU
>FreeBSD CA72_16Gp_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 
>releng/13.0-n244733-ea31abc261ff-dirty: Thu Apr 29 >21:53:20 PDT 2021 
>root@CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R->src/arm64.aarch64/sys/GENERIC-NODBG-CA72
>  arm64 aarch64 1300139 1300139
>
># ~/fbsd-based-on-what-commit.sh
>branch: releng/13.0
>merge-base: ea31abc261ffc01b6ff5671bffb15cf910a07f4b
>merge-base: CommitDate: 2021-04-09 00:14:30 +
>ea31abc261ff (HEAD -> releng/13.0, tag: release/13.0.0, freebsd/releng/13.0) 
>13.0: update to RELEASE
>n244733 (--first-parent --count for merge-base)
>
># uname -apKU
>FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 
>releng/13.0-n244733-ea31abc261ff-dirty: Thu Apr 29 >21:53:20 PDT 2021 
>root@CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R->src/arm64.aarch64/sys/GENERIC-NODBG-CA72
>  arm64 aarch64 1300139 1300139
>
># ~/fbsd-based-on-what-commit.sh
>branch: releng/13.0
>merge-base: ea31abc261ffc01b6ff5671bffb15cf910a07f4b
>merge-base: CommitDate: 2021-04-09 00:14:30 +
>ea31abc261ff (HEAD -> releng/13.0, tag: release/13.0.0, freebsd/releng/13.0) 
>13.0: update to RELEASE
>n244733 (--first-parent --count for merge-base)
>
>From zfs list commands (one machine per line shown):
>
>zopt0/usr/ports   2.13G   236G 2.13G  /usr/ports
>zroot/usr/ports   2.13G   113G 2.13G  /usr/ports
>
>I've no clue if ZFS is important to the odditity
>or not.
>
>The original mount command was on CA72_16Gp_ZFS:
>
># mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/
The likely explanation for this is your use of a "soft" mount.
- If the NFS server is slow to respond or there is a temporary network issue,
   the RPC request can time out and then the
   syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including the
readdir(3) libc functions expect syscalls to fail this way...
Then the cached directory is messed up.
Doing the "ls" read the directory again and fixed the problem.

Try to reproduce it for a mount without the "soft" option.
(If a mount point is hung, due to an unresponsive server "umount -N /mnt"
 can usually get rid of it.)
Personally, I thought "soft" was a bad idea when Sun introduced it in NFS in 
1985
and I still feel that way.
--> If you can reproduce it without "soft" then I can't explain it.
  To be honest, the directory reading/caching code in the NFSv3 client
  hasn't changed significantly in literally decades, as far as I can 
remember.

One additional thing to note is that cached directory contents are invalidated
when the directory's ctime changes. I am not sure how/if/when ZFS changes a
directory's ctime. However, if it was badly broken, I'd hear about this a lot.
(If the ZFS change to ZoL has changed its ctime handling, that might also 
explain it
 and I'll be hearing a lot more soon as FreeBSD13 becomes adopted. I never use 
ZFS and,
 as such, never test with it.)
--> For UFS, if you use mtime, directory caching does not work as well, which is
   why the client directory caching code uses ctime and not mtime to detect 
that
   a directory has changed and cached directory blocks need to be 
invalidated.

Jason Bacon did report a directory reading issue some months ago that never
quite got resolved, although I recall he said he couldn't reproduce it after a
system update, so he thought it was related to some local change he had made.
(I can't remember his email or I'd add him to the cc so he could remind me what
 his case was. I do recall it being somewhat reproducible and happened for both
 UFS and ZFS.)
The network is just a local EtherNet.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

[Mark Millard via freebsd-stable]

fbsd-based-on-what-commit.sh 
branch: releng/13.0
merge-base: ea31abc261ffc01b6ff5671bffb15cf910a07f4b
merge-base: CommitDate: 2021-04-09 00:14:30 +
ea31abc261ff (HEAD -> releng/13.0, tag: release/13.0.0, freebsd/releng/13.0) 
13.0: update to RELEASE
n244733 (--first-parent --count for merge-base)

# uname -apKU
FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 
releng/13.0-n244733-ea31abc261ff-dirty: Thu Apr 29 21:53:20 PDT 2021 
root@CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/sys/GENERIC-NODBG-CA72
  arm64 aarch64 1300139 1300139

# ~/fbsd-based-on-what-commit.sh 
branch: releng/13.0
merge-base: ea31abc261ffc01b6ff5671bffb15cf910a07f4b
merge-base: CommitDate: 2021-04-09 00:14:30 +
ea31abc261ff (HEAD -> releng/13.0, tag: release/13.0.0, freebsd/releng/13.0) 
13.0: update to RELEASE
n244733 (--first-parent --count for merge-base)

>From zfs list commands (one machine per line shown):

zopt0/usr/ports   2.13G   236G 2.13G  /usr/ports
zroot/usr/ports   2.13G   113G 2.13G  /usr/ports

I've no clue if ZFS is important to the odditity
or not.

The original mount command was on CA72_16Gp_ZFS:

# mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/

The network is just a local EtherNet.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Does FreeBSD 13 disable the VEV cache in ZFS ?

[Pete French]

> Could you check the values of the following sysctl variables:
>
> vfs.zfs.vdev.cache_size
> vfs.zfs.vdev.cache_bshift
> vfs.zfs.vdev.cache_max
> kstat.zfs.misc.vdev_cache_stats.misses
> kstat.zfs.misc.vdev_cache_stats.hits
> kstat.zfs.misc.vdev_cache_stats.delegations

As predicted, all zero, apart from bshift which is 16 and max which is 16384


> https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#zfs-vdev-cache-size
>
> Quote:
>
> Note: with the current ZFS code, the vdev cache is not helpful and in some
> cases actually harmful. Thus it is disabled by setting the 
> zfs_vdev_cache_size = 0

Ah, thats the thing I should have found msyelf I guess! :-) Actually, if smarter
people than me think that its not useful on OpenZFS, then I shall probably
leave it disabled. Generally people have a good reason for doing things.

I shall have a read,and thankyou for confirming this (and thankyou for
maintinaing zfs-stats! which is an invalauble tool I have found over the years)


-pete.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Does FreeBSD 13 disable the VEV cache in ZFS ?

[Stefan Esser]

Am 14.05.21 um 10:34 schrieb Pete French:
> 
> Am just upgrading my machiens, and have noticed an oddity.
> This is on a machine runnign 12.2
> 
> # zfs-stats -D
> 
> --------
> ZFS Subsystem Report  Fri May 14 08:30:50 2021
> 
> 
> VDEV Cache Summary:   88.31   m
>   Hit Ratio:  31.20%  27.55   m
>   Miss Ratio: 68.48%  60.47   m
>   Delegations:0.32%   284.97  k
> 
> 
> 
> 
> This is on a machine running 13.0
> 
> # zfs-stats -D
> 
> --------
> ZFS Subsystem Report  Fri May 14 08:32:18 2021
> 
> 
> VDEV cache is disabled
> 
> ----
> 
> Same config on both. So, is it really disabled, or is zfs-stats getting this
> wrong for some reason ? I cant find any refernbce to this in the release notes
> or by googling.

Hi Pete,

zfs-stats has last been modified by me to make it work with the sysctl names
changed by OpenZFS. (But I might have missed a few ...)

Could you check the values of the following sysctl variables:

vfs.zfs.vdev.cache_size
vfs.zfs.vdev.cache_bshift
vfs.zfs.vdev.cache_max
kstat.zfs.misc.vdev_cache_stats.misses
kstat.zfs.misc.vdev_cache_stats.hits
kstat.zfs.misc.vdev_cache_stats.delegations

On my system, vfs.zfs.vdev.cache_size is always 0 - and that results in the
message "VDEV cache is disabled" that you got, too. And I'd guess that the
kstat variables will all report 0, too (not surprising if the size is 0).


A quick web search reveals:

https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#zfs-vdev-cache-size

Quote:

Note: with the current ZFS code, the vdev cache is not helpful and in some
cases actually harmful. Thus it is disabled by setting the zfs_vdev_cache_size 
= 0


You may want to try setting "vfs.zfs.vdev.cache_size" (name changed, it used
to be cache.size instead of cache_size) in /boot/loader.conf and perform a
few performance tests. (I have not verified that the current implementation
actually uses and supports a value specified that way, though.)

Regards, STefan



OpenPGP_signature
Description: OpenPGP digital signature

Does FreeBSD 13 disable the VEV cache in ZFS ?

[Pete French]

Am just upgrading my machiens, and have noticed an oddity.
This is on a machine runnign 12.2

# zfs-stats -D


ZFS Subsystem ReportFri May 14 08:30:50 2021


VDEV Cache Summary: 88.31   m
Hit Ratio:  31.20%  27.55   m
Miss Ratio: 68.48%  60.47   m
Delegations:0.32%   284.97  k




This is on a machine running 13.0

# zfs-stats -D


ZFS Subsystem ReportFri May 14 08:32:18 2021


VDEV cache is disabled




Same config on both. So, is it really disabled, or is zfs-stats getting this
wrong for some reason ? I cant find any refernbce to this in the release notes
or by googling.

-pete.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Install of 13.0-RELEASE i386 with ZFS root hangs up

[Konstantin Belousov]

On Sat, May 08, 2021 at 06:33:02PM +0700, Eugene Grosbein wrote:
> 08.05.2021 2:52, Konstantin Belousov wrote:
> 
> > i386 kernel uses memory up to 24G since 13.0.
> > 
> > PAE only means that devices that can access full 64bit address are allowed
> > to avoid dma bouncing.
> 
> Maybe you could tell something on similar topic?
> 
> There is FreeBSD 12.2-STABLE r369567 Base12 amd64 running
> with Intel Atom CPU capable of long mode and addressing 8GB RAM,
> ASRock A330ION motherboard and two memory modules installed: 4G+2GB.
> Why so small "avail memory"?
> 
> FreeBSD clang version 10.0.1 (g...@github.com:llvm/llvm-project.git 
> llvmorg-10.0.1-0-gef32c611aa2)
> CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz (1600.03-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x106c2  Family=0x6  Model=0x1c  Stepping=2
>   
> Features=0xbfe9fbff
>   Features2=0x40e31d
>   AMD Features=0x2800
>   AMD Features2=0x1
>   TSC: P-state invariant, performance statistics
> real memory  = 6442450944 (6144 MB)
> Physical memory chunk(s):
> 0x0001 - 0x0009dfff, 581632 bytes (142 pages)
> 0x00103000 - 0x001f, 1036288 bytes (253 pages)
> 0x02b0 - 0xd8709fff, 3586170880 bytes (875530 pages)
> avail memory = 3571384320 (3405 MB)
> 
> Also http://www.grosbein.net/freebsd/dmidecode.txt

Some necromancy revealed that this CPU did not have memory controller
on-chip, it was a design from the 2008 where MCH handled memory.  It is
up to the chipset and BIOS to configure and report the memory above 4G
to OS.  As you clearly see from the SMAP printed above, BIOS does not
report anything above 4G.

Might be, look at bios settings.  No other ideas.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Loading zfs module results in hangup on i386

[Yasuhiro Kimura]

From: Yasuhiro Kimura 
Subject: Re: Loading zfs module results in hangup on i386
Date: Sat, 08 May 2021 07:44:15 +0900 (JST)

>> Now I think I know what is the source of problem. After all, on
>> 13.0-RELEASE i386 system simply loading zfs module results in system
>> hang up.
>> 
>> The steps to reproduce it are,
>> 
>> 1. Boot with install media of 13.0-RELEASE i386
>> 2. At the first menu of FreeBSD installer, select 'Shell'.
>> 3. At the shell prompt, type `kldload zfs` and return key.
>> 
>> I confirmed hangup happens with VirtualBox, VMware Player and my bare
>> metal PC environement. So the problem doesn't depend on hardware.
>> 
>> And hangup also happens with 13-STABLE and 14-CURRENT.
> 
> This problem is already reported to Bugzilla.
> 
> Bug 254177 When ZFS is recognized, An i386 machine with a lot of memory hangs.
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254177

Referencing the bug report, I applied attached patch to d474440ab33 of
main (14-CURRENT). built install image and tried install of ZFS root
i386 system with it. Then it completed successfully with 8GB memory.

Additionally GENERIC kernel recognizes 8GB of memory. And ZFS root
system works fine without any tuning.

----------
diff --git a/sys/contrib/openzfs/module/zfs/dbuf.c 
b/sys/contrib/openzfs/module/zfs/dbuf.c
index d48dc7943a2..c85500453fb 100644
--- a/sys/contrib/openzfs/module/zfs/dbuf.c
+++ b/sys/contrib/openzfs/module/zfs/dbuf.c
@@ -796,7 +796,7 @@ dbuf_init(void)
 * By default, the table will take up
 * totalmem * sizeof(void*) / 8K (1MB per GB with 8-byte pointers).
 */
-   while (hsize * zfs_arc_average_blocksize < physmem * PAGESIZE)
+   while (hsize * zfs_arc_average_blocksize < (uint64_t)physmem * PAGESIZE)
hsize <<= 1;
 
 retry:
--

---
Yasuhiro Kimura
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Install of 13.0-RELEASE i386 with ZFS root hangs up

[Eugene Grosbein]

08.05.2021 2:52, Konstantin Belousov wrote:

> i386 kernel uses memory up to 24G since 13.0.
> 
> PAE only means that devices that can access full 64bit address are allowed
> to avoid dma bouncing.

Maybe you could tell something on similar topic?

There is FreeBSD 12.2-STABLE r369567 Base12 amd64 running
with Intel Atom CPU capable of long mode and addressing 8GB RAM,
ASRock A330ION motherboard and two memory modules installed: 4G+2GB.
Why so small "avail memory"?

FreeBSD clang version 10.0.1 (g...@github.com:llvm/llvm-project.git 
llvmorg-10.0.1-0-gef32c611aa2)
CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz (1600.03-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x106c2  Family=0x6  Model=0x1c  Stepping=2
  
Features=0xbfe9fbff
  Features2=0x40e31d
  AMD Features=0x2800
  AMD Features2=0x1
  TSC: P-state invariant, performance statistics
real memory  = 6442450944 (6144 MB)
Physical memory chunk(s):
0x0001 - 0x0009dfff, 581632 bytes (142 pages)
0x00103000 - 0x001f, 1036288 bytes (253 pages)
0x02b0 - 0xd8709fff, 3586170880 bytes (875530 pages)
avail memory = 3571384320 (3405 MB)

Also http://www.grosbein.net/freebsd/dmidecode.txt

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Loading zfs module results in hangup on i386

[Yasuhiro Kimura]

From: Yasuhiro Kimura 
Subject: Loading zfs module results in hangup on i386 (Re: Install of 
13.0-RELEASE i386 with ZFS root hangs up)
Date: Sat, 08 May 2021 07:31:47 +0900 (JST)

> Now I think I know what is the source of problem. After all, on
> 13.0-RELEASE i386 system simply loading zfs module results in system
> hang up.
> 
> The steps to reproduce it are,
> 
> 1. Boot with install media of 13.0-RELEASE i386
> 2. At the first menu of FreeBSD installer, select 'Shell'.
> 3. At the shell prompt, type `kldload zfs` and return key.
> 
> I confirmed hangup happens with VirtualBox, VMware Player and my bare
> metal PC environement. So the problem doesn't depend on hardware.
> 
> And hangup also happens with 13-STABLE and 14-CURRENT.

This problem is already reported to Bugzilla.

Bug 254177 When ZFS is recognized, An i386 machine with a lot of memory hangs.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254177

---
Yasuhiro Kimura
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Loading zfs module results in hangup on i386 (Re: Install of 13.0-RELEASE i386 with ZFS root hangs up)

[Yasuhiro Kimura]

From: Yasuhiro Kimura 
Subject: Install of 13.0-RELEASE i386 with ZFS root hangs up
Date: Fri, 07 May 2021 21:47:59 +0900 (JST)

> Hello,
> 
> Does anyone succeed to install 13.0-RELEASE i386 with ZFS root?
> 
> I tried this with VirtualBox and VMware Player on Windows with
> following VM condition.
> 
> * 4 CPUs
> * 8GB memory
> * 100GB disk
> * Bridge mode NIC
> 
> But in both cases, VM gets high CPU load and hangs up after I moved
> to 'YES' at 'ZFS Configuration' menu and type return key.
> 
> If I select UFS root installation completes successfully. So the
> problem is specific to ZFS root.

Now I think I know what is the source of problem. After all, on
13.0-RELEASE i386 system simply loading zfs module results in system
hang up.

The steps to reproduce it are,

1. Boot with install media of 13.0-RELEASE i386
2. At the first menu of FreeBSD installer, select 'Shell'.
3. At the shell prompt, type `kldload zfs` and return key.

I confirmed hangup happens with VirtualBox, VMware Player and my bare
metal PC environement. So the problem doesn't depend on hardware.

And hangup also happens with 13-STABLE and 14-CURRENT.

---
Yasuhiro Kimura
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Install of 13.0-RELEASE i386 with ZFS root hangs up

[Konstantin Belousov]

On Fri, May 07, 2021 at 09:48:07AM -0700, Freddie Cash wrote:
> On Fri, May 7, 2021 at 5:49 AM Yasuhiro Kimura  wrote:
> 
> > Does anyone succeed to install 13.0-RELEASE i386 with ZFS root?
> >
> > I tried this with VirtualBox and VMware Player on Windows with
> > following VM condition.
> >
> > * 4 CPUs
> > * 8GB memory
> > * 100GB disk
> > * Bridge mode NIC
> >
> > But in both cases, VM gets high CPU load and hangs up after I moved
> > to 'YES' at 'ZFS Configuration' menu and type return key.
> >
> > If I select UFS root installation completes successfully. So the
> > problem is specific to ZFS root.
> >
> 
> Running ZFS on 32-bit OSes is doable (although not recommended) but
> requires a lot of manual configuration and tweaking, especially around
> kernel memory and ARC usage.
> 
> You're limited to 4 GB of memory space, so you need to tune the ARC to use
> less than that.  The auto-tuning has improved a lot over the years, but you
> still need to limit the ARC size to around 2 GB (or less) to keep the
> system stable.  KVA memory space tuning shouldn't be needed anymore, but
> you can do research into that, just in case.
> 
> You can compile a custom kernel to enable PAE support, that will sometimes
> help with memory issues on i386 (and will allow you to use more than 4 GB
> of system RAM, although individual processes are still limited to 4 GB).
i386 kernel uses memory up to 24G since 13.0.

PAE only means that devices that can access full 64bit address are allowed
to avoid dma bouncing.


> 
> If you really need to, you can make ZFS work on i386.  If at all possible,
> though, you really should run it on amd64 instead.
> 
> -- 
> Freddie Cash
> fjwc...@gmail.com
> ___
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Install of 13.0-RELEASE i386 with ZFS root hangs up

[Freddie Cash]

On Fri, May 7, 2021 at 5:49 AM Yasuhiro Kimura  wrote:

> Does anyone succeed to install 13.0-RELEASE i386 with ZFS root?
>
> I tried this with VirtualBox and VMware Player on Windows with
> following VM condition.
>
> * 4 CPUs
> * 8GB memory
> * 100GB disk
> * Bridge mode NIC
>
> But in both cases, VM gets high CPU load and hangs up after I moved
> to 'YES' at 'ZFS Configuration' menu and type return key.
>
> If I select UFS root installation completes successfully. So the
> problem is specific to ZFS root.
>

Running ZFS on 32-bit OSes is doable (although not recommended) but
requires a lot of manual configuration and tweaking, especially around
kernel memory and ARC usage.

You're limited to 4 GB of memory space, so you need to tune the ARC to use
less than that.  The auto-tuning has improved a lot over the years, but you
still need to limit the ARC size to around 2 GB (or less) to keep the
system stable.  KVA memory space tuning shouldn't be needed anymore, but
you can do research into that, just in case.

You can compile a custom kernel to enable PAE support, that will sometimes
help with memory issues on i386 (and will allow you to use more than 4 GB
of system RAM, although individual processes are still limited to 4 GB).

If you really need to, you can make ZFS work on i386.  If at all possible,
though, you really should run it on amd64 instead.

-- 
Freddie Cash
fjwc...@gmail.com
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Install of 13.0-RELEASE i386 with ZFS root hangs up

[Yasuhiro Kimura]

From: 8zwk...@oldach.net (Helge Oldach)
Subject: Re: Install of 13.0-RELEASE i386 with ZFS root hangs up
Date: Fri, 7 May 2021 15:41:45 +0200 (CEST)

> Yasuhiro Kimura wrote on Fri, 07 May 2021 14:47:59 +0200 (CEST):
>> Does anyone succeed to install 13.0-RELEASE i386 with ZFS root?
>   
> 
>> * 8GB memory
> 
> Try with 4GB perhaps?
> 
> Kind regards
> Helge

Thank you for suggestion. But unfortunately hangup still happens with
4GB memory.

---
Yasuhiro Kimura
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Install of 13.0-RELEASE i386 with ZFS root hangs up

[Yasuhiro Kimura]

Hello,

Does anyone succeed to install 13.0-RELEASE i386 with ZFS root?

I tried this with VirtualBox and VMware Player on Windows with
following VM condition.

* 4 CPUs
* 8GB memory
* 100GB disk
* Bridge mode NIC

But in both cases, VM gets high CPU load and hangs up after I moved
to 'YES' at 'ZFS Configuration' menu and type return key.

If I select UFS root installation completes successfully. So the
problem is specific to ZFS root.

---
Yasuhiro Kimura
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zpool list -p 's FREE vs. zfs list -p's AVAIL ? FREE-AVAIL == 6_675_374_080 (199G zroot pool)

[Mark Millard via freebsd-stable]

On 2021-May-5, at 17:01, Yuri Pankov  wrote:

> Mark Millard via freebsd-current wrote:
>> Context:
>> 
>> # gpart show -pl da0
>> =>   40  468862048da0  GPT  (224G)
>> 40 532480  da0p1  efiboot0  (260M)
>> 532520   2008 - free -  (1.0M)
>> 534528   25165824  da0p2  swp12a  (12G)
>>   25700352   25165824  da0p4  swp12b  (12G)
>>   50866176  417994752  da0p3  zfs0  (199G)
>>  468860928   1160 - free -  (580K)
>> 
>> There is just one pool: zroot and it is on zfs0 above.
>> 
>> # zpool list -p
>> NAME   SIZEALLOC  FREE  CKPOINT  EXPANDSZ   FRAG
>> CAP  DEDUPHEALTH  ALTROOT
>> zroot  213674622976  71075655680  142598967296- - 28     
>> 33   1.00ONLINE  -
>> 
>> So FREE: 142_598_967_296
>> (using _ to make it more readable)
>> 
>> # zfs list -p zroot 
>> NAME  USED AVAIL REFER  MOUNTPOINT
>> zroot  71073697792  135923593216 98304  /zroot
>> 
>> So AVAIL: 135_923_593_216
>> 
>> FREE-AVAIL == 6_675_374_080
>> 
>> 
>> 
>> The questions:
>> 
>> Is this sort of unavailable pool-free-space normal?
>> Is this some sort of expected overhead that just is
>> not explicitly reported? Possibly a "FRAG"
>> consequence?
> 
> From zpoolprops(8):
> 
> freeThe amount of free space available in the pool.  By contrast,
>the zfs(8) available property describes how much new data can be
>    written to ZFS filesystems/volumes.  The zpool free property is
>not generally useful for this purpose, and can be substantially
>more than the zfs available space. This discrepancy is due to
>several factors, including raidz parity; zfs reservation, quota,
>refreservation, and refquota properties; and space set aside by
>spa_slop_shift (see zfs-module-parameters(5) for more
>information).

Thanks for pointing to the reference material.

6_675_374_080/213_674_622_976 =approx= 0.03124 =approx= 1.0/32.0

and spa_slop_shift's description reports:

QUOTE
   spa_slop_shift (int)
   Normally, we don't allow the last 3.2%
   (1/(2^spa_slop_shift)) of space in the pool to be consumed.
   This ensures that we don't run the pool completely out of
   space, due to unaccounted changes (e.g. to the MOS).  It
   also limits the worst-case time to allocate space.  If we
   have less than this amount of free space, most ZPL
   operations (e.g. write, create) will return ENOSPC.

   Default value: 5.
END QUOTE

So in my simple context, apparently not much else
contributes and the figures are basically as
expected.

Thanks again.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zpool list -p 's FREE vs. zfs list -p's AVAIL ? FREE-AVAIL == 6_675_374_080 (199G zroot pool)

[Yuri Pankov]

Mark Millard via freebsd-current wrote:
> Context:
> 
> # gpart show -pl da0
> =>   40  468862048da0  GPT  (224G)
>  40 532480  da0p1  efiboot0  (260M)
>  532520   2008 - free -  (1.0M)
>  534528   25165824  da0p2  swp12a  (12G)
>25700352   25165824  da0p4  swp12b  (12G)
>50866176  417994752  da0p3  zfs0  (199G)
>   468860928   1160 - free -  (580K)
> 
> There is just one pool: zroot and it is on zfs0 above.
> 
> # zpool list -p
> NAME   SIZEALLOC  FREE  CKPOINT  EXPANDSZ   FRAG
> CAP  DEDUPHEALTH  ALTROOT
> zroot  213674622976  71075655680  142598967296- - 28 
> 33   1.00ONLINE  -
> 
> So FREE: 142_598_967_296
> (using _ to make it more readable)
> 
> # zfs list -p zroot 
> NAME  USED AVAIL REFER  MOUNTPOINT
> zroot  71073697792  135923593216 98304  /zroot
> 
> So AVAIL: 135_923_593_216
> 
> FREE-AVAIL == 6_675_374_080
> 
> 
> 
> The questions:
> 
> Is this sort of unavailable pool-free-space normal?
> Is this some sort of expected overhead that just is
> not explicitly reported? Possibly a "FRAG"
> consequence?

>From zpoolprops(8):

freeThe amount of free space available in the pool.  By contrast,
the zfs(8) available property describes how much new data can be
written to ZFS filesystems/volumes.  The zpool free property is
not generally useful for this purpose, and can be substantially
more than the zfs available space. This discrepancy is due to
several factors, including raidz parity; zfs reservation, quota,
refreservation, and refquota properties; and space set aside by
spa_slop_shift (see zfs-module-parameters(5) for more
information).
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

zpool list -p 's FREE vs. zfs list -p's AVAIL ? FREE-AVAIL == 6_675_374_080 (199G zroot pool)

[Mark Millard via freebsd-stable]

Context:

# gpart show -pl da0
=>   40  468862048da0  GPT  (224G)
 40 532480  da0p1  efiboot0  (260M)
 532520   2008 - free -  (1.0M)
 534528   25165824  da0p2  swp12a  (12G)
   25700352   25165824  da0p4  swp12b  (12G)
   50866176  417994752  da0p3  zfs0  (199G)
  468860928   1160 - free -  (580K)

There is just one pool: zroot and it is on zfs0 above.

# zpool list -p
NAME   SIZEALLOC  FREE  CKPOINT  EXPANDSZ   FRAGCAP 
 DEDUPHEALTH  ALTROOT
zroot  213674622976  71075655680  142598967296- - 28 33 
  1.00ONLINE  -

So FREE: 142_598_967_296
(using _ to make it more readable)

# zfs list -p zroot 
NAME  USED AVAIL REFER  MOUNTPOINT
zroot  71073697792  135923593216 98304  /zroot

So AVAIL: 135_923_593_216

FREE-AVAIL == 6_675_374_080



The questions:

Is this sort of unavailable pool-free-space normal?
Is this some sort of expected overhead that just is
not explicitly reported? Possibly a "FRAG"
consequence?


For reference:

# zpool status
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:31:48 with 0 errors on Sun May  2 19:52:14 2021
config:

NAMESTATE READ WRITE CKSUM
zroot   ONLINE   0 0 0
  da0p3 ONLINE   0 0 0

errors: No known data errors


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: ZFS rename with associated snapshot present: odd error message

[Mark Millard via freebsd-stable]

On 2021-May-5, at 05:28, Mark Millard  wrote:

> On 2021-May-5, at 02:47, Andriy Gapon  wrote:
> 
>> On 05/05/2021 01:59, Mark Millard via freebsd-current wrote:
>>> I had a:
>>> # zfs list -tall
>>> NAME   USED  AVAIL REFER  MOUNTPOINT
>>> . . .
>>> zroot/DESTDIRs/13_0R-CA72-instwrld-norm  1.44G   117G   96K 
>>>  /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm
>>> zroot/DESTDIRs/13_0R-CA72-instwrld-norm@dirty-style  1.44G  - 1.44G 
>>>  -. . .
>>> . . .
>>> (copied/pasted from somewhat earlier) and then attempted:
>>> # zfs rename zroot/DESTDIRs/13_0R-CA72-instwrld-norm 
>>> zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0
>>> cannot open 'zroot/DESTDIRs/13_0R-CA72-instwrld-norm@dirty-style': snapshot 
>>> delimiter '@' is not expected here
>>> Despite the "cannot open" message, the result looks like:
>>> # zfs list -tall
>>> NAME   USED  AVAIL 
>>> REFER  MOUNTPOINT
>>> . . .
>>> zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0  1.44G   114G  
>>>  96K  /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt-0
>>> zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0@dirty-style  1.44G  - 
>>> 1.44G  -
>>> . . .
>>> Still, it leaves me wondering if everything is okay
>>> given that internal attempt to use the old name with
>>> @dirty-style when it was apparently no longer
>>> available under that naming.
>>> For reference:
>>> # uname -apKU
>>> FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 
>>> releng/13.0-n244733-ea31abc261ff-dirty: Thu Apr 29 21:53:20 PDT 2021 
>>> root@CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/sys/GENERIC-NODBG-CA72
>>>   arm64 aarch64 1300139 1300139
>> 
>> Cannot reproduce here (but with much simpler names and on stable/13):
>> zfs create testz/test
>> zfs snapshot testz/test@snap1
>> zfs rename testz/test testz/test2
>> 
>> All worked.
>> 
> 
> I've noticed that sometimes in my explorations it has been
> silent instead of complaining. I've no clue at this point
> what prior activity (or lack of activity) makes the
> difference for if a message will be generated vs. not.

One difference in context is that your above sort of sequence
generates the after-snapshot context (using some things I have
around now):

zroot/DESTDIRs/13_0R-CA53-poud 1.45G   127G 1.45G  
/usr/obj/DESTDIRs/13_0R-CA53-poud
zroot/DESTDIRs/13_0R-CA53-poud@test   0B  - 1.45G  -

where my example had something more like (hand edited
the above just for illustration):

zroot/DESTDIRs/13_0R-CA53-poud 1.45G   125G   96K  
/usr/obj/DESTDIRs/13_0R-CA53-poud
zroot/DESTDIRs/13_0R-CA53-poud@test1.45G  - 1.45G  -

before the rename. In other words, I'd updated the
original (almost?) completely after the snapshot
(as a side effect of my overall activity). It was
only later that I tried the rename to track a new
purpose/context that I was going to switch to.

I'm not claiming that such is sufficient to
(always? ever?) reproduce the message. I'm
just pointing out that I'd had some significant
activity on the writable file system before the
rename.

Some of my activity has been more like your test
and I'd not seen the problems from such. But it
is not a very good comparison/contrast context
so I'd not infer much. I still can not at-will
set up a context to produce the messages.



===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: ZFS rename with associated snapshot present: odd error message

[Mark Millard via freebsd-stable]

On 2021-May-5, at 02:47, Andriy Gapon  wrote:

> On 05/05/2021 01:59, Mark Millard via freebsd-current wrote:
>> I had a:
>> # zfs list -tall
>> NAME   USED  AVAIL REFER  MOUNTPOINT
>> . . .
>> zroot/DESTDIRs/13_0R-CA72-instwrld-norm  1.44G   117G   96K  
>> /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm
>> zroot/DESTDIRs/13_0R-CA72-instwrld-norm@dirty-style  1.44G  - 1.44G  
>> -. . .
>> . . .
>> (copied/pasted from somewhat earlier) and then attempted:
>> # zfs rename zroot/DESTDIRs/13_0R-CA72-instwrld-norm 
>> zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0
>> cannot open 'zroot/DESTDIRs/13_0R-CA72-instwrld-norm@dirty-style': snapshot 
>> delimiter '@' is not expected here
>> Despite the "cannot open" message, the result looks like:
>> # zfs list -tall
>> NAME   USED  AVAIL 
>> REFER  MOUNTPOINT
>> . . .
>> zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0  1.44G   114G   
>> 96K  /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt-0
>> zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0@dirty-style  1.44G  - 
>> 1.44G  -
>> . . .
>> Still, it leaves me wondering if everything is okay
>> given that internal attempt to use the old name with
>> @dirty-style when it was apparently no longer
>> available under that naming.
>> For reference:
>> # uname -apKU
>> FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 
>> releng/13.0-n244733-ea31abc261ff-dirty: Thu Apr 29 21:53:20 PDT 2021 
>> root@CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/sys/GENERIC-NODBG-CA72
>>   arm64 aarch64 1300139 1300139
> 
> Cannot reproduce here (but with much simpler names and on stable/13):
> zfs create testz/test
> zfs snapshot testz/test@snap1
> zfs rename testz/test testz/test2
> 
> All worked.
> 

I've noticed that sometimes in my explorations it has been
silent instead of complaining. I've no clue at this point
what prior activity (or lack of activity) makes the
difference for if a message will be generated vs. not.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: ZFS rename with associated snapshot present: odd error message

[Andriy Gapon]

On 05/05/2021 01:59, Mark Millard via freebsd-current wrote:

I had a:

# zfs list -tall
NAME   USED  AVAIL REFER  MOUNTPOINT
. . .
zroot/DESTDIRs/13_0R-CA72-instwrld-norm  1.44G   117G   96K  
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm
zroot/DESTDIRs/13_0R-CA72-instwrld-norm@dirty-style  1.44G  - 1.44G  -. 
. .
. . .

(copied/pasted from somewhat earlier) and then attempted:

# zfs rename zroot/DESTDIRs/13_0R-CA72-instwrld-norm 
zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0
cannot open 'zroot/DESTDIRs/13_0R-CA72-instwrld-norm@dirty-style': snapshot 
delimiter '@' is not expected here

Despite the "cannot open" message, the result looks like:

# zfs list -tall
NAME   USED  AVAIL 
REFER  MOUNTPOINT
. . .
zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0  1.44G   114G   
96K  /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt-0
zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0@dirty-style  1.44G  - 
1.44G  -
. . .

Still, it leaves me wondering if everything is okay
given that internal attempt to use the old name with
@dirty-style when it was apparently no longer
available under that naming.

For reference:

# uname -apKU
FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 
releng/13.0-n244733-ea31abc261ff-dirty: Thu Apr 29 21:53:20 PDT 2021 
root@CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/sys/GENERIC-NODBG-CA72
  arm64 aarch64 1300139 1300139


Cannot reproduce here (but with much simpler names and on stable/13):
zfs create testz/test
zfs snapshot testz/test@snap1
zfs rename testz/test testz/test2

All worked.

--
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

ZFS rename with associated snapshot present: odd error message

[Mark Millard via freebsd-stable]

I had a:

# zfs list -tall
NAME   USED  AVAIL REFER  MOUNTPOINT
. . .
zroot/DESTDIRs/13_0R-CA72-instwrld-norm  1.44G   117G   96K  
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm
zroot/DESTDIRs/13_0R-CA72-instwrld-norm@dirty-style  1.44G  - 1.44G  -. 
. .
. . .

(copied/pasted from somewhat earlier) and then attempted:

# zfs rename zroot/DESTDIRs/13_0R-CA72-instwrld-norm 
zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0
cannot open 'zroot/DESTDIRs/13_0R-CA72-instwrld-norm@dirty-style': snapshot 
delimiter '@' is not expected here

Despite the "cannot open" message, the result looks like:

# zfs list -tall
NAME   USED  AVAIL 
REFER  MOUNTPOINT
. . .
zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0  1.44G   114G   
96K  /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt-0
zroot/DESTDIRs/13_0R-CA72-instwrld-alt-0@dirty-style  1.44G  - 
1.44G  -
. . .

Still, it leaves me wondering if everything is okay
given that internal attempt to use the old name with
@dirty-style when it was apparently no longer
available under that naming.

For reference:

# uname -apKU
FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 
releng/13.0-n244733-ea31abc261ff-dirty: Thu Apr 29 21:53:20 PDT 2021 
root@CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R-src/arm64.aarch64/sys/GENERIC-NODBG-CA72
  arm64 aarch64 1300139 1300139


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs native encryption best practices on RELENG13

[Alan Somers]

On Mon, Apr 26, 2021 at 3:04 PM mike tancsa  wrote:

> On 4/23/2021 11:47 PM, Peter Libassi wrote:
> > Yes, I’ve come to the same conclusion. This should be used on a
> > data-zpool and not on the system-pool (zroot). Encryption is per
> > dataset. Also if found that if the encrypted dataset is not mounted of
> > some reason you will be writing to the parent unencrypted dataset.. At
> > least it works for encrypted thumb_drive, i just posted this quick
> > guide
> https://forums.freebsd.org/threads/freebsd-13-openzfs-encrypted-thumb-drive.80008/
> > <
> https://forums.freebsd.org/threads/freebsd-13-openzfs-encrypted-thumb-drive.80008/
> >
> >
> >
> >
>
> Thanks, good points to consider!  I wonder too if there are any
> performance differences
>
> ---Mike
>

Yes there are.  Firstly, if you're using raid, then geli must encrypt both
data and parity.  ZFS crypto, however, only encrypts data because it
operates at a higher level.  That's a pretty substantial performance win
for ZFS during writes.  It's a nonissue for reads from a healthy array,
since ZFS doesn't need to read parity in that case.  Secondly, ZFS crypto
doesn't yet support hardware acceleration.  That's a huge win for geli if
you happen to have a hardware crypto engine (for this purpose AESNI does
not count as hardware, and it works fine with either geli or ZFS).
Thirdly, in my benchmarks I found about a 5% speed advantage for GELI
during reads, though I don't know why.  But of course none of this matters
if you're using a small number of HDDs.  It's only an issue if you have
fast SSDs or a large number of HDDs.
-Alan
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs native encryption best practices on RELENG13

[mike tancsa]

On 4/23/2021 11:47 PM, Peter Libassi wrote:
> Yes, I’ve come to the same conclusion. This should be used on a
> data-zpool and not on the system-pool (zroot). Encryption is per
> dataset. Also if found that if the encrypted dataset is not mounted of
> some reason you will be writing to the parent unencrypted dataset.. At
> least it works for encrypted thumb_drive, i just posted this quick
> guide 
> https://forums.freebsd.org/threads/freebsd-13-openzfs-encrypted-thumb-drive.80008/
> 
>
>
>

Thanks, good points to consider!  I wonder too if there are any
performance differences

    ---Mike


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs native encryption best practices on RELENG13

[mike tancsa]

On 4/23/2021 5:23 PM, Xin Li wrote:
> On 4/23/21 13:53, mike tancsa wrote:
>> Starting to play around with RELENG_13 and wanted explore ZFS' built in
>> encryption.  Is there a best practices doc on how to do full disk
>> encryption anywhere thats not GELI based  ?  There are lots for 
>> GELI,
>> but nothing I could find for native OpenZFS encryption on FreeBSD
>>
>> i.e box gets rebooted, enter in passphrase to allow it to boot kind of
>> thing from the boot loader prompt ?
> I think loader do not support the native OpenZFS encryption yet.
> However, you can encrypt non-essential datasets on a boot pool (that is,
> if com.datto:encryption is "active" AND the bootfs dataset is not
> encrypted, you can still boot from it).
>
> BTW instead of entering passphrase at loader prompt, if / is not
> encrypted, it's also possible to do something like
> https://lists.freebsd.org/pipermail/freebsd-security/2012-August/006547.html
> .
>
> Personally I'd probably go with GELI (or other kind of full disk
> encryption) regardless if OpenZFS's native encryption is used because my
> primary goal is to be able to just throw away bad disks when they are
> removed from production [1].  If the pool is not fully encrypted, there
> is always a chance that the sensitive data have landed some unencrypted
> datasets and never gets fully overwritten.
>
> [1] Also keep in mind: https://xkcd.com/538/

Thanks for the perspective and links.  I have a couple of use case
scenarios.  One, for devices in somewhat physically untrusted
environments.  Someone breaks into the store, and steals the PC.  
I can
see the advantages of GELI to this environment.  The other is the
ability for customers to send me encrypted datasets for offsite backup. 
If its encrypted, I have less exposure if the dataset is encrypted and I
cant see the contents.   Same for making backups to disks to put in cold
storage although yes, I can see GELI having an an advantage again for
full disk encryption. 


    ---Mike


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

(D29934) Reorder commented steps in UPDATING following sequential order. (was: etcupdate -p vs. root on zfs (and bectl use and such): no /usr/src/etc/master.passwd (for example))

[Graham Perrin]

On 23/04/2021 08:39, Mark Millard via freebsd-current wrote:


   [3]



With regard to mounting ZFS file systems in single user mode

What's currently footnote 3 will probably become footnote 4, please see:

<https://reviews.freebsd.org/D29934#inline-186101>

… and so on.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: (D29934) Reorder commented steps in UPDATING following sequential order. (was: etcupdate -p vs. root on zfs (and bectl use and such): no /usr/src/etc/master.passwd (for example))

[Mark Millard via freebsd-stable]

On 2021-Apr-25, at 08:14, Graham Perrin  wrote:

> On 23/04/2021 08:39, Mark Millard via freebsd-current wrote:
> 
>>   [3]
> 
> 
> With regard to mounting ZFS file systems in single user mode
> 
> What's currently footnote 3 will probably become footnote 4, please see:
> 
> <https://reviews.freebsd.org/D29934#inline-186101>
> 
> … and so on.

If it were me, I'd probably do something to make the
mounting of file systems and such have an explicit
reminder as its own step, something like:


[4]
mergemaster -Fp [5]

I just do not think of such as part of :
it is already rebooted in single user at that point in my
view.

Sorry that I missed what was there in UPDATING.

However, /usr/src/Makefile has:

#  1.  `cd /usr/src'   (or to the directory containing your source tree).
#  2.  `make buildworld'
#  3.  `make buildkernel KERNCONF=YOUR_KERNEL_HERE' (default is GENERIC).
#  4.  `make installkernel KERNCONF=YOUR_KERNEL_HERE'   (default is GENERIC).
#   [steps 3. & 4. can be combined by using the "kernel" target]
#  5.  `reboot'(in single user mode: boot -s from the loader prompt).
#  6.  `mergemaster -p'
#  7.  `make installworld'
#  8.  `mergemaster'(you may wish to use -i, along with -U or -F).
#  9.  `make delete-old'
# 10.  `reboot'
# 11.  `make delete-old-libs' (in case no 3rd party program uses them anymore)

without such material, even in footnotes.


Side notes:

"From the bootblocks, boot -s, and then do":
"From the boot loader, boot -s, and then do"?

etcupdate vs. mergemaster and the $FreeBSD$ issue?
Is mergemaster going to stay as the recommented
command to use? If so, with which command line
options?

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs native encryption best practices on RELENG13

[Andrea Venturoli]

On 4/23/21 11:23 PM, Xin Li via freebsd-stable wrote:


I think loader do not support the native OpenZFS encryption yet.
However, you can encrypt non-essential datasets on a boot pool (that is,
if com.datto:encryption is "active" AND the bootfs dataset is not
encrypted, you can still boot from it).


This is what my tests showed too (on 12.2 with OpenZFS from ports).

This is in contrast to what is written here:
https://openzfs.github.io/openzfs-docs/Getting%20Started/FreeBSD.html

Can we get that page corrected?

 bye & Thanks
av.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs native encryption best practices on RELENG13

[Peter Libassi]

> 23 apr. 2021 kl. 23:23 skrev Xin Li via freebsd-stable 
> :
> 
> On 4/23/21 13:53, mike tancsa wrote:
>> Starting to play around with RELENG_13 and wanted explore ZFS' built in
>> encryption.  Is there a best practices doc on how to do full disk
>> encryption anywhere thats not GELI based  ?  There are lots for 
>> GELI,
>> but nothing I could find for native OpenZFS encryption on FreeBSD
>> 
>> i.e box gets rebooted, enter in passphrase to allow it to boot kind of
>> thing from the boot loader prompt ?
> 
> I think loader do not support the native OpenZFS encryption yet.
> However, you can encrypt non-essential datasets on a boot pool (that is,
> if com.datto:encryption is "active" AND the bootfs dataset is not
> encrypted, you can still boot from it).
> 
> BTW instead of entering passphrase at loader prompt, if / is not
> encrypted, it's also possible to do something like
> https://lists.freebsd.org/pipermail/freebsd-security/2012-August/006547.html
> .
> 
> Personally I'd probably go with GELI (or other kind of full disk
> encryption) regardless if OpenZFS's native encryption is used because my
> primary goal is to be able to just throw away bad disks when they are
> removed from production [1].  If the pool is not fully encrypted, there
> is always a chance that the sensitive data have landed some unencrypted
> datasets and never gets fully overwritten.
> 
> [1] Also keep in mind: https://xkcd.com/538/
> 
> Cheers,
> 
Yes, I’ve come to the same conclusion. This should be used on a data-zpool and 
not on the system-pool (zroot). Encryption is per dataset. Also if found that 
if the encrypted dataset is not mounted of some reason you will be writing to 
the parent unencrypted dataset.. At least it works for encrypted thumb_drive, i 
just posted this quick guide 
https://forums.freebsd.org/threads/freebsd-13-openzfs-encrypted-thumb-drive.80008/
 
<https://forums.freebsd.org/threads/freebsd-13-openzfs-encrypted-thumb-drive.80008/>

/Peter
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs native encryption best practices on RELENG13

[Xin Li via freebsd-stable]

On 4/23/21 13:53, mike tancsa wrote:
> Starting to play around with RELENG_13 and wanted explore ZFS' built in
> encryption.  Is there a best practices doc on how to do full disk
> encryption anywhere thats not GELI based  ?  There are lots for 
> GELI,
> but nothing I could find for native OpenZFS encryption on FreeBSD
> 
> i.e box gets rebooted, enter in passphrase to allow it to boot kind of
> thing from the boot loader prompt ?

I think loader do not support the native OpenZFS encryption yet.
However, you can encrypt non-essential datasets on a boot pool (that is,
if com.datto:encryption is "active" AND the bootfs dataset is not
encrypted, you can still boot from it).

BTW instead of entering passphrase at loader prompt, if / is not
encrypted, it's also possible to do something like
https://lists.freebsd.org/pipermail/freebsd-security/2012-August/006547.html
.

Personally I'd probably go with GELI (or other kind of full disk
encryption) regardless if OpenZFS's native encryption is used because my
primary goal is to be able to just throw away bad disks when they are
removed from production [1].  If the pool is not fully encrypted, there
is always a chance that the sensitive data have landed some unencrypted
datasets and never gets fully overwritten.

[1] Also keep in mind: https://xkcd.com/538/

Cheers,



OpenPGP_signature
Description: OpenPGP digital signature

zfs native encryption best practices on RELENG13

[mike tancsa]

Starting to play around with RELENG_13 and wanted explore ZFS' built in
encryption.  Is there a best practices doc on how to do full disk
encryption anywhere thats not GELI based  ?  There are lots for 
GELI,
but nothing I could find for native OpenZFS encryption on FreeBSD

i.e box gets rebooted, enter in passphrase to allow it to boot kind of
thing from the boot loader prompt ?

    ---Mike


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

etcupdate -p vs. root on zfs (and bectl use and such): no /usr/src/etc/master.passwd (for example)

[Mark Millard via freebsd-stable]

FYI: The default bsdinstall result for auto ZFS that I tried
has a separate zroot/usr/src dataset, which zfs mounts at
/usr/src .

UPDATING and such places indicate sequences like:
(think etcupdate where it lists mergemaster and ignore
-F and -Fi)


make buildworld
make buildkernel KERNCONF=YOUR_KERNEL_HERE
make installkernel KERNCONF=YOUR_KERNEL_HERE
[1]
 [3]
mergemaster -Fp [5]
NOTE: What /usr/src/etc/master.passwd here? (for example)
make installworld
mergemaster -Fi [4]
make delete-old [6]


etcupdate has the logic for handling -p:

if [ -n "$preworld" ]; then
# Build a limited tree that only contains files that are
# crucial to installworld.
for file in $PREWORLD_FILES; do
name=$(basename $file)
mkdir -p $1/etc >&3 2>&1 || return 1
cp -p $SRCDIR/$file $1/etc/$name || return 1
done

Note the "$SRCDIR/$file". But for a boot -s after
installing the kernel there is only zroot/ROOT/NAME
and no zroot/usr/src zfs mount so /usr/src/ is empty.

This leads to needing an additional step:

zfs mount zroot/usr/src

(The instructions do not deal with making / writable at this
stage either.)



===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Frequent disk I/O stalls while building (poudriere), processes in "zfs tear" state

[Felix Palmen]

* Dewayne Geraghty  [20210416 06:26]:
> On 16/04/2021 2:29 am, Felix Palmen wrote:
> > Right now, I'm running a test with idprio 0 instead, which still seems
> > to have the desired effect, and so far, I didn't have any of these
> > stalls. If this persists, the problem is solved for me!
> > 
> > I'd still be curious about what might be the cause, and, what this state
> > "zfs tear" actually means. But that's kind of an "academic interest"
> > now.
> 
> Most likely your other processes are pre-empting your build, which is
> what you want :).

Yes, that's exactly the plan.

> Use /usr/bin/top to see the priority of the processes (ie under the  PRI
> column).  Using an idprio 22, means (on my 12.2Stable) a PRI of 146.  If
> your kern.sched.preempt_thresh is using the default (of 80) then
> processes with a PRI of <80 will preempt (for IO).

I was doing that a lot, that's how I found those "global" I/O stalls
were happening when some processes were in that "zfs tear" state (shown
in top only as "zfs te").

> Even with an idprio 0, the PRI is 124. So I suspect that was more a
> matter of timing (ie good luck).

That seems kind of unlikely because the behavior is pretty reproducible.
Having observed builds on idprio 0 (yes, this results in a priority of
124) for a while, I still see from time to time processes getting
"stuck" for a few seconds, mostly ccache processes, but now in state
"zfsvfs" and the rest of the system is not affected, I/O still works.

So, something did change with ZFS and priorities between 12.2 and 13.0.
Running the whole builds on idprio 22 worked fine on 12.2.

> You could increase your pre-emption threshold for the duration of the
> build, to include your nice value. But... (not really a good idea).

That would clearly defeat the purpose, yes ;)

> Re zfs - sorry, I'm peculiar and don't use it ;)

I suspect the relevant change to be exactly in that context, still
thanks for answering :) Now that I have a working solution, it isn't an
important issue for me any more. Curiosity remains…

-- 
 Dipl.-Inform. Felix Palmen ,.//..
 {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
 {pgp public key} http://palmen-it.de/pub.txt   //   """""""""""
 {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A


signature.asc
Description: PGP signature

Re: Frequent disk I/O stalls while building (poudriere), processes in "zfs tear" state

[Dewayne Geraghty]

On 16/04/2021 2:29 am, Felix Palmen wrote:
> After more experimentation, I finally found what's causing these
> problems for me on 13:
> 
> * Felix Palmen  [20210412 11:44]:
>> * Poudriere running on idprio 22 with 8 parallel build jobs
> 
> Running poudriere with normal priority works perfectly fine. Now, I've
> had poudriere running on idprio because there are several other services
> on that machine that shouldn't be slowed down by running a heavy build
> and I still want to use all the CPU resources available for building.
> 
> Right now, I'm running a test with idprio 0 instead, which still seems
> to have the desired effect, and so far, I didn't have any of these
> stalls. If this persists, the problem is solved for me!
> 
> I'd still be curious about what might be the cause, and, what this state
> "zfs tear" actually means. But that's kind of an "academic interest"
> now.
> 

Most likely your other processes are pre-empting your build, which is
what you want :).

Use /usr/bin/top to see the priority of the processes (ie under the  PRI
column).  Using an idprio 22, means (on my 12.2Stable) a PRI of 146.  If
your kern.sched.preempt_thresh is using the default (of 80) then
processes with a PRI of <80 will preempt (for IO).

Even with an idprio 0, the PRI is 124. So I suspect that was more a
matter of timing (ie good luck).

You could increase your pre-emption threshold for the duration of the
build, to include your nice value. But... (not really a good idea).
Better if you run your build using nice (PRI of 76) which should avoid
the stalls, but should also influence your more important services.

Re zfs - sorry, I'm peculiar and don't use it ;)
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Frequent disk I/O stalls while building (poudriere), processes in "zfs tear" state

[Felix Palmen]

After more experimentation, I finally found what's causing these
problems for me on 13:

* Felix Palmen  [20210412 11:44]:
> * Poudriere running on idprio 22 with 8 parallel build jobs

Running poudriere with normal priority works perfectly fine. Now, I've
had poudriere running on idprio because there are several other services
on that machine that shouldn't be slowed down by running a heavy build
and I still want to use all the CPU resources available for building.

Right now, I'm running a test with idprio 0 instead, which still seems
to have the desired effect, and so far, I didn't have any of these
stalls. If this persists, the problem is solved for me!

I'd still be curious about what might be the cause, and, what this state
"zfs tear" actually means. But that's kind of an "academic interest"
now.

-- 
 Dipl.-Inform. Felix Palmen ,.//..
 {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
 {pgp public key} http://palmen-it.de/pub.txt   //   """""""""""
 {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A


signature.asc
Description: PGP signature

Frequent disk I/O stalls while building (poudriere), processes in "zfs tear" state

[Felix Palmen]

Hello all,

since following the releng/13.0 branch, I experience stalled disk I/O
quite often (ca. once per minute) while building packages with poudriere.

What I can see in this case is the CPU going almost idle, and several
processes shown in `top` in state "zfs te" (and procstat shows "zfs
tear" for that). For up to several seconds, no disk I/O completes (even
starting a new process is impossible), then it recovers. Only two times,
I have seen the system going into a deadlock instead, with printing
messages similar to this to the serial console:

  swap_pager: indefinite wait buffer ...

I have this behavior since -RC3 (followed releng/13.0 now up to
-RELEASE). Before that, I had the vnlru-related problem that was fixed
with faa41af1fed350327cc542cb240ca2c6e1e8ba0c.

Some details:

* CPU: Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz
* RAM: 64GB (ECC)
* Four HDDs (Seagate NAS models), 4TB each
* Swap 16GB, striped over the 4 disks
* Pool: 12TB raid-z on GELI-encrypted partitions. NOT upgraded yet, so I
  have a way back to 12.2.
* Two bhyve VMs running with 1GB and 8GB RAM, both wired
* Several jails running services like samba, an MTA, nginx...
* Several NFS shares mounted by other machines
* Poudriere running on idprio 22 with 8 parallel build jobs

Reducing the parallel jobs in poudriere also reduces the frequency of
the problem, but it doesn't seem to completely go away. Also, I have the
impression running into these stalls is more likely when a lot of
compilation jobs can be satisfied from ccache.

Thanks for any ideas and insight (e.g. what this "zfs tear" status
means).

Best regards,
Felix Palmen

-- 
 Dipl.-Inform. Felix Palmen ,.//..
 {web}  http://palmen-it.de  {jabber} [see email]   ,//palmen-it.de
 {pgp public key} http://palmen-it.de/pub.txt   //   """""""""""
 {pgp fingerprint} A891 3D55 5F2E 3A74 3965 B997 3EF2 8B0A BC02 DA2A


signature.asc
Description: PGP signature

Re: ZFS (in virtual machine): $HOME being a dataset causes xauth to timeout- access delays?

[John Kennedy]

On Thu, Apr 01, 2021 at 07:18:56PM -1000, parv/freebsd wrote:
> On Thu, Apr 1, 2021 at 3:38 PM parv/freebsd wrote:
> 
> I am wondering if $SRC_BASE, $MAKEOBJDIRPREFIX, & $WRKDIRPREFIX being
> ZFS datasets now would increase compile time. I will found that out in
> few weeks (in case of buildworld & kernel) as earlier I had both $SRC_BASE &
> $MAKEOBJDIRPREFIX as plain directories & took ~5--6 hours.
> 
> I have FreeBSD 12-STABLE in VirtualBox 5.2.44 on Windows 10. All the "disks"
> > are file-backed virtual disks.
> >
> > I noticed that after making $HOME a ZFS dataset, there were delays ...
> > - generally in start of Xorg;
> > - exhibited by xauth (after using "startx") error messages about timeout
> >   with authority files.
> >
> > After reverting back $HOME not being a dataset on its own as before, there
> > are no delays in start of Xorg; nor are any timeout message from xauth as
> > before.
> >
> > Has anybody else noticed that?

  I can't say that I've noticed any slowness.  I normally don't use xauth
unless I'm doing something exotic (like su to another user but want to have
them able to use my X display), so probably not the best example.  I've
also been using zfs on FreeBSD in favor of any other filesystem since it was
an option, so I'm probably probably totally worthless for speed comparison.

  That being said...

  I often feel like I'm rocking the 1986 X vibes because, while I like eye
candy, that eye candy is coming with a lot of baggage.  Gnome/KDE?  Hah!  I'm
using ctwm.  But if you've got said eye candy that wants to walk your file
system tree and you've got a /home/$USER/.zfs/snapshot with lots of backups
that is magnifying your tree, something might be doing a lot more work than
the designer was originally expecting.

  As far as ZFS datasets go, I keep stock+ to keep my snapshot space low.  I
added a few things onto the stock ZFS partitioning:

zfs create -o canmount=off zroot/usr/home/$USER
zfs create zroot/usr/home/$USER/.cache
zfs create zroot/usr/home/$USER/.mozilla

zfs create zroot/usr/ports/distfiles
zfs create -o canmount=off zroot/usr/obj
zfs create zroot/usr/obj/usr

zfs create zroot/poudriere

  So /home -> /usr/home is a dataset by default, /home/$USER isn't (I'm the
only user, so I wasn't worried there), but because I like doing snapshot
backups of things and I noticed that my snapshots were pretty damn big for
what I was working on, I decided to strip out the browser droppings by
making a dataset for .cache and .mozilla.  Doesn't scale, but just me.

  To keep my zroot/ROOT/default and boot environments clean, I created the
datasets for /usr/obj (kernel objects) because that is generally disposable
(except when I was doing RPI work and that was precious gold since it took
days to rebuild and I might want it to match a kernel I reverted to).  I
added oen for /usr/ports/distfiles to keep it from bloating out /usr/ports,
which is also easily reconstructed (vs the tarballs, which sometimes go AWOL).

  In any case, for /usr/obj, some variation of this is what I was looking at:

# df -h / /usr/obj/usr
FilesystemSizeUsed   Avail Capacity  Mounted on
zroot/ROOT/default 67G4.2G 63G 6%/
zroot/usr/obj/usr  72G8.6G 63G12%/usr/obj/usr

  I do not need 8G of stuff that I'll probably never use again snapshotted with
my root filesystem that I may keep around for extended periods of time.  On
disk, each of the roots generally seems to use ~1.2G of space with what they
have in common, but I wasn't going to get that with /usr/obj.

  Now, I've got a pretty beefy box, so my compiling pain points may not match
yours but you can do things like tune the datasets.  For example, if you're
CPU bound and you have a lot of compression on /usr/obj, you may want to turn
that off (while being able to leave it on for the root partition).

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

ZFS (in virtual machine): $HOME being a dataset causes xauth to timeout- access delays?

[parv]

Hi there,

I have FreeBSD 12-STABLE in VirtualBox 5.2.44 on Windows 10. All the "disks"
are file-backed virtual disks.

I noticed that after making $HOME a ZFS dataset, there were delays ...
- generally in start of Xorg;
- exhibited by xauth (after using "startx") error messages about timeout
  with authority files.

After reverting back $HOME not being a dataset on its own as before, there
are
no delays in start of Xorg; nor are any timeout message from xauth as
before.

Has anybody else noticed that?


  - parv


Before making $HOME (/aux/home/parv) a dataset ...

  aux0 mounted at /aux
  aux0/home at /aux/home

Conversion of $HOME to a dataset ..

  % zfs create aux0/home/parv2
  % mv /aux/home/parv/* /aux/home/parv2/
  % mv /aux/home/parv/.* /aux/home/parv2/
  % rm -rf /aux/home/parv
  % zfs rename aux0/home/parv2 aux0/home/parv
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: ZFS (in virtual machine): $HOME being a dataset causes xauth to timeout- access delays?

[parv/freebsd]

On Thu, Apr 1, 2021 at 3:38 PM parv/freebsd wrote:

I am wondering if $SRC_BASE, $MAKEOBJDIRPREFIX, & $WRKDIRPREFIX being
ZFS datasets now would increase compile time. I will found that out in
few weeks (in case of buildworld & kernel) as earlier I had both $SRC_BASE &
$MAKEOBJDIRPREFIX as plain directories & took ~5--6 hours.

I have FreeBSD 12-STABLE in VirtualBox 5.2.44 on Windows 10. All the "disks"
> are file-backed virtual disks.
>
> I noticed that after making $HOME a ZFS dataset, there were delays ...
> - generally in start of Xorg;
> - exhibited by xauth (after using "startx") error messages about timeout
>   with authority files.
>
> After reverting back $HOME not being a dataset on its own as before, there
> are
> no delays in start of Xorg; nor are any timeout message from xauth as
> before.
>
> Has anybody else noticed that?
>
...

  - parv
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

ZFS (in virtual machine): $HOME being a dataset causes xauth to timeout- access delays?

[parv/freebsd]

 Hi there,

I have FreeBSD 12-STABLE in VirtualBox 5.2.44 on Windows 10. All the "disks"
are file-backed virtual disks.

I noticed that after making $HOME a ZFS dataset, there were delays ...
- generally in start of Xorg;
- exhibited by xauth (after using "startx") error messages about timeout
  with authority files.

After reverting back $HOME not being a dataset on its own as before, there
are
no delays in start of Xorg; nor are any timeout message from xauth as
before.

Has anybody else noticed that?


  - parv


Before making $HOME (/aux/home/parv) a dataset ...

  aux0 mounted at /aux
  aux0/home at /aux/home

Conversion of $HOME to a dataset ..

  % zfs create aux0/home/parv2
  % mv /aux/home/parv/* /aux/home/parv2/
  % mv /aux/home/parv/.* /aux/home/parv2/
  % rm -rf /aux/home/parv
  % zfs rename aux0/home/parv2 aux0/home/parv
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA

[Andriy Gapon]

On 20/03/2021 05:01, Yoshihiro Ota wrote:
> On Tue, 9 Mar 2021 11:24:53 +0200
> Andriy Gapon  wrote:
> 
>> On 08/03/2021 05:24, Yoshihiro Ota wrote:
>>> On Sun, 7 Mar 2021 00:09:33 +0200
>>> Andriy Gapon  wrote:
>>>
>>>> On 06/03/2021 20:09, Yoshihiro Ota wrote:
>>>>> Hi all,
>>>>>
>>>>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
>>>>>
>>>>> After upgrading one in VMWare, 'zfs mount -a' hangs the system.
>>>>> I don't have boottime zfs mount on nor don't have zfsroot.
>>>>> I just simply ran install world/kernel and mergemaster.
>>>>
>>>> Please use procstat -kk to capture a kernel stack trace of the hung 
>>>> process.
>>>
>>> Actually, spining was 'kldload zfs'.
>>> Console doesn't response but ping and sshd sessions still work.
>>> procstat output is below.
>>> In addition, this doesn't happen to systems that I've been following 
>>> 13-CURRENT
>>> but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC.
>>>
>>>
>>> # procstat -kk 1049
>>>   PIDTID COMMTDNAME  KSTACK 
>>>   
>>>  1049 100215 kldload -   spa_init+0xc6 
>>> zfs_kmod_init+0x1a
>>> zfs_modevent+0x34 module_register_init+0x8c linker_load_module+0xaab 
>>> kern_kldload+0xc1
>>> sys_kldload+0x50 syscall+0x17d g_ctx+0xe280bf29 
>>>
>>
>> If you could use kgdb to find out what source code line spa_init+0xc6
>> corresponds to that may help to see what's going on.
>>
> 
> It look me a while to get kgdb working properly.
> At last, I got the output.
> It looks it is spining on a mutex.
> 
> I have few other machines run the same kernel but they can load zfs.ko.
> It is only vmware VM that spins with 'kldload zfs'.
> 
> vmware# kgdb101 /usr/usr/lib/debug/boot/kernel/zfs.ko.debug
> GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD]
> Copyright (C) 2020 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "i386-portbld-freebsd13.0".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <https://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> 
> For help, type "helpType "apropos word" to search for commands related to 
> "word"...
> Reading symbols from zfs.ko.debug...
> (kgdb) info line *spa_init+0xc6
> Line 2345 of "/usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c"
>starts at address 0x2b0461 
>and ends at 0x2b0467 .
> (kgdb) 
> 
> void
> spa_init(spa_mode_t mode)
> {
> mutex_init(_namespace_lock, NULL, MUTEX_DEFAULT, NULL);
> mutex_init(_spare_lock, NULL, MUTEX_DEFAULT, NULL); // <- line 
> 2345
> 

It would highly unusual (and unexplainable) for a thread to get stuck here.
Are you sure that your source tree matches the binary?
Can you disassemble the function to check instructions at and near 0xc6 offset?


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Request: Mount zfs encrypted datasets at boot? Re: FreeBSD 13.0-RC2 Now Available

[Ruben van Staveren via freebsd-stable]

Hi,

> On 13 Mar 2021, at 1:11, Glen Barber  wrote:
> 
> The second RC build of the 13.0-RELEASE release cycle is now available.


Might it be interesting to change the

zfs mount -a / zfs unmount -a

in /etc/rc.d/zfs to

zfs mount -al / zfs unmount -au

so that filesystems using the openzfs encryption feature can be automatically 
mounted?

given their keylocation is not “prompt" and accessible during boot.

This would make it possible for me to move away from my current dual pool boot 
setup, in where the boot pool has the geli keys for my geli encrypted system 
pool, which seems to be incompatible with beadm/bectl handling of things these 
days…
The single geli encrypted pool does not work for me as it doesn’t allow for 
unattended reboots.

Best regards,
Ruben



signature.asc
Description: Message signed with OpenPGP

Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA

[Yoshihiro Ota]

On Tue, 9 Mar 2021 11:24:53 +0200
Andriy Gapon  wrote:

> On 08/03/2021 05:24, Yoshihiro Ota wrote:
> > On Sun, 7 Mar 2021 00:09:33 +0200
> > Andriy Gapon  wrote:
> > 
> >> On 06/03/2021 20:09, Yoshihiro Ota wrote:
> >>> Hi all,
> >>>
> >>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
> >>>
> >>> After upgrading one in VMWare, 'zfs mount -a' hangs the system.
> >>> I don't have boottime zfs mount on nor don't have zfsroot.
> >>> I just simply ran install world/kernel and mergemaster.
> >>
> >> Please use procstat -kk to capture a kernel stack trace of the hung 
> >> process.
> > 
> > Actually, spining was 'kldload zfs'.
> > Console doesn't response but ping and sshd sessions still work.
> > procstat output is below.
> > In addition, this doesn't happen to systems that I've been following 
> > 13-CURRENT
> > but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC.
> > 
> > 
> > # procstat -kk 1049
> >   PIDTID COMMTDNAME  KSTACK 
> >   
> >  1049 100215 kldload -   spa_init+0xc6 
> > zfs_kmod_init+0x1a
> > zfs_modevent+0x34 module_register_init+0x8c linker_load_module+0xaab 
> > kern_kldload+0xc1
> > sys_kldload+0x50 syscall+0x17d g_ctx+0xe280bf29 
> > 
> 
> If you could use kgdb to find out what source code line spa_init+0xc6
> corresponds to that may help to see what's going on.
> 

It look me a while to get kgdb working properly.
At last, I got the output.
It looks it is spining on a mutex.

I have few other machines run the same kernel but they can load zfs.ko.
It is only vmware VM that spins with 'kldload zfs'.

vmware# kgdb101 /usr/usr/lib/debug/boot/kernel/zfs.ko.debug
GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD]
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "i386-portbld-freebsd13.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "helpType "apropos word" to search for commands related to 
"word"...
Reading symbols from zfs.ko.debug...
(kgdb) info line *spa_init+0xc6
Line 2345 of "/usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c"
   starts at address 0x2b0461 
   and ends at 0x2b0467 .
(kgdb) 

void
spa_init(spa_mode_t mode)
{
mutex_init(_namespace_lock, NULL, MUTEX_DEFAULT, NULL);
mutex_init(_spare_lock, NULL, MUTEX_DEFAULT, NULL); // <- line 2345


Hiro
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Updating to 13-stable and existing ZFS pools: any gotchas?

[Dean E. Weimer via freebsd-stable]

On 2021-03-17 9:59 am, tech-lists wrote:

On Sun, Mar 14, 2021 at 09:59:21AM +0100, Stefan Bethke wrote:
I'm planning to upgrade three production machines with existing ZFS 
pools to 13-stable. Is there anything I need to pay attention to wrt 
OpenZFS? Or should it be fully transparent, apart from updating 
loader?


My (limited) testing with VMs went without a hitch, but I want to make 
sure I don't paint myself into a corner.


Hi, I'm interested in this as well.

I'm not using root-on-zfs. The zpool is 5* spinning rust, the booting
media is ssd/ufs2. Is updating the bootcode only relevant for
root-on-zfs? I've not done that for a similarly configured desktop
system, and it seems to be running stable/13 fine. (the desktop was a
stable/12 to stable/13 upgrade).

thanks,


If you are not booting from zfs its seemless. Just remembe once you 
issue zpool upgrade you wont be able to go back to older version. If 
booting from ZFS, just update your boot loader.


I have updated a couple of systems from 12.2-RELEASE-p3 to 13.0-RC2. One 
using UEFI and the other using legacy bios with no problems.


for UEFI I mounted EFI Boot partition

EFI update, New ZFS data set was mounted at ROOT, zpool is mirror. 
Updated both EFI partitions in case primary goes offline.


zpool set bootfs=ssd/ROOT/13.0-RC2 ssd
mount -t msdosfs /dev/ada0p1 /mnt
cp ROOT/boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt
mount -t msdosfs /dev/ada1p1 /mnt
cp ROOT/boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt

Legacy Bios, again new ZFS data set was mounted at ROOT, and zpool is 
mirror, updated both boot partitions

zpool set bootfs=zroot/ROOT/13.0-RC2 zroot
gpart bootcode -b ROOT/boot/pmbr -p ROOT/boot/gptzfsboot -i 1 ada0
gpart bootcode -b ROOT/boot/pmbr -p ROOT/boot/gptzfsboot -i 1 ada1

Everything worked as expected.

--
Thanks,
   Dean E. Weimer
   http://www.dweimer.net/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Updating to 13-stable and existing ZFS pools: any gotchas?

[tech-lists]

On Sun, Mar 14, 2021 at 09:59:21AM +0100, Stefan Bethke wrote:
I'm planning to upgrade three production machines with existing ZFS pools 
to 13-stable. Is there anything I need to pay attention to wrt OpenZFS? 
Or should it be fully transparent, apart from updating loader?


My (limited) testing with VMs went without a hitch, but I want to make sure 
I don't paint myself into a corner.


Hi, I'm interested in this as well.

I'm not using root-on-zfs. The zpool is 5* spinning rust, the booting
media is ssd/ufs2. Is updating the bootcode only relevant for
root-on-zfs? I've not done that for a similarly configured desktop
system, and it seems to be running stable/13 fine. (the desktop was a
stable/12 to stable/13 upgrade).

thanks,
--
J.


signature.asc
Description: PGP signature

Re: Updating to 13-stable and existing ZFS pools: any gotchas?

[Mathias Picker]

I’ve updated two machines and had no problems, read the mailing 
lists and remember no problem apart from the boot loader.


Updated the boot loader before upgrading the pools, found the way 
for EFI is different then I remembered (mount EFI partition, copy 
/boot/loader.efi to it) but it worked just fine..


So, while I cannot 100% guarantee it’s without problems, I would 
expect no hiccups. And the new bootloader is only needed if/once 
you upgrade your pools, I actually run my laptop for a week with 
quite an old loader ;)


Cheers, Mathias

Stefan Bethke  writes:

I'm planning to upgrade three production machines with existing 
ZFS pools to 13-stable. Is there anything I need to pay 
attention to wrt OpenZFS? Or should it be fully transparent, 
apart from updating loader?


My (limited) testing with VMs went without a hitch, but I want 
to make sure I don't paint myself into a corner.



Stefan



--
Mathias Picker
Geschäftsführer

mathias.pic...@virtual-earth.de

virtual earth Gesellschaft für Wissens re/prä sentation mbH
http://www.virtual-earth.de/   HRB126870
supp...@virtual-earth.de   Westendstr. 142
089 / 1250 3943
___

freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Updating to 13-stable and existing ZFS pools: any gotchas?

[Stefan Bethke]

I'm planning to upgrade three production machines with existing ZFS pools to 
13-stable. Is there anything I need to pay attention to wrt OpenZFS? Or should 
it be fully transparent, apart from updating loader?

My (limited) testing with VMs went without a hitch, but I want to make sure I 
don't paint myself into a corner.


Stefan

--
Stefan BethkeFon +49 151 14070811



signature.asc
Description: Message signed with OpenPGP

Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA

[Yoshihiro Ota]

On Tue, 9 Mar 2021 11:24:53 +0200
Andriy Gapon  wrote:

> On 08/03/2021 05:24, Yoshihiro Ota wrote:
> > On Sun, 7 Mar 2021 00:09:33 +0200
> > Andriy Gapon  wrote:
> > 
> >> On 06/03/2021 20:09, Yoshihiro Ota wrote:
> >>> Hi all,
> >>>
> >>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
> >>>
> >>> After upgrading one in VMWare, 'zfs mount -a' hangs the system.
> >>> I don't have boottime zfs mount on nor don't have zfsroot.
> >>> I just simply ran install world/kernel and mergemaster.
> >>
> >> Please use procstat -kk to capture a kernel stack trace of the hung 
> >> process.
> > 
> > Actually, spining was 'kldload zfs'.
> > Console doesn't response but ping and sshd sessions still work.
> > procstat output is below.
> > In addition, this doesn't happen to systems that I've been following 
> > 13-CURRENT
> > but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC.
> > 
> > 
> > # procstat -kk 1049
> >   PIDTID COMMTDNAME  KSTACK 
> >   
> >  1049 100215 kldload -   spa_init+0xc6 
> > zfs_kmod_init+0x1a
> > zfs_modevent+0x34 module_register_init+0x8c linker_load_module+0xaab 
> > kern_kldload+0xc1
> > sys_kldload+0x50 syscall+0x17d g_ctx+0xe280bf29 
> > 
> 
> If you could use kgdb to find out what source code line spa_init+0xc6
> corresponds to that may help to see what's going on.

Hi Andriy,

Could you teach me what command can show such address in kgdb?

Thanks,
Hiro
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA

[Andriy Gapon]

On 08/03/2021 05:24, Yoshihiro Ota wrote:
> On Sun, 7 Mar 2021 00:09:33 +0200
> Andriy Gapon  wrote:
> 
>> On 06/03/2021 20:09, Yoshihiro Ota wrote:
>>> Hi all,
>>>
>>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
>>>
>>> After upgrading one in VMWare, 'zfs mount -a' hangs the system.
>>> I don't have boottime zfs mount on nor don't have zfsroot.
>>> I just simply ran install world/kernel and mergemaster.
>>
>> Please use procstat -kk to capture a kernel stack trace of the hung process.
> 
> Actually, spining was 'kldload zfs'.
> Console doesn't response but ping and sshd sessions still work.
> procstat output is below.
> In addition, this doesn't happen to systems that I've been following 
> 13-CURRENT
> but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC.
> 
> 
> # procstat -kk 1049
>   PIDTID COMMTDNAME  KSTACK   
> 
>  1049 100215 kldload -   spa_init+0xc6 
> zfs_kmod_init+0x1a zfs_modevent+0x34 module_register_init+0x8c 
> linker_load_module+0xaab kern_kldload+0xc1 sys_kldload+0x50 syscall+0x17d 
> g_ctx+0xe280bf29 
> 

If you could use kgdb to find out what source code line spa_init+0xc6
corresponds to that may help to see what's going on.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA

[Yoshihiro Ota]

On Sun, 7 Mar 2021 00:09:33 +0200
Andriy Gapon  wrote:

> On 06/03/2021 20:09, Yoshihiro Ota wrote:
> > Hi all,
> > 
> > I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
> > 
> > After upgrading one in VMWare, 'zfs mount -a' hangs the system.
> > I don't have boottime zfs mount on nor don't have zfsroot.
> > I just simply ran install world/kernel and mergemaster.
> 
> Please use procstat -kk to capture a kernel stack trace of the hung process.

Actually, spining was 'kldload zfs'.
Console doesn't response but ping and sshd sessions still work.
procstat output is below.
In addition, this doesn't happen to systems that I've been following 13-CURRENT
but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC.


# procstat -kk 1049
  PIDTID COMMTDNAME  KSTACK 
  
 1049 100215 kldload -   spa_init+0xc6 
zfs_kmod_init+0x1a zfs_modevent+0x34 module_register_init+0x8c 
linker_load_module+0xaab kern_kldload+0xc1 sys_kldload+0x50 syscall+0x17d 
g_ctx+0xe280bf29 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs mount -a spins the system after upgrading from 12.2 to 13-BETA

[Andriy Gapon]

On 06/03/2021 20:09, Yoshihiro Ota wrote:
> Hi all,
> 
> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
> 
> After upgrading one in VMWare, 'zfs mount -a' hangs the system.
> I don't have boottime zfs mount on nor don't have zfsroot.
> I just simply ran install world/kernel and mergemaster.

Please use procstat -kk to capture a kernel stack trace of the hung process.


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

zfs mount -a spins the system after upgrading from 12.2 to 13-BETA

[Yoshihiro Ota]

Hi all,

I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.

After upgrading one in VMWare, 'zfs mount -a' hangs the system.
I don't have boottime zfs mount on nor don't have zfsroot.
I just simply ran install world/kernel and mergemaster.

Are there any special steps we need to take before upgrading?

Thanks,
Hiro
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: lots of "no such file or directory" errors in zfs filesystem

[Peter Jeremy via freebsd-stable]

On 2021-Feb-23 11:30:58 -0600, Chris Anderson  wrote:
>nope, it led a pretty boring life. that zfs filesystem was created on that
>server and has been on the same two mirrored disks for its lifetime.

Does the server have ECC RAM?  Possibly it's a bitflip somewhere before
the data got to disk.

>prior to the upgrades) the server does have a relatively modest amount of
>ram (2GB). dunno if that makes it more likely that these kinds of issues
>get triggered.

Low amounts of RAM are going to increase the IO load but shouldn't
otherwise impact the filesystem consistency.  I have a FreeBSD test
system that's running ZFS in <1GB RAM and rebuilding itself daily for
multiple years and haven't run into any ZFS corruption issues.

-- 
Peter Jeremy


signature.asc
Description: PGP signature

Re: lots of "no such file or directory" errors in zfs filesystem

[Chris Anderson]

On Tue, Feb 23, 2021 at 4:53 AM Andriy Gapon  wrote:

> On 23/02/2021 05:25, Chris Anderson wrote:
> > so I can't ls -i the file since that triggers the no such file warning.
> if I run
> > zdb - on the inode of a directory which contains one of those
> missing files,
> > I can get the inode of the file from that, but I don't get anything
> particularly
> > interesting in the output.
> >
> > most of the files that are missing are in directories with a large
> number of
> > files (the largest has 180k) but I managed to find a directory which had
> a
> > single file entry that is missing:
> >
> > Dataset tank/home/cva [ZPL], ID 196, cr_txg 163, 109G, 908537 objects,
> rootbp
> > DVA[0]=<0:13210311000:1000> DVA[1]=<0:18b9a02c000:1000> [L0 DMU objset]
> > fletcher4 uncompressed LE contiguous unique double size=800L/800P
> > birth=46916371L/46916371P fill=908537
> > cksum=11fdd21d1d:13cb24c87a6e:da0c9bf1b5df3:715ab2ec45b7b09
> >
> >
> >     Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
> >
> >  382681   128K 1K  0512 1K  100.00  ZFS directory
> >
> >264   bonus  ZFS znode
> >
> > dnode flags: USED_BYTES USERUSED_ACCOUNTED
> >
> > dnode maxblkid: 0
> >
> > uid 1001
> >
> > gid 1001
> >
> > atime   Sun Aug  6 02:00:41 2017
> >
> > mtime   Wed Apr 15 12:12:42 2020
> >
> > ctime   Wed Apr 15 12:12:42 2020
> >
> > crtime  Sat Aug  5 15:10:07 2017
> >
> > gen 23881023
> >
> > mode40755
> >
> > size3
> >
> > parent  38176
> >
> > links   2
> >
> > pflags  4080144
> >
> > xattr   0
> >
> > rdev0x
> >
> > microzap: 1024 bytes, 1 entries
> >
> >
> >
> > hash_test.go = 38274 (type: Regular File)
> >
> >
> > # zdb - tank/home/cva 38274
> >
> > Dataset tank/home/cva [ZPL], ID 196, cr_txg 163, 109G, 908537 objects,
> rootbp
> > DVA[0]=<0:13210311000:1000> DVA[1]=<0:18b9a02c000:1000> [L0 DMU objset]
> > fletcher4 uncompressed LE contiguous unique double size=800L/800P
> > birth=46916371L/46916371P fill=908537
> > cksum=11fdd21d1d:13cb24c87a6e:da0c9bf1b5df3:715ab2ec45b7b09
> >
> >
> > Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
> >
> > zdb: dmu_bonus_hold(38274) failed, errno 2
>
> So, this looks like a "simple" problem.
> Unfortunately, it is very hard to tell retrospectively what bug caused it.
> The directory has an entry for the file, but the file does not actually
> exist
> (or has a different ID).
> This is a logical inconsistency, not a data integrity issue.
> So, a scrub, being a data integrity check, would not detect such an issue.
> Hypothetical zfs_fsck is needed to find and repair such logical problems.
>

ah, I see. that makes sense.


> Does that pool and filesystem have any special history?
> I mean upgrades, replication via send/recv, moving between OS-s, etc.
>

nope, it led a pretty boring life. that zfs filesystem was created on that
server and has been on the same two mirrored disks for its lifetime. it has
had freebsd upgrades applied as they became available. zfs upgrades were
for the most part avoided until quite recently (though the problem existed
prior to the upgrades) the server does have a relatively modest amount of
ram (2GB). dunno if that makes it more likely that these kinds of issues
get triggered.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: lots of "no such file or directory" errors in zfs filesystem

[Andriy Gapon]

On 23/02/2021 05:25, Chris Anderson wrote:
> so I can't ls -i the file since that triggers the no such file warning. if I 
> run
> zdb - on the inode of a directory which contains one of those missing 
> files,
> I can get the inode of the file from that, but I don't get anything 
> particularly
> interesting in the output.
> 
> most of the files that are missing are in directories with a large number of
> files (the largest has 180k) but I managed to find a directory which had a
> single file entry that is missing:
> 
> Dataset tank/home/cva [ZPL], ID 196, cr_txg 163, 109G, 908537 objects, rootbp
> DVA[0]=<0:13210311000:1000> DVA[1]=<0:18b9a02c000:1000> [L0 DMU objset]
> fletcher4 uncompressed LE contiguous unique double size=800L/800P
> birth=46916371L/46916371P fill=908537
> cksum=11fdd21d1d:13cb24c87a6e:da0c9bf1b5df3:715ab2ec45b7b09
> 
> 
>     Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
> 
>      38268    1   128K     1K      0    512     1K  100.00  ZFS directory
> 
>                                                264   bonus  ZFS znode
> 
>         dnode flags: USED_BYTES USERUSED_ACCOUNTED 
> 
>         dnode maxblkid: 0
> 
>         uid     1001
> 
>         gid     1001
> 
>         atime   Sun Aug  6 02:00:41 2017
> 
>         mtime   Wed Apr 15 12:12:42 2020
> 
>         ctime   Wed Apr 15 12:12:42 2020
> 
>         crtime  Sat Aug  5 15:10:07 2017
> 
>         gen     23881023
> 
>         mode    40755
> 
>         size    3
> 
>         parent  38176
> 
>         links   2
> 
>         pflags  4080144
> 
>         xattr   0
> 
>         rdev    0x
> 
>         microzap: 1024 bytes, 1 entries
> 
>         
> 
>                 hash_test.go = 38274 (type: Regular File)
> 
> 
> # zdb - tank/home/cva 38274
> 
> Dataset tank/home/cva [ZPL], ID 196, cr_txg 163, 109G, 908537 objects, rootbp
> DVA[0]=<0:13210311000:1000> DVA[1]=<0:18b9a02c000:1000> [L0 DMU objset]
> fletcher4 uncompressed LE contiguous unique double size=800L/800P
> birth=46916371L/46916371P fill=908537
> cksum=11fdd21d1d:13cb24c87a6e:da0c9bf1b5df3:715ab2ec45b7b09
> 
> 
>     Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
> 
> zdb: dmu_bonus_hold(38274) failed, errno 2

So, this looks like a "simple" problem.
Unfortunately, it is very hard to tell retrospectively what bug caused it.
The directory has an entry for the file, but the file does not actually exist
(or has a different ID).
This is a logical inconsistency, not a data integrity issue.
So, a scrub, being a data integrity check, would not detect such an issue.
Hypothetical zfs_fsck is needed to find and repair such logical problems.

Does that pool and filesystem have any special history?
I mean upgrades, replication via send/recv, moving between OS-s, etc.


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: lots of "no such file or directory" errors in zfs filesystem

[Chris Anderson]

On Mon, Feb 22, 2021 at 9:13 AM Andriy Gapon  wrote:

> On 22/02/2021 16:20, Chris Anderson wrote:
> > On Mon, Feb 22, 2021 at 1:36 AM Andriy Gapon  > <mailto:a...@freebsd.org>> wrote:
> >
> > On 22/02/2021 09:31, Chris Anderson wrote:
> > > None of these files are especially important to me, however I was
> wondering
> > > if there would be any benefit to the community from trying to
> debug this
> > > issue further to understand what might be going wrong.
> >
> > Yes.
> >
> >
> > Could you offer any guidance about what kind of debugging information I
> could
> > collect that would be of use?
>
> You can start with picking a single file that demonstrates the problem.
> Then,
> ls -li the-file
> zdb - file's-filesystem file's-inode-number
> The filesystem can be found out from df output, the inode number is in ls
> -li
> output -- if the command prints anything at all.
> If it does not, then do ls -lid on the file's directory and then zdb -
> for
> the directory's inode number.  In the output there should be the file name
> and
> its number (I think that it's in hex, but not sure).
>

so I can't ls -i the file since that triggers the no such file warning. if
I run zdb - on the inode of a directory which contains one of those
missing files, I can get the inode of the file from that, but I don't get
anything particularly interesting in the output.

most of the files that are missing are in directories with a large number
of files (the largest has 180k) but I managed to find a directory which had
a single file entry that is missing:

Dataset tank/home/cva [ZPL], ID 196, cr_txg 163, 109G, 908537 objects,
rootbp DVA[0]=<0:13210311000:1000> DVA[1]=<0:18b9a02c000:1000> [L0 DMU
objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P
birth=46916371L/46916371P fill=908537
cksum=11fdd21d1d:13cb24c87a6e:da0c9bf1b5df3:715ab2ec45b7b09


Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type

 382681   128K 1K  0512 1K  100.00  ZFS directory

   264   bonus  ZFS znode

dnode flags: USED_BYTES USERUSED_ACCOUNTED

dnode maxblkid: 0

uid 1001

gid 1001

atime   Sun Aug  6 02:00:41 2017

mtime   Wed Apr 15 12:12:42 2020

ctime   Wed Apr 15 12:12:42 2020

crtime  Sat Aug  5 15:10:07 2017

gen 23881023

mode40755

size3

parent  38176

links   2

pflags  4080144

xattr   0

rdev0x

microzap: 1024 bytes, 1 entries



hash_test.go = 38274 (type: Regular File)


# zdb - tank/home/cva 38274

Dataset tank/home/cva [ZPL], ID 196, cr_txg 163, 109G, 908537 objects,
rootbp DVA[0]=<0:13210311000:1000> DVA[1]=<0:18b9a02c000:1000> [L0 DMU
objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P
birth=46916371L/46916371P fill=908537
cksum=11fdd21d1d:13cb24c87a6e:da0c9bf1b5df3:715ab2ec45b7b09


Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type

zdb: dmu_bonus_hold(38274) failed, errno 2


>
> --
> Andriy Gapon
>
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: lots of "no such file or directory" errors in zfs filesystem

[Andriy Gapon]

On 22/02/2021 16:20, Chris Anderson wrote:
> On Mon, Feb 22, 2021 at 1:36 AM Andriy Gapon  > wrote:
> 
> On 22/02/2021 09:31, Chris Anderson wrote:
> > None of these files are especially important to me, however I was 
> wondering
> > if there would be any benefit to the community from trying to debug this
> > issue further to understand what might be going wrong.
> 
> Yes.
> 
> 
> Could you offer any guidance about what kind of debugging information I could
> collect that would be of use?

You can start with picking a single file that demonstrates the problem.
Then,
ls -li the-file
zdb - file's-filesystem file's-inode-number
The filesystem can be found out from df output, the inode number is in ls -li
output -- if the command prints anything at all.
If it does not, then do ls -lid on the file's directory and then zdb - for
the directory's inode number.  In the output there should be the file name and
its number (I think that it's in hex, but not sure).


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: lots of "no such file or directory" errors in zfs filesystem

[Chris Anderson]

On Mon, Feb 22, 2021 at 1:36 AM Andriy Gapon  wrote:

> On 22/02/2021 09:31, Chris Anderson wrote:
> > None of these files are especially important to me, however I was
> wondering
> > if there would be any benefit to the community from trying to debug this
> > issue further to understand what might be going wrong.
>
> Yes.
>

Could you offer any guidance about what kind of debugging information I
could collect that would be of use?


>
> --
> Andriy Gapon
> ___
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
>
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: lots of "no such file or directory" errors in zfs filesystem

[Andriy Gapon]

On 22/02/2021 09:31, Chris Anderson wrote:
> None of these files are especially important to me, however I was wondering
> if there would be any benefit to the community from trying to debug this
> issue further to understand what might be going wrong.

Yes.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

lots of "no such file or directory" errors in zfs filesystem

[Chris Anderson]

I'm in the process of decommissioning an old zfs based file server and I
noticed that around a dozen files with directory entries which fail with
"No such file or directory" when trying to read them.

I can't remember what the original version of freebsd installed was, but
it's been in production for at least 7 years and has been upgraded with
freebsd-update as new FreeBSD releases came available (it is currently on
12.2-RELEASE-p3).

The behavior is perplexing since I've never had any scrub failures on the
pool those files reside in yet from looking at old security run outputs,
the number of files in that state has increased over time.

None of these files are especially important to me, however I was wondering
if there would be any benefit to the community from trying to debug this
issue further to understand what might be going wrong.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: stable/13 and zfs <> openzfs

[tech-lists]

On Tue, Feb 09, 2021 at 03:40:43PM -0700, Alan Somers wrote:


The new ZFS is backwards compatible with the old one.  So your 12.2-p3
system will be able to zfs send, and the stable/13 will be able to zfs
recv.  You can go the other direction too, if you're careful to create the
new pool using the old format, and don't turn on any of the new features.

/etc/rc.conf shouldn't need any changes.

Your upgrade plan sounds fine.  But for the "backup all data" step, I would
recommend using zfs send/recv to do the backup.  That will do a better job
of ensuring your files' integrity than something like rsync or cp.


Hi, thanks for confirming.
--
J.


signature.asc
Description: PGP signature

Re: stable/13 and zfs <> openzfs

[Alan Somers]

On Tue, Feb 9, 2021 at 3:28 PM tech-lists  wrote:

> Hi,
>
> I need to build a new zpool (arch=amd64). The machine needs upgrading
> from stable/12 to stable/13. The process I envisaged following is:
>
> 1. backup all data to a non-zpool disk
> 2. upgrade the OS stable/12 to stable/13
> 3. destroy the original zpool [1]
> 4. build new zpool and restore data
>
> stable/13 seems to have openzfs as the default zfs. Is there anything
> more I need do? [2] The system is havdling remote zfs receive from a
> 12.2-p3 system that uses the earlier default FreeBSD zfs. [3]
>
> [1] the original zpool was going to be destroyed anyway due to wrong
> ashift for these new disks.
>
> [2] like, does /etc/rc.conf need modifying? Things like that.
>
> [3] is the newer openzfs compatible with the older one? I mean will the
> older one still be able to send zfs snapshots for backup, or does
> something need to be changed at either end?
>
> thanks,
> --
> J.
>

The new ZFS is backwards compatible with the old one.  So your 12.2-p3
system will be able to zfs send, and the stable/13 will be able to zfs
recv.  You can go the other direction too, if you're careful to create the
new pool using the old format, and don't turn on any of the new features.

/etc/rc.conf shouldn't need any changes.

Your upgrade plan sounds fine.  But for the "backup all data" step, I would
recommend using zfs send/recv to do the backup.  That will do a better job
of ensuring your files' integrity than something like rsync or cp.

-Alan
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

stable/13 and zfs <> openzfs

[tech-lists]

Hi,

I need to build a new zpool (arch=amd64). The machine needs upgrading
from stable/12 to stable/13. The process I envisaged following is:

1. backup all data to a non-zpool disk
2. upgrade the OS stable/12 to stable/13
3. destroy the original zpool [1]
4. build new zpool and restore data

stable/13 seems to have openzfs as the default zfs. Is there anything
more I need do? [2] The system is havdling remote zfs receive from a 
12.2-p3 system that uses the earlier default FreeBSD zfs. [3]


[1] the original zpool was going to be destroyed anyway due to wrong
ashift for these new disks.

[2] like, does /etc/rc.conf need modifying? Things like that.

[3] is the newer openzfs compatible with the older one? I mean will the
older one still be able to send zfs snapshots for backup, or does
something need to be changed at either end?

thanks,
--
J.


signature.asc
Description: PGP signature

Re: 12.2-RELEASE buildworld fail at cddl/zfs / libuutil.so

[Tomasz CEDRO]

On 27.01.2021 19:31, parv/freebsd wrote:
# git log --pretty=fuller --grep libuutil release/12.2.0. 
.stable/12


commit 30ec3f368d
Author:     Eugene Grosbein 
AuthorDate: Dec.2020.1206-1622 +
Commit:     Eugene Grosbein 
CommitDate: Dec.2020.1206-1622 +

     MFC r364027 by arichardson: Fix linker error in libuutil with 
recent LLVM


     This also fixes nanobsd-style build (cross-compiling).
     Original commit log:

       Not marking the function as static can result in a linker error:
       undefined reference to __assfail [--no-allow-shlib-undefined]
       I noticed this error after updating our CHERI LLVM to the latest 
upstream

       LLVM HEAD revision.

       This change effectively reverts r329984 and marks 
dmu_buf_init_user as

       static (which keeps the GCC build happy).
... above seems to (as I haven't bisected) be the fix in 12-STABLE which 
allows

the build to finish with success.
   - parv


Whoah! Mystery solved! This commit did not jump into the releng/12.2 yet 
it seems. Thank you PARV!! :-)


--
CeDeROM, SQ7MHZ, https://www.tomek.cedro.info
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 12.2-RELEASE buildworld fail at cddl/zfs / libuutil.so

[parv/freebsd]

On Wed, Jan 27, 2021 at 6:10 AM Tomasz CEDRO wrote:

Hi Tomasz,

On 27.01.2021 12:52, parv/freebsd wrote:
>
...

> > # /usr/bin/time -h make buildkernel buildworld NO_CLEAN=1
> > .   ^ ^ ^ ^ ^ ^
> > Perhaps try with cleaning out first buildworld, then kernel?
>
> Well I did remove all from /usr/obj, build failed, then checked out
> /usr/src from svn, build failed, then checked out /usr/src from git, and
> it failed.
>
> Just to make sure this is not a local environment problem I have
> switched to stable/12 branch (that is 12-STABLE) and it builds fine!!
>

Thanks for the clue ...

# git log --pretty=fuller --grep libuutil release/12.2.0..stable/12

commit 30ec3f368d
Author: Eugene Grosbein 
AuthorDate: Dec.2020.1206-1622 +
Commit: Eugene Grosbein 
CommitDate: Dec.2020.1206-1622 +

MFC r364027 by arichardson: Fix linker error in libuutil with recent
LLVM

This also fixes nanobsd-style build (cross-compiling).
Original commit log:

  Not marking the function as static can result in a linker error:
  undefined reference to __assfail [--no-allow-shlib-undefined]
  I noticed this error after updating our CHERI LLVM to the latest
upstream
  LLVM HEAD revision.

  This change effectively reverts r329984 and marks dmu_buf_init_user as
  static (which keeps the GCC build happy).
... above seems to (as I haven't bisected) be the fix in 12-STABLE which
allows
the build to finish with success.

  - parv



> # git branch
>main
>releng/12.2
> * stable/12
>
...

> # /usr/bin/time -h make buildkernel buildworld release -j12
>
...

> --- kernel.full ---
> linking kernel.full
> ctfmerge -L VERSION -g -o kernel.full ...
>text  data   bssdec hex   filename
>23861318   1836970   3798784   29497072   0x1c216f0   kernel.full
> --- kernel.debug ---
> objcopy --only-keep-debug kernel.full kernel.debug
> --- kernel ---
> objcopy --strip-debug --add-gnu-debuglink=kernel.debug  kernel.full kernel
> --
>  >>> Kernel build for GENERIC completed on Wed Jan 27 08:03:22 CET 2021
> --
> `release' is up to date.
>  1h16m12.69s real5h45m44.19s user
>   28m53.59s sys
>
>
> Did anyone manage to build 12.2-RELEASE from repo? :-)
>

--
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: 12.2-RELEASE buildworld fail at cddl/zfs / libuutil.so

[Tomasz CEDRO]

Hello PARV :-)

On 27.01.2021 12:52, parv/freebsd wrote:
> What is the git commit hash (for others)?

For the 12.2-RELEASE:

# git branch
  main
* releng/12.2
  stable/12

# git log --pretty=oneline | head
97eb1441ca7b602510a81fd512d830563998b3e7 PS/2 Synaptics touchpad 
identification fix.
2120d07af09cb830873554ba5405c5d3e51b41cc Fix OpenSSL NULL pointer 
de-reference.
fc825298b84729facaced36d32c3fc8d111648c1 Add UPDATING entries and bump 
version.
85b4a006794d02f6e0e2f53bca3782732932a7c2 Fix multiple vulnerabilities in 
rtsold.
229d311aa42306953eb45bc55a8c5ca97312025e Fix ICMPv6 use-after-free in 
error message handling.
7b3386dba21f57907931b2e6f09f741633ea3349 Fix race condition in callout 
CPU migration.

0b4300a472c89506709341530788ad9815c05bd3 Fix uninitialized variable in ipfw.
d20c61bbc2792d5c256d4d3a1c62156ccc9eb66a Update timezone database 
information.
30710f108d29a61635778e82f300781a097caec4 Fix execve/fexecve system call 
auditing.
1d21fe9cea3f530b4cfca495632d5a1595a32270 - Switch releng/12.2 from RC3 
to RELEASE. - Add the anticipated 12.2-RELEASE date to UPDATING.  Fix 
a missing colon in the previous UPDATING entry while here. - Set a 
static __FreeBSD_version.


The first one is a simple one liner that I want to test for PS/2 
Synaptics mux identification fix.





# /usr/bin/time -h make buildkernel buildworld NO_CLEAN=1
.   ^ ^ ^ ^ ^ ^
Perhaps try with cleaning out first buildworld, then kernel?


Well I did remove all from /usr/obj, build failed, then checked out 
/usr/src from svn, build failed, then checked out /usr/src from git, and 
it failed.


Just to make sure this is not a local environment problem I have 
switched to stable/12 branch (that is 12-STABLE) and it builds fine!!


# git branch
  main
  releng/12.2
* stable/12

# git log --pretty=oneline | head
f56d7f838f5b3aa0f55b10406eaa7eb760a3ba18 elfctl: prefix disable flags 
with "no"
d7e23b5cdd8465bd50f88b1e38cb695a361a26f5 elfctl: allow features to be 
specified by value
49d3dcb041f058880486e3489ca79c9476ac7abf elfctl: add backwards 
compatibility for "no" prefixes
70cdab054c8ffe7f5561ab8f31db33ee62d12c9c rctl(4): support throttling 
resource usage to 0
7baf7a45385962d06cd2a2213109b910d40a7936 ndis: Per user request, delay 
removal to 14
2a0be7b0331c5245418cdf2893c8ba6b42a0afa5 VFS_QUOTACTL: Remove needless 
casts of arg

085997e8f05e69c66a8ffc0804b275a386ba1ca8 contrib/tzdata: import tzdata 2021a
5071cbacfb343f024bc1c9969aa43d20daa8241a safexcel: Simplify request 
allocation
804d63c0ee8f56dfac29e1fe5879740a3549d0cb safexcel: Handle command/result 
descriptor exhaustion gracefully
24ee73604483152baff9e5e3cf61047eb1c3 safexcel: Add counters for some 
resource exhaustion conditions


# /usr/bin/time -h make buildkernel buildworld release -j12
(..)
cc -target x86_64-unknown-freebsd12.2 
--sysroot=/usr/obj/usr/src/amd64.amd64/tmp 
-B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O2 -pipe 
-fno-strict-aliasing  -g -nostdinc  -I. -I/usr/src/sys 
-I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt 
-D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common 
 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer 
-fdebug-prefix-map=./machine=/usr/src/sys/amd64/include 
-fdebug-prefix-map=./x86=/usr/src/sys/x86/include -mcmodel=kernel 
-mno-red-zone -mno-mmx -mno-sse -msoft-float 
-fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector 
-gdwarf-2 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes 
-Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef 
-Wno-pointer-sign -D__printf__=__freebsd_kprintf__ 
-Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas 
-Wno-error-tautological-compare -Wno-error-empty-body 
-Wno-error-parentheses-equality -Wno-error-unused-function 
-Wno-error-pointer-sign -Wno-error-shift-negative-value 
-Wno-address-of-packed-member -Wno-format-zero-length  -mno-aes -mno-avx 
 -std=iso9899:1999 -Werror  vers.c

ctfconvert -L VERSION -g vers.o
--- kernel.full ---
linking kernel.full
ctfmerge -L VERSION -g -o kernel.full ...
  text  data   bssdec hex   filename
  23861318   1836970   3798784   29497072   0x1c216f0   kernel.full
--- kernel.debug ---
objcopy --only-keep-debug kernel.full kernel.debug
--- kernel ---
objcopy --strip-debug --add-gnu-debuglink=kernel.debug  kernel.full kernel
--
>>> Kernel build for GENERIC completed on Wed Jan 27 08:03:22 CET 2021
--
`release' is up to date.
1h16m12.69s real5h45m44.19s user 
 28m53.59s sys



Did anyone manage to build 12.2-RELEASE from repo? :-)

--
CeDeROM, SQ7MHZ, https://www.tomek.cedro.info
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to 

Re: 12.2-RELEASE buildworld fail at cddl/zfs / libuutil.so

[parv/freebsd]

 On Tue, Jan 26, 2021 at 7:45 PM Tomasz CEDRO wrote:

> I cannot buildworld for 12.2-RELEASE. Using /usr/src as svn or git no
> matter (its set to releng/12.2). Did clean /usr/obj. Any hints welcome :-)
>
> # uname -a
> FreeBSD hexagon 12.2-RELEASE-p1 FreeBSD 12.2-RELEASE-p1 GENERIC  amd64
>

What is the git commit hash (for others)?

I had recently compiled 12-STABLE (c243308-ge82353584c58) without issues;
running
as guest in VirtualBox on MS Windows 10.



> # /usr/bin/time -h make buildkernel buildworld NO_CLEAN=1
>
.   ^ ^ ^ ^ ^ ^
Perhaps try with cleaning out first buildworld, then kernel?

  - parv

(..)
> ===> bin/tests (all)
> ===> cddl (all)
> ===> cddl/lib (all)
> ===> cddl/lib/drti (all)
> ===> cddl/lib/libavl (all)
> ===> cddl/lib/libctf (all)
> ===> cddl/lib/libdtrace (all)
> ===> cddl/lib/libnvpair (all)
> ===> cddl/lib/libumem (all)
> ===> cddl/lib/libuutil (all)
> ===> cddl/lib/libzfs_core (all)
> ===> cddl/lib/libzfs (all)
> ===> cddl/lib/libzpool (all)
> ===> cddl/lib/tests (all)
> ===> cddl/sbin (all)
> ===> cddl/sbin/zfs (all)
> cc -target x86_64-unknown-freebsd12.2
> --sysroot=/usr/obj/usr/src/amd64.amd64/tmp
> -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -O2 -pipe -fno-common
> -I/usr/src/cddl/contrib/opensolaris/lib/libzpool/common
> -I/usr/src/cddl/compat/opensolaris/include
> -I/usr/src/cddl/compat/opensolaris/lib/libumem
> -I/usr/src/sys/cddl/compat/opensolaris
> -I/usr/src/cddl/contrib/opensolaris/head
> -I/usr/src/cddl/contrib/opensolaris/lib/libuutil/common
> -I/usr/src/cddl/contrib/opensolaris/lib/libzfs/common
> -I/usr/src/cddl/contrib/opensolaris/lib/libzfs_core/common
> -I/usr/src/cddl/contrib/opensolaris/lib/libumem/common
> -I/usr/src/cddl/contrib/opensolaris/lib/libnvpair
> -I/usr/src/sys/cddl/contrib/opensolaris/uts/common
> -I/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs
> -I/usr/src/sys/cddl/contrib/opensolaris/common/zfs
> -DNEED_SOLARIS_BOOLEAN -g -O0 -std=gnu99 -fstack-protector-strong
> -Wno-pointer-sign -Wno-unknown-pragmas -Wno-empty-body
> -Wno-string-plus-int -Wno-unused-const-variable
> -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality
> -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef
> -Wno-address-of-packed-member -Wno-switch -Wno-switch-enum
> -Wno-knr-promoted-parameter -Wno-parentheses -Qunused-arguments   -o
> zfs.full zfs_main.o zfs_iter.o   -ljail  -lnvpair  -luutil  -lzfs_core
> -lzfs
> ld: error: /usr/obj/usr/src/amd64.amd64/tmp/usr/lib/libuutil.so:
> undefined reference to __assfail
> cc: error: linker command failed with exit code 1 (use -v to see
> invocation)
> *** Error code 1
>
> Stop.
> make[5]: stopped in /usr/src/cddl/sbin/zfs
> *** Error code 1
> (..)
>

--
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

12.2-RELEASE buildworld fail at cddl/zfs / libuutil.so

[Tomasz CEDRO]

Hello world!

I cannot buildworld for 12.2-RELEASE. Using /usr/src as svn or git no 
matter (its set to releng/12.2). Did clean /usr/obj. Any hints welcome :-)


# uname -a
FreeBSD hexagon 12.2-RELEASE-p1 FreeBSD 12.2-RELEASE-p1 GENERIC  amd64

# /usr/bin/time -h make buildkernel buildworld NO_CLEAN=1
(..)
===> bin/tests (all)
===> cddl (all)
===> cddl/lib (all)
===> cddl/lib/drti (all)
===> cddl/lib/libavl (all)
===> cddl/lib/libctf (all)
===> cddl/lib/libdtrace (all)
===> cddl/lib/libnvpair (all)
===> cddl/lib/libumem (all)
===> cddl/lib/libuutil (all)
===> cddl/lib/libzfs_core (all)
===> cddl/lib/libzfs (all)
===> cddl/lib/libzpool (all)
===> cddl/lib/tests (all)
===> cddl/sbin (all)
===> cddl/sbin/zfs (all)
cc -target x86_64-unknown-freebsd12.2 
--sysroot=/usr/obj/usr/src/amd64.amd64/tmp 
-B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin -O2 -pipe -fno-common 
-I/usr/src/cddl/contrib/opensolaris/lib/libzpool/common 
-I/usr/src/cddl/compat/opensolaris/include 
-I/usr/src/cddl/compat/opensolaris/lib/libumem 
-I/usr/src/sys/cddl/compat/opensolaris 
-I/usr/src/cddl/contrib/opensolaris/head 
-I/usr/src/cddl/contrib/opensolaris/lib/libuutil/common 
-I/usr/src/cddl/contrib/opensolaris/lib/libzfs/common 
-I/usr/src/cddl/contrib/opensolaris/lib/libzfs_core/common 
-I/usr/src/cddl/contrib/opensolaris/lib/libumem/common 
-I/usr/src/cddl/contrib/opensolaris/lib/libnvpair 
-I/usr/src/sys/cddl/contrib/opensolaris/uts/common 
-I/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs 
-I/usr/src/sys/cddl/contrib/opensolaris/common/zfs 
-DNEED_SOLARIS_BOOLEAN -g -O0 -std=gnu99 -fstack-protector-strong 
-Wno-pointer-sign -Wno-unknown-pragmas -Wno-empty-body 
-Wno-string-plus-int -Wno-unused-const-variable 
-Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality 
-Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef 
-Wno-address-of-packed-member -Wno-switch -Wno-switch-enum 
-Wno-knr-promoted-parameter -Wno-parentheses -Qunused-arguments   -o 
zfs.full zfs_main.o zfs_iter.o   -ljail  -lnvpair  -luutil  -lzfs_core 
-lzfs
ld: error: /usr/obj/usr/src/amd64.amd64/tmp/usr/lib/libuutil.so: 
undefined reference to __assfail

cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

Stop.
make[5]: stopped in /usr/src/cddl/sbin/zfs
*** Error code 1
(..)
39m18.71s real  33m45.09s user  4m10.23s sys

--
CeDeROM, SQ7MHZ, https://www.tomek.cedro.info
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs panic RELENG_12

[mike tancsa]

On 12/22/2020 10:09 AM, mike tancsa wrote:
> On 12/22/2020 10:07 AM, Mark Johnston wrote:
>> Could you go to frame 11 and print zone->uz_name and
>> bucket->ub_bucket[18]?  I'm wondering if the item pointer was mangled
>> somehow.
> Thank you for looking!
>
> (kgdb) frame 11
>
> #11 0x80ca47d4 in bucket_drain (zone=0xf800037da000,
> bucket=0xf801c7fd5200) at /usr/src/sys/vm/uma_core.c:758
> 758 zone->uz_release(zone->uz_arg, bucket->ub_bucket,
> bucket->ub_cnt);
> (kgdb) p zone->uz_name
> $1 = 0x8102118a "mbuf_jumbo_9k"
> (kgdb) p bucket->ub_bucket[18]
> $2 = (void *) 0xf80de4654000
> (kgdb) p bucket->ub_bucket   
> $3 = 0xf801c7fd5218
>
> (kgdb)
>
Not sure if its coincidence or not, but previously I was running with
arc being limited to ~30G of the 64G of RAM on the box.  I removed that
limit a few weeks ago after upgrading the box to RELENG_12 to pull in
the OpenSSL changes.  The panic seems to happen under disk load. I have
3 zfs pools that are pretty busy receiving snapshots. One day a week, we
write a full set to a 4th zfs pool off some geli attached drives via USB
for offsite cold storage.  The crashes happened with that extra level of
disk work.  gstat shows most of the 12 drives off 2 mrsas controllers at
or close to 100% busy during the 18hrs it takes to dump out the files.

Trying a new cold storage run now with the arc limit back to
vfs.zfs.arc_max=29334498304

    ---Mike



___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs panic RELENG_12

[mike tancsa]

On 12/22/2020 10:07 AM, Mark Johnston wrote:
>
> Could you go to frame 11 and print zone->uz_name and
> bucket->ub_bucket[18]?  I'm wondering if the item pointer was mangled
> somehow.

Thank you for looking!

(kgdb) frame 11

#11 0x80ca47d4 in bucket_drain (zone=0xf800037da000,
bucket=0xf801c7fd5200) at /usr/src/sys/vm/uma_core.c:758
758 zone->uz_release(zone->uz_arg, bucket->ub_bucket,
bucket->ub_cnt);
(kgdb) p zone->uz_name
$1 = 0x8102118a "mbuf_jumbo_9k"
(kgdb) p bucket->ub_bucket[18]
$2 = (void *) 0xf80de4654000
(kgdb) p bucket->ub_bucket   
$3 = 0xf801c7fd5218

(kgdb)

    ---Mike

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs panic RELENG_12

[Mark Johnston]

On Tue, Dec 22, 2020 at 09:05:01AM -0500, mike tancsa wrote:
> Hmmm, another one. Not sure if this is hardware as it seems different ?
> 
> 
> 
> Fatal trap 12: page fault while in kernel mode
> cpuid = 11; apic id = 0b
> fault virtual address   = 0x0
> fault code  = supervisor write data, page not present
> instruction pointer = 0x20:0x80ca0826
> stack pointer   = 0x28:0xfe00bc0f8540
> frame pointer   = 0x28:0xfe00bc0f8590
> code segment    = base 0x0, limit 0xf, type 0x1b
>     = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags    = interrupt enabled, resume, IOPL = 0
> current process = 33 (dom0)
> trap number = 12
> panic: page fault
> cpuid = 11
> time = 1608641071
> KDB: stack backtrace:
> #0 0x80a3fe85 at kdb_backtrace+0x65
> #1 0x809f406b at vpanic+0x17b
> #2 0x809f3ee3 at panic+0x43
> #3 0x80e3fe71 at trap_fatal+0x391
> #4 0x80e3fecf at trap_pfault+0x4f
> #5 0x80e3f516 at trap+0x286
> #6 0x80e19318 at calltrap+0x8
> #7 0x80ca47d4 at bucket_cache_drain+0x134
> #8 0x80c9e302 at zone_drain_wait+0xa2
> #9 0x80ca2bbd at uma_reclaim_locked+0x6d
> #10 0x80ca2af4 at uma_reclaim+0x34
> #11 0x80cc5321 at vm_pageout_worker+0x421
> #12 0x80cc4ee3 at vm_pageout+0x193
> #13 0x809b55be at fork_exit+0x7e
> #14 0x80e1a34e at fork_trampoline+0xe
> Uptime: 5d20h37m16s
> Dumping 16057 out of 65398
> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
> 
> __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
> 55  __asm("movq %%gs:%P1,%0" : "=r" (td) : "n"
> (offsetof(struct pcpu,
> (kgdb) bt

Could you go to frame 11 and print zone->uz_name and
bucket->ub_bucket[18]?  I'm wondering if the item pointer was mangled
somehow.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs panic RELENG_12

[mike tancsa]

:405
    ksi = {ksi_link = {tqe_next = 0x5fe1ea2f, tqe_prev = 0x1},
ksi_info = {si_signo = 76957504, si_errno = -2048, si_code =
-1891515752, si_pid = 506236, si_uid = 0, si_status = 0, si_addr = 0x46,
si_value = {sival_int = 0, sival_ptr = 0x0,
  sigval_int = 0, sigval_ptr = 0x0}, _reason = {_fault =
{_trapno = -1891515752}, _timer = {_timerid = -1891515752, _overrun =
506236}, _mesgq = {_mqd = -1891515752}, _poll = {_band =
2174269467509400}, __spare__ = {
    __spare1__ = 2174269467509400, __spare2__ =
{-1139833872, -512, -2131811593, -1, 1, 0, 0, ksi_flags =
2126206336, ksi_sigq = 0xfe2e4af8}
    td = 0xf80004964740
    dr6 = 
    p = 
    type = 12
    addr = 
    signo = 
    ucode = 
    pf = 
#8  
No locals.
#9  0x80ca0826 in slab_free_item (keg=0xf800037fa380,
slab=0xf80de4656fb0, item=) at
/usr/src/sys/vm/uma_core.c:3357
    dom = 
    freei = 
#10 zone_release (zone=, bucket=0xf801c7fd5218,
cnt=) at /usr/src/sys/vm/uma_core.c:3404
    clearfull = 0
    keg = 0xf800037fa380
    i = 18
    item = 
    mem = 
    slab = 0xf80de4656fb0
#11 0x80ca47d4 in bucket_drain (zone=0xf800037da000,
bucket=0xf801c7fd5200) at /usr/src/sys/vm/uma_core.c:758
    i = 
#12 bucket_cache_drain (zone=0xf800037da000) at
/usr/src/sys/vm/uma_core.c:915
    i = 0
    bucket = 0xf801c7fd5200
    zdom = 
#13 0x80c9e302 in zone_drain_wait (zone=0xf800037da000,
waitok=1) at /usr/src/sys/vm/uma_core.c:1037
No locals.
#14 0x80ca2bbd in zone_drain (zone=0xf800037da000) at
/usr/src/sys/vm/uma_core.c:1056
--Type  for more, q to quit, c to continue without paging--
No locals.
#15 zone_foreach (zfunc=) at /usr/src/sys/vm/uma_core.c:1985
    keg = 0xf800037fa380
    zone = 0xf800037da000
#16 uma_reclaim_locked (kmem_danger=) at
/usr/src/sys/vm/uma_core.c:3737
No locals.
#17 0x80ca2af4 in uma_reclaim () at /usr/src/sys/vm/uma_core.c:3757
No locals.
#18 0x80cc5321 in vm_pageout_lowmem () at
/usr/src/sys/vm/vm_pageout.c:1890
    last = 
    _el = 
    _ep = 
    _t = 
    _v = 
    _tid = 
    _v = 
    _v = 
#19 vm_pageout_worker (arg=) at
/usr/src/sys/vm/vm_pageout.c:1966
    domain = 
    target_met = true
    shortage = 107
    vmd = 
    ofree = 348095
    addl_shortage = 
#20 0x80cc4ee3 in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:2126
    td = 
    p = 0xf800049f8530
    i = 
    first = 0
    error = 
#21 0x809b55be in fork_exit (callout=0x80cc4d50
, arg=0x0, frame=0xfe00bc0f8b00) at
/usr/src/sys/kern/kern_fork.c:1080
    td = 0xf80004964740
    p = 0xf800049f8530
    dtd = 
#22 
No locals.
(kgdb)

On 12/15/2020 4:39 PM, mike tancsa wrote:
> Was doing a backup via zfs send | zfs recv when the box panic'd.  Its a
> not so old RELENG_12 box from last week. Any ideas if this is a hardware
> issue or a bug ? Its r368493 from last Wednesday. I dont see an ECC
> errors logged, so dont think its hardware.
>
> Reading symbols from /boot/kernel/kernel...
> Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
>
> Unread portion of the kernel message buffer:
>
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 1; apic id = 01
> fault virtual address   = 0x0
> fault code  = supervisor read data, page not present
> instruction pointer = 0x20:0x823a554b
> stack pointer   = 0x28:0xfe0343231000
> frame pointer   = 0x28:0xfe03432310c0
> code segment    = base 0x0, limit 0xf, type 0x1b
>     = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags    = interrupt enabled, resume, IOPL = 0
> current process = 87427 (zfs)
> trap number = 12
> panic: page fault
> cpuid = 1
> time = 1608065221
> KDB: stack backtrace:
> #0 0x80a3fa05 at kdb_backtrace+0x65
> #1 0x809f3beb at vpanic+0x17b
> #2 0x809f3a63 at panic+0x43
> #3 0x80e400d1 at trap_fatal+0x391
> #4 0x80e4012f at trap_pfault+0x4f
> #5 0x80e3f776 at trap+0x286
> #6 0x80e19568 at calltrap+0x8
> #7 0x82393a5e at dmu_object_info+0x1e
> #8 0x823983a5 at dmu_recv_stream+0x7b5
> #9 0x8244b706 at zfs_ioc_recv+0xac6
> #10 0x8244dd3d at zfsdev_ioctl+0x62d
> #11 0x808a35e0 at devfs_ioctl+0xb0
> #12 0x80f3becb at VOP_IOCTL_APV+0x7b
> #13 0x80ad1b0a at vn_ioctl+0x16a
> #14 0x808a3bce at devfs_ioctl_f+0x1e
> #15 0x80a5d807 at kern_ioctl+0x2b7
> #16 0x80a5d4aa at sys_ioctl+0xfa
> #17 0x80e40c87 at amd64_syscall+0x387
> Uptime: 3d14h59m52s
> Dumpi

zfs panic RELENG_12

[mike tancsa]

Was doing a backup via zfs send | zfs recv when the box panic'd.  Its a
not so old RELENG_12 box from last week. Any ideas if this is a hardware
issue or a bug ? Its r368493 from last Wednesday. I dont see an ECC
errors logged, so dont think its hardware.

Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address   = 0x0
fault code  = supervisor read data, page not present
instruction pointer = 0x20:0x823a554b
stack pointer   = 0x28:0xfe0343231000
frame pointer   = 0x28:0xfe03432310c0
code segment    = base 0x0, limit 0xf, type 0x1b
    = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = interrupt enabled, resume, IOPL = 0
current process = 87427 (zfs)
trap number = 12
panic: page fault
cpuid = 1
time = 1608065221
KDB: stack backtrace:
#0 0x80a3fa05 at kdb_backtrace+0x65
#1 0x809f3beb at vpanic+0x17b
#2 0x809f3a63 at panic+0x43
#3 0x80e400d1 at trap_fatal+0x391
#4 0x80e4012f at trap_pfault+0x4f
#5 0x80e3f776 at trap+0x286
#6 0x80e19568 at calltrap+0x8
#7 0x82393a5e at dmu_object_info+0x1e
#8 0x823983a5 at dmu_recv_stream+0x7b5
#9 0x8244b706 at zfs_ioc_recv+0xac6
#10 0x8244dd3d at zfsdev_ioctl+0x62d
#11 0x808a35e0 at devfs_ioctl+0xb0
#12 0x80f3becb at VOP_IOCTL_APV+0x7b
#13 0x80ad1b0a at vn_ioctl+0x16a
#14 0x808a3bce at devfs_ioctl_f+0x1e
#15 0x80a5d807 at kern_ioctl+0x2b7
#16 0x80a5d4aa at sys_ioctl+0xfa
#17 0x80e40c87 at amd64_syscall+0x387
Uptime: 3d14h59m52s
Dumping 17213 out of 65366
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55  __asm("movq %%gs:%P1,%0" : "=r" (td) : "n"
(offsetof(struct pcpu,
(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=)
    at /usr/src/sys/kern/kern_shutdown.c:371
#2  0x809f3805 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:451
#3  0x809f3c43 in vpanic (fmt=, ap=)
    at /usr/src/sys/kern/kern_shutdown.c:880
#4  0x809f3a63 in panic (fmt=)
    at /usr/src/sys/kern/kern_shutdown.c:807
#5  0x80e400d1 in trap_fatal (frame=0xfe0343230f40, eva=0)
    at /usr/src/sys/amd64/amd64/trap.c:921
#6  0x80e4012f in trap_pfault (frame=0xfe0343230f40,
    usermode=, signo=, ucode=)
    at /usr/src/sys/amd64/amd64/trap.c:739
#7  0x80e3f776 in trap (frame=0xfe0343230f40)
    at /usr/src/sys/amd64/amd64/trap.c:405
#8  
#9  0x823a554b in dnode_hold_impl (os=0xf805e1d2b800,
    object=, flag=, slots=,
    tag=, dnp=0xfe03432310d8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:1370
#10 0x82393a5e in dmu_object_info (os=0xf80777890070,
    object=18446744071600721588, doi=0xfe03432312e0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:2615
#11 0x823983a5 in receive_read_record (ra=)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:2821
#12 dmu_recv_stream (drc=0xfe0343231430, fp=,
    voffp=, cleanup_fd=8, action_handlep=)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:3203
#13 0x8244b706 in zfs_ioc_recv (zc=)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:4745
#14 0x8244dd3d in zfsdev_ioctl (dev=,
    zcmd=, arg=, flag=,
    td=)
    at
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:6956
#15 0x808a35e0 in devfs_ioctl (ap=0xfe0343231778)
    at /usr/src/sys/fs/devfs/devfs_vnops.c:797
#16 0x80f3becb in VOP_IOCTL_APV (
    vop=0x816a2fe0 , a=0xfe0343231778)
    at vnode_if.c:1067
#17 0x80ad1b0a in vn_ioctl (fp=0xf8001802b5a0,
    com=, data=0xfe0343231910,
    active_cred=0xf80032214300, td=0x2070)
    at /usr/src/sys/kern/vfs_vnops.c:1508
#18 0x808a3bce in devfs_ioctl_f (fp=0xf80777890070,
    com=18446744071600721588, data=0x824e34ed <.L.str+1>, cred=0x0,
    td=0xf8029885) at /usr/src/sys/fs/devfs/devfs_vnops.c:755
#19 0x80a5d807 in fo_ioctl (fp=0xf8001802b5a0, com=3222821403,
    data=0x824e34ed <.L.str+1>, active_cred=0x0,
    td=0xf8029885) at /usr/src/sys/sys/file.h:337
#20 kern_ioctl (td=0x2070, fd=, com=3222821403,
    data=0x824e34ed <.L.str+1> "zrl->zr_mtx")
    at /usr/src/sys/kern/sys_generic.c:805
#21 0x80a5d4aa in sys_ioctl (td=0xf8029885,
    uap=0xf802988503c0) at /usr/src/sys/kern/sys_generic.c:713
#22 0x80e40c87 in syscall

12.2-RC2 crashing on leftover ZFS cache/log on SSD

[Willem Jan Withagen]

Hi,

I managed myself in a somewhat weird situation on a new Epyc server.
It has 2x 4T spinning HDD and 2x intel SSD.

This is what I did:
 - Installed 12.1-RELASE from a USB stick
   - Put the zroot on the 2 HHDs in mirror
 - Downloaded 12/stable, compiled and installed that.
 - Then I added cache and log from  the SSDs (2 gparts)
   to the zroot pool. caches as stripe and log in mirror.

That worked fine.

Only I found out that I had stable, instead of RC2 which I actually wanted
since upgrading with freebsd-upgrade is a lot easiers then when 12.2-RELASE
is out.

So I fetch 12.2-RC2 and installed that on the 2 rusty spinners.
But now the SSDs have leftover ZFS stuff on it that panics the 12.2-RC2.

Simple soluction is to remover the SSDs from their trays, boot and reinsert
the SSDs. That works. I'll be able to clean the SSDs and redo cache and log
stuff.

There are 2 things with the panic:
 - It flies by so fast that I had to film it to actually get the panic 
message.

   no kernel debugger
   No 30/60 secs before autoboot

   Is this a seeting that has changed?
    debug.debugger_on_panic=1 does not seem to work
 - The panic itself:
   panic: solaris assert: nvlist_lookup_uint64(configs[i], 
ZPOOL_CONFIG_POOL_TXG,
 , ) == 0, file 
/usr/src/sys/cddl/contribute/opensolaris/uts/common/fs/zfs/spa.c

 line 5636

Now this could be a case of wrong order and 12/stable has tings in the 
pool that 12.2-RC2

does not like. And should just reuse the SSDs.

Or it is a real bug, and somebody wants to take a look at it? When we 
get into the debugger

we could look at more data
I have a picture of the stack trace, and will leave the system for a day 
as it is.
It there are no takers, than I'll fix things tomorrow and put the system 
into production.


--WjW

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: spa_namespace_lock and concurrent zfs commands

[Eugene Grosbein]

09.09.2020 19:29, Eugene M. Zheganin wrote:

> I'm using sort of FreeBSD ZFS appliance with custom API, and I'm suffering 
> from huge timeouts when large (dozens, actually) of concurrent zfs/zpool 
> commands are issued (get/create/destroy/snapshot/clone mostly).
> 
> Are there any tunables that could help mitigate this ?
> 
> Once I took part in reporting the 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203906 , but the issue that 
> time got resolved somehow. Now I have another set of FreeBSD SANs and it;'s 
> back. I've read the https://wiki.freebsd.org/AndriyGapon/AvgZFSLocking and I 
> realize this probably doesn't have a quick solution, but still...

I think this is some kind of bug/misfeature.
As work-around, try using "zfs destroy -d" instead of plain "zfs destroy".

I suffered from this, too when used ZFS pool over SSD only
instead of HDD+SSD for L2ARC and used SSD sucked really hard
processing BIO_DELETE (trim) with looong delays.

Take a look at "gstat -adI3s" output to monitor amount of delete operations and 
their timings.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: spa_namespace_lock and concurrent zfs commands

[Eugene M. Zheganin]

On 09.09.2020 17:29, Eugene M. Zheganin wrote:

Hello,

I'm using sort of FreeBSD ZFS appliance with custom API, and I'm 
suffering from huge timeouts when large (dozens, actually) of 
concurrent zfs/zpool commands are issued 
(get/create/destroy/snapshot/clone mostly).


Are there any tunables that could help mitigate this ?

Once I took part in reporting the 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203906 , but the 
issue that time got resolved somehow. Now I have another set of 
FreeBSD SANs and it;'s back. I've read the 
https://wiki.freebsd.org/AndriyGapon/AvgZFSLocking and I realize this 
probably doesn't have a quick solution, but still...


This actually looks like this (sometime it takes several [dozens of] 
minutes):



root@cg-mr-prod-stg09:/usr/ports/sysutils/smartmontools # zfs get volmode
load: 3.58  cmd: zfs 70231 [spa_namespace_lock] 16.38r 0.00u 0.00s 0% 3872k
load: 3.58  cmd: zfs 70231 [spa_namespace_lock] 16.59r 0.00u 0.00s 0% 3872k
load: 3.58  cmd: zfs 70231 [spa_namespace_lock] 16.76r 0.00u 0.00s 0% 3872k
load: 3.58  cmd: zfs 70231 [spa_namespace_lock] 16.90r 0.00u 0.00s 0% 3872k
load: 3.58  cmd: zfs 70231 [spa_namespace_lock] 17.04r 0.00u 0.00s 0% 3872k
load: 3.58  cmd: zfs 70231 [spa_namespace_lock] 17.17r 0.00u 0.00s 0% 3872k


root@cg-mr-prod-stg09:~ # ps ax | grep volmode
70231  5  D+    0:00.00 zfs get volmode
70233  6  S+    0:00.00 grep volmode
root@cg-mr-prod-stg09:~ # procstat -kk 70231
  PID    TID COMM    TDNAME KSTACK
70231 101598 zfs - mi_switch+0xe2 sleepq_wait+0x2c 
_sx_xlock_hard+0x459 spa_all_configs+0x1aa zfs_ioc_pool_configs+0x19 
zfsdev_ioctl+0x72e devfs_ioctl+0xad VOP_IOCTL_APV+0x7c vn_ioctl+0x16a 
devfs_ioctl_f+0x1f kern_ioctl+0x2be sys_ioctl+0x15d amd64_syscall+0x364 
fast_syscall_common+0x101

root@cg-mr-prod-stg09:~ #


Thanks.

Eugene.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

spa_namespace_lock and concurrent zfs commands

[Eugene M. Zheganin]

Hello,

I'm using sort of FreeBSD ZFS appliance with custom API, and I'm 
suffering from huge timeouts when large (dozens, actually) of concurrent 
zfs/zpool commands are issued (get/create/destroy/snapshot/clone mostly).


Are there any tunables that could help mitigate this ?

Once I took part in reporting the 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203906 , but the issue 
that time got resolved somehow. Now I have another set of FreeBSD SANs 
and it;'s back. I've read the 
https://wiki.freebsd.org/AndriyGapon/AvgZFSLocking and I realize this 
probably doesn't have a quick solution, but still...



Thanks.


Eugene.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs meta data slowness

[mike tancsa]

On 7/22/2020 9:26 AM, Eugene Grosbein wrote:
>
> It's hard to read due to wrapping plus, it is truncated.
> Maybe additional flag -d3 or similar will help, combined with dedirection < 
> /dev/null > top.out
> so it won't use size of your terminal to wrap/truncate output.
>
> Also, make sure you invoke top while "zfs" command is running.
> Also, procstat -kk for pid of "zfs" command would be useful (but may occur 
> pretty long).
>
> I suppose it blocks waiting for some kernel lock and procstat would show 
> details.

.txt file is attached. I also ran procstat -kk  in a loop for a bit
while it was running.

In this case the list too just under 5min to run.

time /sbin/zfs list -Hp -oname,creation -S creation -t snapshot >
/var/log/snapshots.out
2.311u 24.963s 4:55.13 9.2% 71+179k 2706576+26io 0pf+0w

Ideally, I would like to bias slightly ARC to hold this meta data a
little longer than evict it out of cache so fast.  I put in an nvme disk
hoping that would help, but it makes no difference or doesnt seem to
anyways.

    ---Mike

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs meta data slowness

[Eugene Grosbein]

22.07.2020 20:02, mike tancsa wrote:
> 
> On 7/22/2020 1:29 AM, Eugene Grosbein wrote:
>> 22.07.2020 2:37, mike tancsa wrote:
>>
>>>> Something else special about the setup.
>>>> output of "top -b"
>>>>
>>> ports are right now being built in a VM, but the problem (zrepl hanging)
>>> and zfs list -t snapshots taking forever happens regardless
>>>
>>>   PID USERNAMETHR PRI NICE   SIZERES STATEC   TIMEWCPU
>>> COMMAND
>>>  4439 root 12  40   20  6167M  5762M kqread   3 535:13 200.00% bhyve
>>> 98783 root  2  21016M  5136K hdr->b   4   0:01   1.95% zfs
>>> 76489 root         21  230   738M54M uwait1   2:18   0.88% zrepl
>>> 98784 root  1  21013M  3832K piperd   3   0:01   0.59% zfs
>>> 99563 root  1  20013M  4136K zio->i   4   0:00   0.39% zfs
>> You need top -SHPI
> 
> last pid: 66981;  load averages:  3.44,  3.25,  3.34  up 1+22:49:34   
> 08:58:06
> 2056 threads:  12 running, 1988 sleeping, 3 zombie, 53 waiting
> CPU 0:  7.7% user,  0.1% nice, 15.0% system,  0.7% interrupt, 76.5% idle
> CPU 1:  9.9% user,  0.1% nice, 16.6% system,  0.1% interrupt, 73.2% idle
> CPU 2: 10.0% user,  0.1% nice, 17.5% system,  0.4% interrupt, 71.9% idle
> CPU 3: 10.3% user,  0.1% nice, 21.2% system,  0.1% interrupt, 68.2% idle
> CPU 4:  9.7% user,  0.1% nice, 15.6% system,  0.4% interrupt, 74.3% idle
> CPU 5: 10.2% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.3% idle
> CPU 6:  9.7% user,  0.1% nice, 16.6% system,  0.5% interrupt, 73.1% idle
> CPU 7: 10.1% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.4% idle
> Mem: 4236M Active, 19G Inact, 283M Laundry, 37G Wired, 3248K Buf, 1667M Free
> ARC: 25G Total, 9939M MFU, 12G MRU, 397M Anon, 573M Header, 2143M Other
>  20G Compressed, 29G Uncompressed, 1.43:1 Ratio
> Swap: 20G Total, 20G Free
> 
>   PID USERNAMEPRI NICE   SIZERES STATEC   TIMEWCPU COMMAND
>  4439 root123   20  6167M  5856M CPU2 2  20.2H 100.00%
> bhyve{vcpu 0}
>  4439 root122   20  6167M  5856M CPU3 3  20.2H  99.17%
> bhyve{vcpu 1}
>11 root    155 ki31 0B   128K CPU0 0  35.9H  65.38%
> idle{idle: cpu0}
>11 root155 ki31 0B   128K CPU4 4  34.8H  63.38%
> idle{idle: cpu4}
> 66956 root 90061M54M CPU5 5   0:09  62.70% zfs
>11 root155 ki31 0B   128K CPU6 6  34.3H  58.06%
> idle{idle: cpu6}
>11 root155 ki31 0B   128K RUN  2  33.7H  54.49%
> idle{idle: cpu2}
>11 root155 ki31 0B   128K RUN  1  34.3H  53.76%
> idle{idle: cpu1}
>11 root155 ki31 0B   128K RUN  3  32.0H  53.47%
> idle{idle: cpu3}
>11 root155 ki31 0B   128K CPU7 7  32.0H  50.68%
> idle{idle: cpu7}
>11 root155 ki31 0B   128K RUN  5  32.0H  48.29%
> idle{idle: cpu5}
>56 root-12- 0B  5168K -1   5:49   9.67%
> zpool-zbackup2{zio_write_issue_3}
>56 root-12- 0B  5168K -3   5:48   9.57%
> zpool-zbackup2{zio_write_issue_5}
>56 root-12- 0B  5168K -4   5:48   9.47%
> zpool-zbackup2{zio_write_issue_4}
>56 root-12- 0B  5168K -7   5:48   9.47%
> zpool-zbackup2{zio_write_issue_2}
>56 root-12- 0B  5168K -0   5:49   9.38%
> zpool-zbackup2{zio_write_issue_1}
>56 root-12- 0B  5168K -1   5:48   9.38%
> zpool-zbackup2{zio_write_issue_0}
> 63392 root     230   729M32M uwait0   0:01   4.20%
> zrepl{zrepl}

It's hard to read due to wrapping plus, it is truncated.
Maybe additional flag -d3 or similar will help, combined with dedirection < 
/dev/null > top.out
so it won't use size of your terminal to wrap/truncate output.

Also, make sure you invoke top while "zfs" command is running.
Also, procstat -kk for pid of "zfs" command would be useful (but may occur 
pretty long).

I suppose it blocks waiting for some kernel lock and procstat would show 
details.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs meta data slowness

[mike tancsa]

On 7/22/2020 1:29 AM, Eugene Grosbein wrote:
> 22.07.2020 2:37, mike tancsa wrote:
>
>>> Something else special about the setup.
>>> output of "top -b"
>>>
>> ports are right now being built in a VM, but the problem (zrepl hanging)
>> and zfs list -t snapshots taking forever happens regardless
>>
>>   PID USERNAMETHR PRI NICE   SIZERES STATEC   TIMEWCPU
>> COMMAND
>>  4439 root 12  40   20  6167M  5762M kqread   3 535:13 200.00% bhyve
>> 98783 root  2  21016M  5136K hdr->b   4   0:01   1.95% zfs
>> 76489 root 21  230   738M54M uwait1   2:18   0.88% zrepl
>> 98784 root  1  21013M  3832K piperd   3   0:01   0.59% zfs
>> 99563 root  1  20013M  4136K zio->i   4   0:00   0.39% zfs
> You need top -SHPI

last pid: 66981;  load averages:  3.44,  3.25,  3.34  up 1+22:49:34   
08:58:06
2056 threads:  12 running, 1988 sleeping, 3 zombie, 53 waiting
CPU 0:  7.7% user,  0.1% nice, 15.0% system,  0.7% interrupt, 76.5% idle
CPU 1:  9.9% user,  0.1% nice, 16.6% system,  0.1% interrupt, 73.2% idle
CPU 2: 10.0% user,  0.1% nice, 17.5% system,  0.4% interrupt, 71.9% idle
CPU 3: 10.3% user,  0.1% nice, 21.2% system,  0.1% interrupt, 68.2% idle
CPU 4:  9.7% user,  0.1% nice, 15.6% system,  0.4% interrupt, 74.3% idle
CPU 5: 10.2% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.3% idle
CPU 6:  9.7% user,  0.1% nice, 16.6% system,  0.5% interrupt, 73.1% idle
CPU 7: 10.1% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.4% idle
Mem: 4236M Active, 19G Inact, 283M Laundry, 37G Wired, 3248K Buf, 1667M Free
ARC: 25G Total, 9939M MFU, 12G MRU, 397M Anon, 573M Header, 2143M Other
 20G Compressed, 29G Uncompressed, 1.43:1 Ratio
Swap: 20G Total, 20G Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 4439 root    123   20  6167M  5856M CPU2 2  20.2H 100.00%
bhyve{vcpu 0}
 4439 root    122   20  6167M  5856M CPU3 3  20.2H  99.17%
bhyve{vcpu 1}
   11 root    155 ki31 0B   128K CPU0 0  35.9H  65.38%
idle{idle: cpu0}
   11 root    155 ki31 0B   128K CPU4 4  34.8H  63.38%
idle{idle: cpu4}
66956 root 90    0    61M    54M CPU5 5   0:09  62.70% zfs
   11 root    155 ki31 0B   128K CPU6 6  34.3H  58.06%
idle{idle: cpu6}
   11 root    155 ki31 0B   128K RUN  2  33.7H  54.49%
idle{idle: cpu2}
   11 root    155 ki31 0B   128K RUN  1  34.3H  53.76%
idle{idle: cpu1}
   11 root    155 ki31 0B   128K RUN  3  32.0H  53.47%
idle{idle: cpu3}
   11 root    155 ki31 0B   128K CPU7 7  32.0H  50.68%
idle{idle: cpu7}
   11 root    155 ki31 0B   128K RUN  5  32.0H  48.29%
idle{idle: cpu5}
   56 root    -12    - 0B  5168K -    1   5:49   9.67%
zpool-zbackup2{zio_write_issue_3}
   56 root    -12    - 0B  5168K -    3   5:48   9.57%
zpool-zbackup2{zio_write_issue_5}
   56 root    -12    - 0B  5168K -    4   5:48   9.47%
zpool-zbackup2{zio_write_issue_4}
   56 root    -12    - 0B  5168K -    7   5:48   9.47%
zpool-zbackup2{zio_write_issue_2}
   56 root    -12    - 0B  5168K -    0   5:49   9.38%
zpool-zbackup2{zio_write_issue_1}
   56 root    -12    - 0B  5168K -    1   5:48   9.38%
zpool-zbackup2{zio_write_issue_0}
63392 root 23    0   729M    32M uwait    0   0:01   4.20%
zrepl{zrepl}



___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs meta data slowness

[mike tancsa]

On 7/22/2020 1:04 AM, Rick Macklem wrote:
> mike tancsa wrote:
>> Hi,
>>Thanks for the response. Reply in line
>>
>> On 7/20/2020 9:04 AM, Ronald Klop wrote:
>>> Hi,
>>>
>>> My first suggestion would be to remove a lot of snapshots. But that my
>>> not match your business case.
>> As its a backup server, its sort of the point to have all those snapshots.
> I'm the last guy who should be commenting on ZFS, since I never use it.
> However, it is my understanding that ZFS "pseudo automounts" each
> snapshot when you go there, so I think that might be what is taking
> so long (ie. not really meta data).
>
Thanks Rick, in this case, its just listing snapshots from the command line

zfs list -t snapshot

that is taking forever.

Best case scenario when the box boots up after running it once, it will
take about 25 seconds

but when the box is taking in zfs streams, it really slows down and can
be anywhere up to 30min


0{backup4}# time zfs list -t snapshot > /tmp/snap.out
1.839u 23.211s 3:11.69 13.0%    71+178k 2504801+38io 0pf+0w
0{backup4}# time zfs list -t snapshot > /tmp/snap.out
1.817u 23.612s 0:25.47 99.8%    71+178k 2472088+38io 0pf+0w
0{backup4}# time zfs list -t snapshot > /tmp/snap.out
2.040u 23.314s 0:25.40 99.8%    71+177k 2472105+38io 0pf+0w
0{backup4}#



___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs meta data slowness

[Ronald Klop]

Van: mike tancsa 
Datum: dinsdag, 21 juli 2020 21:37
Aan: Ronald Klop , FreeBSD-STABLE Mailing List 

Onderwerp: Re: zfs meta data slowness


Hi,
Thanks for the response. Reply in line

On 7/20/2020 9:04 AM, Ronald Klop wrote:
> Hi,
>
> My first suggestion would be to remove a lot of snapshots. But that my
> not match your business case.

As its a backup server, its sort of the point to have all those snapshots.


> Maybe you can provide more information about your setup:
> Amount of RAM, CPU?
64G, Xeon(R) CPU E3-1240 v6 @ 3.70GHz
> output of "zpool status"
# zpool status -x

all pools are healthy
 


That is nice to know.
Instead of zpool status -x, the output of "zpool status" is very interesting. And 
"zpool list" also. That gives the reader information about your setup, which helps 
thinking along about the possible cause.

But as somebody else mentioned. Profiling the kernel might be the best thing to 
do. Dtrace can be used for it. But I don't know these commands by heart.

If I remember correctly there is an optimization for "zfs list -o name". This 
is much faster because it does not get extra information from the disks.
See: https://svnweb.freebsd.org/base?view=revision=230438

Regards,
Ronald.

> 

> output of "zfs list" if possible to share

its a big list

# zfs list | wc
 8244120  107511


> Type of disks/ssds?
old school Device Model: WDC WD80EFAX-68KNBN0
> What is the load of the system? I/O per second, etc.
its not cpu bound, disks are sometimes running at 100% based on gstat,
but not always
> Do you use dedup, GELI?

no and no


> Something else special about the setup.
> output of "top -b"
>

ports are right now being built in a VM, but the problem (zrepl hanging)
and zfs list -t snapshots taking forever happens regardless

  PID USERNAMETHR PRI NICE   SIZERES STATEC   TIMEWCPU
COMMAND
 4439 root 12  40   20  6167M  5762M kqread   3 535:13 200.00% bhyve
98783 root  2  21016M  5136K hdr->b   4   0:01   1.95% zfs
76489 root 21  230   738M    54M uwait1   2:18   0.88% zrepl
98784 root  1  21013M  3832K piperd   3   0:01   0.59% zfs
99563 root  1  20013M  4136K zio->i   4   0:00   0.39% zfs
16136 root 18  250   705M56M uwait3  29:58   0.00%
zrepl-freebsd-amd64
 1845 root  1  20012M  3772K nanslp   7   5:54   0.00%
ossec-syscheckd
 1567 root  1  20011M  2744K select   0   2:22   0.00%
syslogd
 1737 root 32  20011M  2844K rpcsvc   6   1:40   0.00% nfsd
 1660 root  1 -52   r011M11M nanslp   5   1:18   0.00%
watchdogd
 1434 root  1  200  9988K   988K select   3   0:27   0.00% devd
 2435 mdtancsa  1  20020M  8008K select   0   0:21   0.00% sshd
 1754 root  3  20018M  3556K select   1   0:11   0.00%
apcupsd
 5917 root  1  20011M  2672K select   2   0:06   0.00%
script
 1449 _pflogd   1  20012M  3572K bpf  3   0:05   0.00%
pflogd

---Mike

> That kind of information.
>
> Regards,
> Ronald.
>
>
> Van: mike tancsa 
> Datum: zondag, 19 juli 2020 16:17
> Aan: FreeBSD-STABLE Mailing List 
> Onderwerp: zfs meta data slowness
>>
>> Are there any tweaks that can be done to speed up or improve zfs
>> metadata performance ? I have a backup server with a lot of snapshots
>> (40,000)  and just doing a listing can take a great deal of time.  Best
>> case scenario is about 24 seconds, worst case, I have seen it up to 15
>> minutes.  (FreeBSD 12.1-STABLE r363078)
>>
>>
>> ARC Efficiency: 79.33b
>> Cache Hit Ratio:92.81%  73.62b
>> Cache Miss Ratio:   7.19%   5.71b
>> Actual Hit Ratio:   92.78%  73.60b
>>
>> Data Demand Efficiency: 96.47%  461.91m
>> Data Prefetch Efficiency:   1.00%   262.73m
>>
>> CACHE HITS BY CACHE LIST:
>>   Anonymously Used: 0.01%   3.86m
>>   Most Recently Used:   3.91%   2.88b
>>   Most Frequently Used: 96.06%  70.72b
>>   Most Recently Used Ghost: 0.01%   5.31m
>>   Most Frequently Used Ghost:   0.01%   10.47m
>>
>> CACHE HITS BY DATA TYPE:
>>   Demand Data:  0.61%   445.60m
>>   Prefetch Data:0.00%   2.63m
>>   Demand Metadata:  99.36%  73.15b
>>   Prefetch Metadata:0.03%   21.00m
>>
>> CACHE MISSES BY DATA TYPE:
>>   Demand Data:  0.29%   16.31m
>>   Pr

Re: zfs meta data slowness

[Eugene Grosbein]

22.07.2020 2:37, mike tancsa wrote:

>> Something else special about the setup.
>> output of "top -b"
>>
> 
> ports are right now being built in a VM, but the problem (zrepl hanging)
> and zfs list -t snapshots taking forever happens regardless
> 
>   PID USERNAMETHR PRI NICE   SIZERES STATEC   TIMEWCPU
> COMMAND
>  4439 root 12  40   20  6167M  5762M kqread   3 535:13 200.00% bhyve
> 98783 root  2  21016M  5136K hdr->b   4   0:01   1.95% zfs
> 76489 root 21  230   738M54M uwait1   2:18   0.88% zrepl
> 98784 root      1  21013M  3832K piperd   3   0:01   0.59% zfs
> 99563 root  1  20013M  4136K zio->i   4   0:00   0.39% zfs

You need top -SHPI

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs meta data slowness

[Rick Macklem]

mike tancsa wrote:
>Hi,
>Thanks for the response. Reply in line
>
>On 7/20/2020 9:04 AM, Ronald Klop wrote:
>> Hi,
>>
>> My first suggestion would be to remove a lot of snapshots. But that my
>> not match your business case.
>
>As its a backup server, its sort of the point to have all those snapshots.
I'm the last guy who should be commenting on ZFS, since I never use it.
However, it is my understanding that ZFS "pseudo automounts" each
snapshot when you go there, so I think that might be what is taking
so long (ie. not really meta data).

Of course I have no idea what might speed that up. I would be
tempted to look in ZFS for the "snapshot mounting code", in
case I could find an obvious problem...

rick


> Maybe you can provide more information about your setup:
> Amount of RAM, CPU?
64G, Xeon(R) CPU E3-1240 v6 @ 3.70GHz
> output of "zpool status"
# zpool status -x

all pools are healthy


> output of "zfs list" if possible to share

its a big list

# zfs list | wc
 8244120  107511


> Type of disks/ssds?
old school Device Model: WDC WD80EFAX-68KNBN0
> What is the load of the system? I/O per second, etc.
its not cpu bound, disks are sometimes running at 100% based on gstat,
but not always
> Do you use dedup, GELI?

no and no


> Something else special about the setup.
> output of "top -b"
>

ports are right now being built in a VM, but the problem (zrepl hanging)
and zfs list -t snapshots taking forever happens regardless

  PID USERNAMETHR PRI NICE   SIZERES STATEC   TIMEWCPU
COMMAND
 4439 root 12  40   20  6167M  5762M kqread   3 535:13 200.00% bhyve
98783 root  2  21016M  5136K hdr->b   4   0:01   1.95% zfs
76489 root 21  23    0   738M54M uwait1   2:18   0.88% zrepl
98784 root  1  21013M  3832K piperd   3   0:01   0.59% zfs
99563 root  1  20013M  4136K zio->i   4   0:00   0.39% zfs
16136 root 18  250   705M56M uwait3  29:58   0.00%
zrepl-freebsd-amd64
 1845 root  1  20012M  3772K nanslp   7   5:54   0.00%
ossec-syscheckd
 1567 root  1  20011M  2744K select   0   2:22   0.00%
syslogd
 1737 root 32  20011M  2844K rpcsvc   6   1:40   0.00% nfsd
 1660 root  1 -52   r011M11M nanslp   5   1:18   0.00%
watchdogd
 1434 root  1  200  9988K   988K select   3   0:27   0.00% devd
 2435 mdtancsa  1  20020M  8008K select   0   0:21   0.00% sshd
 1754 root  3  20018M  3556K select   1   0:11   0.00%
apcupsd
 5917 root  1  20011M  2672K select   2   0:06   0.00%
script
 1449 _pflogd   1  20012M  3572K bpf  3   0:05   0.00%
pflogd

---Mike

> That kind of information.
>
> Regards,
> Ronald.
>
>
> Van: mike tancsa 
> Datum: zondag, 19 juli 2020 16:17
> Aan: FreeBSD-STABLE Mailing List 
> Onderwerp: zfs meta data slowness
>>
>> Are there any tweaks that can be done to speed up or improve zfs
>> metadata performance ? I have a backup server with a lot of snapshots
>> (40,000)  and just doing a listing can take a great deal of time.  Best
>> case scenario is about 24 seconds, worst case, I have seen it up to 15
>> minutes.  (FreeBSD 12.1-STABLE r363078)
>>
>>
>> ARC Efficiency: 79.33b
>> Cache Hit Ratio:92.81%  73.62b
>> Cache Miss Ratio:   7.19%   5.71b
>> Actual Hit Ratio:   92.78%  73.60b
>>
>> Data Demand Efficiency: 96.47%  461.91m
>> Data Prefetch Efficiency:   1.00%   262.73m
>>
>> CACHE HITS BY CACHE LIST:
>>   Anonymously Used: 0.01%   3.86m
>>   Most Recently Used:   3.91%   2.88b
>>   Most Frequently Used: 96.06%  70.72b
>>   Most Recently Used Ghost: 0.01%   5.31m
>>   Most Frequently Used Ghost:   0.01%   10.47m
>>
>> CACHE HITS BY DATA TYPE:
>>   Demand Data:  0.61%   445.60m
>>   Prefetch Data:0.00%   2.63m
>>   Demand Metadata:  99.36%  73.15b
>>   Prefetch Metadata:0.03%   21.00m
>>
>> CACHE MISSES BY DATA TYPE:
>>   Demand Data:  0.29%   16.31m
>>   Prefetch Data:4.56%   260.10m
>>   Demand Metadata:  95.02%  5.42b
>>   Prefetch Metadata:0.14%   7.75m
>>
>>
>> Other than increase the metadata max, I havent really changed any
>> tuneables
>>
>>
&g

Re: zfs meta data slowness

[mike tancsa]

Hi,
    Thanks for the response. Reply in line

On 7/20/2020 9:04 AM, Ronald Klop wrote:
> Hi,
>
> My first suggestion would be to remove a lot of snapshots. But that my
> not match your business case.

As its a backup server, its sort of the point to have all those snapshots.


> Maybe you can provide more information about your setup:
> Amount of RAM, CPU?
64G, Xeon(R) CPU E3-1240 v6 @ 3.70GHz
> output of "zpool status"
# zpool status -x

all pools are healthy


> output of "zfs list" if possible to share

its a big list

# zfs list | wc
 824    4120  107511


> Type of disks/ssds?
old school Device Model: WDC WD80EFAX-68KNBN0
> What is the load of the system? I/O per second, etc.
its not cpu bound, disks are sometimes running at 100% based on gstat,
but not always
> Do you use dedup, GELI?

no and no


> Something else special about the setup.
> output of "top -b"
>

ports are right now being built in a VM, but the problem (zrepl hanging)
and zfs list -t snapshots taking forever happens regardless

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU
COMMAND
 4439 root 12  40   20  6167M  5762M kqread   3 535:13 200.00% bhyve
98783 root  2  21    0    16M  5136K hdr->b   4   0:01   1.95% zfs
76489 root 21  23    0   738M    54M uwait    1   2:18   0.88% zrepl
98784 root  1  21    0    13M  3832K piperd   3   0:01   0.59% zfs
99563 root  1  20    0    13M  4136K zio->i   4   0:00   0.39% zfs
16136 root 18  25    0   705M    56M uwait    3  29:58   0.00%
zrepl-freebsd-amd64
 1845 root  1  20    0    12M  3772K nanslp   7   5:54   0.00%
ossec-syscheckd
 1567 root  1  20    0    11M  2744K select   0   2:22   0.00%
syslogd
 1737 root 32  20    0    11M  2844K rpcsvc   6   1:40   0.00% nfsd
 1660 root  1 -52   r0    11M    11M nanslp   5   1:18   0.00%
watchdogd
 1434 root  1  20    0  9988K   988K select   3   0:27   0.00% devd
 2435 mdtancsa  1  20    0    20M  8008K select   0   0:21   0.00% sshd
 1754 root  3  20    0    18M  3556K select   1   0:11   0.00%
apcupsd
 5917 root  1  20    0    11M  2672K select   2   0:06   0.00%
script
 1449 _pflogd   1  20    0    12M  3572K bpf  3   0:05   0.00%
pflogd

    ---Mike

> That kind of information.
>
> Regards,
> Ronald.
>
>
> Van: mike tancsa 
> Datum: zondag, 19 juli 2020 16:17
> Aan: FreeBSD-STABLE Mailing List 
> Onderwerp: zfs meta data slowness
>>
>> Are there any tweaks that can be done to speed up or improve zfs
>> metadata performance ? I have a backup server with a lot of snapshots
>> (40,000)  and just doing a listing can take a great deal of time.  Best
>> case scenario is about 24 seconds, worst case, I have seen it up to 15
>> minutes.  (FreeBSD 12.1-STABLE r363078)
>>
>>
>> ARC Efficiency: 79.33b
>>     Cache Hit Ratio:    92.81%  73.62b
>>     Cache Miss Ratio:   7.19%   5.71b
>>     Actual Hit Ratio:   92.78%  73.60b
>>
>>     Data Demand Efficiency: 96.47%  461.91m
>>     Data Prefetch Efficiency:   1.00%   262.73m
>>
>>     CACHE HITS BY CACHE LIST:
>>   Anonymously Used: 0.01%   3.86m
>>   Most Recently Used:   3.91%   2.88b
>>   Most Frequently Used: 96.06%  70.72b
>>   Most Recently Used Ghost: 0.01%   5.31m
>>   Most Frequently Used Ghost:   0.01%   10.47m
>>
>>     CACHE HITS BY DATA TYPE:
>>   Demand Data:  0.61%   445.60m
>>   Prefetch Data:    0.00%   2.63m
>>   Demand Metadata:  99.36%  73.15b
>>   Prefetch Metadata:    0.03%   21.00m
>>
>>     CACHE MISSES BY DATA TYPE:
>>   Demand Data:      0.29%   16.31m
>>   Prefetch Data:    4.56%   260.10m
>>   Demand Metadata:  95.02%  5.42b
>>   Prefetch Metadata:    0.14%   7.75m
>>
>>
>> Other than increase the metadata max, I havent really changed any
>> tuneables
>>
>>
>> ZFS Tunables (sysctl):
>>     kern.maxusers   4416
>>     vm.kmem_size    66691842048
>>     vm.kmem_size_scale  1
>>     vm.kmem_size_min    0
>>     vm.kmem_size_max    1319413950874
>>     vfs.zfs.trim.max_interval   1
>>     vfs.zfs.trim.timeout    30
>>     vfs.zfs.trim.tx

Re: zfs meta data slowness

[Eugene Grosbein]

19.07.2020 21:17, mike tancsa wrote:
> Are there any tweaks that can be done to speed up or improve zfs
> metadata performance ? I have a backup server with a lot of snapshots
> (40,000)  and just doing a listing can take a great deal of time.  Best
> case scenario is about 24 seconds, worst case, I have seen it up to 15
> minutes.  (FreeBSD 12.1-STABLE r363078)

I believe you have to perform kernel profiling for your specific use case.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: zfs meta data slowness

[Ronald Klop]

Hi,

My first suggestion would be to remove a lot of snapshots. But that my not 
match your business case.
Maybe you can provide more information about your setup:
Amount of RAM, CPU?
output of "zpool status"
output of "zfs list" if possible to share
Type of disks/ssds?
What is the load of the system? I/O per second, etc.
Do you use dedup, GELI?
Something else special about the setup.
output of "top -b"

That kind of information.

Regards,
Ronald.


Van: mike tancsa 
Datum: zondag, 19 juli 2020 16:17
Aan: FreeBSD-STABLE Mailing List 
Onderwerp: zfs meta data slowness


Are there any tweaks that can be done to speed up or improve zfs
metadata performance ? I have a backup server with a lot of snapshots
(40,000)  and just doing a listing can take a great deal of time.  Best
case scenario is about 24 seconds, worst case, I have seen it up to 15
minutes.  (FreeBSD 12.1-STABLE r363078)


ARC Efficiency: 79.33b
Cache Hit Ratio:92.81%  73.62b
Cache Miss Ratio:   7.19%   5.71b
Actual Hit Ratio:   92.78%  73.60b

Data Demand Efficiency: 96.47%  461.91m
Data Prefetch Efficiency:   1.00%   262.73m

CACHE HITS BY CACHE LIST:
  Anonymously Used: 0.01%   3.86m
  Most Recently Used:   3.91%   2.88b
  Most Frequently Used: 96.06%  70.72b
  Most Recently Used Ghost: 0.01%   5.31m
  Most Frequently Used Ghost:   0.01%   10.47m

CACHE HITS BY DATA TYPE:
  Demand Data:  0.61%   445.60m
  Prefetch Data:0.00%   2.63m
  Demand Metadata:  99.36%  73.15b
  Prefetch Metadata:0.03%   21.00m

CACHE MISSES BY DATA TYPE:
  Demand Data:  0.29%   16.31m
  Prefetch Data:4.56%   260.10m
  Demand Metadata:  95.02%  5.42b
  Prefetch Metadata:0.14%   7.75m


Other than increase the metadata max, I havent really changed any tuneables


ZFS Tunables (sysctl):
kern.maxusers   4416
vm.kmem_size66691842048
vm.kmem_size_scale  1
vm.kmem_size_min0
vm.kmem_size_max1319413950874
vfs.zfs.trim.max_interval   1
vfs.zfs.trim.timeout30
vfs.zfs.trim.txg_delay  32
vfs.zfs.trim.enabled1
vfs.zfs.vol.immediate_write_sz  32768
vfs.zfs.vol.unmap_sync_enabled  0
vfs.zfs.vol.unmap_enabled   1
vfs.zfs.vol.recursive   0
vfs.zfs.vol.mode1
vfs.zfs.version.zpl 5
vfs.zfs.version.spa 5000
vfs.zfs.version.acl 1
vfs.zfs.version.ioctl   7
vfs.zfs.debug   0
vfs.zfs.super_owner 0
vfs.zfs.immediate_write_sz  32768
vfs.zfs.sync_pass_rewrite   2
vfs.zfs.sync_pass_dont_compress 5
vfs.zfs.sync_pass_deferred_free 2
vfs.zfs.zio.dva_throttle_enabled1
vfs.zfs.zio.exclude_metadata0
vfs.zfs.zio.use_uma 1
vfs.zfs.zio.taskq_batch_pct 75
vfs.zfs.zil_maxblocksize131072
vfs.zfs.zil_slog_bulk   786432
vfs.zfs.zil_nocacheflush0
vfs.zfs.zil_replay_disable  0
vfs.zfs.cache_flush_disable 0
vfs.zfs.standard_sm_blksz   131072
vfs.zfs.dtl_sm_blksz4096
vfs.zfs.min_auto_ashift 9
vfs.zfs.max_auto_ashift 13
vfs.zfs.vdev.trim_max_pending   1
vfs.zfs.vdev.bio_delete_disable 0
vfs.zfs.vdev.bio_flush_disable  0
vfs.zfs.vdev.def_queue_depth32
vfs.zfs.vdev.queue_depth_pct1000
vfs.zfs.vdev.write_gap_limit4096
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.aggregation_limit_non_rotating131072
vfs.zfs.vdev.aggregation_limit  1048576
vfs.zfs.vdev.initializing_max_active1
vfs.zfs.vdev.initializing_min_active1
vfs.zfs.vdev.removal_max_active 2
vfs.zfs.vdev.removal_min_active 1
vfs.zfs.vdev.trim_max_active64
vfs.zfs.vdev.trim_min_active1
vfs.zfs.vdev.scrub_max_active   2
vfs.zfs.vdev.scrub_min_active   1
vfs.zfs.vdev