Re: nvme timeout issues with hardware and bhyve vm's

[Warner Losh]

On Fri, Oct 13, 2023 at 11:47 AM Pete Wright  wrote:

>
>
> On 10/13/23 6:24 AM, Warner Losh wrote:
> >
> >
> > On Thu, Oct 12, 2023, 10:53 PM Pete Wright  > > wrote:
> >
> >
> >
> > On 10/12/23 8:45 PM, Warner Losh wrote:
> >  > What version is that kernel?
> >
> > oh dang i sent this to the wrong list, i'm not running current.  the
> > hypervisor and vm are both 13.2 and my workstation is a recent 14.0
> > pre-release build.  i'll do more homework tomorrow and post to
> > questions
> > or a more appropriate list.
> >
> >
> > Are the messages from the VM? Stable/14 should have the important nvme
> > changes I've made lately. The bhyve in 13.2 is lacking a number of nvme
> > fixes that have gone into current and stable/14. It's hard to say where
> > the fault is coming from.
> >
>
>
> the messages i posted in the start of the thread are from the VM itself
> (13.2-RELEASE).  The zpool on the hypervisor (13.2-RELEASE) showed no
> such issues.
>
> Based on your comment about the improvements in 14 I'll focus my efforts
> on my workstation, it seemed to happen regularly so hopefully i can find
> a repo case.
>

Let me now if you see similar messages in stable/14. I think I've fixed all
the
issues with timeouts, though you shouldn't ever seem them in a vm setup
unless something else weird is going on.

Warner

FreeBSD 14.0-RC1 Now Available

[Glen Barber]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

The first RC build of the 14.0-RELEASE release cycle is now available.

Installation images are available for:

o 14.0-RC1 amd64 GENERIC
o 14.0-RC1 i386 GENERIC
o 14.0-RC1 powerpc GENERIC
o 14.0-RC1 powerpc64 GENERIC64
o 14.0-RC1 powerpc64le GENERIC64LE
o 14.0-RC1 powerpcspe MPC85XXSPE
o 14.0-RC1 armv7 GENERICSD
o 14.0-RC1 aarch64 GENERIC
o 14.0-RC1 aarch64 RPI
o 14.0-RC1 aarch64 PINE64
o 14.0-RC1 aarch64 PINE64-LTS
o 14.0-RC1 aarch64 PINEBOOK
o 14.0-RC1 aarch64 ROCK64
o 14.0-RC1 aarch64 ROCKPRO64
o 14.0-RC1 riscv64 GENERIC
o 14.0-RC1 riscv64 GENERICSD

Note regarding arm SD card images: For convenience for those without
console access to the system, a freebsd user with a password of
freebsd is available by default for ssh(1) access.  Additionally,
the root user password is set to root.  It is strongly recommended
to change the password for both users after gaining access to the
system.

Installer images and memory stick images are available here:

https://download.freebsd.org/releases/ISO-IMAGES/14.0/

The image checksums follow at the end of this e-mail.

If you notice problems you can report them through the Bugzilla PR
system or on the -stable mailing list.

If you would like to use Git to do a source based update of an existing
system, use the "releng/14.0" branch.

A summary of changes since 14.0-BETA5 includes:

o A race condition in swap_pager_swapoff_object() had been fixed.

o Various updates to the Linux KPI, 802.11, iwlwifi, and rtw88.

o And other miscellaneous fixes.

A list of changes since 13.2-RELEASE is available in the releng/14.0
release notes:

https://www.freebsd.org/releases/14.0R/relnotes/

Please note, the release notes page is not yet complete, and will be
updated on an ongoing basis as the 14.0-RELEASE cycle progresses.

=== Virtual Machine Disk Images ===

VM disk images are available for the amd64, i386, and aarch64
architectures.  Disk images may be downloaded from the following URL
(or any of the FreeBSD download mirrors):

https://download.freebsd.org/releases/VM-IMAGES/14.0-RC1/

BASIC-CI images can be found at:

https://download.freebsd.org/releases/CI-IMAGES/14.0-RC1/

The partition layout is:

~ 16 kB - freebsd-boot GPT partition type (bootfs GPT label)
~ 1 GB  - freebsd-swap GPT partition type (swapfs GPT label)
~ 20 GB - freebsd-ufs GPT partition type (rootfs GPT label)

The disk images are available in QCOW2, VHD, VMDK, and raw disk image
formats.  The image download size is approximately 135 MB and 165 MB
respectively (amd64/i386), decompressing to a 21 GB sparse image.

Note regarding arm64/aarch64 virtual machine images: a modified QEMU EFI
loader file is needed for qemu-system-aarch64 to be able to boot the
virtual machine images.  See this page for more information:

https://wiki.freebsd.org/arm64/QEMU

To boot the VM image, run:

% qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt  \
-bios QEMU_EFI.fd -serial telnet::,server -nographic \
-drive if=none,file=VMDISK,id=hd0 \
-device virtio-blk-device,drive=hd0 \
-device virtio-net-device,netdev=net0 \
-netdev user,id=net0

Be sure to replace "VMDISK" with the path to the virtual machine image.

=== Amazon EC2 AMI Images ===

FreeBSD/amd64 EC2 AMI IDs are available in the Systems Manager Parameter
Store in each region using the keys:

/aws/service/freebsd/amd64/base/ufs/14.0/RC1
/aws/service/freebsd/amd64/base/zfs/14.0/RC1
/aws/service/freebsd/amd64/cloud-init/ufs/14.0/RC1
/aws/service/freebsd/amd64/cloud-init/zfs/14.0/RC1

FreeBSD/arm64 EC2 AMI IDs are available in the Systems Manager Parameter
Store in each region using the keys:

/aws/service/freebsd/arm64/base/ufs/14.0/RC1
/aws/service/freebsd/arm64/base/zfs/14.0/RC1
/aws/service/freebsd/arm64/cloud-init/ufs/14.0/RC1
/aws/service/freebsd/arm64/cloud-init/zfs/14.0/RC1

=== Vagrant Images ===

FreeBSD/amd64 images are not available for this build.

=== Upgrading ===

The freebsd-update(8) utility supports binary upgrades of amd64, i386,
and aarch64 systems running earlier FreeBSD releases.  Systems running
earlier FreeBSD releases can upgrade as follows:

# freebsd-update upgrade -r 14.0-RC1

During this process, freebsd-update(8) may ask the user to help by
merging some configuration files or by confirming that the automatically
performed merging was done correctly.

# freebsd-update install

The system must be rebooted with the newly installed kernel before
continuing.

# shutdown -r now

After rebooting, freebsd-update needs to be run again to install the new
userland components:

# freebsd-update install

It is recommended to rebuild and install all applications if possible,
especially if upgrading from an earlier FreeBSD release, for example,
FreeBSD 12.x.  Alternatively, the user can install misc/compat12x and
other compatibility 

vfs.zfs.arc.min (was: how to set vfs.zfs.arc.max in 15-current ?)

[Graham Perrin]

On 13/10/2023 07:46, Toomas Soome wrote:



On 13. Oct 2023, at 09:35, void  wrote:

On Fri, Oct 13, 2023 at 08:12:47AM +0200, Juraj Lutter wrote:


Set also vfs.zfs.arc.min to some value higher than zero.

aha! that worked!!! :D

root@beer:/root# sysctl vfs.zfs.arc.min=1073741824
vfs.zfs.arc.min: 0 -> 1073741824
root@beer:/root# sysctl vfs.zfs.arc.max=8589934592
vfs.zfs.arc.max: 0 -> 8589934592
root@beer:/root#
thank you!
--


That is a bit odd, arc min used to be 64MB as lowest value - that is, it should 
never be 0.

rgds,
toomas


0 here, when not set.

 
describes the effect of 0, 
 
on the same page does not.


A gap in documentation upstream?

Re: nvme timeout issues with hardware and bhyve vm's

[Pete Wright]

On 10/13/23 6:24 AM, Warner Losh wrote:



On Thu, Oct 12, 2023, 10:53 PM Pete Wright > wrote:




On 10/12/23 8:45 PM, Warner Losh wrote:
 > What version is that kernel?

oh dang i sent this to the wrong list, i'm not running current.  the
hypervisor and vm are both 13.2 and my workstation is a recent 14.0
pre-release build.  i'll do more homework tomorrow and post to
questions
or a more appropriate list.


Are the messages from the VM? Stable/14 should have the important nvme 
changes I've made lately. The bhyve in 13.2 is lacking a number of nvme 
fixes that have gone into current and stable/14. It's hard to say where 
the fault is coming from.





the messages i posted in the start of the thread are from the VM itself 
(13.2-RELEASE).  The zpool on the hypervisor (13.2-RELEASE) showed no 
such issues.


Based on your comment about the improvements in 14 I'll focus my efforts 
on my workstation, it seemed to happen regularly so hopefully i can find 
a repo case.


thanks warner!
-pete


--
Pete Wright
p...@nomadlogic.org

Re: issue: poudriere jail update fails after recent changes around certctl

[Guido Falsi]

On 13/10/23 17:42, Dag-Erling Smørgrav wrote:

Alexander Leidinger  writes:

some change around certctl (world from 2023-10-09) has broken the
poudriere jail update command. The complete install finishes, certctl
is run, and then there is an exit code 1. This is because I have some
certs listed as untrusted, and this seems to give a retval of 1 inside
certctl.


This only happens if a certificate is listed as both trusted and
untrusted, and I'm pretty sure the previous version would return 1 in
that case as well.  Can you check?



ON an unrelated note, I noticed the new certctl in head requires .pem 
(and any other extension, if I remember correctly) files to contain a 
"Certificate:" line.


While I have no objection to the requirement, it is not documented in 
the man page, or anywhere else, I did have to look at the script source 
to discover this.


Maybe a little note about this requirement in the man page should be added.

--
Guido Falsi 

Re: issue: poudriere jail update fails after recent changes around certctl

[Dag-Erling Smørgrav]

Alexander Leidinger  writes:
> some change around certctl (world from 2023-10-09) has broken the
> poudriere jail update command. The complete install finishes, certctl
> is run, and then there is an exit code 1. This is because I have some
> certs listed as untrusted, and this seems to give a retval of 1 inside
> certctl.

This only happens if a certificate is listed as both trusted and
untrusted, and I'm pretty sure the previous version would return 1 in
that case as well.  Can you check?

DES
-- 
Dag-Erling Smørgrav - d...@freebsd.org

Re: nvme timeout issues with hardware and bhyve vm's

[Warner Losh]

On Thu, Oct 12, 2023, 10:53 PM Pete Wright  wrote:

>
>
> On 10/12/23 8:45 PM, Warner Losh wrote:
> > What version is that kernel?
>
> oh dang i sent this to the wrong list, i'm not running current.  the
> hypervisor and vm are both 13.2 and my workstation is a recent 14.0
> pre-release build.  i'll do more homework tomorrow and post to questions
> or a more appropriate list.
>

Are the messages from the VM? Stable/14 should have the important nvme
changes I've made lately. The bhyve in 13.2 is lacking a number of nvme
fixes that have gone into current and stable/14. It's hard to say where the
fault is coming from.

Warner

-pete
>
> --
> Pete Wright
> p...@nomadlogic.org
>

Re: how to set vfs.zfs.arc.max in 15-current ?

[void]

On Fri, Oct 13, 2023 at 09:46:08AM +0300, Toomas Soome wrote:


That is a bit odd, arc min used to be 64MB as lowest value - that is, it should 
never be 0.


It was the default here (i.e. the value wasn't set in either loader.conf or
sysctl.conf)

world+kernel was built/installed today the 13th main-n265915-be91b4797e2c 
(amd64)
--

issue: poudriere jail update fails after recent changes around certctl

[Alexander Leidinger]

Hi,

some change around certctl (world from 2023-10-09) has broken the 
poudriere jail update command. The complete install finishes, certctl is 
run, and then there is an exit code 1. This is because I have some certs 
listed as untrusted, and this seems to give a retval of 1 inside 
certctl.


Testcase: set a cert as untrusted and try to use "poudriere jail -u -j 
YOUR_JAIL_NAME -m src=/usr/src"


Relevant log:
---snip---
--

Installing everything completed on Fri Oct 13 10:00:04 CEST 2023

--
   83.55 real   103.83 user   109.42 sys
certctl.sh: Skipping untrusted certificate ad088e1d 
(/space/poudriere/jails/poudriere-x11/etc/ssl/untrusted/ad088e1d.0)

[some more untrusted]
*** [installworld] Error code 1

make[1]: stopped in /space/system/usr_src
1 error

make[1]: stopped in /space/system/usr_src

make: stopped in /usr/src
[00:01:32] Error: Failed to 'make installworld'
---snip---

Bye,
Alexander.

--
http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.orgnetch...@freebsd.org  : PGP 0x8F31830F9F2772BF


signature.asc
Description: OpenPGP digital signature

Re: how to set vfs.zfs.arc.max in 15-current ?

[Toomas Soome]

> On 13. Oct 2023, at 09:35, void  wrote:
> 
> On Fri, Oct 13, 2023 at 08:12:47AM +0200, Juraj Lutter wrote:
> 
>> Set also vfs.zfs.arc.min to some value higher than zero.
> 
> aha! that worked!!! :D
> 
> root@beer:/root# sysctl vfs.zfs.arc.min=1073741824
> vfs.zfs.arc.min: 0 -> 1073741824
> root@beer:/root# sysctl vfs.zfs.arc.max=8589934592
> vfs.zfs.arc.max: 0 -> 8589934592
> root@beer:/root# 
> thank you!
> -- 
> 

That is a bit odd, arc min used to be 64MB as lowest value - that is, it should 
never be 0.

rgds,
toomas

Re: how to set vfs.zfs.arc.max in 15-current ?

[void]

On Fri, Oct 13, 2023 at 08:12:47AM +0200, Juraj Lutter wrote:


Set also vfs.zfs.arc.min to some value higher than zero.


aha! that worked!!! :D

root@beer:/root# sysctl vfs.zfs.arc.min=1073741824
vfs.zfs.arc.min: 0 -> 1073741824
root@beer:/root# sysctl vfs.zfs.arc.max=8589934592
vfs.zfs.arc.max: 0 -> 8589934592
root@beer:/root# 


thank you!
--

Re: how to set vfs.zfs.arc.max in 15-current ?

[Juraj Lutter]

> On 13 Oct 2023, at 00:59, void  wrote:
> 
>> Try reducing your arc.max by an order of 10. This suggests that it's
>> probably failing in param_set_arc_max() in the val >= arc_all_memory()
>> comparison..
> 
> Hi, thanks for replying. Sadly, your suggestion doesn't work in this case:
> 
> root@beer:/usr/src# sysctl vfs.zfs.arc.max=8589934592
> vfs.zfs.arc.max: 0
> sysctl: vfs.zfs.arc.max=8589934592: Invalid argument

Set also vfs.zfs.arc.min to some value higher than zero.

—
Juraj Lutter
o...@freebsd.org