Re: Possible regression in main causing poor performance

2023-08-18 Thread Mark Millard 
On Aug 18, 2023, at 19:09, Mark Millard  wrote:

> Glen Barber  wrote on
> Date: Sat, 19 Aug 2023 00:10:59 UTC :
> 
>> I am somewhat inclined to look in the direction of ZFS here, as two
>> things changed:
>> 
>> 1) the build machine in question was recently (as in a week and a half
>> ago) upgraded to the tip of main in order to ease the transition from
>> this machine from building 14.x to building 15.x;
>> 2) there is the recent addition of building ZFS-backed virtual machine
>> and cloud images.
>> 
>> . . .
>> The first machine runs:
>> # uname -a
>> FreeBSD releng1.nyi.freebsd.org 14.0-CURRENT FreeBSD 14.0-CURRENT \
>> amd64 1400093 #5 main-n264224-c84617e87a70: Wed Jul 19 19:10:38 UTC 2023
> 
> I'm confused:
> 
> "the build machine in question was recently (as in a week and a half
> ago) upgraded to the tip of main in order to ease the transition from
> this machine from building 14.x to building 15.x"? But the above
> kernel is from mid July? (-aKU was not used to also get some clue
> about world from the pair of 140009? that would show.)
> 
>> Last week's snapshot builds were completed in a reasonable amount of
>> time:
>> 
>> r...@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c 
>> ./builds-14.conf ; echo ^G
>> 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/logs
>> 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/chroots
>> 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/release
>> 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/ports
>> 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/doc
>> 20230811-00:03:13 INFO: Checking out https://git.FreeBSD.org//src.git (main) 
>> to /releng/scripts-snapshot/release
>> [...]
>> 20230811-15:11:13 INFO: Staging for ftp: 14-i386-GENERIC-snap
>> 20230811-16:27:28 INFO: Staging for ftp: 14-amd64-GENERIC-snap
>> 20230811-16:33:43 INFO: Staging for ftp: 14-aarch64-GENERIC-snap
>> 
>> Overall, 17 hours, including the time to upload EC2, Vagrant, and GCE.
>> 
>> With no changes to the system, no stale ZFS datasets laying around from
>> last week (everything is a pristine environment, etc.), this week's
>> builds are taking forever:
> 
> My confusion may extend to this "no changes" status vs. the uname
> output identifying the kernel is from mid July.
> 
>> r...@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c 
>> ./builds-14.conf ; echo ^G
>> 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/logs
>> 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/chroots
>> 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/release
>> 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/ports
>> 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/doc
>> 20230818-00:15:46 INFO: Checking out https://git.FreeBSD.org//src.git (main) 
>> to /releng/scripts-snapshot/release
>> [...]
>> 20230818-18:46:22 INFO: Staging for ftp: 14-aarch64-ROCKPRO64-snap
>> 20230818-20:41:02 INFO: Staging for ftp: 14-riscv64-GENERIC-snap
>> 20230818-22:54:49 INFO: Staging for ftp: 14-amd64-GENERIC-snap
>> 
>> Note, it is just about 4 minutes past 00:00 UTC as of this writing, so
>> we are about to cross well over the 24-hour mark, and cloud provider
>> images have not yet even started.
>> 
>> . . .
> 
> In:
> 
> https://lists.freebsd.org/archives/freebsd-current/2023-August/004314.html
> ("HEADS UP: $FreeBSD$ Removed from main", Wed, 16 Aug 2023)
> 
> Warner wrote:
> 
> QUOTE
> . . . , but there's no incremental building
> with this change, . . . Also: expect long build times, git fetch times, etc
> after this.
> END QUOTE
> 
> Might this be contributing? How long did those two
> "Checking out . . ." take? Similar time frames?
> 

The build process and information is not available. So
I looked at something I thought might have a chance
of being somewhat invariant and have a limited range of
types of (parallel) activity: time differences for the
CHECKSUM files taht have timestamps after the last
*.img* timestamp, as seen via:

http://ftp3.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/14.0/?C=M=D
(so: most recent to oldest as displayed)

First today's:

CHECKSUM.SHA256-FreeBSD-14.0-ALPHA2-arm64-aarch64-20230819-77013f29d048-264841 
1232 2023-Aug-19 00:26
CHECKSUM.SHA512-FreeBSD-14.0-ALPHA2-arm64-aarch64-20230819-77013f29d048-264841 
1744 2023-Aug-19 00:25
CHECKSUM.SHA256-FreeBSD-14.0-ALPHA2-amd64-20230818-77013f29d048-264841 1168 
2023-Aug-18 22:59
CHECKSUM.SHA512-FreeBSD-14.0-ALPHA2-amd64-20230818-77013f29d048-264841 1680 
2023

Re: Possible regression in main causing poor performance

2023-08-18 Thread Glen Barber 
On Fri, Aug 18, 2023 at 07:09:10PM -0700, Mark Millard wrote:
> Glen Barber  wrote on
> Date: Sat, 19 Aug 2023 00:10:59 UTC :
> 
> > I am somewhat inclined to look in the direction of ZFS here, as two
> > things changed:
> > 
> > 1) the build machine in question was recently (as in a week and a half
> > ago) upgraded to the tip of main in order to ease the transition from
> > this machine from building 14.x to building 15.x;
> > 2) there is the recent addition of building ZFS-backed virtual machine
> > and cloud images.
> > 
> > . . .
> > The first machine runs:
> > # uname -a
> > FreeBSD releng1.nyi.freebsd.org 14.0-CURRENT FreeBSD 14.0-CURRENT \
> > amd64 1400093 #5 main-n264224-c84617e87a70: Wed Jul 19 19:10:38 UTC 2023
> 
> I'm confused:
> 
> "the build machine in question was recently (as in a week and a half
> ago) upgraded to the tip of main in order to ease the transition from
> this machine from building 14.x to building 15.x"? But the above
> kernel is from mid July? (-aKU was not used to also get some clue
> about world from the pair of 140009? that would show.)
> 

Err, right.  I'm confused too, and apparently looked at a different
machine but copied 'uname -a' from the correct machine.  The kernel and
userland are always in sync on these machines.

  # uname -UK
  1400093 1400093

> > Last week's snapshot builds were completed in a reasonable amount of
> > time:
> > 
> > r...@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c 
> > ./builds-14.conf ; echo ^G
> > 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/logs
> > 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/chroots
> > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/release
> > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/ports
> > 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/doc
> > 20230811-00:03:13 INFO: Checking out https://git.FreeBSD.org//src.git 
> > (main) to /releng/scripts-snapshot/release
> > [...]
> > 20230811-15:11:13 INFO: Staging for ftp: 14-i386-GENERIC-snap
> > 20230811-16:27:28 INFO: Staging for ftp: 14-amd64-GENERIC-snap
> > 20230811-16:33:43 INFO: Staging for ftp: 14-aarch64-GENERIC-snap
> > 
> > Overall, 17 hours, including the time to upload EC2, Vagrant, and GCE.
> > 
> > With no changes to the system, no stale ZFS datasets laying around from
> > last week (everything is a pristine environment, etc.), this week's
> > builds are taking forever:
> 
> My confusion may extend to this "no changes" status vs. the uname
> output identifying the kernel is from mid July.
> 

It is admittedly a month versus two weeks difference here.  My fault.

> > r...@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c 
> > ./builds-14.conf ; echo ^G
> > 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/logs
> > 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/chroots
> > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/release
> > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/ports
> > 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/doc
> > 20230818-00:15:46 INFO: Checking out https://git.FreeBSD.org//src.git 
> > (main) to /releng/scripts-snapshot/release
> > [...]
> > 20230818-18:46:22 INFO: Staging for ftp: 14-aarch64-ROCKPRO64-snap
> > 20230818-20:41:02 INFO: Staging for ftp: 14-riscv64-GENERIC-snap
> > 20230818-22:54:49 INFO: Staging for ftp: 14-amd64-GENERIC-snap
> > 
> > Note, it is just about 4 minutes past 00:00 UTC as of this writing, so
> > we are about to cross well over the 24-hour mark, and cloud provider
> > images have not yet even started.
> > 
> > . . .
> 
> In:
> 
> https://lists.freebsd.org/archives/freebsd-current/2023-August/004314.html
> ("HEADS UP: $FreeBSD$ Removed from main", Wed, 16 Aug 2023)
> 
> Warner wrote:
> 
> QUOTE
> . . . , but there's no incremental building
> with this change, . . . Also: expect long build times, git fetch times, etc
> after this.
> END QUOTE
> 
> Might this be contributing? How long did those two
> "Checking out . . ." take? Similar time frames?
> 

Checkout times are negligible.  They are pristine environments, so there
are no delta resolutions, etc.  Everything is clean from the start.

Last week:
20230811-00:03:13   INFO:   Checking out https://git.FreeBSD.org//src.git 
(main) to /releng/scripts-snapshot/release
20230811-00:05:27   INFO:   Creating ZFS snapshot 
zroot/releng/14-src-snap@clone
20230811-00:05:33   INFO:   Checking out https://git.FreeBSD.org//ports.git 
(main) to

Re: Possible regression in main causing poor performance

2023-08-18 Thread Mark Millard 
Glen Barber  wrote on
Date: Sat, 19 Aug 2023 00:10:59 UTC :

> I am somewhat inclined to look in the direction of ZFS here, as two
> things changed:
> 
> 1) the build machine in question was recently (as in a week and a half
> ago) upgraded to the tip of main in order to ease the transition from
> this machine from building 14.x to building 15.x;
> 2) there is the recent addition of building ZFS-backed virtual machine
> and cloud images.
> 
> . . .
> The first machine runs:
> # uname -a
> FreeBSD releng1.nyi.freebsd.org 14.0-CURRENT FreeBSD 14.0-CURRENT \
> amd64 1400093 #5 main-n264224-c84617e87a70: Wed Jul 19 19:10:38 UTC 2023

I'm confused:

"the build machine in question was recently (as in a week and a half
ago) upgraded to the tip of main in order to ease the transition from
this machine from building 14.x to building 15.x"? But the above
kernel is from mid July? (-aKU was not used to also get some clue
about world from the pair of 140009? that would show.)

> Last week's snapshot builds were completed in a reasonable amount of
> time:
> 
> r...@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c 
> ./builds-14.conf ; echo ^G
> 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/logs
> 20230811-00:03:11 INFO: Creating /releng/scripts-snapshot/chroots
> 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/release
> 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/ports
> 20230811-00:03:12 INFO: Creating /releng/scripts-snapshot/doc
> 20230811-00:03:13 INFO: Checking out https://git.FreeBSD.org//src.git (main) 
> to /releng/scripts-snapshot/release
> [...]
> 20230811-15:11:13 INFO: Staging for ftp: 14-i386-GENERIC-snap
> 20230811-16:27:28 INFO: Staging for ftp: 14-amd64-GENERIC-snap
> 20230811-16:33:43 INFO: Staging for ftp: 14-aarch64-GENERIC-snap
> 
> Overall, 17 hours, including the time to upload EC2, Vagrant, and GCE.
> 
> With no changes to the system, no stale ZFS datasets laying around from
> last week (everything is a pristine environment, etc.), this week's
> builds are taking forever:

My confusion may extend to this "no changes" status vs. the uname
output identifying the kernel is from mid July.

> r...@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c 
> ./builds-14.conf ; echo ^G
> 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/logs
> 20230818-00:15:44 INFO: Creating /releng/scripts-snapshot/chroots
> 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/release
> 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/ports
> 20230818-00:15:45 INFO: Creating /releng/scripts-snapshot/doc
> 20230818-00:15:46 INFO: Checking out https://git.FreeBSD.org//src.git (main) 
> to /releng/scripts-snapshot/release
> [...]
> 20230818-18:46:22 INFO: Staging for ftp: 14-aarch64-ROCKPRO64-snap
> 20230818-20:41:02 INFO: Staging for ftp: 14-riscv64-GENERIC-snap
> 20230818-22:54:49 INFO: Staging for ftp: 14-amd64-GENERIC-snap
> 
> Note, it is just about 4 minutes past 00:00 UTC as of this writing, so
> we are about to cross well over the 24-hour mark, and cloud provider
> images have not yet even started.
> 
> . . .

In:

https://lists.freebsd.org/archives/freebsd-current/2023-August/004314.html
("HEADS UP: $FreeBSD$ Removed from main", Wed, 16 Aug 2023)

Warner wrote:

QUOTE
. . . , but there's no incremental building
with this change, . . . Also: expect long build times, git fetch times, etc
after this.
END QUOTE

Might this be contributing? How long did those two
"Checking out . . ." take? Similar time frames?


===
Mark Millard
marklmi at yahoo.com

New FreeBSD snapshots available: main (ALPHA2) (20230819 77013f29d048)

2023-08-18 Thread Glen Barber 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

New FreeBSD development branch installation ISOs and virtual machine
disk images have been uploaded to the FreeBSD Project mirrors.

As with any development branch, the installation snapshots are not
intended for use on production systems.  We do, however, encourage
testing on non-production systems as much as possible.

Please also consider installing the sysutils/panicmail port, which can
help in providing FreeBSD developers the necessary information regarding
system crashes.

Checksums for the installation ISOs and the VM disk images follow at
the end of this email.

=== Installation ISOs ===

Installation images are available for:

o 14.0-ALPHA2 amd64 GENERIC
o 14.0-ALPHA2 i386 GENERIC
o 14.0-ALPHA2 powerpc GENERIC
o 14.0-ALPHA2 powerpc64 GENERIC64
o 14.0-ALPHA2 powerpc64le GENERIC64LE
o 14.0-ALPHA2 powerpcspe MPC85XXSPE
o 14.0-ALPHA2 armv6 RPI-B
o 14.0-ALPHA2 armv7 GENERICSD
o 14.0-ALPHA2 aarch64 GENERIC
o 14.0-ALPHA2 aarch64 RPI
o 14.0-ALPHA2 aarch64 PINE64
o 14.0-ALPHA2 aarch64 PINE64-LTS
o 14.0-ALPHA2 aarch64 PINEBOOK
o 14.0-ALPHA2 aarch64 ROCK64
o 14.0-ALPHA2 aarch64 ROCKPRO64
o 14.0-ALPHA2 riscv64 GENERIC
o 14.0-ALPHA2 riscv64 GENERICSD

Note regarding arm/armv{6,7} images: For convenience for those without
console access to the system, a freebsd user with a password of
freebsd is available by default for ssh(1) access.  Additionally,
the root user password is set to root, which it is strongly
recommended to change the password for both users after gaining
access to the system.

Snapshots may be downloaded from the corresponding architecture
directory from:

https://download.freebsd.org/snapshots/ISO-IMAGES/

Please be patient if your local mirror has not yet caught up with the
changes.

Problems, bug reports, or regression reports should be reported through
the Bugzilla PR system or the appropriate mailing list such as -current@
or -stable@ .

=== Virtual Machine Disk Images ===

VM disk images are available for the following architectures:

o 14.0-ALPHA2 amd64
o 14.0-ALPHA2 i386
o 14.0-ALPHA2 aarch64
o 14.0-ALPHA2 riscv64
o 14.0-ALPHA2 amd64 BASIC-CI

Disk images may be downloaded from the following URL (or any of the
FreeBSD Project mirrors):

https://download.freebsd.org/snapshots/VM-IMAGES/

Images are available in the following disk image formats:

~ RAW
~ QCOW2 (qemu)
~ VMDK (qemu, VirtualBox, VMWare)
~ VHD (qemu, xen)

The partition layout is:

~ 512k - freebsd-boot GPT partition type (bootfs GPT label)
~ 1GB  - freebsd-swap GPT partition type (swapfs GPT label)
~ 24GB - freebsd-ufs GPT partition type (rootfs GPT label)

Note regarding arm64/aarch64 virtual machine images: a modified QEMU EFI
loader file is needed for qemu-system-aarch64 to be able to boot the
virtual machine images.  This is provided by the emulators/qemu port.

To boot the VM image, run:

% qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt  
-bios /usr/local/share/qemu/edk2-aarch64-code.fd 
-serial telnet::,server -nographic 
-drive if=none,file=VMDISK,id=hd0 
-device virtio-blk-device,drive=hd0 
-device virtio-net-device,netdev=net0 
-netdev user,id=net0

Be sure to replace "VMDISK" with the path to the virtual machine image.

BASIC-CI images can be found at:

https://download.freebsd.org/snapshots/VM-IMAGES/

=== Amazon EC2 AMI Images ===

FreeBSD amd64 and aarch64 EC2 AMIs are available for both UFS and ZFS.
The AMI IDs can be retreived from the Systems Manager Parameter Store in
each region using the keys:

/aws/service/freebsd/amd64/base/ufs/14.0/ALPHA2
/aws/service/freebsd/arm64/base/ufs/14.0/ALPHA2

/aws/service/freebsd/amd64/base/zfs/14.0/ALPHA2
/aws/service/freebsd/arm64/base/zfs/14.0/ALPHA2

=== Vagrant Images ===

FreeBSD/amd64 images are available on the Hashicorp Atlas site for the
VMWare Desktop and VirtualBox providers, and can be installed by
running:

% vagrant init freebsd/FreeBSD-14.0-ALPHA2
% vagrant up

== ISO CHECKSUMS ==

o 14.0-ALPHA2 amd64 GENERIC:
  SHA512 (FreeBSD-14.0-ALPHA2-amd64-20230818-77013f29d048-264841-bootonly.iso) 
= 
02b4a7ba3a9520031607e6ebcbdf6cd142044aa1af40ad452c9d9793c5e70ebad94b13e679f5cb0aaa473e20f9da1f63b335b0cd25323aa03b14bd539b565ee1
  SHA512 
(FreeBSD-14.0-ALPHA2-amd64-20230818-77013f29d048-264841-bootonly.iso.xz) = 
966041afcb3df8cc35063592d67ad146453783941bff80fa019d42df6e624dca2a8763940f20a7cc3857bf708e6f76e1420ed18af4a1d287cec004bdd31893ee
  SHA512 (FreeBSD-14.0-ALPHA2-amd64-20230818-77013f29d048-264841-disc1.iso) = 
76f8ff46e679d4920cfe67179a587abdfd323e83c3e3f79f39678f6a956b888cf4a2a1962f9a0549a1fb481912ef41fb666bd518dc1775e90488c31b841d6aca
  SHA512 (FreeBSD-14.0-ALPHA2-amd64-20230818-77013f29d048-264841-disc1.iso.xz) 
= 
c61970abd7f2b7c0a81d5bf702e3cc65129843e3624b1a93a1476ffe535adb8dbb13378b3b049d2e742e47383c3f6bb3b2b0678977ee11873c0b50414070ea1e
  SHA512 (FreeBSD-1

Possible regression in main causing poor performance

2023-08-18 Thread Glen Barber 
I am somewhat inclined to look in the direction of ZFS here, as two
things changed:

1) the build machine in question was recently (as in a week and a half
   ago) upgraded to the tip of main in order to ease the transition from
   this machine from building 14.x to building 15.x;
2) there is the recent addition of building ZFS-backed virtual machine
   and cloud images.

Here is a bit of the history of the build machines:

- the second and first build 13.x and 14.x, respectively.  The third is
  (at the time it was purchased) newer than the other two, and
  significantly fast when it comes to weekly snapshots.  I'll refer to
  these machines as such: first, second, and third.

- The third machine will be eventually used to build 14.0-RELEASE.  The
  will first continue on main, where it will build 15.0-CURRENT
  snapshots.  The second will continue tracking 13-STABLE.

The first machine runs:
  # uname -a
  FreeBSD releng1.nyi.freebsd.org 14.0-CURRENT FreeBSD 14.0-CURRENT \
amd64 1400093 #5 main-n264224-c84617e87a70: Wed Jul 19 19:10:38 UTC 2023

Last week's snapshot builds were completed in a reasonable amount of
time:

  r...@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c 
./builds-14.conf ; echo ^G
  20230811-00:03:11   INFO:   Creating /releng/scripts-snapshot/logs
  20230811-00:03:11   INFO:   Creating /releng/scripts-snapshot/chroots
  20230811-00:03:12   INFO:   Creating /releng/scripts-snapshot/release
  20230811-00:03:12   INFO:   Creating /releng/scripts-snapshot/ports
  20230811-00:03:12   INFO:   Creating /releng/scripts-snapshot/doc
  20230811-00:03:13   INFO:   Checking out https://git.FreeBSD.org//src.git 
(main) to /releng/scripts-snapshot/release
  [...]
  20230811-15:11:13   INFO:   Staging for ftp: 14-i386-GENERIC-snap
  20230811-16:27:28   INFO:   Staging for ftp: 14-amd64-GENERIC-snap
  20230811-16:33:43   INFO:   Staging for ftp: 14-aarch64-GENERIC-snap

Overall, 17 hours, including the time to upload EC2, Vagrant, and GCE.

With no changes to the system, no stale ZFS datasets laying around from
last week (everything is a pristine environment, etc.), this week's
builds are taking forever:

  r...@releng1.nyi:/releng/scripts-snapshot/scripts # ./thermite.sh -c 
./builds-14.conf ; echo ^G
  20230818-00:15:44   INFO:   Creating /releng/scripts-snapshot/logs
  20230818-00:15:44   INFO:   Creating /releng/scripts-snapshot/chroots
  20230818-00:15:45   INFO:   Creating /releng/scripts-snapshot/release
  20230818-00:15:45   INFO:   Creating /releng/scripts-snapshot/ports
  20230818-00:15:45   INFO:   Creating /releng/scripts-snapshot/doc
  20230818-00:15:46   INFO:   Checking out https://git.FreeBSD.org//src.git 
(main) to /releng/scripts-snapshot/release
  [...]
  20230818-18:46:22   INFO:   Staging for ftp: 14-aarch64-ROCKPRO64-snap
  20230818-20:41:02   INFO:   Staging for ftp: 14-riscv64-GENERIC-snap
  20230818-22:54:49   INFO:   Staging for ftp: 14-amd64-GENERIC-snap

Note, it is just about 4 minutes past 00:00 UTC as of this writing, so
we are about to cross well over the 24-hour mark, and cloud provider
images have not yet even started.

I am inclined to do two things:

1) Immediately run a subsequent snapshot build to see if it takes longer
   than 24 hours (my gut tells me it will);
2) Reboot the machine with no other changes, and immediately run this
   week's snapshot builds again (not to be public to avoid confusion).

Some interactive commands are significantly slower, such as systat,
vmstat, even top.  Creating a new window in tmux is also noticeably
slow.

Is there a third option I am overlooking in trying to identify the
drastic cause of this?

Thank you in advance.

Glen



signature.asc
Description: PGP signature

Re: ZFS deadlock in 14

2023-08-18 Thread Mark Millard 
I believe the below quoted messages were reports of deadlocks
based on after the following 2 MFV being in place in their
environments:

Thu, 10 Aug 2023
. . .
• git: cd25b0f740f8 - main - zfs: cherry-pick fix from openzfs Martin 
Matuska 
• git: 28d2e3b5dedf - main - zfs: cherry-pick fix from openzfs Martin 
Matuska

Kevin Bowling  wrote on
Date: Fri, 11 Aug 2023 03:32:51 UTC :

> Spoke too soon still seeing zfs lockups under heavy poudriere workload
> after the MFVs. Regression time matches what has been reported here.
> . . .
> > In message
> >  > om>
> > , Kevin Bowling writes:
> > > The two MFVs on head have improved/fixed stability with poudriere for
> > > me 48 core bare metal.

Cy Schubert  wrote on
Date: Sat, 12 Aug 2023 04:55:40 UTC :

> The poudriere build machine building amd64 packages also panicked. But with:
> 
> Dumping 2577 out of 8122 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91
> %
> 
> __curthread () at /opt/src/git-src/sys/amd64/include/pcpu_aux.h:59
> 59 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct 
> pcpu
> ,
> (kgdb) #0 __curthread () at /opt/src/git-src/sys/amd64/include/pcpu_aux.h:5
> 9
> #1 doadump (textdump=textdump@entry=1)
> at /opt/src/git-src/sys/kern/kern_shutdown.c:407
> #2 0x806c10e0 in kern_reboot (howto=260)
> at /opt/src/git-src/sys/kern/kern_shutdown.c:528
> #3 0x806c15df in vpanic (
> fmt=0x80b6c5f5 "%s: possible deadlock detected for %p (%s), 
> blocked
> for %d ticks\n", ap=ap@entry=0xfe008e698e90)
> at /opt/src/git-src/sys/kern/kern_shutdown.c:972
> #4 0x806c1383 in panic (fmt=)
> at /opt/src/git-src/sys/kern/kern_shutdown.c:896
> #5 0x8064a5ea in deadlkres ()
> at /opt/src/git-src/sys/kern/kern_clock.c:201
> #6 0x80677632 in fork_exit (callout=0x8064a2c0 ,
> arg=0x0, frame=0xfe008e698f40)
> at /opt/src/git-src/sys/kern/kern_fork.c:1162
> #7 
> (kgdb)
> 




===
Mark Millard
marklmi at yahoo.com

Re: ZFS deadlock in 14

2023-08-18 Thread Dag-Erling Smørgrav 
Dag-Erling Smørgrav  writes:
> Plot twist: c47116e909 _without_ the patches also appears to be working
> fine.  The last kernel I know for sure deadlocks is b36f469a15, so I'm
> going to test cd25b0f740 and 28d2e3b5de.

c47116e909 with cd25b0f740 and 28d2e3b5de reverted deadlocks, see
attached ddb.txt.  I'm going to see if reverting only 28d2e3b5de but not
cd25b0f740 changes anything.

DES
-- 
Dag-Erling Smørgrav - d...@freebsd.org



ddb.txt.gz
Description: Binary data

Re: make-memstick.sh creates in 14.0-CURRENT run-away processes

2023-08-18 Thread Matthias Apitz 


I filed this PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273211
about the issue.

El día viernes, agosto 18, 2023 a las 06:17:42p. m. +0200, Matthias Apitz 
escribió:

> 
> I was used to use in 13.0-CURRENT the script "make-memstick.sh" to
> create memstick immages to install the system on smaller devices where
> the OS can't build from the sources, and it always worked fine for many
> years. Now I'm ready to do so with my fresh compiled system (sources
> from git August, 5:
> 
> $ uname -a
> FreeBSD jet 14.0-CURRENT FreeBSD 14.0-CURRENT amd64 1400094 #0 
> main-n264568-1d7ffb373c9d: Sat Aug  5 17:22:47 CEST 2023 
> guru@jet:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
> 
> but the image is not produces and some processes create temp
> files of 800++ GByte. Here are the details:
> 
> root@jet:/usr/src/release/amd64 # ./make-memstick.sh /home/guru/140.root 
> ~guru/memstick.img
> Calculated size of `/home/guru/memstick.img.part': 23795073024 bytes, 263113 
> inodes
> Extent size set to 32768
> /home/guru/memstick.img.part: 22692.8MB (46474752 sectors) block size 32768, 
> fragment size 4096
> using 27 cylinder groups of 869.44MB, 27822 blks, 10240 inodes.
> super-block backups (for fsck -b #) at:
>   192,  1780800,  3561408,  5342016,  7122624,  8903232, 10683840,
>  12464448, 14245056, 16025664, 17806272, 19586880, 21367488, 23148096,
>  24928704, 26709312, 28489920, 30270528, 32051136, 33831744, 35612352,
>  37392960, 39173568, 40954176, 42734784, 44515392, 46296000
> Populating `/home/guru/memstick.img.part'
> Image `/home/guru/memstick.img.part' complete
> Creating `/tmp/efiboot.iFachZ'
> /tmp/efiboot.iFachZ: 65528 sectors in 65528 FAT32 clusters (512 bytes/cluster)
> BytesPerSec=512 SecPerClust=1 ResSectors=32 FATs=2 Media=0xf0 SecPerTrack=63 
> Heads=255 HiddenSecs=0 HugeSectors=66584 FATsecs=512 RootCluster=2 FSInfo=1 
> Backup=2
> Populating `/tmp/efiboot.iFachZ'
> Image `/tmp/efiboot.iFachZ' complete
> 
> It says 'complete' but never ends growing the file /tmp/mkimg-oGNnFb:
> 
> root@jet:/usr/home/guru # ls -ltrah /tmp | tail -6
> drwx--   2 guru wheel  512B Aug 18 15:43 tmux-1001
> -rw---   1 root wheel   33M Aug 18 17:18 efiboot.iFachZ
> -rw---   1 root wheel0B Aug 18 17:18 mkimg-4eMWKW
> drwxrwxrwt  21 root wheel  1.0K Aug 18 17:18 .
> drwxr-xr-x  22 root wheel  1.0K Aug 18 17:43 ..
> -rw---   1 root wheel  850G Aug 18 17:53 mkimg-oGNnFb
> 
> root@jet:/usr/home/guru # ls -ltrh mem*
> -rw-r--r--  1 root wheel   22G Aug 18 17:18 memstick.img.part
> -rw-r--r--  1 root wheel0B Aug 18 17:18 memstick.img
> 
> Only a hard reset and reboot helps.
> 
> What does it mean?
> 
>   matthias
> 
> 
> -- 
> Matthias Apitz, ✉ g...@unixarea.de, http://www.unixarea.de/ +49-176-38902045
> Public GnuPG key: http://www.unixarea.de/key.pub
> 

-- 
Matthias Apitz, ✉ g...@unixarea.de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub

Re: ZFS deadlock in 14

2023-08-18 Thread Mark Millard 
[I had sent to the wrong list. Just fixing that here.]

On Aug 18, 2023, at 09:26, Mark Millard  wrote:

> Dag-Erling_Smørgrav  wrote on
> Date: Fri, 18 Aug 2023 14:16:12 UTC :
> 
>> Dag-Erling Smørgrav  writes:
>>> A kernel built from c47116e909 plus these two patches has so far built
>>> 2,285 packages without a hitch, whereas normally it would have
>>> deadlocked after well before reaching 500 packages. I'll do another run
>>> without the patches tomorrow just to be sure.
>> 
>> Plot twist: c47116e909 _without_ the patches also appears to be working
>> fine. The last kernel I know for sure deadlocks is b36f469a15, so I'm
>> going to test cd25b0f740 and 28d2e3b5de.
> 
> I'm confused:
> 
> Tue, 18 Jul 2023
> . . .
>• git: b36f469a15ec - main - zfs: set autotrim default to 'off' Yuri Pankov
> 
> But you had written:
> 
> Dag-Erling_Smørgrav 
> Date: Tue, 15 Aug 2023 17:37:45 UTC 
> The attached script successfully deadlocks 9228ac3a69c4
> 
> where:
> 
> Thu, 10 Aug 2023
> . . .
>• git: 9228ac3a69c4 - main - ixgbe: Add support for 82599 LS Kevin Bowling
> 
> (You had previously reported a poudriere deadlock there as well.)
> 
> Later on 2023-Aug-10 there is (about 11 or so commits later):
> 
>• git: cd25b0f740f8 - main - zfs: cherry-pick fix from openzfs Martin 
> Matuska 
>• git: 28d2e3b5dedf - main - zfs: cherry-pick fix from openzfs Martin 
> Matuska
> 
> 
> I'll note there was also this:
> 
> Dag-Erling_Smørgrav 
> Date: Sat, 12 Aug 2023 14:11:10 UTC 
> . . .
> Trying to narrow this range down, I did not get a deadlock with
> 4e8d558c9d1c (10 June) but I did with b7198dcfc039 (16 June), albeit
> after building ~1800 packages.
> . . .
> 
> So the count seems to sometimes go well above 500 or so.



===
Mark Millard
marklmi at yahoo.com

make-memstick.sh creates in 14.0-CURRENT run-away processes

2023-08-18 Thread Matthias Apitz 


I was used to use in 13.0-CURRENT the script "make-memstick.sh" to
create memstick immages to install the system on smaller devices where
the OS can't build from the sources, and it always worked fine for many
years. Now I'm ready to do so with my fresh compiled system (sources
from git August, 5:

$ uname -a
FreeBSD jet 14.0-CURRENT FreeBSD 14.0-CURRENT amd64 1400094 #0 
main-n264568-1d7ffb373c9d: Sat Aug  5 17:22:47 CEST 2023 
guru@jet:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

but the image is not produces and some processes create temp
files of 800++ GByte. Here are the details:

root@jet:/usr/src/release/amd64 # ./make-memstick.sh /home/guru/140.root 
~guru/memstick.img
Calculated size of `/home/guru/memstick.img.part': 23795073024 bytes, 263113 
inodes
Extent size set to 32768
/home/guru/memstick.img.part: 22692.8MB (46474752 sectors) block size 32768, 
fragment size 4096
using 27 cylinder groups of 869.44MB, 27822 blks, 10240 inodes.
super-block backups (for fsck -b #) at:
  192,  1780800,  3561408,  5342016,  7122624,  8903232, 10683840,
 12464448, 14245056, 16025664, 17806272, 19586880, 21367488, 23148096,
 24928704, 26709312, 28489920, 30270528, 32051136, 33831744, 35612352,
 37392960, 39173568, 40954176, 42734784, 44515392, 46296000
Populating `/home/guru/memstick.img.part'
Image `/home/guru/memstick.img.part' complete
Creating `/tmp/efiboot.iFachZ'
/tmp/efiboot.iFachZ: 65528 sectors in 65528 FAT32 clusters (512 bytes/cluster)
BytesPerSec=512 SecPerClust=1 ResSectors=32 FATs=2 Media=0xf0 SecPerTrack=63 
Heads=255 HiddenSecs=0 HugeSectors=66584 FATsecs=512 RootCluster=2 FSInfo=1 
Backup=2
Populating `/tmp/efiboot.iFachZ'
Image `/tmp/efiboot.iFachZ' complete

It says 'complete' but never ends growing the file /tmp/mkimg-oGNnFb:

root@jet:/usr/home/guru # ls -ltrah /tmp | tail -6
drwx--   2 guru wheel  512B Aug 18 15:43 tmux-1001
-rw---   1 root wheel   33M Aug 18 17:18 efiboot.iFachZ
-rw---   1 root wheel0B Aug 18 17:18 mkimg-4eMWKW
drwxrwxrwt  21 root wheel  1.0K Aug 18 17:18 .
drwxr-xr-x  22 root wheel  1.0K Aug 18 17:43 ..
-rw---   1 root wheel  850G Aug 18 17:53 mkimg-oGNnFb

root@jet:/usr/home/guru # ls -ltrh mem*
-rw-r--r--  1 root wheel   22G Aug 18 17:18 memstick.img.part
-rw-r--r--  1 root wheel0B Aug 18 17:18 memstick.img

Only a hard reset and reboot helps.

What does it mean?

matthias


-- 
Matthias Apitz, ✉ g...@unixarea.de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub

Re: ZFS deadlock in 14

2023-08-18 Thread Dag-Erling Smørgrav 
Dag-Erling Smørgrav  writes:
> A kernel built from c47116e909 plus these two patches has so far built
> 2,285 packages without a hitch, whereas normally it would have
> deadlocked after well before reaching 500 packages.  I'll do another run
> without the patches tomorrow just to be sure.

Plot twist: c47116e909 _without_ the patches also appears to be working
fine.  The last kernel I know for sure deadlocks is b36f469a15, so I'm
going to test cd25b0f740 and 28d2e3b5de.

DES
-- 
Dag-Erling Smørgrav - d...@freebsd.org

Re: Speed improvements in ZFS

2023-08-18 Thread Mateusz Guzik 
On 8/18/23, Alexander Leidinger  wrote:
> Am 2023-08-16 18:48, schrieb Alexander Leidinger:
>> Am 2023-08-15 23:29, schrieb Mateusz Guzik:
>>> On 8/15/23, Alexander Leidinger  wrote:
 Am 2023-08-15 14:41, schrieb Mateusz Guzik:

> With this in mind can you provide: sysctl kern.maxvnodes
> vfs.wantfreevnodes vfs.freevnodes vfs.vnodes_created vfs.numvnodes
> vfs.recycles_free vfs.recycles

 After a reboot:
 kern.maxvnodes: 10485760
 vfs.wantfreevnodes: 2621440
 vfs.freevnodes: 24696
 vfs.vnodes_created: 1658162
 vfs.numvnodes: 173937
 vfs.recycles_free: 0
 vfs.recycles: 0
>>
>> New values after one rund of periodic:
>> kern.maxvnodes: 10485760
>> vfs.wantfreevnodes: 2621440
>> vfs.freevnodes: 356202
>> vfs.vnodes_created: 427696288
>> vfs.numvnodes: 532620
>> vfs.recycles_free: 20213257
>> vfs.recycles: 0
>
> And after the second round which only took 7h this night:
> kern.maxvnodes: 10485760
> vfs.wantfreevnodes: 2621440
> vfs.freevnodes: 3071754
> vfs.vnodes_created: 1275963316
> vfs.numvnodes: 3414906
> vfs.recycles_free: 58411371
> vfs.recycles: 0
>

so your setup has a vastly higher number of vnodes to inspect than the
number of vnodes it allows to exist at the same time, which further
suggests it easily may be about that msleep.

> Meanwhile if there is tons of recycles, you can damage control by
> bumping kern.maxvnodes.
>>
>> What's the difference between recycles and recycles_free? Does the
>> above count as bumping the maxvnodes?
>
> ^
>

"free" vnodes are just hanging around and can be directly whacked, the
others are used but *maybe* freeable (say a directory with a bunch of
vnodes already established).

 Looks like there are not much free directly after the reboot. I will
 check the values tomorrow after the periodic run again and maybe
 increase by 10 or 100 so see if it makes a difference.

> If this is not the problem you can use dtrace to figure it out.

 dtrace-count on vnlru_read_freevnodes() and vnlru_free_locked()? Or
 something else?

>>>
>>> I mean checking where find is spending time instead of speculating.
>>>
>>> There is no productized way to do it so to speak, but the following
>>> crapper should be good enough:
>> [script]
>>
>> I will let it run this night.
>
> I have a 51MB text file, compressed to about 1MB. Are you interested to
> get it?
>

Yea, put it on freefall for example.

or feed it directly to flamegraph: cat file | ./stackcollapse.pl |
./flamegraph.pl > out.svg

see this repo https://github.com/brendangregg/FlameGraph.git


-- 
Mateusz Guzik

Re: Status for zfs upgrade?

2023-08-18 Thread Tomoaki AOKI 
Hi.

Thanks for the pointer!
Although the data in the page are a bit outdated, but the methods to
determine features described there looks helpful.
(For example, the problematic block_cloning feature is not listed yet.)

And yes, I've noticed the Alpha2 commit, but ATM release schedule page
was not updated yet.

Regards.


On Fri, 18 Aug 2023 12:36:59 +0200 (CEST)
Ronald Klop  wrote:

> Hi,
> 
> This blog might be interesting to you.
> https://vermaden.wordpress.com/2022/03/25/zfs-compatibility/
> 
> BTW: stable/14 is delayed one week: 
> https://www.freebsd.org/releases/14.0R/schedule/
> 
> Regards,
> Ronald.
>  
> Van: Tomoaki AOKI 
> Datum: vrijdag, 18 augustus 2023 00:00
> Aan: freebsd-current@freebsd.org
> Onderwerp: Status for zfs upgrade?
> > 
> > Hi.
> > 
> > Creation of stable/14 is planned at Aug.18, 2023 (Already TODAY in
> > Japan, JST+9).
> > 
> > Is it safe for now to `zfs upgrade `, if tunable
> > vfs.zfs.bclone_enabled is set to 0?
> > 
> > If not, I should wait until next stable branch, /15.
> > (I upgrade pool only when ZFS codes are 100% match between main and
> > latest stable. Meaning doable only when new stable branch is created.)
> > 
> > Regards.
> > 
> > -- 
> > Tomoaki AOKI
> >  
> > 
> > 
> > 
> 
>  


-- 
Tomoaki AOKI

Boot failures with 14.0-ALPHA1

2023-08-18 Thread Graham Perrin 

I'll update to ALPHA2.

In the meantime, briefly: 

– no visible progress beyond the four lines of EFI framebuffer information.

In each case, I worked around by removing a USB flash drive that's used 
for L2ARC.


The most recent incident was preceded by a power button shutdown, in 
response to a drm-510-kmod blackout (GPU lockup etc.) whilst typing in 
Firefox (comparable to  
for drm-515-kmod 5.15.25).


The prior incident was also preceded by a power button shutdown (in 
response to a blackout at Plasma log out time). This was remarkable in 
that there was an unexpected restart, when I expected the computer to 
power off.

Re: Status for zfs upgrade?

2023-08-18 Thread Ronald Klop 

Hi,

This blog might be interesting to you.
https://vermaden.wordpress.com/2022/03/25/zfs-compatibility/

BTW: stable/14 is delayed one week: 
https://www.freebsd.org/releases/14.0R/schedule/

Regards,
Ronald.

Van: Tomoaki AOKI 
Datum: vrijdag, 18 augustus 2023 00:00
Aan: freebsd-current@freebsd.org
Onderwerp: Status for zfs upgrade?


Hi.

Creation of stable/14 is planned at Aug.18, 2023 (Already TODAY in
Japan, JST+9).

Is it safe for now to `zfs upgrade `, if tunable
vfs.zfs.bclone_enabled is set to 0?

If not, I should wait until next stable branch, /15.
(I upgrade pool only when ZFS codes are 100% match between main and
latest stable. Meaning doable only when new stable branch is created.)

Regards.

--
Tomoaki AOKI

Re: Speed improvements in ZFS

2023-08-18 Thread Alexander Leidinger 

Am 2023-08-16 18:48, schrieb Alexander Leidinger:

Am 2023-08-15 23:29, schrieb Mateusz Guzik:

On 8/15/23, Alexander Leidinger  wrote:

Am 2023-08-15 14:41, schrieb Mateusz Guzik:


With this in mind can you provide: sysctl kern.maxvnodes
vfs.wantfreevnodes vfs.freevnodes vfs.vnodes_created vfs.numvnodes
vfs.recycles_free vfs.recycles


After a reboot:
kern.maxvnodes: 10485760
vfs.wantfreevnodes: 2621440
vfs.freevnodes: 24696
vfs.vnodes_created: 1658162
vfs.numvnodes: 173937
vfs.recycles_free: 0
vfs.recycles: 0


New values after one rund of periodic:
kern.maxvnodes: 10485760
vfs.wantfreevnodes: 2621440
vfs.freevnodes: 356202
vfs.vnodes_created: 427696288
vfs.numvnodes: 532620
vfs.recycles_free: 20213257
vfs.recycles: 0


And after the second round which only took 7h this night:
kern.maxvnodes: 10485760
vfs.wantfreevnodes: 2621440
vfs.freevnodes: 3071754
vfs.vnodes_created: 1275963316
vfs.numvnodes: 3414906
vfs.recycles_free: 58411371
vfs.recycles: 0


Meanwhile if there is tons of recycles, you can damage control by
bumping kern.maxvnodes.


What's the difference between recycles and recycles_free? Does the 
above count as bumping the maxvnodes?


^


Looks like there are not much free directly after the reboot. I will
check the values tomorrow after the periodic run again and maybe
increase by 10 or 100 so see if it makes a difference.


If this is not the problem you can use dtrace to figure it out.


dtrace-count on vnlru_read_freevnodes() and vnlru_free_locked()? Or
something else?



I mean checking where find is spending time instead of speculating.

There is no productized way to do it so to speak, but the following
crapper should be good enough:

[script]

I will let it run this night.


I have a 51MB text file, compressed to about 1MB. Are you interested to 
get it?


Bye,
Alexander.

--
http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.orgnetch...@freebsd.org  : PGP 0x8F31830F9F2772BF

Re: www/chromium will not build on a host w/ 8 CPU and 16G mem

2023-08-18 Thread Matthias Apitz 
El día miércoles, agosto 16, 2023 a las 03:34:03p. m. +0200, Matthias Apitz 
escribió:

> Thanks for all the hints I got so far. I started already the build with 
> MAKE_JOBS_UNSAFE=yes. It's running now for some 8 hours and has built
> 32% (as the log says). So it will need some 16 hours more...

After 1,5 days I stopped this again and set the following kernel values.
With these the port www/chromium build fine in 14:27 hours and without
any swap/page problems.

#!/bin/sh
#
# kernel values to apply for running poudriere on my build host
# Dell R210 with 8x 3.30GHz CPU and 15.8 GB memory; applied and
# tested for building the port www/chromium (build time 14:27h)
#
# values found in:
# 
https://lists.freebsd.org/archives/freebsd-questions/2022-February/000666.html# 
publishes by tech-lists  in reply to
# Kevin Oberman , see full thread there;
#
# August, 2023

sysctl vfs.read_max=128
sysctl vfs.aio.max_buf_aio=8192
sysctl vfs.aio.max_aio_queue_per_proc=65536
sysctl vfs.aio.max_aio_per_proc=8192
sysctl vfs.aio.max_aio_queue=65536
sysctl vm.pageout_oom_seq=120
sysctl vm.pfault_oom_attempts=-1 

# the default values are:
#
# vfs.read_max: 64
# vfs.aio.max_buf_aio: 16
# vfs.aio.max_aio_queue_per_proc: 256
# vfs.aio.max_aio_per_proc: 32
# vfs.aio.max_aio_queue: 1024
# vm.pageout_oom_seq: 12
# vm.pfault_oom_attempts: 3

-- 
Matthias Apitz, ✉ g...@unixarea.de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub