Re: Filesystem operations slower in 13.0 than 12.2

[Mark Millard via freebsd-stable]

On 2021-Mar-14, at 11:09, Kevin Oberman  wrote:

> . . .
>  
> Seems to only occur on large r/w operations from/to the same disk. "sp
> big-file /other/file/on/same/disk" or tar/untar operations on large files.
> Hit this today updating firefox.
> 
> I/O starts at >40MB/s. Dropped to about 1.5MB/s. If I tried doing other
> things while it was running slowly, the disk would appear to lock up. E.g.
> pwd(1) seemed to completely lock up the system, but I could still ping it
> and, after about 30 seconds, things came back to life. It was also not
> instantaneous. Disc activity dropped to <1MB/s for a few seconds before
> everything froze.
> 
> During the untar of firefox, I saw; this several times. I also looked at my
> console where I found these errors during :
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 55043, size: 8192
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 51572, size: 4096

Does anyone know:
Are those messages normal "reading is taking a rather long
time" notices or is their presence more useful information
in some way about the type of problem or context for the
problem?

As for the tests:
Are these messages always present when near a time frame
when the problem occurs? Never present in a near time
frame to a period when the problem does not occur?

It appears that the messages are associated with reading
the disk(s), not directly with writing them, where the
reads take more than "hz * 20" time units to complete.
(I'm looking at main (14) code.) What might contribute
to the time taken for the pending read(s)?

/*
 * swap_pager_getpages() - bring pages in from swap
 *
 *  Attempt to page in the pages in array "ma" of length "count".  The
 *  caller may optionally specify that additional pages preceding and
 *  succeeding the specified range be paged in.  The number of such pages
 *  is returned in the "rbehind" and "rahead" parameters, and they will
 *  be in the inactive queue upon return.
 *
 *  The pages in "ma" must be busied and will remain busied upon return.
 */
static int
swap_pager_getpages_locked(vm_object_t object, vm_page_t *ma, int count,
int *rbehind, int *rahead)
{
. . .
/*
 * Wait for the pages we want to complete.  VPO_SWAPINPROG is always
 * cleared on completion.  If an I/O error occurs, SWAPBLK_NONE
 * is set in the metadata for each page in the request.
 */
VM_OBJECT_WLOCK(object);
/* This could be implemented more efficiently with aflags */
while ((ma[0]->oflags & VPO_SWAPINPROG) != 0) {
ma[0]->oflags |= VPO_SWAPSLEEP;
VM_CNT_INC(v_intrans);
if (VM_OBJECT_SLEEP(object, >handle, PSWP,
"swread", hz * 20)) {
printf(
"swap_pager: indefinite wait buffer: bufobj: %p, blkno: %jd, size: %ld\n",
bp->b_bufobj, (intmax_t)bp->b_blkno, bp->b_bcount);
}
}
VM_OBJECT_WUNLOCK(object);
. . .

where:

#define VM_OBJECT_SLEEP(object, wchan, pri, wmesg, timo)\
rw_sleep((wchan), &(object)->lock, (pri), (wmesg), (timo))

and:

#define rw_sleep(chan, rw, pri, wmesg, timo)\
_sleep((chan), &(rw)->lock_object, (pri), (wmesg),  \
tick_sbt * (timo), 0, C_HARDCLOCK)

(I do not claim to be able to interpret the implications
of the code that leads to the messages. But seeing some
of the code might prompt a thought by someone that knows
the code's context and operation.)

> . . .
> Backing off to Mar. 4 was not an improvement. My untar did seem better for a 
> couple of minutes, but then the display froze again for 30 seconds and disk 
> performance dropped to <1M.

You were able to see the disk performance drop while
the display was frozen?

It might not be the best for monitoring but I'll ask
this in terms of top output: Does Inact, Laundry,
Wired, Free, or other such show anything fairly unique
for around the problematical time frame(s)?

> then things got really bad and behaved in a manner that was baffling to me. 
> The screen froze again, but stayed frozen after half a minute. I clicked on a 
> couple of buttons in Firefox to no effect and then hit ctrl-q to quit. After 
> the long pause, I pressed the power button to try to force a shutdown. 
> Suddenly, it started unwinding everything I had done during the freeze. My 
> browser did the updates from my mouse clicks including quitting. It then 
> switched to a different workspace from ctrl-alt-right and did a clean 
> shutdown.  
> 
> Do I also have a graphics issue? Examining log files show no indication that 
> anything was happening. SMART shows no errors and reasonable values for 
> everything. No indication of a HW problem. The system performs well unless I 
> do something that tries a bulk disk data move. Building world takes about 75 
> minutes. I just have a very hard time building big ports.

Re: Filesystem operations slower in 13.0 than 12.2

[Kevin Oberman]

On Sat, Mar 13, 2021 at 9:43 PM Kevin Oberman  wrote:

> No improvement with stable/13-n244880-cec3990d347.  It may be worse.
> worse. An attempt to unpack firefox-86.0.1,2 saw disk rates in the range
> of 800K to 2 MB/s range and with repeated 30 second freezes. I have no idea
> what made it so much worse, but I'm forced to start wondering if it could
> be a hardware issue. The disk drive was already replaced once due to a bad
> bearing. Went from a WD Black to a Seagate. Since it just keeps getting
> worse, I must consider that possibility. It is odd, though, that it was
> suddenly worse with the updated system.
>
> I think I will try going back to n244765-a00bf7d9bba (March 4) and see if
> it improves. If it does, I can likely eliminate bad hardware.
> --
> Kevin Oberman, Part time kid herder and retired Network Engineer
> E-mail: rkober...@gmail.com
> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
>
>
> On Sat, Mar 13, 2021 at 6:00 PM Warner Losh  wrote:
>
>>
>>
>> On Sat, Mar 13, 2021 at 6:37 PM Kevin Oberman 
>> wrote:
>>
>>> I have been dealing with this for a long time since head back in
>>> September through 13-stable of Mar-4. I have seen no improvement over this
>>> time. It seems (my perception without supporting data) that it got worse in
>>> the timeframe of BETA-3 tag. I was running stable, so not quite BETA-3. It
>>> also does not help that I have also been bitten by the P-State related
>>> freeze issue which has some similarities. disabling p-states has almost
>>> eliminated this issue, though, with only three occurrences since I disabled
>>> them in late January.
>>>
>>>  As a result, I don't think it is a recent change, but a problem that
>>> has existed for at least 3 months. This was made worse by two hardware
>>> issues that kept the system unavailable for most of the time between buying
>>> it last spring and getting the keyboard replaced in January. (Both the
>>> mainboard and the disk drive had already been replaced.)  There was another
>>> slow I/O issue that I had assumed was the same as mine, but was reportedly
>>> fixed with BETA-4. A few are still seeing slow I/O, so I assume that there
>>> were different issues with I/O. Since CometLake systems seem pretty
>>> uncommon, it might be related to that.
>>>
>>
>> It was a change from last fall, or set of changes. RC1 or defintely RC2
>> has fixes to regain performance lost. If BETA4 was the last one you
>> evaluated, perhaps you could do a couple tests with RC2 now that it's out
>> to see if it is the same thing?
>>
>> Warner
>>
>>
>>> Kevin Oberman, Part time kid herder and retired Network Engineer
>>> E-mail: rkober...@gmail.com
>>> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
>>>
>>>
>>> On Sat, Mar 13, 2021 at 4:36 PM Warner Losh  wrote:
>>>


 On Sat, Mar 13, 2021 at 5:33 PM Kevin Oberman 
 wrote:

> Just spent a little time looking at my issue and have a few more notes:
>

 What version did you evaluate? There's a number of changes lately that
 could have a big impact on this...

 Warner


> Seems to only occur on large r/w operations from/to the same disk. "sp
> big-file /other/file/on/same/disk" or tar/untar operations on large
> files.
> Hit this today updating firefox.
>
> I/O starts at >40MB/s. Dropped to about 1.5MB/s. If I tried doing other
> things while it was running slowly, the disk would appear to lock up.
> E.g.
> pwd(1) seemed to completely lock up the system, but I could still ping
> it
> and, after about 30 seconds, things came back to life. It was also not
> instantaneous. Disc activity dropped to <1MB/s for a few seconds before
> everything froze.
>
> During the untar of firefox, I saw; this several times. I also looked
> at my
> console where I found these errors during :
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 55043, size: 8192
> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 51572, size: 4096
>
> I should note that some operations continue just fine while this is
> going
> on until I do something that freezes the system. I assume that this
> eliminates the disk drive and low-level driver. Is vfs a possible
> issue. It
> had some serious work in the past few months by markj. That does not
> explain why more people are not seeing this.
>
> I have been seeing this since at least September 2020, so it goes back
> a
> way. As this CometLake system will not run graphics on 12, I can't
> confirm
> operation before 13.
> --
> Kevin Oberman, Part time kid herder and retired Network Engineer
> E-mail: rkober...@gmail.com
> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
>
>
> On Fri, Mar 5, 2021 at 10:47 PM Mark Millard via freebsd-stable <
> freebsd-stable@freebsd.org> wrote:
>
> >
> > Konstantin Belousov kostikbel at gmail.com 

Re: FreeBSD 13.0 RC1 UEFI RAID-10 boot problems under VMware Fusion

[Ruben van Staveren via freebsd-stable]

I’ve updated the installation to RC2, the behaviour has changed in vmware 
fusion to when I installed one of the  BETA releases.

The initial boot fails, the system will reboot again and then it succeeds, 
though more predictable than with the BETAs

I’m trying to see the differences in screen recording I made, and I can only 
see the “Image base” is changed.
Load path, load device, bootcurrent, bootinfo path, currdev (disk0p1, EFI)) has 
not changed.

Either something in the loader(s) was changed, or the RC1 to RC2 update placed 
contents on more “favourable" places on disk that made it possible to boot 
again.

But I do not understand how the loader interacts with the (Vmware) EFI firmware 
to make sense out of it.

> On 10 Mar 2021, at 12:22, Ruben van Staveren  wrote:
> 
> To continue on the subject of UEFI booting weirdness
> 
>> On 9 Mar 2021, at 16:57, Ruben van Staveren  wrote:
>> 
>> If I press escape and end up in VMWare’s UEFI setup screen I can boot from 
>> any ada*p1 drive and continue as normal.
>> Is UEFI with OpenZFS too new, or is this an issue in VMWare?
> 
> I got an off list tip to see whether this was also the case in bhyve, so I 
> also created the setup in there, using UEFI boot, and no problems even with 
> the special/log/cache NVMe vdevs attached to the pool.
> 
> So I’m starting to wonder whether the loader  / VMware UEFI firmware (??) 
> interaction is a bug in VMware or an edge case that needs to be supported too.
> 
> Btw, bhyve is looking nice these days!
> 
> Cheers,
>Ruben
> 



signature.asc
Description: Message signed with OpenPGP

Re: Second pool not mounted automaticly on 13.0-RC2

[Johan Hendriks]

On 13/03/2021 19:43, Warner Losh wrote:



On Sat, Mar 13, 2021 at 9:44 AM Johan Hendriks > wrote:


On 13/03/2021 17:09, Johan Hendriks wrote:
> Hello all, i just upgrade my test server from 12.1 to 13.0-RC2.
This
> is a baremetal server.
> It has two ssd's on ada0 and ada1 using zfs. Also there is a 6 disk
> pool named storage.
>
> After the update the storage pool is not imported any more. I have
> updated the pool, thinking it was necessary, but the result is the
> same, after a reboot the storage pool is not imported.
>
> root@test-server:~ # zpool list
> NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG CAP DEDUP
> HEALTH  ALTROOT
> zroot   107G  1.60G   105G    - - 0% 1% 1.00x
> ONLINE  -
> root@test-server:~ # zpool import
>    pool: storage
>  id: 17039452293665227158
>   state: ONLINE
>  action: The pool can be imported using its name or numeric
identifier.
>  config:
>
>     storage ONLINE
>       mirror-0  ONLINE
>         gpt/disk00  ONLINE
>         gpt/disk01  ONLINE
>       mirror-1  ONLINE
>         gpt/disk02  ONLINE
>         gpt/disk03  ONLINE
>       mirror-2  ONLINE
>         gpt/disk04  ONLINE
>         gpt/disk05  ONLINE
> root@test-server:~ #
>
> After importing the pool all is fine but after a reboot it needs
to be
> imported again.
>
> Did something change?
> I observed this earlier on a HEAD which was 13 at that time (around
> januari) but did not have the time to look at it. That was a
baremetal
> server also.
>
For the completeness, i have the following related settings in
/boot/loader.conf.

# ZFS
zfs_load="YES"

And in my /etc/rc.conf the following.
# ZFS
zfs_enable="YES"


Have you updated your rc scripts? I had the same problem a while ago 
until I did that.


Warner

I did do a mergemaster, but it seems mergemaster is not updating the 
files in /etc/rc.d anymore.
Copied the files from /usr/src/libexec/rc/rc.d to /etc/rc.d make it work 
again.


Thank you for the pointer, time to take a look at etcupdate.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: FreeBSD 13.0-RC2 Now Available

[Nenhum_de_Nos]

On Fri, March 12, 2021 21:11, Glen Barber wrote:
> === Upgrading ===
>
> The freebsd-update(8) utility supports binary upgrades of amd64 and i386
systems running earlier FreeBSD releases.  Systems running earlier
FreeBSD releases can upgrade as follows:
>
>   # freebsd-update upgrade -r 13.0-RC2

Hi Glen,

this announce mentions just amd64 and i386 as able to use freebsd-update.
On the BETA2 announce there was a mention of aarch64 as well
(https://lists.freebsd.org/pipermail/freebsd-stable/2021-February/093081.html).

I then tried to use it now on a 13.0-BETA4 system:

root@rpi4_bsd:~ # freebsd-update -r 13.0-RC2 upgrade
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 13.0-BETA4 from update4.freebsd.org...
done. Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/generic kernel/generic-dbg world/base world/base-dbg

The following components of FreeBSD do not seem to be installed:

Does this look reasonable (y/n)?

The system is:

FreeBSD rpi4_bsd 13.0-BETA4 FreeBSD 13.0-BETA4 #0
releng/13.0-n244592-e32bc253629: Fri Feb 26 05:53:09 UTC 2021
r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64


I did not typed "y" yet. Is it safe to go through with it?

Thanks.

matheus

-- 
"We will call you Cygnus,
the God of balance you shall be."


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Updating to 13-stable and existing ZFS pools: any gotchas?

[Mathias Picker]

I’ve updated two machines and had no problems, read the mailing 
lists and remember no problem apart from the boot loader.


Updated the boot loader before upgrading the pools, found the way 
for EFI is different then I remembered (mount EFI partition, copy 
/boot/loader.efi to it) but it worked just fine..


So, while I cannot 100% guarantee it’s without problems, I would 
expect no hiccups. And the new bootloader is only needed if/once 
you upgrade your pools, I actually run my laptop for a week with 
quite an old loader ;)


Cheers, Mathias

Stefan Bethke  writes:

I'm planning to upgrade three production machines with existing 
ZFS pools to 13-stable. Is there anything I need to pay 
attention to wrt OpenZFS? Or should it be fully transparent, 
apart from updating loader?


My (limited) testing with VMs went without a hitch, but I want 
to make sure I don't paint myself into a corner.



Stefan



--
Mathias Picker
Geschäftsführer

mathias.pic...@virtual-earth.de

virtual earth Gesellschaft für Wissens re/prä sentation mbH
http://www.virtual-earth.de/   HRB126870
supp...@virtual-earth.de   Westendstr. 142
089 / 1250 3943
___

freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Updating to 13-stable and existing ZFS pools: any gotchas?

[Stefan Bethke]

I'm planning to upgrade three production machines with existing ZFS pools to 
13-stable. Is there anything I need to pay attention to wrt OpenZFS? Or should 
it be fully transparent, apart from updating loader?

My (limited) testing with VMs went without a hitch, but I want to make sure I 
don't paint myself into a corner.


Stefan

--
Stefan BethkeFon +49 151 14070811



signature.asc
Description: Message signed with OpenPGP