Re: Possible regression in main causing poor performance

[Mark Millard]

Has any more been learned about this? Is it still an issue?

===
Mark Millard
marklmi at yahoo.com

Re: Speed improvements in ZFS

[Alexander Leidinger]

Am 2023-08-22 18:59, schrieb Mateusz Guzik:

On 8/22/23, Alexander Leidinger  wrote:

Am 2023-08-21 10:53, schrieb Konstantin Belousov:

On Mon, Aug 21, 2023 at 08:19:28AM +0200, Alexander Leidinger wrote:

Am 2023-08-20 23:17, schrieb Konstantin Belousov:
> On Sun, Aug 20, 2023 at 11:07:08PM +0200, Mateusz Guzik wrote:
> > On 8/20/23, Alexander Leidinger  wrote:
> > > Am 2023-08-20 22:02, schrieb Mateusz Guzik:
> > >> On 8/20/23, Alexander Leidinger  wrote:
> > >>> Am 2023-08-20 19:10, schrieb Mateusz Guzik:
> >  On 8/18/23, Alexander Leidinger 
> >  wrote:
> > >>>
> > > I have a 51MB text file, compressed to about 1MB. Are you
> > > interested
> > > to
> > > get it?
> > >
> > 
> >  Your problem is not the vnode limit, but nullfs.
> > 
> >  https://people.freebsd.org/~mjg/netchild-periodic-find.svg
> > >>>
> > >>> 122 nullfs mounts on this system. And every jail I setup has
> > >>> several
> > >>> null mounts. One basesystem mounted into every jail, and then
> > >>> shared
> > >>> ports (packages/distfiles/ccache) across all of them.
> > >>>
> >  First, some of the contention is notorious VI_LOCK in order to
> >  do
> >  anything.
> > 
> >  But more importantly the mind-boggling off-cpu time comes from
> >  exclusive locking which should not be there to begin with -- as
> >  in
> >  that xlock in stat should be a slock.
> > 
> >  Maybe I'm going to look into it later.
> > >>>
> > >>> That would be fantastic.
> > >>>
> > >>
> > >> I did a quick test, things are shared locked as expected.
> > >>
> > >> However, I found the following:
> > >> if ((xmp->nullm_flags & NULLM_CACHE) != 0) {
> > >> mp->mnt_kern_flag |=
> > >> lowerrootvp->v_mount->mnt_kern_flag &
> > >> (MNTK_SHARED_WRITES | MNTK_LOOKUP_SHARED |
> > >> MNTK_EXTENDED_SHARED);
> > >> }
> > >>
> > >> are you using the "nocache" option? it has a side effect of
> > >> xlocking
> > >
> > > I use noatime, noexec, nosuid, nfsv4acls. I do NOT use nocache.
> > >
> >
> > If you don't have "nocache" on null mounts, then I don't see how
> > this
> > could happen.
>
> There is also MNTK_NULL_NOCACHE on lower fs, which is currently set
> for
> fuse and nfs at least.

11 of those 122 nullfs mounts are ZFS datasets which are also NFS
exported.
6 of those nullfs mounts are also exported via Samba. The NFS 
exports

shouldn't be needed anymore, I will remove them.

By nfs I meant nfs client, not nfs exports.


No NFS client mounts anywhere on this system. So where is this 
exclusive

lock coming from then...
This is a ZFS system. 2 pools: one for the root, one for anything I 
need
space for. Both pools reside on the same disks. The root pool is a 
3-way

mirror, the "space-pool" is a 5-disk raidz2. All jails are on the
space-pool. The jails are all basejail-style jails.



While I don't see why xlocking happens, you should be able to dtrace
or printf your way into finding out.


dtrace looks to me like a faster approach to get to the root than 
printf... my first naive try is to detect exclusive locks. I'm not 100% 
sure I got it right, but at least dtrace doesn't complain about it:

---snip---
#pragma D option dynvarsize=32m

fbt:nullfs:null_lock:entry
/args[0]->a_flags & 0x08 != 0/
{
stack();
}
---snip---

In which direction should I look with dtrace if this works in tonights 
run of periodic? I don't have enough knowledge about VFS to come up with 
some immediate ideas.


Bye,
Alexander.

--
http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.orgnetch...@freebsd.org  : PGP 0x8F31830F9F2772BF

Re: zfs autotrim default to off now

[Pete Wright]

On 8/28/23 12:00, Alexander Motin wrote:

On 28.08.2023 13:56, Pete Wright wrote:
So to be clear, if we were using the default autotrim=enabled behavior 
we in fact weren't having our SSDs trimmed?  I think that's my 
concern, as an admin I was under the impression that it was enabled by 
default but apparently that wasn't actually happening.


We wanted autotrim to be enabled by default, but it was not enabled, and 
it was reported as not enabled, so there should be no confusion.  The 
only confusion may have been if you tried to read the code and see it 
should have been enabled.




ok, thanks Alexander!

-pete

--
Pete Wright
p...@nomadlogic.org
@nomadlogicLA

Re: zfs autotrim default to off now

[Bob Bishop]

Hi,

> On 28 Aug 2023, at 20:00, Alexander Motin  wrote:
> 
> On 28.08.2023 13:56, Pete Wright wrote:
>> So to be clear, if we were using the default autotrim=enabled behavior we in 
>> fact weren't having our SSDs trimmed?  I think that's my concern, as an 
>> admin I was under the impression that it was enabled by default but 
>> apparently that wasn't actually happening.
> 
> We wanted autotrim to be enabled by default, but it was not enabled, and it 
> was reported as not enabled, so there should be no confusion.  The only 
> confusion may have been if you tried to read the code and see it should have 
> been enabled.
> 
> -- 
> Alexander Motin

Prompted by this thread, I went to check one of my servers that predates the 
change to OpenZFS. I set autotrim=on at the time of the transition; at some 
point since, it had become unset (and not by me). The system has been upgraded 
from time to time and is currently running 13.2R. 

--
Bob Bishop
r...@gid.co.uk

Re: zfs autotrim default to off now

[Alexander Motin]

On 28.08.2023 13:56, Pete Wright wrote:
So to be clear, if we were using the default autotrim=enabled behavior 
we in fact weren't having our SSDs trimmed?  I think that's my concern, 
as an admin I was under the impression that it was enabled by default 
but apparently that wasn't actually happening.


We wanted autotrim to be enabled by default, but it was not enabled, and 
it was reported as not enabled, so there should be no confusion.  The 
only confusion may have been if you tried to read the code and see it 
should have been enabled.


--
Alexander Motin

Re: Possible issue with linux xattr support?

[James Gritton]

On 2023-08-28 05:17, Alexander Leidinger wrote:

Am 2023-08-28 13:06, schrieb Dmitry Chagin:

On Sun, Aug 27, 2023 at 09:55:23PM +0200, Felix Palmen wrote:

* Dmitry Chagin  [20230827 22:46]:



> I can fix this completely disabling exttatr for jailed proc,
> however, it's gonna be bullshit, though

Would probably be better than nothing. AFAIK, "Linux jails" are used 
a

lot, probably with userlands from distributions actually using xattr.



It might sense to allow this priv (PRIV_VFS_EXTATTR_SYSTEM) for linux
jails by default? What do think, James?


I think the question is more if we want to allow it in jails (not
specific to linux jails, as in: if it is ok for linux jails, it should
be ok for FreeBSD jails too). So the question is what does this
protect the hosts from, if this is not allowed in jails? Some kind of
possibility to DoS the host?


It's definitely an any-jail question, as there's no kernel-level idea
of a Linux jail, in that any jail on a system with the linux module
loaded can run whatever Linux binaries may exist.

- Jamie

Re: zfs autotrim default to off now

[Pete Wright]

On 8/28/23 07:23, Alexander Motin wrote:

Hi Pete,

On 27.08.2023 23:34, Pete Wright wrote:

looking at a recent pull of CURRENT i'm noticing this in the git logs:

#15079 set autotrim default to 'off' everywhere

which references this openzfs PR:
https://github.com/openzfs/zfs/pull/15079


looking at the PR i'm not seeing a reference to a bug report or 
anything, is
anyone able to point me to a bug report for this.  it seems like a 
pretty major

issue:
"As it turns out having autotrim default to 'on' on FreeBSD never 
really worked
due to mess with defines where userland and kernel module were getting 
different
default values (userland was defaulting to 'off', module was thinking 
it's

'on')."

i'd just like to make sure i better understand the issue and can see 
if my

systems are impacted.


You are probably misinterpreting the quote.  There is nothing wrong with 
the autotrim itself, assuming your specific devices properly handle it. 
It is just saying that setting it to "on" by default on FreeBSD, that 
was done to keep pre-OpenZFS behavior, appeared broken for a while.  So 
that commit merely confirmed the status quo.  It should not affect any 
already existing pools.  On a new pool creation the default is now 
officially "off", matching OpenZFS on other platforms, but there is no 
reason why you can not set it to "on", if it is beneficial for your 
devices and workloads.  As alternative, for example, you may run trim 
manually from time to time during any low activity periods.





OK I think that makes sense.

So to be clear, if we were using the default autotrim=enabled behavior 
we in fact weren't having our SSDs trimmed?  I think that's my concern, 
as an admin I was under the impression that it was enabled by default 
but apparently that wasn't actually happening.


-pete



--
Pete Wright
p...@nomadlogic.org
@nomadlogicLA

Re: Possible issue with linux xattr support?

[Dmitry Chagin]

On Mon, Aug 28, 2023 at 08:03:33AM +0200, Felix Palmen wrote:
> * Cy Schubert  [20230827 16:59]:
> > On August 27, 2023 12:55:23 PM PDT, Felix Palmen  wrote:
> > >* Dmitry Chagin  [20230827 22:46]:
> > >> On Sun, Aug 27, 2023 at 07:59:32PM +0200, Felix Palmen wrote:
> > >> > * Dmitry Chagin  [20230827 20:54]:
> > >> > > 1. which fs are you using?
> > >> > 
> > >> > ZFS.
> > >> > 
> > >> > > 2. jailed?
> > >> > 
> > >> > Yes, this is during building ports with poudriere.
> > >> > 
> > >> 
> > >> I think it's a weird prohibition on changing system namespace extattr
> > >> attributes, look to comments in extattr_check_cred()
> > >
> > >Maybe that's when I should finally start trying to understand the stuff
> > >in src.git ;)
> > >
> > >> I can fix this completely disabling exttatr for jailed proc,
> > >> however, it's gonna be bullshit, though
> > >
> > >Would probably be better than nothing. AFAIK, "Linux jails" are used a
> > >lot, probably with userlands from distributions actually using xattr.
> > >
> > >Cheers, Felix
> > >
> > 
> > If we are to break it to fix a problem, maybe a sysctl to enable/disable 
> > then?
> 
> IMHO depends on the exact nature of the problem. If it's confirmed that
> it (always and only) breaks for jailed processes, just disabling it for
> them would be the better workaround. "No-op" calls won't break anything.
> 

please, try: https://people.freebsd.org/~dchagin/xattrerror.patch

etcupdate -p, and other uses of etcupdate(8)

[Graham Perrin]

On 28/08/2023 09:53, Tomoaki AOKI wrote:

On Sun, 27 Aug 2023 18:16:35 +0100
Graham Perrin  wrote:


On 07/08/2023 16:51, Kevin Oberman wrote:

…

…  "etcupdate -p". …

  …


Is this fortune-provided tip outdated?

… for src updates, no [3].



[3]
https://lists.freebsd.org/archives/freebsd-current/2023-August/004210.html



This, in particular:


run "etcupdate" after every upgrade



I mean, I never ran the command without an option, just once at an 
indeterminate stage after an upgrade.


I learnt to run it first with option -p _before_ installworld (i.e. 
during an upgrade), then a run with option -B after installworld.

Re: zfs autotrim default to off now

[Alexander Motin]

Hi Pete,

On 27.08.2023 23:34, Pete Wright wrote:

looking at a recent pull of CURRENT i'm noticing this in the git logs:

#15079 set autotrim default to 'off' everywhere

which references this openzfs PR:
https://github.com/openzfs/zfs/pull/15079


looking at the PR i'm not seeing a reference to a bug report or anything, is
anyone able to point me to a bug report for this.  it seems like a pretty major
issue:
"As it turns out having autotrim default to 'on' on FreeBSD never really worked
due to mess with defines where userland and kernel module were getting different
default values (userland was defaulting to 'off', module was thinking it's
'on')."

i'd just like to make sure i better understand the issue and can see if my
systems are impacted.


You are probably misinterpreting the quote.  There is nothing wrong with 
the autotrim itself, assuming your specific devices properly handle it. 
It is just saying that setting it to "on" by default on FreeBSD, that 
was done to keep pre-OpenZFS behavior, appeared broken for a while.  So 
that commit merely confirmed the status quo.  It should not affect any 
already existing pools.  On a new pool creation the default is now 
officially "off", matching OpenZFS on other platforms, but there is no 
reason why you can not set it to "on", if it is beneficial for your 
devices and workloads.  As alternative, for example, you may run trim 
manually from time to time during any low activity periods.


--
Alexander Motin

Re: Possible issue with linux xattr support?

[Alexander Leidinger]

Am 2023-08-28 13:06, schrieb Dmitry Chagin:

On Sun, Aug 27, 2023 at 09:55:23PM +0200, Felix Palmen wrote:

* Dmitry Chagin  [20230827 22:46]:



> I can fix this completely disabling exttatr for jailed proc,
> however, it's gonna be bullshit, though

Would probably be better than nothing. AFAIK, "Linux jails" are used a
lot, probably with userlands from distributions actually using xattr.



It might sense to allow this priv (PRIV_VFS_EXTATTR_SYSTEM) for linux
jails by default? What do think, James?


I think the question is more if we want to allow it in jails (not 
specific to linux jails, as in: if it is ok for linux jails, it should 
be ok for FreeBSD jails too). So the question is what does this protect 
the hosts from, if this is not allowed in jails? Some kind of 
possibility to DoS the host?


Bye,
Alexander.

--
http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF
http://www.FreeBSD.orgnetch...@freebsd.org  : PGP 0x8F31830F9F2772BF

Re: Possible issue with linux xattr support?

[Dmitry Chagin]

On Sun, Aug 27, 2023 at 09:55:23PM +0200, Felix Palmen wrote:
> * Dmitry Chagin  [20230827 22:46]:
> > On Sun, Aug 27, 2023 at 07:59:32PM +0200, Felix Palmen wrote:
> > > * Dmitry Chagin  [20230827 20:54]:
> > > > 1. which fs are you using?
> > > 
> > > ZFS.
> > > 
> > > > 2. jailed?
> > > 
> > > Yes, this is during building ports with poudriere.
> > > 
> > 
> > I think it's a weird prohibition on changing system namespace extattr
> > attributes, look to comments in extattr_check_cred()
> 
> Maybe that's when I should finally start trying to understand the stuff
> in src.git ;)
> 
> > I can fix this completely disabling exttatr for jailed proc,
> > however, it's gonna be bullshit, though
> 
> Would probably be better than nothing. AFAIK, "Linux jails" are used a
> lot, probably with userlands from distributions actually using xattr.
> 

It might sense to allow this priv (PRIV_VFS_EXTATTR_SYSTEM) for linux
jails by default? What do think, James?

Re: etcupdate -p, and other uses of etcupdate(8) (was: Has the update procedure changed?)

[Tomoaki AOKI]

On Sun, 27 Aug 2023 18:16:35 +0100
Graham Perrin  wrote:

> On 07/08/2023 16:51, Kevin Oberman wrote:
> > UPDATING seems to match the Makefile except that Makefile is far less 
> > detailed. The Makefile even says "See src/UPDATING `COMMON ITEMS' for 
> > more complete information."
> >
> > …  "etcupdate -p". …
> 
>  prompted me to 
> look at 
> .
>  
> 
> 
> Is this fortune-provided tip outdated?

For new install with bsdinstall [1] and binary update by
freebsd-update [2], yes. But for src updates, no [3].


[1]
https://lists.freebsd.org/archives/freebsd-current/2023-August/004194.html

[2]
https://lists.freebsd.org/archives/freebsd-current/2023-August/004208.html

[3]
https://lists.freebsd.org/archives/freebsd-current/2023-August/004210.html


-- 
Tomoaki AOKI

Re: Possible issue with linux xattr support?

[Felix Palmen]

* Cy Schubert  [20230827 16:59]:
> On August 27, 2023 12:55:23 PM PDT, Felix Palmen  wrote:
> >* Dmitry Chagin  [20230827 22:46]:
> >> On Sun, Aug 27, 2023 at 07:59:32PM +0200, Felix Palmen wrote:
> >> > * Dmitry Chagin  [20230827 20:54]:
> >> > > 1. which fs are you using?
> >> > 
> >> > ZFS.
> >> > 
> >> > > 2. jailed?
> >> > 
> >> > Yes, this is during building ports with poudriere.
> >> > 
> >> 
> >> I think it's a weird prohibition on changing system namespace extattr
> >> attributes, look to comments in extattr_check_cred()
> >
> >Maybe that's when I should finally start trying to understand the stuff
> >in src.git ;)
> >
> >> I can fix this completely disabling exttatr for jailed proc,
> >> however, it's gonna be bullshit, though
> >
> >Would probably be better than nothing. AFAIK, "Linux jails" are used a
> >lot, probably with userlands from distributions actually using xattr.
> >
> >Cheers, Felix
> >
> 
> If we are to break it to fix a problem, maybe a sysctl to enable/disable then?

IMHO depends on the exact nature of the problem. If it's confirmed that
it (always and only) breaks for jailed processes, just disabling it for
them would be the better workaround. "No-op" calls won't break anything.

Cheers, Felix

-- 
 Felix Palmen  {private}   fe...@palmen-it.de
 -- ports committer -- {web}  http://palmen-it.de
 {pgp public key}  http://palmen-it.de/pub.txt
 {pgp fingerprint} 6936 13D5 5BBF 4837 B212  3ACC 54AD E006 9879 F231


signature.asc
Description: PGP signature