date:20171128

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

2017-11-28 Thread Rick Macklem

Did my usual and forgot to attach it. Here's the patch, rick

From: Rick Macklem 
Sent: Tuesday, November 28, 2017 6:17:13 PM
To: Emmanuel Vadot
Cc: Konstantin Belousov; FreeBSD Current; freebsd...@freebsd.org; Rick Macklem
Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

I think the attached patch should fix your performance problem.
It simply checks for nfsrv_delegatecnt != 0 before doing all the
locking stuff in nfsrv_checkgetattr().

If this fixes your performance problem (with delegations disabled),
I'll probably add a separate counter for write delegations and
use atomics to increment/decrement it so that it is SMP safe without
acquiring any lock.

If you can test this, please let me know how it goes? rick

From: Rick Macklem 
Sent: Tuesday, November 28, 2017 2:09:51 PM
To: Emmanuel Vadot
Cc: Konstantin Belousov; FreeBSD Current; freebsd...@freebsd.org; Rick Macklem
Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

Emmanuel Vadot wrote:
>I wrote:
>> Since it defaults to "disabled", I don't see why a tunable would be 
>> necessary?
>> (Just do nothing and delegations don't happen. If you want the server
>>  to issue delegations, then use the sysctl to turn them on. If you want to 
>> turn
>>  them off again at the server, reboot the server without setting the sysctl.)
>
>If you need to reboot to make things working again without delegation
>this shouldn't be a sysctl.
Turning them off without rebooting doesn't break anything.
I just wrote it that way since you seemed to want to use a tunable, which
implied rebooting was your preference.
> > > >  The reason behind it is recent problem at work on some on our filer
> > > > related to NFS.
> > > >  We use NFSv4 without delegation as we never been able to have good
> > > > performance with FreeBSD server and Linux client (we need to do test
> > > > again but that's for later).
> Delegations are almost never useful, especially with Linux clients.
Emmanuel Vadot wrote:
>Can you elaborate ? Reading what delegation are I see that this is
>exactly what I'm looking for to have better performance with NFS for my
>workload (I only have one client per mount point).
Delegations allow the client to do Opens locally without contacting the
server. Unless, the delegation is a write delegation, this only applied
to read-only delegations. Also, since most implementors couldn`t agree
on how to check permissions via the delegation, most client implementations
still do an Access check at open, which is almost as much overhead as the
Open RPC. (For example, Solaris servers always specified an empty ACE in the
delegation, forcing the client to do an Access. I have no idea what the
current Linux serveréclient does. If you capture packets when a Linux
client is mounted with delegations enabled, you could look for RPCs like Access 
when
are Opened multiple times. If you see them, then delegations aren`t saving RPCs.
Also, they are `per file`, so are only useful if the client is Opening the
same file multiple times.
Further, if another client Opens the same file and the first client got a Write
delegation, then the write delegation must be recalled, which is a lot of
overhead and one of the few cases where the FreeBSD server must exclusively
lock the state lists, forcing all other RPCs related to Open, Close to wait.

They sounded good in theory and might have worked well if the implementors
had agreed to do them, but that didnèt happen. (Companies pay for servers, but 
the
clients donèt get funded so delegation support in the clients are lacking. I 
tried
to make them useful in the FreeBSD client, but Ièm not sure I succeeded.)

> [stuff snipped]
If I understood your original post, you have a performance problem caused
by lock contention, where the server grabs the state lock to check for 
delegations
for every Getattr from a client.

As below, I think the fix is to add code to check for no delegations issued that
does not require acquisition of the state lock.

Btw, large numbers of Getattrs will not perform well with delegations.
(Again, the client should be able to do Getattr operations locally in the
 client when it has a delegation for the file, but if the client is not doing 
that...)

I wrote:
>
> Having a per-mount version of this would be overkill, I think. It would have 
> to
> disable callbacks on the mount point, since there is no way for a client to 
> say
> "don't give me delegations" except disabling callbacks, which the server
> requires for delegation issue.
> [stuff snipped]
> The case where there has never been delegations issued will result in an
> empty delegation queue. Maybe this case can be handled without acquiring
> the "global NFSv4 state lock", which is what sounds like is causing the
> performance issue. Maybe a global counter of how many delegations are
> issued that is handled by atomic ops.
> --> If

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

2017-11-28 Thread Rick Macklem

I think the attached patch should fix your performance problem.
It simply checks for nfsrv_delegatecnt != 0 before doing all the
locking stuff in nfsrv_checkgetattr().

If this fixes your performance problem (with delegations disabled),
I'll probably add a separate counter for write delegations and
use atomics to increment/decrement it so that it is SMP safe without
acquiring any lock.

If you can test this, please let me know how it goes? rick


From: Rick Macklem 
Sent: Tuesday, November 28, 2017 2:09:51 PM
To: Emmanuel Vadot
Cc: Konstantin Belousov; FreeBSD Current; freebsd...@freebsd.org; Rick Macklem
Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

Emmanuel Vadot wrote:
>I wrote:
>> Since it defaults to "disabled", I don't see why a tunable would be 
>> necessary?
>> (Just do nothing and delegations don't happen. If you want the server
>>  to issue delegations, then use the sysctl to turn them on. If you want to 
>> turn
>>  them off again at the server, reboot the server without setting the sysctl.)
>
>If you need to reboot to make things working again without delegation
>this shouldn't be a sysctl.
Turning them off without rebooting doesn't break anything.
I just wrote it that way since you seemed to want to use a tunable, which
implied rebooting was your preference.
> > > >  The reason behind it is recent problem at work on some on our filer
> > > > related to NFS.
> > > >  We use NFSv4 without delegation as we never been able to have good
> > > > performance with FreeBSD server and Linux client (we need to do test
> > > > again but that's for later).
> Delegations are almost never useful, especially with Linux clients.
Emmanuel Vadot wrote:
>Can you elaborate ? Reading what delegation are I see that this is
>exactly what I'm looking for to have better performance with NFS for my
>workload (I only have one client per mount point).
Delegations allow the client to do Opens locally without contacting the
server. Unless, the delegation is a write delegation, this only applied
to read-only delegations. Also, since most implementors couldn`t agree
on how to check permissions via the delegation, most client implementations
still do an Access check at open, which is almost as much overhead as the
Open RPC. (For example, Solaris servers always specified an empty ACE in the
delegation, forcing the client to do an Access. I have no idea what the
current Linux serveréclient does. If you capture packets when a Linux
client is mounted with delegations enabled, you could look for RPCs like Access 
when
are Opened multiple times. If you see them, then delegations aren`t saving RPCs.
Also, they are `per file`, so are only useful if the client is Opening the
same file multiple times.
Further, if another client Opens the same file and the first client got a Write
delegation, then the write delegation must be recalled, which is a lot of
overhead and one of the few cases where the FreeBSD server must exclusively
lock the state lists, forcing all other RPCs related to Open, Close to wait.

They sounded good in theory and might have worked well if the implementors
had agreed to do them, but that didnèt happen. (Companies pay for servers, but 
the
clients donèt get funded so delegation support in the clients are lacking. I 
tried
to make them useful in the FreeBSD client, but Ièm not sure I succeeded.)

> [stuff snipped]
If I understood your original post, you have a performance problem caused
by lock contention, where the server grabs the state lock to check for 
delegations
for every Getattr from a client.

As below, I think the fix is to add code to check for no delegations issued that
does not require acquisition of the state lock.

Btw, large numbers of Getattrs will not perform well with delegations.
(Again, the client should be able to do Getattr operations locally in the
 client when it has a delegation for the file, but if the client is not doing 
that...)

I wrote:
>
> Having a per-mount version of this would be overkill, I think. It would have 
> to
> disable callbacks on the mount point, since there is no way for a client to 
> say
> "don't give me delegations" except disabling callbacks, which the server
> requires for delegation issue.
> [stuff snipped]
> The case where there has never been delegations issued will result in an
> empty delegation queue. Maybe this case can be handled without acquiring
> the "global NFSv4 state lock", which is what sounds like is causing the
> performance issue. Maybe a global counter of how many delegations are
> issued that is handled by atomic ops.
> --> If it is 0, nfsrv_checkgetattr() can just return without acquiring the 
> lock.
>
> I'll take a look at this, rick
>
rick
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

witness_lock_list_get: witness exhausted

2017-11-28 Thread Michael Jung


Hi!

I've recently up'd my processor count on our poudriere box and have 
started noticing the error
"witness_lock_list_get: witness exhausted" on the console.  The kernel 
*DOES NOT* crash but I

thought the report may be useful to someone.

$ uname -a
FreeBSD poudriere 12.0-CURRENT FreeBSD 12.0-CURRENT #1 r325999: Sun Nov 
19 18:41:20 EST 2017

mikej@poudriere:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

The machine is pretty busy running four poudriere build instances.

last pid: 76584;  load averages: 115.07, 115.96, 98.30   
 
 up 6+07:32:59  14:44:03

763 processes: 117 running, 581 sleeping, 2 zombie, 63 lock
CPU: 59.0% user,  0.0% nice, 40.7% system,  0.1% interrupt,  0.1% idle
Mem: 12G Active, 2003M Inact, 44G Wired, 29G Free
ARC: 28G Total, 11G MFU, 16G MRU, 122M Anon, 359M Header, 1184M Other
 25G Compressed, 32G Uncompressed, 1.24:1 Ratio

Let me know what additional information I might supply.


Thanks!
--mikej
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #1 r325999: Sun Nov 19 18:41:20 EST 2017
mikej@poudriere:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM 
5.0.0svn)
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E7- 4850  @ 2.00GHz (1995.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206f2  Family=0x6  Model=0x2f  Stepping=2
  
Features=0x1fa3fbff
  
Features2=0x83b82203
  AMD Features=0x28100800
  AMD Features2=0x1
  Structured Extended Features=0x2
  TSC: P-state invariant
Hypervisor: Origin = "VMwareVMware"
real memory  = 96636764160 (92160 MB)
avail memory = 93873786880 (89525 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: 
FreeBSD/SMP: Multiprocessor System Detected: 60 CPUs
FreeBSD/SMP: 6 package(s) x 10 core(s)
random: unblocking device.
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0  irqs 0-23 on motherboard
SMP: AP CPU #46 Launched!
SMP: AP CPU #16 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #51 Launched!
SMP: AP CPU #20 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #48 Launched!
SMP: AP CPU #23 Launched!
SMP: AP CPU #27 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #55 Launched!
SMP: AP CPU #17 Launched!
SMP: AP CPU #52 Launched!
SMP: AP CPU #18 Launched!
SMP: AP CPU #39 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #41 Launched!
SMP: AP CPU #58 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #26 Launched!
SMP: AP CPU #37 Launched!
SMP: AP CPU #45 Launched!
SMP: AP CPU #40 Launched!
SMP: AP CPU #35 Launched!
SMP: AP CPU #53 Launched!
SMP: AP CPU #50 Launched!
SMP: AP CPU #44 Launched!
SMP: AP CPU #34 Launched!
SMP: AP CPU #30 Launched!
SMP: AP CPU #31 Launched!
SMP: AP CPU #47 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #28 Launched!
SMP: AP CPU #49 Launched!
SMP: AP CPU #54 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #57 Launched!
SMP: AP CPU #29 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #24 Launched!
SMP: AP CPU #33 Launched!
SMP: AP CPU #38 Launched!
SMP: AP CPU #59 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #42 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #43 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #19 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #25 Launched!
SMP: AP CPU #56 Launched!
SMP: AP CPU #36 Launched!
SMP: AP CPU #22 Launched!
SMP: AP CPU #32 Launched!
SMP: AP CPU #21 Launched!
random: entropy device external interface
netmap: loaded module
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0x80f95c20, 0) error 19
kbd1 at kbdmux0
nexus0
vtvga0:  on motherboard
cryptosoft0:  on motherboard
acpi0:  on motherboard
acpi0: Power Button (fixed)
hpet0:  iomem 0xfed0-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
cpu0:  numa-domain 0 on acpi0
cpu1:  numa-domain 5 on acpi0
cpu2:  numa-domain 5 on acpi0
cpu3:  numa-domain 5 on acpi0
cpu4:  numa-domain 5 on acpi0
cpu5:  numa-domain 5 on acpi0
cpu6:  numa-domain 5 on acpi0
cpu7:  numa-domain 5 on acpi0
cpu8:  numa-domain 5 on acpi0
cpu9:  numa-domain 5 on acpi0
cpu10:  numa-domain 5 on acpi0
cpu11:  numa-domain 4 on acpi0
cpu12:  numa-domain 4 on acpi0
cpu13:  numa-domain 4 on acpi0
cpu14:  numa-domain 4 on acpi0
cpu15:  numa-domain 4 on acpi0
cpu16:  numa-domain 4 on acpi0
cpu17:  numa-domain 4 on acpi0
cpu18:  numa-domain 4 on acpi0
cpu19:  numa-domain 4 on acpi0

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

2017-11-28 Thread Rick Macklem

Emmanuel Vadot wrote:
>I wrote:
>> Since it defaults to "disabled", I don't see why a tunable would be 
>> necessary?
>> (Just do nothing and delegations don't happen. If you want the server
>>  to issue delegations, then use the sysctl to turn them on. If you want to 
>> turn
>>  them off again at the server, reboot the server without setting the sysctl.)
>
>If you need to reboot to make things working again without delegation
>this shouldn't be a sysctl.
Turning them off without rebooting doesn't break anything.
I just wrote it that way since you seemed to want to use a tunable, which
implied rebooting was your preference.
> > > >  The reason behind it is recent problem at work on some on our filer
> > > > related to NFS.
> > > >  We use NFSv4 without delegation as we never been able to have good
> > > > performance with FreeBSD server and Linux client (we need to do test
> > > > again but that's for later).
> Delegations are almost never useful, especially with Linux clients.
Emmanuel Vadot wrote:
>Can you elaborate ? Reading what delegation are I see that this is
>exactly what I'm looking for to have better performance with NFS for my
>workload (I only have one client per mount point).
Delegations allow the client to do Opens locally without contacting the
server. Unless, the delegation is a write delegation, this only applied
to read-only delegations. Also, since most implementors couldn`t agree
on how to check permissions via the delegation, most client implementations
still do an Access check at open, which is almost as much overhead as the
Open RPC. (For example, Solaris servers always specified an empty ACE in the
delegation, forcing the client to do an Access. I have no idea what the
current Linux serveréclient does. If you capture packets when a Linux
client is mounted with delegations enabled, you could look for RPCs like Access 
when
are Opened multiple times. If you see them, then delegations aren`t saving RPCs.
Also, they are `per file`, so are only useful if the client is Opening the
same file multiple times.
Further, if another client Opens the same file and the first client got a Write
delegation, then the write delegation must be recalled, which is a lot of
overhead and one of the few cases where the FreeBSD server must exclusively
lock the state lists, forcing all other RPCs related to Open, Close to wait.

They sounded good in theory and might have worked well if the implementors
had agreed to do them, but that didnèt happen. (Companies pay for servers, but 
the
clients donèt get funded so delegation support in the clients are lacking. I 
tried
to make them useful in the FreeBSD client, but Ièm not sure I succeeded.)

> [stuff snipped]
If I understood your original post, you have a performance problem caused
by lock contention, where the server grabs the state lock to check for 
delegations
for every Getattr from a client.

As below, I think the fix is to add code to check for no delegations issued that
does not require acquisition of the state lock.

Btw, large numbers of Getattrs will not perform well with delegations.
(Again, the client should be able to do Getattr operations locally in the
 client when it has a delegation for the file, but if the client is not doing 
that...)

I wrote:
>
> Having a per-mount version of this would be overkill, I think. It would have 
> to
> disable callbacks on the mount point, since there is no way for a client to 
> say
> "don't give me delegations" except disabling callbacks, which the server
> requires for delegation issue.
> [stuff snipped]
> The case where there has never been delegations issued will result in an
> empty delegation queue. Maybe this case can be handled without acquiring
> the "global NFSv4 state lock", which is what sounds like is causing the
> performance issue. Maybe a global counter of how many delegations are
> issued that is handled by atomic ops.
> --> If it is 0, nfsrv_checkgetattr() can just return without acquiring the 
> lock.
>
> I'll take a look at this, rick
> 
rick
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CURRENT: core dumped no process name

2017-11-28 Thread Konstantin Belousov

On Tue, Nov 28, 2017 at 08:37:44AM -0800, Gleb Smirnoff wrote:
>   Hi Alex,
> 
> On Tue, Nov 28, 2017 at 06:09:16PM +0700, Alex V. Petrov wrote:
> A> You were right.
> A> After revert to r325188 worked:
> A> Nov 28 18:05:18 alex kernel: pid 23618 (mc), uid 0: exited on signal 6
> A> (core dumped)
> 
> Can you please send over to me the syslogd binary and syslogd.core?

Syslogd does not crash, it removes part of the kernel message which it
perceives as the host name.  Read the whole thread.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CURRENT: core dumped no process name

2017-11-28 Thread Gleb Smirnoff

  Hi Alex,

On Tue, Nov 28, 2017 at 06:09:16PM +0700, Alex V. Petrov wrote:
A> You were right.
A> After revert to r325188 worked:
A> Nov 28 18:05:18 alex kernel: pid 23618 (mc), uid 0: exited on signal 6
A> (core dumped)

Can you please send over to me the syslogd binary and syslogd.core?


-- 
Gleb Smirnoff
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

2017-11-28 Thread Emmanuel Vadot

On Tue, 28 Nov 2017 14:12:38 +
Rick Macklem  wrote:

> Konstantin Belousov wrote:
> >On Tue, Nov 28, 2017 at 02:26:10PM +0100, Emmanuel Vadot wrote:
> >> On Tue, 28 Nov 2017 13:04:28 +0200
> >> Konstantin Belousov  wrote:
> >>
> >> > On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote:
> >> > >
> >> > >  Hello,
> >> > >
> >> > >  I would like to switch the vfs.nfsd.issue_delegations sysctl to a
> >> > > tunable.
> Since it defaults to "disabled", I don't see why a tunable would be necessary?
> (Just do nothing and delegations don't happen. If you want the server
>  to issue delegations, then use the sysctl to turn them on. If you want to 
> turn
>  them off again at the server, reboot the server without setting the sysctl.)

 If you need to reboot to make things working again without delegation
this shouldn't be a sysctl.

> > > >  The reason behind it is recent problem at work on some on our filer
> > > > related to NFS.
> > > >  We use NFSv4 without delegation as we never been able to have good
> > > > performance with FreeBSD server and Linux client (we need to do test
> > > > again but that's for later).
> Delegations are almost never useful, especially with Linux clients.

 Can you elaborate ? Reading what delegation are I see that this is
exactly what I'm looking for to have better performance with NFS for my
workload (I only have one client per mount point).

> [stuff snipped]
> > > Perhaps make nodeleg per-mount flag ?
> The sysctl vfs.nfsd.issue_delegations only affects the server.
> If you have a FreeBSD client that is mounting a delegations enabled server and
> does not want delegations, just don't run the "nfscbd" daemon on the client.
> The only time you want the "nfscbd" daemon running is for delegations and
> pNFS layouts. (I suppose there is the case of a client using NFSv4.1/pNFS 
> against
> a server with delegations enabled, but since delegations aren't very useful,
> I'd just disable delegations on the server for this case.)
> 
> Having a per-mount version of this would be overkill, I think. It would have 
> to
> disable callbacks on the mount point, since there is no way for a client to 
> say
> "don't give me delegations" except disabling callbacks, which the server
> requires for delegation issue.
> [stuff snipped]
> The case where there has never been delegations issued will result in an
> empty delegation queue. Maybe this case can be handled without acquiring
> the "global NFSv4 state lock", which is what sounds like is causing the
> performance issue. Maybe a global counter of how many delegations are
> issued that is handled by atomic ops.
> --> If it is 0, nfsrv_checkgetattr() can just return without acquiring the 
> lock.
> 
> I'll take a look at this, rick
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


-- 
Emmanuel Vadot  
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

2017-11-28 Thread Emmanuel Vadot

On Tue, 28 Nov 2017 15:40:09 +0200
Konstantin Belousov  wrote:

> On Tue, Nov 28, 2017 at 02:26:10PM +0100, Emmanuel Vadot wrote:
> > On Tue, 28 Nov 2017 13:04:28 +0200
> > Konstantin Belousov  wrote:
> > 
> > > On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote:
> > > > 
> > > >  Hello,
> > > > 
> > > >  I would like to switch the vfs.nfsd.issue_delegations sysctl to a
> > > > tunable.
> > > >  The reason behind it is recent problem at work on some on our filer
> > > > related to NFS.
> > > >  We use NFSv4 without delegation as we never been able to have good
> > > > performance with FreeBSD server and Linux client (we need to do test
> > > > again but that's for later). We recently see all of our filers with NFS
> > > > enabled perform pourly, resulting in really bad performance on the
> > > > service.
> > > >  After doing some analyze with pmcstat we've seen that we spend ~50%
> > > > of our time in lock delay, called by nfsrv_checkgetattr (See [1]).
> > > >  This function is only usefull when using delegation, as it search for
> > > > some write delegations issued to client, but it's called anyway when
> > > > there so no delegation and cause a lot of problem when having a lot of
> > > > activities.
> > > >  We've patched the kernel with the included diff and now everything is
> > > > fine (tm) (See [2]).
> > > >  The problem for upstreaming this patch is that since issue_delegations
> > > > is a sysctl we cannot know if the checkgetattr called is needed, hence
> > > > the question to switch it to a TUNABLE. Also maybe some other code path
> > > > could benefit from it, I haven't read all the source of nfsserver yet.
> > > > 
> > > >  Note that I won't MFC the changes as it would break POLA.
> > > Perhaps make nodeleg per-mount flag ?
> > > The you can check it safely by dereferencing vp->v_mount->mnt_data without
> > > acquiring any locks, while the vnode lock is owned and the vnode is not
> > > reclaimed.
> > 
> >  That won't work with current code.
> Why ?
> 
> >  Currently if you have delegation enabled and connect one client to a
> > mountpoint, then disable delegation, the current client will still have
> > delegation while new ones will not. I have not tested restarting nfsd in
> > this situation but I suspect that client will behave badly. This is a
> > another +1 for making it a tunable I think.
> It is up to the filesystem to handle remount, in particular, fs can disable
> changing mount options on mount upgrade if the operation is not supported.
> In other words, you would do
>   mount -o nodeleg ... /mnt
> and
>   mount -u -o nonodeleg ... /mnt
> needs to return EINVAL.

 You are talking about client here while I'm talking about server.

> > 
> > > > 
> > > >  Cheers,
> > > > 
> > > > [1] https://people.freebsd.org/~manu/m8.svg
> > > > [2] https://people.freebsd.org/~manu/m8-new.svg
> > > > 
> > > > >From 0cba277f406d3ccf3c9e943a3d4e17b529e31c89 Mon Sep 17 00:00:00 2001
> > > > From: Emmanuel Vadot 
> > > > Date: Fri, 24 Nov 2017 11:17:18 +0100
> > > > Subject: [PATCH 2/4] Do not call nfsrv_checkgetttr if delegation isn't
> > > > enable.
> > > > 
> > > > Signed-off-by: Emmanuel Vadot 
> > > > ---
> > > >  sys/fs/nfsserver/nfs_nfsdserv.c | 3 ++-
> > > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c
> > > > b/sys/fs/nfsserver/nfs_nfsdserv.c index 8102c5810a9..8daf0142360 100644
> > > > --- a/sys/fs/nfsserver/nfs_nfsdserv.c
> > > > +++ b/sys/fs/nfsserver/nfs_nfsdserv.c
> > > > @@ -54,6 +54,7 @@ extern struct timeval nfsboottime;
> > > >  extern int nfs_rootfhset;
> > > >  extern int nfsrv_enable_crossmntpt;
> > > >  extern int nfsrv_statehashsize;
> > > > +extern int nfsrv_issuedelegs;
> > > >  #endif /* !APPLEKEXT */
> > > >  
> > > >  static int nfs_async = 0;
> > > > @@ -240,7 +241,7 @@ nfsrvd_getattr(struct nfsrv_descript *nd, int
> > > > isdgram, if (nd->nd_flag & ND_NFSV4) {
> > > > if (NFSISSET_ATTRBIT(,
> > > > NFSATTRBIT_FILEHANDLE)) nd->nd_repstat = nfsvno_getfh(vp, , p);
> > > > -   if (!nd->nd_repstat)
> > > > +   if (nd->nd_repstat == 0 && nfsrv_issuedelegs
> > > > == 1) nd->nd_repstat = nfsrv_checkgetattr(nd, vp,
> > > > , , nd->nd_cred, p);
> > > > if (nd->nd_repstat == 0) {
> > > > -- 
> > > > 2.14.2
> > > > 
> > > > 
> > > > -- 
> > > > Emmanuel Vadot  
> > > > ___
> > > > freebsd...@freebsd.org mailing list
> > > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > > > To unsubscribe, send any mail to "freebsd-fs-unsubscr...@freebsd.org"
> > > ___
> > > freebsd-current@freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-current

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

2017-11-28 Thread Rick Macklem

Konstantin Belousov wrote:
>On Tue, Nov 28, 2017 at 02:26:10PM +0100, Emmanuel Vadot wrote:
>> On Tue, 28 Nov 2017 13:04:28 +0200
>> Konstantin Belousov  wrote:
>>
>> > On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote:
>> > >
>> > >  Hello,
>> > >
>> > >  I would like to switch the vfs.nfsd.issue_delegations sysctl to a
>> > > tunable.
Since it defaults to "disabled", I don't see why a tunable would be necessary?
(Just do nothing and delegations don't happen. If you want the server
 to issue delegations, then use the sysctl to turn them on. If you want to turn
 them off again at the server, reboot the server without setting the sysctl.)

> > >  The reason behind it is recent problem at work on some on our filer
> > > related to NFS.
> > >  We use NFSv4 without delegation as we never been able to have good
> > > performance with FreeBSD server and Linux client (we need to do test
> > > again but that's for later).
Delegations are almost never useful, especially with Linux clients.
[stuff snipped]
> > Perhaps make nodeleg per-mount flag ?
The sysctl vfs.nfsd.issue_delegations only affects the server.
If you have a FreeBSD client that is mounting a delegations enabled server and
does not want delegations, just don't run the "nfscbd" daemon on the client.
The only time you want the "nfscbd" daemon running is for delegations and
pNFS layouts. (I suppose there is the case of a client using NFSv4.1/pNFS 
against
a server with delegations enabled, but since delegations aren't very useful,
I'd just disable delegations on the server for this case.)

Having a per-mount version of this would be overkill, I think. It would have to
disable callbacks on the mount point, since there is no way for a client to say
"don't give me delegations" except disabling callbacks, which the server
requires for delegation issue.
[stuff snipped]
The case where there has never been delegations issued will result in an
empty delegation queue. Maybe this case can be handled without acquiring
the "global NFSv4 state lock", which is what sounds like is causing the
performance issue. Maybe a global counter of how many delegations are
issued that is handled by atomic ops.
--> If it is 0, nfsrv_checkgetattr() can just return without acquiring the lock.

I'll take a look at this, rick
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

2017-11-28 Thread Konstantin Belousov

On Tue, Nov 28, 2017 at 02:26:10PM +0100, Emmanuel Vadot wrote:
> On Tue, 28 Nov 2017 13:04:28 +0200
> Konstantin Belousov  wrote:
> 
> > On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote:
> > > 
> > >  Hello,
> > > 
> > >  I would like to switch the vfs.nfsd.issue_delegations sysctl to a
> > > tunable.
> > >  The reason behind it is recent problem at work on some on our filer
> > > related to NFS.
> > >  We use NFSv4 without delegation as we never been able to have good
> > > performance with FreeBSD server and Linux client (we need to do test
> > > again but that's for later). We recently see all of our filers with NFS
> > > enabled perform pourly, resulting in really bad performance on the
> > > service.
> > >  After doing some analyze with pmcstat we've seen that we spend ~50%
> > > of our time in lock delay, called by nfsrv_checkgetattr (See [1]).
> > >  This function is only usefull when using delegation, as it search for
> > > some write delegations issued to client, but it's called anyway when
> > > there so no delegation and cause a lot of problem when having a lot of
> > > activities.
> > >  We've patched the kernel with the included diff and now everything is
> > > fine (tm) (See [2]).
> > >  The problem for upstreaming this patch is that since issue_delegations
> > > is a sysctl we cannot know if the checkgetattr called is needed, hence
> > > the question to switch it to a TUNABLE. Also maybe some other code path
> > > could benefit from it, I haven't read all the source of nfsserver yet.
> > > 
> > >  Note that I won't MFC the changes as it would break POLA.
> > Perhaps make nodeleg per-mount flag ?
> > The you can check it safely by dereferencing vp->v_mount->mnt_data without
> > acquiring any locks, while the vnode lock is owned and the vnode is not
> > reclaimed.
> 
>  That won't work with current code.
Why ?

>  Currently if you have delegation enabled and connect one client to a
> mountpoint, then disable delegation, the current client will still have
> delegation while new ones will not. I have not tested restarting nfsd in
> this situation but I suspect that client will behave badly. This is a
> another +1 for making it a tunable I think.
It is up to the filesystem to handle remount, in particular, fs can disable
changing mount options on mount upgrade if the operation is not supported.
In other words, you would do
mount -o nodeleg ... /mnt
and
mount -u -o nonodeleg ... /mnt
needs to return EINVAL.

> 
> > > 
> > >  Cheers,
> > > 
> > > [1] https://people.freebsd.org/~manu/m8.svg
> > > [2] https://people.freebsd.org/~manu/m8-new.svg
> > > 
> > > >From 0cba277f406d3ccf3c9e943a3d4e17b529e31c89 Mon Sep 17 00:00:00 2001
> > > From: Emmanuel Vadot 
> > > Date: Fri, 24 Nov 2017 11:17:18 +0100
> > > Subject: [PATCH 2/4] Do not call nfsrv_checkgetttr if delegation isn't
> > > enable.
> > > 
> > > Signed-off-by: Emmanuel Vadot 
> > > ---
> > >  sys/fs/nfsserver/nfs_nfsdserv.c | 3 ++-
> > >  1 file changed, 2 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c
> > > b/sys/fs/nfsserver/nfs_nfsdserv.c index 8102c5810a9..8daf0142360 100644
> > > --- a/sys/fs/nfsserver/nfs_nfsdserv.c
> > > +++ b/sys/fs/nfsserver/nfs_nfsdserv.c
> > > @@ -54,6 +54,7 @@ extern struct timeval nfsboottime;
> > >  extern int nfs_rootfhset;
> > >  extern int nfsrv_enable_crossmntpt;
> > >  extern int nfsrv_statehashsize;
> > > +extern int nfsrv_issuedelegs;
> > >  #endif   /* !APPLEKEXT */
> > >  
> > >  static int   nfs_async = 0;
> > > @@ -240,7 +241,7 @@ nfsrvd_getattr(struct nfsrv_descript *nd, int
> > > isdgram, if (nd->nd_flag & ND_NFSV4) {
> > >   if (NFSISSET_ATTRBIT(,
> > > NFSATTRBIT_FILEHANDLE)) nd->nd_repstat = nfsvno_getfh(vp, , p);
> > > - if (!nd->nd_repstat)
> > > + if (nd->nd_repstat == 0 && nfsrv_issuedelegs
> > > == 1) nd->nd_repstat = nfsrv_checkgetattr(nd, vp,
> > >   , , nd->nd_cred, p);
> > >   if (nd->nd_repstat == 0) {
> > > -- 
> > > 2.14.2
> > > 
> > > 
> > > -- 
> > > Emmanuel Vadot  
> > > ___
> > > freebsd...@freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > > To unsubscribe, send any mail to "freebsd-fs-unsubscr...@freebsd.org"
> > ___
> > freebsd-current@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
> 
> 
> -- 
> Emmanuel Vadot  
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

2017-11-28 Thread Emmanuel Vadot

On Tue, 28 Nov 2017 13:04:28 +0200
Konstantin Belousov  wrote:

> On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote:
> > 
> >  Hello,
> > 
> >  I would like to switch the vfs.nfsd.issue_delegations sysctl to a
> > tunable.
> >  The reason behind it is recent problem at work on some on our filer
> > related to NFS.
> >  We use NFSv4 without delegation as we never been able to have good
> > performance with FreeBSD server and Linux client (we need to do test
> > again but that's for later). We recently see all of our filers with NFS
> > enabled perform pourly, resulting in really bad performance on the
> > service.
> >  After doing some analyze with pmcstat we've seen that we spend ~50%
> > of our time in lock delay, called by nfsrv_checkgetattr (See [1]).
> >  This function is only usefull when using delegation, as it search for
> > some write delegations issued to client, but it's called anyway when
> > there so no delegation and cause a lot of problem when having a lot of
> > activities.
> >  We've patched the kernel with the included diff and now everything is
> > fine (tm) (See [2]).
> >  The problem for upstreaming this patch is that since issue_delegations
> > is a sysctl we cannot know if the checkgetattr called is needed, hence
> > the question to switch it to a TUNABLE. Also maybe some other code path
> > could benefit from it, I haven't read all the source of nfsserver yet.
> > 
> >  Note that I won't MFC the changes as it would break POLA.
> Perhaps make nodeleg per-mount flag ?
> The you can check it safely by dereferencing vp->v_mount->mnt_data without
> acquiring any locks, while the vnode lock is owned and the vnode is not
> reclaimed.

 That won't work with current code.
 Currently if you have delegation enabled and connect one client to a
mountpoint, then disable delegation, the current client will still have
delegation while new ones will not. I have not tested restarting nfsd in
this situation but I suspect that client will behave badly. This is a
another +1 for making it a tunable I think.

> > 
> >  Cheers,
> > 
> > [1] https://people.freebsd.org/~manu/m8.svg
> > [2] https://people.freebsd.org/~manu/m8-new.svg
> > 
> > >From 0cba277f406d3ccf3c9e943a3d4e17b529e31c89 Mon Sep 17 00:00:00 2001
> > From: Emmanuel Vadot 
> > Date: Fri, 24 Nov 2017 11:17:18 +0100
> > Subject: [PATCH 2/4] Do not call nfsrv_checkgetttr if delegation isn't
> > enable.
> > 
> > Signed-off-by: Emmanuel Vadot 
> > ---
> >  sys/fs/nfsserver/nfs_nfsdserv.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c
> > b/sys/fs/nfsserver/nfs_nfsdserv.c index 8102c5810a9..8daf0142360 100644
> > --- a/sys/fs/nfsserver/nfs_nfsdserv.c
> > +++ b/sys/fs/nfsserver/nfs_nfsdserv.c
> > @@ -54,6 +54,7 @@ extern struct timeval nfsboottime;
> >  extern int nfs_rootfhset;
> >  extern int nfsrv_enable_crossmntpt;
> >  extern int nfsrv_statehashsize;
> > +extern int nfsrv_issuedelegs;
> >  #endif /* !APPLEKEXT */
> >  
> >  static int nfs_async = 0;
> > @@ -240,7 +241,7 @@ nfsrvd_getattr(struct nfsrv_descript *nd, int
> > isdgram, if (nd->nd_flag & ND_NFSV4) {
> > if (NFSISSET_ATTRBIT(,
> > NFSATTRBIT_FILEHANDLE)) nd->nd_repstat = nfsvno_getfh(vp, , p);
> > -   if (!nd->nd_repstat)
> > +   if (nd->nd_repstat == 0 && nfsrv_issuedelegs
> > == 1) nd->nd_repstat = nfsrv_checkgetattr(nd, vp,
> > , , nd->nd_cred, p);
> > if (nd->nd_repstat == 0) {
> > -- 
> > 2.14.2
> > 
> > 
> > -- 
> > Emmanuel Vadot  
> > ___
> > freebsd...@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > To unsubscribe, send any mail to "freebsd-fs-unsubscr...@freebsd.org"
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


-- 
Emmanuel Vadot  
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CURRENT: core dumped no process name

2017-11-28 Thread Konstantin Belousov

On Tue, Nov 28, 2017 at 06:09:16PM +0700, Alex V. Petrov wrote:
> You were right.
> After revert to r325188 worked:
> Nov 28 18:05:18 alex kernel: pid 23618 (mc), uid 0: exited on signal 6
> (core dumped)
Thank you for confirming.

Adding the committer.

> 
> 28.11.2017 17:38, Konstantin Belousov пишет:
> > On Tue, Nov 28, 2017 at 05:13:10PM +0700, Alex V. Petrov wrote:
> >> from /var/log/messages
> >>
> >> # grep core /var/log/messages
> >> Nov 21 06:26:56 alex kernel: 1001: exited on signal 11 (core dumped)
> >> Nov 22 00:46:31 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> >> Nov 22 16:01:32 alex kernel: 1001: exited on signal 11 (core dumped)
> >> Nov 23 08:14:45 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> >> Nov 24 04:32:17 alex kernel: 1001: exited on signal 11 (core dumped)
> >> Nov 24 04:41:44 alex kernel: 1001: exited on signal 11 (core dumped)
> >> Nov 24 04:48:03 alex kernel: 1001: exited on signal 11 (core dumped)
> >> Nov 24 04:49:55 alex kernel: 1001: exited on signal 11 (core dumped)
> >> Nov 24 11:33:45 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> >> Nov 26 02:50:35 alex kernel: 1001: exited on signal 11 (core dumped)
> >> Nov 26 03:47:56 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> >> Nov 26 12:05:01 alex kernel: 1001: exited on signal 11 (core dumped)
> >> Nov 26 14:16:26 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> >> Nov 26 14:27:05 alex kernel: 0: exited on signal 6 (core dumped)
> >> Nov 26 23:49:56 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> >> Nov 26 23:50:26 alex kernel: 0: exited on signal 6 (core dumped)
> >> Nov 27 12:13:02 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> >> Nov 27 15:28:11 alex kernel: 0: exited on signal 6 (core dumped)
> >> Nov 27 15:50:54 alex kernel: 0: exited on signal 6 (core dumped)
> >> Nov 27 16:34:02 alex kernel: 0: exited on signal 6 (core dumped)
> >> Nov 28 03:07:42 alex kernel: 0: exited on signal 6 (core dumped)
> >> Nov 28 04:20:03 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> >> Nov 28 04:51:08 alex kernel: 0: exited on signal 6 (core dumped)
> >> Nov 28 10:28:38 alex kernel: 1001: exited on signal 11 (core dumped)
> >> Nov 28 15:55:42 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> >> Nov 28 17:05:35 alex kernel: 0: exited on signal 6 (core dumped)
> >>
> > Do you use syslogd after r325558 ?  If yes, try to revert it.
> > 
> > 
> 
> -- 
> -
> Alex.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CURRENT: core dumped no process name

2017-11-28 Thread Alex V. Petrov

You were right.
After revert to r325188 worked:
Nov 28 18:05:18 alex kernel: pid 23618 (mc), uid 0: exited on signal 6
(core dumped)

28.11.2017 17:38, Konstantin Belousov пишет:
> On Tue, Nov 28, 2017 at 05:13:10PM +0700, Alex V. Petrov wrote:
>> from /var/log/messages
>>
>> # grep core /var/log/messages
>> Nov 21 06:26:56 alex kernel: 1001: exited on signal 11 (core dumped)
>> Nov 22 00:46:31 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
>> Nov 22 16:01:32 alex kernel: 1001: exited on signal 11 (core dumped)
>> Nov 23 08:14:45 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
>> Nov 24 04:32:17 alex kernel: 1001: exited on signal 11 (core dumped)
>> Nov 24 04:41:44 alex kernel: 1001: exited on signal 11 (core dumped)
>> Nov 24 04:48:03 alex kernel: 1001: exited on signal 11 (core dumped)
>> Nov 24 04:49:55 alex kernel: 1001: exited on signal 11 (core dumped)
>> Nov 24 11:33:45 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
>> Nov 26 02:50:35 alex kernel: 1001: exited on signal 11 (core dumped)
>> Nov 26 03:47:56 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
>> Nov 26 12:05:01 alex kernel: 1001: exited on signal 11 (core dumped)
>> Nov 26 14:16:26 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
>> Nov 26 14:27:05 alex kernel: 0: exited on signal 6 (core dumped)
>> Nov 26 23:49:56 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
>> Nov 26 23:50:26 alex kernel: 0: exited on signal 6 (core dumped)
>> Nov 27 12:13:02 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
>> Nov 27 15:28:11 alex kernel: 0: exited on signal 6 (core dumped)
>> Nov 27 15:50:54 alex kernel: 0: exited on signal 6 (core dumped)
>> Nov 27 16:34:02 alex kernel: 0: exited on signal 6 (core dumped)
>> Nov 28 03:07:42 alex kernel: 0: exited on signal 6 (core dumped)
>> Nov 28 04:20:03 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
>> Nov 28 04:51:08 alex kernel: 0: exited on signal 6 (core dumped)
>> Nov 28 10:28:38 alex kernel: 1001: exited on signal 11 (core dumped)
>> Nov 28 15:55:42 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
>> Nov 28 17:05:35 alex kernel: 0: exited on signal 6 (core dumped)
>>
> Do you use syslogd after r325558 ?  If yes, try to revert it.
> 
> 

-- 
-
Alex.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

2017-11-28 Thread Konstantin Belousov

On Tue, Nov 28, 2017 at 11:41:36AM +0100, Emmanuel Vadot wrote:
> 
>  Hello,
> 
>  I would like to switch the vfs.nfsd.issue_delegations sysctl to a
> tunable.
>  The reason behind it is recent problem at work on some on our filer
> related to NFS.
>  We use NFSv4 without delegation as we never been able to have good
> performance with FreeBSD server and Linux client (we need to do test
> again but that's for later). We recently see all of our filers with NFS
> enabled perform pourly, resulting in really bad performance on the
> service.
>  After doing some analyze with pmcstat we've seen that we spend ~50%
> of our time in lock delay, called by nfsrv_checkgetattr (See [1]).
>  This function is only usefull when using delegation, as it search for
> some write delegations issued to client, but it's called anyway when
> there so no delegation and cause a lot of problem when having a lot of
> activities.
>  We've patched the kernel with the included diff and now everything is
> fine (tm) (See [2]).
>  The problem for upstreaming this patch is that since issue_delegations
> is a sysctl we cannot know if the checkgetattr called is needed, hence
> the question to switch it to a TUNABLE. Also maybe some other code path
> could benefit from it, I haven't read all the source of nfsserver yet.
> 
>  Note that I won't MFC the changes as it would break POLA.
Perhaps make nodeleg per-mount flag ?
The you can check it safely by dereferencing vp->v_mount->mnt_data without
acquiring any locks, while the vnode lock is owned and the vnode is not
reclaimed.

> 
>  Cheers,
> 
> [1] https://people.freebsd.org/~manu/m8.svg
> [2] https://people.freebsd.org/~manu/m8-new.svg
> 
> >From 0cba277f406d3ccf3c9e943a3d4e17b529e31c89 Mon Sep 17 00:00:00 2001
> From: Emmanuel Vadot 
> Date: Fri, 24 Nov 2017 11:17:18 +0100
> Subject: [PATCH 2/4] Do not call nfsrv_checkgetttr if delegation isn't
> enable.
> 
> Signed-off-by: Emmanuel Vadot 
> ---
>  sys/fs/nfsserver/nfs_nfsdserv.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c
> b/sys/fs/nfsserver/nfs_nfsdserv.c index 8102c5810a9..8daf0142360 100644
> --- a/sys/fs/nfsserver/nfs_nfsdserv.c
> +++ b/sys/fs/nfsserver/nfs_nfsdserv.c
> @@ -54,6 +54,7 @@ extern struct timeval nfsboottime;
>  extern int nfs_rootfhset;
>  extern int nfsrv_enable_crossmntpt;
>  extern int nfsrv_statehashsize;
> +extern int nfsrv_issuedelegs;
>  #endif   /* !APPLEKEXT */
>  
>  static int   nfs_async = 0;
> @@ -240,7 +241,7 @@ nfsrvd_getattr(struct nfsrv_descript *nd, int
> isdgram, if (nd->nd_flag & ND_NFSV4) {
>   if (NFSISSET_ATTRBIT(,
> NFSATTRBIT_FILEHANDLE)) nd->nd_repstat = nfsvno_getfh(vp, , p);
> - if (!nd->nd_repstat)
> + if (nd->nd_repstat == 0 && nfsrv_issuedelegs
> == 1) nd->nd_repstat = nfsrv_checkgetattr(nd, vp,
>   , , nd->nd_cred, p);
>   if (nd->nd_repstat == 0) {
> -- 
> 2.14.2
> 
> 
> -- 
> Emmanuel Vadot  
> ___
> freebsd...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Switch vfs.nfsd.issue_delegations to TUNABLE ?

2017-11-28 Thread Emmanuel Vadot


 Hello,

 I would like to switch the vfs.nfsd.issue_delegations sysctl to a
tunable.
 The reason behind it is recent problem at work on some on our filer
related to NFS.
 We use NFSv4 without delegation as we never been able to have good
performance with FreeBSD server and Linux client (we need to do test
again but that's for later). We recently see all of our filers with NFS
enabled perform pourly, resulting in really bad performance on the
service.
 After doing some analyze with pmcstat we've seen that we spend ~50%
of our time in lock delay, called by nfsrv_checkgetattr (See [1]).
 This function is only usefull when using delegation, as it search for
some write delegations issued to client, but it's called anyway when
there so no delegation and cause a lot of problem when having a lot of
activities.
 We've patched the kernel with the included diff and now everything is
fine (tm) (See [2]).
 The problem for upstreaming this patch is that since issue_delegations
is a sysctl we cannot know if the checkgetattr called is needed, hence
the question to switch it to a TUNABLE. Also maybe some other code path
could benefit from it, I haven't read all the source of nfsserver yet.

 Note that I won't MFC the changes as it would break POLA.

 Cheers,

[1] https://people.freebsd.org/~manu/m8.svg
[2] https://people.freebsd.org/~manu/m8-new.svg

>From 0cba277f406d3ccf3c9e943a3d4e17b529e31c89 Mon Sep 17 00:00:00 2001
From: Emmanuel Vadot 
Date: Fri, 24 Nov 2017 11:17:18 +0100
Subject: [PATCH 2/4] Do not call nfsrv_checkgetttr if delegation isn't
enable.

Signed-off-by: Emmanuel Vadot 
---
 sys/fs/nfsserver/nfs_nfsdserv.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c
b/sys/fs/nfsserver/nfs_nfsdserv.c index 8102c5810a9..8daf0142360 100644
--- a/sys/fs/nfsserver/nfs_nfsdserv.c
+++ b/sys/fs/nfsserver/nfs_nfsdserv.c
@@ -54,6 +54,7 @@ extern struct timeval nfsboottime;
 extern int nfs_rootfhset;
 extern int nfsrv_enable_crossmntpt;
 extern int nfsrv_statehashsize;
+extern int nfsrv_issuedelegs;
 #endif /* !APPLEKEXT */
 
 static int nfs_async = 0;
@@ -240,7 +241,7 @@ nfsrvd_getattr(struct nfsrv_descript *nd, int
isdgram, if (nd->nd_flag & ND_NFSV4) {
if (NFSISSET_ATTRBIT(,
NFSATTRBIT_FILEHANDLE)) nd->nd_repstat = nfsvno_getfh(vp, , p);
-   if (!nd->nd_repstat)
+   if (nd->nd_repstat == 0 && nfsrv_issuedelegs
== 1) nd->nd_repstat = nfsrv_checkgetattr(nd, vp,
, , nd->nd_cred, p);
if (nd->nd_repstat == 0) {
-- 
2.14.2


-- 
Emmanuel Vadot  
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CURRENT: core dumped no process name

2017-11-28 Thread Konstantin Belousov

On Tue, Nov 28, 2017 at 05:13:10PM +0700, Alex V. Petrov wrote:
> from /var/log/messages
> 
> # grep core /var/log/messages
> Nov 21 06:26:56 alex kernel: 1001: exited on signal 11 (core dumped)
> Nov 22 00:46:31 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> Nov 22 16:01:32 alex kernel: 1001: exited on signal 11 (core dumped)
> Nov 23 08:14:45 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> Nov 24 04:32:17 alex kernel: 1001: exited on signal 11 (core dumped)
> Nov 24 04:41:44 alex kernel: 1001: exited on signal 11 (core dumped)
> Nov 24 04:48:03 alex kernel: 1001: exited on signal 11 (core dumped)
> Nov 24 04:49:55 alex kernel: 1001: exited on signal 11 (core dumped)
> Nov 24 11:33:45 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> Nov 26 02:50:35 alex kernel: 1001: exited on signal 11 (core dumped)
> Nov 26 03:47:56 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> Nov 26 12:05:01 alex kernel: 1001: exited on signal 11 (core dumped)
> Nov 26 14:16:26 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> Nov 26 14:27:05 alex kernel: 0: exited on signal 6 (core dumped)
> Nov 26 23:49:56 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> Nov 26 23:50:26 alex kernel: 0: exited on signal 6 (core dumped)
> Nov 27 12:13:02 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> Nov 27 15:28:11 alex kernel: 0: exited on signal 6 (core dumped)
> Nov 27 15:50:54 alex kernel: 0: exited on signal 6 (core dumped)
> Nov 27 16:34:02 alex kernel: 0: exited on signal 6 (core dumped)
> Nov 28 03:07:42 alex kernel: 0: exited on signal 6 (core dumped)
> Nov 28 04:20:03 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> Nov 28 04:51:08 alex kernel: 0: exited on signal 6 (core dumped)
> Nov 28 10:28:38 alex kernel: 1001: exited on signal 11 (core dumped)
> Nov 28 15:55:42 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
> Nov 28 17:05:35 alex kernel: 0: exited on signal 6 (core dumped)
> 
Do you use syslogd after r325558 ?  If yes, try to revert it.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CURRENT: core dumped no process name

2017-11-28 Thread Alex V. Petrov

from /var/log/messages

# grep core /var/log/messages
Nov 21 06:26:56 alex kernel: 1001: exited on signal 11 (core dumped)
Nov 22 00:46:31 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
Nov 22 16:01:32 alex kernel: 1001: exited on signal 11 (core dumped)
Nov 23 08:14:45 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
Nov 24 04:32:17 alex kernel: 1001: exited on signal 11 (core dumped)
Nov 24 04:41:44 alex kernel: 1001: exited on signal 11 (core dumped)
Nov 24 04:48:03 alex kernel: 1001: exited on signal 11 (core dumped)
Nov 24 04:49:55 alex kernel: 1001: exited on signal 11 (core dumped)
Nov 24 11:33:45 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
Nov 26 02:50:35 alex kernel: 1001: exited on signal 11 (core dumped)
Nov 26 03:47:56 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
Nov 26 12:05:01 alex kernel: 1001: exited on signal 11 (core dumped)
Nov 26 14:16:26 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
Nov 26 14:27:05 alex kernel: 0: exited on signal 6 (core dumped)
Nov 26 23:49:56 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
Nov 26 23:50:26 alex kernel: 0: exited on signal 6 (core dumped)
Nov 27 12:13:02 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
Nov 27 15:28:11 alex kernel: 0: exited on signal 6 (core dumped)
Nov 27 15:50:54 alex kernel: 0: exited on signal 6 (core dumped)
Nov 27 16:34:02 alex kernel: 0: exited on signal 6 (core dumped)
Nov 28 03:07:42 alex kernel: 0: exited on signal 6 (core dumped)
Nov 28 04:20:03 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
Nov 28 04:51:08 alex kernel: 0: exited on signal 6 (core dumped)
Nov 28 10:28:38 alex kernel: 1001: exited on signal 11 (core dumped)
Nov 28 15:55:42 alex kernel: FreeBSD/SMP: 1 package(s) x 8 core(s)
Nov 28 17:05:35 alex kernel: 0: exited on signal 6 (core dumped)




28.11.2017 15:29, Konstantin Belousov пишет:
> On Tue, Nov 28, 2017 at 10:37:45AM +0700, Alex V. Petrov wrote:
>> SUBJ:
>> Nov 28 10:28:38 alex kernel: 1001: exited on signal 11 (core dumped)
>> It was a thunerbird
>>
>>
>> 12.0-CURRENT #0 r326286
>> kernel GENERIC-NODEBUG
> 
> The format strings is following:
> "pid %d (%s), uid %d: exited on signal %d%s\n"
> In other words, your line misses whole prefix, I suppose that '1001:' part
> is uid printout.
> 
> Where do you see this line ?  Is it from /var/log/messages, remote syslog
> service, console ?
> 

-- 
-
Alex.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CURRENT: core dumped no process name

2017-11-28 Thread Konstantin Belousov

On Tue, Nov 28, 2017 at 10:37:45AM +0700, Alex V. Petrov wrote:
> SUBJ:
> Nov 28 10:28:38 alex kernel: 1001: exited on signal 11 (core dumped)
> It was a thunerbird
> 
> 
> 12.0-CURRENT #0 r326286
> kernel GENERIC-NODEBUG

The format strings is following:
"pid %d (%s), uid %d: exited on signal %d%s\n"
In other words, your line misses whole prefix, I suppose that '1001:' part
is uid printout.

Where do you see this line ?  Is it from /var/log/messages, remote syslog
service, console ?
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

witness_lock_list_get: witness exhausted

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

Re: CURRENT: core dumped no process name

Re: CURRENT: core dumped no process name

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

Re: CURRENT: core dumped no process name

Re: CURRENT: core dumped no process name

Re: Switch vfs.nfsd.issue_delegations to TUNABLE ?

Switch vfs.nfsd.issue_delegations to TUNABLE ?

Re: CURRENT: core dumped no process name

Re: CURRENT: core dumped no process name

Re: CURRENT: core dumped no process name

18 matches

Site Navigation

Mail list logo

Footer information