Re: svn commit: r326758 - in head/sys/i386: conf include

[Bruce Evans]

On Wed, 20 Dec 2017, John Baldwin wrote:


On Wednesday, December 20, 2017 10:16:58 AM Nathan Whitehorn wrote:

...
With GCC 4, it takes a little while, but trying to build ports over NFS
is a sure-fire way to bring down the kernel. I haven't tried any other
compilers.


Ah, I have only done things like run binaries over NFS and compile simple
test programs over NFS with GCC 4 (I do run a gdb binary over NFS against
itself which probably involves a bit of I/O due to debug symbols, etc. but
still not as onerous as building lots of ports.  I cross-build the GDB on
the host due to qemu being too slow).


I use gcc-4.2.1 on i386 and amd64 with no problems except source code
breakage.   I use nfs a lot, but not zfs or newer networking.


clang insta-panics for even trivial
things like 'ls' and tab-completion though.  It's definitely much worse
than either version of GCC.


It is is indeed worse in every way :-) except it can build kernels
that run a couple of percent faster, so I don't use it.  It doesn't
support standard flags like -fno-inline-functions-called-once.  I use
anti-inlining flags like this mainly to simplify debugging, but they
help reduce stack bloat.  Kernels build with excessive inlining still
don't crash here.

The mechanism for stack bloat by inlining might be that when inlining
a bunch of functions, say all leaf functions, the stacks for all the
functions get added to the callee and don't get combined.

I wonder if -fstack-protector is any use.  I tried using the correct value
for KSTACK_PAGES (1) on i386, but there was always too much bloat for
that so it crashed fast.  -fstack-protector didn't help, and it crashed
with a double fault so there was no usable stack trace.  Automatically
expanding the stack by 1 page in the double fault handler would be not
much harder than finding the old stack to trace it.

Bruce
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Warner Losh]

On Wed, Dec 20, 2017 at 12:15 PM, Adrian Chadd  wrote:

> Hi,
>
> I kinda bet that this will just get worse over time, so maybe it's
> time we revisited figuring out a better way of dispatching work
> instead of (a) lots of kernel threads for different subsystems and (b)
> lots of deep stack frames.
>
> eg - NFS over wifi == hilarious pain
>

non-hilarious pain can be had by adding IPSEC to that recipe.

Warner
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Adrian Chadd]

Hi,

I kinda bet that this will just get worse over time, so maybe it's
time we revisited figuring out a better way of dispatching work
instead of (a) lots of kernel threads for different subsystems and (b)
lots of deep stack frames.

eg - NFS over wifi == hilarious pain




-adrian
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[John Baldwin]

On Wednesday, December 20, 2017 10:16:58 AM Nathan Whitehorn wrote:
> 
> On 12/20/17 09:14, John Baldwin wrote:
> > On Wednesday, December 20, 2017 09:59:26 AM David Chisnall wrote:
> >> On 16 Dec 2017, at 18:05, John Baldwin  wrote:
> >>> When I build a FreeBSD/mips64 kernel with clang,
> >>> _any_ simple NFS op triggers a kernel stack overflow.  Kernels compiled
> >>> with GCC do not.
> >> That is not my experience.  I haven’t tried a MIPS64 kernel built with 
> >> clang, but with in-tree gcc I get kernel panics as soon as I try to use 
> >> NFS, unless I use Stacey’s patches that increase the kernel stack size.
> > I have primarily been using modern GCC for GCC once that was working, but at
> > least when running a MALTA64 kernel under qemu I was not triggering panics
> > even with old GCC.  With the in-tree clang 5.0 or with CHERI clang, just
> > doing an 'ls' of a NFS directory or even a tab-complete of a path that
> > is on NFS reliably triggers a kernel stack overflow for MALTA64 in qemu.
> >
> > With Stacey's kstack pages, a clang kernel does survive, but those are not
> > in stock FreeBSD which is where I have been testing this.
> >
> 
> With GCC 4, it takes a little while, but trying to build ports over NFS 
> is a sure-fire way to bring down the kernel. I haven't tried any other 
> compilers.

Ah, I have only done things like run binaries over NFS and compile simple
test programs over NFS with GCC 4 (I do run a gdb binary over NFS against
itself which probably involves a bit of I/O due to debug symbols, etc. but
still not as onerous as building lots of ports.  I cross-build the GDB on
the host due to qemu being too slow).  clang insta-panics for even trivial
things like 'ls' and tab-completion though.  It's definitely much worse
than either version of GCC.

-- 
John Baldwin
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Nathan Whitehorn]

On 12/20/17 09:14, John Baldwin wrote:

On Wednesday, December 20, 2017 09:59:26 AM David Chisnall wrote:

On 16 Dec 2017, at 18:05, John Baldwin  wrote:

When I build a FreeBSD/mips64 kernel with clang,
_any_ simple NFS op triggers a kernel stack overflow.  Kernels compiled
with GCC do not.

That is not my experience.  I haven’t tried a MIPS64 kernel built with clang, 
but with in-tree gcc I get kernel panics as soon as I try to use NFS, unless I 
use Stacey’s patches that increase the kernel stack size.

I have primarily been using modern GCC for GCC once that was working, but at
least when running a MALTA64 kernel under qemu I was not triggering panics
even with old GCC.  With the in-tree clang 5.0 or with CHERI clang, just
doing an 'ls' of a NFS directory or even a tab-complete of a path that
is on NFS reliably triggers a kernel stack overflow for MALTA64 in qemu.

With Stacey's kstack pages, a clang kernel does survive, but those are not
in stock FreeBSD which is where I have been testing this.



With GCC 4, it takes a little while, but trying to build ports over NFS 
is a sure-fire way to bring down the kernel. I haven't tried any other 
compilers.

-Nathan
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[John Baldwin]

On Wednesday, December 20, 2017 09:59:26 AM David Chisnall wrote:
> On 16 Dec 2017, at 18:05, John Baldwin  wrote:
> > 
> > When I build a FreeBSD/mips64 kernel with clang,
> > _any_ simple NFS op triggers a kernel stack overflow.  Kernels compiled
> > with GCC do not.
> 
> That is not my experience.  I haven’t tried a MIPS64 kernel built with clang, 
> but with in-tree gcc I get kernel panics as soon as I try to use NFS, unless 
> I use Stacey’s patches that increase the kernel stack size.

I have primarily been using modern GCC for GCC once that was working, but at
least when running a MALTA64 kernel under qemu I was not triggering panics
even with old GCC.  With the in-tree clang 5.0 or with CHERI clang, just
doing an 'ls' of a NFS directory or even a tab-complete of a path that
is on NFS reliably triggers a kernel stack overflow for MALTA64 in qemu.

With Stacey's kstack pages, a clang kernel does survive, but those are not
in stock FreeBSD which is where I have been testing this.

-- 
John Baldwin
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[David Chisnall]

On 16 Dec 2017, at 18:05, John Baldwin  wrote:
> 
> When I build a FreeBSD/mips64 kernel with clang,
> _any_ simple NFS op triggers a kernel stack overflow.  Kernels compiled
> with GCC do not.

That is not my experience.  I haven’t tried a MIPS64 kernel built with clang, 
but with in-tree gcc I get kernel panics as soon as I try to use NFS, unless I 
use Stacey’s patches that increase the kernel stack size.

David

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Nathan Whitehorn]

On 12/12/17 11:32, John Baldwin wrote:

On 12/11/17 5:26 AM, Eugene Grosbein wrote:

On 11.12.2017 16:19, Konstantin Belousov wrote:

On Mon, Dec 11, 2017 at 04:32:37AM +, Conrad Meyer wrote:

Author: cem
Date: Mon Dec 11 04:32:37 2017
New Revision: 326758
URL: https://svnweb.freebsd.org/changeset/base/326758

Log:
   i386: Bump KSTACK_PAGES default to match amd64

i386 is not amd64, the change is wrong.

i386 has the word size two times smaller than amd64, which makes typical
frame smaller by 30-40% over same code on amd64. Also i386 has much
smaller available KVA size (tens of MB) and KVA fragmentation is both
more severe and more fatal due to this. I expect that your change will
make any non-trivial load which creates enough threads to either fail
randomly or deadlock.

If somebody tries to fit large load onto i386 machine, he must know what to
do and how to configure the kernel to adapt to the load (which does not
require the recompilation).

Its very easy to get kernel stack overflow with 11-STABLE/i386
without any significant load due to abuse of kernel stack in many kernel 
subsystems
as shown in the https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219476

Contrary, "enough threads" seems to be very non-trivial number of threads
and pretty unusual load pattern for i386 as I run several such systems
with kern.kstack_pages=4 for quite long time and have no problems.
No random fails, no deadlocks. And with 2 pages only 11-STABLE/i386 is just 
unusable
if one utilizes SCTP, IPv6, some WiFi connectivity, IPSEC or even very small 
ZFS pool.

I still wonder if there is really such load pattern that creates "enough 
threads"
for i386 to make 4-pages stack troublesome.

I suspect two things are at play in running out of stack in 10.x and later.
1) Usage of XSAVE for AVX, etc. state uses more of the kstack (as kib@ alludes
to), and 2) clang.  Certainly for MIPS I have found that compiling with clang
instead of gcc for mips64 gives a kernel that panics for stack overflow for any
use of NFS.  It might be that this is due to something MIPS-specific, but it
might be worthwhile retesting with kstack_pages=2 and building the kernel
with CROSS_TOOLCHAIN=i386-gcc after installing the appropriate package.


For what it's worth, I see the same NFS-related stack overflows on 
mips64 with in-tree GCC 4.2.1.

-Nathan

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[John Baldwin]

On 12/14/17 3:34 AM, Eugene Grosbein wrote:
> On 13.12.2017 04:55, John Baldwin wrote:
>> On 12/12/17 3:09 PM, Eugene Grosbein wrote:
>>> On 13.12.2017 02:32, John Baldwin wrote:
>>>
 Certainly for MIPS I have found that compiling with clang
 instead of gcc for mips64 gives a kernel that panics for stack overflow 
 for any
 use of NFS.  It might be that this is due to something MIPS-specific, but 
 it
 might be worthwhile retesting with kstack_pages=2 and building the kernel
 with CROSS_TOOLCHAIN=i386-gcc after installing the appropriate package.
>>>
>>> You may want to check NFS code that uses stack heavily.
>>> Here are numbers for i386 (bytes-on-stack, module, what function):
>>>
>>> 1344 nfs_nfsdport.o :
>>> 1152 nfs_nfsdserv.o :
>>> 1128 nfs_nfsdserv.o :
>>> 952 nfs_nfsdserv.o :
>>> 664 nfs_nfsdserv.o :
>>> 640 nfs_nfsdserv.o :
>>> 624 nfs_nfsdserv.o :
>>> 608 nfs_nfsdserv.o :
>>> 600 nfs_clvfsops.o :
>>
>> My point is that you should compare gcc with clang as 10.x switched to
>> clang and that may be a factor in the stack overflows beginning with 10.x.
> 
> I think thats's NFS code who is guilty. You can see example of amd64 (sic!) 
> kstack exhaustion
> due to 40+ frames deep call chain here:
> 
> https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087429.html

In case it is not clear, it is not _just_ NFS that is broken.  clang is
_known_ to be broken.  When I build a FreeBSD/mips64 kernel with clang,
_any_ simple NFS op triggers a kernel stack overflow.  Kernels compiled
with GCC do not.  I would really appreciate investigating the clang vs gcc
on i386 to determine if this issue is platform-specific or not.  If clang
is stack hungry on all architectures then we need to be aware of that
problem.  (clang 5 on MIPS also has the lovely property that it doesn't
save $RA when it calls a dead2 function, so the stack trace for the kernel
stack overflow from ddb is useless and ends when the trap handler calls
panic, it never makes it back into the original stack which overflowed.)

-- 
John Baldwin
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

14.12.2017 23:00, Konstantin Belousov wrote:

> Sigh. This would make i386 even less usable for everybody, perhaps
> except you. Because default 3G of UVA is too small for some common tasks
> (thanks clang, but also e.g. pypy), and you reduce the user address
> space even more.
 Those who need 3GB of UVA within single process should not use 32 bit 
 system in first place, should they?
>>> Why do you even consider it acceptable to break the configuration just
>>> because you are not interested in the workload ?
>> I do not consider it is acceptable. I'm trying to find compromise.
> If you do not find the proposed change acceptable, why do you propose
> it at all ?

I do not consider breakage acceptable but not the change.
I am still in doubt and that's why I'm asking questions.
I doubt, how average i386 workstation or server can generate 1000+ threads
and what is real threshold to get problems because of KVA fragmentation?

>>> 386 arch configuration is not perfect but it provides the main intended
>>> service of general-purpose workstation, with some limitation caused by
>>> 32bit of address space being somewhat low for modern code. Appliance
>>> flavoring of the default 386 config is unacceptable.  Keep your tweaks
>>> local.
>>
>> Do you think that kstack overflow of general-purpose workstation using NFS
>> or IPv6 or WiFi with SCTP is OK? I do not.
> 
> Kernel stack overflow is not 'OK', but two things are equally troublesome:
> - papering over the problem of large structures allocated on stack by
>   increasing the stack size;
> - ignoring the explanation why the stack increase is problematic by itself

The change was submitted for discussion before any kind of explanation was 
given.

>   and generating thread consisting of 50+ content-less messages claiming
>   that 'it fixes the problem for me'.

Not for "me". For average i386 user of IPv6 or SCTP or NFS etc. reporting 
double faults.

> The proper route to fix the issues was already explained more than
> several times, and you (and I, and other people) participated in the
> ongoing efforts there. If it is so much important to you, wave the flag
> and herd the action, by asking relevant people to help with fixes, be it
> writing the patches, providing reviews or just finding the place where
> the stack is abused. It requires identification and time to think and
> develop specific changes for each place, and cannot happen in one day.

Exactly. The problem is officialy noted in our docs since 10.2-RELEASE at least.
And I doubt we can fix all the code in some foreseeable future and 11.2-RELEASE
will still panic same way. And 12.0-RELEASE too, I'm afraid. And that is more
important than keeping old defaults for i386 workstation users
*unless* there are really lots of applications creating large number
of threads/subprocesses.

> But it also would not break KVA-starved architectures and provide fixes
> relevant to even KVA-ample machines, as you other example shown.
> 
> For the case of extreme use, there is still the tunable (and ZFS on i386
> is extreme).

Sadly, we have too many "extreme" cases, as user reports and code analisys 
shows.
That's the problem.


___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Konstantin Belousov]

On Thu, Dec 14, 2017 at 10:39:18PM +0700, Eugene Grosbein wrote:
> On 14.12.2017 22:23, Konstantin Belousov wrote:
> 
> >>> Sigh. This would make i386 even less usable for everybody, perhaps
> >>> except you. Because default 3G of UVA is too small for some common tasks
> >>> (thanks clang, but also e.g. pypy), and you reduce the user address
> >>> space even more.
> >>
> >> Those who need 3GB of UVA within single process should not use 32 bit 
> >> system in first place, should they?
> > 
> > Why do you even consider it acceptable to break the configuration just
> > because you are not interested in the workload ?
> 
> I do not consider it is acceptable. I'm trying to find compromise.
If you do not find the proposed change acceptable, why do you propose
it at all ?

> 
> > 3G cumulative VA does not translate into 3G of usable addresses: the bss
> > is reserved, the address space is fragmented due to need of growing
> > stack to not conflict with mmaped regions. As result, applications get
> > slightly less than 2G usable without tricks, and to apply the tricks
> > apps must know details of the AS layout, i.e. breaking the portability.
> > 
> > 386 arch configuration is not perfect but it provides the main intended
> > service of general-purpose workstation, with some limitation caused by
> > 32bit of address space being somewhat low for modern code. Appliance
> > flavoring of the default 386 config is unacceptable.  Keep your tweaks
> > local.
> 
> Do you think that kstack overflow of general-purpose workstation using NFS
> or IPv6 or WiFi with SCTP is OK? I do not.

Kernel stack overflow is not 'OK', but two things are equally troublesome:
- papering over the problem of large structures allocated on stack by
  increasing the stack size;
- ignoring the explanation why the stack increase is problematic by itself
  and generating thread consisting of 50+ content-less messages claiming
  that 'it fixes the problem for me'.

The proper route to fix the issues was already explained more than
several times, and you (and I, and other people) participated in the
ongoing efforts there. If it is so much important to you, wave the flag
and herd the action, by asking relevant people to help with fixes, be it
writing the patches, providing reviews or just finding the place where
the stack is abused. It requires identification and time to think and
develop specific changes for each place, and cannot happen in one day.
But it also would not break KVA-starved architectures and provide fixes
relevant to even KVA-ample machines, as you other example shown.

For the case of extreme use, there is still the tunable (and ZFS on i386
is extreme).
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

On 14.12.2017 22:23, Konstantin Belousov wrote:

>>> Sigh. This would make i386 even less usable for everybody, perhaps
>>> except you. Because default 3G of UVA is too small for some common tasks
>>> (thanks clang, but also e.g. pypy), and you reduce the user address
>>> space even more.
>>
>> Those who need 3GB of UVA within single process should not use 32 bit system 
>> in first place, should they?
> 
> Why do you even consider it acceptable to break the configuration just
> because you are not interested in the workload ?

I do not consider it is acceptable. I'm trying to find compromise.

> 3G cumulative VA does not translate into 3G of usable addresses: the bss
> is reserved, the address space is fragmented due to need of growing
> stack to not conflict with mmaped regions. As result, applications get
> slightly less than 2G usable without tricks, and to apply the tricks
> apps must know details of the AS layout, i.e. breaking the portability.
> 
> 386 arch configuration is not perfect but it provides the main intended
> service of general-purpose workstation, with some limitation caused by
> 32bit of address space being somewhat low for modern code. Appliance
> flavoring of the default 386 config is unacceptable.  Keep your tweaks
> local.

Do you think that kstack overflow of general-purpose workstation using NFS
or IPv6 or WiFi with SCTP is OK? I do not.

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Konstantin Belousov]

On Thu, Dec 14, 2017 at 07:59:03PM +0700, Eugene Grosbein wrote:
> On 14.12.2017 19:26, Konstantin Belousov wrote:
> 
> > Sigh. This would make i386 even less usable for everybody, perhaps
> > except you. Because default 3G of UVA is too small for some common tasks
> > (thanks clang, but also e.g. pypy), and you reduce the user address
> > space even more.
> 
> Those who need 3GB of UVA within single process should not use 32 bit system 
> in first place, should they?

Why do you even consider it acceptable to break the configuration just
because you are not interested in the workload ?

3G cumulative VA does not translate into 3G of usable addresses: the bss
is reserved, the address space is fragmented due to need of growing
stack to not conflict with mmaped regions. As result, applications get
slightly less than 2G usable without tricks, and to apply the tricks
apps must know details of the AS layout, i.e. breaking the portability.

386 arch configuration is not perfect but it provides the main intended
service of general-purpose workstation, with some limitation caused by
32bit of address space being somewhat low for modern code. Appliance
flavoring of the default 386 config is unacceptable.  Keep your tweaks
local.
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

On 14.12.2017 19:26, Konstantin Belousov wrote:

> Sigh. This would make i386 even less usable for everybody, perhaps
> except you. Because default 3G of UVA is too small for some common tasks
> (thanks clang, but also e.g. pypy), and you reduce the user address
> space even more.

Those who need 3GB of UVA within single process should not use 32 bit system in 
first place, should they?

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Konstantin Belousov]

On Thu, Dec 14, 2017 at 07:04:57PM +0700, Eugene Grosbein wrote:
> On 14.12.2017 18:51, Konstantin Belousov wrote:
> 
> >> I think thats's NFS code who is guilty. You can see example of amd64 
> >> (sic!) kstack exhaustion
> >> due to 40+ frames deep call chain here:
> >>
> >> https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087429.html
> > 
> > Yes, NFS crosses network/VFS and often VM boundaries, so each subsystem
> > adds its usual stack use footprint to the overall picture.  NFS reconnect
> > is especially hard in this regard, and in case the direct dispatch is
> > triggered (in this case over loopback) machine has no chance.
> > 
> > The backtrace you cited just reinforces the point that the i386 commit
> > is wrong. It breaks the workload we aims as the main FreeBSD target,
> > which is generic-purpose Unix workstation or server. The commit tries
> > to make defaults fit for specific appliance load of router with IPSEC
> > or ZFS on i386, which require extensive tuning on i386 anyway. Worse,
> > as you prove above, the commit in fact does not fix the issues, it only
> > papers over them and move easily triggered faults from one configuration
> > to another.
> 
> Modern FreeBSD usage as workstation/server should not exclude IPv6, SCTP, 
> WiFi,
> and even ZFS nor IPSEC for i386. GENERIC kernel should not panic due to low 
> volume
> network activity with default settings.
And two ponies should be given to everybody who wishes for them.

> 
> Perhaps, it's time to make KVA_PAGES loader tunnable too?
Sure, make it the tunable.  Just to make you know in advance, this is quite
delicate.

> And/or increase its default for i386 upto some value corresponding to stable 
> management
> of kern.threads.max_threads_per_proc=1500 (by default) with kstack_pages=4 ?
> 
> Maybe, KVA_PAGES=384 (1.5GB for 1500 threads)?
Sigh. This would make i386 even less usable for everybody, perhaps
except you. Because default 3G of UVA is too small for some common tasks
(thanks clang, but also e.g. pypy), and you reduce the user address
space even more.
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

On 14.12.2017 18:51, Konstantin Belousov wrote:

>> I think thats's NFS code who is guilty. You can see example of amd64 (sic!) 
>> kstack exhaustion
>> due to 40+ frames deep call chain here:
>>
>> https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087429.html
> 
> Yes, NFS crosses network/VFS and often VM boundaries, so each subsystem
> adds its usual stack use footprint to the overall picture.  NFS reconnect
> is especially hard in this regard, and in case the direct dispatch is
> triggered (in this case over loopback) machine has no chance.
> 
> The backtrace you cited just reinforces the point that the i386 commit
> is wrong. It breaks the workload we aims as the main FreeBSD target,
> which is generic-purpose Unix workstation or server. The commit tries
> to make defaults fit for specific appliance load of router with IPSEC
> or ZFS on i386, which require extensive tuning on i386 anyway. Worse,
> as you prove above, the commit in fact does not fix the issues, it only
> papers over them and move easily triggered faults from one configuration
> to another.

Modern FreeBSD usage as workstation/server should not exclude IPv6, SCTP, WiFi,
and even ZFS nor IPSEC for i386. GENERIC kernel should not panic due to low 
volume
network activity with default settings.

Perhaps, it's time to make KVA_PAGES loader tunnable too?
And/or increase its default for i386 upto some value corresponding to stable 
management
of kern.threads.max_threads_per_proc=1500 (by default) with kstack_pages=4 ?

Maybe, KVA_PAGES=384 (1.5GB for 1500 threads)?

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Konstantin Belousov]

On Thu, Dec 14, 2017 at 06:34:21PM +0700, Eugene Grosbein wrote:
> On 13.12.2017 04:55, John Baldwin wrote:
> > On 12/12/17 3:09 PM, Eugene Grosbein wrote:
> >> On 13.12.2017 02:32, John Baldwin wrote:
> >>
> >>> Certainly for MIPS I have found that compiling with clang
> >>> instead of gcc for mips64 gives a kernel that panics for stack overflow 
> >>> for any
> >>> use of NFS.  It might be that this is due to something MIPS-specific, but 
> >>> it
> >>> might be worthwhile retesting with kstack_pages=2 and building the kernel
> >>> with CROSS_TOOLCHAIN=i386-gcc after installing the appropriate package.
> >>
> >> You may want to check NFS code that uses stack heavily.
> >> Here are numbers for i386 (bytes-on-stack, module, what function):
> >>
> >> 1344 nfs_nfsdport.o :
> >> 1152 nfs_nfsdserv.o :
> >> 1128 nfs_nfsdserv.o :
> >> 952 nfs_nfsdserv.o :
> >> 664 nfs_nfsdserv.o :
> >> 640 nfs_nfsdserv.o :
> >> 624 nfs_nfsdserv.o :
> >> 608 nfs_nfsdserv.o :
> >> 600 nfs_clvfsops.o :
> > 
> > My point is that you should compare gcc with clang as 10.x switched to
> > clang and that may be a factor in the stack overflows beginning with 10.x.
> 
> I think thats's NFS code who is guilty. You can see example of amd64 (sic!) 
> kstack exhaustion
> due to 40+ frames deep call chain here:
> 
> https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087429.html

Yes, NFS crosses network/VFS and often VM boundaries, so each subsystem
adds its usual stack use footprint to the overall picture.  NFS reconnect
is especially hard in this regard, and in case the direct dispatch is
triggered (in this case over loopback) machine has no chance.

The backtrace you cited just reinforces the point that the i386 commit
is wrong. It breaks the workload we aims as the main FreeBSD target,
which is generic-purpose Unix workstation or server. The commit tries
to make defaults fit for specific appliance load of router with IPSEC
or ZFS on i386, which require extensive tuning on i386 anyway. Worse,
as you prove above, the commit in fact does not fix the issues, it only
papers over them and move easily triggered faults from one configuration
to another.
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

On 14.12.2017 18:34, Eugene Grosbein wrote:
> On 13.12.2017 04:55, John Baldwin wrote:
>> On 12/12/17 3:09 PM, Eugene Grosbein wrote:
>>> On 13.12.2017 02:32, John Baldwin wrote:
>>>
 Certainly for MIPS I have found that compiling with clang
 instead of gcc for mips64 gives a kernel that panics for stack overflow 
 for any
 use of NFS.  It might be that this is due to something MIPS-specific, but 
 it
 might be worthwhile retesting with kstack_pages=2 and building the kernel
 with CROSS_TOOLCHAIN=i386-gcc after installing the appropriate package.
>>>
>>> You may want to check NFS code that uses stack heavily.
>>> Here are numbers for i386 (bytes-on-stack, module, what function):
>>>
>>> 1344 nfs_nfsdport.o :
>>> 1152 nfs_nfsdserv.o :
>>> 1128 nfs_nfsdserv.o :
>>> 952 nfs_nfsdserv.o :
>>> 664 nfs_nfsdserv.o :
>>> 640 nfs_nfsdserv.o :
>>> 624 nfs_nfsdserv.o :
>>> 608 nfs_nfsdserv.o :
>>> 600 nfs_clvfsops.o :
>>
>> My point is that you should compare gcc with clang as 10.x switched to
>> clang and that may be a factor in the stack overflows beginning with 10.x.
> 
> I think thats's NFS code who is guilty. You can see example of amd64 (sic!) 
> kstack exhaustion
> due to 40+ frames deep call chain here:
> 
> https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087429.html

Final post in the thread describes amd64 crash with kstack_pages=5 due to 75 
frames:
https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087477.html

He increased kstack_pages upto 6 and stopped posting, as it obviously helped 
him finally.

By the way, I finally realised why I do not have KVA problems with my i386 
hosts:
I run all of them (no PAE) with options KVA_PAGES=512 (2GB) at least
and even 768 (3GB) in case of ZFS instead of default 256 (1GB) to make it 
stable.

Tell me more of untweaked FreeBSD stability...

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

On 13.12.2017 04:55, John Baldwin wrote:
> On 12/12/17 3:09 PM, Eugene Grosbein wrote:
>> On 13.12.2017 02:32, John Baldwin wrote:
>>
>>> Certainly for MIPS I have found that compiling with clang
>>> instead of gcc for mips64 gives a kernel that panics for stack overflow for 
>>> any
>>> use of NFS.  It might be that this is due to something MIPS-specific, but it
>>> might be worthwhile retesting with kstack_pages=2 and building the kernel
>>> with CROSS_TOOLCHAIN=i386-gcc after installing the appropriate package.
>>
>> You may want to check NFS code that uses stack heavily.
>> Here are numbers for i386 (bytes-on-stack, module, what function):
>>
>> 1344 nfs_nfsdport.o :
>> 1152 nfs_nfsdserv.o :
>> 1128 nfs_nfsdserv.o :
>> 952 nfs_nfsdserv.o :
>> 664 nfs_nfsdserv.o :
>> 640 nfs_nfsdserv.o :
>> 624 nfs_nfsdserv.o :
>> 608 nfs_nfsdserv.o :
>> 600 nfs_clvfsops.o :
> 
> My point is that you should compare gcc with clang as 10.x switched to
> clang and that may be a factor in the stack overflows beginning with 10.x.

I think thats's NFS code who is guilty. You can see example of amd64 (sic!) 
kstack exhaustion
due to 40+ frames deep call chain here:

https://lists.freebsd.org/pipermail/freebsd-stable/2017-July/087429.html

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

13.12.2017 3:43, Rodney W. Grimes wrote:

> Or are you thinking we have something that is so deep even if it
> only uses 32 bytes of stack we are going to run ourselfs out of
> kstack under some work loads?  I hope not.

No, I'm not.

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Rodney W. Grimes]

> On 13.12.2017 00:32, Rodney W. Grimes wrote:
> 
> >> I am not sure if there are tools that can analyze stack requirements for
> >> possible call chains rather than individual functions.
> > 
> > Call graphs can be used to find deep chains.  Combine the above
> > with a call graph and we should be able to come up with some
> > data.
> > 
> > This also sounds like a good caniate for GSOC, creation
> > of a call graph with static analysis of local variable size +
> > call frame size requirements as a O(1) stack space estimator.
> 
> "Longitude" of call chain greatly depends of run-time system configuration.
> For example, application sendto() call can result in variable number
> of system calls performing IPSEC transformations (ESP, AH, IPCOMP)
> then custom PFIL processing then multiple NETGRAPH calls then complex
> WiFi stack operations and each of such calls can result in kernel stack 
> exhaustion.

That was why I expressed it as an O(1) estimator.  Attempting to do
this more complex problem you describe would probably best be left
to dtrace and some coverage metrics to make sure we hit most of the
paths.

At this time I don't think we need that.  I think what you did with
your very quick turn around of the simple static data is probably
going to lead to rapid closure of the problem, or at least greatly
reduce it.

Or are you thinking we have something that is so deep even if it
only uses 32 bytes of stack we are going to run ourselfs out of
kstack under some work loads?  I hope not.

-- 
Rod Grimes rgri...@freebsd.org
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Rodney W. Grimes]

[ Charset UTF-8 unsupported, converting... ]
> On 12/12/2017 19:01, Rodney W. Grimes wrote:
> > We should probably start looking for those, something someone who is
> > offerring help but doesnt know what they might be good at, but can
> > read C code.  They could be utililized t simply scan through the
> > code looking for this type of thing and bring it to the attention
> > of a area of expertise commiter.
> 
> By the way, there are tools that can analyze binary code (e.g. a kernel binary
> or a module) and determine stack requirements of each function.  I think that
> debug symbols might be required too.

That would be an even better first start, even if it just could
tell us what the local var stack space request of each function
was that might point out some easly low hanging fruit.

Can it also tell us how big the call frame is?

> 
> I am not sure if there are tools that can analyze stack requirements for
> possible call chains rather than individual functions.

Call graphs can be used to find deep chains.  Combine the above
with a call graph and we should be able to come up with some
data.

This also sounds like a good caniate for GSOC, creation
of a call graph with static analysis of local variable size +
call frame size requirements as a O(1) stack space estimator.


-- 
Rod Grimes rgri...@freebsd.org
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[John Baldwin]

On 12/12/17 3:09 PM, Eugene Grosbein wrote:
> On 13.12.2017 02:32, John Baldwin wrote:
> 
>> Certainly for MIPS I have found that compiling with clang
>> instead of gcc for mips64 gives a kernel that panics for stack overflow for 
>> any
>> use of NFS.  It might be that this is due to something MIPS-specific, but it
>> might be worthwhile retesting with kstack_pages=2 and building the kernel
>> with CROSS_TOOLCHAIN=i386-gcc after installing the appropriate package.
> 
> You may want to check NFS code that uses stack heavily.
> Here are numbers for i386 (bytes-on-stack, module, what function):
> 
> 1344 nfs_nfsdport.o :
> 1152 nfs_nfsdserv.o :
> 1128 nfs_nfsdserv.o :
> 952 nfs_nfsdserv.o :
> 664 nfs_nfsdserv.o :
> 640 nfs_nfsdserv.o :
> 624 nfs_nfsdserv.o :
> 608 nfs_nfsdserv.o :
> 600 nfs_clvfsops.o :

My point is that you should compare gcc with clang as 10.x switched to
clang and that may be a factor in the stack overflows beginning with 10.x.

-- 
John Baldwin
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Rodney W. Grimes]

> On 13.12.2017 00:01, Rodney W. Grimes wrote:
> 
> >> But many other parts of kernel think it's OK to allocate big arrays or
> >> structures on stack.
> > 
> > We should probably start looking for those, something someone who is
> > offerring help but doesnt know what they might be good at, but can
> > read C code.  They could be utililized t simply scan through the
> > code looking for this type of thing and bring it to the attention
> > of a area of expertise commiter.
> 
> These can be found with simple script:
> 
> cd /usr/src
> make TARGET_ARCH=i386 KERNCONF=GENERIC buildkernel
> 
> cd /usr/obj/i386.i386/home/src/sys/GENERIC
> for o in *.o
> do
>   objdump -d $o | awk -vn=$o '/sub .*,%esp/ {split ($NF,a,/[,$]/); printf "%u 
> %s %s\n", a[2], a[2], n}'
> done | sort -rn > top.sub
> 
> head -50 top.sub | while read d h o
> do
>   objdump -d $o | fgrep -B7 "$h,%esp" | awk -vo=$o -vd=$d '/>:$/ {print d, o, 
> $2}'
> done > top2.sub
> 
> Here is what do we get in top2.sub:
> 
> 4328 in6_proto.o :
> 3312 in6_proto.o :
> 2232 sctp_pcb.o :
...

Thank you Thank you THANK YOU!  And imp has already removed one of these.
I wonder if fxing some of these sctp*.o would lead to the removal of the 
complaints
about that feature filling the stack.

> ...
> 
> First column is total stack usage of kernel function (decimal),
> second is module name and third contains function name it question.

Can we get this put into a CI job?
Output anything that gets added to the list of things with a stack
usage greater than some N to an email sent.  That way new violators
get flagged quickly and can be cleaned up by the commiter, then
slowly work down this list until things are stable again.

-- 
Rod Grimes rgri...@freebsd.org
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

On 13.12.2017 02:32, John Baldwin wrote:

> Certainly for MIPS I have found that compiling with clang
> instead of gcc for mips64 gives a kernel that panics for stack overflow for 
> any
> use of NFS.  It might be that this is due to something MIPS-specific, but it
> might be worthwhile retesting with kstack_pages=2 and building the kernel
> with CROSS_TOOLCHAIN=i386-gcc after installing the appropriate package.

You may want to check NFS code that uses stack heavily.
Here are numbers for i386 (bytes-on-stack, module, what function):

1344 nfs_nfsdport.o :
1152 nfs_nfsdserv.o :
1128 nfs_nfsdserv.o :
952 nfs_nfsdserv.o :
664 nfs_nfsdserv.o :
640 nfs_nfsdserv.o :
624 nfs_nfsdserv.o :
608 nfs_nfsdserv.o :
600 nfs_clvfsops.o :


___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[John Baldwin]

On 12/11/17 5:26 AM, Eugene Grosbein wrote:
> On 11.12.2017 16:19, Konstantin Belousov wrote:
>> On Mon, Dec 11, 2017 at 04:32:37AM +, Conrad Meyer wrote:
>>> Author: cem
>>> Date: Mon Dec 11 04:32:37 2017
>>> New Revision: 326758
>>> URL: https://svnweb.freebsd.org/changeset/base/326758
>>>
>>> Log:
>>>   i386: Bump KSTACK_PAGES default to match amd64
>> i386 is not amd64, the change is wrong.
>>
>> i386 has the word size two times smaller than amd64, which makes typical
>> frame smaller by 30-40% over same code on amd64. Also i386 has much
>> smaller available KVA size (tens of MB) and KVA fragmentation is both
>> more severe and more fatal due to this. I expect that your change will
>> make any non-trivial load which creates enough threads to either fail
>> randomly or deadlock.
>>
>> If somebody tries to fit large load onto i386 machine, he must know what to
>> do and how to configure the kernel to adapt to the load (which does not
>> require the recompilation).
> 
> Its very easy to get kernel stack overflow with 11-STABLE/i386
> without any significant load due to abuse of kernel stack in many kernel 
> subsystems
> as shown in the https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219476
> 
> Contrary, "enough threads" seems to be very non-trivial number of threads
> and pretty unusual load pattern for i386 as I run several such systems
> with kern.kstack_pages=4 for quite long time and have no problems.
> No random fails, no deadlocks. And with 2 pages only 11-STABLE/i386 is just 
> unusable
> if one utilizes SCTP, IPv6, some WiFi connectivity, IPSEC or even very small 
> ZFS pool.
> 
> I still wonder if there is really such load pattern that creates "enough 
> threads"
> for i386 to make 4-pages stack troublesome.

I suspect two things are at play in running out of stack in 10.x and later.
1) Usage of XSAVE for AVX, etc. state uses more of the kstack (as kib@ alludes
to), and 2) clang.  Certainly for MIPS I have found that compiling with clang
instead of gcc for mips64 gives a kernel that panics for stack overflow for any
use of NFS.  It might be that this is due to something MIPS-specific, but it
might be worthwhile retesting with kstack_pages=2 and building the kernel
with CROSS_TOOLCHAIN=i386-gcc after installing the appropriate package.

For people wanting to use 32-bit binaries on a modern x86 box, we should 
probably
add support for the x32 ABI permitting you to use 32-bit pointers but with the
full complement of 64-bit registers (and still using the 64-bit kernel so
there is not KVA pressure for the kernel itself).

-- 
John Baldwin
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

On 13.12.2017 00:32, Rodney W. Grimes wrote:

>> I am not sure if there are tools that can analyze stack requirements for
>> possible call chains rather than individual functions.
> 
> Call graphs can be used to find deep chains.  Combine the above
> with a call graph and we should be able to come up with some
> data.
> 
> This also sounds like a good caniate for GSOC, creation
> of a call graph with static analysis of local variable size +
> call frame size requirements as a O(1) stack space estimator.

"Longitude" of call chain greatly depends of run-time system configuration.
For example, application sendto() call can result in variable number
of system calls performing IPSEC transformations (ESP, AH, IPCOMP)
then custom PFIL processing then multiple NETGRAPH calls then complex
WiFi stack operations and each of such calls can result in kernel stack 
exhaustion.

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

On 13.12.2017 00:01, Rodney W. Grimes wrote:

>> But many other parts of kernel think it's OK to allocate big arrays or
>> structures on stack.
> 
> We should probably start looking for those, something someone who is
> offerring help but doesnt know what they might be good at, but can
> read C code.  They could be utililized t simply scan through the
> code looking for this type of thing and bring it to the attention
> of a area of expertise commiter.

These can be found with simple script:

cd /usr/src
make TARGET_ARCH=i386 KERNCONF=GENERIC buildkernel

cd /usr/obj/i386.i386/home/src/sys/GENERIC
for o in *.o
do
  objdump -d $o | awk -vn=$o '/sub .*,%esp/ {split ($NF,a,/[,$]/); printf "%u 
%s %s\n", a[2], a[2], n}'
done | sort -rn > top.sub

head -50 top.sub | while read d h o
do
  objdump -d $o | fgrep -B7 "$h,%esp" | awk -vo=$o -vd=$d '/>:$/ {print d, o, 
$2}'
done > top2.sub

Here is what do we get in top2.sub:

4328 in6_proto.o :
3312 in6_proto.o :
2232 sctp_pcb.o :
2208 xform_esp.o :
2176 ip6_output.o :
2092 ar9300_eeprom.o :
2080 kern_linker.o :
1696 md_ddf.o :
1664 cryptosoft.o :
1592 sctp_auth.o :
1420 zlib.o :
1416 ar9300_paprd.o :
1360 scsi_sa.o :
1352 scsi_da.o :
1344 nfs_nfsdport.o :
1328 vm_object.o :
1320 scsi_cd.o :
1312 scsi_enc_ses.o :
1312 fortuna.o :
1232 cam_periph.o :
1232 bt.o :
1224 zlib.o :
1192 cam_xpt.o :
1184 cam_xpt.o :
1184 ata_da.o :
1168 ata_da.o :
1160 sctp_output.o :
1160 cam_periph.o :
1152 nfs_nfsdserv.o :
1136 sctp_output.o :
1128 nfs_nfsdserv.o :
1088 rt2860.o :
1088 blkback.o :
1080 rt2860.o :
1080 pseudofs_vnops.o :
1072 cardbus_cis.o :
...

First column is total stack usage of kernel function (decimal),
second is module name and third contains function name it question.

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Andriy Gapon]

On 12/12/2017 19:01, Rodney W. Grimes wrote:
> We should probably start looking for those, something someone who is
> offerring help but doesnt know what they might be good at, but can
> read C code.  They could be utililized t simply scan through the
> code looking for this type of thing and bring it to the attention
> of a area of expertise commiter.

By the way, there are tools that can analyze binary code (e.g. a kernel binary
or a module) and determine stack requirements of each function.  I think that
debug symbols might be required too.

I am not sure if there are tools that can analyze stack requirements for
possible call chains rather than individual functions.

-- 
Andriy Gapon
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Rodney W. Grimes]

[ Charset windows-1252 unsupported, converting... ]
> 12.12.2017 22:30, Rodney W. Grimes:
> 
> >>> Now I run FreeBSD 11/i386 as my home router with IPSEC and torrent
> >>> client, and I run several virtualized routers with IPSEC tunnels,
> >>> jabber and mail server, squid and ZFS for src/obj/ports compression
> >>> and they all easily crash unless kern.kstack_pages raised upto 4. Same
> >>> for some other my i386 installations having IPSEC tunnels.
> >>
> >> IPSEC definitely used to wwith with kstack_pages=2 since I ran that way
> >> for a number of years.  I haven't used IPSEC since I upgraded from
> >> FreeBSD 8.x to 10.x a while back, so it could be broken now.
> > 
> > I think this comes as a regression in 10.x or perhaps later.  So that
> > atleast narrows down what has triggered the need for more kernel stack
> > space.
> 
> Once again, that's not about IPSEC only that, indeed, had this kind of 
> "regression"

I didnt mean to make it sound like it was IPSEC only, I simply meant to
say that in general we have a kstack use regression that appears to have
started after 10.x.  I hope this clarifies my point.

> with overhaul of its code between 11.0 and 11.1 releases with r315514.
> It was already polished in stable/11 with later r319118 plus there is
> https://reviews.freebsd.org/D9721 that introduces new
> sysctl net.inet.ipsec.use_netisr=1
> to convert long path of direct function calls requiring large stack to
> queuing of outgoing to-be-encrypted traffic using NETISR
> at cost of some performance penalty when enabled.
> 
> But many other parts of kernel think it's OK to allocate big arrays or
> structures on stack.

We should probably start looking for those, something someone who is
offerring help but doesnt know what they might be good at, but can
read C code.  They could be utililized t simply scan through the
code looking for this type of thing and bring it to the attention
of a area of expertise commiter.

-- 
Rod Grimes rgri...@freebsd.org
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

12.12.2017 22:30, Rodney W. Grimes:

>>> Now I run FreeBSD 11/i386 as my home router with IPSEC and torrent
>>> client, and I run several virtualized routers with IPSEC tunnels,
>>> jabber and mail server, squid and ZFS for src/obj/ports compression
>>> and they all easily crash unless kern.kstack_pages raised upto 4. Same
>>> for some other my i386 installations having IPSEC tunnels.
>>
>> IPSEC definitely used to wwith with kstack_pages=2 since I ran that way
>> for a number of years.  I haven't used IPSEC since I upgraded from
>> FreeBSD 8.x to 10.x a while back, so it could be broken now.
> 
> I think this comes as a regression in 10.x or perhaps later.  So that
> atleast narrows down what has triggered the need for more kernel stack
> space.

Once again, that's not about IPSEC only that, indeed, had this kind of 
"regression"
with overhaul of its code between 11.0 and 11.1 releases with r315514.
It was already polished in stable/11 with later r319118 plus there is
https://reviews.freebsd.org/D9721 that introduces new sysctl 
net.inet.ipsec.use_netisr=1
to convert long path of direct function calls requiring large stack to
queuing of outgoing to-be-encrypted traffic using NETISR
at cost of some performance penalty when enabled.

But many other parts of kernel think it's OK to allocate big arrays or 
structures on stack.

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Rodney W. Grimes]

> On 12 Dec, Eugene Grosbein wrote:
> 
> > Now I run FreeBSD 11/i386 as my home router with IPSEC and torrent
> > client, and I run several virtualized routers with IPSEC tunnels,
> > jabber and mail server, squid and ZFS for src/obj/ports compression
> > and they all easily crash unless kern.kstack_pages raised upto 4. Same
> > for some other my i386 installations having IPSEC tunnels.
> 
> IPSEC definitely used to wwith with kstack_pages=2 since I ran that way
> for a number of years.  I haven't used IPSEC since I upgraded from
> FreeBSD 8.x to 10.x a while back, so it could be broken now.

I think this comes as a regression in 10.x or perhaps later.  So that
atleast narrows down what has triggered the need for more kernel stack
space.


-- 
Rod Grimes rgri...@freebsd.org
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

On 12.12.2017 06:00, Don Lewis wrote:
> On 12 Dec, Eugene Grosbein wrote:
> 
>> Now I run FreeBSD 11/i386 as my home router with IPSEC and torrent
>> client, and I run several virtualized routers with IPSEC tunnels,
>> jabber and mail server, squid and ZFS for src/obj/ports compression
>> and they all easily crash unless kern.kstack_pages raised upto 4. Same
>> for some other my i386 installations having IPSEC tunnels.
> 
> IPSEC definitely used to wwith with kstack_pages=2 since I ran that way
> for a number of years.  I haven't used IPSEC since I upgraded from
> FreeBSD 8.x to 10.x a while back, so it could be broken now.

That's all about FreeBSD 11 code base that growed more "fat" comparing with 8.x 
times.
It is not "broken", it just needs more stack :-)

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Don Lewis]

On 12 Dec, Eugene Grosbein wrote:

> Now I run FreeBSD 11/i386 as my home router with IPSEC and torrent
> client, and I run several virtualized routers with IPSEC tunnels,
> jabber and mail server, squid and ZFS for src/obj/ports compression
> and they all easily crash unless kern.kstack_pages raised upto 4. Same
> for some other my i386 installations having IPSEC tunnels.

IPSEC definitely used to wwith with kstack_pages=2 since I ran that way
for a number of years.  I haven't used IPSEC since I upgraded from
FreeBSD 8.x to 10.x a while back, so it could be broken now.

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

12.12.2017 2:12, Rodney W. Grimes wrote:

>>> Not sure about IPSEC and ZFS. But I was NOT able to reproduce the issue by 
>>> just using
>>> an SCTP association on a 32-bit VM using FreeBSD 11.1. So right now I don't 
>>> know what
>>> is wrong and therefore what could be fixed...
>>
>> You just got lucky. Kernel stack abuse by multiple parts of FreeBSD kernel 
>> it matter of fact.
>> But it is not like you always get a panic because sometimes network traffic
>> is passed through NETISR queues to reduce stack usage and sometimes it is 
>> not,
>> depending on several run-time factors. You may find it stable with one NIC 
>> driver
>> and crashing with another even just because of different stack usage.
>>
> 
> I agree with that, there is and has been a kernel stack space issue on
> i386 for some time.  But lets all agree just cranking the size of the
> kernel stack up is a poor long term solution as the larger you crank
> this the less number of processes you can have due to limited KVA.

OTOH, it seems to be totally unrealistic to hope the problem can be solved in 
feasible future
as such crashes were not fixed for several *years* (at least since 10.x era).

> So, first off, ZFS, well that is fairly well documented to not be
> a good idea on i386, and well documented to be a bad idea with only
> 4G of memory.

That's just incorrect. With 4 kstack pages, you could utilize ZFS
for compressed src/obj/ports with virtualized FreeBSD 9 and 1GB memory
using following loader.conf settings:

zfs_load="YES"
vfs.zfs.arc_max="40M"
vfs.zfs.vdev.cache.size="8M"
vfs.zfs.prefetch_disable="1"

And that was pretty stable as ARC was really limited.
This was broken in 10.x and later so that ARC grows order of magnitude (x10) 
larger, though.

>  So make sure that this stays documented, along with
> what one should tweak to make it work, most of which is already in
>   https://wiki.freebsd.org/ZFSTuningGuide#i386
> someone please verify that you really need to increase KSTACK, cause
> last time I played with ZFS on i386 I just used that tuning guide and
> all was happy.  If it is found necessary to add KSTACK pages, document
> that above and include in that document the side effects of doing so.
> 
> As far as IPSEC and SCTP, maybe someone can take a once over through
> the code looking for stack abuse with local variables that
> really should be malloced, passing large stuff in function call
> arguments rather than a pointer.
> 
> Possibly even doing some dtrace to find out how close we get to the limits.

That's not just ZFS, IPSEC or SCTP. It spreads all over kernel parts:
wifi subsystem and drivers, libkern/zlib, crypto, scsi/cam/da/ada drivers,
vm subsystem itself, NFS etc. 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219476


___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Rodney W. Grimes]

> 12.12.2017 1:52, Michael Tuexen wrote:
> 
> > Not sure about IPSEC and ZFS. But I was NOT able to reproduce the issue by 
> > just using
> > an SCTP association on a 32-bit VM using FreeBSD 11.1. So right now I don't 
> > know what
> > is wrong and therefore what could be fixed...
> 
> You just got lucky. Kernel stack abuse by multiple parts of FreeBSD kernel it 
> matter of fact.
> But it is not like you always get a panic because sometimes network traffic
> is passed through NETISR queues to reduce stack usage and sometimes it is not,
> depending on several run-time factors. You may find it stable with one NIC 
> driver
> and crashing with another even just because of different stack usage.
> 

I agree with that, there is and has been a kernel stack space issue on
i386 for some time.  But lets all agree just cranking the size of the
kernel stack up is a poor long term solution as the larger you crank
this the less number of processes you can have due to limited KVA.

So, first off, ZFS, well that is fairly well documented to not be
a good idea on i386, and well documented to be a bad idea with only
4G of memory.  So make sure that this stays documented, along with
what one should tweak to make it work, most of which is already in
https://wiki.freebsd.org/ZFSTuningGuide#i386
someone please verify that you really need to increase KSTACK, cause
last time I played with ZFS on i386 I just used that tuning guide and
all was happy.  If it is found necessary to add KSTACK pages, document
that above and include in that document the side effects of doing so.

As far as IPSEC and SCTP, maybe someone can take a once over through
the code looking for stack abuse with local variables that
really should be malloced, passing large stuff in function call
arguments rather than a pointer.

Possibly even doing some dtrace to find out how close we get to the limits.

-- 
Rod Grimes rgri...@freebsd.org
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Michael Tuexen]

> On 11. Dec 2017, at 17:21, Eugene Grosbein  wrote:
> 
> 11.12.2017 22:08, Konstantin Belousov пишет:
>> On Mon, Dec 11, 2017 at 06:03:36PM +0700, Eugene Grosbein wrote:
>>> I do not try to contradict other usage patterns. In fact, I'm eager to know
>>> a practical example of such pattern: a task, an application, anything real?
>> Plain workstation use, like X11+browser+editor+some other programs easily
>> allocates 1000+ threads.  It was still possible to use 32bit x86 for that,
>> of course in max memory config without PAE, and without ZFS.  Add some
>> load that involves network, for instance torrent client, to establish the
>> pressure on KVA.
>> 
>> I am almost sure that users would get troubles now.
> 
> Understood. While I'm sure that modern internet browsers make it unconfortable
> to browse with less than 4G total RAM (e.g. 2GB) available for the system 
> thus requiring amd64,
> I can imagine some old hardware without 64bit-enabled CPU used as workstation.
> 
> We should add Release Notes and UPDATING entry for such i386 users about
> setting "kern.kstack_pages=2" back in /boot/loader.conf (or 3?)
> and warn them against combined usage of ZFS/SCTP/IPSEC until these subsystems 
> get fixed
> to not abuse kernel stack.
Not sure about IPSEC and ZFS. But I was NOT able to reproduce the issue by just 
using
an SCTP association on a 32-bit VM using FreeBSD 11.1. So right now I don't 
know what
is wrong and therefore what could be fixed...

Best regards
Michael
> 
> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: svn commit: r326758 - in head/sys/i386: conf include

[Rodney W. Grimes]

...
> > 
> > We need to break the developers model that i386 is dead and that i386 is
> > not running on extremly modern hardware due to the factor of virtualization.
> > 
> > Output from one of my VM's running inside bhyve:
> > 
> >  # uname -a
> > FreeBSD filestore.dnsmgr.net 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: 
> > Fri Jul 21 04:10:47 UTC 2017 
> > r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  i386
> > 
> >  # dmesg | head -24
> > Copyright (c) 1992-2017 The FreeBSD Project.
> > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> > The Regents of the University of California. All rights reserved.
> > FreeBSD is a registered trademark of The FreeBSD Foundation.
> > FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017
> > r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
> > FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 
> > 4.0.0)
> > VT(vga): resolution 640x480
> > info: [drm] Initialized drm 1.1.0 20060810
> > CPU: Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz (2494.39-MHz K8-class CPU)
> This is quite old processor, IvyBridge is 6 generations old :).

I think you missed the point, this is a i386 guest running on an amd64
host inside of a hypervisor, bottom line, your gona see advanced amd64
features in i386 running in such situations, and we should probably stop
ignoreing this fact and embrace it as reality.

Another words you are going to see i386 running on the 6 generation newer
CPU that I dont have, and to say that this is unlikely is just a false
statement.

-- 
Rod Grimes rgri...@freebsd.org
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Rodney W. Grimes]

[ Charset UTF-8 unsupported, converting... ]
> 11.12.2017 23:45, Alexey Dokuchaev wrote:
> 
> >> Understood. While I'm sure that modern internet browsers make it
> >> uncomfortable to browse with less than 4G total RAM (e.g. 2GB) available
> >> for the system thus requiring amd64,
> > 
> > Browsing just fine on 2G RAM with Firefox, both under GNU/Linux and FreeBSD.
> > Where does this "uncomfortable to browse with less than 4G total RAM" false
> > narrative come from? :-/
> 
> My own experience. I use FreeBSD/amd64 as my desktop and workstation since 
> 1999
> and Firefox under FreeBSD always leaks memory like animal female.
> Some Firefox releases are little better than others but it's always greedy 
> memory hog.

I suspect your amd64 version of firefox leaks memory 1.8 times faster than
my i386 version of firefox???

Do you run any i386? 

They (mozilla) did finally wake up and realize it is a memory pig and did this 
quantum
thing, but in the process broke a lot of things in the process, should we call 
this
progrress by regression?

I browse quiet confortable in 3.25G on a 32Bit OS on even an old E5800, using 
the
pig that is FireFox 45.3.0 ESR.

-- 
Rod Grimes rgri...@freebsd.org
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

11.12.2017 23:45, Alexey Dokuchaev wrote:

>> Understood. While I'm sure that modern internet browsers make it
>> uncomfortable to browse with less than 4G total RAM (e.g. 2GB) available
>> for the system thus requiring amd64,
> 
> Browsing just fine on 2G RAM with Firefox, both under GNU/Linux and FreeBSD.
> Where does this "uncomfortable to browse with less than 4G total RAM" false
> narrative come from? :-/

My own experience. I use FreeBSD/amd64 as my desktop and workstation since 1999
and Firefox under FreeBSD always leaks memory like animal female.
Some Firefox releases are little better than others but it's always greedy 
memory hog.

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Alexey Dokuchaev]

On Mon, Dec 11, 2017 at 11:21:06PM +0700, Eugene Grosbein wrote:
> 11.12.2017 22:08, Konstantin Belousov пишет:
> > Plain workstation use, like X11+browser+editor+some other programs easily
> > allocates 1000+ threads.  It was still possible to use 32bit x86 for that,
> > of course in max memory config without PAE, and without ZFS.  Add some
> > load that involves network, for instance torrent client, to establish the
> > pressure on KVA.
> > 
> > I am almost sure that users would get troubles now.
> 
> Understood. While I'm sure that modern internet browsers make it
> uncomfortable to browse with less than 4G total RAM (e.g. 2GB) available
> for the system thus requiring amd64,

Browsing just fine on 2G RAM with Firefox, both under GNU/Linux and FreeBSD.
Where does this "uncomfortable to browse with less than 4G total RAM" false
narrative come from? :-/

./danfe
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

11.12.2017 22:09, Rodney W. Grimes write:

> THis is a mistake, there is a huge worled of i386 deployment, not all
> the world needs, or even wants amd64, especially in teh virtualization
> world when you are running anything with less than 4G of memory, which
> I would argue is a huge depolyement base.

It is very easy for FreeBSD11/i386 to crash due to overflow of kernel stack
no matter if it is virtualized or not. And that's very bad.
Please take a look at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219476 
for details.

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

11.12.2017 22:08, Konstantin Belousov пишет:
> On Mon, Dec 11, 2017 at 06:03:36PM +0700, Eugene Grosbein wrote:
>> I do not try to contradict other usage patterns. In fact, I'm eager to know
>> a practical example of such pattern: a task, an application, anything real?
> Plain workstation use, like X11+browser+editor+some other programs easily
> allocates 1000+ threads.  It was still possible to use 32bit x86 for that,
> of course in max memory config without PAE, and without ZFS.  Add some
> load that involves network, for instance torrent client, to establish the
> pressure on KVA.
> 
> I am almost sure that users would get troubles now.

Understood. While I'm sure that modern internet browsers make it unconfortable
to browse with less than 4G total RAM (e.g. 2GB) available for the system thus 
requiring amd64,
I can imagine some old hardware without 64bit-enabled CPU used as workstation.

We should add Release Notes and UPDATING entry for such i386 users about
setting "kern.kstack_pages=2" back in /boot/loader.conf (or 3?)
and warn them against combined usage of ZFS/SCTP/IPSEC until these subsystems 
get fixed
to not abuse kernel stack.


___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Konstantin Belousov]

On Mon, Dec 11, 2017 at 07:33:08AM -0800, Rodney W. Grimes wrote:
> > On Mon, Dec 11, 2017 at 07:09:10AM -0800, Rodney W. Grimes wrote:
> > > The current comment about a pcb, I thought that code was changed
> > > so we only put the pointer to a pcb on the stack.
> > 
> > pcb is on top of the stack, followed by the userspace FPU registers save
> > area.  I do not see any sense in existence of pcb in modern kernel, it is
> > a remnant of the user area that was swappable.  Currently pcb is swappable
> > as well, but the value of this is much less then the overhead we pay by
> > the stack space reduction.
> 
> How about KSTACK_PAGES should be KSTAKE_BYTES/PAGESIZE, and we
> need a much better formula for define of KSTACK_BYTES so that
> these facts are more accurately defined?
The FPU save area size depends on CPU we booted on, so if we increase
(or decrease) it statically, we make a decision that is not good for
one or another generation of processors.

> 
> And a compile time assert if this ever growes to something unreasonable
> that would cause other issues.
So far it is an overkill, Intel did not passed the page boundary with all
extensions defined.

> 
> I fully agree with you that just bumping KSTACK_PAGES is very much
> the wrong way to fix the i386 issue of certain code not running,
> it is that code that should be examined for over usage of the stack.
In fact there is a tunable which I added to avoid recompiling kernel if
specific load requires large stack, keeping ZFS in mind.

> 
> I can assert that the base i386 system is very usable for tons of
> things without this change, I have at least 30 VM's running FreeBSD
> 11.1/i386 in some very small footprints, typically 64MB, that have
> zero issues.  But then they are not using any of the code that
> sited as problem areas.
> 
> > FPU save area is the on of the problem which makes us increase the amd64
> > stack size, AVX or even AVX512 make the things much worse.  It is unlikely
> > that somebody would run 32bit kernel on machine capable of that extensions,
> > i.e. Haswell or Skylake.
> 
> Your igonoring the virutalization world, host is a skylake or haswell, guest
> is i386 as it has small memory needs and no use to waste half of all pointers.
If you are serious about avoiding the waste, then you would also disable
unused extensions eating save area space, e.g. AVX512.  AVX might be still
useful (but not required) if AESNI or SHA extensions are used.

> 
> We need to break the developers model that i386 is dead and that i386 is
> not running on extremly modern hardware due to the factor of virtualization.
> 
> Output from one of my VM's running inside bhyve:
> 
>  # uname -a
> FreeBSD filestore.dnsmgr.net 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: 
> Fri Jul 21 04:10:47 UTC 2017 
> r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  i386
> 
>  # dmesg | head -24
> Copyright (c) 1992-2017 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017
> r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
> FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 
> 4.0.0)
> VT(vga): resolution 640x480
> info: [drm] Initialized drm 1.1.0 20060810
> CPU: Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz (2494.39-MHz K8-class CPU)
This is quite old processor, IvyBridge is 6 generations old :).
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Alexey Dokuchaev]

On Mon, Dec 11, 2017 at 07:09:10AM -0800, Rodney W. Grimes wrote:
> [ Charset UTF-8 unsupported, converting... ]
> > New Revision: 326758
> > URL: https://svnweb.freebsd.org/changeset/base/326758
> > 
> > Log:
> >   i386: Bump KSTACK_PAGES default to match amd64
> >   
> >   Logically, extend r286288 to cover all threads, by default.
> >   
> >   The world has largely moved on from i386.  Most FreeBSD users and
> >   developers ...
> 
> This is a mistake, there is a huge world of i386 deployment, not all
> the world needs, or even wants amd64, especially in the virtualization
> world when you are running anything with less than 4G of memory, which
> I would argue is a huge depolyement base.
> 
> 64-bit code is simply wasteful on these platforms, and though maybe
> software developers are ok with that waste, (smart) business people
> are not.

Big +1.  Most of my hw is 32-bit and I like it that way.  It works
fine, it helps to maintains/check sw sanity, and remains our default
reference platform.

> >   ... test on amd64 hardware.

Then they should start doing a better job and test on i386 (at least)
as well.

> For better or worse, we have written a non-trivial amount of kernel
> code that relies on stacks larger than 8 kB, and it "just works" on
> amd64, so there has been little incentive to shrink it.

If the code is broken, it should be fixed.  "Just works on amd64" is
not a good enough excuse to deprecate^Wput quick "fixes" for i386 that
did not undergo sufficient review.

./danfe
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Rodney W. Grimes]

> On Mon, Dec 11, 2017 at 07:09:10AM -0800, Rodney W. Grimes wrote:
> > The current comment about a pcb, I thought that code was changed
> > so we only put the pointer to a pcb on the stack.
> 
> pcb is on top of the stack, followed by the userspace FPU registers save
> area.  I do not see any sense in existence of pcb in modern kernel, it is
> a remnant of the user area that was swappable.  Currently pcb is swappable
> as well, but the value of this is much less then the overhead we pay by
> the stack space reduction.

How about KSTACK_PAGES should be KSTAKE_BYTES/PAGESIZE, and we
need a much better formula for define of KSTACK_BYTES so that
these facts are more accurately defined?

And a compile time assert if this ever growes to something unreasonable
that would cause other issues.

I fully agree with you that just bumping KSTACK_PAGES is very much
the wrong way to fix the i386 issue of certain code not running,
it is that code that should be examined for over usage of the stack.

I can assert that the base i386 system is very usable for tons of
things without this change, I have at least 30 VM's running FreeBSD
11.1/i386 in some very small footprints, typically 64MB, that have
zero issues.  But then they are not using any of the code that
sited as problem areas.

> FPU save area is the on of the problem which makes us increase the amd64
> stack size, AVX or even AVX512 make the things much worse.  It is unlikely
> that somebody would run 32bit kernel on machine capable of that extensions,
> i.e. Haswell or Skylake.

Your igonoring the virutalization world, host is a skylake or haswell, guest
is i386 as it has small memory needs and no use to waste half of all pointers.

We need to break the developers model that i386 is dead and that i386 is
not running on extremly modern hardware due to the factor of virtualization.

Output from one of my VM's running inside bhyve:

 # uname -a
FreeBSD filestore.dnsmgr.net 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri 
Jul 21 04:10:47 UTC 2017 
r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  i386

 # dmesg | head -24
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017
r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 
4.0.0)
VT(vga): resolution 640x480
info: [drm] Initialized drm 1.1.0 20060810
CPU: Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz (2494.39-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306a9  Family=0x6  Model=0x3a  Stepping=9
  
Features=0xbfebfbff
  
Features2=0x7fbae3bf
  AMD Features=0x28100800
  AMD Features2=0x1
  Structured Extended Features=0x281
  XSAVE Features=0x1
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 12884901888 (12288 MB)
avail memory = 12114096128 (11552 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: 
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs

-- 
Rod Grimes rgri...@freebsd.org
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Konstantin Belousov]

On Mon, Dec 11, 2017 at 07:09:10AM -0800, Rodney W. Grimes wrote:
> The current comment about a pcb, I thought that code was changed
> so we only put the pointer to a pcb on the stack.

pcb is on top of the stack, followed by the userspace FPU registers save
area.  I do not see any sense in existence of pcb in modern kernel, it is
a remnant of the user area that was swappable.  Currently pcb is swappable
as well, but the value of this is much less then the overhead we pay by
the stack space reduction.

FPU save area is the on of the problem which makes us increase the amd64
stack size, AVX or even AVX512 make the things much worse.  It is unlikely
that somebody would run 32bit kernel on machine capable of that extensions,
i.e. Haswell or Skylake.
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Rodney W. Grimes]

[ Charset UTF-8 unsupported, converting... ]
> Author: cem
> Date: Mon Dec 11 04:32:37 2017
> New Revision: 326758
> URL: https://svnweb.freebsd.org/changeset/base/326758
> 
> Log:
>   i386: Bump KSTACK_PAGES default to match amd64
>   
>   Logically, extend r286288 to cover all threads, by default.
>   
>   The world has largely moved on from i386.  Most FreeBSD users and developers

THis is a mistake, there is a huge worled of i386 deployment, not all
the world needs, or even wants amd64, especially in teh virtualization
world when you are running anything with less than 4G of memory, which
I would argue is a huge depolyement base.

64bit code is simply wasteful on these platforms, and though maybe
software developers are ok with that waste, (smart) business people
are not.

>   test on amd64 hardware.  For better or worse, we have written a non-trivial
>   amount of kernel code that relies on stacks larger than 8 kB, and it "just
>   works" on amd64, so there has been little incentive to shrink it.
>   
>   amd64 had its KSTACK_PAGES bumped to 4 back in Peter's initial AMD64 commit,
>   r114349, in 2003.  Since that time, i386 has limped along on a stack half
>   the size.  We've even observed the stack overflows years ago, but neglected
>   to fix the issue; see the 20121223 and 20150728 entries in UPDATING.

This makes perfect sense to have sizeofstackamd64 == 2 * sizofstacki386 if
you think about it for very long, most of what should be on your stack are
pointers and ints, the sizedof pointers/int is 2x on amd64.   If we are
storing lots of none pointer stuff that is very large on the stack something
is probably designed wrong.

Does it makes since to change the KSTACK_PAGES value to be more MI
in how it is defined (I am not saying move it to MI code) so a
better understanding can be had, as in:

KSTACK_PAGES=(sizeof(void *)/2)

The comment that should go with this is left as an excersize for
the commiter.

The current comment about a pcb, I thought that code was changed
so we only put the pointer to a pcb on the stack.

>   
>   If anyone is concerned with this change, I suggest they configure their
>   AMD64 kernels with KSTACK_PAGES 2 and fix the fallout there first.  Eugene
>   has identified a list of high stack usage functions in the first PR below.

Um, that is the wrong approach for the reason stated above, you would end
up pessimizing on amd64.  Thought getting amd64 to run in 2 pages of stack
would defanitly allow the i386 to run in 2 pages.


>   PR: 219476, 224218
>   Reported by:eugen@, Shreesh Holla 
>   Relnotes:   maybe
>   Sponsored by:   Dell EMC Isilon
> 
> Modified:
>   head/sys/i386/conf/NOTES
>   head/sys/i386/include/param.h
> 
> Modified: head/sys/i386/conf/NOTES
> ==
> --- head/sys/i386/conf/NOTES  Mon Dec 11 02:44:15 2017(r326757)
> +++ head/sys/i386/conf/NOTES  Mon Dec 11 04:32:37 2017(r326758)
> @@ -968,7 +968,7 @@ devicendis
>  # KSTACK_PAGES is the number of memory pages to assign to the kernel
>  # stack of each thread.
>  
> -options  KSTACK_PAGES=3
> +options  KSTACK_PAGES=5
>  
>  # Enable detailed accounting by the PV entry allocator.
>  
> 
> Modified: head/sys/i386/include/param.h
> ==
> --- head/sys/i386/include/param.h Mon Dec 11 02:44:15 2017
> (r326757)
> +++ head/sys/i386/include/param.h Mon Dec 11 04:32:37 2017
> (r326758)
> @@ -113,7 +113,7 @@
>  #define IOPAGES  2   /* pages of i/o permission bitmap */
>  
>  #ifndef KSTACK_PAGES
> -#define KSTACK_PAGES 2   /* Includes pcb! */
> +#define KSTACK_PAGES 4   /* Includes pcb! */
>  #endif
>  #define KSTACK_GUARD_PAGES 1 /* pages of kstack guard; 0 disables */
>  #if KSTACK_PAGES < 4
> 
> 

-- 
Rod Grimes rgri...@freebsd.org
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Konstantin Belousov]

On Mon, Dec 11, 2017 at 06:03:36PM +0700, Eugene Grosbein wrote:
> I do not try to contradict other usage patterns. In fact, I'm eager to know
> a practical example of such pattern: a task, an application, anything real?
Plain workstation use, like X11+browser+editor+some other programs easily
allocates 1000+ threads.  It was still possible to use 32bit x86 for that,
of course in max memory config without PAE, and without ZFS.  Add some
load that involves network, for instance torrent client, to establish the
pressure on KVA.

I am almost sure that users would get troubles now.

> 
> I already know how to bring FreeBSD down to its kneels using stress tests
> but that's not what I'm looking for in this case of kstack_pages.
> 
Stress test is useful due to its canary nature.
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

On 11.12.2017 17:52, Konstantin Belousov wrote:

>> I still wonder if there is really such load pattern that creates "enough 
>> threads"
>> for i386 to make 4-pages stack troublesome.
> Yes, there is such load pattern, it is when you do create threads. Your
> load, as described, is static. Peter' stress2 includes some tests which
> will highlight the change.
> 
> I am quite impressed by your ability to make generalization from single data
> point.  Moving issues around because you care about your load, and do not
> care about other usage patterns, is certainly the way to go.

I do not try to contradict other usage patterns. In fact, I'm eager to know
a practical example of such pattern: a task, an application, anything real?

I already know how to bring FreeBSD down to its kneels using stress tests
but that's not what I'm looking for in this case of kstack_pages.


___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Konstantin Belousov]

On Mon, Dec 11, 2017 at 05:26:12PM +0700, Eugene Grosbein wrote:
> On 11.12.2017 16:19, Konstantin Belousov wrote:
> > On Mon, Dec 11, 2017 at 04:32:37AM +, Conrad Meyer wrote:
> >> Author: cem
> >> Date: Mon Dec 11 04:32:37 2017
> >> New Revision: 326758
> >> URL: https://svnweb.freebsd.org/changeset/base/326758
> >>
> >> Log:
> >>   i386: Bump KSTACK_PAGES default to match amd64
> > i386 is not amd64, the change is wrong.
> > 
> > i386 has the word size two times smaller than amd64, which makes typical
> > frame smaller by 30-40% over same code on amd64. Also i386 has much
> > smaller available KVA size (tens of MB) and KVA fragmentation is both
> > more severe and more fatal due to this. I expect that your change will
> > make any non-trivial load which creates enough threads to either fail
> > randomly or deadlock.
> > 
> > If somebody tries to fit large load onto i386 machine, he must know what to
> > do and how to configure the kernel to adapt to the load (which does not
> > require the recompilation).
> 
> Its very easy to get kernel stack overflow with 11-STABLE/i386
> without any significant load due to abuse of kernel stack in many kernel 
> subsystems
> as shown in the https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219476
> 
> Contrary, "enough threads" seems to be very non-trivial number of threads
> and pretty unusual load pattern for i386 as I run several such systems
> with kern.kstack_pages=4 for quite long time and have no problems.
> No random fails, no deadlocks. And with 2 pages only 11-STABLE/i386 is just 
> unusable
> if one utilizes SCTP, IPv6, some WiFi connectivity, IPSEC or even very small 
> ZFS pool.
> 
> I still wonder if there is really such load pattern that creates "enough 
> threads"
> for i386 to make 4-pages stack troublesome.
Yes, there is such load pattern, it is when you do create threads. Your
load, as described, is static. Peter' stress2 includes some tests which
will highlight the change.

I am quite impressed by your ability to make generalization from single data
point.  Moving issues around because you care about your load, and do not
care about other usage patterns, is certainly the way to go.
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Eugene Grosbein]

On 11.12.2017 16:19, Konstantin Belousov wrote:
> On Mon, Dec 11, 2017 at 04:32:37AM +, Conrad Meyer wrote:
>> Author: cem
>> Date: Mon Dec 11 04:32:37 2017
>> New Revision: 326758
>> URL: https://svnweb.freebsd.org/changeset/base/326758
>>
>> Log:
>>   i386: Bump KSTACK_PAGES default to match amd64
> i386 is not amd64, the change is wrong.
> 
> i386 has the word size two times smaller than amd64, which makes typical
> frame smaller by 30-40% over same code on amd64. Also i386 has much
> smaller available KVA size (tens of MB) and KVA fragmentation is both
> more severe and more fatal due to this. I expect that your change will
> make any non-trivial load which creates enough threads to either fail
> randomly or deadlock.
> 
> If somebody tries to fit large load onto i386 machine, he must know what to
> do and how to configure the kernel to adapt to the load (which does not
> require the recompilation).

Its very easy to get kernel stack overflow with 11-STABLE/i386
without any significant load due to abuse of kernel stack in many kernel 
subsystems
as shown in the https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219476

Contrary, "enough threads" seems to be very non-trivial number of threads
and pretty unusual load pattern for i386 as I run several such systems
with kern.kstack_pages=4 for quite long time and have no problems.
No random fails, no deadlocks. And with 2 pages only 11-STABLE/i386 is just 
unusable
if one utilizes SCTP, IPv6, some WiFi connectivity, IPSEC or even very small 
ZFS pool.

I still wonder if there is really such load pattern that creates "enough 
threads"
for i386 to make 4-pages stack troublesome.

Eugene Grosbein

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r326758 - in head/sys/i386: conf include

[Konstantin Belousov]

On Mon, Dec 11, 2017 at 04:32:37AM +, Conrad Meyer wrote:
> Author: cem
> Date: Mon Dec 11 04:32:37 2017
> New Revision: 326758
> URL: https://svnweb.freebsd.org/changeset/base/326758
> 
> Log:
>   i386: Bump KSTACK_PAGES default to match amd64
i386 is not amd64, the change is wrong.

i386 has the word size two times smaller than amd64, which makes typical
frame smaller by 30-40% over same code on amd64. Also i386 has much
smaller available KVA size (tens of MB) and KVA fragmentation is both
more severe and more fatal due to this. I expect that your change will
make any non-trivial load which creates enough threads to either fail
randomly or deadlock.

If somebody tries to fit large load onto i386 machine, he must know what to
do and how to configure the kernel to adapt to the load (which does not
require the recompilation).
>   
>   Logically, extend r286288 to cover all threads, by default.
>   
>   The world has largely moved on from i386.  Most FreeBSD users and developers
>   test on amd64 hardware.  For better or worse, we have written a non-trivial
>   amount of kernel code that relies on stacks larger than 8 kB, and it "just
>   works" on amd64, so there has been little incentive to shrink it.
>   
>   amd64 had its KSTACK_PAGES bumped to 4 back in Peter's initial AMD64 commit,
>   r114349, in 2003.  Since that time, i386 has limped along on a stack half
>   the size.  We've even observed the stack overflows years ago, but neglected
>   to fix the issue; see the 20121223 and 20150728 entries in UPDATING.
>   
>   If anyone is concerned with this change, I suggest they configure their
>   AMD64 kernels with KSTACK_PAGES 2 and fix the fallout there first.  Eugene
>   has identified a list of high stack usage functions in the first PR below.
This suggestion is completely bogus.

>   
>   PR: 219476, 224218
>   Reported by:eugen@, Shreesh Holla 
>   Relnotes:   maybe
>   Sponsored by:   Dell EMC Isilon
> 
> Modified:
>   head/sys/i386/conf/NOTES
>   head/sys/i386/include/param.h
> 
> Modified: head/sys/i386/conf/NOTES
> ==
> --- head/sys/i386/conf/NOTES  Mon Dec 11 02:44:15 2017(r326757)
> +++ head/sys/i386/conf/NOTES  Mon Dec 11 04:32:37 2017(r326758)
> @@ -968,7 +968,7 @@ devicendis
>  # KSTACK_PAGES is the number of memory pages to assign to the kernel
>  # stack of each thread.
>  
> -options  KSTACK_PAGES=3
> +options  KSTACK_PAGES=5
>  
>  # Enable detailed accounting by the PV entry allocator.
>  
> 
> Modified: head/sys/i386/include/param.h
> ==
> --- head/sys/i386/include/param.h Mon Dec 11 02:44:15 2017
> (r326757)
> +++ head/sys/i386/include/param.h Mon Dec 11 04:32:37 2017
> (r326758)
> @@ -113,7 +113,7 @@
>  #define IOPAGES  2   /* pages of i/o permission bitmap */
>  
>  #ifndef KSTACK_PAGES
> -#define KSTACK_PAGES 2   /* Includes pcb! */
> +#define KSTACK_PAGES 4   /* Includes pcb! */
>  #endif
>  #define KSTACK_GUARD_PAGES 1 /* pages of kstack guard; 0 disables */
>  #if KSTACK_PAGES < 4
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"