Re: CVS commit: src/sys
Please pull up to netbsd-8 and netbsd-9 release branches. > On Nov 11, 2020, at 23:44, Simon Burge wrote: > > Module Name: src > Committed By: simonb > Date: Thu Nov 12 07:44:01 UTC 2020 > > Modified Files: > src/sys/conf: param.c > src/sys/kern: init_main.c > src/sys/sys: param.h > > Log Message: > Set a better default for MAXFILES on larger RAM machines if not > otherwise specified the kernel config file. Arbitary numbers are > 20,000 files for 16GB RAM or more and 10,000 files for 1GB RAM or > more. > > TODO: Adjust this and other values totally dynamically. > > > To generate a diff of this commit: > cvs rdiff -u -r1.68 -r1.69 src/sys/conf/param.c > cvs rdiff -u -r1.532 -r1.533 src/sys/kern/init_main.c > cvs rdiff -u -r1.678 -r1.679 src/sys/sys/param.h > > Please note that diffs are not public domain; they are subject to the > copyright notices on the relevant files. >
Re: CVS commit: src
On Sun, Nov 08, 2020 at 09:56:48PM +, Nia Alarie wrote: > Module Name: src > Committed By: nia > Date: Sun Nov 8 21:56:48 UTC 2020 > > Modified Files: > src/external/bsd/kyua-cli: Makefile.inc > src/external/ibm-public/postfix: Makefile.inc > src/external/public-domain/sqlite: Makefile.inc > src/external/public-domain/sqlite/bin: Makefile > src/external/public-domain/sqlite/lib: Makefile sqlite3.pc.in > src/usr.sbin/makemandb: Makefile > > Log Message: > sqlite: do not build without multithreading support So the core issue here is that it actually does two separate things. It does not only enable threadsafety (which is good), but also enables the worker thread support (which is bad). There is little reason for wanting the latter in a general build, at least the way it is implemented right now. Just doing the former is a lot less intrusive... Joerg diff -r 2bb2635be785 external/public-domain/sqlite/Makefile.inc --- a/external/public-domain/sqlite/Makefile.incThu Nov 12 23:24:18 2020 +0100 +++ b/external/public-domain/sqlite/Makefile.incThu Nov 12 23:28:36 2020 +0100 @@ -15,6 +15,7 @@ -DHAVE_STRERROR_R=1 \ -DHAVE_USLEEP=1 \ -DHAVE_SYS_ENDIAN_H=1 \ + -DSQLITE_THREADSAFE \ -DSQLITE_MAX_WORKER_THREADS=0 \ -DSQLITE_ENABLE_COLUMN_METADATA \ -DSQLITE_ENABLE_FTS3_PARENTHESIS \
Re: CVS commit: src
On 2020/11/13 2:35, nia wrote: I'll revert the commit shortly. Thank you very much for your quick response. I wasn't expecting such major breakage (obviously) and actually did run a build but ran into different problems with libstdc++. Yeah, this is why we need tests, and also listen to others before making somewhat big changes. No one is perfect... Thanks, rin
Re: CVS commit: src
On Thu, Nov 12, 2020 at 07:58:13PM +, Taylor R Campbell wrote: > But there's a snag with heimdal. > > Heimdal exposes the sqlite3 library to clients via libgssapi.so which > links against libkrb5.so which brings in libsqlite3.so. So we get nice > situations like this: > > % ldd /pkg/2020Q2/bin/svn | grep sqlite > -lsqlite3.0 => /pkg/2020Q2/lib/libsqlite3.so.0 > -lsqlite3.1 => /usr/lib/libsqlite3.so.1 > > This is also why the change to base sqlite3 affected, e.g., su(1). How about making sqlite3 just a private lib statically linked into whatever part of heimdahl actually needs it, and not exposing any symbols from it? Martin
Re: CVS commit: src
> Date: Thu, 12 Nov 2020 11:21:43 -0800 > From: Jason Thorpe > > > On Nov 12, 2020, at 9:40 AM, nia wrote: > > > > For the record I'm just trying to fix things so that running > > third-party software on NetBSD sucks less. If fewer third-party > > libraries were exposed by the base system this wouldn't be > > necessary. > > Why do we even have sqlite in the base system in the first place? apropos(1); a handful of other things also use it (heimdal, postfix). I suggested that we leave the bells & whistles out of base sqlite3, maybe even stop installing the .so symlink, and teach pkgsrc to just use pkgsrc sqlite3 (with all bells & whistles included) rather than base sqlite3 -- after all, sqlite3 is meant to be embedded and has lots of optional bells & whistles that apropos(1) has no need of, so it is silly to impose those costs on apropos(1) or make sqlite3 part of the NetBSD API. But there's a snag with heimdal. Heimdal exposes the sqlite3 library to clients via libgssapi.so which links against libkrb5.so which brings in libsqlite3.so. So we get nice situations like this: % ldd /pkg/2020Q2/bin/svn | grep sqlite -lsqlite3.0 => /pkg/2020Q2/lib/libsqlite3.so.0 -lsqlite3.1 => /usr/lib/libsqlite3.so.1 This is also why the change to base sqlite3 affected, e.g., su(1). I have no idea what heimdal does with sqlite3, but it puts us in a pretty pickle: we can't reliably use pkgsrc sqlite3 on existing NetBSD installs for packages that use gssapi, and we can't rely on sqlite3 in existing NetBSD installs for packages that need newer/threaded/jsonned sqlite3 as we often encounter.
Re: CVS commit: src
> On Nov 12, 2020, at 9:40 AM, nia wrote: > > For the record I'm just trying to fix things so that running > third-party software on NetBSD sucks less. If fewer third-party > libraries were exposed by the base system this wouldn't be > necessary. Why do we even have sqlite in the base system in the first place? -- thorpej
Re: CVS commit: src
On Thu, Nov 12, 2020 at 05:35:44PM +, nia wrote: > I'll revert the commit shortly. > > I wasn't expecting such major breakage (obviously) and actually > did run a build but ran into different problems with libstdc++. For the record I'm just trying to fix things so that running third-party software on NetBSD sucks less. If fewer third-party libraries were exposed by the base system this wouldn't be necessary.
Re: CVS commit: src
I'll revert the commit shortly. I wasn't expecting such major breakage (obviously) and actually did run a build but ran into different problems with libstdc++.
Re: CVS commit: src
On 2020/11/12 23:03, Rin Okuyama wrote: The backtrace reads: | (gdb) bt | #0 0xfc2961bca308 in ?? () | #1 0xfc2961b9ec10 in __deregister_frame_info_bases () | from /usr/lib/libgcc_s.so.1 | #2 0xf88425b4 in _rtld_call_function_void (obj=0xfc2962917400, | addr=277254663434168) at /usr/src/libexec/ld.elf_so/rtld.h:514 | ... The fault address is anon memory w/o exec permission when su exits: | # pmap -p xxx | ... | FC2961BB1000 4K read/write /altroot/lib/libgcc_s.so.1.0 | FC2961BB6000 40K read/write [ anon ] | FC2961BE6000 40K read/write [ anon ] | ... Oops, correction: the fault address is not mapped when su exits; FC2961BB6000 + 40K = FC2961BC. The discussion thereafter does not change. Thanks, rin
Re: CVS commit: src
On 2020/11/11 1:50, nia wrote: On Tue, Nov 10, 2020 at 04:29:21PM +, Taylor R Campbell wrote: Module Name:src Committed By: nia Date: Sun Nov 8 21:56:48 UTC 2020 Modified Files: src/external/bsd/kyua-cli: Makefile.inc src/external/ibm-public/postfix: Makefile.inc src/external/public-domain/sqlite: Makefile.inc src/external/public-domain/sqlite/bin: Makefile src/external/public-domain/sqlite/lib: Makefile sqlite3.pc.in src/usr.sbin/makemandb: Makefile Log Message: sqlite: do not build without multithreading support at least a few pkgsrc packages avoid base sqlite because it fails this check, and it's probably a surprising performance penalty for unsuspecting users Why do we even expose NetBSD's libsqlite3.so to pkgsrc? Why not just have pkgsrc always use pkgsrc sqlite3, and change base to go back to the smaller non-threaded sqlite3 with only the options the things in base actually need? Also: What is the performance penalty? The sqlite3 documentation (https://www.sqlite.org/faq.html#q6) suggests that _enabling_ SQLITE_THREADSAFE has a performance penalty, not the other way around. It seems to me we've had a lot of problems over the years trying to use base sqlite3 for everything in pkgsrc. It's not clear to me that any of the recent changes are helpful for the three things in base that need sqlite3, and it's also not clear that they will help with running newer pkgsrc on older NetBSD. libevent is also problematic. Maybe there's just too many third-party libraries that are exposed that sholdn't be. A failure for cvs, suspected to be a fallout of threaded sqlite3 was reported in port-arm@. See thread starts with: http://mail-index.netbsd.org/port-arm/2020/11/12/msg007046.html Also, I encountered another fallout: ``su -'' receives SIGSEGV when exits (on aarch64, but I bet this is MI problem): | $ su | # exit | $ su - | # exit | [2] Segmentation fault (core dumped) su - The backtrace reads: | (gdb) bt | #0 0xfc2961bca308 in ?? () | #1 0xfc2961b9ec10 in __deregister_frame_info_bases () |from /usr/lib/libgcc_s.so.1 | #2 0xf88425b4 in _rtld_call_function_void (obj=0xfc2962917400, | addr=277254663434168) at /usr/src/libexec/ld.elf_so/rtld.h:514 | ... The fault address is anon memory w/o exec permission when su exits: | # pmap -p xxx | ... | FC2961BB1000 4K read/write/altroot/lib/libgcc_s.so.1.0 | FC2961BB6000 40K read/write [ anon ] | FC2961BE6000 40K read/write [ anon ] | ... However, when su is running, libpthread is loaded at this address: | # pmap -p xxx | ... | FC2961BB1000 4K read/write/altroot/lib/libgcc_s.so.1.0 | FC2961BB6000 40K read/write [ anon ] | FC2961BC 76K read/exec /altroot/lib/libpthread.so.1.4 | FC2961BD3000 64K /altroot/lib/libpthread.so.1.4 | ... su is not explicitly linked to libpthread, but it is loaded via libkrb5 via libpam. It registers its own deconstructor, however, it is unloaded before su exits. Then, SIGSEGV takes place as a result. IMO, this commit should be reverted ASAP. This kind of change needs more tests, and discussion in appropriate lists before commit. Unfortunately, the committer did not even carry out ``build.sh'' before commit (martin fixed the build failure due to this). This is not what we, at least, I expect our developers. Thanks, rin
