Re: numpy failure on netbsd-9, rintl

[Greg Troxel]

It looks like this is more ok than I thought.  I don't find anything
different from 10 to current.   Looking at 9 to current and pruning diff
noise, there are a number of additional aliases in 10.

It seems that there are two strategies for this.Can we pick one
and document it?  It seems like the choices are:

1) ldbl_dummy approach

In ldbl_dummy.c, define a weak_alias from fool to _fool
In namespace.h, define fool to _fool, but only for some

It's not clear how these interact, as namespace.h is included first.


2) per-function alias approach

e.g See s_modf.c,

#ifndef __HAVE_LONG_DOUBLE
__strong_alias(_modfl, modf)
__weak_alias(modfl, modf)
#endif




Index: src/ldbl_dummy.c
===
RCS file: /cvsroot/src/lib/libm/src/ldbl_dummy.c,v
retrieving revision 1.2
retrieving revision 1.2.26.1
diff -u -p -r1.2 -r1.2.26.1
--- src/ldbl_dummy.c13 Nov 2014 21:43:27 -  1.2
+++ src/ldbl_dummy.c11 Aug 2023 14:44:19 -  1.2.26.1
@@ -43,8 +43,11 @@ __weak_alias(atan2l, _atan2l)
 __weak_alias(hypotl, _hypotl)
 __weak_alias(logl, _logl)
 __weak_alias(log10l, _log10l)
+__weak_alias(log2l, _log2l)
+__weak_alias(log1pl, _log1pl)
 __weak_alias(expl, _expl)
 __weak_alias(exp2l, _exp2l)
+__weak_alias(expm1l, _expm1l)
 __weak_alias(powl, _powl)
 __weak_alias(cosl, _cosl)
 __weak_alias(sinl, _sinl)
@@ -60,6 +63,8 @@ __weak_alias(asinhl, _asinhl)
 __weak_alias(atanhl, _atanhl)
 __weak_alias(erfl, _erfl)
 __weak_alias(erfcl, _erfcl)
+__weak_alias(lgammal, _lgammal)
+__weak_alias(tgammal, _tgammal)
 
 long double
 atan2l(long double y, long double x)
@@ -86,6 +91,18 @@ log10l(long double x)
 }
 
 long double
+log2l(long double x)
+{
+   return log2(x);
+}
+
+long double
+log1pl(long double x)
+{
+   return log1p(x);
+}
+
+long double
 expl(long double x)
 {
return exp(x);
@@ -98,6 +115,12 @@ exp2l(long double x)
 }
 
 long double
+expm1l(long double x)
+{
+   return expm1(x);
+}
+
+long double
 powl(long double x, long double y)
 {
return pow(x, y);
@@ -187,3 +210,27 @@ erfcl(long double x)
 {
return erfc(x);
 }
+
+long double
+lgammal(long double x)
+{
+   return lgamma(x);
+}
+
+long double
+tgammal(long double x)
+{
+   return tgamma(x);
+}
+
+long double
+remainderl(long double x, long double y)
+{
+   return remainder(x, y);
+}
+
+long double
+remquol(long double x, long double y, int *quo)
+{
+   return remquo(x, y, quo);
+}
Index: src/namespace.h
===
RCS file: /cvsroot/src/lib/libm/src/namespace.h,v
retrieving revision 1.14
retrieving revision 1.16.2.1
diff -u -p -r1.14 -r1.16.2.1
--- src/namespace.h 22 Mar 2017 23:11:09 -  1.14
+++ src/namespace.h 11 Aug 2023 14:44:19 -  1.16.2.1
@@ -10,6 +10,7 @@
 #define exp _exp
 #define expf _expf
 #define expl _expl
+#define expm1l _expm1l
 #define log _log
 #define logf _logf
 #define logl _logl
@@ -22,6 +23,9 @@
 #define finite _finite
 #define finitef _finitef
 #endif /* notyet */
+#define sincos _sincos
+#define sincosf _sincosf
+#define sincosl _sincosl
 #define sinh _sinh
 #define sinhf _sinhf
 #define sinhl _sinhl
@@ -70,15 +74,17 @@
 #define tanhl _tanhl
 #define atanhl _atanhl
 #define log10l _log10l
+#define log1pl _log1pl
+#define log2l _log2l
 
-#define erfl   _erfl
-#define erfcl  _erfcl
+#define erfl _erfl
+#define erfcl _erfcl
+
+#define lgammal _lgammal
+#define tgammal _tgammal
 
 #define feclearexcept _feclearexcept
-#define fedisableexcept _fedisableexcept
-#define feenableexcept _feenableexcept
 #define fegetenv _fegetenv
-#define fegetexcept _fegetexcept
 #define fegetexceptflag _fegetexceptflag
 #define fegetround _fegetround
 #define feholdexcept _feholdexcept
@@ -88,3 +94,7 @@
 #define fesetround _fesetround
 #define fetestexcept _fetestexcept
 #define feupdateenv _feupdateenv
+
+#define fedisableexcept _fedisableexcept
+#define feenableexcept _feenableexcept
+#define fegetexcept _fegetexcept

Re: numpy failure on netbsd-9, rintl

[Greg Troxel]

I extracted defined symbols ending in l from libm.so.


Diffing from netbsd-9 amd64 to netbsd-9 earmv7hf-el

-__ieee754_sqrtl
+_fmal
+_frexpl
+_rintl
-fabsl
-nearbyintl
-nexttowardl


I suspect the + is my local changes and strong vs weak aliases.

numpy failure on netbsd-9, rintl

[Greg Troxel]

As background, I am using numpy, not really intentionally, but because
homeassistant depends on it indirectly, as does gpsd via py-matplotlib.

I have a RPI3, netbsd-9, and pkgsrc 2023Q3.  Importing  numpy results
in the following error (full output at end):

  ImportError: 
/usr/pkg/lib/python3.11/site-packages/numpy/core/_multiarray_umath.so: 
Undefined PLT symbol "rintl" (symnum = 157)

In the sources, the definition is guarded by
  #ifdef __HAVE_LONG_DOUBLE

See src/lib/libm/src/s_rintl.c which seems to be the same 9/10/current.

I have previously accumulated some diffs in my sources to deal with this
kind of issue:

Index: s_frexp.c
===
RCS file: /cvsroot/src/lib/libm/src/s_frexp.c,v
retrieving revision 1.13
diff -u -p -r1.13 s_frexp.c
--- s_frexp.c   28 Sep 2008 18:54:55 -  1.13
+++ s_frexp.c   16 Nov 2023 13:59:37 -
@@ -31,6 +31,11 @@ __RCSID("$NetBSD: s_frexp.c,v 1.13 2008/
 static const double
 two54 =  1.8014398509481984e+16; /* 0x4350, 0x */
 
+#ifndef __HAVE_LONG_DOUBLE
+__strong_alias(_frexpl, frexp)
+__weak_alias(frexpl, _frexpl)
+#endif
+
 double
 frexp(double x, int *eptr)
 {


As I look at POSIX, which defers to C99, it seems that long double as a
type and long double function variants are required, even if long double
can be equal to double.

On arm, long double seems to be just double, and _HAVE_LONG_DOUBLE isn't
defined, leading to these symbols being missing from libm.   But they
are still in the headers.


I suspect that the right approach is to either systematicaly add the
above, or something similar, but I'm not adequately certain to commit vs
patch locally.

What do our floating point experts think?





-
$ python3.11
Python 3.11.5 (main, Sep 30 2023, 19:59:56) [GCC 7.4.0] on netbsd9
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy;
Traceback (most recent call last):
File "/usr/pkg/lib/python3.11/site-packages/numpy/core/__init__.py", line 23, 
in 
from . import multiarray
File "/usr/pkg/lib/python3.11/site-packages/numpy/core/multiarray.py", line 10, 
in 
from . import overrides
File "/usr/pkg/lib/python3.11/site-packages/numpy/core/overrides.py", line 8, 
in 
from numpy.core._multiarray_umath import (
ImportError: 
/usr/pkg/lib/python3.11/site-packages/numpy/core/_multiarray_umath.so: 
Undefined PLT symbol "rintl" (symnum = 157)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in 
File "/usr/pkg/lib/python3.11/site-packages/numpy/__init__.py", line 139, in 

from . import core
File "/usr/pkg/lib/python3.11/site-packages/numpy/core/__init__.py", line 49, 
in 
raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

* The Python version is: Python3.11 from "/usr/pkg/bin/python3.11"
* The NumPy version is: "1.25.2"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: 
/usr/pkg/lib/python3.11/site-packages/numpy/core/_multiarray_umath.so: 
Undefined PLT symbol "rintl" (symnum = 157)

Re: new certificate stuff

[Greg Troxel]

Taylor R Campbell  writes:

> The critical part I had missed is that certctl can claim _either_ a
> directory it has already claimed, _or_ an empty directory, so it works
> for new installations and to update pristine but old installations.

Sorry, I should have said that out loud; I was thinking that.
> Let me know if any of this seems wrong, or if the implementation seems
> to behave differently from what I described.

Sounds good.

> Regarding etcupdate: I agree that interactive prompting is bad for
> deployment.  I don't actually use etcupdate myself, partly because it
> is too interactive but mainly because it can't do three-way merges --
> instead, I use a bespoke script called etcmerge that does three-way
> merges using the old and new etc.tgz.
>
> https://mumble.net/cgit/campbell/etcmerge
>
> (Maybe some day I will propose to import this into base, but it needs
> a lot more polish and testing first, and some tweaks to the usage
> model which currently has too many things going on at once.)

You may also wish to look at etcmanage, which has a concept of marking
things manually maintained and handling all sorts of cases.  But it does
not merge.  So some logical merge of all of these would be good.

> But while I agree with your criticism of etcupdate, it's what we have
> in base and what we recommend in the guide.  So that's what we have to
> work with as a baseline to gauge the impact of changes like this on
> update procedures; it's hard to meaningfully gauge if I have to guess
> everything that anyone might try to do.

Yes, it's the standard approach, but there are a number of well-known
update workflows.

> That said, you're right that it's better not to create things that
> rely on the interactive prompt.

Thanks for adjusting; I think we're in a good place now.

Re: new certificate stuff

[Greg Troxel]

The other alternative is to decide that we are going to do unsafe things
and to put it super loudly in the release notes that any
sysadmin-configured trust anchors will be blown away.  Compared to
pkgdb, I expect that most admins both have backups, and have such certs
elsewhere, and recovery is not too hard.  I do expect a lot of people to
have trouble and to complain.

Doing it this way would be a eparture from longstanding practice that is
so ingrained that we haven't ever talked about whether it is policy.

Re: new certificate stuff

[Greg Troxel]

Taylor R Campbell  writes:

> How is using /etc/openssl/certs/.certctl as the token different from
> using /etc/openssl/certs.conf as the token?

Because normal updates merge etc in various ways, and if certs.conf
comes along with that (because it is in etc.tgz) then that is automatic
and not an admin action.

> How does this satisfy the two criteria I set down in the previous
> message?  Repeating here:
>
> (a) new installations get /etc/openssl/certs populated out of the box,

that dir is empty and certctl can write to it and set the sentinel.
There is no problem populating an empty dir because that's not removing
admin config.

> and
>
> (b) on _future_ updates (like 10.0 to 10.1, where both releases have
> certctl(8) and a default certs.conf), /etc/openssl/certs gets
> updated too (unless you set `manual' in /etc/openssl/certs.conf).

Once there is a sentinel file, certctl works like it does now.  So I
don't see the concern here.

> Please be specific either about what mechanism you mean and how it
> meets these criteria, or about scenarios that you think the current
> mechanism will inadequately address.

The current one fails to address an update where etc is merged (as it
should be), causing certs.conf to exist, in the case where the admin has
configured trust anchors.

I also don't see how the situation of having mozilla-rootcerts-openssl
installed is handled, and what happens when it is later de-installed.

> The status quo in HEAD (if pulled up to netbsd-10) will be:
>
> 1. New installations get /etc/openssl/certs populated out of the box.
>
> 2. Updates to existing installations that follow the long-standing
>procedure of
>
>- update kernel
>- unpack non-etc sets
>- run etcupdate
>- run postinstall
>
>will interactively prompt during etcupdate to install the new
>/etc/openssl/certs.conf.

That is one longstanding procedure among many in that there are various
flavors of etcupate and private scripts that do the same thing.  It is
not ok for upgrades to need to be interactive in many situations, and
hence etcupdate cannot be the one true path.

And, prompting to install a new config file, which happens all the time,
is not the same as asking if it's ok to overwrite something.

>The file has comments explaining what it does and how to override
>it -- here's the current content:

Sure, once it's looked at.

With pkgdb, there was the idea that people might have to do things.
But it was a disaster in practice for many.

(I realize this is easier to recover from.)

> 3. If:
>- you have /etc/openssl/certs.conf from 10.0,
>- you haven't set it to manual,
>- there are changes to mozilla-certdata from 10.0 to 10.1, and
>- you update to 10.1,
>then those changes will be reflected in /etc/openssl/certs on
>postinstall (but etcupdate won't prompt anything about certs.conf
>-- unless something has also changed in that, of course).

sure that's fine

> If you follow a different procedure for update, like naively unpacking
> etc sets or running a bespoke etcupdate replacement, well, you should
> expect to have to deal with the fallout yourself.

I don't think that is at all reasonable.  Yes, unpacking etc.tgz into
etc is nuts, but taking a file that exists in the new set and not in
/etc and silently putting it in /etc is not dismissible as bespoke; it's
the right thing to do and IMHO it's a bug that etcupdate prompts itstead
of just doing it.  This is based on experience using NetBSD in large
scale testbeds where more machines than people can think about get
updated to new builds frequuently; this "change file in /etc to match
etc.tgz, as long as the file in /etc either doesn't exist and hasn't
existed, or has the same bits as it did when it was last put there from
an install or automated update" procedure works extremely well.  It's
been available in pkgsrc since 2006.

And, I don't even think "etcupdate asks if it's ok to install some new
file you don't necesssarily understand the consequence of" avoids this
concern.   We do not have a history of adding new config files via this
path that then overwrite admin config.

Maybe there's some other way for certctl to recognize things that it
didn't add, without a sentinel file.

Re: new certificate stuff

[Greg Troxel]

Taylor R Campbell  writes:

> Currently, if /etc/openssl/certs.conf doesn't exist, `certctl rehash'
> (the crux of `postinstall fix opensslcerts') will print an error
> message and then exit with status 0.  This combination is a bug --
> need to think a bit about it, but probably better to exit nonzero than
> to suppress the error message.

sure, that's fine.

> So if you unpack new _non-etc_ sets, `postinstall fix' won't
> clobber your /etc/openssl/certs directory.

ok

> The etc.tgz set, however, will have /etc/openssl/certs.conf.  So if
> you naively unpack etc.tgz, `postinstall fix' will clobber your
> /etc/openssl/certs directory.

What I do is unpack etc.tgz and xetc.tgz to /usr/netbsd-etc and then use
etcmanage, which does automatic updates which are safe.  However, safe
is not turing complete; it is "this file has newly appeared in the new
etc and does not exist in the real etc so copy it in".  This happens all
the time with new rc.d and things like that.

> That said, I think if you use etcupdate(8), it will interactively
> prompt you before creating the new /etc/openssl/certs.conf.  (Have
> made a note to add this in my etcmerge(8) tool to do a three-way merge
> for updating (x)etc sets too.)

etcmanage won't prompt.  It has a different design philosophy, which is
that it is run non-interactively to allow administering large numbers of
computers.  The initial impetus was a testbed of 20 and later 100.  So
it does what can be done safely and just doesn't do the rest, which
works out remarkably well.

ANd the prompt is not

  this is installing a config file that will cause later steps to
  overwrite manual configuration

it's just

  there's a new config file you don't have; install it?



> I'm open to other suggestions about how to handle the transition from
> manually maintained /etc/openssl/certs on (say) 9.x with no certs.conf
> or certctl(8) to 10.0 with new default certs.conf and certctl(8),
> provided that
>
> (a) new installations get /etc/openssl/certs populated out of the box,
>
> and
>
> (b) on _future_ updates (like 10.0 to 10.1, where both releases have
> certctl(8) and a default certs.conf), /etc/openssl/certs gets
> updated too (unless you set `manual' in /etc/openssl/certs.conf).

I think the only issue is a one-time transition from

  sysadmin controls contents of /etc/openssl/certs

to

  certctl controls contents

and I can't think of anything safe other than

  sentinel file in /etc/openssl/certs declares it certctl managed

  certctl will auto-write the sentinel if it's sure there are no manual
  contents

  certctl will rm/replace if sentinel present

  certctl -f will rm/replace and write sentinel



This works for (a) and it works for (b), with postinstall fix because
there is a sentinel file, and it will also work if certs.conf gets
updated via etcupdate/etmanage/etcmerge/whatever-sysadmin-uses.

Re: new certificate stuff

[Greg Troxel]

Manuel Bouyer  writes:

>> The etc.tgz set, however, will have /etc/openssl/certs.conf.  So if
>> you naively unpack etc.tgz, `postinstall fix' will clobber your
>> /etc/openssl/certs directory.
>
> As it will clobber others /etc/ files, so that's fine.

Maybe this is too much, but perhaps certctl should look for a .certctl
in /etc/openssl/certs and only if present rm/replace.  Or really only
limit the rm; adding to an empty dir is fine.   Basically a token that
says the dir is under the control of certctl.  -f can override the
check and write the token.

I know this sounds like extra work, but the lesson I took from the pkgdb
change is that things like that this are at least 10x harder than you
think.


Also people will have to uninstall mozilla-rootcerts-openssl.

Re: [PATCH] HTTPS/TLS CA certificates in base

[Greg Troxel]

With the certctl patch on the table, I think it will be possible for
anybody who wants to

  install mozilla-rootcerts
  change certctl.conf to point to it

and get what abs@ wants for updates (which is different that everybody
getting it by default).

I am now in the "this is not really different from any other serious
vulnerability in case" camp.

I have long believed that installing any particular release and leaving
it indefinitely is not reasonable.  My own practice is to run the
netbsd-N stable branch and routinely update along the branch every 2
months, which means I am never far out of date and also in a position to
update/build/rsync/update quickly when fixes for serious CVEs appear on
the branch.  So it's the same timeline as updating pkgsrc (update,
pkg_rr, create summary, sync, pkgin) with different steps.

Re: [PATCH] HTTPS/TLS CA certificates in base

[Greg Troxel]

Taylor R Campbell  writes:

> This is exactly what you get if you populate a directory
> /usr/local/mycerts with the .pem certificates you want and then add
> the line
>
> path /usr/local/mycerts
>
> to /etc/openssl/certs.conf, alongside the line
>
> path /usr/share/certs/mozilla/server
>
> which is already in it.

That's totally fine.

> The only restriction is that the base names must be distinct across
> all the paths specified.

heh, but as long as the man page says that doesn't upset me.

>>   a "trust mozilla root certificate set" as a yes/no option in
>>   configuration where you turn on sshd/ntpd, and I don't really care how
>>   it defaults.
>
> An unobtrusive optional item in the sysinst utility and/or config menu
> would be fine by me, along with a note in afterboot(8).  (I feel like
> the config menu ought to have a reference to afterboot(8) so you know
> how to go back and reconsider the config later.)  But I don't think
> the initial commit needs to block on this.

initial commit seems fine, but I think it should be present before
pullup to 10.  However, I may be an extreme outlier.

>> 3) If a user wants to configure (as an example)
>>The US DOD root (a "real CA" not in the mozilla set)
>>a personal CA that they created
>>Let's Encrypt, and 8 other specific CAs from mozilla
>>
>> then that should be supported somehow in the certctl config file.  I
>> suspect you've enabled that, and it's just omitting the "use the mozilla
>> set" and adding lines for "use this" and "use that".
>
> Create a directory of .pem files or symlinks to them and put it in a
> `path' line instead of `path /usr/share/certs/mozilla/server' in
> certs.conf.
>
> Or, create three directories and add three `path' lines.

Totally straightforward, thanks.

> The patch I attached only adds the certificates, certctl(8), and
> automatic tests.  It does not change whether any applications use
> /etc/openssl/certs or require validation.
>
> Currently, as of February, ftp(1) has validation enabled by default,
> with a default-off option `sslnoverify' to disable validation.  That
> means it just doesn't work out of the box right now on real-world web
> sites.
>
> I don't know offhand whether pkg_add(1) has validation enabled or
> disabled by default, but we should enable it after this goes in.
> (That's aside from signing packages, which we should also do, but my
> plan for that is not ready yet.)

That make sense and I'm all for keeping separable things sepearate.

>> C) The openssl nomenclature /etc/openssl/certs is confusing because it
>> blurs what is a certificate and what certificates are trust anchors; it
>> really (I think) means the latters.  I realize that few understand this
>> naming distinction, but a CA cert is just a cert whose private key can
>> sign other certs, and you may or may not care, unless it is a trust
>> anchor or reachable from one.
>
> I agree the nomenclature is confusing.
>
> However, the path /etc/openssl/certs is baked into applications
> already.  Changing it would require patching OpenSSL (again),
> coordinating with pkgsrc, and dealing with who knows what other
> existing applications.  And it's overridden in OpenSSL by the
> SSL_CERT_DIR environment variable, which would presumably be even
> riskier to rename.
>
> So I'm not proposing to tug at that thread.  I'm only proposing to
> populate the directory out of the box.

I did not mean to change directory names or code at all.

I meant that it would good if the documentation said, at least the first
time it appears "configure certificates as trust anchors" rather than
"install CA certificates".   Certainly we don't want to confuse normal
people who are used to sloppy wording, but somebody who has read pkix
should be able to tie our wording to the spec in a way that is not
confusing or ambiguous.

This wording problem is not new to your proposed change.  As an example
of exising wording, mozilla-rootcerts-openssl says:

  This package configures the Mozilla rootcerts bundle CAs as trust
  anchors in OpenSSL, so that programs using OpenSSL will be able to use
  them to validate SSL certificates.

which I intended to be accurate for those that understand and clear
enough for those that are fuzzy on "install certificates".

Re: Android-like NetBSD

[Greg Troxel]

Chris Hanson  writes:

> On Aug 5, 2023, at 12:11 PM, Greg Troxel  wrote:
>> 
>> As for 'external SDK', that would be "install some other OS and cross
>> build", but when you cross build for android or ios, you do that from a
>> system which is a full install.
>
> This isn’t the case for the Apple platforms; Xcode comes with
> sysroot-style SDKs for Apple’s platforms that contain only headers and
> stub libraries (these days in the form of .tbd files, previously in
> the form of stub Mach-O files) since those are all that are needed to
> build.

I meant that the host system that is doing the compilation is a full
normal system.

But interesting point about not having a proper destdir, and the
presence of stub libraries.


I don't really understand what Bruno is trying to do.

Re: [PATCH] HTTPS/TLS CA certificates in base

[Greg Troxel]

Overall I like this.  Thank you for listening to the various comments
and coming up with a mechanism that is configurable for almost all
possible policies that have been expressed.


I'd like to see three things handled (which might be already):

1)

  a way for a user to install a CA cert (as a trust anchor -- I think it
  would be good for docs to use pkix terminology) that is not part of
  the mozilla root set, such that as updates/etc. happen, the trust
  anchor set remains
union of
  user-configured
  mozilla set, if opted into, minus those excluded

This is just code and I don't care if the user has to put the cert in
/etc/openssl/manual-trust-anchors or someplace like that to be available
for unioning, which would make certctl be able to just write /certs.
I am guessing your patch already handles this somehow, but I wanted to
comment quickly before reading this since I have ranted so much.

2) On installation, either

  a yes/no question "do you want to configure the Mozilla root
  certificate set as trusted"?

or

  a "trust mozilla root certificate set" as a yes/no option in
  configuration where you turn on sshd/ntpd, and I don't really care how
  it defaults.


We should absolutely install them in their own dir always; this is only
about whether the certctl.conf file has "include mozilla" or doesn't,
and is thus super easy to change.

3) If a user wants to configure (as an example)
   The US DOD root (a "real CA" not in the mozilla set)
   a personal CA that they created
   Let's Encrypt, and 8 other specific CAs from mozilla
   
then that should be supported somehow in the certctl config file.  I
suspect you've enabled that, and it's just omitting the "use the mozilla
set" and adding lines for "use this" and "use that".


minor comments:

A) I think it's fine to only have "postinstall fix" run certctl in the
DESTDIR=/ case and not at all urgent to address.   This is sort of like
how we do'n build locate/man db etc. in images but only on real systems.

(Alternatively we could run certctl rehash at boot if certctl=YES,
default YES and skip it from postinstall.  I have not thought through
pros and cons.)

B) There is talk of enforcing validation, but we already enforce
validation with an empty list.  I think the only change is installing a
bunch of CAs as trust anchors (in a manageable way), and there is no
change to validation.  If I'm confused on this point please tell me.


C) The openssl nomenclature /etc/openssl/certs is confusing because it
blurs what is a certificate and what certificates are trust anchors; it
really (I think) means the latters.  I realize that few understand this
naming distinction, but a CA cert is just a cert whose private key can
sign other certs, and you may or may not care, unless it is a trust
anchor or reachable from one.

Re: epoll exposure

[Greg Troxel]

Mouse  writes:

>> The problem is third-party software assumes epoll == Linux,
>
> Software that makes stupid assumptions will never go away.
>
> Is it better to work around it (not ship epoll.h), or to get it fixed
> (report it upstream as the bug it is)?  I could argue that either way.

I don't really see it as a bug.  You'd have to have all those problems
have configure logic that says

  if we find an epoll implementation, then we have a list of operating
  systems that have implemented an epoll that has different semantics
  and we have to reject it

It seems far more reasonable to say that if an OS implements a different
epoll, then it should call it something else.

Re: epoll exposure

[Greg Troxel]

Jonathan Perkin  writes:

> * On 2023-08-13 at 18:10 BST, Tobias Nygren wrote:
>
>>On Sat, 12 Aug 2023 19:21:06 -0400
>>Christos Zoulas  wrote:
>>
>>> I really want to understand what's going on here (why do we think that
>>> our epoll implementation is broken in a way that will affect applications).
>>
>>jperkin@ might be able to explain what the issues with Illumos are,
>>but I guess the problems are of different nature than the NetBSD case.
>
> The problem is third-party software assumes epoll == Linux, and once

I see it that Linux was first with epoll and thus is the specification.
In an ideal world, it would have gone through POSIX, but whatever.
Once that exists, I think an epoll implementation in other operating
systems has to have the exact same semantics, or it's asking for
trouble, perhaps hard-to-find ways.

I don't really find it problematic that the world has assigned the word
epoll to Linux's implementation.  To object to that one has to object to
any non-POSIX feature in any OS.

> our epoll implementation landed to support LX binaries, a bunch of
> packages that previously built natively broke.

Is that because while it appeared to offer the same API, it did not
exactly?

> In hindsight we would have not shipped sys/epoll.h, it provides no
> benefits, and obviously it's preferable to use native features (event
> ports in our case, kqueue in yours).

Was your epoll intended to be Linux compatible, or something else?  It
would seem that if so, it should be exactly compatible, and if not it
should just have a different name.  It really feels like you have gone
down the path we are on, so your experience seems useful.

> So now I'm stuck with shipping a faked-up epoll.h in my build chroots
> that #error out forever, and still have to patch up some software that
> finds it and continues on regardless thinking we're Linux.
>
> Just Say No.

Do you mean:

  do not implement epoll in NetBSD native at all?

or

  if you implement epoll, it needs to be 100% compatible with Linux
  semantics so that programs that use it will work as one expects?


There are some programs out there that support epoll but not kqueue;
guile-fibers used to be one but grew kqueue support.

Re: epoll exposure

[Greg Troxel]

nia  writes:

> On Fri, Aug 11, 2023 at 06:52:41PM -, Christos Zoulas wrote:
>> I see that you removed with without further discussion which is not the
>> way we do things on NetBSD. Do you have an example where the epoll emulation
>> breaks, because either forking matters or the implementation is
>> incorrect/different?
>
> We maybe need some clarification there - it's come up before
> that changes should be backed out while the discussion is ongoing.
> I've generally complied and backed out my changes when someone
> wants to discuss them.

As someone who has given others a hard time about this I'd like to
strongly second what nia@ said.  I agree with and sympathize with the
idea that we don't randomly revert, but the total situation is more
subtle.

First, we have a norm that changes that others object to should be
proposed and discussed, and only if there is pretty strong rough
consensus or better that they get committed.  For weaker rough
consensus, I see it as a call for core or pmc.  A real problem is when
people just commit anyway, and people object.  The right thing is for
the committer to say "didn't realize this was so controversial" (often
it's hard to tell) and revert.  It can easily be an honest mistake to
not realize that other people will have issues with something.

I believe it is very important to have the discussion with the
controversial code not in tree, because that frames the argument as "I
would like to commit this, is that ok" rather than "someone else wants
to revert this thing that we already have".  In some sense that's the
same, but we are talking people and it's really clear to me that it is
not the same.

I view committing controversial things and not reverting them as a
bigger problem than someone else doing the revert after it has been
called for.  In my view core should be asking for things to be reverted
for discussion when they turn out to be controversial.

> Riastradh raised that having a symbol named "epoll_create" in libc
> may be enough to change the way build systems behave, so we need
> to be really careful here. In my opinion it's better to back out
> ABI changes that may be problematic early before we're stuck with
> something we might regret later.

Agreed.  In this case, the complaints were rapid and significant and it
was obviously tricky.

And, my read of the discussion was that adding an emulated epoll for
linux binaries was ok but adding native epoll had nowhere near reached
consensus.

>> I would agree on principle that it is better to use kqueue on BSD systems,
>> but if it is not broken, why not advertise it?
>
> We need to be 100% sure that this code will remain 100% compatible
> with Linux for the forceeable future. Exposing epoll involves
> compiling code that has only ever been tested on Linux on
> NetBSD for the first time, while avoiding tried and tested code.
>
> This is really scary, and can create all sorts of headaches in
> porting work, especially if the epoll-enabled source code uses
> all sorts of other Linuxisms.

Agreed.  Porting linuxy code is hard enough.  I think we need to talk
about exact semantics match, and if not to really have the rationale
super clear and widely accepted.   I think a  lot of people are just not
wanting to bite off that trouble.

Re: Android-like NetBSD

[Greg Troxel]

Bruno Melo  writes:

> Is it possible install a minimum NetBSD with no manpage, no compilers
> and toolchain and provide all these stuff as an external SDK and then
> being able to build pkgsrc software using this external SDK? I think
> this is what commercial systems like Android and iOS are
> doing. Android itself doesn't even provide a su command.

You can:

  install a normal netbsd system and build packages

  install on another machine only "base.tgz" (and kernel) and not the
  rest, and install binary packages.  That might work, or mostly work,
  and you might need a few more sets.


As for 'external SDK', that would be "install some other OS and cross
build", but when you cross build for android or ios, you do that from a
system which is a full install.

Re: Architecture neutral packages (mozilla-rootcerts-openssl)

[Greg Troxel]

David Brownlee  writes:

Please drop tech-pkg from this.  This is a base system issue.

> Another unmentioned local elephant is installations which do not use
> sysinst (though I'm happy to suggest we specifically exclude those for
> the first pass).

People who do things manually do things manually, which means they make
choices about all sorts of things.  I think that's fine and doesn't need
addressing.

Re: /etc/services losses

[Greg Troxel]

Taylor R Campbell  writes:

> `smtp(s)' and `submission(s)' are subtly different protocols and
> should not be aliases:
>
> - smtp(s) is for MTA<->MTA exchange of fully formed internet mail
>   messages with complete headers.
>
> - submission(s) is for an MUA to submit new messages, which may not
>   have complete headers or fully qualified addresses or otherwise be
>   fully formed, via a mail submission agent into the internet mail
>   system.

Yes, they are different, however my take from reading
https://datatracker.ietf.org/doc/html/rfc8314 is that the orignal label
of 465 as smtps was confused, and that IANA now labels 465 as start

> I'm also not sure it matters if a TLS session is preceded by the ten
> bytes `STARTTLS\r\n' on the wire or not.

It very clearly does not matter.  I think the concern was
implementations that treated TLS as optional and would continue.  But
that's all rationale for why RFC8314 recommends as it does.  Where we
are now is that 465/tcp is submissions and 587 is submission.  And our
services file has smtps for 465, but that no longer exists in IANA-land.

> I'm not sure if there is any port number that can rightly be called
> `smtps' today.

Agreed.  I know of no usage that is MUA-to-MUA SMTP over prenegotiated
TLS.   It's basically a STARTTLS over 25 world -- which is what I think
you are thinkig

For this thread, I think all that matters is that we find a reasonable
way to update services to match IANA while maintaining local diffs.  We
have gotten into a bad state.

Re: /etc/services losses

[Greg Troxel]

Mouse  writes:

>> I'm also not sure it matters if a TLS session is preceded by the ten
>> bytes `STARTTLS\r\n' on the wire or not.

> In theory, it matters because the conversation is not conformant to the
> protocol otherwise; a receiver-SMTP would be entirely justified in
> dropping a connection which attempts to start a TLS session without
> STARTTLS, and, while I don't have specific knowledge of any (I don't
> use TLS), it would surprise me if there weren't implementations that
> did.  (Playing fast and loose with standards conformance is in large
> part how email became the disaster it currently is; doing so more just
> makes it worse.)

I am pretty sure Taylor menat that there is no meaningful difference
between:

  connect to 465, negotiate TLS, speak SMTP/submission inside of TLS

and

  connect to 587, send "STARTTLS", negotiate TLS, speak SMTP/submission
  inside of TLS.  If STARTTLS/negotation fails, error out.

and I agree.  Nobody is suggesting that 587 speak TLS without STARTTLS
or that 465 accept STARTTLS.

Re: /etc/services losses

[Greg Troxel]

Hauke Fath  writes:

> On Mon, 31 Jul 2023 18:20:40 +0200, Steffen Nurpmeso wrote:
>> Greg Troxel wrote in :
>>  |Hauke Fath  writes:
>>  |> attached is a diff with services that for some reason or other got 
>>  |> dropped from /etc/services - in particular Amanda* and AppleTalk.
>>  |
>>  |The really big question here is the relationship between our
>>  |/etc/services and
>>  | 
>>  | 
> <https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt>
>
> Indeed, and our /etc/services looks a lot more like a copycat of that 
> file than like the output of Steffen's script - thanks for that. Has it 
> ever been used for a commit, though?

You should write to the people that did the commits and ask them.

> IANA appears to have settled on submissions for port 465 (which, 
> coincidentally, was assigned to 'urd' in the netbsd-5 version, and the 
> NetBSD addition then declared the smtps alias). A web search for 
> 'smtps' confirms widespread use, so this should be maintained as an 
> alias for 'submissions' IMO.

I must have misread.  The iana link above clearly has submissions =
465/tcp.  It also has urd.  So our file should of course match.


There's a huge point here that I want to make explicit.  We need to have
a sane process for updating from IANA, and that involves

  - having a script to convert IANA into our format

  - having that script checked into the sources

  - some mechanism to maintain the "local additions" section at the end

  - probably a scheme to maintain any diffs from IANA, if we really want
any

  - probably a way to build a new file from iana-converted and local
  

Re: /etc/services losses

[Greg Troxel]

Hauke Fath  writes:

> attached is a diff with services that for some reason or other got 
> dropped from /etc/services - in particular Amanda* and AppleTalk.

The really big question here is the relationship between our
/etc/services and

  
https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt

The format you don't like seems to be sort of similar to the one from
IANA.  But we are quite out of sync from that, so I wonder about the
other BSDs.

Looking at the CVS history of src/etc/services (you didn't mention this,
but it is of course highly informative):

  revision 1.104
  date: 2022-11-27 21:48:33 -0500;  author: jschauma;  state: Exp;  lines: 
+4876 -2122;  commitid: 32OhFLVJATNRMp3E;
  regen from IANA 2022-11-22
  
  revision 1.103
  date: 2021-04-08 15:03:43 -0400;  author: christos;  state: Exp;  lines: 
+2167 -1938;  commitid: OWMzgP5LIijfrwOC;
  refresh with latest
  
  revision 1.102
  date: 2019-10-09 18:42:14 -0400;  author: sevan;  state: Exp;  lines: +73 
-2613;  commitid: yPvhkD2M6kinefGB;
  sync with latest from IANA
  
  revision 1.101
  date: 2019-06-15 11:22:56 -0400;  author: christos;  state: Exp;  lines: 
+2697 -42;  commitid: 1DqxZezrQZeHwirB;
  branches:  1.101.2;
  - sync with latest from iana
  - add ttcp
  
  revision 1.100
  date: 2019-01-03 12:30:06 -0500;  author: christos;  state: Exp;  lines: 
+7802 -9512;  commitid: WE0FBhgRwFbjtm6B;
  regen

So I would talk to them and see what they did and why; it seems like
there must be a script from an iana file, and then there's supposed to
be the "local additions" section.  Probably the real bug is losing that
and it can be put back.  But editing without understanding that flow
seems unwise.

Files generated like this usually have a Big Scary Warning, and this
doesn't; probably someone(tm) should fix that.

On the substance:

The use of mail for port 465 was apparently assigned briefly and then
not (we have STARTTLS now), and how is assigned to urd.  I never thought
it was for submission.  It is not in the current IANA file.  So it's a
good question why it remains at all.  I am therefore not ok with adding smtps.

I wonder why kamanda is commented out.  Probably because it conflicts
and it is not assigned by IANA.


> The repeated loss of entries is an immediate consequence of the file's 
> shape - look at the version history, no committer would look at the 
> diffs.

I think the issue is the semi-automated nature and not everybody being
clear on that.

> Anyway... okay to commit?

For now, I would say not ok about smtps -- I think we need to understand
the larger plan of tracking upstream and if we are off plan how that
happened.

Let's hear from the people that have been maintaining this.

Re: ipsec: slight inconsistency

[Greg Troxel]

tlaro...@polynum.com writes:

> The two functions are said "inverse" from each other but the problem is
> that if one gives a delimiter to ipsec_dump_policy(3) that is neither
> a blank nor a new line, the string obtained can not be an input
> to ipsec_set_policy(3). So there are not really inverse from each other.
>
> Wouldn't it be more logical whether to have no delimiter to
> ipsec_dump_policy(3) (defaulting to '\n' for separating the elementary
> statements) or to allow a delimiter to ipsec_set_policy(3) when parsing
> the policy passed?

I think it would be most logical to document in ipsec_dump_policy that
the default delimeter matches what is expected by ipsec_set_policy, and
that alternate delimiters might be useful for people but do not produce
valid syntax.  That resolves your consternation but does not break
anyone relying on the current behavior.  This problem is surely
longstanding and that you seem to be the first to notice or care, so the
severity would seem to be extremely low.

missing long double functions in netbsd-9

[Greg Troxel]

I'm using various things that uses numpy.  I have had a problem with
long double versions of log functions on x86, and with other functions
on arm.  I don't actually need these functions, but numpy wraps them and
thus the shlib won't load.  The numpy package build succeeds.  I think
it's a bug by inspection that they are missing so the why I care bit is
not important.

1) One case is
  log1pl
  log2l
  expm1l
being missing in libm.  I've worked around that by a
too-gross-too-commit kludge that I've appended; basically implementing
them by calling the double version.  I now realize that I should perhaps
instead define aliases, following how long double function are provided
on arm, and perhaps actually have code the processes long doubles,
perhaps stealing from FreeBSD.


2) The other case is frexpl.  It seems the plan is for each function to
have a fool impl that is #ifdef HAVE_LONG_DOUBLE, and for the foo impl
to have aliases from fool #ifndef HAVE_LONG_DOUBLE.  But some of these,
include frexpl, are missing.   I have appended a patch that adds this.

There appear to be multiple similar cases of missing long double
variants on arm.

3) I'm boggled at how this works in the sources.  s_frexp.c is not
included in libm; a comment says it is in libc.  I can't find it in C
sources or arm assembly.  On earmv7hf-el, I find frexp and frexpf in
libm and frexp in libc.  Trying to rebuild libm and libc after my change
results in nothing happening.


I would appreciate:

  clues to how frexp flows from source to libs

  advice on the right way to address this.





 frexpl

Index: s_frexp.c
===
RCS file: /cvsroot/src/lib/libm/src/s_frexp.c,v
retrieving revision 1.13
diff -u -p -r1.13 s_frexp.c
--- s_frexp.c   28 Sep 2008 18:54:55 -  1.13
+++ s_frexp.c   28 Jun 2023 17:05:46 -
@@ -31,6 +31,11 @@ __RCSID("$NetBSD: s_frexp.c,v 1.13 2008/
 static const double
 two54 =  1.8014398509481984e+16; /* 0x4350, 0x */
 
+#ifndef __HAVE_LONG_DOUBLE
+__strong_alias(_frexpl, frexp)
+__weak_alias(frexpl, _frexpl)
+#endif
+
 double
 frexp(double x, int *eptr)
 {


 log1pl etc.

Index: src/b_exp.c
===
RCS file: /cvsroot/src/lib/libm/src/b_exp.c,v
retrieving revision 1.1
diff -u -p -r1.1 b_exp.c
--- src/b_exp.c 5 May 2012 17:54:14 -   1.1
+++ src/b_exp.c 28 Jun 2023 16:58:41 -
@@ -135,3 +135,41 @@ __exp__D(double x, double c)
/* exp(INF) is INF, exp(+big#) overflows to INF */
return( finite(x) ?  scalb(1.0,5000)  : x);
 }
+
+
+/*
+ * Fake up long double version to let numpy build, and blow off it
+ * being more accurate.  Put it here because s_log1p.c is really in
+ * asm.
+ *   gdt, 20220811
+ */
+long double
+log1pl(long double x)
+{
+  long double y;
+
+  y = log1p(x);
+
+  return y;
+}
+
+long double
+log2l(long double x)
+{
+  long double y;
+
+  y = log2l(x);
+
+  return y;
+}
+
+long double
+expm1l(long double x)
+{
+  long double y;
+
+  y = expm1l(x);
+
+  return y;
+}
+

Re: shutdown sequence and UPS poweroff

[Greg Troxel]

David Brownlee  writes:

> If only we had a reliable way to remount filesystems from read-write
> to readonly after flushing data, that could be an option on shutdown,
> then the ups command could be triggered with no writable filesystems
> :-P

That is what ought to happen, yes.  I am sort of surprised that doesn't
work.

Re: shutdown sequence and UPS poweroff

[Greg Troxel]

David Holland  writes:

> On Fri, Jan 06, 2023 at 10:13:36AM -0500, Greg Troxel wrote:
>  > Does it seem reasonably safe that mountall through root will be fast, < 
> 10s?
>
> We've been seeing scattered reports of unmounting taking fast amounts
> of time, particularly on nvme devices, and I don't think we know why
> yet.

So usually < 10s, but could be 10 minutes in the bad case?

> It's a bug, but since not cutting the power to SSDs midflight is one
> of the primary reasons to have a UPS these days... seems like a
> dangerous on in this context. So it seems like a good idea to be
> cautions.

It seems then we should add some instrumentation to print scary warnings
(or even panic?) if unmount is slow, to be run by all users.

> Not sure what to recommend. The last time there was an issue like this
> (which turned out to be a bad bug with cached data not being written
> back until unmount time) it was possible to trigger the writing by
> attempting an unmount you know will fail with EBUSY, but I don't think
> we've ascertained if that works this go.

The other side of the coin is that shutdown starts when there is limited
runtime and most people want to use most of their runtime.

I'm going to live-test my system soonish.


Another thing is that mountall doesn't do anything on shutdown.  It
seems like it should unmount what it mounted, or unmount everything
which is not critical_filesystems_foo.  That would unmount big
filesystems before getting to power shudown command, at least for those
without whole-disk root.

I think nut's norm is 20s so may head to that.   Only affects those who
set up nut.  Without that, though, you'll lose power when the UPS shuts
down, with no warning, which isn't great either.

shutdown sequence and UPS poweroff

[Greg Troxel]

Currently, pkgsrc/sysutils/ups-nut works well on NetBSD (somewhat shaky
testing on some points, but my current belief):

  UPS is monitored and data is available
  configuration of UPS params
  messages on loss of comms and restore
  messages on transition to battery and return
  initiation of shutdown on low battery

What is missing is

  during shutdown due to low battery, at some point run `upsdrvctl
  shutdown` to tell the UPS to turn off power (and thus preserve battery
  life) until power is back

Most UPS units can be set up to e.g. wait 20s after the command to turn
off power, and then to power on some delay after power returns.

The question is how to add this in as an rc.d script.

It should be as late as possible in shutdown so that the system is
overwhelmingly likely to be in an ok state (filesystems unmounted or
mounted ro) before power goes away.

To run `upsdrvctl shutdown`, guarded by `upsmon -K` to find out if we
should, then /usr/pkg or whatever needs to be available.

rcorder * shows the following, with '/ -' noting what happens on shutdown

  root / -
  DISKS
  swap1 / also unmounts tmpfs without device nodes
  mountcritlocal / -
  NETWORKING
  mountcritremote / -
  mountall / "umount -a" !!
  swap2
  SERVERS
  DAEMONS
  LOGIN
  upsd

It seems the ups files should have some BEFORE/AFTER to put them
between SERVERS and DAEMONS, so let's assume I've done that -- but
that's a separable issue.

(It also seems that things like nfsd, that aren't needed to get *this*
machine up should REQUIRE: DAEMONS but that's off point and not
important.)


I think the right place to kill power, as things stand now, is (on
shutdown) after swap2 and before mountall, because mountall might
unmount /usr.

It seems like a bug that mountall unmounts critlocal/critremote.  With
that fixed, it seems like just before mountcritlocal is the right place.

Does it seem reasonably safe that mountall through root will be fast, < 10s?

Re: mkdir -p, autoconf

[Greg Troxel]

ignat...@cs.uni-bonn.de writes:

> On Fri, Oct 07, 2022 at 01:25:57PM -0400, Greg Troxel wrote:
>  
>> Sorry, I meant not documented in mkdir(1).
>
> "Create intermediate directories as required. ... 
> Do not consider it an error if the argument directory already
> exists."
>
> Hm... our implementation doesn't consider it an error if any
> intermediate directory in that path already exists; I'd argue
> that the "as required" part covers this.

The point is not that they exist, but that the program is robust against
something else creating them.  Given autoconf macros have concerns about
parallel execution, it's reasonable to expect

  if (!directory_exists())
make_it()

and a possible returned error if it doesn't exist but then creation
fails.


signature.asc
Description: PGP signature

Re: mkdir -p, autoconf

[Greg Troxel]

ignat...@cs.uni-bonn.de writes:

> hi,
>
> On Fri, Oct 07, 2022 at 06:40:19AM -0400, Greg Troxel wrote:
>> 
>>   1) Is NetBSD's mkdir -p safe, in that it is immune to
>> check for dir
>> call mkdir(2) which fails because another mkdir did it in between
>> return -1 because that was an error
>> 
>>   I don't see this documented in the (9) manpage.
>
> ... because that's not where it's handled.
>
> Code inspection shows that mkdir itself, if mkdir(2) failed, 
> checks whether the (possibly intermediate) path exists and is
> a directory, and in this case handles it the same af it was
> created successfully.
>
> So the answer to this is "yes".

Sorry, I meant not documented in mkdir(1).


signature.asc
Description: PGP signature

mkdir -p, autoconf

[Greg Troxel]

I am chasing a bug in the postgis install phase, found by a bulk build,
while it ran fine on my system -- which happened to have coreutils.
There are a lot of moving pieces in postgis's install becaues it's
autoconf *and* pgxs, but there's one aspect that I want to bring up
here.


autoconf has a macro AC_PROG_MKDIR_P that sets MKDIR_P.  However it is
all paranoid about races, wanting two simultaneous 'mkdir -p foo'
invocations to result in foo being created and neither returning
failure, because...  specculating somehow there are makefiles that don't
serialize that like they do actual build steps, and somehow those
makefiles are not considered buggy?

autoconf rejects NetBSD mkdir -p and thus there is some install-sh and
that would have been ok but this somehow interacts with pgxs and blows
up.

So my real questions are:

  1) Is NetBSD's mkdir -p safe, in that it is immune to
check for dir
call mkdir(2) which fails because another mkdir did it in between
return -1 because that was an error

  I don't see this documented in the (9) manpage.

  2) If so, should we be doing anything about autoconf?

  3) If so or if not, is it reasonable to force ac_cv_path_mkdir to
  /bin/mkdir on NetBSD?
  


signature.asc
Description: PGP signature

Re: [Christos Zoulas] CVS commit: src/usr.bin/ftp

[Greg Troxel]

I just sent a big note about the default trust anchor issue to
tech-security.  Please follow up there about that, vs this specific ftp
change.

Re: [Christos Zoulas] CVS commit: src/usr.bin/ftp

[Greg Troxel]

Martin Husemann  writes:

> On Fri, Sep 02, 2022 at 06:23:48PM +0300, Christos Zoulas wrote:
>> I think we should be installing the anchors by default. I also think
>> that people think that https gets validated by default.
>
> I agree. The problem is that we need to suply anchors now with new
> installations and have a way to keep them updated (and optionaly disabled).

This belongs in a new thread.  This one is about ftp choosing to
validate when we haven't done that.


signature.asc
Description: PGP signature

Re: [Christos Zoulas] CVS commit: src/usr.bin/ftp

[Greg Troxel]

Christos Zoulas  writes:

>> On Sep 2, 2022, at 3:57 PM, Greg Troxel  wrote:
>> 
>> Did I miss discussion on this?  I am getting the impression that we now
>> have defaults:
>>  no trust anchors installed
>>  require verification
>> 
>> which really doesn't make sense.  If I am following correctly this is a
>> major behavior change in a controversial area, which isn't ok without
>> discussion/consensus.
>
> I did not realize it was controversial, but let's have the discussion.

There's been a large amount of discussion about having the base system
provide a pre-configured set of trust anchors.  Without that already
agreed on and in place, it seems clear that changing ftp(1)'s behavior
is going to be a breaking change for a lot of people, and thus well over
the the needing-discussion line.

>> Plus, this is a negative option, usually frowned upon.
>
> grep NO_ /usr/src/usr.bin/ftp/*.c

I guess, and I didn't really say this earlier, but this is not really
about ftp.  pkix more or less says TLS should do certificate validation,
but it's longstanding practice in many client programs not to validate.
For the most part this is about a sysadmin/user either

  configuring a set of trust anchors, and deciding that that all SSL/TLS
  and other uses of pkix (x509) certificates should be subject to
  validation

  deciding not to play the configured trust anchor game and not turning
  on the "fail if not validated" switch.

>> So (absent confusion on my part, as always), it sounds like one of the
>> following should happen:
>> 
>>  1) just revert this until we have discussion
>>  2) change the environment variable to CERTIFICATE_VALIDATION to use the term
>> from the RFC
>>https://www.rfc-editor.org/rfc/rfc5280#section-6
>> and default to FALSE, enabling if set and TRUE.
>
> I don't care much about the environment variable name, and yes calling
> it what the RFC suggests is probably better.

Do you think it matters if it's an environment variable?  Do you think
this needs to be a user option, or system?  Why environment vs
comand-line like wget?

>> If at some point the system installs trust anchors by default, the
>> default can change.
>
> I think we should be installing the anchors by default. I also think
> that people think that https gets validated by default.

That is a separate discussion, which is totally fine to have in a new
thread.  This discussion is about changing the behavior of ftp to fail
without validation while the system does not install trust anchors.

Right now, this change acts like "disable https in ftp" for people that
do not have configured trust anchors.

>> Plus, I think it's reasonable to have some config file in /etc/openssl
>> that signals "user has configured trust anchors and wishes to routinely
>> validate certificates".  The existence of /etc/openssl/VALIDATE might be
>> a good trigger for validation, or some other color file.  That would
>> mean that the code, running on a system with old config, would not be
>> surprising.   Using this file now in option 2 instead of an environment
>> variable seems fine.
>
> That is a good idea.

So are you ok with:

  1) Right now, in ftp(1), enable certificate validation if
  /etc/openssl/VALIDATE exists (and remove the env var).  Tell people
  who want this to touch /etc/openssl/VALIDATE?

  2) Sort of have a plan that if/when we configure trust anchors by
  default and perhaps if a package configures them
  (mozilla-rootcerts-openssl) that this file be created, to turn on
  validation for all pkix users, as the system default?This would be
  the interface from all trust anchor configuration schemes to all
  validators.


signature.asc
Description: PGP signature

[Christos Zoulas] CVS commit: src/usr.bin/ftp

[Greg Troxel]

Did I miss discussion on this?  I am getting the impression that we now
have defaults:
  no trust anchors installed
  require verification

which really doesn't make sense.  If I am following correctly this is a
major behavior change in a controversial area, which isn't ok without
discussion/consensus.

Plus, this is a negative option, usually frowned upon.

So (absent confusion on my part, as always), it sounds like one of the
following should happen:

  1) just revert this until we have discussion
  2) change the environment variable to CERTIFICATE_VALIDATION to use the term
 from the RFC
https://www.rfc-editor.org/rfc/rfc5280#section-6
 and default to FALSE, enabling if set and TRUE.

If at some point the system installs trust anchors by default, the
default can change.

Plus, I think it's reasonable to have some config file in /etc/openssl
that signals "user has configured trust anchors and wishes to routinely
validate certificates".  The existence of /etc/openssl/VALIDATE might be
a good trigger for validation, or some other color file.  That would
mean that the code, running on a system with old config, would not be
surprising.   Using this file now in option 2 instead of an environment
variable seems fine.



--- Begin Message ---
Module Name:src
Committed By:   christos
Date:   Tue Aug 30 08:51:28 UTC 2022

Modified Files:
src/usr.bin/ftp: ftp.1 ssl.c

Log Message:
Add cert verification, together with an environment variable "NO_CERT_VERIFY",
to turn it off.


To generate a diff of this commit:
cvs rdiff -u -r1.146 -r1.147 src/usr.bin/ftp/ftp.1
cvs rdiff -u -r1.10 -r1.11 src/usr.bin/ftp/ssl.c

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

--- End Message ---

Re: fidocrypt(1): `storing' cgd keys on U2F/FIDO keys

[Greg Troxel]

Taylor R Campbell  writes:

>> Date: Sat, 6 Aug 2022 18:47:47 -0400
>> From: Gabriel Rosenkoetter 

[comments reordered]

>> I guess my follow-up Devil's advocacy question would be: why does this 
>> need to be in base, rather than provided via ports?
>
> cgdconfig runs early at boot before most file systems are mounted --
> often before the file systems on which any packages are installed are
> mounted.  The cgdroot ramdisk, for instance, has a very minimal NetBSD
> userland in a ramdisk just to configure cgd(4) volumes before mounting
> the `real' root from them and chrooting into it.  fidocrypt could be
> baked into this ramdisk.

First, I think this is totally fine to add to the base set, given the
justification of using it to store keys for cgd that would be used for
beyond / and /usr, and it being small.

That raises two semi-separable questions:

  1) should it be in /, so that if / is not on cgd but /usr is, things
  work?

  2) what is the path to getting it into the cgdroot filesystem for
  root-on-cgd?  Does the / and /usr boundary matter?   I have never
  tried this, but it seems that it's just selecting a bunch of paths for
  the ramdisk and which of / and /usr they are on is not important in
  this process.  Being small still is important.

I'm not sure if 1 matters that much or not, but it's plausible that
people want that; leaving a separate / not encrypted and encrypting all
the rest seems reasonable, but OTOH if cgdroot is easy enough that seems
better.

>> [...], whereas I think what you're proposing to add to base is an
>> interface to a standards-compliant (and somewhat-open) device
>> specification. Right?
>
> The _file format_ used by fidocrypt(1) is bespoke, based on sqlite3.

I can see why you did this, but I am not sure we want to put sqlite3 in
/.  I realize there are people who either have the one-big-filesystem
approach, and people with a merged / and /usr, compared to the
traditional Unix layout.  On most systems, I tend to have / and /usr
separate.  I believe NetBSD still supports the traditional layout and
considers it the standard approach.

Therefore I wonder if a sqlite3-based format is necessary, because while
I haven't read the code, I would expect this is just an encrypted key
plus one table with a row for {each device that stores a KEK}, which
seems easy enough for lines with printed hex and spaces, or a bunch of
structs, and whole-file-rewriting/rename to modify.  Or using proplib
which seems to fit.

But maybe that is a lot of work for no point and we shouldn't try to
avoid sqlite3 in /.  Maybe we're going to rewrite proplib to use
sqlite3; I dimly remember someone saying that but I am not sure if it
was a joke.

After looking at /lib and what is already there:

  I'm a bit horrified.

  sqlite3 doesn't seem as big as I thought, relatively speaking; it's
  0.9 MB more on 9.3 MB -- but it's still 10% growth.

So I guess it's somewhat 10% growth, and somewhat the disruption of
moving it, but if unpacking the sets and running postinstall fix is
fully satisfactory, that's not really disruption.

I would say that if you propose to move sqlite3 to /, that deserves a
thread with that as the title.  I don't personally object (I have pruned
off retrocomputing as a hobby), but I think it's a bigger deal than
adding fidocrypt.




signature.asc
Description: PGP signature

Re: disable HPN in sshd for the -10 branch?

[Greg Troxel]

mlel...@serpens.de (Michael van Elst) writes:

> Part of the HPN patches is to optionally strip encryption (and now even
> integrity checks) for the data transfer. Doesn't fit into what
> the OpenSSH people want, not even as an option.

I would say that doesn't really fit with what we want either, certainly
without somebody really trying.  It breaks the rule that using ssh can
count on confidentiality and integrity and makes systems with ssh as  a
component harder to reason about.


signature.asc
Description: PGP signature

Re: disable HPN in sshd for the -10 branch?

[Greg Troxel]

nia  writes:

> I've heard some reports that the HPN-SSH patches to sshd are
> not quite working as well as expected, with some users getting
> mildly worse results.  They're apparently supposed to improve
> performance:
>
> https://www.psc.edu/hpn-ssh-home/
>
> With "HPNDisabled" in sshd_config I notice no particular difference
> in sshd performance, but I don't have access to any particularly
> amazingly large machines currently.

I view HPN as not the standard approach; it hasn't been merged upstream
and PSC's agenda does not even seem to include merging any of it
upstream -- which I see as a huge clue.

My quick reaction is that unless we are confident that the HPN patches:

  - make things better (or neutral) for almost everybody
  - have zero downsides

then it is just as well to have them disabled by default in the config
file.  People with massively fast networks with long delays but no
congestion generally know who they are.

But it would be very interesting to hear test results from people both
ways.


signature.asc
Description: PGP signature

Re: sysupgrade in src?

[Greg Troxel]

nia  writes:

> On Fri, Apr 15, 2022 at 01:20:04PM -0400, Greg Troxel wrote:
>> Use of RCS IDs seems fragile/unsound in that you can't conclude from
>> matching IDs that the files match, or really the other way around, given
>> people storing sources in !cvs with local modes, local builds, etc.
>> (Not saying doing local things in !cvs is bad, just that the RCS ID
>> match assumption assumes more than is true.)
>> 
>> Seems like this should be based on a database of hashes of the old
>> upstream and the new upstream, as well as the previous version in /etc,
>> and perhaps have a way to record that the admin has agreed to the
>> current contents of a file in /etc.
>
> Unfortunately this is the 'state of the art' for the past 20 years,
> etcupdate is what NetBSD provides as stock (I'm not proposing to add
> it here because it's already been added). There is a lot of ways
> it can be improved, agreed, it's by far the most irritable part
> of the update process.

I agree that what you are proposing and improving etcupdate are
separable, and do not object at all to adding sysupgrade before fixing
everything about etcupdate.

I would suggest that those interested in etcupdate behavior try out
etcmanage so they can understand what they do and don't like about how
etcmanage does it.   I think etcmanage does well, except it needs:

  a way to produce a comprehensive diff of stock to installed, somehow
  avoiding generated files and things a human admin would say don't count

  three-way merge when old-stock/installed differs and
  old-stock/new-stock differs as well, in a way that is somehow aware of
  which files are safe to merge if there aren't untouched context lines
  and which aren't

  some way to list/show things that need a merge but for which the merge
  algorithm didn't do it.  (Arguably this should precede implementing
  merge, effectively using a 3-way merge algorithm that always says it
  can't merge safely.)


And etcupdate may well be close to right, and need a slightly different
set of improvments; this is a complicated lattice space.


signature.asc
Description: PGP signature

Re: sysupgrade in src?

[Greg Troxel]

nia  writes:

> On Fri, Apr 15, 2022 at 08:05:54AM -0400, Greg Troxel wrote:
>> To me, the right behavior is to know if each file in etc has been put
>> there as a copy of a file that appeared in etc.tgz, and to change it to
>> the new version without prompting if so, and if not, to note to the user
>> that it might need attention.  Perhaps sysupgrade/etcupdate does this,
>> and if so great.
>> 
>
> -a etcupdate can automatically update files which have not
>been modified locally.  The -a flag instructs etcupdate to
>store MD5 checksums in /var/etcupdate and use these
>checksums to determine if there have been any local
>modifications.

Seems like that should be the default.

> -l Automatically skip files with unchanged RCS IDs.  This has
>the effect of leaving alone files that have been altered
>locally but which have not been changed in the reference
>files.  Since this works using RCS IDs, files without RCS
>IDs will not be skipped even if only modified locally.
>This flag may be used together with the -a flag described
>above.

Use of RCS IDs seems fragile/unsound in that you can't conclude from
matching IDs that the files match, or really the other way around, given
people storing sources in !cvs with local modes, local builds, etc.
(Not saying doing local things in !cvs is bad, just that the RCS ID
match assumption assumes more than is true.)

Seems like this should be based on a database of hashes of the old
upstream and the new upstream, as well as the previous version in /etc,
and perhaps have a way to record that the admin has agreed to the
current contents of a file in /etc.

> Downside: this doesn't work for modified files without RCS IDs, which
> of course means that etcupdate prompts about /etc/passwd (etc.)
> every single time.

The other thing I should have said is that etcmanage never prompts: it
is totally non-interactive which means it can be run as a batch job to
ssh to 20 machines and update them all and then reboot them.  That's why
it got written (in 2004, when perl was ok).  It got spiffed up in
2010/2011 for a testbed of 50 machines (that would get a full system
update pretty often).

Yes, I know that's technically dangerous, but being able to see diffs
and reduce them when you want to pay attention, and have it be on
autopilot the rest of the time, has been in practice very reliable.
And having the admin do it by hand is also dangerous, for an admin that
is not infallible.   And there's only one of those, and they do things
other than sysadmin most of the time :-)



Perhaps I might figure out to improve etcupdate so it can do what I
want.  It seems like it is reasonably close.



signature.asc
Description: PGP signature

Re: sysupgrade in src?

[Greg Troxel]

I don't object at all, but I want to point out etcmanage, in pkgsrc,
(which needs xz help), and does the same thing.  A more or less fatal
downside is that it is in perl, but I think it has very good semantics
about etc merging.  I have not kept up with the other methods, but in
the past I found them too manual.

To me, the right behavior is to know if each file in etc has been put
there as a copy of a file that appeared in etc.tgz, and to change it to
the new version without prompting if so, and if not, to note to the user
that it might need attention.  Perhaps sysupgrade/etcupdate does this,
and if so great.

But either way, sysupgrade is something people use, and upstream not
doing maintenance and even worse requiring signing an agreement with
google is a good reason to fork it.




signature.asc
Description: PGP signature

Re: to bump or note to bump

[Greg Troxel]

Brett Lymn  writes:

> So, the point of all this blather is even though there have been no
> interface changes as a result of my work there is the potential for a
> significant visual change.  Should I mark this with a libcurses major
> bump? I don't think it necessary but I am putting it out there for
> discussion...

In packaging, I generally find that ABI bumps cause more work/problems
than people expect.  It feels to me like what you did was a bugfix, and
any program that relied on the old behavior is buggy.  So I lean to no
bump, and if something else needs fixing, so be it.


signature.asc
Description: PGP signature

Re: ZFS - mounting filesystems (Was CVS commit: src/etc)

[Greg Troxel]

Taylor R Campbell  writes:

>> Date: Thu, 17 Mar 2022 08:32:40 -0400
>> From: Greg Troxel 
>> 
>> Simon Burge  writes:
>> 
>> >  5. Move all local mounts to /etc/rc.d/mountcritlocal (ala
>> > FreeBSD) and possibly rename this to /etc/rc.d/mountlocal .
>> 
>> I think the only thing we lose with this is the ability to mount local
>> filesystems on top of mount points that are in remote filesystems,
>> without playing the noauto/rc.local game.  I am ok with this personally,
>> but it feels hard to be sure it won't cause trouble, and I do expect
>> some trouble.
>
> Does anyone actually do this -- have local mounts on top of remote
> mounts?
>
> I keep hearing about the theoretical possibility of /usr on nfs and
> /usr/src or /usr/local on local ffs.  But you'd really have to go out
> of your way to set that up -- certainly sysinst won't do that for you.
> Not sure it's too much to ask that if you do set something up like
> that you use noauto/rc.local or a local rc.d script to manage it.
>
> Is this obscure use case actually worth worrying about enough to add
> new knobs, invent new NetBSD-specific zfs properties, and/or keep a
> confusing series of mount stages in rc.d?
>
> I think it would be better to just nix the `critical' distinction:
> mount root, then mount all local, then start networking and mount all
> remote.  Keep it simple unless there's a strong reason not to.

It is really hard to assess the costs of these two paths over all users
and uses.

I don't object to mounting all local in mountcritlocal (and I don't care
if it is renamed to mounlocal).  I don't want to commit that personally,
since I don't want to bite off responsibility for fixing problems I
didn't anticipate.

Right now, mountcritremote comes before servers/etc. and the rest can
happen later, in mountall.  Right now for almost everyone
mountcritremote does nothing.  That does't have a compelling reason for
change, so there's only risk of breaking things we don't understand, and
I'm inclined to leave it.



signature.asc
Description: PGP signature

Re: ZFS - mounting filesystems (Was CVS commit: src/etc)

[Greg Troxel]

Simon Burge  writes:

> Lots of interesting discussion!  Thanks all.

As a loud ranter I'll comment briefly but thanks for the summary and I
think we're heading for a good place.

> Broadly I think I can summarise to the following options:
>
>  1. The existing critical_filesystems_zfs rc.conf variable, which
> mixes ZFS configuration in both rc.conf and with ZFS itself.

If this works, which it by all accounts seems to, the only real problem
is that some people (and I can see why) would like to manage the
critical property with a zfs property to keep all their zfs config in
the same place.  Am I misperceiving?

>  2. Add ZFS "critical" properties for filesystems and mount those
> in /etc/rc.d/mountcritlocal .
>  3. Move all ZFS mounts to /etc/rc.d/mountcritlocal .

3 is the only thing here I object to because it is architecturally
unclean, giving special semantics to zfs.

>  4. Optionally move all ZFS mounts to /etc/rc.d/mountcritlocal
> controlled by an rc.conf variable (eg zfs_critical).

I view this as syntactic sugar for putting all zfs mounts in the
critical_filesystems_zfs variable.  It's a little to close to option 3
for my taste, but I won't say I object.

>  5. Move all local mounts to /etc/rc.d/mountcritlocal (ala
> FreeBSD) and possibly rename this to /etc/rc.d/mountlocal .

I think the only thing we lose with this is the ability to mount local
filesystems on top of mount points that are in remote filesystems,
without playing the noauto/rc.local game.  I am ok with this personally,
but it feels hard to be sure it won't cause trouble, and I do expect
some trouble.

>  6. "Intelligent" ordering based on prefixes, tsort, etc.

> In terms of what to do for the up-and-coming netbsd-10 (although the
> actual release will still be a little whiles away), I think there
> appeared to be broad concensus on that option 2 was reasonable.  I will
> try to get an implementation of that ready.  If that's not ready in time
> for netbsd-10 I think option 4 is probably least intrusive fallback
> especially because it's optional.

It may be that after that is done, there is no real reason based on the
zfs concerns to change anything else.

> critical_filesystems_zfs is still available now, and will be available
> going forward.
>
> Option 3 doesn't seem to buy anything if option 4 is available and
> option 5 seems like it could be more impactful to existing mixed local
> and NFS setups.

Agreed.

> Option 6 I think is out of scope of what I want to get root-on-ZFS
> working.  That doesn't stop others from fleshing this out of course!
>
> Thoughts?  Anything I've missed?

I think you got it exactly right.

> And somewhat related that came out of this discussion - add a "noerror"
> or "notfatal" or some similar keyword that says a failed fsck or mount
> doesn't automatically fail the boot and fall back to single user mode.

That would be good.

> Although Ignatios had an interesting solution to this with "noauto" and
> using @reboot crontab entries.

Sure, you work around the lack of the feature, but that doesn't mean a
simpler way is bad.


signature.asc
Description: PGP signature

Re: ZFS - mounting filesystems (Was CVS commit: src/etc)

[Greg Troxel]

Brad Spencer  writes:

> Martin Husemann  writes:
>
>> On Tue, Mar 15, 2022 at 08:30:11AM -0400, Greg Troxel wrote:
>>> I still don't understand and object to this "zfs is special" notion.
>>
>> It is special because it just does not use /etc/fstab (usually).
>> I don't like this part either, but we probably don't want to make our
>> ZFS different from others (or force "legacy" markers).

I understand that.   But I  don't think it is justification to treat it
differently.   We have a design that's obviously ok (critical boolean),
and the only objections are:

  1) it's work to do it

  2) having tunables for which there is no point is just complexity for
  no gain.

However, point 2 applies to the entire notion of not mounting any local
filesystem early.

> The dynamic inheritance that one can do with filesets (allowing a new
> fileset to be carved out of an existing one perhaps with different
> properties) makes the use of the /etc files very troubled especially
> when you delegate fileset administration to someone other than root.
> The ZFS legacy marker more or less says that I don't want to do that
> with this entity and won't make use of these abilities.

Yes, but we aren't talking about prohibiting using normal zfs admin.
Just a simple way to logically add a zfs volume into
critical_filesystems_local with a property.

I realize why things have to be done a bit different, and I don't object
to that.  What I object to is "zfs is special and should have different
semantics from other local filesystems".


signature.asc
Description: PGP signature

Re: ZFS - mounting filesystems (Was CVS commit: src/etc)

[Greg Troxel]

I had another thought, which might very little work and make everybody
happy:

Add a variable zfs_critical, which if

  yes:  zfs mount -a happens in mountcritlocal
  no:   doesn't happen

and maybe

  not set:  zfs mount -a happens if root is zfs


I think I prefer the per-mountpoint property, but this seems useful with
less work, and should allow everything that works now to continue to
work.


signature.asc
Description: PGP signature

Re: ZFS - mounting filesystems (Was CVS commit: src/etc)

[Greg Troxel]

Brad Spencer  writes:

>> But seriously, I think tsorting the filesystems is necessary, and then
>> there's putting filesystems into one of
>>
>>   required
>>   optional
>>
>> and then some sort of label for
>>
>>   before-starting-networking
>>   after-networking-before-daemons
>>
>> and so on.
>
> Ya, that is mostly what I would suspect is desirable if you mean sorting
> by type.  The real world problem I see, however, is figuring out what
> the type is.  Failing to do that means making a hard choice..  in a
> practical sense it probably means sorting such that ZFS is first and
> then Not-ZFS or the other way around.

I still don't understand and object to this "zfs is special" notion.  If
it's a heuristic that ffs is mounted on zfs more than zfs is mounted on
non-root ffs, that's just a guess.  If it's something else, I really
don't understand.

I am suggesting labels for filesystems that

  1) need to be mounted before networking can start (because they
  contain files that are needed to start networking).   This is what
  mountcritlocal is for.

  2)  need to be mounted before daemons can start, but need networking
  to mount.   This is what mountcritremote is for.



There's a larger philosophical point which is if remote filesystems
matter *for the operation of systems*.   Surely remote filesystems
matter and should matter more over time -- it's kind of a bug that we
don't have fully satisfactory remote filesystems now (think disconnected
like coda, no kludges, good security properties, robust).

Storage has gotten plentiful, so one might say that the local system can
be local.  But, dealing with configuration is still hard, and remote
storage for computers is sort of like virtualization.  If we were to say
"there is no such concept as mountcritremote any more" that would mean

  if you are operating diskless (maybe for testing, maybe for lack of
  admin of local storage), then you have to have the whole system on /,
  because /usr and /var don't get mounted before daemons and thus cannot
  contain logging or config on remote filesystems (other than /), which 

  similarly for /usr/pkg probably in /.

This means no sharing of /usr, no sharing of /usr/pkg.  That sort of
sharing and ability to be read-only is part of the historical BSD design
and I think it still makes sense.


The thing that might make sense is to say that while all the above is
valid, there is no reason to refrain from mounting everything that is
local during the mountcritlocal phase.  Except that if you want to mount
local things on top of remote things.

Which brings me back to 1) tsorting the mountpoints and 2) having
mountcritlocal and mountcritremote properties to enforce the barriers.

So if we just add a mountcritlocal boolean property to zfs filesystems
(because we can't cleanly list them in critical_filesystems_local when
they are mounted from the zfs database), I think we're back to where we
were, which is still a pretty good place.

People with zfs / can put the critical property on zfs /var, /usr, and
maybe /usr/pkg, and on random things like /usr/obj can add it or not as
they please.  People who want to to stack remote/local in odd ways can
do that as well as they can today.






Some data about our current rc.d system.  Doing rcorder * and reading it
is useful, as is grepping for REQUIRES on key things.   (Surely there
must be a rcorder->graphviz script?)

Things that depend on mountcritlocal follow.  Note that routed is
missing from this list, probably as an optimization that it is in /sbin
which can be relied on to be mounted.

  bootconf.sh:# REQUIRE: mountcritlocal
  dhcpcd:# REQUIRE: network mountcritlocal
  ip6addrctl:# REQUIRE: root bootconf mountcritlocal tty
  ipfilter:# REQUIRE: root bootconf mountcritlocal tty
  ipsec:# REQUIRE: root bootconf mountcritlocal tty
  iscsid:# REQUIRE: NETWORKING mountcritlocal
  mdnsd:# REQUIRE: network mountcritlocal staticroute
  mountcritremote:# REQUIRE: NETWORKING root mountcritlocal
  network:# REQUIRE: ipfilter ipsec mountcritlocal root tty sysctl
  npf:# REQUIRE: root bootconf mountcritlocal tty network
  npf_boot:# REQUIRE: root bootconf mountcritlocal tty
  pf:# REQUIRE: root bootconf mountcritlocal tty network dhcpcd
  pf_boot:# REQUIRE: root bootconf mountcritlocal tty
  random_seed:# REQUIRE: mountcritlocal
  route6d:# REQUIRE: network mountcritlocal
  smtoff:# REQUIRE: root bootconf mountcritlocal tty
  wpa_supplicant:# REQUIRE: network mountcritlocal

These are the things that depend on mountcritrmote:

  SERVERS:# REQUIRE: mountcritremote
  altqd:# REQUIRE: mountcritremote
  dmesg:# REQUIRE: mountcritremote
  fccache:# REQUIRE: mountcritremote
  hostapd:# REQUIRE: mountcritremote
  ifwatchd:# REQUIRE: mountcritremote syslogd
  ipfs:# REQUIRE: ipnat mountcritremote downinterfaces
  ipnat:# REQUIRE: ipfilter mountcritremote
  mixerctl:# REQUIRE: mountcritremote
  motd:# REQUIRE: mountcritremote
  mountall:# REQUIRE: mountcritremote named ypbind
  named:# 

Re: ZFS - mounting filesystems (Was CVS commit: src/etc)

[Greg Troxel]

Robert Elz  writes:

> So, add me to the list of people who'd like a "mount if possible"
> switch, with nothing more than a boot warning if some of them cannot
> be found at all, or have unfixable fsck issues.

Agreed - that would be great.

> I'd actually prefer even more - for most of those, if the filesystem
> isn't clean, and it isn't just a matter of "replay the log" to deal
> with it, I'd prefer that the filesys simply be skipped until the
> system is up multi-user, and then the necessary fsck fix be attempted
> while the system is carrying on with its real work, and if successful
> the mount be performed later (so some of that junk might be missing for
> a while after a boot following an unclean shutdown, but most of it
> will come back later).

That could be ok, or to complicated - if it's optional just leaving it
for the admin seems fine.

> Certainly all this can be accomplished by ad-hoc scripts, but it
> seems like something that many people actually would use, and we'd
> benefit if this was done in a standard way.

Well said.


signature.asc
Description: PGP signature

Re: ZFS - mounting filesystems (Was CVS commit: src/etc)

[Greg Troxel]

Brad Spencer  writes:

> The point is that by need /usr/sources has to be realized in the system
> after / and /usr are available and right now those can't be a ZFS pool
> themselves (well, /usr COULD be, but I can't really see how / could).
> This, at least in my opinion, is not a complicated nor unusual use
> case.  However, if it were possible for / and /usr to be a ZFS fileset
> or pool then the requirements change a bit.

I think you have argued that it is necessary to sort filesystems
topologically, but you left out the patch :-)


But seriously, I think tsorting the filesystems is necessary, and then
there's putting filesystems into one of

  required
  optional

and then some sort of label for

  before-starting-networking
  after-networking-before-daemons

and so on.


signature.asc
Description: PGP signature

Re: ZFS - mounting filesystems (Was CVS commit: src/etc)

[Greg Troxel]

Paul Goyette  writes:

>> Do we have any valid need to have non-critical local filesystems?
>
> Well, I have a dedicated filesystem for builds, separate from my
> OS.  The /build happens to be my nvme SSD.
>
> Building (or being able to build) is not critical to having the
> machine running (and receiving mail).
>
> So, yeah, I think non-critical local filesystems are meaningful.

Yes, but I think the big question is:

  if that filesystem got mounted during boot where mountcritlocal
  happens now, woudl anything bad happen

  if that filesystem can't be mounted, does the boot fail now, just
  later, or is there some meaningful resilience.


I have a machine with a USB disk and I put /d0 as "noauto" and kick off
a script to try to mount it.  So if the USB disk fails, is disconnected,
doesn'tt have power, the (remote unattended) machine still boots.




signature.asc
Description: PGP signature

Re: ZFS - mounting filesystems (Was CVS commit: src/etc)

[Greg Troxel]

Simon Burge  writes:

> I'm using ZFS as my root filesystem, with the EFI boot loader reading
> the kernel from the ZFS root filesystem.  This is based on the the
> FreeBSD libsa ZFS code. https://github.com/snarkophilus/src/tree/zfsboot
> is this work, and I'm planning on merging into main NetBSD sources soon.

Very cool - I had no idea that was about to land.

> Later, Greg Troxel wrote:
>
>> It makes sense to have a boolean "critical" property (the name you
>> suggested is fine) for zfs volumes that specify a mount point, so that
>> such volumes would be mounted in mountcritlocal.  I am 100% happy for
>> someone to add that and see no big problems.
>
> I will look into doing more on this, depending on the outcome of
> discussion on just "mountlocal" below.

It would be a safer change, not doing a critical rototill.

> Another idea that came up in off-list discussion with Taylor is in the
> same vein as this - the whole notion of "critical".  So changing from
>
> mountcritlocal
> mountcritremote
> mountall
>
> to
>
> mountlocal
> mountcritremote
> mountremote
>
> or even just a simpler
>
> mountlocal
> mountremote
>
> Do we have any valid need to have non-critical local filesystems?

Historically this is tied up with two things:

  separate / and /usr (the longstanding tradition, which needs to be
  supporrted)

  diskless workstations and system partitions over NFS, especially where
  / and /usr are separate, because /usr is shared

critical filesystems are mounted before NETWORKING, and then mountall
happens before some other barrier before daemons start.  So the idea is
that critical filesystems are those needed to bring up networking at
least, and that other filesystems might be then mounted via a mechanism
that *requires networking*.

> For what it's worth, FreeBSD appears to do things in this order:
>
> /etc/rc.d/zpool
> load all cached zpools
> /etc/rc.d/zvol
> load zvols for swapping onto
> /etc/rc.d/root
> mount -uw /
> /etc/rc.d/mountcritlocal
> mount all non-NFS filesystems (except non-legacy ZFS)

So freebsd has decided to call all non-remote filesystems in fstab
during mountcritlocal.  That's basically deciding that there is no point
to refrain from mounting local filesystems (in fstab)

> /etc/rc.d/zfs
> zfs mount -a (and zfs share -a)
> [ ... much later on ... ]

this can be viewed as mountcritzfs and belonging in mountcritlocal

> /etc/rc.d/mountcritremote
> mount all NFS filesystems
> [ ... nearly last thing ... ]

and where does bringing up networking fit?

> /etc/rc.d/mountlate
> mount filesystems marked as "late" in /etc/fstab

wow, late - never heard of that.

> The "crit" in mountcritlocal and mountcritremote don't seem to have any
> current meaning, so is similar to just the mountlocal/mountremote idea
> above.

except that critlocal is before networking.


All that said, I don't see a real problem with deciding to mount all
local filesystems (marked auto of course)  at mouncritlocal time.I
can try to reason through diskless/nfs to see if that's really true.




signature.asc
Description: PGP signature

Re: math.h, copysign, visibility defines

[Greg Troxel]

Joerg Sonnenberger  writes:

> Am Thu, Feb 24, 2022 at 07:59:22AM -0500 schrieb Greg Troxel:
>> 
>> I'm trying to build wip/ocaml on NetBSD 9 amd64 and came across:
>> 
>>   gcc -c -O2 -fno-strict-aliasing -fwrapv  -pthread  -Wall 
>> -Wdeclaration-after-statement -Werror -fno-common 
>> -fexcess-precision=standard -fno-tree-vrp -fPIC -ffunction-sections -g -O2 
>> -fPIC -D_FORTIFY_SOURCE=2 -I/usr/include -D_FILE_OFFSET_BITS=64 
>> -DCAML_NAME_SPACE  -DCAMLDLLIMPORT= -I/usr/include -o floats.b.o floats.c
>>   floats.c: In function 'caml_copysign':
>>   floats.c:1074:10: error: implicit declaration of function 'copysign' 
>> [-Werror=implicit-function-declaration]
>>  return copysign(x, y);
>> ^~~~
>>   floats.c:1074:10: error: incompatible implicit declaration of built-in 
>> function 'copysign' [-Werror]
>>   floats.c:1074:10: note: include '' or provide a declaration of 
>> 'copysign'
>>   cc1: all warnings being treated as errors
>>   gmake[2]: *** [Makefile:366: floats.b.o] Error 1
>
> This was fixed in current almost exactly two years ago.

And apparently not fixed in the releaase branch, which is what we expect
most people to run :-)

Do you think it's wise to pullup of src/include/math.h 1.65-1.66?  If
so, could you request it (since you obviously understand these details
better than I do)?


signature.asc
Description: PGP signature

math.h, copysign, visibility defines

[Greg Troxel]

I'm trying to build wip/ocaml on NetBSD 9 amd64 and came across:

  gcc -c -O2 -fno-strict-aliasing -fwrapv  -pthread  -Wall 
-Wdeclaration-after-statement -Werror -fno-common -fexcess-precision=standard 
-fno-tree-vrp -fPIC -ffunction-sections -g -O2 -fPIC -D_FORTIFY_SOURCE=2 
-I/usr/include -D_FILE_OFFSET_BITS=64 -DCAML_NAME_SPACE  -DCAMLDLLIMPORT= 
-I/usr/include -o floats.b.o floats.c
  floats.c: In function 'caml_copysign':
  floats.c:1074:10: error: implicit declaration of function 'copysign' 
[-Werror=implicit-function-declaration]
 return copysign(x, y);
^~~~
  floats.c:1074:10: error: incompatible implicit declaration of built-in 
function 'copysign' [-Werror]
  floats.c:1074:10: note: include '' or provide a declaration of 
'copysign'
  cc1: all warnings being treated as errors
  gmake[2]: *** [Makefile:366: floats.b.o] Error 1


I found that floats.c set visibility defines:

  /* Needed for uselocale */
  #define _XOPEN_SOURCE 700

  /* Needed for strtod_l */
  #define _GNU_SOURCE

and if I added _NETBSD_SOURCE the build succeeded.  In general I have
the attitude that most use of visibility defines are wrong because
e.g. asking for _XOPEN_SOURCE 700 means that things not defined in that
standard  should be hidden, and thus other visibility defines must also
be turned on, but perhaps this is not one of those cases.

Looking, I find that POSIX says copysign is taken from C99

  https://pubs.opengroup.org/onlinepubs/9699919799/

Looking in math.h, I find that the copysign decl is guarded with
_NETBSD_SOURCE.   However copysignf is not guarded.


This test program results in a warning for copysign but not copysignf.
Without _XOPEN_SOURCE, or with _NETBSD_SOURCE, it compiles cleanly.


#define _XOPEN_SOURCE 700

#include 

int
main(int argc, char **argv)
{
  double a, x, y;
  float af, xf, yf;

  x = xf = 9.0;
  y = yf = -2.0;

  af = copysignf(xf, yf);
  a = copysign(x, y);

  return 0;
}

cs.c: In function ‘main’:
cs.c:15:7: warning: implicit declaration of function ‘copysign’ 
[-Wimplicit-function-declaration]
   a = copysign(x, y);
   ^~~~
cs.c:15:7: warning: incompatible implicit declaration of built-in function 
‘copysign’
cs.c:15:7: note: include ‘’ or provide a declaration of ‘copysign’



So I conclude that math.h is buggy on NetBSD 9.  But can I really be the first 
to notice?


signature.asc
Description: PGP signature

Re: openssl 3

[Greg Troxel]

nia  writes:

> There are likely problems mixing different OpenSSL shared object
> versions in pkgsrc, no?

If you mean openssl 1.1.1 in base, and (eventually) 3 in pkgsrc, with

/usr/lib/libopenssl.so.[1.1.1-version]
/usr/pkg/lib/libopenssl.so.[3-version]

and things in base linked with base, things in pkgsrc linked with
pkgsrc, and some mess with both in the same binary, then agreed that
this is going to be not good, but maybe not super terrible in practice.

> If NetBSD 10 is to have OpenSSL 1.1 I think it's critical we
> establish a flow for maintaining it in the long term (whether
> it's by pulling patches from Red Hat, etc), so it doesn't rot
> like netbsd-8.

Agreed.  I think we're headed for openssl 3 being security/openssl3 and
some way to select 1.1.1 vs 3 (globally for pkgsrc), with builtin
processing to use base isntead for both cases, and choosing pkgsrc
different from base not really being supported, but non-gross patches
that don't break other cases welcome.

That leaves the "who is going to patch 1.1.1 after openssl project
stops" open, but I think ti's the usual "people who care, grabbing
patches from other places that care" and we'll muddle through.

Does that sound reasonable to you?  To others?



signature.asc
Description: PGP signature

Re: openssl 3

[Greg Troxel]

Reinoud Zandijk  writes:

>> My impression is that work to make things build with 3.0.0 is in
>> progress, but that the fallout from a (not committed, in testing) switch
>> is over 5000 packages.  But I expect that will rapidly get better.
>> 
>>   
>> https://us-east.manta.joyent.com/pkgsrc/public/reports/trunk/bulktest/20210928.1522/meta/report.html
>
> Those is quite a lot of fallout.

yes.

> Just scouted a few, but some of the logs state that it thinks its building on
> x86_64-unknown-solaris2 (ghc, wayland). Is this intentional?

That's a bulk build by jperkin@, so it seems intentional.  Anyone is
welcome to do a bulk build on other systems, and that would be most welcome.

>> So if netbsd-current moves to openssl3 before pkgsrc is ready, that's
>> going to cause a lot of packages to break on current.   So then people
>> who care can fix them, as usual.
>
> Can't pkgsrc do a fallback? Ie if the package is not openssl3 compatible it
> can fall back on an openssl 1.1 in pkgsrc until upstream is converted?

So far no one has spoken up in favor of having two openssl
implementations at once.

We probably could allow people to select pkgsrc openssl on
netbsd-current, so packages get 1.1.1.  That may require other things
that use openssl to come from pkgsrc.


signature.asc
Description: PGP signature

Re: openssl 3

[Greg Troxel]

chris...@astron.com (Christos Zoulas) writes:

> My thoughts are:
> - It is too late to put OpenSSL-3.0.0 un current, to become part of NetBSD-10.
> - After the NetBSD-10 branch, I will move OpenSSL-1.1.1 to openssl.old and
>   import OpenSSL-3.0.0 in openssl. Every port will point to openssl.old.
> - I will provide OpenSSL-3.0.0 source compatibility to OpenSSL-1.1.1 if needed
>   (like I did for OpenSSL-1.1.x and OpenSSL-1.0.x) by adding the missing
>   functionality if needed (and if possible)
> - I will make HEAD work with both OpenSSL-3.0.0 and OpenSSL-1.1.1.
> - I will switch all ports to use OpenSSL-3.0.0
> - Unfortunately pkgsrc will suffer the same way it did in the previous
>   upgrade.

So, if it happens that way, then it really does not matter from the
NetBSD side if pkgsrc moves to 3 this branch, or adds 3 as an option
alongside 1.1.1, or doesn't do either.   But, as you switch things, it
will be good for pkgsrc to be able to use the native 3 and have other
packages build against it.


signature.asc
Description: PGP signature

Re: openssl 3

[Greg Troxel]

Martin Husemann  writes:

>> I don't know what you mean exactly.   Certainly at some point pkgsrc
>> builds on 9 will use pkgsrc openssl.  Perhaps long before that.   But I
>> don't see how e.g. postfix in base is going to use pkgsrc openssl.
>
> Right, so the admin of an affected machine would have to choose:
>
>  - update to newer netbsd
>  - update to use pkgsrc postfix
>  - live with outdated openssl

Which more or less amounts to "netbsd 9 is no longer supported".  So we
should plan to have 11 out by spring 2024 -- but we should be doing that
(or earlier) anyway.

(Personally, assuming 10 happens early/mid 2022, I expect to have
updated all my systems long before end of 2023.)


signature.asc
Description: PGP signature

Re: openssl 3

[Greg Troxel]

Martin Husemann  writes:

> On Thu, Sep 30, 2021 at 08:44:22AM -0400, Greg Troxel wrote:
>> What are people thinking about
>> 
>>   updating openssl to 3.0.0 in current
>
> Yes, someone(tm) should do that! Early to catch fallout quickly, but
> we'd need commitment from the pkgsrc team to make pkgsrc usefull with
> that. I don't think we need to sequence this, thing will follow naturally
> from people hitting more fallout.

My impression is that work to make things build with 3.0.0 is in
progress, but that the fallout from a (not committed, in testing) switch
is over 5000 packages.  But I expect that will rapidly get better.

  
https://us-east.manta.joyent.com/pkgsrc/public/reports/trunk/bulktest/20210928.1522/meta/report.html

So if netbsd-current moves to openssl3 before pkgsrc is ready, that's
going to cause a lot of packages to break on current.   So then people
who care can fix them, as usual.

The real question is what kind of pace of update is best, as maintained
upstreams are going to make releases that work with openssl3, and not
being first makes life easier.

I suspect in a few weeks we'll have a better idea.


>>   pulling up openssl 3 to 9?
>
> We can't do that. Instead, at some point (probably a bit past the two
> years 1.1 will still receive updates) we will have to bite and switch
> netbsd-9 over to pkgsrc based openssl.

I don't know what you mean exactly.   Certainly at some point pkgsrc
builds on 9 will use pkgsrc openssl.  Perhaps long before that.   But I
don't see how e.g. postfix in base is going to use pkgsrc openssl.

> It is a pitty that security projects like openssl don't care for ABI
> compatibility.

Indeed.  Even API compat would be a big improvement.



signature.asc
Description: PGP signature

openssl 3

[Greg Troxel]

This is a software engineering question, not a security question and
hence here.

openssl 3.0.0 is out, and it has a lot of compat issues.
I hear that openssl 1.1.1 only has two years of maintenance left.

history: 8 was released in July 2018 and 9 in february 2020.  At that
pace, 10 will be released in September 2021, but there are only 12 hours
left :-)

I observe that 10, if released in April 2022 (just making that up), can
be expected to need support until mid 2026.  And 9 will need support until
2024.

Hence, I'm going to ignore 8, as it will be out of support long before
1.1.1 is desupported upstream (but don't quote on that in fall of 2023).


What are people thinking about

  updating openssl to 3.0.0 in current

  if so, the effects on building pkgsrc and how to sequence that

  pulling up openssl 3 to 9?


I am guessing:

  pkgsrc needs to be able to cope with 3.0.0 first

  openssl 3.0.0 should go in current, for 10

  9 and esp 8 will not get pullups.  It's an ABI break and not allowed.
  

(Asking with pkgsrc-pmc hat on as we have similar questions in pkgsrc
and all of this is a bit tangled.)



signature.asc
Description: PGP signature

Re: Java servers and rc.d?

[Greg Troxel]

Paul Ripke  writes:

> Another thought:
> Should _find_processes in /etc/rc.subr pay attention to a deamon's
> configured user?

I'm not quite ready to conclude that, but I agree that we should have a
way that rc.d scripts for programs that use java can work correctly.

> I run unifi from pkgsrc, which is java. I've also got a couple of
> minecraft servers (also java) running to keep my son happy. I've just
> discovered that the default rc.d script for unifi thinks that unifi
> is running if any of the minecraft servers are running.

The root cause here is the java attitude that people should invoke
programs in java via a java command line that they have to
semi-understand and carry from documentation, rather than having an
executable -- even if a script that encapsulates this.  But of course
that's too hard to fix in general.

I wonder how hard it would be to have /usr/pkg/bin/unifi that takes
start and stop and has the long java command, and which then might be
matchable via ps.

> They both run under different users - should find_processes pay
> attention to the configured user? Do we have a solution for java
> servers and procname in rc.d?

Your suggestion seems like a reasonable workaround.

One could match on not just $command but also $command_args.  While
"start" is a traditional ccommand argument, the rest are really code
that belongs in a program rather than command-line args, and this could
make the match specific enough.

Another possible approach is to add a "status" operator along with stop
to unifi, but that is likely harder.

A patch to improve this would be welcome for consideration.


signature.asc
Description: PGP signature

Re: inetd Enhancements Followup

[Greg Troxel]

James Browning  writes:

> maintainers at their own discretion. Some of you brought up the concern of
> over-automating the system at the potential risk of the configuration system
> becoming too opaque and administrators allowing packages to configure inetd
> without their knowledge of what is happening. However, we believe that with

As I read it, that was mostly mouse@.  My take was that most of the
concerns were about not making it super complicated and maintaining
backwards compat.


signature.asc
Description: PGP signature

Re: inetd Enhancements

[Greg Troxel]

Mouse  writes:

> Of course, any setup can ultimately be understood.  But the more
> complexity there is, the harder that is to do; and the more automation
> is provided by someone else, the more it encourages administration
> without understanding - in extreme cases it actively obstructs
> administration *with* understanding.  (While I haven't seen it often, I
> have seen people asking about underlying mechanisms answered with,
> basically, "just use the automated tool".  While there is a place for
> automation, using it as a substitute for understanding is, in my
> opinion, a disaster in the making.)

I am sympathetic, but a directory of fragments is very thin syntactic
sugar over having it all in a file.  It does mean that "update this file
if it hasn't changed" is likely to work on each fragment, rather than
failing on something which is a textual collision but not a semantic
one.

If what  you object to is programs coming with  default configs that are
active without the admin passing a test, then please say that.   Making
the config less fragile  about things that are textual but not semantic
merge conflicts isn't the core problem, and we shouldn't not do that
because of concern it might be abused.  That would be like banning cars
because they might be used in a bank robbery.


signature.asc
Description: PGP signature

Re: inetd Enhancements

[Greg Troxel]

Keep in mind as you consider "changing" rather than "extending" that
there are many systems with existing files, and the NetBSD way is that
if you upgrade the system to a new version, then as little breakage as
possible should ensue.  Yes, technically you should read all the release
notes, and you should merge the differences from all updated etc files
to your etc, but in reality this often does not happen.

So if this is more like

  add a new line to something like ".includedir /etc/inetd.d" which says
  to act is if every file in that directory was part of inetd.conf

  perhaps, have additional syntax so that a line that starts with some
  special token introduces a config stanza that is logically equivalent
  to a current line


The big point is that the new inetd, when run with an old config file,
should behave as we would have hoped, not fail to start anything, and
not failing open.

  


signature.asc
Description: PGP signature

Re: ssp, __strcpy_ck: just to be sure

[Greg Troxel]

tlaro...@polynum.com writes:

> On Tue, Nov 17, 2020 at 10:16:47AM -0500, Greg Troxel wrote:
>> 
>> tlaro...@polynum.com writes:
>> 
>> > But then I realized that I have not changed the offending line: the
>> > strcpy() call since there was no overflow problem: it was doing inplace
>> > rewrite, suppressing a prefix (doing basename(1) essentially).
>> 
>> Not sure what "inplace" means, but it sounds like it runs afoul of:
>
> simply something like : strcpy(buf, buf + shift);

That is undefined behavior, and thus wrong and needs to be fixed,
fortify or no.

>> and thus it is fair to alert on it.
>
> Yes, but the question was that it alerts only with arrays---but this is
> perhaps simply because it would be too costly to call strlen(3) or
> equivalent at runtime, and so, when the size is not known at compilation
> time, the alert doesn't work.

Perfectly ok to have a discussion  about how the tool can be improved,
but the code is UB and thus an alert is fair, being a member of the set
of all behaviors.


signature.asc
Description: PGP signature

Re: ssp, __strcpy_ck: just to be sure

[Greg Troxel]

tlaro...@polynum.com writes:

> But then I realized that I have not changed the offending line: the
> strcpy() call since there was no overflow problem: it was doing inplace
> rewrite, suppressing a prefix (doing basename(1) essentially).

Not sure what "inplace" means, but it sounds like it runs afoul of:

  If copying takes place between objects that overlap, the behavior is
  undefined.

and thus it is fair to alert on it.

See:

https://pubs.opengroup.org/onlinepubs/9699919799/functions/strcpy.html

and there is similar language in our man page.


signature.asc
Description: PGP signature

Re: Proposal to remove catman(8)

[Greg Troxel]

Kamil Rytarowski  writes:

> I wish good luck finding user-base/target-audience (if you like, in any
> age) that relies on the slowest of slow hardware and cannot use anything
> else to study the system documentation.

You are missing the other larger point.   It may well be that catpages
should go entirely.

However, the other thing that people pay attention to when deciding to
be part of a community is how people conduct themselves.  Aggressive
removals in the face of objections is something that is likely to cause
people to find a community with higher standards.


signature.asc
Description: PGP signature

Re: Proposal to remove catman(8)

[Greg Troxel]

Kamil Rytarowski  writes:

> I am surprised that the proposal to remove MK${FOO} is read as removal
> of the Makefile conditionals and keep ${FOO} in the base. With that
> bizarre interpretation the whole proposal renders into useless idea.
>
> I would be very surprised to interpret that e.g. proposal to remove
> MKX11 would not mean to remove X11 from the base but to enable it by
> default.

That is a ridiculous strawman and not the same thing at all.  MKX11
means to build X11.  MKCATPAGES means to have the step of generating
catpages at OS release time.  We don't have a switch to say "don't build
the tools that could make catpages".   I thought it was obvious that
your proposal was just "remove the build.sh/Makefile glue that generates
catpages at release time, and setlist contents that expects them",
nothing more.


signature.asc
Description: PGP signature

Re: Proposal to remove catman(8)

[Greg Troxel]

Kamil Rytarowski  writes:

> It's not a selling point to any regular user, born after A.D. 2000 to
> optimize reading man pages.

This ageist comment is offensive and not appropriate on NetBSD lists.

It's also remarkably off base; the notion that most NetBSD users are
less than 20 years old seems entirely disconnected from my perception.


signature.asc
Description: PGP signature

Re: Proposal to import window(1) into the base

[Greg Troxel]

Tobias Nygren  writes:

> On Thu, 22 Oct 2020 17:12:39 +0200
> Kamil Rytarowski  wrote:
>
>> I propose to import window(1) into the base.
>> 
>> tmux is a similar program to tmux, but much simpler and traditional in
>> the BSD environment. Personally I use tmux as a screen replacement for
>> detached consoles, but for management of windows I prefer window(1).
>> window(1) does one thing and does it well.
>
> window(1) was removed in NetBSD-6.0, almost 9 years ago. I think most
> users have, begrudgingly or not, migrated to tmux(1). There is little
> merit in resurrecting window after this much time has passed. Unless
> it covers a specific use case that tmux cannot support my opinion is
> that it should stay in pkgsrc.

I basically agree with tnn@.  Base should be on one hand minimal and
functional on the other. Plus, I think tmux is actually quite easy to
use in a simple manner.


signature.asc
Description: PGP signature

Re: package build failures on netbsd-8 using nbtar, pax, bsdtar

[Greg Troxel]

Greg Troxel  writes:

> This is about a system with netbsd-8 built with sources updated on
> 2020-02-14.
>
> I updated pkgsrc, and several sphinx packages are failing to build.
> With help from adam@, I have narrowed it down to bad extraction from the
> upstream tarball.

>From on and offlist comments, the consensus is that our pax does not
understand newer format headers that are appearing, and hance pax-as-tar
(NetBSD <= 8) does not either.

Thus, packages like this need to use gtar or tar-with-libarchive, found
in NetBSD >=9.

Moving to tech-pkg to talk about how best to do this.

Re: package build failures on netbsd-8 using nbtar, pax, bsdtar

[Greg Troxel]

tlaro...@polynum.com writes:

>> tar: File extended headers posix ustar archive. Extracting as plain files. 
>> Following files might be in the wrong directory or have wrong attributes.
>> --  1 root wheel156 Dec 31  1969 ././@PaxHeader
>> -rw-r--r--  1 502  staff   1007 Dec 31  1969 
>> sphinxcontrib-serializinghtml-1.1.4/sphinxcontrib/serializinghtml/locales/zh_TW/LC_MESSAGES/sphinxco
>
> Have you seen the dates? It pre-dates the epoq? Is it supported? Because
> this seems to imply that the date is negative (probably -1)...

The epoch started at 1900 on December 31, 1969, in my timezone.

But good point  that this seems odd.

package build failures on netbsd-8 using nbtar, pax, bsdtar

[Greg Troxel]

This is about a system with netbsd-8 built with sources updated on
2020-02-14.

I updated pkgsrc, and several sphinx packages are failing to build.
With help from adam@, I have narrowed it down to bad extraction from the
upstream tarball.

The package is /usr/pkgsrc/textproc/py-sphinxcontrib-serializinghtml
which was recently updated to 1.1.4.  The 1.1.3 tarball looks fine with
NetBSD's /bin/tar but the 1.1.4 throws lots of:

tar: File extended headers posix ustar archive. Extracting as plain files. 
Following files might be in the wrong directory or have wrong attributes.
--  1 root wheel 27 Dec 31  1969 ././@PaxHeader
drwxr-xr-x  2 502  staff  0 Dec 31  1969 
sphinxcontrib-serializinghtml-1.1.4/sphinxcontrib/serializinghtml/locales/zh_TW
tar: File extended headers posix ustar archive. Extracting as plain files. 
Following files might be in the wrong directory or have wrong attributes.
--  1 root wheel 28 Dec 31  1969 ././@PaxHeader
drwxr-xr-x  2 502  staff  0 Dec 31  1969 
sphinxcontrib-serializinghtml-1.1.4/sphinxcontrib/serializinghtml/locales/zh_TW/LC_MESSAGES
tar: File extended headers posix ustar archive. Extracting as plain files. 
Following files might be in the wrong directory or have wrong attributes.
--  1 root wheel156 Dec 31  1969 ././@PaxHeader
-rw-r--r--  1 502  staff755 Dec 31  1969 
sphinxcontrib-serializinghtml-1.1.4/sphinxcontrib/serializinghtml/locales/zh_TW/LC_MESSAGES/sphinxco
tar: File extended headers posix ustar archive. Extracting as plain files. 
Following files might be in the wrong directory or have wrong attributes.
--  1 root wheel156 Dec 31  1969 ././@PaxHeader
-rw-r--r--  1 502  staff   1007 Dec 31  1969 
sphinxcontrib-serializinghtml-1.1.4/sphinxcontrib/serializinghtml/locales/zh_TW/LC_MESSAGES/sphinxco

and there is indeed a "sphinxco" file when unpacked.

Using gtar to extract results in an OK build.

So I think something is wrong with the tar on netbsd-8, and I'm not sure
what we should be doing.

The files can be grabbed from:

https://pypi.org/project/sphinxcontrib-serializinghtml/#history
https://files.pythonhosted.org/packages/cd/cc/fd7d17cfae18e5a92564bb899bc05e13260d7a633f3cffdaad4e5f3ce46a/sphinxcontrib-serializinghtml-1.1.3.tar.gz
https://files.pythonhosted.org/packages/ac/86/021876a9dd4eac9dae0b1d454d848acbd56d5574d350d0f835043b5ac2cd/sphinxcontrib-serializinghtml-1.1.4.tar.gz


(There are similar problems with a few other sphinxcontrib-foo packages,
and I'm guessing theya are the same thing.)

Re: Expose max_align_t unconditionally

[Greg Troxel]

Kamil Rytarowski  writes:

> On 09.03.2020 20:05, Martin Husemann wrote:
>> This is pretty stupid, but IMHO no big deal. We can
>> 
> [...]
>>  - just not support (i.e. #error) on older C++ standard compiles
>
> There are still many programs in pkgsrc that set older C++ release. Some
> of them are hard to upgrade.
>
> $ git grep c++03|wc -l
>  206
> $ git grep c++98|wc -l
>9

Given that there are multiple sublanguages, a program written (more or
less correctly) in an older one that specifies --std for it seems
entirely ok, and not in need of fixing.  True?

Re: Expose max_align_t unconditionally

[Greg Troxel]

Michał Górny  writes:

> Given that Clang defaults to C++14 nowadays, I suppose it could be fixed
> by finding old -std= and replacing it.

I am not really following the issue, but I don't understand why what the
default C++ sublanguage is matters.  It would seem that behavior should
be correct for any --std passed to compilers, per that sublanguages's
specification, and what you get by default is just what happens to you
if you don't pass --std.  (And I basically think a  build that does not
pass an explicit --std is buggy, but that's a separate issue.)

Re: Solving the syslogd problem

[Greg Troxel]

Piotr Meyer  writes:

> On Wed, Jan 29, 2020 at 09:32:30PM +, Alexander Nasonov wrote:
> [...]
>
>> I like it when fsck doesn't take ages to check /. With bigger /,
>> it's going to be problematic.
>
> IMVHO moving /usr/bin, /usr/sbin and /usr/lib to / is reasonable,
> and it will not lead to excessive growth of / - but /usr should
> remain as separated mountpoint, convenient place for extra storage
> needed for src/, xsrc/, pkgsrc/, share/, X11R7/, local/, pkg/ and
> - in some cases - home/ directories.

I think this is the crux of it.  People put things in /usr, like
/usr/pkg, and it is normal that /usr be large.

So "moving /usr/bin to /" just can't be done.  If you mean "move all
programs in /usr/bin to /bin", that's something else.

We have longstannding practice, and at this point I'd favor the smallest
changes to fix the actual problems.

remquo(3) issues

[Greg Troxel]

# summary

(Note that unless you already understand remquo or you read the docs I
am pointing to very carefully, this is not going to make a lot of
sense.)

While running tests for PROJ 6.3.0 (wip/proj) I got some failures.  PROJ
includes geodesticlib and has replacement code for C99-required math
functions, or can use the system-provided functions.  I narrowed the
failure down to using the replacement vs system remquo, and found that
they returned different results.   I wrote a test program, but I am not
convinced that my test program provokes the root cause of the
PROJ/geographiclib failure.

The issues are the precise semantics of the returned partial quotient

  https://pubs.opengroup.org/onlinepubs/9699919799/functions/remquo.html

and in C++, not in use here, but ~obviously text trying to say the same
thing:

  https://en.cppreference.com/w/cpp/numeric/math/remquo

So for the language lawyers, assuming n=3 and thus modulo 8, it seems
that if the quotient is 0, then the answers 0, 8, 16, and so on are all
entirely acceptable.

For arguments x, y of -45.0, 90, the quotient is -0.5, and hence the
sign is negative.

We do not appear to have any atf tests exercising remquo.

It seems obviously in order to look at our remquo implementation and
compare to FreeBSD's.

So I wonder if anyone understands this.  Once again, I hope I am not the
expert!

# results from PROJ regression testing

PROJ regression tests (geodtest, specifically) failed on netbsd-8 amd64,
and passed on macos 10.13.  They also pass on Linux and FreeBSD, say the
PROJ people and their CI infrastructure.

I get some lines (see my printf below) when input is nan, and n (in this
code, the modulo-2^3 quotient) differs.  However, I suspect the partial
quotient is not valid for a nan input.

  remquo mismatch delta nan z nan z_c99 nan n 0 n_c99 2146959360 IN nan 
90.00
  remquo mismatch delta nan z nan z_c99 nan n 0 n_c99 1 IN nan 90.00
  remquo mismatch delta nan z nan z_c99 nan n 0 n_c99 0 IN nan 90.00
  remquo mismatch delta nan z nan z_c99 nan n 0 n_c99 0 IN nan 90.00
  remquo mismatch delta nan z nan z_c99 nan n 0 n_c99 0 IN nan 90.00
  remquo mismatch delta nan z nan z_c99 nan n 0 n_c99 30716 IN nan 90.00
  remquo mismatch delta nan z nan z_c99 nan n 0 n_c99 30716 IN nan 90.00

I also get a lot of lines that look like:

  remquo mismatch delta 0.00 z 39.214461 z_c99 39.214461 n 8 n_c99 0 IN 
39.214461 90.00

which appear to be an odd choice of n, but not wrong according to the
spec as I read it.

Here is my modified remquox (with original copyright), which basically is the 
code in PROJ and
then a second call to the system remquo and comparison.

/*  

   
 * This is a C implementation of the geodesic algorithms described in   

   
 *  

   
 *   C. F. F. Karney,   

   
 *   Algorithms for geodesics,  

   
 *   J. Geodesy 87, 43--55 (2013);   

   
 *   https://doi.org/10.1007/s00190-012-0578-z  

   
 *   Addenda: https://geographiclib.sourceforge.io/geod-addenda.html

   
 *  

   
 * See the comments in geodesic.h for documentation.

   
 *  

   
 * Copyright (c) Charles Karney (2012-2019)  and licensed   

   
 * under the MIT/X11 License.  For more information, see

   
 * 

wip/proj test failures, possible C99 math trouble

[Greg Troxel]

proj is a program/library for geographic coordinate transforms.  There
is a new version in wip/proj, 6.3.0.  (This is not yet in pkgsrc proper
because of compat issues with programs that depend on proj, but that's
orthogonal to this message.)

proj can use extra data files, but test vectors are without, so one runs
proj tests as

  make PKG_OPTIONS.proj=-proj-grids test

Through 6.2.1, this passed, and 6.3.0 has new tests and new reliance on
some C99 math.  Now, the "geodtest" program fails, with wrong
calculation results.   This is on netbsd-8/amd64.  On 8/i386 I get the
same failures, but not quite the same wrong value in one test.

paranoia is ok, but that doesn't test the math functions in question.

These tests pass on Linux, and proj's CI also successfully tests on
FreeBSD 11, MSVC 2015 & 2017 and MacOSX.  That doesn't 100% prove this
is a NetBSD bug, but it's quite strong fingerpointing.

I built with (in wip/proj/Makefile)

  CFLAGS+=-DHAVE_C99_MATH=0
  
which causes src/geodesic.c to replace some C99 functions, and then the
tests pass.

I hope I am not the math expert at the moment!

I will test on 8/earmv7hf-el.

I will see if bisecting on the various "if c99, use it, otherwise use
this replacement" stanzas is useful.

I am curious about test results from 9 or current, or other
architectures.

I also wonder if anyone knows of C99 math test suites.

Thanks,
Greg

Re: stdio.h's P_tmpdir definition - make it /tmp?

[Greg Troxel]

u...@stderr.spb.ru (Valery Ushakov) writes:

> Greg Troxel  wrote:
>
>> It seems that long long ago, /tmp was small and on /, and /var was often
>> bigger, and people used /var/tmp for larger stuff.   Also I remember a
>> notion of clearing /tmp on boot and not clearing /var/tmp.   Now, /tmp
>> is a tmpfs and large/fast, and the right place for things that really
>> belong in /tmp.
>
> That's true for some people/systems.  Not true for others.

A fair point, but I think it's also a reasonable expectation that you
have to have a /tmp that's big enough for what you want to do, vs the
expectation that programs that use space use some other tmp by default.

> libiberty does check TMPDIR as the very first thing.  If the build
> breaks that, may be the build should be fixed?

And, I agree that programs that use /tmp or especially programs that use
a lot of tmp space, should respect TMPDIR.

But here, I think we're talking about changing the OS default in stdio
that /tmp is the default place, absent config, and by omission it sounds
like you are ok with that?
\

Re: stdio.h's P_tmpdir definition - make it /tmp?

[Greg Troxel]

Thor Lancelot Simon  writes:

> On Fri, Aug 30, 2019 at 02:40:06PM +1000, matthew green wrote:
>> hi folks.
>> 
>> 
>> a very long time ago in netbsd years tls@ patched GCC to use
>> /tmp over /var/tmp, for the compiler temporaries.
>> 
>> i tried to keep these patches, but they got lost at some
>> point and when i recently wanted to deal with it (i had one
>> machine with some several hundred /var/tmp/cc*.o files that
>> were mostly over a year old), i discovered it's not nearly
>> as simple as a quick patch.
>> 
>> most of the toolchain consumers that create temp files don't
>> use $TMPDIR from the env but use stdio.h's P_tmpdir defintion
>
>
> Well, this is... well... I'm embarassed I missed it.  I think
> you should change it now.
>
> Only thing to watch out for I suppose is that vi's expectation
> of where vi.recover goes doesn't change -- though that should
> never have even been in /var/tmp in the first place.

The change sounds good to me.

It seems that long long ago, /tmp was small and on /, and /var was often
bigger, and people used /var/tmp for larger stuff.   Also I remember a
notion of clearing /tmp on boot and not clearing /var/tmp.   Now, /tmp
is a tmpfs and large/fast, and the right place for things that really
belong in /tmp.  /var/tmp should probably go away, but that's harder.

Re: More compatibility for refuse

[Greg Troxel]

co...@sdf.org writes:

I too was not happy about the notion that librefuse should be
deprecated.

> librefuse is matching the newer FUSE APIs. it's not matching the older.
> The filesystems want the older high-level API.

I have looked into this slightly.  I had the impression from the list
that filesystems/fuse had at any time a single high-level API revision,
and did not support backwards compat.  That, combined with an unstable
API, didn't seem like a good idea, since it imposes a duty of prompt
updating on all fuse clients and leads to a need for synchronized
updates.  I was wondering why, if that were true, it made sense for
librefuse to be more accomodating.

But, as I understand it now, filesystems/fuse has a default API and a
way for fuse clients to declare their API version.

I think you're suggesting that librefuse should have the same kind of
compatibility, at least for one rev back, via the same declaration.
Then we will have packages building cleanly with either, and not needing
to carry so many patches.  And librefuse will be a more complete and
thus better implementation of the family of FUSE APIs.  Did I get that
right?

If I did understand, that seems like obviously a good thing to happen,
and the only question is who feels like doing it.

> Or like ntfs-3g they mysteriously have duplicate code to use the
> lowlevel API. That can be avoided and still provide NTFS functionality
> (but no configure way to do it).

> We should provide a package with just the parts of ntfs-3g only using
> the high level FUSE API because other FUSE libraries also don't provide
> the low level API (like openbsd).

It would be nice to fix fuse-ntfs-3g to only use high-level API (perhaps
by default, if there's no really good reason to use the low-level API).
Ideally this would be fixed upstream.

Re: More compatibility for refuse

[Greg Troxel]

m...@netbsd.org (Emmanuel Dreyfus) writes:

>  wrote:
>
>> I'm just going to commit code without code review if all the responses
>> are going to be "please don't work on X, work on Y instead".
>> I am interested in librefuse because that's what gets used.
>
> Sure, go ahead for the short term, but my point is that in the long run,
> your life could be easier by just linking FUSE filesystems with the real
> libfuse instead of librefuse. 

I am not following two things:

  Do you think that (for things that use the high-level API) it is
  better to use (upstream, via pkgsrc) libfuse and perfuse, instead of
  using librefuse?  If so, could you explain why?  It seems to me to add
  a layer of indirection and a daemon.

  It seems the underlying problem is that upstream libfuse is not good
  about compatibility, and thus filesystems that haven't update to the
  30 API are troubled.

Re: More compatibility for refuse

[Greg Troxel]

Taylor R Campbell  writes:

>> Date: Wed, 10 Apr 2019 19:50:02 +
>> From: co...@sdf.org

>> I'm just going to commit code without code review if all the responses
>> are going to be "please don't work on X, work on Y instead".
>> I am interested in librefuse because that's what gets used.
>
> May I suggest the following rephrasing?
>
> `Cool, thanks, how do I make fuse-ntfs-3g and fuse-ext2 use it?
> What's the costs/benefits to using libfuse via perfused, versus
> librefuse?  Is there a reason to prefer one over the other going
> forward?'

That's a fair question and request.  But, in this case, separately from
valid concerns about discourse, I think the situation is:

librefuse via puffs is and has been the standard approach on NetBSD for
high-level API access.  Using the high-level API is the standard
approach for filesystems (I am aware of only gluster needing the
low-level API).

As I understand things, the FUSE high-level API defined by the FUSE
project is not as stable as one would like, and backwards compat is not
as good as one would like.  Therefore filesystems more or less have to
stay current with the FUSE API (but apparently some don't quite).

Therefore:

  perfuse has to stay and be supported (but there is no actual
  discussion otherwise, just a single question)

  librefuse should support the current high-level API.

  There is no real need for compat to older APIs, in that filesystems
  that would benefit from that compat won't run without patching on
  other operating systems.

  It would perhaps be nice to allow building against older APIs,
  basically emulating what upstream FUSE probably should have done, and
  I gather opinions differ about the merits of this.


So:

  Obviously maya@ thinks that having compat for old APIs in librefuse is
  good.

  I am unclear on whether the cognitive load is worth the gain, as in
  pkgsrc any filesystem needs to be updated to the current FUSE
  high-level API anyway.  Given that it seems like not a lot of lines of
  .h/.c, I don't see much harm.

  I don't know if manu@ objects to API compat, or merely objects to the
  notion the perfuse isn't necessary.

  I don't know what anybody else thinks about API compat (in our
  librefuse; opinions about the wisdom of upstream are sort of
  interesting but not really on point).


signature.asc
Description: PGP signature

Re: More compatibility for refuse

[Greg Troxel]

co...@sdf.org writes:

> No, upstream FUSE loves having every downstream filesystem get adjusted
> whenever it decides an argument isn't useful any more.

So you are proposing to provide more compat than mainstream fuse, and we
would add a -D argument to packages that need it, on NetBSD only, and
they would continue to fail on other-than-NetBSD?

Is the problem that uptreams of useful things are dead, and this is
easier than patching them?

Sorry if I am being dense, but the big picture is missing.

Re: More compatibility for refuse

[Greg Troxel]

co...@sdf.org writes:

> On Wed, Apr 10, 2019 at 02:16:30PM +, co...@sdf.org wrote:
>>  
>> +#if FUSE_USE_VERSION < 30
>> +#define fuse_unmount fuse_unmount_compat30
>> +#endif
>
> Open to (FUSE_USE_VERSION < 30) && (FUSE_USE_VERSION > 22) :-)

Don't you need defined(FUSE_USE_VERSION) also?  Or is the fuse rule that
all programs that include fuse.h have to define that anyway?

Re: More compatibility for refuse

[Greg Troxel]

co...@sdf.org writes:

> If anything we should get rid of perfuse.

Certainly not.  It is there because gluster uses /dev/fuse directly, or
via some "low level" API, rather than using the standard FUSE API (the
"high level" API).  I am unclear on why, but my impression is that there
are good reasons vs it being gratuitous.  So perfuse - which is only in
pkgsrc - has to stay.

But, that's not really related to the best way forward here.

Is it a fair characterization that what you are proposing is about
adding support for older versions of the API, to accomodate programs
that are not written to the current FUSE API version?

Other than a bit more header defs, and a few compat functions, are there
downsides?

Is this notion of compat APIs normal in FUSE?  With real modern fuse,
are the old API functions available always, or does one have to define
something?

Re: More compatibility for refuse

[Greg Troxel]

co...@sdf.org writes:

> On Wed, Apr 10, 2019 at 02:22:22PM +, Emmanuel Dreyfus wrote:
>> On Wed, Apr 10, 2019 at 02:16:30PM +, co...@sdf.org wrote:
>> > Tell me what you think. I will commit it soon otherwise.
>> 
>> You know we support vnilla libfuse through libperfuse, right?
>> 
>> -- 
>> Emmanuel Dreyfus
>> m...@netbsd.org
>
> No.
> If there's some alternative interface that works better, why doesn't
> pkgsrc use it?

Works better is complicated.

libperfuse ends up with a daemon, as I understand it, and real FUSE
user-space libraries.

It seems that having FUSE API compat in librefuse is a good thing, if it
doesn't cause trouble.

Re: Pressure sensors

[Greg Troxel]

Artturi Alm  writes:

> On Tue, Jan 01, 2019 at 12:03:31PM -0500, Greg Troxel wrote:
>> So, I wonder why you are choosing kPa vs hPa, and if that choice ends 
>> up being a framework choice for everything.  And how the rest of the
>> world deals with this issue.
>
> By accident, i guess, and after having read about pascals on wiki[0],
> it felt like a good choice, for being "not just for meteorologists",
> nor imperial, but i'm from metric .eu, fwiw..

Interesting article, and it seems right.

Atmospheric pressure is basically meteorology, IMHO.

> I actually got *= 1000 for mPa in the driver, so i don't really care
> about the unit to be used, as long as it's not loosing any precision/
> limiting range (for kernel -> user, so Pa would work just as well, and
> i'm not sure whether mPa does buy anything w/r.t. future sensors, it
> was more about minimal diff looking reasonable).

The comments about integers, range and precision are compelling.  It's
hard to imagine anything in envsys delivering more than 1 Pa of
precision.

I didn't mean to give you a really hard time about this.   It just felt
like a decision that was likely to control how all envsys pressure
reporting is done in NetBSD, for all time.  But maybe I'm misperceiving
that.

Re: Pressure sensors

[Greg Troxel]

Martin Husemann  writes:

> On Wed, Jan 02, 2019 at 06:38:21PM +, David Holland wrote:
>> Most of the world uses millibars and occasionally labels them hPa.
>> Since judging by the patch the framework isn't up to handling both,
>> the best course is probably to use and label as millibars and just
>> duck the occasional brickbat from metrication nuts.

That's probably fair, and I resemble being called a metrication nut.

I do agree that millibar is normal usage in the US.

> I would make it Pa in the API and let userland translate to whatever
> values we want to display by default.

That seems somewhat reasonable, except that in my experience everybody
wants to see atmospheric pressure in mb/hPa (if they aren't using inches
or mm of Hg).

Re: Proposed modification to seq

[Greg Troxel]

Brian Ginsbach  writes:

> It has been a while since I wrote that code but my recollection is
> that it isn't necessarily a bug. That GNU copied and changed the
> meaning of -s (again provided my recollection is correct) isn't
> surprising either. I'd need to dig back to see what GNU seq had 20
> years ago when I originally wrote seq.
>
> I will agree that the -s behavior may violate POLA.
>
> The default "separator" is a newline ('\n').  The -s was to change
> this to something else but not assume that the last "separator" be
> a terminating newline.  This is why I added the -t option.
>
> The current -s option allows for using interesting separators like
> '\r' (carriage return) for a "spinning counter".
>
> Note that FreeBSD has picked up the NetBSD version of seq (and by
> that so has Apple for OSX). GNU shouldn't necessarily be considered
> as the 'standard'.

Interesting about FreeBSD and MacOS, and history.

So would you propose changing the man page to describe the traditional
BSD seq behavior instead?

Re: Proposed modification to seq

[Greg Troxel]

a...@100acres.us writes:

> The seq command behaves a little differently than I expect.  I needed a comma
> separated list of integers, but seq gave me this:
>
>  $seq -s , 1 3
>  1,2,3,$
>
> Notice the extra comma and no trailing return.  The comma is troublesome for 
> my

>From reading the man page, the extra comma is a bug, and the lack of a
trailing return is correct.

> particular purpose.  It seems that the item separator (-s) is really an item
> terminator.  I was expecting this:
>
>  $seq -s , 1 3
>  1,2,3
>  $
>
> I'm sure there is some not-obvious-to-me reason for the current behavior.

It could just be a bug.  But the man page is inconsistent.

I suspect there is almost no use of -s.  But I wonder.

If this change makes seq with -s behave like GNU seq, and more closely
aligns with seq's own man page, that seems like a reasonable thing.

> However, I implemented what I expected and build a distribution without issue.
> The attached diff implements the proposed behavior.
>
> Aran

> Index: seq.1
> ===
> RCS file: /cvsroot/src/usr.bin/seq/seq.1,v
> retrieving revision 1.9
> diff -u -r1.9 seq.1
> --- seq.1 18 Aug 2016 22:55:28 -  1.9
> +++ seq.1 13 Dec 2018 17:35:48 -
> @@ -117,8 +117,7 @@
>  can contain character escape sequences in backslash notation as
>  defined in
>  .St -ansiC .
> -This option is useful when the default separator
> -does not contain a
> +The default is
>  .Cm \en .
>  .It Fl w
>  Equalize the widths of all numbers by padding with zeros as necessary.
> Index: seq.c
> ===
> RCS file: /cvsroot/src/usr.bin/seq/seq.c,v
> retrieving revision 1.10
> diff -u -r1.10 seq.c
> --- seq.c 29 Oct 2017 01:28:46 -  1.10
> +++ seq.c 13 Dec 2018 17:35:48 -
> @@ -132,7 +132,7 @@
>   fprintf(stderr,
>   "usage: %s [-w] [-f format] [-s string] [-t string] [first 
> [incr]] last\n",
>   getprogname());
> - exit(1);
> +exit(1);

Why the whitespace change?

The rest seems ok, except I'm not sure what happens if the for loop
would not run the first time, but perhaps that can't happen.

Re: Moving telnet/telnetd from base to pkgsrc

[Greg Troxel]

Taylor R Campbell  writes:

> Given that a large fraction of respondents (though not all) indicated
> that their primary use of telnet is to test reachability of a server
> or manually enter SMTP or HTTP requests over the internet -- a use
> which is adequately served by the much smaller and much more
> confidence-inspiring usr.bin/nc -- I think this _does_ constitute a
> serious danger that warrants the scrutiny it is getting.
>
> [*] Whether it can lead to arbitrary code execution, I don't know, and
> I'm not interested in studying further to find out; it doesn't
> take much to get arbitrary code execution, like a single null byte
> heap buffer overflow:
> 
> https://googleprojectzero.blogspot.com/2014/08/the-poisoned-nul-byte-2014-edition.html

If somebody knows the details of such a bug and wants to fix it, that
seems uncontroversial.  But we don't seem to be talking about that.

Re: Moving telnet/telnetd from base to pkgsrc

[Greg Troxel]

> What's the deal wiht IPSEC?

The protoocol is called IPsec (and often miscapitalized), and our kernel
option is IPSEC.

> I've never used it, but I was under the impression it gives encryption
> for free for things that otherwise don't have it.

It provides confidentiality and data origin authentication at the IP
level, via a per-packet protocol called Encapsulating Security Protocol.

In this respect it is sort of like TLS, but operating at the IP layer
rather than the TCP layer.

However, implementations of it are OS services, rather than code in user
space.  (But the key management is in user space.)

> Do all the programs need to have ipsec-specific goo to use it? telnet
> does, as well as having its own encryption code.

No.  One configures the use of IPsec via Security Policy Database
entries, which in NetBSD are managed via setkey(8).

The encryption is telnet is I believe Kerberos.  Kerberos predates IPsec
by a lot, and is based on symmetric cryptography only (which is all that
was feasible in the early 80s).   As far as I know, Kerberos processing
is always done within the application program rather than being a kernel
service.

Re: Moving telnet/telnetd from base to pkgsrc

[Greg Troxel]

Robert Elz  writes:

> It does no harm as it is, if you don't use the client, all it does is
> occupy a couple of hundred blocks (nothing), the server is not
> enabled by default, and it is even smaller.

I agree.  I use it often, to see if TCP ports are open and hand-type
smtp or http.

Another point is that as a BSD system, we at least used to have a
respect for history and tradition.  That has to have a balance - we did
get rid of sendmail.  But removing longstanding programs (that don't run
by default) because some people don't like them, as part of what feels
like a larger deletionist crusade, is not ok.

tr -C and posix

[Greg Troxel]

Our tr, at least in 8, does not seem to implement the -C flag.

Reading posix:

  https://pubs.opengroup.org/onlinepubs/9699919799/utilities/tr.html

I am a bit lost in a twisty maze of "value" vs "character", that I
expect has something to do with multibyte characters.

Can anyone explain where NetBSD is and might be going w.r.t. this
specification?

Re: Default PAGER -> less

[Greg Troxel]

co...@sdf.org writes:

> I thought of making the default .profile snippet set PAGER=less.
> thoughts?

I tend to think we shouldn't have a default .profile at all.  (I don't
use the default file, and I don't set PAGER in my own files.)   But I
see that it contains only PATH.

In general, I think people tend to configure dotfiles how they want
them, and that we should avoid changing defaults and imposing choices on
people.  Of course not setting things is a choice, but BSD systems have
a longstanding practice of following tradition when it's arguable.



signature.asc
Description: PGP signature

Re: Shipping SSL certificates in the base system

[Greg Troxel]

Benny Siegert  writes:

> The question of root certificates for OpenSSL in base came up recently
> in pkgsrc. That got me thinking: why does NetBSD not come with a set
> of certificates in the base system? The set that mozilla-rootcerts
> delivers would be a reasonable thing to put there, because
> (a) that’s what literally everyone ends up installing anyway and 
> (b) it does not require us to make a moral judgement about individual CAs.

The comparision to tzdata is not quite right.  Timezones are just facts
about what names mean.   The mozilla CA set, not configured as trust
anchors, is arguably the same conceptually.  But once configured as
trust anchors, it's a trust decision.   So it's like the ssh
fingerprints for TNF hosts in /etc/ssh/ssh_known_hosts, but with a level
of indirection.

I agree that the perl issue is easy to address.  Maybe kre can rewrite
the script in sh/sed/awk :-)

Overall, I think this is a difficult issue.  Part of the problem is that
the whole CA situation is a bit surreal, having a large number of CAs
that are in theory all trustworthy when logic defies that conclusion.
But, it is how people validate X.509.

There are several questions that I think need answering as part of a
proposal to add the mozilla set:

1) What do other Free opereating systems do?  What was their thought
process in terms of balancing convenience, good engineering judgement
and security?  How has it worked out?

2) Do any programs in the base system validate certificate chains, or
fail to accept unvalidated certificates, by default?  If not, why is this
a base issue?  Or are you also proposing to change those defaults?

3) Do other operating systems just use the mozilla set?  One
controversial issue is the US government CA hierarchy, which I run into
on government sites.  As I understand it, they have't met the mozilla
criteria, but they seem well run, and the risk of government
misbhehavior seems significant for all CAs associated with governments
or in countries where government/CA is blurred, and I have the
impression quite a few CAs for which government misbehavior is a
rational concern are in the mozilla set.  Probably the same issue exists
for other national CAs.


I'm not really oppposed, more very reluctant to conclude this is ok, but
I'm not sure that's rational.


signature.asc
Description: PGP signature