Re: getconf LONG_BIT in NetBSD
At Sun, 19 May 2024 22:08:27 +0200, Ramiro Aceves wrote:
Subject: getconf LONG_BIT in NetBSD
>
> I have been playing with an autoconf
> ./configure script that I want get running
> in several OSes. The script uses "getconf
> LONG_BIT" to get the bits of the system.
What is the purpose of this test in the context where it is needed?
I would think that if it's a C program then there's no need ever for any
autoconf test. Just write pure portable C that uses properly.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpie9pZv4ybc.pgp
Description: OpenPGP Digital Signature
Re: Mail delivery from Postfix to remote IMAP
At Tue, 23 Apr 2024 01:41:11 +0200, Steffen Nurpmeso wrote:
Subject: Re: Mail delivery from Postfix to remote IMAP
>
> SPF should never have been introduced
I agree _VERY_ much! It still does absolutely nothing to reduce SMTP
abuse or increase trust in any way whatsoever.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpXhX5pfLz6i.pgp
Description: OpenPGP Digital Signature
Re: Mail delivery from Postfix to remote IMAP
First off let me second what Steffen says!
At Mon, 22 Apr 2024 21:15:08 +0200, Rhialto wrote:
Subject: Re: Mail delivery from Postfix to remote IMAP
>
> and neither can you change the
> envelope FROM address because bounces (as far as they happen) won't work.
I haven't verified this works right with Postfix, but if you're doing
forwarding with ~/.forward files then this should happen automatically.
It does of course mean bounces do end up going to the account on the
forwarding host, not the original sender, but this is (in theory) what
people using ~/.forward files want -- the forwarding itself caused the
bounce, not the initial delivery to the forwarded account, so sending
the message back to the original sender is arguably wrong.
Maybe you can increase your storage capacity and simply run local IMAP
service for all your domains and users? Every modern IMAP client (MUA)
I've encountered has been able to easily handle multiple IMAP accounts,
and many of them have simple ways to aggregate all INBOXes, for example,
into one meta INBOX.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgplmxEfEXlMt.pgp
Description: OpenPGP Digital Signature
Re: Mail delivery from Postfix to remote IMAP
At Mon, 22 Apr 2024 19:39:05 +0200, Rhialto wrote: Subject: Re: Mail delivery from Postfix to remote IMAP > > On Mon 22 Apr 2024 at 19:29:13 +0200, Hauke Fath (SPG) wrote: > > You would have to have a local delivery > > agent that (working with stored user credentials) also acts as an IMAP > > _client_. Interesting concept... > > I'd expect some little program already exists that, say, takes a mail > message on stdin and puts it in some configured imap mailbox... but I > haven't found one yet. Then I could just use an alias: > > u...@isp.tld: "|imap-delivery user:passw...@imap.isp.tld" That's what a local delivery agent does, effectively, but without needing the each user's credentials. Local delivery agents work on a different premise. The LDA uses its own master key, so to speak, to authenticate and authorize the MTA to do delivery to _any_ _local_ mailbox. I don't know the specifics of Dovecot, but in the Cyrus IMAP server it's called "lmtpd", and it uses the LMTP protocol, and it uses (or can use) LMTP AUTH to authenticate the MTA, and it can be configured to listen either locally on a UNIX (filesystem) socket, or on an internet socket (IP:port). However no mailbox provider in their right mind would ever allow any third party to have LMTP access to their IMAP server! The whole point of allowing a network connection to LMTP is so that you can have a farm of SMTP servers all accepting incoming mail and delivering to one IMAP server. I once maintained a system with a half dozen incoming SMTP servers all feeding one big IMAP server. SMTP goes between unaffiliated systems. LMTP is "local" to one . There are programs that can mirror IMAP mailboxes between IMAP servers, (e.g. mail/isync) but I think that's a different use case than yours. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpcIV51Rytvl.pgp Description: OpenPGP Digital Signature
Re: Mail delivery from Postfix to remote IMAP
At Mon, 22 Apr 2024 18:27:05 +0200, Rhialto wrote:
Subject: Mail delivery from Postfix to remote IMAP
>
> I have a local IMAP server running as well, so I could deliver to that
> and then do an IMAP-move, but that seems rather roundabout and requires
> creating some service which gets somehow triggered to do this. I don't
> have much space available though. A direct delivery would seem better to
> me.
Just keep doing what you're doing. Anything else _is_ more roundabout.
Why complicate things? SMTP forwarding is the way to keep it working!
Of course fixing your mail server to do proper DKIM, or even just
futzing with SPF (and PTR) records enough to get normal SMTP port#25
through, i.e. without heavier AUTH and use of the submission service,
would be even simpler. I've done the latter, and hope to do more with
DKIM soon (but _NOT_ with the milter mess!).
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpmA9lRU9qc9.pgp
Description: OpenPGP Digital Signature
Re: compile kernel
At Tue, 9 Apr 2024 07:24:11 +, Todd Gruhn wrote:
Subject: Re: compile kernel
>
> On Mon, Apr 8, 2024 at 4:56 PM Brad Spencer wrote:
> >
> > [personal nit... it would be nice if the isa bus would be eliminated
> > entirely, but there appears to be an edge case when compiling a
> > XEN3_DOM0 kernel that requires it to be there... don't remember why...]
>
> I saw this. Split it. I only use 'pciide* at pci? ...' AND
> 'ahcisata* at pci? ...'
I've got several production and test servers (Dell machines), running
Xen, that need pkcb0:
pckbc0 at isa0 port 0x60-0x64
Some also have com ports on isa0 (here only one as Xen gets the other):
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, 1-byte FIFO
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpTmx1KFLoLI.pgp
Description: OpenPGP Digital Signature
Re: Official cloud/live images ?
At Mon, 25 Mar 2024 20:53:49 +0100, "Enrico Weigelt, metux IT consult"
wrote:
Subject: Official cloud/live images ?
>
> are there any official live images that allow direct login via ssh
> (no password), which can directly be used for cloud / continous
> integration jobs ?
I would think that would be a very bad idea for a publicly distributed
official OS image!
> I'm currently setting up CI jobs for building Xorg on NetBSD, but I've
> only found an amd64 live image, where sshd is pretty locked down
> (no root login, etc), so I had to manually log in on console and change
> sshd config. That's quite unpleasant - I'd rather directly use the
> official release images instead.
I would think it should be trivial to take a copy of the official image,
modify it as desired, then use that modified copy for one's own uses.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpATMipTjTLS.pgp
Description: OpenPGP Digital Signature
Re: x86_64 assembly Hello World
At Sun, 11 Feb 2024 10:39:36 +0100, Fekete Zoltán
wrote:
Subject: x86_64 assembly Hello World
>
> I have played with GNU as and ld, and
> subsequently created a "Hello World!"
> program, which I could not find anywhere
> else so far.
Very nice!
Here's my hello.s written in x86_64 assembler for NetBSD:
https://github.com/robohack/experiments/blob/master/thello.s
Therein you will find a reference to the following page from whence I
borrowed most of the implementation:
https://polprog.net/blog/netbsdasmprog/
Also here's my example true(1) and false(1) written in x86_64 assembler:
https://github.com/robohack/experiments/blob/master/ttrue.s
https://github.com/robohack/experiments/blob/master/tfalse.s
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgp7uoDGQGKR6.pgp
Description: OpenPGP Digital Signature
Re: Reverse of promoting to root: downgrade root to unprivileged
t out then it could be possible to add this feature to pkg_comp. You'd need another general pkgsrc makefile target to get the list of dependencies in the correct order such that they can be built and then installed in a sandbox from binary packages. The print-build-depends-list is close, but not sorted in the necessary order (and it currently includes verbiage just for humans). So, for pkgsrc, the library or tool you're looking for has its beginnings in pkg_comp, but there's still work to do. > The question arises when I asked (wanting to write something for my > own): OK, but _what_ unprivileged user exists that I can safely "su" > to and accomplish the unprivileged part as? "nobody" does not seem the > answer; "operator" neither. This opened a can of worm-questions ;-) Indeed, "nobody" has, or at least had, a very specific and singular purpose (though it seems some programs are misusing it now). Typically one creates a new user for each specific "unprivileged" operation or group of operations. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpTJ6O0YyZ1I.pgp Description: OpenPGP Digital Signature
Re: Reverse of promoting to root: downgrade root to unprivileged
At Sat, 27 Jan 2024 20:00:24 +0100, tlaro...@kergis.com wrote:
Subject: Reverse of promoting to root: downgrade root to unprivileged
>
> Starting some operation as common user (for example compiling/building)
> before promoting to privileged (generally root) by su'ing or sudo'ing
> (for example to install) is common.
>
> But does somebody know of an established program or library that allows
> to start a process as root and to automatically downgrade rights for
> tasks (I mean identified chunks of whatever code) that do not require
> privileges?
Lots of programs that are run as root do this by design, e.g. login(1)
as well as daemons like cron(8), sshd(8), etc.
There are also other system programs that start as setuid-root (or some
other special-purpose user) in order to do some privileged operation,
such as opening a protected socket or file, and then return to running
as the invoking user or some other (possibly less privileged) UID. This
is exactly what su(1) does in fact.
(There are also a number of programs following a largely mistaken and
dangerous idea that they should swap back and forth between running in
privileged mode and running as the user, some to an absurd extreme, like
lpr(1). This is obviously not safe and is a pure idiotic fallacy. The
kernel _should_ force a processes that drop privileges to permanently do
so and to never try to regain them except through execve(2), as indeed
earlier real Unix(TM) kernels always did, as does my NetBSD variant.)
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpdoSaFqm2dD.pgp
Description: OpenPGP Digital Signature
Re: groff issue after upgrade to NetBSD-9.2
At Fri, 4 Mar 2022 17:26:23 + (UTC), st...@prd.co.uk (Steve Blinkhorn) wrote: Subject: Re: groff issue after upgrade to NetBSD-9.2 > > Unpacking the textproc set overwrites files like > /usr/share/groff_font/devps/DESC and devps/download, and maybe other > files which have been adapted or expanded locally. The unpacking > process follows any symbolic link that devps has been set to rather > than overwriting the symbolic link with a hard directory. Fortunately > I have backups. Would this not be worth a warning in the installation > guide - it's a similar issue to /etc, where precious lolcalisations > risk being lost? Yeah, I would say most of those are not normally files that any end user would be expected to localise. I think the best you can hope for is, perhaps, in a future upgrade if/when syspkgs are used, that there may someday be some conflict detection for locally modified system files. That said, you could also add any system files you've customised to /etc/mtree/special.local and they'll be backed up, with complete daily automatic version control, into /var/backups by /etc/security. See "check_changelist" in security.conf(5). > I know thered is a move not to includee groff etc. in the main > distribution, but some of us use it extensively: I have substantial > software systems which emit *roff source files, it's not just a > manpage generator. Perhaps you would be a lot happier with a more modern troff? I would suggest trying out pkgsrc/textproc/heirloom-doctools Despite the name, these are quite modernised versions of the original true AT Troff and related tools from what was effectively the Documenter's Workbench. These tools even have a special "groff" compatability mode if indeed you depend on any Groff extensions. See https://n-t-roff.github.io/heirloom/doctools.html (There is also a port of old DWB-3 (3.3b) in pkgsrc/textproc/DWB, but it has not been modernised nearly so much.) One potentially huge advantage of using doctools over the base-system groff would be that you can much more easily customise (and test!) the tools and their configuration by applying local patches via pkgsrc. That said I've long argued for these heirloom-doctools to be used to replace the base system Groff, and I would still strongly suggest that be done. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpgQ8QLh5CkY.pgp Description: OpenPGP Digital Signature
Re: Release
At Sun, 19 Dec 2021 20:23:20 -0500, Greg Troxel wrote: Subject: Re: Release > > What's messy is the idea that when replying to the list one should send > to *only* the list. That has some merit, but the standards are murkier > (Mail-Followup-To:) and implementation of them somewhat sparse. Well, no, there's nothing murky about it _in_the_standards_, even going all of the way back to RFC-822. It's called "Reply-To": 4.4.3. REPLY-TO / RESENT-REPLY-TO This field provides a general mechanism for indicating any mailbox(es) to which responses are to be sent. [[ ... ]] A somewhat different use may be of some help to "text message teleconferencing" groups equipped with automatic distribution services: include the address of that service in the "Reply- To" field of all messages submitted to the teleconference; then participants can "reply" to conference submissions to guarantee the correct distribution of any submission of their own. (To be even more pedantic, "Mail-Followup-To", and the even more bogus "mail-reply-to" are stupid inventions by people who didn't understand RFC 822 clearly enough, and were, in some part, clueless attempts to abuse Usenet headers that were somewhat over-specified again by people who apparently didn't understand RFC 822 clearly enough. Of course some of the problem was exacerbated by software that had been designed and implemented by people who didn't understand (or maybe appreciate) RFC 822 clearly enough, which sadly included BSD mail and some mailing list software.) -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgp0C_BlNgW_z.pgp Description: OpenPGP Digital Signature
Re: mount_union(8) vs. open(O_RDWR)
At Sun, 5 Dec 2021 10:38:54 +0100, "J. Hannken-Illjes"
wrote:
Subject: Re: mount_union(8) vs. open(O_RDWR)
>
> There are (up to) three vnodes involved:
>
> - The union vnode attached to the union mount.
Ah, OK, I think I understand.
Just to be sure: Does that mean a vnode representing the "virtual"
object in the union filesystem itself? I.e. the vnode resulting from a
namei() lookup within the namespace of the union filesystem?
> Operation union_access() reuses "vp", it starts as union vnode
> and then gets assigned the upper and then the lower vnode making
> this operation difficult to read.
Indeed, that code is unnecessarily obtuse.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpM9Q5e9iHwc.pgp
Description: OpenPGP Digital Signature
Re: mount_union(8) vs. open(O_RDWR)
At Sat, 4 Dec 2021 10:25:27 +0100, "J. Hannken-Illjes" wrote: Subject: Re: mount_union(8) vs. open(O_RDWR) > > Ok some comments: > > - union_copyup() works and is needed for regular nodes only. > > - it copies the node attributes up so checking the access on > the (copied) upper node should be sufficient. > > - the switch() on top checks for read-only mounted upper layer, > not the lower layer. Ah ha! That was not obvious to me at all. I assumed, incorrectly apparently, that (struct vop_access_args).a_vp would be the vnode for the lower layer in the case I was interested in since the upper one did not yet exist. So what is "a_vp" pointing to in the case where the file does not yet exist in the upper layer? vnodeops(9) says (for VOP_ACCESS) "The argument vp is the vnode of the file to check", but if it doesn't exist yet (i.e. hasn't been copied up yet)??? (in the "recursive" calls made by union_access() it is indeed either the upper vnode or lower vnode, depending on what is going to be used) > The attached diff just copies up in the VWRITE/VREG case and > this should be sufficient -- please give it a try. It works perfectly, thank you very much! -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpUYQcLnc8ke.pgp Description: OpenPGP Digital Signature
Re: mount_union(8) vs. open(O_RDWR)
un->un_path);
+#endif
return (EROFS);
+ }
break;
case VBAD:
case VBLK:
@@ -744,7 +808,14 @@
if ((vp = un->un_uppervp) != NULLVP) {
ap->a_vp = vp;
- return (VCALL(vp, VOFFSET(vop_access), ap));
+ error = VCALL(vp, VOFFSET(vop_access), ap);
+#ifdef UNION_DIAGNOSTIC
+ if (error) {
+ printf("union_access: failed upper access check for %s:
%d\n",
+ un->un_path, error);
+ }
+#endif
+ return (error);
}
if ((vp = un->un_lowervp) != NULLVP) {
@@ -758,8 +829,13 @@
}
}
VOP_UNLOCK(vp);
- if (error)
+ if (error) {
+#ifdef UNION_DIAGNOSTIC
+ printf("union_access: failed lower access check for %s:
%d\n",
+ un->un_path, error);
+#endif
return (error);
+ }
}
return (error);
@@ -1231,7 +1307,7 @@
*/
vp = NULLVP;
if (dun->un_uppervp == NULLVP)
-panic("union: null upperdvp?");
+panic("union: null uppervp?");
error = relookup(ap->a_dvp, , ap->a_cnp, 0);
if (error) {
VOP_UNLOCK(ap->a_vp);
@@ -1242,6 +1318,10 @@
* The name we want to create has
* mysteriously appeared (a race?)
*/
+#ifdef UNION_DIAGNOSTIC
+ printf("union_link: %s: already
exists?\n",
+ un->un_path);
+#endif
error = EEXIST;
VOP_UNLOCK(ap->a_vp);
vput(vp);
[ 2.3112534] root file system type: cd9660
[ 2.3262533] kern.module.path=/stand/amd64/9.99.81/modules
[ 2.3362553] warning: no /dev/console
[ 2.3362553] warning: no /dev/constty
Created tmpfs /dev (2064384 byte, 4000 inodes)
[ 4.2142574] union_mount(mp = 0x888007b3e000)
[ 4.2142574] union_mount: from :/tmp/uvar, on /var
[ 4.2142574] union_statvfs(mp = 0x888007b3e000, lvp = 0x888008cfcd40,
uvp = 0x888008cfcac0)
[ 4.3342534] union_newsize: wtmpx: lower size now 0
[ 4.3352568] union: copied up wtmpx
[ 4.3352568] union_newsize: wtmpx: upper size now 520
[ 4.3352568] union_newsize: wtmp: lower size now 0
[ 4.3352568] union: copied up wtmp
[ 4.3402531] union_newsize: wtmp: upper size now 0
[ 4.3452562] union_newsize: wtmp: upper size now 40
[ 4.3502564] union_newsize: utmpx: lower size now 0
[ 4.3502564] union: copied up utmpx
[ 4.3502564] union_newsize: utmpx: upper size now 0
[ 4.3602577] union_newsize: utmpx: upper size now 520
[ 4.3602577] union_newsize: utmpx: upper size now 1040
[ 4.3602577] union_newsize: utmpx: upper size now 1560
[ 4.3742532] union_newsize: utmpx: upper size now 2080
init: kernel security level changed from 0 to 1
[ 4.4112513] union_newsize: utmpx: upper size now 2600
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpyyCmQZR7Zx.pgp
Description: OpenPGP Digital Signature
Re: mount_union(8) vs. open(O_RDWR)
tories?), with the caveat that their manual page still warns of the seriously experimental nature of their code. I've yet to try their code. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgp9daI8VmLUT.pgp Description: OpenPGP Digital Signature
mount_union(8) vs. open(O_RDWR)
s vaguely saying? For the record this is with a (slightly dated) 9.99.81 kernel on amd64. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpi8Dxi5kce7.pgp Description: OpenPGP Digital Signature
Re: what is systems programming?
At Sun, 28 Nov 2021 05:04:44 GMT, Mayuresh Kathe wrote:
Subject: what is systems programming?
>
> i have heard a lot of references to systems
> programming throughout my time as a
> programmer in the past 30 years, but even
> on inquiring haven't received a clear and
> simple answer to what makes a systems
> programmer.
I don't agree with the whole of the Wikipedia article about "Systems
Programming", but I think this part is an excellent summary:
The primary distinguishing characteristic of systems programming
when compared to application programming is that application
programming aims to produce software which provides services to the
user directly (e.g. word processor), whereas systems programming
aims to produce software and software platforms which provide
services to other software.
The Encyclopedia Britannica description is also reasonably good (though
I would definitely leave out the phrase "especially as used in computer
networks" -- that is not a requirement):
Systems programming: Development of computer software that is part
of a computer operating system or other control program, especially
as used in computer networks. Systems programming covers data and
program management, including operating systems, control programs,
network software, and database management systems.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgp_qspCGQ7R8.pgp
Description: OpenPGP Digital Signature
Re: Modern Cyrus IMAP and Apache Guacamole on NetBSD?
At Thu, 9 Sep 2021 12:20:15 +0200, Matthias Petermann
wrote:
Subject: Re: Modern Cyrus IMAP and Apache Guacamole on NetBSD?
>
>
> In the future, I would like
> to bring this to a newer version - 3.4.2
> is now stable.
A new IMAPd is one of the last remaining requirements I need to get
working before I upgrade my enormously old version.
I've just managed to finally get a successful build of Cyrus IMAPd 3.4.2
on a reasonably recent NetBSD x86_64 machine.
I have not yet tried to run it -- I need to make a pkgsrc module with my
few necessary patches and then install it on a test machine.
It still after all these years has some remarkably egregious portability
and build problems. Turning on a few extra compiler warnings generated
nearly 200,000 lines of output during the build (though perhaps 90% of
it was repeats due to the errors being in common headers).
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpAjzw8JMy41.pgp
Description: OpenPGP Digital Signature
Re: Compiling NetBSD-HEAD kernel sources
At Thu, 1 Jul 2021 14:43:26 +0200 (CEST), neit...@hackett.marshlabs.gaertner.de (Martin Neitzel) wrote: Subject: Re: Compiling NetBSD-HEAD kernel sources > > > I *think* the correct way is to make your _own_, almost empty kernel > > config, and include (say, `GENERIC'.) then override settings in your own > > config. I've done it, but somehow it feels more hazy than plain > > modifying GENERIC > > Even easier: the stock GENERIC config contains the conditional include > > # Pull in optional local configuration > cinclude "arch/amd64/conf/GENERIC.local" > > So just create a GENERIC.local file with your tweaks to build > an "almost-GENERIC" kernel. Unfortunately that line is not at the end of the GENERIC file, so it can be a little less useful than it could be -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgp3RSI6YFwiX.pgp Description: OpenPGP Digital Signature
Re: Status of syspkg or similar
At Sat, 22 May 2021 15:19:36 +0300, Staffan Thomén wrote: Subject: Re: Status of syspkg or similar > > I've often thought about how the not-exactly-core utilities like tmux > (but window would probably not have fallen into this category > were it still there), postfix and bind (possibly also ssh, openssl might be > too difficult to extract) _that have more alternate/newer versions in pkgsrc_ > could live in their own set so I wouldn't have to install them and then just > install the pkgsrc version and be stuck with two. There are many ways to install and select alternative software, and I think the well known and used ones (e.g. pkgsrc and adjusting one's PATH) are quite sufficient. There are also a great many different things in the base system that might be replaced or superceded by such alternatives, and (as you noted) not all of these base system things are (or can be?) easily separated into individual "system" packages. I think the main thing that would be useful would be a tool that could easily and safely(*) remove a "syspkg" group of files from a system, which I think also implies that a little more information from the "lists" files needs to be included in the /etc/mtree "sets" files. This should be sufficient to cover any issues with, for example, security flaws in any of these major subsystems that could be replaced with something from pkgsrc. I.e. this deals with the case of wanting to entirely remove one of the more complex base-system subsystems. Other smaller or less intertwined things are just as easily manually removed (and optionally replaced with something from pkgsrc), at least I think so. (*) "safely" as in without loss of any local, e.g. configuration, changes, etc., which is easily managed using the existing signature in the mtree files. As a side note -- the only thing I think is still a useful option is to set LOCALBASE=/usr, though I haven't actively done that since the 1.6 days (even though it should be easier now with fixes to pkgsrc to allow one to more transparently handle hierarchy differences between /usr and /usr/pkg). These day's I'm happy enough with setting PKG_SYSCONFBASE=/etc -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpwY34lMz7zk.pgp Description: OpenPGP Digital Signature
Re: Status of syspkg or similar
At Sat, 22 May 2021 10:39:07 +0100, Sad Clouds
wrote:
Subject: Re: Status of syspkg or similar
>
> Thanks for the info. From what I understand, there are two separate
> packaging systems - traditional NetBSD sets, and syspkgs. The two
> systems cannot really co-exist together, so if/when more granular
> packaging system is implemented, the sets should be deprecated.
No, "sets" shouldn't be deprecated.
The two schemes may not be compatible within the same instance, but the
"sets" packaging is, and has long proven to be, a perfectly adequate and
very usable alternative.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpUkn5knG8xw.pgp
Description: OpenPGP Digital Signature
Re: toupper and warnings
At Thu, 06 May 2021 07:06:43 -0400, Greg Troxel wrote: Subject: Re: toupper and warnings > > Johnny Billquist writes: > > >> See CAVEATS in ctype(3). > > > > Right. But is gcc really smart enough to understand at compile time if > > something else than -1 is the negative value, and that toupper in fact > > is more limited than what the signature says? > > > > The *signature* of the function is int toupper(int). If you pass a > > char to that, I can't see that there would ever be a warning about any > > problems. Well the warning is actually an almost accidental side effect due to the implementation as a macro and due to the way the particular way the macro is expanded into an array reference. The potential to access an array with a negative index is obvious to the compiler in the current standard NetBSD implementation. Note too that actually implementing some of the ctype.h style interfaces as real functions on a system using signed chars could be problematic as they could then not internally distinguish between -1 and 0xFF when passed a plain un-cast signed char value (a value of 0xFF is of course a valid character, but due to the default integer promotions for all function parameters the sign will be extended and the result will be (int) -1). This matters for iscntrl(), isalpha(), and potentially also toupper() and tolower() depending on exactly how the return value is used. > To answer Greg Woods, It is not "conservative" to silently accept > UB-causing inputs and turn them into UB array reads. Well I don't actually do that. Which is to say my implementation's goal is to turn them into properly promoted values and access the flags array in such a way that this does not and cannot "cause" UB. My implementation carefully tests for a -1 and then if it is not -1 it uses a cast and assignment to an unsigned char, the value of which is then used to access the array. I.e. totally safe _and_ UB-free. Thus I take the conservative approach of not causing anything bad or unexpected to happen, including not causing UBSAN to abort the program. The only bad thing is that I do is to silently coerce truly badly behaving code into continuing to work without complaint, even from UBSAN. My test code for my implementation is here: https://github.com/robohack/experiments/blob/master/tctype.c So far it's only been tested with a Clang and a range of GCC versions. I can send the patch that adds it to -current too if you'd like to see it in-situ. > Another approach would be for NetBSD to adust the code to bounds check > before indexing and do something if the value is out of bounds. Yes, and my implementation could also easily be adjusted to do that. > Options > are returning false, returning a random bit, calling abort(3), trying to > send all of the user's files to a random IP addreess, and so on. All > are allowed by the spec :-) Yeah, "Undefined Behaviour" should be undefined -- i.e. removed from the spec -- i.e. become either fully defined or at least implementation defined. It is not helpful at all -- it was a very VERY bad idea. E.g. for ctype.h interfaces the spec should just say that values outside the recognized range will simply be truncated as if by assignment to an unsigned char. > Probably returning false with an envvar option to abort is best, similar > to how UB from mutex ops is handled. On the other hand perhaps this would better be a job for the UBSAN module to do, but perhaps that's just because I've become much more comfortable with using UBSAN with Clang in recent years. SIGILL for the losers. I'm not sure how my implementation could be so hooked into UBSAN tests though, especially in a less compiler-dependent way. > When we started with abort it > seemed there was a vast amount of incorrect code written and debugged on > other operating systems that allow code specified to hav UB. I haven't had quite that experience, though there have been some rare but stunning examples of problematic code -- though all I can remember were actually caught while porting the code to run on NetBSD/alpha, or in a few cases when porting to platforms that actually SIGBUS on unaligned accesses (like sparc IIRC?). What I am pretty sure of though is that there's a vast difference between the massive number of warnings spit out by the compiler vs. the relatively low number of actual cases of passing values outside of -1..255. We certainly wouldn't want to claim UB and abort for all of the warnings! -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpWHDkukp9tx.pgp Description: OpenPGP Digital Signature
Re: toupper and warnings
At Wed, 5 May 2021 20:33:30 - (UTC), chris...@astron.com (Christos Zoulas)
wrote:
Subject: Re: toupper and warnings
>
> gcc hides warnings from system headers by default (my guess is because
> the Linux system headers have a concealed firearms license).
>
> Our bsd.sys.mk turns it on... Try gcc -Wall -Wsystem-headers
Does that really make the difference?
(I can't quickly test it at the moment as I've fixed my test system such
that I don't ever get these warnings from the ctype macros. I.e. I went
the other way and chose to make my implementation's "undefined
behaviour" to be to behave as conservative as possible and allow the
caller to use these macros in the traditional naive way without fear and
without noisy warnings from picky compilers when char is signed.)
> I've been considering to make it the default for our native gcc,
> but then again I am against gratuitous user facing changes...
Note that -Wsystem-headers doesn't go well with -Wundef on NetBSD yet.
I find -Wundef to be quite helpful in writing more robust CPP
expressions -- it's saved me more than once from embarrassingly
non-portable ones, though I didn't start using it nearly soon enough.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpzulPfCalY4.pgp
Description: OpenPGP Digital Signature
Re: Setting ACCEPTABLE_LICENSES in /etc/mk.conf
At Fri, 16 Apr 2021 18:09:44 +, Todd Gruhn wrote:
Subject: Setting ACCEPTABLE_LICENSES in /etc/mk.conf
>
> Can ACCEPTABLE_LICENSES have several values separated
> by commas-- or must they all be on separate lines?
The canonical way to add values to it is the same way as is used in the
initial setting in mk/defaults/mk.conf, i.e. "+=", e.g. from there:
ACCEPTABLE_LICENSES+= adobe-acrobat-license
ACCEPTABLE_LICENSES+= amap-license
ACCEPTABLE_LICENSES+= amiwm-license
ACCEPTABLE_LICENSES+= apple-admin-license
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgphNo9C5jv6s.pgp
Description: OpenPGP Digital Signature
how do I mount a read-only filesystem from the "root device" prompt?
So with Xen one can export a "disk" (disk, file, LVM partiion, etc.)
with "access=ro", and that is enforced.
However if one tries to mount such a disk in a domU as root, it fails.
When one first looks at the code which does the initial vfs_mountroot it
would appear to be correct -- i.e. it is trying to open the root
filesystem device for reading it uses VOP_OPEN() to open the root
device with FREAD (which I think means "only for reading"):
error = VOP_OPEN(rootvp, FREAD, FSCRED);
if (error) {
printf("vfs_mountroot: can't open root device, error =
%d\n", error);
return (error);
}
However something assumes that if it is like a disk (i.e. but not a
CD-ROM/DVD) then it tries to open for write too as we get:
root on dk1
vfs_mountroot: can't open root device, error = 30
cannot mount root, error = 30
(errno #30 is of course EROFS)
I'm not even sure where this is happening.
vfs_rootmountalloc() does indeed set MNT_RDONLY, but this error is
happening before vfs_mountroot() calls ffs_mountroot (through the
vfs_mountroot pointer).
So I'm lost -- any hints? Is it from bounds_check_with_label()? How?
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgp7lVzPDrO_W.pgp
Description: OpenPGP Digital Signature
Re: blocklistd: How to keep my dynamic IP from getting blocked
At Sat, 3 Apr 2021 12:30:46 +0530, Mayuresh wrote:
Subject: Re: blocklistd: How to keep my dynamic IP from getting blocked
>
> Between these two: 1. Let blocklistd try to block and let npf overrule vs
> 2. Let blocklistd not block. Isn't the latter more economical?
I would include worrying about the economies of human understanding.
If you have two layers of control were one can say one thing and another
can possibly reverse the first's decision, then perhaps that is more
complex and more difficult to understand, especially during an emergency
debugging session.
However if you have one layer that entirely controls the other (with
respect to whatever features that controlling layer controls), then you
should only have to look in one place and understand one set of
conditions and rules to figure out what is going on.
So if you're going to use blocklistd to manage and control network
abusers, and that's going to be manipulating your firewall rules, then I
would say that you want to manage every aspect of what blocklistd does
at the blocklistd layer. I.e. make the firewall entirely subservient to
blocklistd w.r.t. what responsibilities you've assigned to blocklistd.
Perhaps as an allegory, when I first started using network level packet
filtering firewalls I already had a long experience with using TCP
Wrappers (i.e. libwrap with /etc/hosts.deny) to control access to
services. However I continued to use TCP Wrappers as I introduced
firewalling to my servers, and as a result I found that I could rapidly
confuse myself when something didn't behave as expected if I didn't keep
the domains of responsibility between these two access control features
quite well separated.
That all said, there are still a million ways to cut a cake, and not
necessarily any one better than the other, especially if you're just
going to eat the whole thing anyway. In this example you could even
modify the easily modified back-end blocklistd-helper script to filter
out those actions you don't want it to take, perhaps also controlling it
with yet another configuration file or list, etc., etc., etc.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpJi9w8rTU7Y.pgp
Description: OpenPGP Digital Signature
Re: blocklistd: How to keep my dynamic IP from getting blocked
At Sat, 3 Apr 2021 11:45:59 +0530, Mayuresh wrote: Subject: Re: blocklistd: How to keep my dynamic IP from getting blocked > > Just looked at man blacklistd.conf > > I guess nfail=* (means never) is what I have to use? And this entry with > ip address would be in [remote], right? Yes, correct. The EXAMPLES section in blocklistd.conf(5) should hopefully make it more clear. > What is unclear is the precedence - when one spec says block it and > another one says don't, how does blocklistd resolve it? > > I do see this: > > Matching is done first by checking the local rules individually, in > the order of the most specific to the least specific. If a match is > found, then the remote rules are applied. The name, nfail, and > disable fields can be altered by the remote rule that matched. > > Does it mean [remote] simply overrides [local]? Yes, rules in the [remote] section should override anything in the [local] section, and in particular since the rule in the [remote] section can set a new "nfail" value, using "*" will mean "never block". -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpIgGaUJkfdK.pgp Description: OpenPGP Digital Signature
Re: blocklistd: How to keep my dynamic IP from getting blocked
At Fri, 2 Apr 2021 11:24:53 +0530, Mayuresh wrote:
Subject: Re: blocklistd: How to keep my dynamic IP from getting blocked
>
> I can store a whitelist in a file and when it changes I can trigger (say)
> reload of npf. (I might possibly do something like tail -f on a file to
> trigger this. And a client side job will update the file.)
>
> But the next question is, I need npf to not entertain request from
> blocklistd to block a whitelisted ip stored in a file. Can someone suggest
> how to do this?
The way you are asking the question makes it sound like you are trying
to make the most complex, convoluted, confusing, and difficult solution
possible.
It can be much easier than that!
Just tell blocklistd not to block that IP! (and of course also
trigger blocklistd to reload its configuration at the same time)
I.e. have the script which is triggered by a change in the remote
network IP edit /etc/blocklist.conf every time that IP changes to
replace the old IP with the new IP, and then after it has saved the
change it can just run "/etc/rc.d/blocklistd reload".
I.e. if you don't want blocklistd to update your firewall rules for a
certain IP or range, then just ask it not to do so in the first place!
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgps9LyHlOhA8.pgp
Description: OpenPGP Digital Signature
Re: blocklistd: How to keep my dynamic IP from getting blocked
At Wed, 31 Mar 2021 11:13:51 - (UTC), mlel...@serpens.de (Michael van Elst) wrote: Subject: Re: blocklistd: How to keep my dynamic IP from getting blocked > > mayur...@acm.org (Mayuresh) writes: > > > > Strangely autossh manages to fail auth irking blocklistd and that ends up > > blocking access to all devices at home as they share the same external > > dynamic IP. (Let's keep aside why autossh manages to fail auth for now.) Well, that is the very root of the problem, is it not? :-) SSHd and blocklistd are doing exactly what you asked them to do and they're reporting and blocking "abuse" where that's been defined as some persistent attempt to authenticate a connection that's been explicitly (or implicitly, or accidentally) denied. Fix the authentication problems (and perhaps tune blocklistd's sensitivity so as to allow as many fat-finger failed authentications as you feel you might need), and your problem magically disappears entirely and hopefully permanently. And even more magically this solution is not affected in any way by whether or not either or both the target and/or source IPs are dynamically or statically assigned. It Just Works. > > Alternatively does it need to be done at npf's level? > > That's the more logical way. blocklistd works as designed and the login > failures trigger an entry in the blocklist. If you don't want to block > specific IPs, allow them by a specific rule, then it's also more clear > what is allowed and what is not by looking at a single place. Blocklistd also has the ability to be configured to not block any given addresses or networks. So depending on how the firewall rules are designed, it may actually make more sense to keep blocklistd from injecting its own blocking rules into the same firewall that is also trying to avoid blocking those same addresses or networks. Either way you'll need to update the new address in one or more files and trigger one or more actions that probably have to be done as root. That becomes more complicated if it's the remote (client) side that has the changing address and you don't already have a pre-determined way to do these updates and actions based on a remote trigger or some other kind of locally initiated monitoring. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpyH120xPMzp.pgp Description: OpenPGP Digital Signature
Re: kern/54969 (Disk cache is no longer flushed on shutdown)
RNING: some file systems would
not unmount
[Wed Mar 24 20:43:02 2021][ 715718.5284461] forcefully unmounting
/dev/mapper/scratch-build from /build...
[Wed Mar 24 20:43:02 2021][ 715718.5284461] forcefully unmounted
/dev/mapper/scratch-build from /build, type ffs
[Wed Mar 24 20:43:02 2021][ 715718.5384534] unmount of / (/dev/dk0) failed with
error 16
[Wed Mar 24 20:43:02 2021][ 715718.5384534] WARNING: some file systems would
not unmount
[Wed Mar 24 20:43:02 2021][ 715718.5384534] forcefully unmounting /dev/dk0 from
/...
[Wed Mar 24 20:43:02 2021][ 715718.5384534] forcefully unmounted /dev/dk0 from
/, type ffs
[Wed Mar 24 20:43:02 2021][ 715718.5384534] unmounting done
[Wed Mar 24 20:43:02 2021][ 715718.5384534] turning off swap... done
[Wed Mar 24 20:43:02 2021][ 715718.5384534] dk0 at sd0 (/) deleted
[Wed Mar 24 20:43:02 2021][ 715718.5384534] sd0: detached
[Wed Mar 24 20:43:02 2021][ 715718.5384534] scsibus0: detached
[Wed Mar 24 20:43:02 2021][ 715718.7184994] mfi0: detached
[Wed Mar 24 20:43:02 2021][ 715718.7184994] pci8: detached
[Wed Mar 24 20:43:02 2021][ 715718.7184994] ppb7: detached
[Wed Mar 24 20:43:02 2021][ 715718.7184994] unmounting done
[Wed Mar 24 20:43:02 2021][ 715718.7184994] turning off swap... done
[Wed Mar 24 20:43:02 2021][ 715718.7184994] rebooting...
[[ ... why is "turning off swap" seen twice? .. ]]
[[ ... and then the reboot, until rc scripts say ... ]]
[Wed Mar 24 20:44:51 2021]Starting root file system check:
[Wed Mar 24 20:44:51 2021]/dev/rdk0: file system is clean; not checking
[Wed Mar 24 20:44:51 2021]start / wait fsck_ffs -p /dev/rdk0
[Wed Mar 24 20:44:52 2021]Starting file system checks:
[Wed Mar 24 20:44:52 2021]/dev/rdk2: file system is clean; not checking
[Wed Mar 24 20:44:52 2021]/dev/rdk3: file system is clean; not checking
[[ ... here I hit ^T on the console as it was taking too long ... ]]
[Wed Mar 24 20:44:58 2021][ 15.0201108] load: 0.08 cmd: sleep 345 [nanoslp]
0.00u 0.00s 0% 512k
[Wed Mar 24 20:44:58 2021]/dev/mapper/rscratch-build: phase 1: cyl group 24 of
345 (6%)
[Wed Mar 24 20:46:09 2021]/dev/mapper/rscratch-build: phase 1: cyl group 284 of
345 (82%)
[Wed Mar 24 20:49:30 2021]/dev/mapper/rscratch-build: 1400986 files, 36172587
used, 28347707 free (17403 frags, 3541288 blocks, 0.0% fragmentation)
[Wed Mar 24 20:49:30 2021]/dev/mapper/rscratch-build: MARKING FILE SYSTEM CLEAN
[Wed Mar 24 20:49:30 2021]start /var nowait fsck_ffs -p /dev/rdk2
[Wed Mar 24 20:49:30 2021]start /build nowait fsck_ffs -p
/dev/mapper/rscratch-build
[Wed Mar 24 20:49:30 2021]done ffs: /dev/rdk2 (/var) = 0x0
[Wed Mar 24 20:49:30 2021]start /usr/pkg nowait fsck_ffs -p /dev/rdk3
[Wed Mar 24 20:49:30 2021]done ffs: /dev/rdk3 (/usr/pkg) = 0x0
[Wed Mar 24 20:49:30 2021]done ffs: /dev/mapper/rscratch-build (/build) = 0x0
[Wed Mar 24 20:49:30 2021]Script /etc/rc.d/fsck running
[Wed Mar 24 20:49:30 2021]Currently sourcing /etc/rc.d/fsck
[Wed Mar 24 20:49:30 2021]exec: mount_ffs -o rw /dev/dk2 /var
[Wed Mar 24 20:49:30 2021]exec: mount_ffs -o rw /dev/dk2 /var
[Wed Mar 24 20:49:30 2021]/dev/dk2 on /var type ffs (local, fsid: 0xa802/0x78b,
reads: sync 1 async 0, writes: sync 2 async 0)
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpd4g7W5tdUY.pgp
Description: OpenPGP Digital Signature
a reminder: upgrade Xen in single-user mode, or with Xen disabled!
So I just upgraded Xen to xenkernel413-4.13.2nb5, but without first
upgrading the Xen tools, as otherwise how would one safely shut down any
running domUs, etc.? :-)
Once upgrading to xentools413-4.13.2nb4 I immediately got stuck:
# xl list
NameID Mem VCPUs State Time(s)
[ 578.9865720] load: 0.27 cmd: xl 2027 [tstile] 0.00u 0.01s 0% 3080k
and I mean "really" stuck -- xl is unkillable (and unstoppable) in that
state!
At first I had grave misgivings that the old tstile deadlock was back,
but at the moment only dom0 is running
So thinking, h the old xenstored is started on boot and will
still be running and so I need to restart that from another xterm with
"/etc/rc.d/xencommons restart", and voila, that unstuck xl.
Probably xl shouldn't get stuck like that if it can't connect to
xenstored properly -- as I said it's unkillable in that state!
I then tried "/etc/rc.d/xenwatchdog restart" but it didn't restart (for
some reason I've yet to diagnose -- I had this happen once before -- it
seems to have trouble restarting sometimes, perhaps especially after
restarting xencommons).
That meant that a few moments later the Xen kernel decided dom0 was dead
and promptly (and I mean PROMPTLY) rebooted the machine -- kaboom!
(XEN) [2021-03-25 04:16:26.951] Watchdog timer fired for domain 0
(XEN) [2021-03-25 04:16:26.951] Hardware Dom0 shutdown: watchdog rebooting
machine
At least on this next reboot all the right versions of the right bits
started!
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpKwzMGQ5iLa.pgp
Description: OpenPGP Digital Signature
Re: ispell-british (for 9.0/amd64) has broken hash files
kre mentioned giving up on ispell, but I found it "hard" to use anything
but ispell with emacs, so as I've upgraded systems and ended up tripping
over this breakage, I've eventually persevered to make it work again.
I don't remember why aspell didn't work for me (I have hacks in my
.emacs.el to try to use it), but I do remember it was much easier to fix
ispell than to make aspell work.
At Wed, 03 Feb 2021 08:33:11 -0500, Greg Troxel wrote:
Subject: Re: ispell-british (for 9.0/amd64) has broken hash files
>
> If you follow upstream's build instructions, not using pkgsrc, does it
> work?
I guess so? That's more or less how I've changed the package to make it
work -- I.e. just use the supplied makefile.
> It sounds like you should be sending your fixes to ispell upstream, but
> maybe there is a pkgsrc bug, vs, a lack of pkgsrc workaround for an
> upstream bug.
Here's the PR with my fixes: pkg/55972
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpPApJwQ9YCp.pgp
Description: OpenPGP Digital Signature
ispell-british (for 9.0/amd64) has broken hash files
I've been trying to use the ispell-british dictionaries but I've been
having problems (note that the default "english" dictionary works A-OK).
I see similar reports for other dictionaries for various linux packages,
but I don't see any related to NetBSD or pkgsrc.
I have a rather radical set of changes to the way the ispell
dictionaries are built and installed, and when I use my updated
makefiles then the resulting dictionary works A-OK
I'm wondering if anyone can reproduce this with the standard packages,
and if so then perhaps my fixes would make for a good send-pr.
$ DICTIONARY=british ispell quotes
Illegal format hash table /usr/pkg/lib/british.hash - expected magic2 0x9602,
got 0x0
$ ls -l /usr/pkg/lib/british*
-rw-r--r-- 2 root wheel 3458112 Jul 18 2020 /usr/pkg/lib/british.hash
-rw-r--r-- 2 root wheel 3458112 Jul 18 2020 /usr/pkg/lib/britishxlg.hash
$ file /usr/pkg/lib/*.hash
/usr/pkg/lib/american.hash: symbolic link to americanmed+.hash
/usr/pkg/lib/americanmed+.hash: little endian ispell 3.1 hash file, 7-bit,
capitalization, 52 flags and 512 string characters
/usr/pkg/lib/british.hash: little endian ispell 3.1 hash file, 7-bit,
capitalization, 52 flags and 512 string characters
/usr/pkg/lib/britishxlg.hash: little endian ispell 3.1 hash file, 7-bit,
capitalization, 52 flags and 512 string characters
/usr/pkg/lib/english.hash: symbolic link to americanmed+.hash
$ uname -a
NetBSD xentastic 9.99.64 NetBSD 9.99.64 (GENERIC) #11: Sat Jul 4 16:12:53 PDT
2020
woods@xentastic:/build/woods/xentastic/current-amd64-amd64-obj/build/src-current/sys/arch/amd64/compile/GENERIC
amd64
$ mktable /usr/pkg/etc/pkgin/repositories.conf
http://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/amd64/9.0/All
$ /usr/sbin/pkg_info ispell
Information for ispell-3.4.00:
Comment:
Interactive spelling checker
Required by:
ispell-british-3.4.00
Description:
Ispell is a fast screen-oriented spelling checker that shows you your
errors in the context of the original file, and suggests possible
corrections when it can figure them out. Compared to UNIX spell, it
is faster and much easier to use. Ispell can also handle languages
other than English.
Homepage:
http://ficus-www.cs.ucla.edu/geoff/ispell.html
$ /usr/sbin/pkg_info ispell-british
Information for ispell-british-3.4.00:
Comment:
British dictionary for interactive spelling checker
Requires:
ispell>=3.3.02
Description:
This package provides the British-spelling dictionaries for ispell.
Homepage:
http://ficus-www.cs.ucla.edu/geoff/ispell.html
(BTW, the server for that homepage doesn't seem reachable any more.)
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpbexpAby1lr.pgp
Description: OpenPGP Digital Signature
Re: Git pkgsrc - setting file locations
At Wed, 15 Jul 2020 15:40:40 -0400 (EDT), Bob Bernstein
wrote:
Subject: Git pkgsrc - setting file locations
>
> I have been working awhile now with the GIT pkgsrc, and it occurs to
> me that it might be an advantage to provide different locations for
> things such as distfiles, packages, and...what else?
>
> What is suggested, and where/how is the optimum method for altering
> these values?
These are my relevant hacks which I make directly to
pkgsrc/mk/defaults/mk.conf:
PKGMAKECONF = /etc/mk.conf
PKGSRC_MAKE_ENV += USER=${USER:Q}
WRKOBJDIR ?=/var/package-obj/${USER}
DISTDIR ?= /var/package-distfiles
PACKAGES ?=
/var/packages/${USER}/${OPSYS}/${OS_VERSION:C/9.0.*/9.0/:C/9.99..*/9.99/}/${MACHINE_ARCH}
On my build servers these /var directories are then often symlinks to
separate directories on either shared or other local filesystem(s).
Then in /etc/mk.conf I wrap local pkgsrc-only things in an if testing
BSD_PKG_MK, e.g. as follows:
.if defined(BSD_PKG_MK)
# I.e. the rest is for pkgsrc things that are truly local to this host
# environment: (as opposed to the site-specific stuff in
/usr/pkgsrc/mk)
# XXX N.B.: It is expected that mk/defaults/mk.conf will have set
#
# PKGMAKECONF = /etc/mk.conf
# use pkgtools/autoswc to cache some autoconf results
#
.sinclude "/usr/pkg/share/autoswc/autoswc.mk"
PKG_SYSCONFBASE = /etc
PKG_RCD_SCRIPTS = YES # install rc.d scripts immediately.
.endif
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpfQNnaC3ghq.pgp
Description: OpenPGP Digital Signature
Re: pkgsrc build server
At Fri, 10 Jul 2020 14:24:20 +0300, Denys Nykula wrote: Subject: Re: pkgsrc build server > > "Greg A. Woods" 4 July 2020, 23:58:24: > > > # use pkgtools/autoswc to cache some autoconf results > > # > > .sinclude "/usr/pkg/share/autoswc/autoswc.mk" > > Hello, after your letter I went to look what autoswc is. Its Makefile > hardcodes CACHEDIR and MKCONF to root paths, /var/db/autoswc and > /etc/mk.conf. What is the reason they're set this way and not to > VARBASE/db/autoswc and LOCALBASE/etc? Would it be possible to > pull in paths from mk.conf and cache autoconf results as an > unprivileged pkgsrc user? I'm sure it would be possible, but I'm guessing this is an indication of how well, or how poorly, that unprivileged pkgsrc use is supported at the moment. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpzIh9AXupZh.pgp Description: OpenPGP Digital Signature
Re: pkgsrc build server
At Sat, 4 Jul 2020 21:01:26 +0300, Dima Veselov wrote: Subject: Re: pkgsrc build server > > What I want is to run "make install" on fast real production server > with lot of resources once and then have all of those packages built in > NFS DISTDIR so any VM can just install them. Yes, you can do that, provided the fast server runs the same OS that will run on the target VM. I share my /build FS from the build server(s) and install binary packages built there on production systems. You may have to arrange for pkgsrc to build binary packages for you, and I think that's relatively easy now with something like this in your /etc/mk.conf (i.e. on the build server), though I must warn I may have missed somthing that I've more permanently hacked on in my pkgsrc tree (there are some other nice things shown here too). # This section is just for pkgsrc # .if defined(BSD_PKG_MK) # carefully share host settings for packages with BSD-Style makefiles # PKGMAKECONF = /etc/mk.conf INSTALL_UNSTRIPPED =YES DEFAULT_DEPENDS_TARGET =package DEFAULT_UPDATE_TARGET ?=package DEPENDS_TARGET =update UPDATE_TARGET = package-install # uncomment this if you sometimes manually build packages and might miss # some dependency that was built without "make update" or "make package" # and thus which still needs a "make package" run to build the binary # package. # #CLEANDEPENDS= no # use pkgtools/autoswc to cache some autoconf results # .sinclude "/usr/pkg/share/autoswc/autoswc.mk" .endif # BSD_PKG_MK As others have warned though, be careful about mixing packages from different pkgsrc trees. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpkECIGDsJhb.pgp Description: OpenPGP Digital Signature
Re: Use network printer from NetBSD
-sDEVICE=pxlmono'. I must still be missing some critical secret to the
mostly undocumented XL2HB format. The printer starts up and warms up,
then silently gives up without even ejecting a blank page.
If your printer really does understand postscript then this script
should print something (and show things on the telnet output):
#!/bin/sh
{
printf '\033%%-12345X@PJL\n'
printf '@PJL INITIALIZE\n'
printf '\033%%-12345X@PJL\n'
printf '@PJL INFO ID\n'
printf '@PJL INFO CONFIG\n'
printf '@PJL INFO VARIABLES\n'
printf '@PJL INFO STATUS\n'
printf '@PJL INFO FEATURES\n'
printf '@PJL INFO OPTIONS\n'
printf '@PJL INFO FILESYS\n'
printf '@PJL INFO LOG\n'
printf '@PJL INFO PRODINFO\n'
printf '@PJL INFO SUPPLIES\n'
printf '@PJL USTATUS DEVICE = VERBOSE\n@PJL USTATUS JOB = ON\n@PJL
USTATUS PAGE = ON\n@PJL INFO USTATUS\n'
printf '@PJL JOB NAME = "test job"\n'
printf '@PJL INFO PAGECOUNT\n'
printf '@PJL ENTER LANGUAGE = POSTSCRIPT\n'
cat <<__EOF__
%!PS
0.5 setgray
clippath pathbbox moveto lineto stroke
newpath
100 200 moveto
200 250 lineto
100 300 lineto
2 setlinewidth
stroke
showpage
__EOF__
printf '\004'
printf '\033%%-12345X@PJL\n@PJL EOJ NAME = "test job"\n';
printf '@PJL USTATUSOFF\n@PJL INFO PAGECOUNT\n@PJL INFO STATUS\n';
printf '\033%%-12345X';
sleep 30
} | telnet 10.0.1.19 9100
BTW, the PJL language is documented for my printer in the "Command
Reference Guide for Software Developers":
https://download.brother.com/welcome/doc002907/Tech_Manual_AC.pdf
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpqXEdz_anOE.pgp
Description: OpenPGP Digital Signature
Re: does anyone have a working mozilla firefox-74.0 on 9.0 amd64?
At Thu, 2 Jul 2020 22:27:49 + (GMT), r0ller wrote: Subject: Re: does anyone have a working mozilla firefox-74.0 on 9.0 amd64? > > Mine works fine. However, my system is an upgraded one from 8.1 to 9.0 > if it makes a difference. My system is a stock fresh 9.0/amd64 install, from the downloaded ISO. (well it was at the time I reported the crash -- I've since upgraded it to my own 9.99.64 build.) > Then I installed firefox-74.0 and removed 68 > (I guess that was its version). I tried installing firefox68. It dumps core in exactly the same way, though now with library debug symbols from my own build I do see a tiny wee fraction more info, but in this case it's not helpful: Reading symbols from /usr/pkg/lib/firefox68/firefox68... (No debugging symbols found in /usr/pkg/lib/firefox68/firefox68) Core was generated by `firefox68'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x7e313cb798ca in _lwp_kill () from /usr/lib/libc.so.12 [Current thread is 1 (process 12887)] (gdb) bt #0 0x7e313cb798ca in _lwp_kill () from /usr/lib/libc.so.12 #1 0x7e312b6486a1 in ?? () from /usr/pkg/lib/firefox68/libxul.so #2 0x7e313caa5c10 in _opendir (name=) at /build/src-current/lib/libc/gen/opendir.c:72 #3 0x0001000b in ?? () #4 0x in ?? () (gdb) I don't get any error messages from it, just a near instant kaboom. LDD shows it is gtk-3 that depends on libepoxy, and that's using the installed one from the X11 system sets. $ ldd /usr/pkg/lib/libgtk-3.so /usr/pkg/lib/libgtk-3.so: -lgdk-3.0 => /usr/pkg/lib/libgdk-3.so.0 -lpangocairo-1.0.0 => /usr/pkg/lib/libpangocairo-1.0.so.0 -lpango-1.0.0 => /usr/pkg/lib/libpango-1.0.so.0 -lm.0 => /usr/lib/libm.so.0 -lc.12 => /usr/lib/libc.so.12 -lglib-2.0.0 => /usr/pkg/lib/libglib-2.0.so.0 -lpcre.1 => /usr/pkg/lib/libpcre.so.1 -lintl.1 => /usr/lib/libintl.so.1 -lpthread.1 => /usr/lib/libpthread.so.1 -lgobject-2.0.0 => /usr/pkg/lib/libgobject-2.0.so.0 -lffi.7 => /usr/pkg/lib/libffi.so.7 -lfribidi.0 => /usr/pkg/lib/libfribidi.so.0 -lharfbuzz.0 => /usr/pkg/lib/libharfbuzz.so.0 -lfreetype.19 => /usr/X11R7/lib/libfreetype.so.19 -lz.1 => /usr/lib/libz.so.1 -lbz2.1 => /usr/lib/libbz2.so.1 -lgraphite2.3 => /usr/pkg/lib/libgraphite2.so.3 -lstdc++.9 => /usr/lib/libstdc++.so.9 -lgcc_s.1 => /usr/lib/libgcc_s.so.1 -lpangoft2-1.0.0 => /usr/pkg/lib/libpangoft2-1.0.so.0 -lfontconfig.2 => /usr/X11R7/lib/libfontconfig.so.2 -lexpat.2 => /usr/lib/libexpat.so.2 -lcairo.2 => /usr/pkg/lib/libcairo.so.2 -lpixman-1.2 => /usr/X11R7/lib/libpixman-1.so.2 -lpng16.16 => /usr/pkg/lib/libpng16.so.16 -lxcb-shm.0 => /usr/X11R7/lib/libxcb-shm.so.0 -lxcb.2 => /usr/X11R7/lib/libxcb.so.2 -lXau.7 => /usr/X11R7/lib/libXau.so.7 -lXdmcp.7 => /usr/X11R7/lib/libXdmcp.so.7 -lxcb-render.0 => /usr/X11R7/lib/libxcb-render.so.0 -lXrender.2 => /usr/X11R7/lib/libXrender.so.2 -lXext.7 => /usr/X11R7/lib/libXext.so.7 -lX11.7 => /usr/X11R7/lib/libX11.so.7 -lrt.1 => /usr/lib/librt.so.1 -lgdk_pixbuf-2.0.0 => /usr/pkg/lib/libgdk_pixbuf-2.0.so.0 -lgmodule-2.0.0 => /usr/pkg/lib/libgmodule-2.0.so.0 -lgio-2.0.0 => /usr/pkg/lib/libgio-2.0.so.0 -lcairo-gobject.2 => /usr/pkg/lib/libcairo-gobject.so.2 -lXinerama.2 => /usr/X11R7/lib/libXinerama.so.2 -lXi.7 => /usr/X11R7/lib/libXi.so.7 -lXrandr.3 => /usr/X11R7/lib/libXrandr.so.3 -lXcursor.2 => /usr/X11R7/lib/libXcursor.so.2 -lXcomposite.2 => /usr/X11R7/lib/libXcomposite.so.2 -lXdamage.2 => /usr/X11R7/lib/libXdamage.so.2 -lXfixes.4 => /usr/X11R7/lib/libXfixes.so.4 -lxkbcommon.0 => /usr/pkg/lib/libxkbcommon.so.0 -lwayland-cursor.0 => /usr/pkg/lib/libwayland-cursor.so.0 -lwayland-client.0 => /usr/pkg/lib/libwayland-client.so.0 -lwayland-egl.1 => /usr/pkg/lib/libwayland-egl.so.1 -lepoxy.0 => /usr/X11R7/lib/libepoxy.so.0 -latk-1.0.0 => /usr/pkg/lib/libatk-1.0.so.0 -latk-bridge-2.0.0 => /usr/pkg/lib/libatk-bridge-2.0.so.0 -ldbus-1.3 => /usr/pkg/lib/libdbus-1.so.3 -lexecinfo.0 => /usr/lib/libexecinfo.so.0 -lelf.2 => /usr/lib/libelf.so.2 -latspi.0 => /usr/pkg/lib/libatspi.so.0 -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgp1FfdANnqtv.pgp Description: OpenPGP Digital Signature
Re: does anyone have a working mozilla firefox-74.0 on 9.0 amd64?
At Thu, 2 Jul 2020 14:03:26 -0700 (PDT), Hisashi T Fujinaka
wrote:
Subject: Re: does anyone have a working mozilla firefox-74.0 on 9.0 amd64?
>
> OK, I don't know why this works, but I did "cat /dev/null >
> /etc/ld.so.conf" and it started right up.
>
Not for me, though maybe it didsimplify the stack backtrace a wee bit:
Core was generated by `firefox'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x7a520d3798ca in _lwp_kill () from /usr/lib/libc.so.12
[Current thread is 1 (process 17800)]
(gdb) bt
#0 0x7a520d3798ca in _lwp_kill () from /usr/lib/libc.so.12
#1 0x7a51fb5fec81 in ?? () from /usr/pkg/lib/firefox/libxul.so
#2 0x7a520d2a5c10 in _opendir (name=)
at /build/src-current/lib/libc/gen/opendir.c:72
#3 0x0001000b in ?? ()
#4 0x in ?? ()
(gdb)
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpneNpHCUapB.pgp
Description: OpenPGP Digital Signature
does anyone have a working mozilla firefox-74.0 on 9.0 amd64?
So, does anyone have a working mozilla firefox-74.0 on 9.0 amd64?
I've been rebuilding another amd64 system starting with a stock 9.0
install, and I decided I should try firefox, since I haven't tried it in
a very long time, so I just used "pkgin install firefox" to try
However it just dumps core when I start it:
Reading symbols from /usr/pkg/lib/firefox/firefox...
(No debugging symbols found in /usr/pkg/lib/firefox/firefox)
(gdb) run
Starting program: /usr/pkg/lib/firefox/firefox
[New process 17803]
[Detaching after fork from child process 2656]
[New LWP 2 of process 17803]
Thread 1 "" received signal SIGSEGV, Segmentation fault.
0x7b1ac69aa96a in ?? () from /usr/pkg/lib/firefox/libxul.so
(gdb) bt
#0 0x7b1ac69aa96a in ?? () from /usr/pkg/lib/firefox/libxul.so
#1 0x7b1ac980e571 in ?? () from /usr/pkg/lib/firefox/libxul.so
#2 0x7b1ac980b18b in ?? () from /usr/pkg/lib/firefox/libxul.so
#3 0x7b1ac9810278 in ?? () from /usr/pkg/lib/firefox/libxul.so
#4 0x7b1ac98105d9 in ?? () from /usr/pkg/lib/firefox/libxul.so
#5 0x0001e8c0a810 in ?? ()
#6 0x0001e8c068fd in _start ()
(gdb)
(stripped binaries are a curse)
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpBClgjl3aHS.pgp
Description: OpenPGP Digital Signature
Re: So it seems "umount -f /nfs/mount" still doesn't work.....
At Tue, 30 Jun 2020 14:28:38 -0700, "Greg A. Woods" wrote: Subject: Re: So it seems "umount -f /nfs/mount" still doesn't work. > > At Tue, 30 Jun 2020 12:52:37 -0700, "Greg A. Woods" wrote: > Subject: So it seems "umount -f /nfs/mount" still doesn't work. > > > So, I should have mentioned that "umount -f nfs.server:/remotefs" does work (i.e. it does not hang waiting for the server to reconnect, and provided that there are no processes with cwd or open files on the remote filesystem, it can unmount the filesystem). I.e. the problem is in how umount(8) looks up the parameters of the mount point. If it looks at the mount point it hangs, but if it looks through the mount table, it works. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpy_74PnJtrP.pgp Description: OpenPGP Digital Signature
Re: So it seems "umount -f /nfs/mount" still doesn't work.....
At Tue, 30 Jun 2020 12:52:37 -0700, "Greg A. Woods" wrote:
Subject: So it seems "umount -f /nfs/mount" still doesn't work.
>
Curiously the kernel now does something I didn't quite expect when one
tries to reboot a system with a stuck mount. I was able to see this as
I was running a kernel that verbosely documents all its shutdown
unmounts and detaches. In prior times I had reached for the power switch.
At first it just hangs:
lilbit# reboot -q
[ 1131744.8297338] syncing disks... 3 3 done
[ 1131744.9797408] unmounting 0xc1f27000 /more/work (more.local:/work)...
[ 1131744.9907053] ok
[ 1131744.9907053] unmounting 0xc1f24000 /more/archive (more.local:/archive)...
[ 1131745.0004431] ok
[ 1131745.0004431] unmounting 0xc1f21000 /more/home (more.local:/home)...
[ 1131745.0097426] ok
[ 1131745.0097426] unmounting 0xc1f1f000 /once/build (once.local:/build)...
[ 1131745.0097426] ok
[ 1131745.0210854] unmounting 0xc1f1b000 /future/build (future.local:/build)...
[ 1131745.0210854] ok
[ 1131745.0304676] unmounting 0xc1f11000 /building/build
(building.local:/build)...
this is me hitting ^T to try to see what's going on
[ 1131753.2800902] load: 0.52 cmd: reboot 7414 [fstcnt] 0.00u 0.16s 0% 424k
[ 1132107.6651517] load: 0.48 cmd: reboot 7414 [fstcnt] 0.00u 0.16s 0% 424k
[ 1133247.8436109] load: 0.48 cmd: reboot 7414 [fstcnt] 0.00u 0.16s 0% 424k
then I hit ^C and immediately it proceeded
^C[ 1133249.3636755] unmounting 0xc1f0f000 /proc (procfs)...
[ 1133249.3636755] ok
[ 1133249.3636755] unmounting 0xc1f0d000 /dev/pts (ptyfs)...
[ 1133249.3788641] unmounting 0xc1ecb000 /kern (kernfs)...
[ 1133249.3843127] ok
[ 1133249.3843127] unmounting 0xc1ec9000 /cache (/dev/wd1a)...
[ 1133249.7636916] ok
[ 1133249.7636916] unmounting 0xc1ec6000 /home (/dev/wd0g)...
[ 1133249.7736976] unmounting 0xc1dd7000 /usr/pkg (/dev/wd0f)...
[ 1133250.0737098] unmounting 0xc1ab1000 /var (/dev/wd0e)...
[ 1133250.1537121] unmounting 0xc1804000 / (/dev/wd0a)...
[ 1133251.0337515] unmounting 0xc1f11000 /building/build
(building.local:/build)...
[ 1133251.0469644] unmounting 0xc1f0d000 /dev/pts (ptyfs)...
[ 1133251.0469644] unmounting 0xc1ec6000 /home (/dev/wd0g)...
[ 1133251.0579007] unmounting 0xc1dd7000 /usr/pkg (/dev/wd0f)...
[ 1133251.0637673] unmounting 0xc1ab1000 /var (/dev/wd0e)...
[ 1133251.0637673] unmounting 0xc1804000 / (/dev/wd0a)...
[ 1133251.0750403] sd0: detached
[ 1133251.0750403] scsibus0: detached
[ 1133251.0750403] gpio1: detached
[ 1133251.0853614] sysbeep0: detached
[ 1133251.0853614] midi0: detached
[ 1133251.0853614] wd1: detached
[ 1133251.0949369] uhub0: detached
[ 1133251.0949369] com1: detached
[ 1133251.0949369] usb0: detached
[ 1133251.1045456] gpio0: detached
[ 1133251.1045456] ohci0: detached
[ 1133251.1045456] pchb0: detached
[ 1133251.1151702] unmounting 0xc1f11000 /building/build
(building.local:/build)...
[ 1133251.1151702] unmounting 0xc1f0d000 /dev/pts (ptyfs)...
[ 1133251.1279509] unmounting 0xc1ec6000 /home (/dev/wd0g)...
[ 1133251.1279509] unmounting 0xc1dd7000 /usr/pkg (/dev/wd0f)...
[ 1133251.1393918] unmounting 0xc1ab1000 /var (/dev/wd0e)...
[ 1133251.1448739] unmounting 0xc1804000 / (/dev/wd0a)...
[ 1133251.1448739] forcefully unmounting /building/build
(building.local:/build)...
[ 1133251.1587138] forceful unmount of /building/build failed with error -3
[ 1133251.1653872] rebooting...
So it seems there's some contention between the internal attempt to
unmount the stuck NFS filesystem(s), and the reboot system call itself,
but if the reboot command is interrupted, then the kernel can get on
with its shutdown procedures, and eventually it actually forces the
unmount of the stuck NFS filesystem.
Another interesting thing to note is that /future/build was also stuck
as future.local is offline at this time. However that's the filesystem
I tried to clear first by hand with "umount -f /future/build", but that
was stuck, apparently in the same call to nfs_reconnect(). It seems it
had done enough that when the reboot() triggered unmounting that it
could complete the unmount without problems. (The other mounts on
more.local and once.local were responding so they unmounted normally.)
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgp0PN1eJ9ZMC.pgp
Description: PGP signature
So it seems "umount -f /nfs/mount" still doesn't work.....
rshd -L 12 16008 9130 85 0 7860 1140 kqueue I? 0:00.01 pickup -l -t unix -u 0 16131 10040 85 0 2620644 select I? 0:00.01 rshd -L 1000 20090 152700 85 0 25056 7028 select Is ? 0:00.24 xterm -class UXTerm 1000 1768 6090 85 0 11708 1056 ttyraw Is+ pts/1 0:00.09 -ksh 0 2940 75840 117 0 17264236 nfscn2 D+ pts/2 0:00.14 umount -f /future/build 1000 4103 40070 85 0 4120 1064 pause Is pts/2 0:00.09 -ksh 0 7584 41030 85 0 7656 1088 pause Ipts/2 0:00.35 ksh 0 6722 276390 127 0 16768440 tstile D+ pts/3 0:00.00 fstat /future/build 1000 21172 14064 489 85 0 3728 1060 pause Is pts/3 0:00.09 -ksh 0 27639 211720 85 0 9648 1048 pause Ipts/3 0:00.08 ksh 1000 13000 20090 722 85 0 3600 1056 ttyraw Is+ pts/4 0:00.11 -ksh 0 3707 19523 1057 85 0 11736 1044 pause Spts/5 0:00.08 ksh 0 4176 3707 1057 43 0 12900624 - O+ pts/5 0:00.00 ps -alx 1000 19523 1002 3550 85 0 3188 1056 pause Ss pts/5 0:00.09 -ksh 0 1013 1 527 85 0 2660652 ttyraw Is+ ttyE0 0:00.08 -ksh 0 822 1 126 85 0 2524412 ttyraw Is+ ttyE1 0:00.00 /usr/libexec/getty Ws ttyE1 0 828 1 126 85 0 2524416 ttyraw Is+ ttyE2 0:00.00 /usr/libexec/getty Ws ttyE2 0 957 1 126 85 0 2528412 ttyraw Is+ ttyE3 0:00.00 /usr/libexec/getty Ws ttyE3 0 862 1 126 85 0 4188408 ttyraw Is+ ttyE4 0:00.00 /usr/libexec/getty Ws ttyE4 0 1023 1 126 85 0 2524424 ttyraw Is+ ttyE5 0:00.00 /usr/libexec/getty Ws ttyE5 0 1050 1 126 85 0 2524428 ttyraw Is+ ttyE6 0:00.00 /usr/libexec/getty Ws ttyE6 0 668 10 85 0 2528416 ttyraw Is+ xencons0:00.01 /usr/libexec/getty console constty 12:00 [1.61] # crash Crash version 8.99.32, image version 8.99.32. Output from a running system is unreliable. crash> bt /t 0t2940 trace: pid 2940 lid 1 at 0xaf808a4748f0 sleepq_block() at sleepq_block+0xfd kpause() at kpause+0xdf nfs_reconnect() at nfs_reconnect+0x8b nfs_request() at nfs_request+0xf3a nfs_getattr() at nfs_getattr+0x175 VOP_GETATTR() at VOP_GETATTR+0x49 vn_stat() at vn_stat+0x3d do_sys_statat() at do_sys_statat+0x97 sys___lstat50() at sys___lstat50+0x25 syscall() at syscall+0x9c --- syscall (number 441) --- 43292a: crash> bt /t 0t6722 trace: pid 6722 lid 1 at 0xaf808a488920 sleepq_block() at sleepq_block+0x99 turnstile_block() at turnstile_block+0x337 rw_vector_enter() at rw_vector_enter+0x169 genfs_lock() at genfs_lock+0x3c VOP_LOCK() at VOP_LOCK+0x71 vn_lock() at vn_lock+0x90 nfs_root() at nfs_root+0x2b lookup_once() at lookup_once+0x38e namei_tryemulroot() at namei_tryemulroot+0x453 namei() at namei+0x29 fd_nameiat.isra.2() at fd_nameiat.isra.2+0x54 do_sys_statat() at do_sys_statat+0x87 sys___stat50() at sys___stat50+0x28 syscall() at syscall+0x9c --- syscall (number 439) --- 43c94a: crash> So it would seem that even though umount is trying to force an unmount of an NFS mount, the kernel is first trying to reconnect to the server! BTW, I have another system running a quite recent i386 build where crash(8) is unable to do a backtrace: # ktrace crash Crash version 9.99.64, image version 9.99.64. Kernel compiled without options LOCKDEBUG. Output from a running system is unreliable. crash> trace /t 0t4003 crash: kvm_read(0x4, 4): kvm_read: Bad address trace: pid 4003 lid 4003 crash> -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpFB0EHtrOEo.pgp Description: PGP signature
Re: How to configure npf to restrict nfs to localhost
At Tue, 30 Jun 2020 15:36:07 +0200, Hauke Fath wrote:
Subject: Re: How to configure npf to restrict nfs to localhost
>
> On 2020-06-29 23:24, Greg A. Woods wrote:
> > Stopping rpcbind from revealing ports other RPC servers are listening on
> > is the primary thing you need to do. You can do this with filters
> > blocking TCP and UDP ports #111, and/or with rpcbind itself using its
> > built-in libwrap support, like so:
> >
> > In your /etc/hosts.allow file you can restrict rpcbind to given
> > networks:
> >
> > rpcbind:PARANOID:DENY
> > rpcbind:0.0.0.0, 127.0.0.1, 10.0.1.0/255.255.255.0 :ALLOW
> > rpcbind:ALL:DENY
>
> In order for rpcbind(8) to actually heed /etc/hosts.{allow,deny} it
> needs to be started with
>
> -W Enable libwrap (TCP wrappers) support.
>
> which for whatever reason is not the default.
Ah, yes! Very good point! Thank you!
This is one of the problems with "fixing" one's local source tree and
forgetting what fixes are there!
--
Greg A. Woods
Planix, Inc.
+1 250 762-7675http://www.planix.ca/
pgpmwIqewIzlx.pgp
Description: PGP signature
Re: fork and COW
At Mon, 29 Jun 2020 21:48:13 +0200, Jaromír Dole ek
wrote:
Subject: Re: fork and COW
>
> However COW is only relevant before php-fpm does exec of ImageMagick.
> Immediately after fork the child shares the address space with the
> parent having the memory pages marked as COW, after exec the child no
> longer shares the virtual address space with the parent. There is
> never any address space sharing for the independently executed
> ImageMagick processes.
Note that there is in fact address space sharing between independently
executed processes, but only for their text (code) segments (and of
course that should include code pages for shared libraries too, iff the
program is dynamically linked). Stack and heap pages cannot be shared
between independently executed processes, of course.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpaEwh6BYNrF.pgp
Description: PGP signature
Re: How to configure npf to restrict nfs to localhost
At Mon, 29 Jun 2020 10:00:06 +0530, Mayuresh wrote:
Subject: How to configure npf to restrict nfs to localhost
>
> Looking to share host FS with a qemu guest using NFS.
>
> Do not want to expose the NFS ports to outside world. Firstly, what all
> ports are in question - is it 111, 1000 and 2049 (rpc,mount,nfs
> respectively) or is there anything else involved?
>
> Any hints for how to block these ports for outside world and keep open for
> localhost?
Stopping rpcbind from revealing ports other RPC servers are listening on
is the primary thing you need to do. You can do this with filters
blocking TCP and UDP ports #111, and/or with rpcbind itself using its
built-in libwrap support, like so:
In your /etc/hosts.allow file you can restrict rpcbind to given
networks:
rpcbind:PARANOID:DENY
rpcbind:0.0.0.0, 127.0.0.1, 10.0.1.0/255.255.255.0 :ALLOW
rpcbind:ALL:DENY
Make sure you do not run rpcbind(8) with its "-i" (insecure) option!
Note you may want to enable NFS server locking support with
"lockd=${nfs_server}" and "statd=${nfs_server}" in /etc/rc.conf, i.e. if
your virtual machine runs an OS that has client support for NFS locking
(NetBSD does not).
In your /etc/exports file you can further restrict an exported
filesystem to a specified network range like this example:
/ -alldirs -maproot=nobody -network 10.0.1.0 -mask 255.255.255.0
Further filtering external traffic to/from all possible RPC ports,
i.e. all of those in the range 600-1023 (IPPORT_RESERVEDMIN to
IPPORT_RESERVED-1), 49152-65535 (sysctl net.inet.ip.anonportmin to
sysctl net.inet.ip.anonportmax), and 2049 (NFS_PORT), is another added
layer of protection. Filtering the whole ranges of reserved and
anonymous ports might be a bit too strict though. Unfortunately
rpcbind(8) doesn't have hooks to register filters for registered RPC
services, though one could periodically run "rpcinfo -p" to get the list
of actual RPC ports in use and use that to update the filters.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpWz6ZBhEFy0.pgp
Description: PGP signature
Re: cvs better than git?
At Mon, 22 Jun 2020 09:37:30 +0200, a l3x wrote: Subject: Re: cvs better than git? > > git follows a snapshot like approach to version control. but this view of > history bites you as can be seen in "merge commits". requiring "rebasing" > things and actually "rewriting history". this is what i dislike about the > design > of git. it is just a hash based object store. maybe that's the reason > why the cli is > cluttered with lots of details. merge commits call for trouble and for > rebase, this is why i consider the design of git as VCS broken at best. I wouldn't call it broken, not by a long shot -- it's just an outgrowth of our history of using lesser tools which provide a per-file snapshot. Indeed the enhanced version of CVS that NetBSD has been using is also effectively just a snapshot approach (as opposed to plain CVS which is just an easier way of managing a whole lot of per-file snapshots). > another approach e.g. is using a partial order of patches as utilized > by darcs, or pijul. Both Git and NetBSD's version of CVS can be treated as if they are managing a series of ordered patches, but for Git doing that also loses some of its ability to track history of changes, albeit only slightly. Indeed that's kind of what happens in NetBSD CVS now with movement of changes from the trunk to release branches, though the formalization of doing it would require additional tools and more discipline. In effect for Git in GitHub the way pull requests have to be merged sometimes does the same thing, but it's hard to tie together the pull request history with what's in the DAG. Personally I don't think partially ordered sets of patches have the right level of integrity to lead to good software hygiene though. I think snapshots ala Git are much better, though even in Git it's easy (because of all the extra features supporting practical usage) to have it bite your butt if you don't test things in a more structured way (which is only really practical with full automation of testing, and that's mostly outside of Git's purview). > i wish they had written darcs in idiomatic straight line c/c++. haskell, and > in > case of pijul rust are kind of show stoppers for me. Indeed. I've no idea why many (except noteably Larry McVoy and Linux Torvalds) who have set out to build a better VCS have chosen to use some language far more esoteric and unwieldy and practically un-portable than they could have. Even Common Lisp would likely have been better. Monotone is one exception, though I personally still object to C++ with vehemence. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgp5n4pSdR66B.pgp Description: OpenPGP Digital Signature
Re: cvs better than git?
At Sun, 21 Jun 2020 17:45:44 +0200, Johnny Billquist wrote:
Subject: Re: cvs better than git?
>
> Something done on your local repository is not truly committed to
> start with. And it should be run through unit tests and so on, on the
> central repository, before being committed to the central repository.
Actually that's not true, for any DVCS. The hash that identifies a
commit remains the same no matter where or when it is pushed or pulled,
and of course if the hash doesn't change then neither does the code.
Once you have a commit-ID you know _exactly_ what you have in every file
and in every directory of the working tree, no matter where you take it.
So with a DVCS of any kind you have to learn to distinguish between
various sources of various kinds of authority.
In most any DVCS, I think, and certainly in Git, it is easy to set up a
process that strictly implements a review process such that what ends up
on an authoritative branch in an authoritative repository (e.g. "trunk"
in the main project's primary repository on a central Git server) has
been fully tested and fully reviewed by as many people as required, yet
it should have the same commit-ID as it did when it was first committed
to some developer's repository on whatever machine they were on when
they completed it (there may be another node or two on the DAG to
account for merges, but the original node represents the code exactly as
it was on that final commit).
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgp3Oz8QFxccu.pgp
Description: OpenPGP Digital Signature
Re: Checking out src with Mercurial
At Fri, 19 Jun 2020 21:51:35 +0200, Johnny Billquist wrote: Subject: Re: Checking out src with Mercurial > > On 2020-06-19 20:19, matthew sporleder wrote: > > git clone with --depth 1, over http (instead of ssh), and with a few > > simple settings changes will make it work inside of 128M. > > Well, the whole point of virtual memory and demand paging is that you > don't have to have enough physical memory. I would hope that still > applies... Well, one still should prefer to keep the working set of pages well within the capacity of physical memory, but, yes, VM should allow simpler, but reasonable, programming models to work with larger data sets in smaller amounts of physical memory. Secondary storage vs. main storage is like molasses (even on a hot day) vs. jet streams. > My comment about have 128M (which, by the way, can be > considered a lot, when we talk about VAXen), was just about the > potential speed I possibly could expect. If git really requires that > people have at least 128M of physical memory to work, then I'd first > ask when did NetBSD break so badly that the amount of physical memory > becomes a limitation in this way, (a) History -- there's a _lot_ of it! (b) size -- since NetBSD supports so many platforms, and is such a usefully complete system, the total number and size of files of source code eventually adds up to a quite sizeable collection! > and second, why would a tool like > this require that much memory in the first place? (c) modern change tracking tools try to track changes to whole sets of files at once, so if you have lots of files, and lots of history, this combinatorial problem can sometimes bite at a bad time for the user of a tool trying to manage it all. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpOEHVSGjyCJ.pgp Description: OpenPGP Digital Signature
Re: Checking out src with Mercurial
At Fri, 19 Jun 2020 21:07:24 +0200, Jan Danielsson wrote: Subject: Re: Checking out src with Mercurial > >The tl;dr for those who don't want to read all that: If, five years > earlier, Rust had been in the shape it was when that post was written, > the Mercurial developers may have opted to port to Rust rather than try > to bend Python 3 to their will -- because many common assumptions they > made about Python were true in 2.x, but not 3.x. > >Regarding choosing Rust over "well designed C/C++ code": Going by > the blog, it seems some of their core developers really like Rust. I > imagine that the OsString/OsStr issue is probably quite relevant to > [portable] source management systems; probably things like that matter > to them. heh. I always thought at the beginnings of Mercurial that Python was only a suitable prototyping language for it and that a rewrite to C or better, at least for core functionality, should have been a VERY early step in a production implementation. But it just never got there, and then they suffered through the Python version gyrations with no appreciable gain. However considering Rust is, well Rust is a stupidly horribly overly complex language (and in a convoluted way unlike, say, D) that is currently very hard to support on a wide variety of platforms, unlike, say, Python which does run _everywhere_, even if slowly. However it isn't too hard to see how a Python-only developer would be intrigued by a language that's at least as capable as Python (in a "mine is as big as yours" kind of way). Now why they wouldn't prefer a truly safe language like, say, Ada (or Oberon or Eiffel, etc.), if that's their concern, I couldn't begin to guess. These days a decent well supported, very capable, and much more widely available language like Go would seem, to me at least, to be a literally more infinitely better choice. Of course a truly small and elegant language like V (or maybe Wren) would be more along my line of choice, though for the kind of project like Mercurial, well, as I said, I would have a very hard time arguing against Go. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpo6CCmIfbCq.pgp Description: OpenPGP Digital Signature
dealing with rewriting of history in Git repos (was: cvs better than git?)
At Fri, 19 Jun 2020 09:56:50 +1000, Paul Ripke wrote:
Subject: Re: cvs better than git?
>
> I've tripped over this, too. I use git specifically for speed, having
> just spinning rust and no ssd means other VCS are atrociously slow.
> I got annoyed enough that I figured a clean way out without blowing the
> repo away and refetching:
>
> https://www.stix.id.au/wiki/git_pull_merge_conflicts
>
> Basically, recreating a clean branch tied to the remote branch.
> I'm now "comfortable" enough with git that I haven't had to blow my
> repo away and start again, except in one case of a corrupted repo
> which I managed to pin on bad RAM.
Ah, yes, that should work well!
I use a little add-on called git-up(1), which automatically does the
stash/unstash part. It should probably also do the re-creation of the
local tracking branch too, and in fact it could always do that under
normal circumstances, just for safety (i.e. make the assumption that one
never commits to the local tracking branch of any repository you don't
normally also push to so there should never be any need to rebase or
merge it).
(1) git-up is:
$ pip show git-up
Name: git-up
Version: 1.6.1
Summary: A python implementation of 'git up'
Home-page: https://github.com/msiemens/PyGitUp
Author: Markus Siemens
Author-email: mar...@m-siemens.de
License: MIT
Location: /usr/pkg/lib/python2.7/site-packages
Requires: GitPython, colorama, termcolor, six, click
Required-by:
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpwYx3XLTvc6.pgp
Description: OpenPGP Digital Signature
don't bother to use NetBSD Git repos for anything but testing (was: cvs better than git?)
At Thu, 18 Jun 2020 13:00:50 -0400, Andrew Cagney wrote: Subject: Re: cvs better than git? > > On Wed, 17 Jun 2020 at 23:23, Mayuresh wrote: > > > > On Wed, Jun 17, 2020 at 03:42:44PM +0530, Mayuresh wrote: > > > For pkgsrc I prefer the git mirror, as I don't have to push anything > > > anyway and a few hours of latency doesn't matter to me. > > > > Don't know whether it's relevant to say on this thread. May be it is. > > > > When I use pkgsrc git mirror, after every few days I get this error when I > > pull: "fatal: refusing to merge unrelated histories" > > It isn't you. > > This is a problem with the script that converts the repo from CVS (be it HG > or GIT). For some reason, from time to time, commit history gets a > complete rewrite. It is probably not the fault of the conversion process, nor CVS per se. It's probably someone, i.e. some NetBSD developer with sufficient privileges, messing around directly in the CVS repository and doing things that are contrary to even CVS best practices, let alone what's good for a repo that undergoes regular conversion. Such conversion processes typically require absolutely frozen and stable history, and often even innocuous-seeming changes can cause a ripple in the middle of the history that must then cause all the changeset hashes to be recomputed. Only sometimes, and only with extreme care, can such conversion processes be used securely for any kind of continuous migration. I've witnessed this regularly with the little tiny SCCS to Git conversion tool I've been maintaining. There's really no point to actually trying to use the Git repos for anything except infrequent testing, and possibly no point to using the Hg repos either, until the conversion is 100% frozen with no possibility of any new change ever being introduced to the original CVS repo again. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpd3iGSGiuNF.pgp Description: OpenPGP Digital Signature
Re: "hg clone https://anonhg.netbsd.org/src/" still aborts, but...
At Thu, 11 Jun 2020 20:41:58 -0700, bch wrote:
Subject: Re: "hg clone https://anonhg.netbsd.org/src/; still aborts, but...
>
> Nb: you’ll want to have an .hg/hgrc w a line:
> default = https://anonhg.netbsd.org/src
>
> ...so that in the future you can just “hg pull” from w/i that repo.
Thanks for the tip -- I figured something would have to be set like that.
Looks like it's not quite right though:
$ cat .hg/hgrc
default = https://anonhg.netbsd.org/src
$ hg incoming
abort: repository default not found!
The manual section for "hg clone" does say:
The location of the source is added to the new repository's .hg/hgrc
file, as the default to be used for future pulls.
However hgrc(5) suggests the syntax might have to be a bit different,
more like a .git/config.
Ah ha! It looks like this has to be in the "[paths]" section, and MUST
NOT be proceeded by a tab or other whitespace (which .git/config allows):
$ cat .hg/hgrc
[paths]
default = https://anonhg.netbsd.org/src
$ hg incoming | head
comparing with https://anonhg.netbsd.org/src
searching for changes
changeset: 931876:26c8f37631b6
branch: trunk
user:maxv
date:Sat May 02 11:12:49 2020 +
summary: Remove unused.
changeset: 931877:42596ac89b6e
branch: trunk
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpxaVF0Hfpg2.pgp
Description: OpenPGP Digital Signature
Re: "hg clone https://anonhg.netbsd.org/src/" still aborts, but...
At Fri, 12 Jun 2020 07:34:55 -0400, matthew sporleder wrote: Subject: Re: "hg clone https://anonhg.netbsd.org/src/; still aborts, but... > > On Thu, Jun 11, 2020 at 11:34 PM Greg A. Woods wrote: > > > > Just a wee while ago it was again mentioned that 'hg clone' would be a > > suitable way to download NetBSD sources, but I've been trying this off > > and on for over a month now and always end up with a failure and abort > > just like this attempt today: > > > > $ hg clone https://anonhg.netbsd.org/src/ h-NetBSD-src > > applying clone bundle from > > https://cdn.NetBSD.org/_bundles/src/77d2a2ece3a06d837da45acd0fda80086ab4113c.zstd.hg > > adding changesets > > adding manifests > > manifests [> ] 751718/931876 > > 48m52s > > transaction abort! > > rollback completed > > abort: stream ended unexpectedly (got 32754 bytes, expected 32768) > > curl > 'https://cdn.NetBSD.org/_bundles/src/77d2a2ece3a06d837da45acd0fda80086ab4113c.zstd.hg' > |md5 > 394158db5733b878963e9d6a5eb721d5 > > what did you get? why does that clone fail? $ md5 /build/work-tmp/77d2a2ece3a06d837da45acd0fda80086ab4113c.zstd.hg MD5 (/build/work-tmp/77d2a2ece3a06d837da45acd0fda80086ab4113c.zstd.hg) = 394158db5733b878963e9d6a5eb721d5 As shown above it would seem that something on the server side, or between me and the server, simply drops the connection after transmitting some portion of the data or after some time has elapsed. My guess is that when "hg clone" opens the connection then it cannot fetch the data fast enough and something somewhere times the connection out. However when the data is fetched first by direct means then the connection stays open for long enough to pull it all. Probably it is a load balancer, a NAT, or some other kind of tunnel that times out. I think only someone familiar with the whole infrastructure in front of cdn.NetBSD.org would be able to make a better guess. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpMGKhGNQOs9.pgp Description: OpenPGP Digital Signature
Re: "hg clone https://anonhg.netbsd.org/src/" still aborts, but...
At Fri, 12 Jun 2020 00:26:26 -0700, "Greg A. Woods" wrote: Subject: Re: "hg clone https://anonhg.netbsd.org/src/; still aborts, but... > > I'll now fire up a new "git clone" next for a more up-to-date > comparison. There will be another rsync && cvs start during this, just > to be fair. :-) And it's done. The whole "git clone" completed far faster than just the last "hg checkout trunk" step (i.e. in just under 1.5hrs, 36mins faster than the HG checkout alone, and 3.5hrs faster than the whole "hg clone"): 00:01 [685] $ time git clone https://github.com/NetBSD/src g-NetBSD-src-test Cloning into 'g-NetBSD-src-test'... remote: Enumerating objects: 1274, done. remote: Counting objects: 100% (1274/1274), done. remote: Compressing objects: 100% (875/875), done. remote: Total 5117234 (delta 642), reused 675 (delta 396), pack-reused 5115960 Receiving objects: 100% (5117234/5117234), 1.87 GiB | 2.42 MiB/s, done. Resolving deltas: 100% (3876350/3876350), done. Checking out files: 100% (171999/171999), done. 5061.00s real 510.15s user 409.84s system 01:26 [686] $ The last step, the checkout, took most of the time, at least an hour, so on a local fast disk that wasn't otherwise being bombarded by background activity, it would have been decently fast and arguably "usable", all things considered. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgp_oCXWmEm4A.pgp Description: OpenPGP Digital Signature
Re: "hg clone https://anonhg.netbsd.org/src/" still aborts, but...
At Fri, 12 Jun 2020 09:50:27 +0200, Andreas Kusalananda Kähäri wrote: Subject: Re: "hg clone https://anonhg.netbsd.org/src/; still aborts, but... > > Well, you're asking to fetch all the history for all the sources to an > actively developed operating system project running since at least 1993 > (27 years). But of course. That's the point. The suggestion earlier on this list (netbsd-users) was to do exactly: hg clone https://anonhg.netbsd.org/src/ a) That doesn't actually work at the moment (and hasn't, ever?). b) Even when you work around the problem, it's TERRIBLY slow. Arguably it is unusably slow. > With git, you can use --depth=1 when cloning to create a shallow clone. > This would limit the amount of data fetched. I imagine Mercurial has > some similar option (but I don't really know). Yes, but the intent of comparison with Git is to do the same as with HG, i.e. in the same normal way that most DVCS-using projects expect users to fetch their sources -- i.e. fetch _all_ the history _all_ the time. From this initial experience with HG in such a "large" project, it seems the problem isn't really with the amount of data fetched, but rather with the amount of processing required to convert it from a "bundle" format back into an on-disk repository format, plus some unexplained amount of extra time it takes to extract the files from an HG repository when doing the initial checkout (since obviously the amount of data output to the filesystem must be identical between the two checkouts). That's why I asked if there's a more efficient mode of supplying whatever data is needed for an HG clone to take place (i.e. instead of supplying it in the "bundle" format). At the moment it looks like Git is going to finish about 3.3 times as fast while suffering equivalent amounts of background activity occurring separately on my NFS server, i.e. in under 1.5hrs compared to HG's 5hrs. Once I can get my fastest server rebooted I'll be able to compare both without having competing file server activity, though ultimately what I've done this evening is probably a more fair test in an environment that might better mimic what other users will do. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpkaa7G9mBeN.pgp Description: OpenPGP Digital Signature
Re: "hg clone https://anonhg.netbsd.org/src/" still aborts, but...
OK, so the initial "hg clone" equivalent took almost exactly 3 hrs. 17:46 [663] $ ftp https://cdn.NetBSD.org/_bundles/src/77d2a2ece3a06d837da45acd> Requesting https://cdn.NetBSD.org/_bundles/src/77d2a2ece3a06d837da45acd0fda80086ab4113c.zstd.hg 100% |***| 1285 MiB6.86 MiB/s00:00 ETA 1347521045 bytes retrieved in 03:07 (6.84 MiB/s) 17:49 [664] $ 17:55 [669] $ hg init h-NetBSD-src 17:55 [668] $ cd h-NetBSD-src 17:55 [669] $ hg unbundle /build/work-tmp/77d2a2ece3a06d837da45acd0fda80086ab4> adding changesets adding manifests adding file changes added 931876 changesets with 2425841 changes to 439702 files (+417 heads) new changesets 8cec458d70ff:77d2a2ece3a0 (run 'hg heads' to see heads) 20:48 [670] $ The final "hg checkout trunk" took almost exactly 2 hrs: 20:48 [671] $ time hg checkout trunk 171685 files updated, 0 files merged, 0 files removed, 0 files unresolved 7204.00s real 501.37s user 1645.00s system 22:48 [672] $ So, that's 5hrs total to do an "hg clone". Wow. Now this was to an NFS-hosted filesystem, and there was another "rsync && cvs update" running during some of that time on the NFS server, but still (the rsync and other cvs activity normally only occupies about 1/2 or less of the file server's capacity according to sysstat). I'll now fire up a new "git clone" next for a more up-to-date comparison. There will be another rsync && cvs start during this, just to be fair. :-) -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpMP1GK6S4dV.pgp Description: OpenPGP Digital Signature
"hg clone https://anonhg.netbsd.org/src/" still aborts, but...
Just a wee while ago it was again mentioned that 'hg clone' would be a suitable way to download NetBSD sources, but I've been trying this off and on for over a month now and always end up with a failure and abort just like this attempt today: $ hg clone https://anonhg.netbsd.org/src/ h-NetBSD-src applying clone bundle from https://cdn.NetBSD.org/_bundles/src/77d2a2ece3a06d837da45acd0fda80086ab4113c.zstd.hg adding changesets adding manifests manifests [> ] 751718/931876 48m52s transaction abort! rollback completed abort: stream ended unexpectedly (got 32754 bytes, expected 32768) So after reading the line of output again and noticing it's a URL I decided to try downloading th bundle directly. This worked fine, taking about 3 minutes for me. I then did: "hg init src && cd src && hg unbundle $bundle" The unbundle finally finished the first step (unpacking all 931,876 changesets) after about 15 minutes and began on the manifests, just as with the clone operation did. The manifests then completed, unlike the attempts via the network, though I don't know how long that took, and now it's checking out 439,702 "file changes", saying it will be taking over 2hrs to complete. HG has the world's second-worst task time estimator though -- it's been waffling between 10 to 30 mins for the past hour or more (on the "files" step). Overall that's _STUNNINGLY SLOW_ compared to a "git clone" of the same NetBSD source tree -- especially since I've eliminated the network for the HG test case! I'll have to try an up-to-date "git clone" again to be compareing more apples-to-apples, but IIRC the "git clone" of the src tree works at least an order of magnitude faster on the same platform, and direct from the network. Perhaps this slowness is because even the initial clone attempt is/was working from the one big complete bundle-format file? Does HG have any more efficient way to supply a clone? And also, why does the network clone fail, but a fetch+unbundle work? -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpYsihqdaY8J.pgp Description: OpenPGP Digital Signature
Re: requesting support for persistent memory
At Wed, 10 Jun 2020 14:13:57 +0200, "mayur...@kathe.in"
wrote:
Subject: requesting support for persistent memory
>
> i noticed; https://pmem.io/
>
> they are supporting windows and linux using some 'dax' (direct-access)
> technology. would netbsd experts too work towards bringing-in support
> for persistent memory? i believe it could pave the way for a new
> breed of applications since it's a whole new programming paradigm.
Heh. Kind of hokey looking.
Though I agree it is time to stop viewing data through the eye of a
needle all the time and instead be able to just address it like memory.
However this PMEM thing sure as heck isn't Multics-like enough for me to
consider it.
I think things like NVDIMMs, etc., should be used as performance
enhancements for a more orthogonal single-level storage model, not as an
alternative storage device.
I also really don't see the point of trying to bolt such APIs onto
POSIX-like operating systems -- much smarter to build an underlying OS
with single-level storage and then bolt POSIX compatibility onto that
(which is incredibly easy to do by comparison).
In fact that's exactly what I want to do.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpT33iXhOWyB.pgp
Description: OpenPGP Digital Signature
Re: Optional crunchgen base
At Mon, 8 Jun 2020 15:09:47 +0100, David Brownlee wrote: Subject: Re: Optional crunchgen base (Was: Postfix and local mail delivery - still relevant in 2020?) > > On Sun, 7 Jun 2020 at 18:35, Greg A. Woods wrote: > > [...] > > To really make this useful for general NetBSD builds will take some more > > hacking on the build system (basically it might go something like always > > building a ".cro" file for every program possible, then generating a > > crunchgen config to link them all together and generate an mtree file to > > generate all the (sym)links; and possibly doing it on a per-set basis > > (e.g. one binary for base, one for the toolchain, one for x11, or > > something like that). > > Would it be possible to persuade you to take a look at this for > current NetBSD? If it was possible to build base like this it would > give an opportunity for people to test on a variety of older and > memory constrained boxes... I am slowly working at it! A little less slowly now that I don't have a $dayjob to interfere. The way I did the experiment for netbsd.NET5501.EMBED was far too hackish as it was based on the installer image build, and that requires re-compiling every program for every image. It didn't take a lot of work to do initially though as the ramdisk image builds already held all the necessary infrastructure and machinery to do it. I've only recently come up with the idea of doing the *.cro builds in parallel with regular builds. I fact it would probably be nearly as efficient to also build both dynamic-linked and static binaries all at the same time too. However I've since been struggling with how to create sets and/or final bootable images in a more efficient and flexible manner. I recently realised that it might be possible to have the sets-building scripts generate the sets dynamically with symlinks and to also do the final link step of the ramdiskbin itself (based on what programs were being pulled into the set file). As-is this might require some info be made available showing where the objdir for each program is (and this might be automatically generated when the normal binary is installed, e.g. as part of the mtree file). Sets could then also be used to create the ramdiskbin for all ramdisk images that currently use separately compiled crunchgen files. That way even if you wanted to build a bootable image with some different combinations of programs, e.g. base+misc+text, all the required programs could still be linked into the same ramdiskbin. Alternatively a "master" destdir could be built where every program is actually in a sub-directory (e.g. as in macOS .app dirs), and then the sets builders and image builders could choose either the static binary, the dynamic-linked binary, or the .cro file when creating the final filesystem image. And of course only one kind of binary would have to be built, with the others only built as options. All-told though any of this may be a pretty invasive change to the build overall, but would probably worthwhile in order to achieve the necessary efficiencies. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpFpWHY5TVQT.pgp Description: OpenPGP Digital Signature
shared libraries vs. static binaries (Was: Postfix and local mail delivery - still relevant in 2020?)
At Mon, 8 Jun 2020 08:58:51 +0100, Sad Clouds wrote: Subject: Re: Postfix and local mail delivery - still relevant in 2020? > > On Sun, 07 Jun 2020 15:12:56 -0700 > "Greg A. Woods" wrote: > > > However when you put _all_ the code for _all_ the system's programs > > into one single lone binary, with no shared libraries, then _all_ > > text pages are shared entirely for _all_ processes all of the time, > > no matter what program they are running as. > > OK thanks for the explanation. So as long as it is a single executable, > text pages are shared, but if you create multiple executables on disk, > i.e. different inodes, then you do get duplicate text pages loaded into > RAM? Yes, though "duplicate text pages" is a misnomer -- there will be some fragments of code duplicated into text pages in each binary, and each of those pages from each binary will have to be separately paged in when a process starts from each binary. So there will be duplicate code in separate pages in memory when you have two different programs running, but each uses, say, some functions from libc. (but only one copy per process for each given program) In my personal experience the tradeoff of having all static-linked but separate binaries per program is still well worthwhile (provided you're not using an extremely RAM-starved system). The disk use is probably the biggest loss. ~2GB for i386 install (without x11, another 600MB for x11). Link time and time to build sets and their checksums is also longer, but only developers do that every day. :-) There's currently still also a fair chunk of local diffs in my source tree to make it all work, and that requires merging on every update, but for now that's just my own personal overhead. However because the static linker is very efficient when given the kinds of libraries NetBSD has (where each function is in its own compilation unit and so there's usually only one function per .o file), each program contains only just the code for the functions it needs, and also because demand paging for executables is also a wondrous idea that means each running process can have a relatively small resident set size of text pages in memory. (It could be even more efficient if less-used code were put into separate pages, but that's a tricky proposition -- though I've heard of experiments with re-adjusting page layout after having collected stats from typical use of the system.) So, for example, that little Soekris i386 machine with only 512MB of RAM running a fully static linked individual binary install still has only 3% of its memory (4400 pages) in use for executable code pages once it's running a normal multi-user boot and with a couple of shell sessions open from locally running xterm processes (open to the Xserver on my desktop machine). Meanwhile because nothing needs to run ld.so for every process exec, the startup time of commonly used processes (i.e. ones that can be cached in cached executable pages) is incredibly fast and snappy, especially for a system with a 500MHz clock. It "feels" faster for command-line use than a machine nearly 10 times as fast that's running all dynamic-linked binaries (and don't even get me started on the brain damage of paging that happens in an even more aggressively dynamic-linked OS like macOS when you exec or re-exec a big program that uses several libraries and frameworks, even with "prelinking", e.g. a modern browser). Starting static-linked X11 processes like xterm is incredibly fast, as they typically need even more libraries than, for example, the postfix programs. Long long ago I posted some raw performance tests, taken on an old Sun3 I think, for an X program that didn't have to actually open a connection and could be run in a shell loop for timing, but still needed a half dozen libraries or more. If I remember right the loop running the static-linked version was nearly two orders of magnitude faster. The real fun begins with a bigger more modern server, e.g. my Dell systems with 32GB of RAM and 3,160MHz CPUs. Now the savings from not having to run ld.so for every process exec is amazing, and while one might lose a few percent of that vast memory to exec pages, the tradeoff is still very good. A static-linked tool-chain still means faster builds (despite the longer link and sets times)! -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpo6tYKPWZNC.pgp Description: OpenPGP Digital Signature
Re: Postfix and local mail delivery - still relevant in 2020?
At Sun, 7 Jun 2020 20:19:59 +0100, Sad Clouds wrote: Subject: Re: Postfix and local mail delivery - still relevant in 2020? > > On Sun, 07 Jun 2020 10:35:09 -0700 > "Greg A. Woods" wrote: > > > Now ideally what I want to do for embedded systems is static-link > > every binary into one crunchgen binary. I've done this for 5.2 on > > i386, and the whole base system (or most of it, no compiler, tests, > > or x11; and no ntpd or named or postfix) is just 7.4MB compressed. > > Yes, just 7.4 megabytes: > > So when you run this binary as multiple concurrent processes, aren't > they all going to use more RAM as they load duplicate copies of shared > libraries that have been statically linked into it? No, as this is one static-linked binary. There are no shared libaries. Also "shared" libraries are a bit of a misnomer. Yes, their pages are shared, _but_ some of the most expensive work in using them must be done for _every_ exec (at least in NetBSD), even repeated execs of the same program. Meanwhile _all_ text pages are also shared for each binary. I.e. NetBSD demand-pages text pages from program files. So all processes running, say, /bin/sh will only have one shared set of text pages in use between them all (plus one set of pages for each shared library), no matter how many /bin/sh processes there are running at any one time. Now if you static-link /bin/sh then you can start a new instance without ever having to go to disk to run it, and also without first having to run the dynamic linker. Individual static-linked programs start very fast, especially if another instance is already running -- once the process is set up it's literally just jumping to code already in memory and getting right to work immediately. However when you put _all_ the code for _all_ the system's programs into one single lone binary, with no shared libraries, then _all_ text pages are shared entirely for _all_ processes all of the time, no matter what program they are running as. The time it takes to exec and start a new unique program (one not yet running) is now just the time it takes to maybe load the one page of code containing its main() function. As it branches out and runs more of itself then it might need to load a few more unique pages, but as it makes library calls it will likely be jumping to code that some other process already loaded. No matter how many of those 274 programs that I put into that one 10MB binary you run, and no matter how many instances of each you run, there will NEVER EVER be more than 10MB of text pages in memory for _all_ processes simultaneously. 10MB total. For the whole base system. The whole base system is really just 10MB of code, total. Yes each process will allocate more pages for data and stack, but those are just memory allocations, never paging in from disk. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpRtzkSL2OhQ.pgp Description: OpenPGP Digital Signature
Re: Postfix and local mail delivery - still relevant in 2020?
At Sun, 7 Jun 2020 08:45:51 +0100, Sad Clouds
wrote:
Subject: Re: Postfix and local mail delivery - still relevant in 2020?
>
> I'm curious, what sort of essential information do these emails
> provide on a daily basis? Is it simply that some cron job completed
> successfully?
It's the unsuccessful ones that are most interesting. If I get mail
from cron it's because something is broken or otherwise need attention.
> Personally, I would like to see graphs and charts of
> cpu/memory/disk/network usage. I would like to archive various security
> and cron logs in a different location, so that hackers cannot easily
> delete them. If I'm running email/web/database services, I would like
> to archive all logs, statistics and performance metrics, on a frequent
> (maybe hourly?) basis. If the database has performance issues at
> specific times, I would like to be able to go back in history and
> analyse all logs and metrics. Some of those metrics could be stored in
> binary files and may need specific tools to extract visual graphs, etc.
Actually an excellent way to collect system information and reports, one
many of us have used for years, is to email them to a central repository
where they can be processed. This is probably more work for just one
system, but as soon as you have more than one system, it becomes more
and more appealing.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgp8cAaptNY8f.pgp
Description: OpenPGP Digital Signature
Re: Postfix and local mail delivery - still relevant in 2020?
At Sun, 7 Jun 2020 12:46:51 +0200, Johnny Billquist wrote: Subject: Re: Postfix and local mail delivery - still relevant in 2020? > > On 2020-06-07 07:32, Greg A. Woods wrote: > > > > At Sun, 7 Jun 2020 01:53:34 +0200, Johnny Billquist > > wrote: > > Is the slow startup you observe happening with just the default > > configuration (for local delivery)? > > It's happening with any kind of configuration. > > Basically, anything written in C++ is almost a lost cause even before > starting. Do you mean you're under the impression Postifix is written in C++? Perhaps you should look at the Postfix code -- it's entirely pure C. > postfix alone takes close to a minute to get going at boot time on my > VAXen... > And that is just counting before rc continues to the next item. I > haven't checked if postfix are then still being busy actually getting > things sorted in the background... That's very interesting. The slowest machine I have running at the moment is a little old Soekris box, with a FLASH disk as root: # time /etc/rc.d/postfix restart postfix/postfix-script: stopping the Postfix mail system postfix/postfix-script: waiting for the Postfix mail system to terminate postfix/postfix-script: starting the Postfix mail system 3.31s real 1.21s user 0.76s system Now that's a little unfair for two reasons. First off the programs were just running so the good sized parts of their text segments were already in memory. So I stopped Postfix, flushed it from RAM by reading a bunch of big files from an attached cache disk and started it again: # time /etc/rc.d/postfix start postfix/postfix-script: starting the Postfix mail system 3.29s real 1.08s user 0.77s system So, that's still pretty fast. Now the second unfairness is this: # file /usr/libexec/postfix/master /usr/libexec/postfix/master: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for NetBSD 8.99.32, stripped I.e. my whole build is static-linked. This is perhaps the big problem since the two processes that have to start are like this on stock builds: $ ldd /usr/libexec/postfix/master /usr/libexec/postfix/master: -lsaslc.0 => /usr/lib/libsaslc.so.0 -lcrypto.14 => /usr/lib/libcrypto.so.14 -lcrypt.1 => /lib/libcrypt.so.1 -lc.12 => /usr/lib/libc.so.12 -lgcc_s.1 => /usr/lib/libgcc_s.so.1 -lssl.14 => /usr/lib/libssl.so.14 -lgssapi.11 => /usr/lib/libgssapi.so.11 -lkrb5.27 => /usr/lib/libkrb5.so.27 -lhx509.6 => /usr/lib/libhx509.so.6 -lasn1.10 => /usr/lib/libasn1.so.10 -lcom_err.8 => /usr/lib/libcom_err.so.8 -lroken.20 => /usr/lib/libroken.so.20 -lutil.7 => /usr/lib/libutil.so.7 -lwind.1 => /usr/lib/libwind.so.1 -lheimbase.2 => /usr/lib/libheimbase.so.2 -lsqlite3.1 => /usr/lib/libsqlite3.so.1 -lheimntlm.5 => /usr/lib/libheimntlm.so.5 -lldap.4 => /usr/lib/libldap.so.4 -llber.3 => /usr/lib/liblber.so.3 $ ldd /usr/libexec/postfix/qmgr /usr/libexec/postfix/qmgr: -lsaslc.0 => /usr/lib/libsaslc.so.0 -lcrypto.14 => /usr/lib/libcrypto.so.14 -lcrypt.1 => /lib/libcrypt.so.1 -lc.12 => /usr/lib/libc.so.12 -lgcc_s.1 => /usr/lib/libgcc_s.so.1 -lssl.14 => /usr/lib/libssl.so.14 -lgssapi.11 => /usr/lib/libgssapi.so.11 -lkrb5.27 => /usr/lib/libkrb5.so.27 -lhx509.6 => /usr/lib/libhx509.so.6 -lasn1.10 => /usr/lib/libasn1.so.10 -lcom_err.8 => /usr/lib/libcom_err.so.8 -lroken.20 => /usr/lib/libroken.so.20 -lutil.7 => /usr/lib/libutil.so.7 -lwind.1 => /usr/lib/libwind.so.1 -lheimbase.2 => /usr/lib/libheimbase.so.2 -lsqlite3.1 => /usr/lib/libsqlite3.so.1 -lheimntlm.5 => /usr/lib/libheimntlm.so.5 -lldap.4 => /usr/lib/libldap.so.4 -llber.3 => /usr/lib/liblber.so.3 That's an awful lotta libraries to rattle through! The runtime linker isn't terribly fast as it has quite a bit of computation to do, and just doing the I/O alone to page in enough of all those files on an older machine is going to take quite a long time. Now I don't know what your storage situation is on your VAXen, but if you can possibly afford to static-link your build you'll find things start so much faster you'll be VERY surprised. Static linked those two programs are just
Re: Postfix and local mail delivery - still relevant in 2020?
At Sun, 7 Jun 2020 01:53:34 +0200, Johnny Billquist wrote:
Subject: Re: Postfix and local mail delivery - still relevant in 2020?
>
> That I disagree with. Being a person who actually runs NetBSD on some
> older hardware, postfix is a real resource hog. It takes forever to
> start up, while sendmail takes off in no time. I actually like
> sendmail, while I find postfix to be way over complicated.
> (But yes, I can understand people not liking sendmail.cf, it takes a
> bit to grow on you.)
Is the slow startup you observe happening with just the default
configuration (for local delivery)?
Even static-linked it's only a couple of still relatively small
processes, just over 1MB RSS each, on my machines. Though I suppose
they do have to open and parse a few more config files, and the qmgr
process is rather bloated in total size (but it doesn't seem to have to
read in much more than 1/4 of itself to get going).
As for complexity, well sendmail.cf is just the most visible surface of
the problem -- the real issue is inside the code. Postfix does have a
somewhat more complex overall design, depending on how you look at it,
as it's not just mostly one big monolithic program that's mostly a state
machine driver. However the Postfix code is almost trivial to read and
comprehend in comparison to Sendmail's code. The whole package is just
over 133,000 lines (pure code, comments stripped) of extremely well
documented easy-reading code; with the actual programs that run in the
default config accounting for fewer than a total of 8000 lines total of
code together (i.e. not including the project libs they also make use of).
The Research Unix (and Plan-9) UPAS mailer is possibly more elegant than
Postfix in both design and implementation, and incredibly small in
comparison (at only 9k lines of code _total_), but ultimately it is
_far_ less flexible and _far_ less featureful; and probably less
scalable. For small machines though it would be a good basic MTA.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpdoLF25SV19.pgp
Description: OpenPGP Digital Signature
Re: Postfix and local mail delivery - still relevant in 2020?
Software and systems (and often sysadmins and possibly other users) send
email. Some such software is even part of the base NetBSD system
distribution. This is as true in 2020 as it was in 2000 or even 1995.
Email is often much more conveniently delivered to a remote or central
server (i.e. as opposed to being delivered to a locally mounted
filesystem).
Delivering mail to a remote mail server requires MTA software that is
capable of delivering mail via the network.
Postfix is a most excellent MTA that can be configured to deliver mail
via the network.
Conveniently Postfix is also a most excellent MTA that can be configured
to (also) _receive_ mail via the network.
Postfix's licensing and portability and ease of configuration and
operation (as well as its extensive features and scalability) make it
most excellent software for NetBSD.
Ergo Postfix is in the base NetBSD system distribution.
(in case you didn't know, once upon a time the MTA in NetBSD was
Sendmail -- Postfix is more excellent than Sendmail in every way)
See also the "Mail Aliases" and "Postfix" sections in afterboot(8).
(BTW, when I set a standards-compliant "reply-to" header in my email, it
means I want any reply to go there, unless I've pointed it to a public
list and the reply is not to be public. I don't need to be personally
CCed on replies that I've directed to a public list.)
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgp_hwG5Hikee.pgp
Description: OpenPGP Digital Signature
Re: Postfix and local mail delivery - still relevant in 2020?
At Sat, 6 Jun 2020 20:05:09 +0100, Sad Clouds wrote: Subject: Re: Postfix and local mail delivery - still relevant in 2020? > > OK, but does this really require the entire Postfix infrastructure? > A small mail delivery tool would be sufficient, e.g somebody mentioned > Dragonfly mail agent. The NetBSD mailing list archives contain long discussions about this very issue. I would suggest it is still very relevant to have a network capable MTA in the base system distribution, and for that purpose Postfix is a most excellent choice. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpTYQ8z9hkcf.pgp Description: OpenPGP Digital Signature
Re: Securing DNS traffic
At Mon, 25 May 2020 19:51:52 -0400, "Aaron B." wrote:
Subject: Re: Securing DNS traffic
>
> Again, I'd prefer to run my own resolvers, but can't justify the
> expense.
I would recommend begging or borrowing _any_ old used computer that can
run any open-source OS (though ideally NetBSD, of course) and support at
least two Ethernet ports, and set it up as a firewall (with NAT) between
your home network and your ISP's router. Hook the cable modem to it and
run all your own networking through it.
Then you can run your own DHCP server and resolver (e.g. unbound), your
own NTP server, and possibly even some other services, such as SSH
(perhaps on a non-standard port for the ISP-facing interface); as well
as of course using it as a proper firewall too. With a WiFi card it can
also be your access point.
I currently use my Apple Time Capsule as the router/firewall/DHCP server
and run the resolver, etc. on a cheap old used server (actually on a VM
running on Xen on that cheap old used server). The time capsule is
technically using NetBSD too. (Though now that Apple has dumbed down
the AirPort Utility to basically cripple it, I'll soon have to migrate
to a newer machine for routing -- something with better gigabit-speed
throughput, as keeping the old laptop to run the old AirPort Utility is
not viable.)
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgprTIEkuyggn.pgp
Description: OpenPGP Digital Signature
Re: NetBSD Jails
At Thu, 21 May 2020 00:17:27 -0400, "Aaron B." wrote: Subject: Re: NetBSD Jails > > On Wed, 20 May 2020 14:47:52 -0700 > "Greg A. Woods" wrote: > > > Well if all your chroot tree of processes runs as a single unique user > > then from what I understand secmodel_extensions "Curtain Mode" already > > does actually do all of the rest of what you need. > > > > Curtain mode does not. I think you may have misunderstood what I meant by "tree of processes runs as a single _unique_ user". I mean "unique from any other user running any other process on the system, chrooted or not"; i.e. absolutely unique to that chroot instance and that one alone. > Some applications run as multiple processes with multiple users, by > design. Curtain mode as I understand it would tear these apart. So, that's covered by the rest of my discussion and depends on exactly how those multiple users get their credentials, and exactly what they need to be able to do and not do with their different IDs. 1. You have a master control process that runs as root and does the chroot() and then forks a number of processes each of which then calls setuid() as necessary before execing each application process that runs together in the chroot instance. 2. You have a program or script that calls chroot(8) (or its moral equivalent) and starts your application which then sometimes has to use enhanced privileges and to do so it forks and execs a setuid binary within the chroot area, and note this will not be for superuser privs, but just some other application user. In both cases there may need to be per-chroot assigned UIDs. The "disk is cheap" comment was meant to suggest that each chroot area be a unique stand-alone sub-directory and thus each per-chroot instance gets its own unique new UIDs assigned at the time it is first created. If such a scheme requires too much disk space then that's where the option of using mount_umap(8) (perhaps via non-superuser mounts) might solve the problem by sharing a single copy of the application storage directory but presenting unique ownerships (including for setuid binaries) to each chroot instance. However, still, in _all_ of these cases I think curtain mode should still provide all of the extra process and (non-network) resource isolation necessary between each chroot instance (where normal per-user controls and limits will still provide the foundations for controlling the process(es) within the chroot). If this is not true for you then perhaps you could be very specific about exactly what a chrooted process running as a unique user-ID, and behind a secmodel curtain, can still do that you don't want it to be able to do? > In addition, if I'm reading the documentation correctly - multiple > instances of the same application- or any two applications sharing the > same UID - would be able to interact. That is not isolation. Obviously -- even FreeBSD Jails do not really offer that kind of totally secure isolation (it's quite a bit better, I think, but not total, and that's also why it has to spread its tendrils in an ugly and (arguably) unmaintainable way throughout the whole FreeBSD kernel). > > So, is it necessary to support setuid binaries in a shared filesystem > > used to underly chrooted processes? > > > > For my specific use case, I don't think so. Processes are launched from > an external command that calls chroot, then switches to an unprivileged > UID as needed, then calls exec. Then you wouldn't/couldn't have multiple processes with different UIDs in any individual chroot instance, unless it's actually more like what I described in #1 above where it starts several processes and at least one or more of them runs as a different UID than the others. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpOhZJMMeuYW.pgp Description: OpenPGP Digital Signature
Re: NetBSD Jails
At Tue, 19 May 2020 10:21:52 +0200, Niels Dettenbach wrote: Subject: Re: NetBSD Jails > > Am Dienstag, 19. Mai 2020, 03:15:53 CEST schrieb Greg A. Woods: > > > > I still think the security and complexity issues with containers, are a > > very much bigger concern than the pure efficiency losses of running full > > VMs. > This is - from my view - a bit outdated view, because of the development. Sure, doing things smart/clean/elegant is definitely outdated when compared to the way many choose to work. As I said, most seem to see the apparent surface simplicity of "docker pull nginx" as elegant enough. > I would switch over more setups to NetBSD if jails would be available, > because i still prefer NetBSD over FreeBSD on such servers because it > is more Xen (PV) "friendly" at all. I think FreeBSD-style jails would be nice too, but only in theory. They seem to offer much more uniform APIs than that mess in Linux. However after looking at just how much "interference" jails cause in so many (most?) parts of the FreeBSD kernel, I just can't see it as any kind of good idea. It's like a fungus sending its tendrils everywhere adding complexity everywhere. The problem is that it seems unix-y monolithic kernels just are not architected well enough to make it easy to add such "isolation" features without either impacting whole swaths of subsystems, and/or introducing new APIs for practically everything (beyond POSIX). Maybe there's a better way to integrate jails using something like secmodel_extensions(9) (where we already have "curtain mode" and non-superuser CPU affinity control)? The odd thing is I think the advantage Linux containers (CGroups and namespaces, etc.) have is that their unique APIs allow them to be more cleanly integrated into the Linux kernel (though I can't say for certain, having not really looked at the code). I haven't looked enough at Solaris yet either to see how zones impact its kernel, though given what else I've seen of Solaris in general, I do hold out some hope that it is integrated in a more structured way, and I think they avoid at least some of the necessity of doing container management with unique new APIs the way Linux must. One of the things I've been hoping to learn in this discussion is more concretely what the true low-level requirements are, over and above what can be done with existing chroot and user/login-class rlimits in order to provide useful isolation of applications. If I look at an example list of "features" of various isolation and virtualization technologies, such as the on on Wikipedia[1], most of what I see in the comparison columns are things that OS academics would find cool and interesting, rather than requirements driven features. So what more is needed, beyond chroot and login classes, to make possible the kinds things like allowing a customer to install web-app "plugins" to their instance of a web server? I can't think of _anything_ else that's _actually_ needed, other than management tooling to make it all clickety-web-GUI-ish. You certainly don't need/want to give them root in their chroot. [1] https://en.wikipedia.org/wiki/Operating_system%E2%80%93level_virtualization_implementations#Implementations -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpRzSuesV8Hr.pgp Description: OpenPGP Digital Signature
Re: NetBSD Jails
At Sun, 17 May 2020 21:46:39 +0100, Sad Clouds
wrote:
Subject: Re: NetBSD Jails
>
> Your main gripe about jails/zones/containers is added complexity, well
> guess what, with Xen/VMware/VirtualBox the complexity is still there,
> you just pushed it over to the hypervisor vendor.
Actually that's not true at all, at least not for a type-1 (bare metal)
hypervisor like Xen. The hypervisor is a micro-kernel -- well contained
and actually quite simple (and this is even more-so since 4.11). Things
do get a bit more hairy if you have to add qemu to the stack, but that's
the exception for poorly conceived environment, and not (supposed to be)
the rule. Speaking of qemu, PVH dom0 will get rid of a whole lot more
complexity from dom0 as well.
Xen is also less complex at all the layers above the hypervisor as well
since you can use the exact same OS(es), the exact same APIs, and so on,
to build each VM instance and the applications programs that run within,
as well as for the control domain, e.g. for network and storage
configuration and assignment (though of course VM instantiation and
control over some hardware resources still requires a unique API for
each type of virtualization, i.e. Xen, VirtualBox, VMWare, etc.).
Xen further offers to eliminate complexity under the hood for the upper
layers too, e.g. with para-virtual device drivers for guest systems that
then use simple protocols to communicate with the hypervisor and dom0.
So I would say a good hypervisor offers a clear structure that controls
and eliminates complexity and also "reduces attack surface" (to use that
favourite phrase of security researchers).
> If you run multiple instances of the same OS version in Xen/VMware,
> that is a pretty inefficient way to partition your application domains.
Yes, this is true, but there is some debate. E.g. on ARM with Linux
using KVM and Xen vs. Docker the conclusion of one study was:
"a slightly better performance for containers in CPU bound workloads
and request/response networking; conversely, thanks to their caching
mechanisms, hypervisors perform better in most disk I/O operations
and TCP streaming benchmark."
https://doi.org/10.1109/AIEEE.2015.7367280
(and what always dominates performance? I/O dominates!)
(Other studies I've scanned suggest there is even less performance
difference than most people seem to assume must be there.)
I still think the security and complexity issues with containers, are a
very much bigger concern than the pure efficiency losses of running full
VMs. When it's all hidden behind a single command ("docker pull nginx")
then it's too easy to ignore the problems and so that's what people do
-- they take the easy street.
Never the less even VMs are not as secure and simple as bare hardware.
Those who have a good handle on their application software and customer
requirements, and who are able to keep things more uniform, are able to
bundle similar services together on the same bare metal all with one OS
kernel and one shared application configuration, with no loss of
efficiency and with no added complexity.
The really interesting things w.r.t. performance and efficiency with
shared hardware and without total loss of isolation (i.e. allowing
multi-tenant applications) happen when you start to look at application
specific unikernels in combination with a hypervisor. This of course is
a little bit like vaporware still, even for Xen on NetBSD, though
Rumprun[1] shows great promise.
[1] https://github.com/rumpkernel/rumprun
Personally I think some combination of virtualization, unikernels,
capabilities, and old-made-new again single-level-store techniques are
the brightest future for shared computing, i.e. multi-tenant clouds.
(BTW, from what I understand, third-hand so it may or may not be true,
both Google and Amazon are actually running each "container" inside a
full VM just to ensure better isolation and security. There's even
open-source software originally from Intel that does this. Now where
did the efficiency go?)
(Also, if you really want to see some crazy way to do things, have a
look for "hypervisor-based containers" -- i.e. "let's keep all the crazy
extra APIs and kernel complexity and also run it together with the
application code all in a unikernel VM environment!")
> Also forget about chroot, it is not an enterprise solution.
Well, I guess that depends on what requirements one has.
If by "enterprise" one means: "it has a clicky-GUI driving console",
well, no, of course not.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpvF_vCwj_cN.pgp
Description: OpenPGP Digital Signature
Re: NetBSD Jails
At Sun, 17 May 2020 21:52:58 +0100, Sad Clouds wrote: Subject: Re: NetBSD Jails > > On Sun, 17 May 2020 14:07:21 -0500 > Ted Spradley wrote: > > > How well will all this modern container and virtualization stuff work > > on the older platforms that only have megabytes of memory, not > > gigabytes? > > Quite well, since containers are very lightweight. It's not the > container technology that sucks beyond belief, but the bloated user > applications that need gigabytes of memory to run basic tasks. Since Ted asked about older platforms where memory is an issue, well the kind of "container" technology in Linux certainly isn't "lightweight" -- there's a ton of extra code in the kernel to make it work. (CGroups) I _think_ FreeBSD Jails are less heavy-weight but it does get its tendrils deep into quite a wide variety of kernel subsystems. Both incur a huge amount of code complexity to the kernel. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpwoyQv47Yzf.pgp Description: OpenPGP Digital Signature
Re: NetBSD Jails
At Sun, 17 May 2020 11:11:22 +0200, Niels Dettenbach wrote: Subject: Re: NetBSD Jails > > Am 17.05.2020 um 06:01 schrieb Greg A. Woods : > > > > I know some people do allow human users to login to FreeBSD "jails", but > > I really have to wonder why. I think if you want to give human users > > the idea that they have their own machine then you really do need to > > give them a whole VM (at least with Unix/POSIX systems -- modernized > > multics-like systems might be a better way). > > if you really wonder, take a look at i.e. FreeNAS as other projects > which uses BSD jails as containers for virtual multi host environments > (i.e. mailservers, LAMP stuff, Database servers, Samba stuff and > proprietary / binary software etc) which all have their own IPs as > root as user contexts in fs as userspace and security isolation > (system as net / firewalling etc) is a major reason. This is one of > the most used scenarios today. Indeed, as I say, I know people do this and I've seen lots of it. I have friends and colleagues who have tried to tell me how and why. I've gone to talks at BSDCan about the how's and why's and I've chatted to people in the halls after about these talks. But what I've been trying to express in my questions on this thread is that I still don't understand the deep reasons why this is seen as a _necessary_ approach. Many folks are doing it because others do it. Well, all I can say to that is have fun on your bandwagon, and don't let me stop you! Some think there are some security benefits. I continue to see security issues which are a direct result of more complex code, more complex configurations, and more complex management overhead. Chroot isn't 100% secure, especially not for processes with superuser privileges, and jails are no better. I.e. I think chroot environments are good, in so far as they go, but I'm less trusting of even FreeBSD jails because of the added complexity, both under the hood, and in configuration and management. Other competing technologies on other platforms such as those on Solaris and Linux are even more complex and convoluted. In the end there is inherently less security with any and all forms of virtualization and/or sharing of resources. If absolute security is your requirement then you really need separate hardware for each circle of trust (especially as we've seen with the issues coming from the very fundamentals of modern CPU internals). Some think there are performance benefits. I do see there are performance tradeoffs, but if chroot is enough then why add even the additional layers of code needed for FreeBSD jails? If you actually really need a fully isolated and completely full featured environment where you can run complex applications in "reasonably secure" sandbox style isolation then why not choose the best possible hardware you can afford that supports a full virtual machine environment such as Xen, or nvmm/bhyve with qemu or virtualbox, etc.? (e.g. I bought a used Dell server for about $500 and I can run Xen with many domUs on it very efficiently) -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgp7hfhm4r2Ut.pgp Description: OpenPGP Digital Signature
Re: NetBSD Jails
tralize authentication services and logging. Tools > like Ansible come into play to ensure VM's don't drift away from a > common specification. Different OS versions cause random > incompatibilites - once you're past a ten or twenty VM's, it's > very unlikely you can upgrade all of them at once. I would argue full VMs are actually easier to manage, especially when they all share the same installed software and only have minor configuration differences (and all those config files also live adjacent to each other on a big file server). I only ever upgrade one root filesystem. I only ever install packages once, in one place. All like-minded VMs share these things -- one copy, read-only. I only ever run one program that edits all the similar configurations at once (with specific details taken into account automatically by a data driven config tool for each instance of course). > Anyway, none of computing on this scale is simple. It should be. It can be. It probably is the only sane/safe/efficient way. Complexity breeds disaster. > The complexity will > exist somewhere, it's a question of where. I don't agree -- I think it's just a matter of seeing requirements in a different light so that the complexity can be cut away for good. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgphw9hh8Mvjd.pgp Description: OpenPGP Digital Signature
Re: NetBSD Jails
At Fri, 15 May 2020 20:18:28 -0400, "Aaron B." wrote: Subject: Re: NetBSD Jails > > - Processes can "see" each other; I have to be careful not to reuse > UID numbers. For example: if I build a chroot with an instance of nginx > that runs as UID 2505, and then deploy multiple copies of that chroot, > all of them can call kill(2) on a process in a different chroot. Perhaps all that's required is a tool which extracts the minimum required entries from the real /etc/master.passwd for each chroot? (and some way to maintain chroot copies?) (Another way would be a new service behind nsdispatch(3) which would provide access through the chroot, e.g. via a socket, to the shared /etc/master.passwd, though that would assume all chrooted programs use only the "standard" interfaces supported by nsdispatch(3).) > - All chroots share the same network stack. If I tell nginx to bind to > '0.0.0.0' or '::', the first instance will startup fine, the others > will fail with "address already in use." Well if you're chrooting multiple instances of the same service, isn't it obvious that each has to listen on one and only one specific address? If I understand correctly one could also route a subnet via a bridge interface to each chrooted service. Maybe a chrooted process should also be prevented from listening to a wildcard address? I've heard FreeBSD folks go on for days about how FreeBSD's "jails" make network management simpler, but I still don't have any real understanding of exactly what this means. The only thing that seems to be of interest is in allowing a chrooted "root" user(*) (e.g. to allow someone inside the jail to muck around with interface addresses, routes, etc.), but I would suggest that allowing root inside a chroot is a very very bad idea no matter what "jails" features you might think will protect you (i.e. I would never trust a chrooted root user, "jails" or no jails). (*) "chrooted root user" -- i.e. the "root" user in the jail can only do things (as superuser ID#0) to those resources, e.g. interfaces, routes, etc. that are delegated to the jail. > The wiki's projects list has a > clean solution to this particular point, which may or may not be within > scope of jails: > > https://wiki.netbsd.org/projects/project/virtual_network_stacks/ Virtual network stacks seem to be a rather complex solution looking for a problem -- i.e. in actual fact more of a problem looking for trouble. > - Some way to set per-chroot resource limits would be helpful. I can > manipulate ulimits, but that is basically driving screws with a hammer. > It's simply the wrong tool. Well here's where /etc/login.conf solves most problems for normal chroot environments, since only ordinary users should be running inside the chroot. (Or it could, if there were resource controls related to networking. :-)) For anything beyond that I'm pretty certain that a full virtual machine is the real solution. Personally I think full VMs are the K.I.S.S. of death for "containers" (especially once you see the nightmare of complexity that underlies them in the kernel, i.e. CGroups). I would much rather have the clearly structured overhead of the VM instead of the hidden overhead and excessive complexity of "containers", or even just some of the bits of them like virtual network stacks. All this said though I would note that perhaps re-engineering the whole network stack in the netgraph way (perhaps directly using netgraph[1]), provides some form of "virtualization" for network things in a clean and structured way. [1] https://en.wikipedia.org/wiki/Netgraph One thing I would like to have for making VMs easier to manage though is a filesystem that can be shared read-only as a block device -- i.e. through xbdback(4) to xbd(4), or the equivalent for other kinds of VMs. (I say this because I abhor the use of NFS to share to VM domUs.) -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgppMReEo1poV.pgp Description: OpenPGP Digital Signature
Re: NetBSD Jails
At Fri, 15 May 2020 16:56:04 -0400, "Aaron B." wrote:
Subject: Re: NetBSD Jails
>
> > Can't wait to have jails on NetBSD.
>
> I have also wanted this feature for a long time. Currently I manage a
> lot of applications in running in self contained chroot'ed
> environments, less for security and more for organization. I can
> tarball them up and move to a different server with ease; or completely
> reinstall a new major version of NetBSD and run the same chroots without
> modification.
>
> Jails would turn my chroots into true containers.
I'm curious about what this means to you -- what do you need/want in
addition to the chroot environments you now have?
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpS1Y_UNTDYZ.pgp
Description: OpenPGP Digital Signature
Re: pkg and downloaded files
At Fri, 15 May 2020 01:09:31 +0200, Riccardo Mottola
wrote:
Subject: Re: pkg and downloaded files
>
>
> Actually the mapping is done on the server side, so on the client I
> see the file as owned "by root" but the server considers it
> "nobody"... if I had seen the mapping with "ls" I would have noticed
> immediately.
Wow, that's an incredibly useful insight!
I wish I would have thought of it. It seems so obvious in hindsight now!
Now I wonder just how difficult it might be to implement, perhaps as
an(other) option.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpOdU6w9rvfr.pgp
Description: OpenPGP Digital Signature
Re: NetBSD install experiences
At Wed, 13 May 2020 07:03:41 +0200, Martin Husemann wrote:
Subject: Re: NetBSD install experiences
>
> That means in a very minimal setup where you boot from an UEFI install
> image on USB and you have a target SATA disk that already has some GPT
> partitions you would get devices like:
>
> sd0 the USB stick you booted from
> wd0 the SATA disk
> dk0 some GPT partition on wd0
> dk1 another GPT partition on wd0
> dk2 the EFI boot partition on the USB boot medium
> dk3 the boot/root partition on the USB boot medium (not offered
> by the installer)
>
> in mostly random order.
>[[]]
> I'm thinking about making that list of potential target devices more like
> a tree so the selection becomes more obvious.
Ah! Yes, a tree-like view of the devices would be immensely useful!
(especially since by that point getting a look at the dmesg output again
is, well, painful)
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpWWygdbHDbg.pgp
Description: OpenPGP Digital Signature
netbsd (i386 9.0) vs. USB Keyboard/Video/Mouse switches (and X11)
So as mentioned elsewhere in response to some weirdness with 32-bit i386
builds not working as well as 64-bit amd64 builds, I installed the stock
9.0 i386 release on my old Dell PE2650 server.
I use a serial console on this machine, but the VGA and USB are
connected to a USB Keyboard/Video/Mouse switch that I use for firmware
access (not quite all the firmware works with serial console, especially
not on these old machines, and I don't know how to type function keys
via the serial console either, something that's necessary to drive some
firmware UIs).
Before the KVM, and back in the 5.x days, I had X running on this
machine, so for a whim I thought I'd enable X11 with "xdm" via sysinst
to see if it would still work, and if so, how well it would look.
It sort of worked after an initial manual kill of the first X server
process (at first there was just a black screen, and the running X
server process seemed to be spinning). The second one didn't display
correctly on the monitor I'm using though -- it was "scrolled"
horizontally. Asking the monitor to self-adjust seemed to fix it,
though pressing keys also caused weird video artifacts and more
scrolling. I'll attach the X log below -- the only thing that looks odd
to me in it is the manufacturing year "2010" -- that must be for the
monitor as the machine itself was considered quite old even in 2010.
However things got worse when I cycled the KVM to another server and
back. Suddenly it seemed as if a key was held down and repeating in the
login window. Pressing other keys changed which key was repeating, but
didn't stop the repeating. I didn't seem to be albe to switch away to a
different (plain) console screen either. Occasionally after cycling the
KVM to another system and back the X server starts spinning and eating
CPU again, requiring another manual kill.
The following kernel messages appear when cycling the KVM and look
"normal"/OK (to me):
[ 96163.108666] wsmouse0: detached
[ 96163.108666] ums0: detached
[ 96163.108666] uhidev0: detached
[ 96163.108666] uhidev0: at uhub0 port 1 (addr 2) disconnected
[ 96163.208734] wskbd0: disconnecting from wsdisplay0
[ 96163.208734] wskbd0: detached
[ 96163.208734] ukbd0: detached
[ 96163.208734] uhidev1: detached
[ 96163.208734] uhidev1: at uhub0 port 2 (addr 3) disconnected
[ 96170.263343] uhidev0 at uhub0 port 1 configuration 1 interface 0
[ 96170.263343] uhidev0: ELAN Microelectronics (0x4f3) USB+PS/2 Optical Mouse
(0x230), rev 1.10/24.58, addr 2, iclass 3/1
[ 96170.273352] ums0 at uhidev0: 3 buttons and Z dir
[ 96170.273352] wsmouse0 at ums0 mux 0
[ 96171.824385] uhidev1 at uhub0 port 2 configuration 1 interface 0
[ 96171.824385] uhidev1: CHICONY (0x4b3) USB NetVista Full Width Keyboard
(0x3025), rev 1.10/1.02, addr 3, iclass 3/1
[ 96171.824385] ukbd0 at uhidev1
[ 96172.234655] wskbd0 at ukbd0 mux 1
[ 96172.234655] wskbd0: connecting to wsdisplay0
The KVM itself doesn't seem to show up as as USB device, just the
keyboard and mouse attached:
# usbdevs -vd
Controller /dev/usb0:
addr 1: full speed, self powered, config 1, OHCI root hub(0x),
NetBSD(0x), rev 1.00(0x0100)
uhub0
port 1 addr 2: low speed, power 100 mA, config 1, USB+PS/2 Optical
Mouse(0x0230), ELAN Microelectronics(0x04f3), rev 24.58(0x2458)
uhidev0
port 2 addr 3: low speed, power 100 mA, config 1, USB NetVista Full Width
Keyboard(0x3025), CHICONY(0x04b3), rev 1.02(0x0102)
uhidev1
port 3 powered
port 4 powered
So, does anyone have any advice as to how to better use a USB KVM with
NetBSD, or indeed if it's expected to work at all?
Getting X stable with the built-in video card and the monitor I have
doesn't seem worthwhile until the keyboard issue is fixed, but just for
fun, as promised, the xdm, Xserver, and dmesg logs follow.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
-- one instance from /var/log/xdm --
xdm info (pid 476): Starting X server on :0
X.Org X Server 1.20.5
X Protocol Version 11, Revision 0
Build Operating System: NetBSD/i386 9.0 - The NetBSD Foundation, Inc.
Current Operating System: NetBSD once.local 9.0 NetBSD 9.0 (GENERIC) #0: Fri
Feb 14 00:06:28 UTC 2020
mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/i386/compile/GENERIC i386
Build Date: 03 March 2019 07:11:23AM
Current version of pixman: 0.38.4
Before reporting problems, check http://wiki.x.org
to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Tue Apr 28 14:27:46 2020
failed to open /dev/ttyE0: Device not configured
(==) Using default built-in configuration (39 lines)
The XKEYBOARD keymap compiler (xkbcomp) r
Re: Linux compat and swap
At Fri, 24 Apr 2020 10:18:10 +0100, Sad Clouds
wrote:
Subject: Re: Linux compat and swap
>
> But what is the value of vm.filemax? It tells the system what
> percentage of memory to steal from other uses, but this is counter
> productive, since by stealing from vm.anon and vm.exec it can result in
> swapping and extra disk I/O, which is what vm.filemax is trying to
> avoid in the first place, that is unnecessary disk I/O.
>
> I think the behaviour needs fixing, i.e. if vm.filemax is about to push
> pages to swap, then it should stop.
Well, you (i.e. the system administrator) are in charge of setting the
balance.
If your systems are experiencing paging because they are both bulging
with constantly scanned anon pages and trolling through the filesystem
like crazy then you can lower vm.filemax.
On the other hand if you've got filemax too low, and the anon pages are
actually sitting idle most of the time, then lowering anonmin and
raising filemax might improve filesystem performance dramatically.
On my edit/build/test server I have a couple of GB out on swap because
it's basically all idle memory -- there are very few page faults to read
from swap during even "-j 12" builds. After 132 days uptime, with
numerous system and pkgsrc builds, all while running some rather bloated
long-running emacs processes, these are the paging stats (from vmstat -s)
480372 pagein requests
180517 pageout requests
0 pages swapped in
1442574 pages swapped out
and just:
15 faults with no memory
28 faults had to wait on pages
all a tiny fraction of:
1271146431 total faults taken
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpdAkaC0B8Hd.pgp
Description: OpenPGP Digital Signature
Re: Linux compat and swap
At Fri, 24 Apr 2020 14:56:37 +0200, tlaro...@polynum.com wrote:
Subject: Re: Linux compat and swap
>
> Does somebody know what are the main source files implementing it so
> that if no in depth documentation is available, the C files would give
> the picture?
again, from my sysctl.conf: :-)
# See also:
#
#
http://web.archive.org/web/20181008110324/http://www.selonen.org/arto/netbsd/vm_tune.html
# http://chuck.cranor.org/p/diss.pdf
#
http://usenix.org/publications/library/proceedings/usenix99/full_papers/cranor/cranor.pdf
BTW, I've used the settings I posted basically since UVM was integrated
on all my servers, and even on some small machines like a little Soekris
board with 512MB of RAM.
In hindsight I think the vm.exec* values are a bit too high for most
uses, but at the same time they don't seem to hurt for the kinds of uses
I put my machines to. I admit though I haven't extremely stressed any
smaller machines (virtual or otherwise) to see how they do. The domU I
edit/email/build on has 8GB assigned, and though it runs with a fair
number of pages out on swap, it never thrashes even with "-j 12" builds.
(and that's with a fully static-linked userland too)
I would say generally speaking that UVM does better than any other VM
management system I've used, including even macOS with its compressed
memory feature; though at some level macOS is hard to judge because of
its massive overreliance on dynamic linking and zillions of frameworks.
macOS only really starts to shine on 2+ CPU machines with SSD system
storage.
Indeed I suspect adding compressed memory support to UVM, i.e. as an
intermediate step before paging to disk, could make it the very best of
the best (for multi-core SMP systems, at least, though even a single CPU
can probably swap compressed pages faster than it can write and read
them from spinning rust).
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpDlyF0KHhnc.pgp
Description: OpenPGP Digital Signature
Re: Linux compat and swap
At Fri, 24 Apr 2020 12:23:14 +1000, Paul Ripke wrote:
Subject: Re: Linux compat and swap
>
> To the best of my knowledge, vm.execmax just defines the maximum amount
> of memory for mapped executable pages, and won't cause processes to be
> killed. Since mapped executable pages are generally read-only, they can
> just be dropped from RAM and paged in again from the executables as
> needed.
Well I think more accurately the vm.*max values are better described as
the maximum percentage of pages that the VM system will attempt to
reclaim from other uses for the named use. I.e. if you want to start a
fairly big and long-running program (i.e. one that has a lot of text
pages in terms of percent of the whole system's capacity, and which uses
many of its text pages frequently) then you want the system to be able
to push out other pages (e.g. especially file cache pages) and keep them
for the busy executable text pages while the program runs, so you want
to set vm.execmax large enough to accommodate the full compliment of
necessary text pages for that big program (or a set of simultaneously
running but different programs).
If executable pages are pushed out (i.e. down to vm.execmin) due to
other memory pressure (e.g. your running programs start to do a lot of
filesystem I/O or they allocate a lot of heap and start using it while
at the same time allowing the majority of their active text pages to go
idle), then the idle text pages are indeed simply dropped (down to
vm.execmin) to make room for the other uses; then, as you say, they will
be simply paged in from the program files on disk as needed again.
However page faults on exec pages are is still very expensive for a
running program, especially if it is CPU bound.
Depending on where your real-world workload's memory pressure is, it may
make sense to have vm.execmin relatively high to accommodate really
large programs or very large numbers of _different_ programs
simultaneously (many instances of the same program all share the same
text pages), but for many use cases, especially on larger-memory
systems, the number of average executable pages in use will be quite a
small percentage of total pages. I rarely see my systems go above 1-2%
exec pages (even on my systems where every program is static-linked),
but I still keep vm.execmax fairly high as these are, along with
anonymous pages, far more important than file cache pages for many kinds
of running programs to actually be able to make progress without hitting
page faults during pure computation.
On the other hand the vm.*max percentages are just limits to how many
pages will be reclaimed from other uses when a given category faces
pressure from extensive and immediate use -- they do not set the maximum
use for a given category. On my rsync backup host I normally see 85% of
pages allocated to file cache even though vm.filemax is just 50.
One should keep an eye on the percentages shown by a "systat vm" running
in another window during normal workloads to see how memory is being
used.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgp0fCCIE_PsZ.pgp
Description: OpenPGP Digital Signature
Re: Linux compat and swap
At Thu, 23 Apr 2020 11:56:08 +0100, Sad Clouds wrote: Subject: Re: Linux compat and swap > > On Thu, 23 Apr 2020 10:56:15 +0100 > Mike Pumford wrote: > > > If you have both memory intensive and filesystem intensive processes > > running on a NetBSD system the kernel filesystem cache can end up > > evicting programs running in the background that are inactive which > > then take a LONG time to recover. For a system with a reasonable > > amount of memory I found the vm.filemin and vm.filemax needed to be > > tweaked so that filesystem cache was more likely to be tweaked than > > program code. > > Is this correct? I always thought that file cache was opportunistic, > i.e. it will use all free memory, but under no circumstances it should > evict any pages of running programs. The opposite should happen, i.e. > any program using memory should be allowed to steal it from file cache > at any time. > Mike's description is correct. Processes can only take over memory from the file cache if the value for vm.filemin is small enough. That is, after all, what vm.filemin means. Here is the VM section of my sysctl.conf, with notes that help explain some of my rationale: # the rest are VM pager tuning parameters # # Questions: # # Does the pager eventually always free inactive pages # above the vm.*max limits? # # It will depend upon the order of pages on the list(?). It will # free pages until there are enough(?) free pages, in the order they # appear on the list. # # Is this worse for exec and file pages than it is for anon pages? # # # Does it even matter if inactive anon pages are preserved for very # long? # # See also: # # http://web.archive.org/web/20181008110324/http://www.selonen.org/arto/netbsd/vm_tune.html # http://chuck.cranor.org/p/diss.pdf # http://usenix.org/publications/library/proceedings/usenix99/full_papers/cranor/cranor.pdf # N.B.: On a live system make sure to order changes to these values so # that you always lower any values from their default first, and then # raise any that are to be raised above their defaults. This way, the # sum of the minimums will stay within the 95% limit. # the minimum percentage of memory always (made) available for the # file data cache # # The default is 10, which is much too high, even for a large-memory # system... # vm.filemin=5 # the maximum percentage of memory that will be reclaimed from others # for file data cache # # The default is 50, which may be too high for small-memory systems # but may be about right for large-memory systems... # #vm.filemax=25 # the minimum percentage of memory always (made) available for # anonymous pages # # The default is 10, which is way too low... # vm.anonmin=40 # the maximum percentage of memory that will be reclaimed from others # for anonymous pages # # The default is 80, which seems just about right, but then again it's # unlikely that the majority of inactive anonymous pages will ever be # reactivated so maybe this should be lowered? # vm.anonmax=80 # the minimum percentage of memory always (made) available for text pages # # The default is 5, which may be far too low on small-RAM systems... # vm.execmin=20 # the maximum percentage of memory that will be reclaimed from others # for text pages # # The default is 30, which seems way too low, esp. for big programs... # vm.execmax=40 # It may also be useful to set the bufmem high-water limit to a number # which may actually be less than 5% (vm.bufcache / options BUFCACHE) # on large-memory systems (as BUFCACHE cannot be set below 5%). # # note this value is given in bytes. # #vm.bufmem_hiwater= -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpgmDnDncIRl.pgp Description: OpenPGP Digital Signature
Re: SMTP servers receiving from gmail
At Thu, 16 Apr 2020 09:14:30 +0200, ignat...@cs.uni-bonn.de wrote:
Subject: Re: SMTP servers receiving from gmail
>
> On Wed, Apr 15, 2020 at 10:55:25PM +0200, Rhialto wrote:
> [Google "Mail"]
>
> > They demand DKIM or similar configurations, which I refuse to use,
> > because its a lot of work to configure, and it often doesn't even work
> > with mail forwarding.
>
> It might help to have a PTR record for the smtp clients' outgoing address;
I doubt they demand either.
I can send from my server to gmail without problem or delay, and I
currently have neither a PTR, nor an SPF record (which are stupid), nor
does my mailer do anything special to create DKIM headers, etc.
Vice versa I can also send from my gmail account to my server A-OK.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgp0Mf0jKIvK9.pgp
Description: OpenPGP Digital Signature
Re: default search path for shared libraries
At Mon, 23 Mar 2020 10:41:14 +0100, Martin Husemann wrote: Subject: Re: default search path for shared libraries > > On Mon, Mar 23, 2020 at 01:41:52AM -0400, Jeffrey Walton wrote: > > > > Under LD_LIBRARY_PATH, the man page says: > > > > A colon separated list of directories, overriding the > > default search path for shared libraries. > > > > What is the default search path? > > That is RPATH from the binary (or RUNPATH, they are the same on newer NetBSD, > older NetBSD does not support RUNPATH at all). In other words, what's shown by "readelf -d /bin/path | fgrep PATH", if anything, else the default path of "/usr/lib" (assuming LD_LIBRARY_PATH is not set in the environment and /etc/ld.so.conf is empty or missing). See ld.elf_so(1). (which isn't perfectly clear on search order, unlike the latest FreeBSD ld.so(1)) -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpxtbGvoAX7u.pgp Description: OpenPGP Digital Signature
Re: Shared object "libintl.so.9" not found (but its there)
At Wed, 11 Mar 2020 20:34:11 -0400, Jeffrey Walton wrote: Subject: Re: Shared object "libintl.so.9" not found (but its there) > > I don't follow. Please forgive my ignorance. > > I'm building using my build scripts from > https://github.com/noloader/Build-Scripts . They feature test, set > package options, configure, make and make install. Everything someone > would do manually. Nothing fancy. > > This does not make sense, either: > > $ ldd /usr/pkg/bin/git > /usr/pkg/bin/git: > -lpcre2-8.0 => /usr/pkg/lib/libpcre2-8.so.0 > -lpthread.1 => /usr/lib/libpthread.so.1 > -lc.12 => /usr/lib/libc.so.12 > -lz.1 => /usr/lib/libz.so.1 > -lintl.1 => /usr/lib/libintl.so.1 "git" is not "git-submodule". They are likely separate binaries, where "git submodule" invokes "git-submodule". > I can't seem to get more information though: > > $ LD_DEBUG=files git submodule --init > Shared object "libintl.so.9" not found > Shared object "libintl.so.9" not found > Shared object "libintl.so.9" not found > Shared object "libintl.so.9" not found I'm going to guess that "git-submodule" was compiled to use the "libintl.so.9" version that you have/had installed in /usr/local/lib, but the build didn't bake in the runtime path for that library, so the runtime linker can't find it (it doesn't look in /usr/local/lib by default). You could probably work around this by configuring /etc/ld.so.conf, or even by setting LD_RUN_PATH in the environment, but the best fix is to make sure the linker gets the required "-rpath=" options when building the "git-submodule" binary. Or just static link everything. That's what I do. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpjPZBX8DOs6.pgp Description: OpenPGP Digital Signature
Re: tcsh and accented characters (strange problem)
At Mon, 9 Mar 2020 15:52:06 +0100, tlaro...@polynum.com wrote:
Subject: Re: tcsh and accented characters (strange problem)
>
> If I'm not mistaken, the internationalization code (citrus) is only
> available via dynamic shared objects loading. If you compile all
> statically, you don't have the internationalization (iconv and the
> rest).
Note the patches in PR#18151 by Anders Hjalmarsson solved this problem
for static-linked binaries way back in 2002 (by allowing a set number of
locales to be supported at compile time).
I've been keeping those patches up-to-date in my own tree since about
2008 (though I'm still back on ~8.99) and they work well!
Note this task is also mentioned in:
https://wiki.netbsd.org/projects/project/static-locale/
But unfortunately there's no mention in that project item about PR#18151
and it's still listed as a 1-month task -- it's actually now only a few
hours work.
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpRkRsdR5Qzn.pgp
Description: OpenPGP Digital Signature
Re: green lines hell
At Wed, 26 Feb 2020 16:43:04 - (UTC), mlel...@serpens.de (Michael van Elst)
wrote:
Subject: Re: green lines hell
>
> /dev/console can be redirected with the TIOCCONS ioctl. That's how
> xconsole works, it opens a pty, redirects console output to it
> and then logs things written to the pty to an X window.
>
> sysinst could do the same and then e.g. present console output in a
> curses window.
That would seem to be the best overall solution...
But, it does seem like a lot of extra complication too.
Since sysinst is a full-screen program, redrawing the screen should work
for any kind of "console" connection.
Doesn't pressing already work to manually trigger a screen
redraw in many instances, or am I mis-remembering?
Maybe just a wee hint about using ^L in one of the earlier messages
would suffice?
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpanbMqI5D_Q.pgp
Description: OpenPGP Digital Signature
Re: NetBSD and User Private Groups (Unique Groups)
At Wed, 29 Jan 2020 09:36:02 +, Ottavio Caruso
wrote:
Subject: NetBSD and User Private Groups (Unique Groups)
>
> [1]
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/deployment_guide/s1-users-groups-private-groups
> [2] https://www.freebsd.org/cgi/man.cgi?adduser(8)
The way those documents are worded seem to me as if they were written by
people who did not understand the use of Unix file permissions and
ownership very well.
As others have said, the user's default "umask" is the correct solution
to the problem of having un-related users being able, by default, to
have read (and search for directories) access to each other's files.
Note that a user's default umask can be set in their shell's startup
script, as well as in /etc/login.conf (which adds yet another way of
"grouping" users).
As for the policy set in the default /etc/usermgmt.conf for "useradd",
well, it's definitely a policy issue and not a technical issue.
Personally I would say it would be rather obnoxious to change it now
after twenty years, at least without a far better argument. That said,
I can't remember ever having used "useradd" except to test it. :-)
--
Greg A. Woods
Kelowna, BC +1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgptpALXByYyc.pgp
Description: OpenPGP Digital Signature
Re: "route_enqueue: queue full, dropped message" blast from a 8.99.32 amd64 domU
At Fri, 10 May 2019 13:02:05 -0700, "Greg A. Woods" wrote: Subject: Re: "route_enqueue: queue full, dropped message" blast from a 8.99.32 amd64 domU > > During normal operation I see just over 8000/s (and with the console > spewing there are only about 200/s more), and it all seems to add up > since this is an 8-vcpu domU, and I have HZ=1000. Before I rebooted I > was seeing numbers ranging up to 25000/s. So this happened again today. However I didn't manage to run "systat vm" and see if there was an interrupt storm again, and if so what device it was from. But neither did I have to reboot. It lasted long enough and strong enough to kill all my X11 connections, though by the time emacs was dumping core (as it always does when it loses touch with the X11 server), NFS was running well enough to store the core files. I was in the midst of trying to login on the console to investigate further when things seemed to have returned to normal. The console isn't fully usable though -- I see output sent to it, but I can't seem to send anything to the getty running on it. I'm not yet sure if this is a conserver, xen, or NetBSD problem. This time there were occasional "xennet0: xennet_watchdog" messages too. There's one other possibly related symptom -- sometimes X11 clients running on this host seem to have trouble painting characters, particularly "links" (but it has no trouble displaying images or scrolling images). Unfortunately I have not thought to check for a slightly less voracious interrupt storm while this symptom is present, but hopefully I will remember to do so next time it appears. Until now I've thought this might be some kind of problem with volumes of network traffic interfering with the font server, though I've not observed any truly excessive network traffic. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgp1bux9EI1Me.pgp Description: OpenPGP Digital Signature
Re: csapp, really good?
At Wed, 19 Jun 2019 12:39:50 -0700, "Greg A. Woods" wrote: Subject: Re: csapp, really good? > > If you want to learn to program for any Unix-like system the very best > book would probably be Stevens' "Advanced Programming in the UNIX > Environment", possibly along with Steven's 2-volume "UNIX Network > Programming" set. I have recently acquired a new (to me) book that may also be interesting for anyone wanting to learn "systems programming" in any Unix-like environment. While it's not really all that new (copyright 2003), it is none the less somewhat newer than Stevens' APUE; has many still-relevant examples and exercises; and is perhaps more oriented as a learning textbook: UNIX SYSTEMS Programming (Communication, Concurrency, and Threads) Kay A. Robbins, Steven Robbins Second Edition Prentice Hall PTR ISBN 0-13-042411-0 I got my copy via alibris.com where more hardcover copies can be found for as little as $5 or less. I did find a full PDF online too without too much google fu. I still very much like all the late W. Richard Stevens' books and admire them for their completeness and accuracy and attention to detail. However they are now somewhat dated. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgp_ITpCnoo_3.pgp Description: OpenPGP Digital Signature
Re: csapp, really good?
At Sat, 8 Jun 2019 03:57:02 GMT, Mayuresh Kathe wrote: Subject: csapp, really good? > > i stumbled upon "computer systems: a programmer's perspective" > (url: csapp.cs.cmu.edu) and it looks like a really interesting > book for a newbie to systems programming under unix. > is it really good to warrant a purchase (expensive), or would > the book by "maurice bach" be considered good enough, though > the style of "c" used in the book seems a bit strange. > thanks. > From what I see from the preface of "Computer Systems: A Programmer's Perspective" it seems to be an entirely different kind of introduction to systems programming than a book like Bach's, and possibly much more general, at least with respect to reasonably modern kinds of hardware. It does indeed look rather interesting. Bach's book is more specifically about the various common aspects of the internals of Unix itself and alone. Bach's pseudo-code for presenting algorithms is reasonably readable and for that book's main purpose they are the main content; however examples of user-land programs presented in actual C are all toy programs to the extreme. If you want to learn to program for any Unix-like system the very best book would probably be Stevens' "Advanced Programming in the UNIX Environment", possibly along with Steven's 2-volume "UNIX Network Programming" set. -- Greg A. Woods +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpk66hmN_ViN.pgp Description: OpenPGP Digital Signature
Re: BSDCan meets!
At Tue, 14 May 2019 17:56:11 +, co...@sdf.org wrote:
Subject: BSDCan meets!
>
> Hi netbsd people,
>
> I'm going to be at Ottawa for BSDCan on 15-20 May (that's tomorrow).
> If you are coming and want to join, please reply.
> (Several other NetBSD'ers are coming, too)
I'll be arriving Thursday evening, see you there!
--
Greg A. Woods
+1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgp9omHZ399t9.pgp
Description: OpenPGP Digital Signature
Re: "route_enqueue: queue full, dropped message" blast from a 8.99.32 amd64 domU
At Fri, 10 May 2019 13:05:04 +0100, Roy Marples wrote:
Subject: Re: "route_enqueue: queue full, dropped message" blast from a 8.99.32
amd64 domU
>
> I would imagine that if an interface is interupting that much then
> it's constantly sending messages to route(4) to say that it's up/down
> and addresses are detached/tentative in a tight loop. The queueing
> mechanism has a fixed length and while we go out of our way to notify
> userland if there's an error sending these messages, we can't send
> this one at all so we just log it.
>
> So it's an artifact of your interupt storm, but not the cause.
Thanks Roy!
I'm afraid I was too distracted with other things at the time to look to
see if I could identify which device might have accounted for the extra
interrupts.
I kind of assumed they were all from the console driver, though I just
did a test and now realize that even at full speed it can't account for
nearly the magnitude of interrupts per second that I was seeing.
During normal operation I see just over 8000/s (and with the console
spewing there are only about 200/s more), and it all seems to add up
since this is an 8-vcpu domU, and I have HZ=1000. Before I rebooted I
was seeing numbers ranging up to 25000/s.
So, what in the world could make either the single xennet0 interface, or
the lo0 interface for that matter, bounce in such a fashion I wonder.
As far as I can tell from old console log files this did not happen in
the first few years of life of this system (from about 2015) when it
(and its dom0) was running 7.99.various.
--
Greg A. Woods
+1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpWS5wsy4cXw.pgp
Description: OpenPGP Digital Signature
"route_enqueue: queue full, dropped message" blast from a 8.99.32 amd64 domU
So, I had a weird thing happen on one of my regularly used Xen-hosted
virtual servers this morning...
The host is a Dell PE2950, running Xen-4.5 with 8.99.32 amd64.
The domU is also 8.99.32 amd64.
(I'm in the slow process of upgrading packages so I can upgrade Xen, but
that's not been completed quite yet.)
From all observations on shell and xterm sessions to the domU just
seemed very sluggish and sometimes non-responsive. "systat vm" reported
a very high "interrupts" count, and a rather high "sys" CPU use.
The console seemed completely dead, but had reported a stream of
messages like:
[Thu May 9 09:24:08 2019][ 6442662.0806318] route_enqueue: queue full, dropped
message
There were thousands of identical lines, all separated by a few
microseconds. No doubt this spew was the real cause of the apparent
interrupt storm and the resulting sluggishness.
The other domUs and the dom0 seemed A-OK.
So I decided to reboot it from the dom0 and it did the right thing:
[Thu May 9 10:09:46 2019][ 6445400.3265991] xenbus_shutdown_handler: xenbus_rm
13
[Thu May 9 10:09:46 2019]May 9 10:09:46 future shutdown: poweroff by root:
power button pressed
[Thu May 9 10:10:05 2019]May 9 10:10:05 future syslogd[155]: Exiting on
signal 15
[Thu May 9 10:10:40 2019][ 6445454.6233182] syncing disks... 2 done
[Thu May 9 10:10:40 2019][ 6445454.8073215] unmounting 0xbe00102cb008
/more/archive (more.local:/archive)...
[Thu May 9 10:10:40 2019][ 6445454.9233295] ok
[Thu May 9 10:10:40 2019][ 6445454.9233295] unmounting 0xbe00102c6008
/more/home (more.local:/home)...
But "Because NFS" it stuck there trying to unmount /home and I ended up
typing the unfortunate command:
xl destroy future
I've never had to be quite so emphatic before! :-)
However rebooting got the "future" running quite happily again!
As mentioned it's been taking a while to upgrade, and the whole Xen
server and all its production domains has been running for 87 days.
However when I looked back through the console log I was surprised to
find another blast of these messages from two months ago (after nearly a
month of uptime). However that spew stopped without me knowingly
intervening, after nearly 7000 lines (but just 20 seconds elapsed),
though curiously there's another odd message within seconds of the spew
stopping.
[Wed Mar 20 16:19:01 2019][ 2147554.5719048] route_enqueue: queue full, dropped
message
[Wed Mar 20 16:19:09 2019][ 2147562.9851727] pid 28947 (emacs): user write of
1019904@0x364 at 48052784 failed: 28
If that last message is from a core dump, it might have been caused by
the route_enquue problem (because it lost its X11 connection and emacs
likes to dump core when that happens), or it might have caused the
problem since it would have been dumping to an NFS server (because emacs
on rare occasions ups and dumps core when you least expect it to, though
thankfully far less so in recent releases).
Today though I don't think there was a core dump -- I was using two
different emacs sessions on that host while experiencing the sluggish
behaviour right up until it got too sluggish to use. There are no other
interesting messages in my console logs.
Does anyone have any clues/suggestions/questions for me?
--
Greg A. Woods
+1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgp4HNA6PNhyb.pgp
Description: OpenPGP Digital Signature
Re: mutt and gpg2
At Tue, 7 May 2019 02:01:51 -0400, Bob Bernstein wrote: Subject: mutt and gpg2 > > If built from pkgsrc mutt requires gnupg2. The supplied > executable then is 'gpg2'. > > On the other hand, among the documents that ship with > this mutt is the highly useful 'gpg.rc', which is > easily included into one's .muttrc. > > Problem: The latter is writeen in terms of 'gpg', not > 'gpg2'. I don't use mutt so I'm not sure of the reason for the discrepancy, but I suspect this is just a sign of age of when the example .muttrc file was last updated. It doesn't seem common for pkgsrc to make such patches in example files (though perhaps it should be done more carefully in more cases). My understanding is that the gpg->gpg2 API differences more or less mandated a change of name in the program and that "gpg2" is indeed the official name of the GnuPG-2.x program. If mutt works with that version under the "gpg2" name then the best thing is probably to explicitly list it as such in your .muttrc file. However as for the "official" pkgsrc method for aliasing such commands, there's pkg_alternatives(8). I haven't used it much myself, except for when it is automatically used by some packages. It's been a part of pkgsrc since 2005, so seems stable and widely accepted. On my current system I see it is being used for Go and Python. -- Greg A. Woods +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpNf9dWkW9Tc.pgp Description: OpenPGP Digital Signature
Re: upgrade - what will happen?
At Tue, 30 Apr 2019 09:23:44 -0400, Greg Troxel wrote: Subject: Re: upgrade - what will happen? > > "Greg A. Woods" writes: > > > Isn't there's one small caveat about old shared library versions? > > > > Postinstall(8), which will be run by sysinst during an upgrade, or if > > you do a manual upgrade and then similarly run "postinstall fix" as is > > recommended, will remove "obsolete" files, including old system shared > > libraries. > > The highest minor of any particular major is not supposed to be marked > obsolete, for this very reason. If so, it's a bug; please report it. Ah, of course, the highest minor for every major (and its major-only symlink), i.e. the one that was still pointed to by the symlinks just _before_ the upgrade, and the major-only links of which should be the one directly referenced by any dynamic-linked programs built before the upgrade, will be kept. I too have at least one example of this on one of my systems which started life as 7.0 or so and was recently upgraded to 8.99 with a full "postinstall fix" (I didn't spot it on an earlier check): $ ls -l /usr/lib/libmagic.so* lrwxr-xr-x 1 root wheel 15 Feb 4 14:06 /usr/lib/libmagic.so -> libmagic.so.6.0 lrwxr-xr-x 1 root wheel 15 Feb 20 2015 /usr/lib/libmagic.so.5 -> libmagic.so.5.1 -r--r--r-- 1 root wheel 126225 Feb 20 2015 /usr/lib/libmagic.so.5.1 lrwxr-xr-x 1 root wheel 15 Feb 4 14:06 /usr/lib/libmagic.so.6 -> libmagic.so.6.0 -r--r--r-- 1 root wheel 149016 Feb 3 18:17 /usr/lib/libmagic.so.6.0 (For the record, here if there had been a "libmagic.so.5.0" then it could have been marked as obsolete and have been removed by postinstall with no loss to software built when it was all that existed.) (Shows you how quickly I forget about stuff I try to avoid using!) -- Greg A. Woods +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpc22TlmlIUd.pgp Description: OpenPGP Digital Signature
Re: upgrade - what will happen?
At Sun, 28 Apr 2019 11:02:30 +0200, Benny Siegert wrote:
Subject: Re: upgrade - what will happen?
>
> When you upgrade, the old library versions stay around, so old
> packages continue to work -- with the exception of things depending on
> osabi, which might break.
Isn't there's one small caveat about old shared library versions?
Postinstall(8), which will be run by sysinst during an upgrade, or if
you do a manual upgrade and then similarly run "postinstall fix" as is
recommended, will remove "obsolete" files, including old system shared
libraries.
--
Greg A. Woods
+1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpCHjG1afCZ7.pgp
Description: OpenPGP Digital Signature
Re: SSL_library_init not found in libssl in current
At Fri, 26 Apr 2019 08:38:30 +0530, Mayuresh wrote:
Subject: Re: SSL_library_init not found in libssl in current
>
> On Thu, Apr 25, 2019 at 10:47:15AM -0700, Greg A. Woods wrote:
> > Checking to see if the symbol mentioned in the message is present in the
> > library you think will be used for linking is of course a good idea, but
> > you should also always examine the "config.log" file in the build
> > directory to see the real error message(s) from the configure test.
>
> Yes, that was the first thing I did.
Not a bad idea to post the actual error messages, and maybe the command
that failed too, and maybe even the code of the test file that failed! :-)
--
Greg A. Woods
+1 250 762-7675 RoboHack
Planix, Inc. Avoncote Farms
pgpKAYPhAklbr.pgp
Description: OpenPGP Digital Signature
