Re: Any TCP VTW users?

[Thor Lancelot Simon]

On Thu, Sep 15, 2022 at 04:33:09PM +0900, Ryota Ozaki wrote:
> Hi,
> 
> Are there any users of TCP Vestigial Time-Wait (VTW)?
> The feature is disabled by default and we need to explicitly
> enable via sysctl to use it.
> 
> I just want to know if we should still maintain it.

Have you read the original discussion at
https://mail-index.netbsd.org/tech-net/2011/04/08/msg002566.html ?  I
believe the rationale for this feature is still valid, and I think it
is unfortuanate it was never enabled by default, and thus seems to have
rotted.

Thor

Re: Any TCP VTW users?

[Thor Lancelot Simon]

On Fri, Sep 16, 2022 at 10:33:31AM +0900, Ryota Ozaki wrote:
> 
> Thank you for the report!
> 
> Just curious. Does it improve performance? (or reduce CPU/memory usage?)

The Coyote Point loadbalancers couldn't survive our load testing (which
was modeled on traces of real world workloads) without it.  But Coyote
Point was bought by Fortinet and I believe the NetBSD-based firmware is
long gone.

Coyote Point did much more with the fat pointer stuff, but I don't think any
of it even got into the internal tree before they were purchased and
basically shut down.  A lot of it was actually aimed at connection placement
to improve concurrency within the stack - it is not surprising that if you
are looking at only a small fragment of it, it makes that much _harder_.

Thor

Re: regarding the changes to kernel entropy gathering

[Thor Lancelot Simon]

On Sun, Apr 04, 2021 at 11:02:02PM +, Taylor R Campbell wrote:
> 
> Lots of SoCs have on-board RNGs these days; there are Intel and ARM
> CPU instructions (no ARMv8.5 hardware yet that I know of, but we're
> ready for its RNG!); some crypto decelerators like tpm(4), ubsec(4),
> and hifn(4) have RNGs; and there are some dedicated RNG devices like
> ualea(4).

Can we actually use the TPM RNG from in-kernel?  Whether we should is a
different, interesting question, given how it is typically implemented.

-- 
 Thor Lancelot Simon t...@panix.com
  "Whether or not there's hope for change is not the question.  If you
   want to be a free person, you don't stand up for human rights because
   it will work, but because it is right."  --Andrei Sakharov

Re: regarding the changes to kernel entropy gathering

[Thor Lancelot Simon]

On Tue, Apr 06, 2021 at 10:54:51AM -0700, Greg A. Woods wrote:
> At Mon, 5 Apr 2021 23:18:55 -0400, Thor Lancelot Simon  wrote:
> 
> > But what you're missing is that neither does what you
> > think.  When rndctl -L runs after the system comes up multiuser, all
> > entropy samples that have been added (which are in the per-cpu pools)
> > are propagated to the global pool.  Every stream RNG on the system then
> > rekeys itself - they are _not_ just using the entropy from the seed on
> > disk.  Even if nothing does so earlier, when rndctl -S runs as the system
> > shuts down, again all entropy samples that have been added (which, again,
> > are accumulating in the per-cpu pools) are propagated to the global pool;
> > all the stream RNGs rekey themselves again; then the seed is extracted.
> 
> That's all great, and more or less what I've assumed from all the
> previous discussion
> 
> Except it seems to be useless in practice without an initial seed,

Again there's really little I can do other than suggest you read the code.
You are certainly competent to do so, and the code does not do what you
keep claiming it does.  Read the code, all of it -- it's only a few hundred
lines -- and have a think.

When rndctl -L runs, or you perform a sufficiently long write to /dev/random,
all the per-CPU pools, which, counter to what you keep claiming, *do* accumulate
samples from all the same sources they used to, are coalesced into the global
pool.  When rndctl -S runs, all the per-CPU pools, which, counter to what you
keep claiming, *do* accumulate samples from all the same sources they used to,
are coalesced into the global pool.  If you'd like those samples coalesced
into the global pool more frequently, you can use the sysctl to do so.

Thor

Re: regarding the changes to kernel entropy gathering

[Thor Lancelot Simon]

On Mon, Apr 05, 2021 at 02:13:31PM -0700, Greg A. Woods wrote:
> At Mon, 5 Apr 2021 15:37:49 -0400, Thor Lancelot Simon  wrote:
> Subject: Re: regarding the changes to kernel entropy gathering
> >
> > On Sun, Apr 04, 2021 at 03:32:08PM -0700, Greg A. Woods wrote:
> > >
> > > BTW, to me reusing the same entropy on every reboot seems less secure.
> >
> > Sure.  But that's not what the code actually does.
> >
> > Please, read the code in more depth (or in this case, breadth), then argue
> > about it.
> 
> Sorry, I was eluding to the idea of sticking the following in
> /etc/rc.local as the brain-dead way to work around the problem:
> 
>   echo -n "" > /dev/random
> 
> However I have not yet read and understood enough of the code to know
> if:
> 
>   dd if=/dev/urandom of=/dev/random bs=32 count=1

It's no better.  But what you're missing is that neither does what you
think.  When rndctl -L runs after the system comes up multiuser, all
entropy samples that have been added (which are in the per-cpu pools)
are propagated to the global pool.  Every stream RNG on the system then
rekeys itself - they are _not_ just using the entropy from the seed on
disk.  Even if nothing does so earlier, when rndctl -S runs as the system
shuts down, again all entropy samples that have been added (which, again,
are accumulating in the per-cpu pools) are propagated to the global pool;
all the stream RNGs rekey themselves again; then the seed is extracted.

It is neither the case that samples added with a 0 entropy estimate go
nowhere, nor that they do not add entropy to the seed file such that it
is _not_ "reusing the same entropy on every boot".

If you'd like to propagate samples from the per-CPU pool to the global
pool and force the stream generators to rekey more often, you can
sysctl -w kern.entropy.consolidate=1 from cron.

Re: regarding the changes to kernel entropy gathering

[Thor Lancelot Simon]

On Sun, Apr 04, 2021 at 03:32:08PM -0700, Greg A. Woods wrote:
> 
> BTW, to me reusing the same entropy on every reboot seems less secure.

Sure.  But that's not what the code actually does.

Please, read the code in more depth (or in this case, breadth), then argue
about it.

Re: regarding the changes to kernel entropy gathering

[Thor Lancelot Simon]

On Mon, Apr 05, 2021 at 09:30:16AM -0700, Greg A. Woods wrote:
> At Mon, 5 Apr 2021 10:46:19 +0200, Manuel Bouyer  
> wrote:
> Subject: Re: regarding the changes to kernel entropy gathering
> >
> > If I understood it properly, there's no need for such a knob.
> > echo 0123456789abcdef0123456789abcdef > /dev/random
> >
> > will get you back to the state we had in netbsd-9, with (pseudo-)randomness
> > collected from devices.
> 
> Well, no, not quite so much randomness.  Definitely pseudo though!
> 
> My patch on the other hand can at least inject some real randomness into
> the entropy pool, even if it is observable or influenceable by nefarious
> dudes who might be hiding out in my garage.

No.  You are confused.

All those inputs are *already* being injected into the entropy pool.  If you
don't understand that, you need to read the code more.

All echoing crap into /dev/random does is goose the system's entropy estimate
so it will give you the _output_ of the pool when it thought it shouldn't yet.

Thor

Re: regarding the changes to kernel entropy gathering

[Thor Lancelot Simon]

On Sun, Apr 04, 2021 at 01:08:20PM -0700, Greg A. Woods wrote:
> 
> I trust the randomness and in-observability and isolation of the
> behaviour of my system's fans far more than I would trust Intel's RDRAND
> or RDSEED instructions.

I do not.  However, I do differ with Taylor in that I believe that system
fans are a very good example of a case where there is a well understood
_physical_ basis -- turbulence -- for the existence of the entropy being
collected, and that we should count it, in a very conservative way.

If your system has an audio device, and you mute all the inputs and turn
the gain all the way up, then feed that in as entropy samples (we do not
currently have a device driver to do this, but you could from userslace)
I'd make the same argument, since if you don't get all-zeroes, you're then
sampling the amplifier noise.

I also think there is a case to be made for skew sources since measuring
independently sourced clocks against one another is a basic technique in
many hardware RNGs; but I am not sure the one we have is good enough.

It's important to note that most hardware RNG designs *do not* come with
a real physics model alleging to _compute_ their underlying entropy; rather
they come with an explanation in physical terms of where the entropy comes
from, an explanation of how it is sampled (since this can impact the degree
to which samples are truly independent of one another), and _empirically
derived_ estimates of the minimum expected output entropy.  These are usually
obtained by taking the output data prior to any hashing or "whitening" stage
and running specialized statistical tests, compressing it, etc.

I would _personally_ support counting the entropy from environmental sources,
certainly fans but possibly also thermal sensors, if someone wanted to do the
work of characterizing it in a way which is as rigorous as what's done for
commercial RNGs (the Hifn 799x paper on this is a relatively simple worked
example) _and_ empirically general across a wide array of systems.  And I
think the audio thing is worth exploring.

There is also Peter Gutmann's view that what matters is not really what
mathematicians call "entropy" but, really, the work required for an adversary
to predict the output.  This line of thought can lead to very different
views of what bits should count, on a per-system basis; and I think it would
not be wrong to let the system administrator override this with rndctl,
source by source, as previously they could - but the problem remains that
nobody knows _when_ to count such samples and it is very hard to know you
are not overcounting.

Re: nothing contributing entropy in Xen domUs? (causing python3.7 rebuild to get stuck in kernel in "entropy" during an "import" statement)

[Thor Lancelot Simon]

On Wed, Mar 31, 2021 at 11:24:07AM +0200, Manuel Bouyer wrote:
> On Tue, Mar 30, 2021 at 10:42:53PM +, Taylor R Campbell wrote:
> > 
> > There are no virtual RNG devices on the system in question, according
> > to the quoted `rndctl -l' output.  Perhaps the VM host needs to be
> > taught to expose a virtio-rng device to the guest?
> 
> There is no such thing in Xen.

Is the CPU so old that it doesn't have RDRAND / RDSEED, or is Xen perhaps
masking these CPU features from the guest?

Thor

Re: "wireguard" implementation improperly merged and needs revert

[Thor Lancelot Simon]

On Sat, Aug 22, 2020 at 08:35:39PM +0200, Jason A. Donenfeld wrote:
> 
> In its current form, there are implementation flaws and violations
> that I do not consider acceptable, and deploying this kind of thing is
> highly irresponsible and harmful to your users.

Can you please explain what these (heck, what even _one_ of these) are?

As far as I know the code was written following the published documentation.
That's how Internet protocol development works, is it not?

Thor

Re: Status of 8.99.12

[Thor Lancelot Simon]

On Mon, Feb 12, 2018 at 08:48:32AM +0800, Paul Goyette wrote:
> 
> 1. Starting the gnucash program (from pkgsrc finance/gnucash) now takes
>about 3 times as long as before.  Even after successfully loading
>the image (to get libraries etc into the file system cache) it take
>more than three full minutes for the program to initialize.

It previously took 1 full minute?

How does it look without LOCKDEBUG?

Thor

Re: netbsd 8 (beta) failing to load ixg device

[Thor Lancelot Simon]

On Thu, Nov 09, 2017 at 09:15:53AM -0500, Derrick Lobo wrote:
> The daily beta version of nebtsd 8 does not support ixg 5gb NIC's, the
> support was enabled in 7.99 

That doesn't make sense - if it's in 7.anything, it's in 8.  When we cut
the 8 branch, we move the version number on HEAD to 8.99.

Thor

Re: Running NetBSD in Google Cloud?

[Thor Lancelot Simon]

On Mon, Nov 06, 2017 at 07:07:26PM +, m...@netbsd.org wrote:
> https://github.com/google/netbsd-gce

Oh, hey, I can fix one of the bugs they think that stuff has.  It's just a
quick, nearly trivial command-line change to qemu to make it use a
vioscsi disk instead of an IDE disk...

Thanks!

Thor

Running NetBSD in Google Cloud?

[Thor Lancelot Simon]

It appears with the current rolling Amazon reboot, PV NetBSD instances
may fail to restart, and may not be restartable unless either Amazon
changes their Xen settings to allow "linear" mappings by guests, or
someone steps up to extensively rototill our x86 pmap (or separates
the Xen pmap from it) to not use such mappings.

That leaves me with stuff I've got to move elsewhere, including stuff
I was actually still working to move _to_ EC2 from physical machines.

Some will go to Panix, but for things like nameservers I prefer to have
two separate providers.  As I understand it it should currently be possible
to run NetBSD VMs in Google Cloud, since they are roughly KVM/QEMU and we
support the various virtio devices they provide.  Can anyone offer step
by step instructions for doing so?

Azure looks a little harder.

-- 
  Thor Lancelot Simont...@panix.com
 "The two most common variations translate as follows:
illegitimi non carborundum = the unlawful are not silicon carbide
illegitimis non carborundum = the unlawful don't have silicon carbide."

Re: Crash related to VLANs in Oct 18th -current

[Thor Lancelot Simon]

On Mon, Oct 23, 2017 at 12:18:32PM +0100, Roy Marples wrote:
> 
> I don't know anything about 802.1q trunks.
> How can I tell that it is one, and why shouldn't it have a local address?

Generally speaking, configurations that run tagged ("trunk") and
untagged ("native") traffic on the same interface are poor practice.

One reason in particular (there are others) is that this kind of
configuration is notorious for facilitating attacks in which
double-VLAN-encapsulated packets are fed to an unsuspecting link
partner which then obligingly processes them as if they came in from
a VLAN to which the malicious sender actually had no configured access.
Not good.

Adding addresses by default to physical interfaces that have virtual
interfaces stacked on top of them also breaks important virtual
interface types such as agr, which then can't attach.

I think it is safe to say that an interface which is participating
in an interface stack such as vlan or agr should never be given an
address unless the user has explicitly configured the system to do
so.  The sane default is to give addresses to the leaf interfaces
only (e.g. vlan) not the root nor intermediate nodes (wm, agr, etc --
noting of course that any of these interfaces _could_ be the leaf,
but in fact are not).

Surely if the user has configured the system such that only certain
interfaces should be given addresses, that should be respected.  Even
listening on other interfaces could cause unexpected security exposure
if the client software has bugs.

Thor

Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Thor Lancelot Simon]

On Wed, Oct 11, 2017 at 07:56:04PM -, Michael van Elst wrote:
> t...@panix.com (Thor Lancelot Simon) writes:
> 
> >It probably has to do with our small maximum transfer size.  The disk is
> >probably trying to be safer and *not* caching tagged writes as aggressively,
> >but with only 32 commands in-flight (SCSI/SAS allow 256) and a maximum
> >transfer size of 64K (our MAXPHYS), that's only 2MB of in-flight data;
> >maybe not enough to keep up with the actual bandwidth of the media at
> >its real latency.
> 
> With a larger MAXPHYS there wouldn't be as many transfers in flight.

That depends on the workload and the clustering behavior of the filesystem.

Remember, in the relevant sense, data the disk has already accepted is
"in flight" from the point of view of absorbing latency, in the non-NCQ,
write-cache-enabled case.  The host can't tell it's "in flight", but it
still is -- so the host will continue to write aggressively even if the
filesystem's clustering behavior would otherwise throttle it -- because
to the host, these writes appear to be "complete".

If the drive is not aggressively caching writes in the same way when NCQ
is in use, this won't happen, since less writes will appear, to the host,
to be completed.  With only 32 tags available, we may need a larger
transfer per tag to allow the same amount of in-flight data (if the disk,
with NCQ enabled, has stopped lying about whether the transfers are
really complete).

-- 
  Thor Lancelot Simont...@panix.com
 "The two most common variations translate as follows:
illegitimi non carborundum = the unlawful are not silicon carbide
illegitimis non carborundum = the unlawful don't have silicon carbide."

Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)

[Thor Lancelot Simon]

On Tue, Oct 10, 2017 at 11:11:54PM +0200, Jarom??r Dole??ek wrote:
> I've fixed the compilation for ALL kernels.
> 
> 2017-10-10 17:34 GMT+02:00 Michael :
> > I tried sequential reads ( dd if=/dev/rwd0c ... ) and throughput took a
> > significant hit. I used to get about 120MB/s with the siisata, now it
> > fluctuates between 80 and 90MB/s, ahcisata dropped from about 80MB/s to
> > 70MB/s. Both spinning rust of varying vintage.
> > I should probably do a bonnie run on either one before & after to see
> > if there's any change in random access.
> 
> I've seen this on one of my disks, too. It seems it's much slower in NCQ
> mode. I think the firmware might not utilise the disk cache properly when
> in NCQ mode.

It probably has to do with our small maximum transfer size.  The disk is
probably trying to be safer and *not* caching tagged writes as aggressively,
but with only 32 commands in-flight (SCSI/SAS allow 256) and a maximum
transfer size of 64K (our MAXPHYS), that's only 2MB of in-flight data;
maybe not enough to keep up with the actual bandwidth of the media at
its real latency.

Most other operating systems will hand the drive 32 x 128K transfers, or
even larger.  If you want to attack the -- fairly minor at this point,
I think -- problems of rebasing and merging the tls-maxphys branch,
I bet it would help here.

I have the interest, but not the time, I'm afraid.  Real life interferes
again.

Thor

Re: AMD Ryzen and NetBSD?

[Thor Lancelot Simon]

On Sun, Jul 02, 2017 at 10:57:20PM +0100, Patrick Welche wrote:
> On Fri, Jun 30, 2017 at 12:00:45PM -0400, Thor Lancelot Simon wrote:
> 
> I shoved a rather newer ST2000DM001-1CH164 in, which according to its
> marketing bumpf can manage "Max SustainableTransfer Rate 210MB/s"
> and not so bad:
> 
> # dd if=/dev/zero ibs=64k | progress -l 976751887b dd of=/dev/rdk15 obs=64
> k
>  99% |** |   465 GiB  116.74 MiB/s00:00 
> ETAd

This is already effectively double buffered, because of the way you used
"progress".  You could try using a larger blocksize for the reads from
/dev/zero (1m perhaps) and also for the writes to rdk15 - the kernel
will buffer up and dispatch the MAXPHYS sized I/Os.

To get 200MB out of that drive you likely need larger writes, which we
currently can't do.  It might perform slightly better through the
filesystem, though.

Thor

Re: AMD Ryzen and NetBSD?

[Thor Lancelot Simon]

On Fri, Jun 30, 2017 at 04:55:40PM +0100, Patrick Welche wrote:
> On Fri, Jun 30, 2017 at 03:41:39PM +, m...@netbsd.org wrote:
> > On Fri, Jun 30, 2017 at 09:25:19AM +0100, Patrick Welche wrote:
> > > I'm happily running -current on a Ryzen 7 1700 / Asus PRIME X370-PRO.
> > > I wonder about its i/o though: I remember dd'ing zeros over old disks, and
> > > thought "this is slow", so I dd'd over 3 disks at once, with no change
> > > in individual "progress" bit rate, i.e. 3 times the bandwidth...
> > 
> > Are you specifying block size? dd is terribly slow without it.
> 
> I was using 64k - essentially the example in the "progress" man page.

Latency.  You want a double-buffered pipeline.

-- 
  Thor Lancelot Simont...@panix.com

  "We cannot usually in social life pursue a single value or a single moral
   aim, untroubled by the need to compromise with others."  - H.L.A. Hart

Re: NetBSD-current amd64 with dhcpcd connects only partially

[Thor Lancelot Simon]

On Tue, Jun 13, 2017 at 09:20:24PM +, Thomas Mueller wrote:
> 
> I don't run dhcpcd or anything like that from /etc/rc.conf but ran 
> dhcpcd -dM re0 :

It appears dhcpcd sends DISCOVER messages for a full minute and receives
no responses.  Are you sure the DHCP server is OK?

Thor

Re: disabling mediaopts for if_wm(4)

[Thor Lancelot Simon]

On Tue, May 09, 2017 at 11:05:19PM -0400, Thor Lancelot Simon wrote:
> On Wed, May 10, 2017 at 09:35:44AM +0800, Paul Goyette wrote:
> > 
> > So, what is the magic incantation to set the interface to 100TX FDX, but
> > with flow/pause disabled?
> 
> You can't.  Or, more rightly, you can't do it the way you probably want to.
> 
> If you explicitly set a speed and duplex, you will disable autonegotiation,

Sorry, I should be more clear: if you explicitly set a speed *or* duplex...

Thor

Re: disabling mediaopts for if_wm(4)

[Thor Lancelot Simon]

On Wed, May 10, 2017 at 09:35:44AM +0800, Paul Goyette wrote:
> 
> So, what is the magic incantation to set the interface to 100TX FDX, but
> with flow/pause disabled?

You can't.  Or, more rightly, you can't do it the way you probably want to.

If you explicitly set a speed and duplex, you will disable autonegotiation,
because we do not separately control autonegotiation.  So you can force the
use of 100baseTX/FDX, but we will then not do nway, parallel autodetection
may fail to correctly guess the link duplex, and if you can't hardwire the
link partner you are likely to end up with a duplex mismatch and terrible
performance.

Indeed it's possible that if you've actually already explicitly set the speed
and duplex, what you're actually seeing is the consequence not of flow control
but of a duplex mismatch.

What we should probably do is what most other OSes do: if the user explicltly
sets a speed/duplex or other media options, perform autonegotiation but
restrict the modes we are willing to negotiate to only what the user chose.

Expressly disabling autonegotation should require another deliberate choice
by the user -- or just be impossible, as it has been on e.g. OS X for years.

But we can't do that now.  It'd be lovely if someone cared to make it so we
could.

Thor

Re: rng padlock changes causes NetBSD to crash

[Thor Lancelot Simon]

On Thu, Feb 16, 2017 at 01:04:33PM +0200, Andrius V wrote:
> Hi,
> 
> I have tested the fix.  lcr4(rcr4() | CR4_OSFXSR); helps indeed and
> system boots but if statement seems to be not correct, at least on
> VT-310DP board it ended up in the same error.

I checked in an unconditional version of the fix.

It's interesting to note (again as pointed out by jak@) 64 bit
kernels would not have had this problem, since they enable SSE
very early in CPU startup.

The underlying hardware cause for this seems to also be why we
must mess with the coprocessor enables in CR0 before calling
any ACE/RNG instructions -- in Jonathan's testing, even a single
call to xstorerng causes a coprocessor (DNA) fault if the FPU's
off.  Unfortunate, since all that monkeying around (interrupt
disable, turn off preemption, whack coproc regs) increases the
cost of getting 8 random bytes by a factor of at least 10.

The VIA manual suggests they intended to move PadLock out of
the FPU in later designs but I think at this point it is
fairly clear there won't be any of those.

Thor

Re: rng padlock changes causes NetBSD to crash

[Thor Lancelot Simon]

On Wed, Feb 15, 2017 at 10:25:36AM +0200, Andrius V wrote:
> Hello,
> 
> I have recently decided to test changes in this commit
> https://mail-archive.com/source-changes@netbsd.org/msg64898.html.
> Unfortunately NetBSD (i386) crashes on boot in all systems I have
> tried with which includes VIA VT-310DP (two C5P based Eden-N 1GHz
> CPUs), EPIA-M900 (Nano X2 1.6GHz), Jetway JNF76 (Nano U2300) in the
> same fashion. I have a question if these changes have been ever tested
> on real hardware and does it work for any of you? Should I make a new
> bug report for this?

Despite repeated requests, nobody ever came forward with real hardware
to test on, except a few people who turned out to have very old chips
that lacked the RNG.

It'd be nice to have the output of DDB "trace" and "regs", but maybe I
can find the issue by code inspection -- if you crashed _at_ the
XSTORERNG instruction odds are I flipped two of the arguments to the
asm statement or something.

What happens if you boot an amd64 kernel on the Nano machines?

Thor

Re: rnd entropy estimate running low?

[Thor Lancelot Simon]

On Tue, Jan 31, 2017 at 05:54:37PM +0100, Martin Husemann wrote:
> On Tue, Jan 31, 2017 at 11:45:55AM -0500, Thor Lancelot Simon wrote:
> > The only time we've ever really dug into it, I believe, the user decided
> > the failures were right around the expected failure rate.  Can you help
> > gather more data?
> 
> Good point, and I am not sure this might cover my case as well - will have
> a look and start gathering better data.

Maybe we should alert in a more sophisticated way -- monitor the failure
rate and alert only if it is significantly above the expected rate.

I might even remember enough statistics to do that.

-- 
 Thor Lancelot Simon  t...@panix.com

Ring the bells that still can ring.

Re: rnd entropy estimate running low?

[Thor Lancelot Simon]

On Tue, Jan 31, 2017 at 05:40:01PM +0100, Martin Husemann wrote:
> On Tue, Jan 31, 2017 at 11:38:02AM -0500, Thor Lancelot Simon wrote:
> > The statistical failures later in system run might indicate a memory
> > integrity issue, a race condition of some kind, or just be the expected
> > roughly 3/1 random occurrences.  Hard to say without more information.
> 
> I see statistical test failures on various hardware during later system run
> a lot. Others have stated the same, IIRC.

The only time we've ever really dug into it, I believe, the user decided
the failures were right around the expected failure rate.  Can you help
gather more data?

-- 
 Thor Lancelot Simon  t...@panix.com

Ring the bells that still can ring.

Re: rnd entropy estimate running low?

[Thor Lancelot Simon]

On Tue, Jan 31, 2017 at 05:16:33PM +0100, Havard Eidnes wrote:
> >> Meanwhile the hardware random generator sits there unused.
> >
> > Does it sit there completely unused, or did it get used a little at
> > boot time?
> 
> It generated some bits at boot time, but apparently not early
> enough, because on each reboot the kernel log looks like this:

It looks like nothing's actually calling for bits except the start-up
statistical test (which itself creates demand) before the hardware RNG
attaches, so there shouldn't be a practical problem.  The question is,
could the hardware RNG attach earlier or the statistical test happen
later -- or doesn't it matter?

The statistical failures later in system run might indicate a memory
integrity issue, a race condition of some kind, or just be the expected
roughly 3/1 random occurrences.  Hard to say without more information.

Thor

Re: Wapbl correct and stable again?

[Thor Lancelot Simon]

Could the discards be entered into the log?

On Wed, Oct 19, 2016 at 09:50:57AM +0200, Jarom??r Dole??ek wrote:
> Can you confirm whether you had the filesystem mounted with 'discard'
> option by chance?
> 
> FYI, discard hides block deallocation changes from wapbl log until
> discard is finished for the blocks, so pretty much throws consistency
> out of window. I plan to actually change FFS to disable discard when
> logging is enabled as stopgap measure, because of this. Later, discard
> needs to be changed, so that it doesn't interfere with block free
> maps.
> 
> Jaromir
> 
> 2016-10-18 19:10 GMT+02:00 bch <brad.har...@gmail.com>:
> > Yesterday I had an unclean shutdown while ./build.sh distribution (battery
> > on laptop died). On reboot i had corrupted files similar to other week after
> > wapbl refactor. Could wapbl stand to have more demanding testing thrown at
> > it? Can we simulate unclean shutdown with vn file-based filesystem and run
> > aggressive tests on it?

-- 
  Thor Lancelot Simont...@panix.com

"The dirtiest word in art is the C-word.  I can't even say 'craft'
 without feeling dirty."-Chuck Close

Re: Gumstix Pepper

[Thor Lancelot Simon]

On Sat, Oct 15, 2016 at 05:02:50PM +, Michael van Elst wrote:
> kiyoh...@kk.iij4u.or.jp (KIYOHARA Takashi) writes:
> 
> >By the way, how should I handle a sensor?  Is interface of a Linux
> >compatible provide so that application of Linux can access?
> >Who is doing such work at present?
> 
> Sensors are currently supported by the envsys framework.
> There is nothing like the Linux input devices.

I think it is a very good question whether the Linux abstractions
are better than ours, in this case.  I am no fan of envsys.

Thor

Re: USB serial problems

[Thor Lancelot Simon]

On Wed, Sep 21, 2016 at 07:56:05AM -0700, John Nemeth wrote:
> On Sep 21, 10:30am, Thomas Klausner wrote:
> } 
> } I wanted to look at the serial console of a second machine, so I
> } plugged in a USB serial dongle into my NetBSD (7.99.38/amd64):
> } 
> } uftdi0 at uhub4 port 3
> } uftdi0: FTDI FT232R USB UART, rev 2.00/6.00, addr 3
> } ucom0 at uftdi0 portno 1
> } 
> } Then I looked at ucom(4) and saw that the first mentioned entry is
> } /dev/dtyU?, so I set up 'minicom -s' using that device. Then I
> } connected using minicom. The screen stayed empty, and the process was
> } unkillable. I also tried 'cu -l /dev/dtyU0' which had the same result.
> } (Even though there should have been something visible on the serial
> } line.)
> 
>  Use /dev/ttyU0 for that purpose.  dty devices are for dialup.

Please do not answer questions on the lists if you do not actually know
the correct answer.

Thor

Re: bind -> unbound/nsd

[Thor Lancelot Simon]

On Sun, Aug 28, 2016 at 06:24:41AM +, David Holland wrote:
> 
> So for what it's worth: I don't see any need to have a DNS server in
> base. It may be traditional, but few people use it; the landscape's

As a guy who spent the best part of a decade building embedded products
out of NetBSD: I'll believe all this "you can just add a package" stuff
when all this stuff I should supposedly be able to use from pkgsrc
cross-compiles as cleanly as our base system does.

I had to integrate *PHP* to our "base system" build.  That sucked,
but it sucked a lot less than the logistical and non-reproducibility 
issues which stemmed from building parts of our product outside the
wonderful, clean, cross-compile-from-anywhere-to-anywhere NetBSD
system build.

Thor

Re: bind -> unbound/nsd

[Thor Lancelot Simon]

On Fri, Aug 19, 2016 at 06:13:13PM +0200, Joerg Sonnenberger wrote:
> On Fri, Aug 19, 2016 at 09:55:48AM +0100, Roy Marples wrote:
> > For example, I would use nsd on exactly one machine in my environment,
> > my public facing DNS server which is exactly where it belongs.
> > 
> > On the other hand, all my other BSD machines run unbound as a local
> > caching resolver.
> 
> To slightly expand that. You don't need nsd if you just want to serve a
> few local host names for a local network. You only need nsd if you want
> to provide an authoritive DNS server. IMO that is a decently small use
> case that it doesn't justify the incluse into the base system.

I am strongly opposed to removing basic server functionality present
in BSD Unix for over 30 years -- and still in widespread use -- from NetBSD.
I don't mind replacing BIND but all its functionality should be replacd.

If you want to have to guess which version of basic Internet server
software might happen to be on the system you're working on today, Linux
is -->over there.

Thor

Re: Building on OS X - how?

[Thor Lancelot Simon]

On Sat, Aug 13, 2016 at 10:06:35PM +0200, Jarom??r Dole??ek wrote:
> FWIW, build of tools for both i386 and sparc64 finished without
> problems for me on Mac OS X host (10.11.6), building from clean
> sources.

The problem is not with the tools build.

Thor

New build break on OS X

[Thor Lancelot Simon]

For the past several days, building on OS X 10.9.5 has failed with what
looks like a symptom of host/target confusion that could have more serious
consequences (like bad builds on other platforms).  This is new.  Hubert
has also seen it on OS X 10.5.

Build command is "build.sh -m amd64 -U -u distribution".  Happens in a
clear (non-update) build too.

#  link  backend/genmatch
c++ -O
-I/Volumes/NB/build-src/external/gpl3/gcc/usr.bin/host-libiberty/obj/libiberty
-I.
-I/Volumes/NB/build-src/external/gpl3/gcc/usr.bin/backend/../gcc/arch/x86_64
-DIN_GCC -DHAVE_CONFIG_H -I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/.
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/../include
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/../libcpp/include
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/../libdecnumber
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/../libdecnumber/dpd
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/../libbacktrace
-DGENERATOR_FILE -I/Volumes/NB/build-src/external/gpl3/gcc/usr.bin/backend/..
-o genmatch genmatch.lo build-errors.lo build-vec.lo build-hash-table.lo
-L/Volumes/NB/build-src/obj/tooldir.Darwin-13.4.0-x86_64/lib -lnbcompat
/Volumes/NB/build-src/external/gpl3/gcc/usr.bin/host-libcpp/obj/libcpp/libcpp.a
/Volumes/NB/build-src/external/gpl3/gcc/usr.bin/host-libiberty/obj/libiberty/libiberty.a
Undefined symbols for architecture x86_64:
  "_iconv", referenced from:
  convert_using_iconv(void*, unsigned char const*, unsigned long,
_cpp_strbuf*) in libcpp.a(charset.o)
 (maybe you meant: __Z14cpp_init_iconvP10cpp_reader, __cpp_destroy_iconv )
  "_iconv_close", referenced from:
  __cpp_destroy_iconv in libcpp.a(charset.o)
  __cpp_convert_input in libcpp.a(charset.o)
  "_iconv_open", referenced from:
  init_iconv_desc(cpp_reader*, char const*, char const*) in
libcpp.a(charset.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see
invocation)

*** Failed target:  genmatch
*** Failed command: c++ -O
-I/Volumes/NB/build-src/external/gpl3/gcc/usr.bin/host-libiberty/obj/libiberty
-I.
-I/Volumes/NB/build-src/external/gpl3/gcc/usr.bin/backend/../gcc/arch/x86_64
-DIN_GCC -DHAVE_CONFIG_H -I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/.
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/../include
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/../libcpp/include
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/../libdecnumber
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/../libdecnumber/dpd
-I/Volumes/NB/build-src/external/gpl3/gcc/dist/gcc/../libbacktrace
-DGENERATOR_FILE -I/Volumes/NB/build-src/external/gpl3/gcc/usr.bin/backend/..
-o genmatch genmatch.lo build-errors.lo build-vec.lo build-hash-table.lo
-L/Volumes/NB/build-src/obj/tooldir.Darwin-13.4.0-x86_64/lib -lnbcompat
/Volumes/NB/build-src/external/gpl3/gcc/usr.bin/host-libcpp/obj/libcpp/libcpp.a
/Volumes/NB/build-src/external/gpl3/gcc/usr.bin/host-libiberty/obj/libiberty/libiberty.a
*** Error code 1

Stop.
nbmake[8]: stopped in /Volumes/NB/build-src/external/gpl3/gcc/usr.bin/backend
/bin/sh: line 1: 46823 Segmentation fault: 11
/Volumes/NB/build-src/obj/tooldir.Darwin-13.4.0-x86_64/bin/nbmake
_THISDIR_="${this}" "$@" ${target}

*** Failed target:  dependall-backend
*** Failed command: _makedirtarget() { dir="$1"; shift; target="$1"; shift;
case "${dir}" in /*) this="${dir}/"; real="${dir}" ;; .)
this="external/gpl3/gcc/usr.bin/";
real="/Volumes/NB/build-src/external/gpl3/gcc/usr.bin" ;; *)
this="external/gpl3/gcc/usr.bin/${dir}/";
real="/Volumes/NB/build-src/external/gpl3/gcc/usr.bin/${dir}" ;; esac;
show=${this:-.}; echo "${target} ===> ${show%/}${1:+ (with: $@)}"; cd
"${real}" && /Volumes/NB/build-src/obj/tooldir.Darwin-13.4.0-x86_64/bin/nbmake
_THISDIR_="${this}" "$@" ${target}; }; _makedirtarget backend dependall
*** Error code 139

Stop.
nbmake[7]: stopped in /Volumes/NB/build-src/external/gpl3/gcc/usr.bin

Re: Building on OS X - how?

[Thor Lancelot Simon]

On Thu, Aug 11, 2016 at 04:05:06PM +0100, Robert Swindells wrote:
> 
> >2) /usr/bin/cc:
> >Undefined symbols for architecture x86_64: "_iconv"
> >in external/gpl3/gcc/usr.bin/backend
> 
> This should be in libc.

For what value of "should"?  _iconv is in the implementation-defined
namespace.

It's curious that this doesn't break the tools build, and doesn't
prevent using the built tools to build a kernel!  If this can break
the cross-build of the target compiler, I think we must have suddenly
sprouted a rather serious instance of host/target confusion.

Thor

Re: Building on OS X - how?

[Thor Lancelot Simon]

On Thu, Aug 11, 2016 at 09:00:35AM -0700, Michael Plass wrote:
> 
> That sounds like it may be related to a problem I ran into when cross-building
> from FreeBSD (11 beta something). With a package for libiconv installed, the 
> compile
> must have found the header in /usr/local/include, but the link step did not 
> use the 
> right library.  I worked around this by building in a jail with no packages 
> installed.

If that is the case, then it means that the compilation of a *target*
executable is using *host* header files.  That will eventually cause corrupt
builds even in a NetBSD-on-NetBSD build.

Thor

Re: Building on OS X - how?

[Thor Lancelot Simon]

On Thu, Aug 11, 2016 at 04:29:54PM +0200, Hubert Feyrer wrote:
> 
> 2) /usr/bin/cc:
>Undefined symbols for architecture x86_64: "_iconv"
>in external/gpl3/gcc/usr.bin/backend

This bug has appeared within the past few days and breaks my builds
on OS X 10.9.5 as well.  I'm not having much luck tracking it down.

Thor

Re: Identifying the NetBSD shell

[Thor Lancelot Simon]

On Mon, Mar 21, 2016 at 02:14:05PM -0400, Greg Troxel wrote:
> 
> I find using 7.99.X awkward, as that's a version that means something
> for the kernel (and userland more or less), and this is really something
> quite different.

I strongly agree.  How about just giving it a date rather than a version
number?  You can still bump the date only when something significant
changes.  Then scripts can just numerically require that NBSH > 20160401
or the like.

Thor

Re: kqueue: SIGIO?

[Thor Lancelot Simon]

On Wed, Sep 30, 2015 at 12:30:36AM +0200, Joerg Sonnenberger wrote:
> On Tue, Sep 29, 2015 at 10:09:51PM +0200, Rhialto wrote:
> > On Tue 29 Sep 2015 at 13:22:08 +0200, Tobias Nygren wrote:
> > > Here is the relevant bit of the talk if you are curious:
> > > 
> > > https://www.youtube.com/watch?v=t400SmZlnO8=youtu.be=1888
> > 
> > So he wants a signal when a message is available in a kqueue, in other
> > words, can be read with kevent(2).
> 
> Why oh why. I thought the X server finally got rid of the
> overcomplicated signal handlers. If there is any kind of load going on,
> the signal sending is more costly than occassional querying the kqueue
> for (other active) entries. If there is no load, it doesn't make a
> difference.

What he said.  I've owned some fairly performance-critical single threaded
event driven code in my time, and it is my opinion that trying to use
signals to achieve client fairness is almost always a huge mistake.

The trick is almost always to structure the event loop so that checking
for work from another client is nearly zero-cost, and scales much less
than linearly with the number of clients.  Given the very high cost of
handling a signal, it is pretty darned hard to do worse.

The way to get yourself in trouble is to chase false "optimizations"
involving processing-to-completion of too much work from a single client
at once, or shortcuts involving handing off data directly from one
client to another.  The latter, at least, are really classic priority
inversion bugs in disguise.

In practice, the X server has a shared memory transport to most clients
and a shared memory interface to the display hardware; it should seldom
have syscalls to do.  Arranging for nearly zero-cost "look aside" at
some other *properly designed and structured* shared memory source of
client requests should be pretty easy.  Does the problem actually have
to do with the mouse and keyboard?  Mouse's idea of having the kernel
write a flag word instead of interrupting the process seems like a 
very nice fit if so.

-- 
  Thor Lancelot Simont...@panix.com

  "We cannot usually in social life pursue a single value or a single moral
   aim, untroubled by the need to compromise with others."  - H.L.A. Hart

Re: old file system , old partition tables, and some old scsi drive

[Thor Lancelot Simon]

On Mon, Sep 21, 2015 at 04:33:21AM +, David Holland wrote:
> 
> In any case, there is zero chance that anything from the 80s that has
> ffs on it has a MSDOS FDISK partition table. You're better off using
> disklabel(8) and hoping you can find a BSD-type disklabel.

But we have compatibility support for most "foreign" label formats of
that time, including SunOS, NeXT, HP/UX, and Ultrix labels.  It's not
all compiled into the i386 or amd64 NetBSD kernels, but with a little
hacking, it can be.

And failing that, you can grep around until you find the superblock.

What kind of system did the old drive come from?

Thor

Re: current status of ixg(4)

[Thor Lancelot Simon]

On Tue, Mar 31, 2015 at 03:38:45PM +0200, 6b...@6bone.informatik.uni-leipzig.de 
wrote:
 On Fri, 27 Mar 2015, Masanobu SAITOH wrote:
 
 This change have commited now.
 
 New patch:
 
  http://www.netbsd.org/~msaitoh/ixg-20150327-0.dif
 
 
 I have tested the patch and found no problems.
 
 My server (HP G5) can handle with the new driver package rates up to 200,000
 packets per second. Then CPU0 is running at 100% with interrupts.
 
 If I have not charged me, it comes with 200,000 packets per second and an
 MTU of 1500 bytes to a maximum of 2.4GB.
 
 Is it possible to optimize some parameters for the interrupt throttling?

I think for your workload you may actually want interrupt distribution over
multiple CPUs.  But we have some support for that in -current now, right?

The next thing you're likely to run into is contention for the kernel lock
in ip_input.  But if I recall correctlly how this driver's built, I don't
think you should be seeing that time charged to the interrupt handler, so
I *think* you are not there yet.

Thor

Re: language lawyering: ftello/fseek/pread edition

[Thor Lancelot Simon]

On Mon, Mar 23, 2015 at 04:29:35PM +, David Holland wrote:
 On Mon, Mar 23, 2015 at 04:55:52PM +0100, Thomas Klausner wrote:
   Thanks. So if it survives a build, can I commit the attached patch?
 
 I question the merits of _LARGEFILE_SOURCE (are we going to start
 mucking with _FILE_OFFSET_BITS too?) but whatever.

I object to the presence of any of that LARGEFILE/FILE_OFFSET_BITS
crap anywhere in our source tree.

Thor

Re: DoS attack against TCP services

[Thor Lancelot Simon]

On Sat, Mar 14, 2015 at 06:53:51AM +0800, Paul Goyette wrote:
 On Fri, 13 Mar 2015, Christos Zoulas wrote:
 
 On Mar 13,  6:32pm, hann...@eis.cs.tu-bs.de (J. Hannken-Illjes) wrote:
 -- Subject: Re: DoS attack against TCP services
 
 | What about the attached diff.  It adds a counter of busy items and
 | stops enqueueing more work if an item is still busy.
 |
 | Adds a short time lock to protect this counter and keeps sme_mtx as
 | long time lock.
 
 LGTM!
 
 A cursory review from here also looks good.

A cursory review from here looks like sysmon should be deleted.

But the patch seems OK.

Thor

Re: kernel panic: uvm_fault

[Thor Lancelot Simon]

On Mon, Jan 26, 2015 at 04:28:22PM +0100, Thomas Klausner wrote:
 On Mon, Jan 26, 2015 at 10:01:41AM -0500, Thor Lancelot Simon wrote:
  alloc_bouncebus?  On amd64?  I think you've got a trashed pointer
  somewhere.
 
 I have
 makeoptions   DEBUG=-g  # compile full symbol table
 
 # grep -r _bus_dma_alloc_bouncebuf /usr/src/sys/arch/amd64/
 Binary file /usr/src/sys/arch/amd64/compile/obj/KERNELNAME/bus_dma.o matches

Yes, but why would you be trying to bounce PCI DMA on a 64-bit platform?

Thor

Re: Removing openldap?

[Thor Lancelot Simon]

On Thu, Oct 02, 2014 at 10:33:02AM +, Michael van Elst wrote:
 w...@netbsd.org (Thomas Klausner) writes:
 
 I think there is no particular need to have openldap in the base
 system; I don't see any particular integration, and it puts more
 burden of maintenance on us. I think that installing openldap from
 pkgsrc should be good enough.
 
 openldap is used by postfix, sshd and amd. There is also pam-ldap in
 pkgsrc that we might want to import into base.
 
 All this is only using the client part of openldap.

I would support removing the server parts of openldap but wonder whether
this would actually reduce maintenance burden.

It would reduce attack surface on some systems.

-- 
 Thor Lancelot Simon  t...@panix.com
From the tooth paste you use in the morning to the salt on your evening meal,
it's easy to take for granted the many products brought to us with explosives.
- Institute of Manufacturers of Explosives, Explosives Make It Possible 

Re: a separate build of libc

[Thor Lancelot Simon]

On Sat, Sep 20, 2014 at 01:03:05PM +0200, u-6...@aetey.se wrote:
 On Fri, Sep 19, 2014 at 08:10:31PM -0400, Thor Lancelot Simon wrote:
  On Fri, Sep 19, 2014 at 04:04:34PM +0200, u-6...@aetey.se wrote:
   Background:
   
 building an independent/standalone toolchain able to produce binaries
 runnable on NetBSD
  
  Why don't you just run build.sh -m machine tools and use the result?
 
 Because I aim to be able to rebuild the tools without having
 everything needed/assumed by build.sh.
 
 The intention is to have an independent and self-contained toolchain, to
 avoid any explicit or implicit restrictions/assumptions of the existing
 building routines.

I don't understand.  You will work very, very hard to construct a cross
build framework as robust as build.sh, and why?  Because you think there
might be some restrictions/assumptions nobody might have noticed or
be willing to fix?

If you tell build.sh to build the tools, that's all it does: build an
independent and self-contained toolchain.  Everything you need ends up
in the tooldir.  No need to build any other parts of NetBSD.

Why duplicate effort?

Thor

Re: a separate build of libc

[Thor Lancelot Simon]

On Fri, Sep 19, 2014 at 04:04:34PM +0200, u-6...@aetey.se wrote:
 Background:
 
   building an independent/standalone toolchain able to produce binaries
   runnable on NetBSD

Why don't you just run build.sh -m machine tools and use the result?

Thor

Re: Add MKCTF

[Thor Lancelot Simon]

On Mon, Mar 24, 2014 at 07:27:33PM +0900, Ryota Ozaki wrote:
 Hi,
 
 I'm adding a new build variable MKCTF, which
 was discussed on the chat some days ago.
 
 Currently CTF tools are built and used to generate
 and manipulate CTF data of ELF binaries when
 we build with MKDTRACE=yes. Unfortunately,
 current CTF tools don't work on i386/amd64/arm,
 and that adds a burden to try DTrace by users.
 
 The new variable is intended to separate CTF
 stuffs from MKDTRACE; we can build DTrace
 solely without worrying about the CTF issues.
 Fortunately, CTF data are not used yet by
 current DTrace of NetBSD, so DTrace still
 works without CTF data.

The CTF data for the kernel are needed in order to use
DTrace on the kernel, no?

Thor

Re: Build broken with new gcc 4.8.3.

[Thor Lancelot Simon]

On Sun, Mar 02, 2014 at 03:36:57PM -0500, Christos Zoulas wrote:
 
 Anything you compile with -g will now need -gdwarf-2 if you are using
 ctf{convert,merge} on them because our libdwarf understands dwarf-2,
 partially dwarf-3 and no dwarf-4 which is the default for gcc-4.8.

This looks relevant:

http://lists.freebsd.org/pipermail/freebsd-toolchain/2014-January/001107.html

Or we could pitch dtrace entirely.  But that seems like it would not be
a very good choice.

Thor

Re: kernel crashes because crypto unloading?

[Thor Lancelot Simon]

On Sun, Jan 19, 2014 at 07:13:12PM +, David Laight wrote:
 
 open/close (well probably the vnode) needs to hold a reference count
 against the device.

This is complicated somewhat because the device clones.

Thor