Re: Specify swap device priority in fstab?
Valery Ushakov writes: > On Wed, Jun 05, 2024 at 06:38:04 -0700, Paul Goyette wrote: > >> Yup, that works, even when using ``sw,dp,priority=100'' (ie, the >> dp portion can intervene and it still works). >> >> Thanks - the obvious answer is the hardest to find! > > A sign that fstab(5) could use some clarifications. :) One can sort of > make the necessary inferences, but explicit text (and examples) are > better. > > Can someone, please, tweak fstab(5) to explicitly say that some > options from the options column are processed by the mount(8) frontend > itself and the rest is passed to the appropriate mount program (where > swapctl(8) is the "mount" program for swaps) via -o. We already have > the text that explains fs_type, though that could use a bit of > copyediting too. > > Don't worry about the markup too much, I'll clean it up if necessary. > > -uwe While it may not be the most obvious, the required information is already in swapctl(8) (at least in -current) as far as I can tell. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Specify swap device priority in fstab?
Paul Goyette writes: > Is it possible to do this? > > I have two swap files, and I want to designate one of them as also a > dump file. I tried to put ``sw,dp,-p=50'' on the dump file entry but > ``swapctl -A'' still says it is using priority 0. > > I don't want swapping to overwrite a crash dump ... > > > +-+--+--+ > | Paul Goyette (.sig) | PGP Key fingerprint: | E-mail addresses:| > | (Retired) | 1B11 1849 721C 56C8 F63A | p...@whooppee.com| > | Software Developer | 6E2E 05FD 15CE 9F2D 5102 | pgoye...@netbsd.org | > | & Network Engineer | | pgoyett...@gmail.com | > +-+--+--+ Do something like this (an example using a file as swap): /swap/swap.1none swapsw,priority=2 0 0 -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Minor sysupgrade issue
Chavdar Ivanov writes: [snip] > But then wouldn't be > there a chance that the old userland disagrees with the new kernel? > As a general statement, that should not be a problem as long as the kernel has the COMPAT_* options set. My own personal experience is that since about 1995, I have encountered exactly two instances where the newer kernel would not run the older userland completely correctly assuming the new kernel had the COMPAT_* options. One was very early where some programs from a 0.9 userland had trouble with a 1.0 kernel. I never understood what was going on, and a recompile of the programs fixed them. The second was around the 4.x - 6.x time frame where a system call was removed completely and old userland had trouble. So if you were going from 4.x to something more recent, you might have some problems especially with pkgsrc. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Upgrading a 90s laptop from 5.1 to 10 -- no FD or CDROM
jo...@sdf.org writes: >> Any chance that BIOS can do netboot? It's a long shot perhaps, >> but I've owned several non-Intel machines of approximately that >> vintage that I've successfully netbooted NetBSD onto, for >> reasons similar to yours: the other possibilities required >> no-longer-functioning hardware. >> >> regards, tom lane >> > > Unfortunately, no PXE BIOS option. > > This was my first laptop and it ran Win98 originally (yep, scary). It > wasn't long before I realized I could run a BSD on it. > > Maybe it's time to realize I can just enjoy a previous great release: > NetBSD 5.1. I have compiler tools on it, so still useful. > > Thanks - Joel I have done a couple of 4->7 updates in one leap without any problems. Since you are on 5 that would be after the Schedular Activation change to threads so if the newer kernel boots about 99% of userland will likely just work (that wasn't the case with 4.x, as anything linked against the older threads shared library would fail badly). I have not tried leaping to 9, but it should be pretty painless to just boot the kernel and see. I would place the 9.x kernel in root fs as something like /netbsd_9.x and put the 9.x modules in /stand. Then I would break into the boot loader when it is booting and do something like "boot netbsd_9.x". As mentioned, some of the networking may not come up, but if the kernel is bootable with your boot blocks and boot loader it has a fair chance of working out for you. If you want to proceed with the update, check dmesg and make sure that your devices are probed and attached correctly. A varient of this would be to set rc_configured to NO in /etc/rc.conf before trying the 9.x kernel as the boot will then stop in single user. You can always do a "mount -u /" later and edit /etc/rc.conf again to get multiuser up (maybe after mounting /var and /usr if needed). With the new kernel running you can place the sets on a thumb drive (from another system) and unpack them onto the target system (this can be tricky, as you may be overwritting the tar command that you are using... usually what I do is make a copy of /rescue/tar somewhere else and use that copy. That would be statically linked and even a 5.x version should be able to unpack the 9.x sets. Just make sure you use something like '-xpf' when you do it to get the permissions correct). Then reboot again with the new kernel to get the new userland running. Then do postinstall (lots of ways to run this one, but something like "postinstall -s /thumb/etc.tar.xz -s /thumb/xetc.tar.xz -d /" is what you want) and then reboot again with the new kernel (somewhere along the way, sway the kernel names around so the 9.x kernel is /netbsd and the older one is /netbsd_5.x). One of the 4->7 updates I did was a very old laptop with a very small hard drive in it that could not be upgraded as its physical size was not standard. It used a cardbus card for networking. There was just enough room for the newer kernel and modules in / but nothing else and no other room for the sets anywhere. I used a thumb drive to transport the sets into the system. Updated in one leap more or less... -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes: [snip] > Can you try the attached patch on a dom0 and see if you still observe > drift? I had a little time to mess with trying 10.99.6 on the DOM0.. I applied your patch to 10.99.6 and built a XEN3_DOM0 kernel (XEN3_DOM0 + bigger IFQ_MAXLEN). The result was, while the kernel booted up, when a 32 bit 9.x PVH+PVSHIM guest performed its ifconfig against a xennet interface, the DOM0 rebooted. No DDB, no obvious panic message,... just a hang for a bit and a reboot. I also observed the DOM0 appear to hang up hard before the guest started while poking at other stuff. I don't know if the patch caused the trouble, or if 10.99.6 has trouble for me in a general sense (or if I am not allowed any more to make IFQ_MAXLEN larger than the default). I won't have time to mess with this again for a while, but the results were not encouraging. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes: > [1:text/plain Hide] > >> Date: Tue, 01 Aug 2023 16:02:17 -0400 >> From: Brad Spencer >> >> Taylor R Campbell writes: >> >> > So I just added a printf to the kernel in case this jump happens. Can >> > you update to xen_clock.c 1.15 (and sys/arch/x86/include/cpu.h 1.135) >> > and try again? >> >> Sure... > > Correction: xen_clock.c 1.16 and sys/arch/x86/include/cpu.h 1.136 > (missed a spot). I noticed the second update and believe that I am running the latest of those files. I have xen_clock.c at 1.17 and cpu.h at 1.136. >> >> If the dtrace does continue to run, sometimes, it is impossible to exit >> >> with CTRL-C. The process seems stuck in this: >> >> >> >> [ 4261.7158728] load: 2.64 cmd: dtrace 3295 [xclocv] 0.01u 0.02s 0% 7340k >> > >> > Interesting. If this is reproducible, can you enter crash or ddb and >> > get a stack trace for the dtrace process, as well as output from ps, >> > ps/w, and `show all tstiles'? >> >> It appears to be reproduceable.. in the sense that I encountered it a >> couple of times doing exactly the same workload test. I am more or less >> completely unsure as to what the trigger is, however. I probably should >> have mentioned, but when this happened the last time, I did have other >> newly created processes hang in tstile (the one in particular that I >> noticed was 'fortune' from a ssh attempt .. it got stuck on login and >> when I did a CTRL-T tstile was shown). > > `show all tstiles' output in crash or ddb would definitely be helpful > here. I started the last dtrace that was mentioned before the latest abuse test and it has already exited claiming a unresponsive system. I highly suspect that if I start it again, it will hang as that seems to be what occurred previously. However, that will have to wait a bit until the abuse is done. If there are no additional messages about negative runtime in the morning, I will try starting the dtrace and see if it hangs up and perform the ddb outputs. This will also result in the guest rebooting to get it all unhung again. >> I also probably should have mentioned that the DOM0 (NOT the DOMU) that >> the target system is running under has HZ set to 1000. This is mostly >> to help keep the ntpd and chronyd happy on the Xen guests. If the DOM0 >> is left at 100 the drift can be too much on the DOMU systems. Been >> running like this for a long time... > > Interesting. Why would the dom0's HZ choice a difference? Nothing > in the guest should depend substantively on the host's tick rate. I really have no idea why it matters, but it did seem to help some. > A NetBSD XEN3_DOM0 kernel periodically updates the hypervisor with a > real-time clock derived from NTP (the `timepush' callout in > xen_clock.c), but the period between updates is 53 sec + 3 ticks, and > it's hard to imagine that setting the clock every 53.03 sec vs every > 53.003 sec should make any difference for whether guests drift. > > The resolution of the real-time clock sent to NTP is 1/hz, because > resettodr uses getmicrotime instead of microtime, but while that might > introduce jitter from rounding, I'm not sure it should cause > persistent drift in one direction or the other and I don't think > guests are likely to periodically query the Xen wall clock time often > enough for this jitter to matter. > > Does the dom0 have any substantive continuous influence on domU > scheduling and timing? I always assumed the hypervisor would have all > the say in that. > > As an aside, I wonder whether it's even worthwhile to run ntpd or > chronyd on the domU instead of just letting the dom0 set it and > arranging to do the equivalent of inittodr periodically in the domU? Something will need to keep the time synced. I can tell you with 100% assurance that ntpd will report differences in its statistics and metrics between a DOM0 and a PV DOMU on the same hardware. I have been running some metric gathering for years from ntpq output and it this is 100% the case. Currently I have been using ntpd on single processor DOMUs and chronyd on multiprocessor DOMUs (because ntpd very often can not deal with the drift present with DOMU vcpu > 1). I can also say with 100% certainty that for a PV DOMU one of those, or something like it, will be required. Your suggestion is something like what Solaris zones do, where there you don't run ntpd in the zone, and it gets it time from the global zone. This would probably be a very useful thing to do for Xen if it were possible. > Can you try the attached patch on a dom0 and see if you still observe > drift? For the DOM0 that is a little more involved than
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
>> Date: Mon, 31 Jul 2023 12:47:20 -0400
>>
>> # dtrace -x nolibs -n 'sdt:xen:hardclock:jump { @ = quantize(arg1 - arg0) }
>> sdt:xen:hardclock:jump /arg2 >= 430/ { printf("hardclock jump violated
>> timecounter contract") }'
>> dtrace: description 'sdt:xen:hardclock:jump ' matched 2 probes
>> dtrace: processing aborted: Abort due to systemic unresponsiveness
>
> Well! dtrace might be unhappy if the timecounter is broken too, heh.
> So I just added a printf to the kernel in case this jump happens. Can
> you update to xen_clock.c 1.15 (and sys/arch/x86/include/cpu.h 1.135)
> and try again?
Sure...
>> The system is fine just after a reboot, it certainly seems to be a
>> requirment that a fair bit of work must be done before it gets into a
>> bad state.
>>
>> If the dtrace does continue to run, sometimes, it is impossible to exit
>> with CTRL-C. The process seems stuck in this:
>>
>> [ 4261.7158728] load: 2.64 cmd: dtrace 3295 [xclocv] 0.01u 0.02s 0% 7340k
>
> Interesting. If this is reproducible, can you enter crash or ddb and
> get a stack trace for the dtrace process, as well as output from ps,
> ps/w, and `show all tstiles'?
It appears to be reproduceable.. in the sense that I encountered it a
couple of times doing exactly the same workload test. I am more or less
completely unsure as to what the trigger is, however. I probably should
have mentioned, but when this happened the last time, I did have other
newly created processes hang in tstile (the one in particular that I
noticed was 'fortune' from a ssh attempt .. it got stuck on login and
when I did a CTRL-T tstile was shown).
I also probably should have mentioned that the DOM0 (NOT the DOMU) that
the target system is running under has HZ set to 1000. This is mostly
to help keep the ntpd and chronyd happy on the Xen guests. If the DOM0
is left at 100 the drift can be too much on the DOMU systems. Been
running like this for a long time...
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
>> Date: Sun, 30 Jul 2023 14:56:53 -0400
>> From: Brad Spencer
>>
>> Taylor R Campbell writes:
>>
>> > Can you please try running with the attached patch and share the
>> > warnings it produces? Should give slightly more information.
>>
>> Caught another one. As far as I know the system is up to date with all
>> of the requested patches:
>>
>> [ 19419.647972] WARNING: lwp 16 (system idle/1) flags 0xa020:
>> timecounter went backwards from (19420 + 0x9e37cf0149d8f7bb/2^64) sec at
>> netbsd:mi_switch+0x11e on cpu1 to (19419 + 0xad917b77bd0a7cd3/2^64) sec at
>> netbsd:mi_switch+0x11e on cpu1
>
> Can you run this dtrace script for a while (say, for a day, or from
> start of boot until you see the WARNING above which only happens once
> per boot), and then hit ^C?
>
> dtrace -x nolibs -n 'sdt:xen:hardclock:jump { @ = quantize(arg1 - arg0) }
> sdt:xen:hardclock:jump /arg2 >= 430/ { printf("hardclock jump violated
> timecounter contract") }'
>
> If my hypothesis is correct, you can just leave this running over any
> particular workload and you'll get:
>
> (a) a message printed whenever the hardclock delay is too long, and
> (b) when you hit ^C at the end, a histogram of all the >1-tick
> hardclock jump delays.
>
> (Avoiding the tick-10s probe, like I used in the last dtrace
> suggestion, means you won't get updates printed every 10sec to your
> terminal -- you'll have to hit ^C to see the results -- but as an
> upside it won't instantly crash your kernel owing to the Xen/!Xen
> module ABI mismatch for CLKF_USERMODE/PC.)
Struggling a little with this one...
The usual work load I have been doing is a couple of world builds at a
time sometimes followed by a cvs update of -current and NetBSD 10.x and
a rebuild of cscope indexes. I suspect, but can't confirm, that one of
the recent patches (probably the one to call hardclock more often, if I
remember correctly) is causing some trouble with stability. The system
starts to act pretty bad mostly when accessing disk after the world
builds are done. Often the cvs update will not complete at all. I can
start the above dtrace, but it will often exit with the following:
# dtrace -x nolibs -n 'sdt:xen:hardclock:jump { @ = quantize(arg1 - arg0) }
sdt:xen:hardclock:jump /arg2 >= 430/ { printf("hardclock jump violated
timecounter contract") }'
dtrace: description 'sdt:xen:hardclock:jump ' matched 2 probes
dtrace: processing aborted: Abort due to systemic unresponsiveness
The system is fine just after a reboot, it certainly seems to be a
requirment that a fair bit of work must be done before it gets into a
bad state.
If the dtrace does continue to run, sometimes, it is impossible to exit
with CTRL-C. The process seems stuck in this:
[ 4261.7158728] load: 2.64 cmd: dtrace 3295 [xclocv] 0.01u 0.02s 0% 7340k
At that point, the guest must be destroyed and rebooted as more and more
processes will start to get stuck.
If I don't put just too much of a heavy load on the system (or don't
wait to long after loading up the system) I can interrupt the dtrace and
it does produce a graph. Here is one after about 2.5 hours of world
building:
# dtrace -x nolibs -n 'sdt:xen:hardclock:jump { @ = quantize(arg1 - arg0) }
sdt:xen:hardclock:jump /arg2 >= 430/ { printf("hardclock jump violated
timecounter contract") }'
dtrace: description 'sdt:xen:hardclock:jump ' matched 2 probes
^C
value - Distribution - count
8388608 | 0
16777216 | 220301
33554432 | 0
I saw the WARNING about negative runtime just once so far again, but the
dtrace had exited with the "systemic unresponsiveness" message and I
could not interrupt it.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes: > Can you please try running with the attached patch and share the > warnings it produces? Should give slightly more information. Caught another one. As far as I know the system is up to date with all of the requested patches: [ 19419.647972] WARNING: lwp 16 (system idle/1) flags 0xa020: timecounter went backwards from (19420 + 0x9e37cf0149d8f7bb/2^64) sec at netbsd:mi_switch+0x11e on cpu1 to (19419 + 0xad917b77bd0a7cd3/2^64) sec at netbsd:mi_switch+0x11e on cpu1 The system had done a couple of world builds and was doing a cvs update of -current and NetBSD 10.x and/or perhaps rebuilding cscope indexes. I am getting the general sense that some of the trigger for this problem may be disk activity, but I can't really prove it. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
>> Date: Fri, 28 Jul 2023 07:42:04 -0400
>> From: Brad Spencer
>>
>> dtrace: invalid probe specifier sdt:xen:hardclock:jump { @ = quantize(arg1 -
>> arg0) } tick-10s { printa(@) }: probe description :::tick-10s does not match
>> any probes
>
> modload dtrace_profile
Another data point or three... as mentioned in the previous email, the
above probe panics the -current PV kernel after a few seconds. It
appears to work if you leave off the '-10s' part, however. Further, it
appears to work as presented with the GENERIC kernel in PVH mode. I
suspect that running in PVH mode changes a lot of things around, so the
comparison is likely not valid. In fact, I run a PVH NetBSD 10_BETA
system with 2 VCPUs and it never has the negative runtime problem (as
far as I know). The work load is quite a bit different, but it may be
that the problem is limited to just DOMU PV guests.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
>> Date: Fri, 28 Jul 2023 07:42:04 -0400
>> From: Brad Spencer
>>
>> dtrace: invalid probe specifier sdt:xen:hardclock:jump { @ = quantize(arg1 -
>> arg0) } tick-10s { printa(@) }: probe description :::tick-10s does not match
>> any probes
>
> modload dtrace_profile
Thanks, that allowed the dtrace to execute, but I never have time to
execute the second probe, as this kernel panic occures within a few
seconds of the first probe being run (probably on the order of 4 - 5
seconds):
[ 213.9904671] fatal page fault in supervisor mode
[ 213.9904671] trap type 6 code 0 rip 0x824d29ad cs 0xe030 rflags
0x10283 cr2 0xd8 ilevel 0x7 rsp 0x92032de8db58
[ 213.9904671] curlwp 0x92001784e9c0 pid 0.16 lowest kstack
0x92032de892c0
kernel: page fault trap, code=0
Stopped in pid 0.16 (system) at cyclic:cyclic_clock+0x86: movqd8(%rbx)
,%rax
cyclic_clock() at cyclic:cyclic_clock+0x86
hardclock() at netbsd:hardclock+0x8f
xen_timer_handler() at netbsd:xen_timer_handler+0x95
evtchn_do_event() at netbsd:evtchn_do_event+0x11c
do_hypervisor_callback() at netbsd:do_hypervisor_callback+0x17e
Xhandle_hypervisor_callback() at netbsd:Xhandle_hypervisor_callback+0x1a
--- interrupt ---
hypercall_page() at netbsd:hypercall_page+0x3aa
idle_loop() at netbsd:idle_loop+0x147
cpu_hatch() at netbsd:cpu_hatch+0x19c
ds 58f5
es f72a
fs 196
gs 246
rdi 92032de8db78
rsi 0
rbp 92032de8dbb8
rbx 0
rdx d5
rcx 658f5
rax 1
r8 920017834000
r9 399fcd6a
r10 a9d32cec93
r11 963bca08
r12 1
r13 31d2d365f4
r14 823f9460solaris_cpu+0x40
r15 321e02ef47
rip 824d29adcyclic_clock+0x86
cs e030
rflags 10283
rsp 92032de8db58
ss e02b
cyclic:cyclic_clock+0x86: movqd8(%rbx),%rax
db{1}> bt
cyclic_clock() at cyclic:cyclic_clock+0x86
hardclock() at netbsd:hardclock+0x8f
xen_timer_handler() at netbsd:xen_timer_handler+0x95
evtchn_do_event() at netbsd:evtchn_do_event+0x11c
do_hypervisor_callback() at netbsd:do_hypervisor_callback+0x17e
Xhandle_hypervisor_callback() at netbsd:Xhandle_hypervisor_callback+0x1a
--- interrupt ---
hypercall_page() at netbsd:hypercall_page+0x3aa
idle_loop() at netbsd:idle_loop+0x147
cpu_hatch() at netbsd:cpu_hatch+0x19c
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
>> Date: Thu, 27 Jul 2023 11:17:30 -0400
>> From: Brad Spencer
>>
>> On the system that I have that exhibits the negative runtime problem, it
>> may very well be the case that hardclocks are missed for 4.3sec. The
>> system has to have been up for a while and busy as a prereq., but if I
>> then run:
>>
>> dtrace -x nolibs -n 'sdt:xen:clock:, sdt:xen:hardclock:,
>> sdt:xen:timecounter: { printf("%d %d %d %d %d %d %d %d", arg0, arg1, arg2,
>> arg3, arg4, arg5, arg6, arg7) }'
>
> Update xen_clock.c, and try this now (one in each of two terminals):
>
> dtrace -x nolibs -n 'sdt:xen:hardclock:jump { @ = quantize(arg1 - arg0) }
> tick-10s { printa(@) }'
The first one errors for me:
# cat d3a
#!/bin/sh
dtrace -x nolibs -n 'sdt:xen:hardclock:jump { @ = quantize(arg1 - arg0) }
tick-10s { printa(@) }'
# ./d3a
dtrace: invalid probe specifier sdt:xen:hardclock:jump { @ = quantize(arg1 -
arg0) } tick-10s { printa(@) }: probe description :::tick-10s does not match
any probes
with
$NetBSD: xen_clock.c,v 1.15 2023/07/28 10:39:14 riastradh Exp $
I updated nothing else other than xen_clock.c and rebuilt the kernel and
booted it.
> dtrace -x nolibs -n 'sdt:xen:hardclock:jump /arg2 >= 430/ {
> printf("TIMECOUNTER BAD: hardclock delayed by %d ns (%d tick)", arg1 - arg0,
> arg2) }'
This one executed.
> Let it run for a while until the second one fires, or until you get
> bored, then share the last histogram from the first one?
>
> (Note: I added a probe sdt:xen:hardclock:tick which fires on every
> tick, so the original dtrace script will be a little too noisy. If
> you want to do it again for some reason, narrow the scope from
> `sdt:xen:hardclock:' to just `sdt:xen:hardclock:missed,
> sdt:xen:hardclock:jump'.)
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
[snip]
> I don't know what the sparc timecounter frequency is, but the Xen
> system timecounter returns units of nanoseconds, i.e., runs at 1 GHz,
> well within these bounds. So this kind of wraparound leading to
> apparently negative runtime -- that is, l->l_stime going backwards --
> should not be possible as long as we are calling tc_windup() at a
> frequency of at least 1 GHz / (2^k / 2) = 0.47 Hz.
>
> That said, at a 32-bit timecounter frequency of 1 GHz, if there is a
> period of about 2^32 / 1 GHz ~= 4.3sec during which we miss all
> consecutive hardclock ticks, that would violate the timecounter(9)
> assumptions, and tc_delta(th) may go backwards if that happens.
>
> So I think we need to find out why we're missing Xen hardclock timer
> interrupts. Should also make the dtrace probe show exactly how many
> hardclock ticks in a batch happened, and should raise an alarm (with
> or without dtrace) if it exceeds a threshold.
[snip]
On the system that I have that exhibits the negative runtime problem, it
may very well be the case that hardclocks are missed for 4.3sec. The
system has to have been up for a while and busy as a prereq., but if I
then run:
dtrace -x nolibs -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: {
printf("%d %d %d %d %d %d %d %d", arg0, arg1, arg2, arg3, arg4, arg5, arg6,
arg7) }'
on what is otherwise an idle system there are a continous stream of
hardclock missed events.
And... 'vmstat -e | grep -e tsc -e systime -e hardclock' shows:
vcpu0 missed hardclock 6882953 74 intr
vcpu1 missed hardclock 322680 intr
.. that second number, 74, in this case, is a rate.. that is, there are
continuous missed hardclocks even while idle.
If the system is freshly rebooted, the vmstat will not show anything
until after the system has been busy for a while and likewise the dtrace
is nearly free of events.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes: > Can you please try running with the attached patch and share the > warnings it produces? Should give slightly more information. I applied the patch and caught this: [ 76783.161208] WARNING: lwp 16 (system idle/1) flags 0xa020: timecounter went backwards from (76784 + 0x219ae38ce039d545/2^64) sec at netbsd:mi_switch+0x11e on cpu1 to (76783 + 0x30f9c248a810cadf/2^64) sec at netbsd:mi_switch+0x11e on cpu1 The system had just performed three builds of the world and was in the process of doing cvs updates against -current, the NetBSD 10 branch and the NetBSD 9 branch. After doing the cvs updates, it then rebuild the cscope indexes for each branch. There would be a lot of disk activity during this, but the CPU load would not be very high. Chronyd, which is what is being used on that system instead of ntpd, noted this: Jul 27 07:15:06 samwise chronyd[1144]: Forward time jump detected! Right now the system is somewhat unresponsive for any new logins, but the console still works (NFS might be mucked up or something). I am going to let it sit for a while and see if ssh comes back. The actual time on the system is not off by much (if any). -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
>> Date: Thu, 20 Jul 2023 21:50:26 -0400
>> From: Brad Spencer
>>
>> With a DOMU kernel compiled with KDTRACE_HOOKS I get the following with
>> either of those dtrace probes on the DOMU:
>>
>> dtrace -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: {
>> printf("%d %d %d %d %d %d %d", arg0, arg1, arg2, arg3, arg4, arg5, arg6,
>> arg7) }'
>> dtrace: invalid probe specifier sdt:xen:clock:, sdt:xen:hardclock:,
>> sdt:xen:timecounter: { printf("%d %d %d %d %d %d %d", arg0, arg1, arg2,
>> arg3, arg4, arg5, arg6, arg7) }: "/usr/lib/dtrace/psinfo.d", line 46: syntax
>> error near "u_int"
>
> Can you please file a PR about that, and pass `-x nolibs' to dtrace in
> the mean time?
>
> dtrace -x nolibs -n 'sdt:xen:clock:...'
Back to the original problem
I got this on the console:
[ 159564.607341] WARNING: lwp 105 (system pgdaemon) flags 0x20020020:
timecounter went backwards from (159565 + 0x60917da1f41f3971/2^64) sec to
(159564 + 0x9b7aac0ce33a822d/2^64) sec in netbsd:mi_switch+0x112
I assume that this indicates some or other is going off the rails.
$ vmstat -e | grep -e tsc -e systime -e hardclock
vcpu0 missed hardclock16031340 92 intr
vcpu1 missed hardclock 534620 intr
The dtrace probes produced some amount of timecounter:backwards and tons
of hardclock:missed events.
The system is mostly idle right now with out the only thing going on
being a cvs update. Some some amount of disk activity.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
>> Date: Thu, 20 Jul 2023 21:50:26 -0400
>> From: Brad Spencer
>>
>> With a DOMU kernel compiled with KDTRACE_HOOKS I get the following with
>> either of those dtrace probes on the DOMU:
>>
>> dtrace -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: {
>> printf("%d %d %d %d %d %d %d", arg0, arg1, arg2, arg3, arg4, arg5, arg6,
>> arg7) }'
>> dtrace: invalid probe specifier sdt:xen:clock:, sdt:xen:hardclock:,
>> sdt:xen:timecounter: { printf("%d %d %d %d %d %d %d", arg0, arg1, arg2,
>> arg3, arg4, arg5, arg6, arg7) }: "/usr/lib/dtrace/psinfo.d", line 46: syntax
>> error near "u_int"
>
> Can you please file a PR about that, and pass `-x nolibs' to dtrace in
> the mean time?
>
> dtrace -x nolibs -n 'sdt:xen:clock:...'
[trying again... the attachments were apparently too large]
Some initial results...
I set up a -current DOMU running in normal PV mode. The userland is
9.x and the kernel is a custom build adding KDTRACE_HOOKS but is
otherwise XEN3_DOMU.
I used this probe:
dtrace -x nolibs -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: {
printf("%d %d %d %d %d %d %d %d", arg0, arg1, arg2, arg3, arg4, arg5, arg6,
arg7) }'
Note that the initial probe I was presented with either was missing a %d
or had too many arguments. The above balances out the %d and argument
in what I suspect should be the proper way.
This first result is:
http://anduin.eldar.org/~brad/dtrace_neg/d2_idle.out
This is the above dtrace probe running on the system when it is mostly
idle.
The second result is:
http://anduin.eldar.org/~brad/dtrace_neg/d2_working.out
This is the dtrace probe running when the system was made to be busy
doing three world builds of the NetBSD tree. Each build was allowed to
use 2 processors. If you look at the xentop output on the DOM0 it will
be noted that the guest is almost always 199% or 200% busy as was
intended. The system was rebooted and the artifact area for the builds
cleared out before this run.
There is a third result that was generated to get the dtrace working.
The system was busy doing builds at the time, but probably had been
doing that longer than the "working" result mention above. This initial
result caught some hardclock miss events that I have not seen again
(although apparently vmstat -e did). This result has only 6 of the 7
arguments in it as I was trying to work out what was wrong with the
probe.
http://anduin.eldar.org/~brad/dtrace_neg/d2_initial.out
At no time, so far, has the system reported the 'WARNING: negative
runtime...' message. With a 9.x kernel, I can usual get though a small
number of builds and when the system is idle after that, the message is
generated.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
>> Date: Thu, 20 Jul 2023 21:50:26 -0400
>> From: Brad Spencer
>>
>> With a DOMU kernel compiled with KDTRACE_HOOKS I get the following with
>> either of those dtrace probes on the DOMU:
>>
>> dtrace -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: {
>> printf("%d %d %d %d %d %d %d", arg0, arg1, arg2, arg3, arg4, arg5, arg6,
>> arg7) }'
>> dtrace: invalid probe specifier sdt:xen:clock:, sdt:xen:hardclock:,
>> sdt:xen:timecounter: { printf("%d %d %d %d %d %d %d", arg0, arg1, arg2,
>> arg3, arg4, arg5, arg6, arg7) }: "/usr/lib/dtrace/psinfo.d", line 46: syntax
>> error near "u_int"
>
> Can you please file a PR about that, and pass `-x nolibs' to dtrace in
> the mean time?
>
> dtrace -x nolibs -n 'sdt:xen:clock:...'
Sorry ... ment to include this too:
The "vmstat -e | grep -e tsc -e systime -e hardclock" during the time
the system is working on doing builds:
vcpu0 missed hardclock 20 intr
vcpu1 missed hardclock 790 intr
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
>> Date: Thu, 20 Jul 2023 21:50:26 -0400
>> From: Brad Spencer
>>
>> With a DOMU kernel compiled with KDTRACE_HOOKS I get the following with
>> either of those dtrace probes on the DOMU:
>>
>> dtrace -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: {
>> printf("%d %d %d %d %d %d %d", arg0, arg1, arg2, arg3, arg4, arg5, arg6,
>> arg7) }'
>> dtrace: invalid probe specifier sdt:xen:clock:, sdt:xen:hardclock:,
>> sdt:xen:timecounter: { printf("%d %d %d %d %d %d %d", arg0, arg1, arg2,
>> arg3, arg4, arg5, arg6, arg7) }: "/usr/lib/dtrace/psinfo.d", line 46: syntax
>> error near "u_int"
>
> Can you please file a PR about that, and pass `-x nolibs' to dtrace in
> the mean time?
>
> dtrace -x nolibs -n 'sdt:xen:clock:...'
Ya, a PR can be filed. Even when I did Solaris / SmartOS for $DAYJOB I
never really used dtrace all that often. I see it is very powerful, but
my paths never crossed much with it.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
>> Date: Sat, 08 Jul 2023 14:34:56 -0400
>> From: Brad Spencer
>>
>> Taylor R Campbell writes:
>>
>> > Can you either:
>>
>> Yes, I can perform as much of this as needed after I get some other
>> stuff in life dealt with more towards the end of the month. I really
>> won't have any time before then.
>
> No worries!
I looking at this problem again having returned from my road trip. I
built a -current from 2023-07-20 and am working with that.
>> > 1. share the output of `vmstat -e | grep -e tsc -e systime -e
>> >hardclock' after you get the console warning;
>>
>> The DOMU currently only has 1 vcpu, but here is the output now:
>>
>> vcpu0 raw systime went backwards 465790 intr
>>
>> When I have real time later I will force the negative runtime to happen
>> and run the above again.
>
> This is evidence that the hypervisor is doing something wrong with the
> clock it exposes to the guest. However, on a single-vCPU system, we
> work around this by noting the last Xen systime recorded on the
> current vCPU, and pretending the clock just hadn't changed since then.
>
> On a multi-vCPU system, we also try to work around it by recording a
> clock skew in xen_global_systime_ns and applying it to ensure the
> timestamp is monotonic, but perhaps that's not working right -- or
> perhaps it is working for 64-bit timestamps, but the jumps are so
> large that they wrap around the 32-bit timecounter arithmetic.
>
>> > 2. run
>> >
>> >dtrace -n 'sdt:xen:clock: { printf("%d %d %d %d %d %d %d",
>> >arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7) }'
>
> Note: this should now be
>
> dtrace -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: {
> printf("%d %d %d %d %d %d %d", arg0, arg1, arg2, arg3, arg4, arg5, arg6,
> arg7) }'
With a DOMU kernel compiled with KDTRACE_HOOKS I get the following with
either of those dtrace probes on the DOMU:
dtrace -n 'sdt:xen:clock:, sdt:xen:hardclock:, sdt:xen:timecounter: {
printf("%d %d %d %d %d %d %d", arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7)
}'
dtrace: invalid probe specifier sdt:xen:clock:, sdt:xen:hardclock:,
sdt:xen:timecounter: { printf("%d %d %d %d %d %d %d", arg0, arg1, arg2, arg3,
arg4, arg5, arg6, arg7) }: "/usr/lib/dtrace/psinfo.d", line 46: syntax error
near "u_int"
Now, this would have been with a 9.x userland and a -current (10.99.6)
kernel and modules from 2023-07-20. The -current dtrace binary did the
same thing, but I didn't replace the libraries.
>> >on the system, and leave it running with output directed to a file,
>> >and share the output when you see the console warning; or
>>
>> The DOMU is a 9.3_STABLE from around November 8th and when I attempted
>> to run the above dtrace it didn't work. I got this in the messsages:
>>
>> [ 1792486.921759] kobj_checksyms, 988: [dtrace]: linker error: symbol
>> `dtrace_invop_calltrap_addr' not found
>> [ 1792486.921759] kobj_checksyms, 988: [dtrace]: linker error: symbol
>> `dtrace_invop_jump_addr' not found
>> [ 1792486.921759] kobj_checksyms, 988: [dtrace]: linker error: symbol
>> `dtrace_trap_func' not found
>> [ 1792486.921759] WARNING: module error: unable to affix module `dtrace',
>> error 8
>
> Looks like nobody has wired up dtrace to Xen! That's a pretty serious
> regression of Xen vs native x86. Someone needs to hook these up.
Ya, this appears to just be KDTRACE_HOOKS, as mentioned in the other
email (I should have remembered this, as I had seen it before). That is
also needed to get the solaris module to load, which is required by the
zfs module. The effect right now is that zfs won't work out of the box
on a DOMU.
> In the mean time, I've add a little more diagnostics to HEAD -- if you
> can boot a current kernel, that might help, or I could try to make the
> corresponding changes on netbsd-9.
>
> https://mail-index.netbsd.org/source-changes/2023/07/13/msg145973.html
> https://mail-index.netbsd.org/source-changes/2023/07/13/msg145974.html
I am current trying to abuse the system to make the negative runtime
thing happen. Usually this has occured a day or so after building the
world for a couple of different system types. We will see if it still
triggers.
If you think I should have to replace userland DOMU, I can probably do
that too. Updaing the DOM0 kernel is a bit more involved but can also
happen with some planning.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
Taylor R Campbell writes:
>> Date: Wed, 04 Jan 2023 14:43:25 -0500
>> From: Brad Spencer
>>
>> So... I have a PV+PVSHIM DOMU running a pretty recent 9.x on a DOM0
>> running a 9.99.xx kernel. The DOM0 is not large, a 4 processor E-2224
>> with 32GB of memory. The DOMU has 2 VCPUs and 8GB of memory. About
>> every day a very particular DOMU tosses the:
>>
>> WARNING: negative runtime; monotonic clock has gone backwards
>
> Does this still happen?
As far as I know it still happens. I ended up running just a single CPU
much of the time on the DOMU, until the system needed more and then I
reboot the DOMU with two vcpus. The negative runtime does not happen
when there is only one vcpu on the DOMU.
When I forget to reboot back to one CPU in the DOMU, the negative
runtime message has always happened within a couple of days.
> Can you either:
Yes, I can perform as much of this as needed after I get some other
stuff in life dealt with more towards the end of the month. I really
won't have any time before then.
> 1. share the output of `vmstat -e | grep -e tsc -e systime -e
>hardclock' after you get the console warning;
The DOMU currently only has 1 vcpu, but here is the output now:
vcpu0 raw systime went backwards 465790 intr
When I have real time later I will force the negative runtime to happen
and run the above again.
> 2. run
>
>dtrace -n 'sdt:xen:clock: { printf("%d %d %d %d %d %d %d",
>arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7) }'
>
>on the system, and leave it running with output directed to a file,
>and share the output when you see the console warning; or
The DOMU is a 9.3_STABLE from around November 8th and when I attempted
to run the above dtrace it didn't work. I got this in the messsages:
[ 1792486.921759] kobj_checksyms, 988: [dtrace]: linker error: symbol
`dtrace_invop_calltrap_addr' not found
[ 1792486.921759] kobj_checksyms, 988: [dtrace]: linker error: symbol
`dtrace_invop_jump_addr' not found
[ 1792486.921759] kobj_checksyms, 988: [dtrace]: linker error: symbol
`dtrace_trap_func' not found
[ 1792486.921759] WARNING: module error: unable to affix module `dtrace', error
8
When I have time to get to this I can build a newer 9.x world, I just
need to know if I need to do that.
> 3. put `#define XEN_CLOCK_DEBUG 1' in sys/arch/xen/xen/xen_clock.c and
>build a new kernel, and share the dmesg output when you get the
>console warning?
>
> This should tell us whether it's the Xen host's fault or something
> wrong in NetBSD.
Some more data points from the vmstat command mentioned above (because
it is simple and quick to run):
1) Another system with the same generation 9.x BUT on a different DOM0
produces:
vcpu0 raw systime went backwards 329090 intr
This second system has also been known to have negative runtime and is
also currently running with one vcpu.
2) On the same DOM0 as the one mentioned in #1 there is a 10.x_BETA from
January 22nd. This guest is a PVH DOMU and the above vmstat produces no
output. This full PVH DOMU has two vcpus and I have never known it to
produce negative runtime.
3) A third 9.x DOMU that is just a normal PV (no PVSHIM involved)
produces the following with that vmstat command:
vcpu0 raw systime went backwards 1415320 intr
vcpu0 missed hardclock70 intr
I have never known this system to have negative runtime.
4) Another 10.x_BETA DOMU PV guest with 1 vcpu from April 22nd on the
same DOM0 as #3 does not produce any output from the vmstat command.
Thanks for asking about this... it is more than a little annoying. I
apologize that I won't be able to be very receptive to doing much more
with this until later.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: /usr/bin/ftp crash on -current (10.00.4) aarch64
Chavdar Ivanov writes: > On Thu, 18 May 2023 at 11:31, Robert Swindells wrote: >> >> >> Chavdar Ivanov wrote: >> > The weird and suspicious thing is that /usr/bin/ftp is linked to both >> > existing libcrypto.so versions: >> > >> > ldd /usr/bin/ftp >> > /usr/bin/ftp: >> >-ledit.3 => /usr/lib/libedit.so.3 >> >-lterminfo.2 => /usr/lib/libterminfo.so.2 >> >-lc.12 => /usr/lib/libc.so.12 >> >-lssl.15 => /usr/lib/libssl.so.15 >> >-lcrypto.14 => /usr/lib/libcrypto.so.14 >> >-lcrypt.1 => /lib/libcrypt.so.1 >> >-lcrypto.15 => /usr/lib/libcrypto.so.15 >> >> I'm guessing you did an update build not a clean one. Also, did you use >> the -j flag to build in parallel? > > I indeed usually do update builds; in this case however, I had deleted > the entire obj (it is on a zfs, removing takes a lot of time, so > subsequently I replaced it with a dedicated zfs so that I can just > destroy and recreate it...). I also 'make cleandir' in [x]src in > advance; that usually is enough. Snapshot the zfs fileset when it is empty and rollback when you want to make it empty again. Very quick works best when obj is its own fileset. >> >> Do a clean build and everything should work. > > That's next. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: ipmi0: incorrect critical max
Stephen Borrill writes: > On an HP Microserver Gen10 Plus, I found that soon after booting, I get > the following alert: > > ipmi0: critical over limit on '11-LOM-CORE' > > If powerd is running (the default), it shuts the machine down (so > basically as soon as it hits multi-user). > > envstat shows that CritMax is zero: > >Current CritMax WarnMax WarnMin CritMin Unit > [ipmi0] > 11-LOM-CORE:59.2530.000 110.471degC > > Seen on 9.3_STABLE, but also in 10 BETA. > > I suppose one simple fix would be to ensure that if CritMax is lower > than WarnMax, it should be set to the value of WarnMax. > > Any other things to look at? The machine won't be put into production for > a few days, so it's good time to experiment > > I have put the latest BIOS on the machine If that server has a independent out of band "system" in it, a BMC with a command line interface or web browser, I would get into that and see if it reports the sensors there just to see if the ipmi driver pulls them correctly. The BMC may not have a way to specifiy or tell you what the Crit and Warn values are, but it would be worth looking around for that too. Failing any of that, I think you should be able to set what NetBSD thinks the CritMax is in /etc/envsys.conf. See envsys.conf(5) for details. I have a ASrockRack board that doesn't report one of the sensors correctly and/or the APMI driver doesn't pull it correctly. It is a fixed values that never changes for nothing... -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: About kern/57136, panic assertion, probably a diagnotic panic
Brian Buhrow writes: > hello Brad. In reading about your panics on day 6 of uptime, I wonder > if the issue might > be related to memory allocation? Specifically, by day 6, I expect that > memory allocations in > the system are pretty fragmented and thus more CPU time is spent doing things > like cleaning > memory, freeing memory and, perhaps, paging. What happens if you give the > domu in question > more or less memory in terms of its stability? I'm guessing less memory > makes it panic faster, > more memory makes it stay up longer. Of course, this is pure speculation on > my part and I hve > no idea what's going on. :) > -Brian I suspected that and the DOMU has moved from 8GB to the current 12GB (living at 10GB for while) and it really didn't seem to make any notable difference. The panics appeared to happen at pretty much the same rate. I can't discount memory fragmentation, but I can't really say that is much of a factor either. The system really should not need anything like 12GB, and in fact top says that almost 9GB is free at this point. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
About kern/57136, panic assertion, probably a diagnotic panic
gnats-ad...@netbsd.org writes: I wrote this PR a little while ago. kern/57136: NPF panic probably on a NPF table list call >>Category: kern >>Responsible:kern-bug-people >>Synopsis: NPF panic probably on a NPF table list call >>Arrival-Date: Fri Dec 23 20:20:00 + 2022 I was finally able to catch more of the panic message. It appears to be a diagnostic assertion in kern_synch.c. Copied from an image this is the assert: panic: kernel diagnostic assertion "ci->ci_mtx_count == -1" failed: file "../../../../kern/kern_synch.c", line 726 mi_switch: cpu0: ci_mtx_count (-2) != -1 (block with spin-mutex held) (following that is the panic I copied into the PR) The system is a pure PVH DOMU running a 10.0_BETA mostly GENERIC kernel with 2 VCPUS and 12GB of memory. What was going on at the time was a tar-pipe copy "tar -cf - . | (cd /someplace;tar -xvf -)" and probably a pretty large network burst of activity from another system on the lan into the DOMU (and a bunch of other stuff, like being a router and firewall). The trigger of the panic is as mentioned in the PR, a "npfctl table sometable list" was performed from a cron job, but that wasn't where the actual panic happened, it seems. ... so lots of Xen guest disk and some Xen network activity was present. I have other DOMUs that perform the same npfctl actions just as often and they never panic... but they also only have 1 VCPU. The DOMU that panics is the only one with more than 1 VCPU. The problem does seem to get worse, in the sense that the panic happens more often, if 3 VCPUs are given to the DOMU. As it is right now, the panic can't exactly be reproduced upon demand, but pretty much WILL happen at day 6 or so of uptime if some sort of activity ties up the CPUS (this may be a bit subjective on my part, but it seems like the system can handle activity better in days 1 to 5 and only around day 6 does the chances of the panic increase, but I may also be smoking something...). With 3 VCPUs, it was happening about every 30 hours. Any help or hints would be greatly appreciated. With some planning I can perform tests. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
What to do about "WARNING: negative runtime; monotonic clock has gone backwards"
So... I have a PV+PVSHIM DOMU running a pretty recent 9.x on a DOM0 running a 9.99.xx kernel. The DOM0 is not large, a 4 processor E-2224 with 32GB of memory. The DOMU has 2 VCPUs and 8GB of memory. About every day a very particular DOMU tosses the: WARNING: negative runtime; monotonic clock has gone backwards on its console and sort of falls of the network. A reboot "fixes" it, but nothing else does. The DOM0 is running a mostly stock XEN3_DOM0 kernel except that IFQ_MAXLEN is larger than the default and HZ is 2000 (which helped to keep the clocks on the DOMU syncing properly all other things being equal). The DOMU is running a pretty stock XEN3_DOMU, that is, HZ is 100 there. Over the years, this problem has been mentioned from time to time, but I was wondering if anyone has a actual solution to this sort of thing. The DOM0 hosts a small number of guests, all PV+PVSHIM, but this one in particular is the only one with 2 VCPUs, all the rest have only 1 and have never exhibited the problem of the clock going backwards. Nothing much is really going on with the DOMU right now, as it is one of my build guests. It is running ZFS and does backups each day and does a 'cvs update' sort of thing from time to time, but than that is it right now. Any thoughts?? -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Two small problems noticed with 10.0_BETA and probably -current
I run a lot of DOMUs, but this is the first time I noticed these particular problems. These probably effect -current too. 1) The stock XEN3_DOMU will fail to load the solaris module with this error: nbsd10test# modload solaris [ 927.7196996] kobj_checksyms, 1004: [solaris]: linker error: symbol `sdt_provider_sdt' not found [ 927.7196996] WARNING: module error: unable to affix module `solaris', error 8 modload: solaris: Exec format error This will result in the inability to load the zfs module which could be a suprise if you have zfs filesystems. If you compile a kernel with: options KDTRACE_HOOKS all will be well again, but I don't know what the right answer here should be so I hesitate to commit the simple change of adding that option. 2) It does not appear that a NetBSD x86_64 pure PVH guest using the GENERIC kernel can see more than 4GB of memory. I believe that this worked in the past, but I currently do not have very old 9.99.xx guests right now to try it on. That is, if you set memory= and/or maxmem= the guest will see exactly 4GB. If you make the guest a pvshim PVH guest and use XEN3_DOMU you get all of the memory asked for. This does not work at least from 9.99.104. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: npf table loads slow??
nia writes: > There is a PR for bad behavior of NPF with large tables: > > http://gnats.netbsd.org/55892 Thanks ya, it is simular to that... I let the table replace run and after 65 minutes, npfctl exited with: # /usr/bin/time npfctl table blocklist replace /etc/blocklist npfctl: npf_table_replace(): Argument list too long 3924.20 real 3820.09 user 0.76 sys so at least it wasn't garbage, but it wasn't useful either. This is a 9.3_STABLE built a short time ago. Since this is just a test, I am going to try a loop against the list adding the ip addresses one at a time to see if that actually works. Clearly large full table loads will not work right now. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
npf table loads slow??
So I am experimenting with NPF, converting a small rule set from
IPF+IPPOOL to NPF with its tables.
The IPPOOL is a hash table with a bit more than 20 ip addresses in
it. This is auto generated from bad behavior. Loading this with
ippool(8) takes just a couple of seconds. Doing the same thing with
"npfctl table reload ..." takes a very long time (unknown at this point,
but maybe 1/2 hour at least) and drives the CPU to 100% while doing that
load.
Am I doing something obviously wrong?? The /etc/npf.conf file is
nothing more than this:
table type ipset
procedure "log" {
log: npflog0
}
group default {
pass in all
pass out all
block in from apply "log"
block out to apply "log"
}
If a file is used in /etc/npf.conf for the blocklist the start up
(i.e. npfctl reload) hangs up driving the CPU to 100%, probably loading
the large table...
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Any TCP VTW users?
Ryota Ozaki writes: > Hi, > > Are there any users of TCP Vestigial Time-Wait (VTW)? > The feature is disabled by default and we need to explicitly > enable via sysctl to use it. > > I just want to know if we should still maintain it. > > ozaki-r I do use it on one system, but it is not likely to be critical to anything. The system in question creates a whole ton of short lived connections and I think I was trying to get them to expire quicker then normal. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: How to BIOS-boot from NVMe device?
Chavdar Ivanov writes: [snip] > > If anyone wants to play with UEFI booting and has access to a recent Xen > > DOM0 system you can install the pkgsrc/sysutils/ovmf package and point a > pkgsrc/sysutils/ovmf does not build on -current at least - and hasn't been > building for a long while. Hmm... unfortunate... it does build just fine on 9.2ish from 2022Q2 pkgsrc. > I checkout the source and build it manually under Debian; obviously the > resulting firmware files are the same. I didn't try doing that, but I did find that the version from Archlinux from their package manager won't work on a -current NetBSD kernel, which panics somehere in the ACPI code. -current, after removing PVH and PVHVM, worked fine on the version I built on 9.x. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: How to BIOS-boot from NVMe device?
Paul Goyette writes: > On Thu, 8 Sep 2022, Paul Goyette wrote: > >> Would be nice to get my menu back and have it default to the NetBSD >> system partition, but at least it boots! > > I got my menu back. Just had to put it at /efi/boot.cfg (ie, at the > root of the EFI partition). > > > ++--+--+ > | Paul Goyette | PGP Key fingerprint: | E-mail addresses:| > | (Retired) | FA29 0E3B 35AF E8AE 6651 | p...@whooppee.com| > | Software Developer | 0786 F758 55DE 53BA 7731 | pgoye...@netbsd.org | > | & Network Engineer | | pgoyett...@gmail.com | > ++--+--+ This is a clear indication of a UEFI boot. There is confusion in the docs about where the boot.cfg file should be located. I have also found that the desired behavior works if it is in the root of the EFI filesystem. If anyone wants to play with UEFI booting and has access to a recent Xen DOM0 system you can install the pkgsrc/sysutils/ovmf package and point a Xen HVM guest at that for its bios and get a more or less working UEFI setup without physical hardware (you will have to compile a custom kernel without PVM or PVHVM support). I posted a note on port-xen a bit ago about what works and what had trouble. You can probably do the same thing with qemu and/or nvmm but I have not messed with those. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: raspberry pi zero W serial port overlay fun
Patrick Welche writes:
[snip]
> - replace firmware:
>
> mount /dev/sd0e /mnt
> cd /mnt
> for sfx in elf bin dat; do
> rm *.${sfx}
> cp /tmp/firmware/boot/*.${sfx} .
> done;
> rm dtb/*
> cp /tmp/firmware/boot/bcm2708-rpi-zero-w.dtb dtb
>
> NB in rpi.img.gz was called: bcm2835-rpi-zero-w.dtb, bcm2708, yet
> dtc -I dtb -O dts ./boot/overlays/disable-bt.dtbo
> shows
[snip]
>
> Cheers,
>
> Patrick
Hello...
I really wish it was simpler to tell what was happening with this FDT
stuff... like really super verbose debugging messsages...
I have the same speed shifting problem on some "Raspberry Pi 3 Model B
Rev 1.2" that you have on the Zero[*]. A third method with the RPI3
that can be used to work around this problem is to set forced_turbo=1 in
/boot/config.txt.. you won't be able to step speed any more, but the
serial port console works.
In an effort to do better than forcing turbo on all of the time I did
the above replacement that your posted about on my RPI3 that exhibits
this speed shifting on the console and all I had to do was replace the
firmware with the upstream version. The only difference between what
you did and what I did was I included in /boot/dtb ALL of the dtb files
from upstream and did not include /boot/overlays at all. I can't tell
any differences in dmesg between the two firmware types but the one from
NetBSD clearly exhibits the speed shifting problem while upstream does
not have the problem and allows for the frequency to be changed. Adding
'dtdebug=1' to /boot/config.txt didn't add any additional messages as to
which DTB file was being used, if any... I assume that something got
loaded because if I empty out /boot/dtb completely, the system doesn't
boot at all, it seems.
I did other experiements ... and it seems that just replacing the
*.bin, *.elf and *.dat files and loaders is enough to "fix" this problem
on the RPI3.. that is the NetBSD dtb files (from -current 9.99.99, BTW)
worked fine with the upstream loaders and other files (I assume that is
what the *.bin, *.elf and *.dat stuff is mostly boot loaders).
Both -current and 9.2_STABLE have a working console.
[*] - A complete aside... I have a number of RPI3 systems and only some
of them exhibit this speed shifting... can't explain that as they claim
to all be the same version on the board and using the same SDCARD.
Literally using the same SDCARD on some has this problem and others do
not.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: "zfs send" freezes system (was: Re: pgdaemon high CPU consumption)
Matthias Petermann writes: [snip] > Roundabout one week after removing the patch, my system with ZFS is > behaving "normally" for the most part and the freezes have disappeared. > What is the recommended way given the 10 branch? If it is not > foreseeable that the basic problem can be solved shortly, would it also > be an option to withdraw the patch in the sources to get at least a > stable behavior? (Not only) on the sidelines, I would still be > interested in whether this "zfs send" problem occurs in general, or > whether certain hardware requirements have a favorable effect on it. > > Kind regards > Matthias My personal experience with this problem is with Xen PV/PVH guests on my build system(s), but I think others have experienced it with physical systems. The only particular facts that I have observed are: a) If there is more memory given to the guest they last longer. That is the one with 8GB does not have as much trouble as the one with 4GB. b) reducing the number of vnodes makes helps keep the system up. I usually run 4096 to 16384 kern.maxvnodes. Without this the OS build system can do about 1.5 "build.sh release" runs before it hangs and with a reduction in vnodes it can get away with 4 to 6 before there are problems. c) Seen only once and on -currentish (most systems are 9.2), but a zfs receive into a compressed fileset hung after a while (in the usual manor that I observe) even with the mentioned patch reverted and a reduced maxvnodes. Receiving into a non-compressed fileset worked as expected. The above a, b and c was 100% reproduceable for me, just takes a while as a build release isn't quite.. the zfs receive thing happened much faster. Both of my build systems uses zfs send through a ssh pipe to a file on another system as a backup method and both have compressed filesets, but they are rarely received into and never sent from those. I have had no trouble zfs sending and receiving locally from a non-compressed fileset into a compressed one, which makes that one case a little strange. On a -currentish test system with zfs filesets for object artifacts and build artifacts if I didn't revert the mentioned patch to arc.c the system could not make it though a build of qemu from pkgsrc and maybe not even though unpacking the source for building (sorry, don't exactly remember how far it would get). -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: iscsi target on a zfs zvol?
Brian Buhrow writes: > hello. Yes, I was vaguely aware of the lack of extended attributes for > NetBSD-Zfs, but > what I was suggesting was just using a flat file, exported via iscsi through > istgt or your > initiator of choice, on top of zfs, rather than a zvol, because you'll find > the read/write speed > to be so much faster. Unfortunately, it seems the upstream zfs maintainers > have decided that > zvols are not worth the time to optimize, so while they're functional, > they're not performant > under any openzfs-using implementation. This makes me sad because zvols are > such a tidy way to > manage so many different kinds of things. > > -thanks > -Brian I freely admit that I don't use zvols very much in NetBSD, but did you mess with the volblocksize any on the volume?? -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: pgdaemon high CPU consumption
Matthias Petermann writes: > Hello, > > On 01.07.22 12:48, Brad Spencer wrote: >> "J. Hannken-Illjes" writes: >> >>>> On 1. Jul 2022, at 07:55, Matthias Petermann wrote: >>>> >>>> Good day, >>>> >>>> since some time I noticed that on several of my systems with NetBSD/amd64 >>>> 9.99.97/98 after longer usage the kernel process pgdaemon completely >>>> claims a CPU core for itself, i.e. constantly consumes 100%. >>>> The affected systems do not have a shortage of RAM and the problem does >>>> not disappear even if all workloads are stopped, and thus no RAM is >>>> actually used by application processes. >>>> >>>> I noticed this especially in connection with accesses to the ZFS set up on >>>> the respective machines - for example after checkout from the local CVS >>>> relic hosted on ZFS. >>>> >>>> Is there already a known problem or what information would have to be >>>> collected to get to the bottom of this? >>>> >>>> I currently have such a case online, so I would be happy to pull >>>> diagnostic information this evening/afternoon. At the moment all info I >>>> have is from top. >>>> >>>> Normal view: >>>> >>>> ``` >>>> PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND >>>> 0 root 1260 0K 34M CPU/0 102:45 100% 100% >>>> [system] >>>> ``` >>>> >>>> Thread view: >>>> >>>> >>>> ``` >>>> PID LID USERNAME PRI STATE TIME WCPUCPU NAME COMMAND >>>> 0 173 root 126 CPU/1 96:57 98.93% 98.93% pgdaemon [system] >>>> ``` >>> >>> Looks a lot like kern/55707: ZFS seems to trigger a lot of xcalls >>> >>> Last action proposed was to back out the patch ... >>> >>> -- >>> J. Hannken-Illjes - hann...@mailbox.org >> >> >> Probably only a slightly related data point, but Ya, if you have a >> system / VM / Xen PV that does not have a whole lot of RAM and if you >> don't back out that patch your system will become unusable in a very >> short order if you do much at all with ZFS (tested with a recent >> -current building pkgsrc packages on a Xen PVHVM). The patch does fix a >> real bug, as NetBSD doesn't have the define that it uses, but the effect >> of running that code will be needed if you use ZFS at all on a "low" RAM >> system. I personally suspect that the ZFS ARC or some pool is allowed >> to consume nearly all available "something" (pools, RAM, etc..) without >> limit but have no specific proof (or there is a leak somewhere). I >> mostly run 9.x ZFS right now (which may have other problems), and have >> been setting maxvnodes way down for some time. If I don't do that the >> Xen PV will hang itself up after a couple of 'build.sh release' runs >> when the source and build artifacts are on ZFS filesets. > > Thanks for describing this use case. Apart from the fact that I don't > currently use Xen on the affected machine, it performs similiar > workload. I use it as pbulk builder with distfiles, build artifacts and > CVS / Git mirror stored on ZFS. The builders themself are located in > chroot sandboxes on FFS. Anyway, I can trigger the observations by doing > a NetBSD src checkout from ZFS backed CVS to the FFS partition. > > The maxvnodes trick first led to pgdaemon behave normal again, but the > system freezed shortly after with no further evidence. > > I am not sure if this thread is the right one for pointing this out, but > I experienced further issues with NetBSD current and ZFS when I tried to > perform a recursive "zfs send" of a particular snapshot of my data sets. > After it initially works, I see the system freeze after a couple of > seconds with no chance to recover (could not even enter the kernel > debugger). I will come back and need to prepare a dedicated test VM for > my cases. > > Kind regards > Matthias I saw something like that with a "zfs send..." and "zfs receive..." locking up just one time. I do that sort of thing fairly often to move filesets between one system and another and it has worked fine for me, except in one case... the destination was a NetBSD-current with a ZFS fileset set to use compression. The source is a FreeBSD with a ZFS fileset created in such a manor that NetBSD is happy with it and it also is set to use compression. No amount of messing around would let 'zfs send | ssh destination "zfs receive "' complete without locking up the destination. When I changed the destination to not use compression I was able to perform the zfs send / receive pipeline without any problems. The destination is a pretty recent -current Xen PVHVM guest and the source is a FreeBSD 12.1 (running minio to back up my Elasticsearch cluster). -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: pgdaemon high CPU consumption
"J. Hannken-Illjes" writes: >> On 1. Jul 2022, at 07:55, Matthias Petermann wrote: >> >> Good day, >> >> since some time I noticed that on several of my systems with NetBSD/amd64 >> 9.99.97/98 after longer usage the kernel process pgdaemon completely claims >> a CPU core for itself, i.e. constantly consumes 100%. >> The affected systems do not have a shortage of RAM and the problem does not >> disappear even if all workloads are stopped, and thus no RAM is actually >> used by application processes. >> >> I noticed this especially in connection with accesses to the ZFS set up on >> the respective machines - for example after checkout from the local CVS >> relic hosted on ZFS. >> >> Is there already a known problem or what information would have to be >> collected to get to the bottom of this? >> >> I currently have such a case online, so I would be happy to pull diagnostic >> information this evening/afternoon. At the moment all info I have is from >> top. >> >> Normal view: >> >> ``` >> PID USERNAME PRI NICE SIZE RES STATE TIME WCPUCPU COMMAND >>0 root 1260 0K 34M CPU/0 102:45 100% 100% [system] >> ``` >> >> Thread view: >> >> >> ``` >> PID LID USERNAME PRI STATE TIME WCPUCPU NAME COMMAND >>0 173 root 126 CPU/1 96:57 98.93% 98.93% pgdaemon [system] >> ``` > > Looks a lot like kern/55707: ZFS seems to trigger a lot of xcalls > > Last action proposed was to back out the patch ... > > -- > J. Hannken-Illjes - hann...@mailbox.org Probably only a slightly related data point, but Ya, if you have a system / VM / Xen PV that does not have a whole lot of RAM and if you don't back out that patch your system will become unusable in a very short order if you do much at all with ZFS (tested with a recent -current building pkgsrc packages on a Xen PVHVM). The patch does fix a real bug, as NetBSD doesn't have the define that it uses, but the effect of running that code will be needed if you use ZFS at all on a "low" RAM system. I personally suspect that the ZFS ARC or some pool is allowed to consume nearly all available "something" (pools, RAM, etc..) without limit but have no specific proof (or there is a leak somewhere). I mostly run 9.x ZFS right now (which may have other problems), and have been setting maxvnodes way down for some time. If I don't do that the Xen PV will hang itself up after a couple of 'build.sh release' runs when the source and build artifacts are on ZFS filesets. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: XEN devices included in kernel even if not XEN
Manuel Bouyer writes: > On Tue, Dec 21, 2021 at 07:44:59AM -0800, Paul Goyette wrote: >> I've noticed that device drivers listed in arch/xen/conf/files.xen >> (or, at least, most of those devices) seem to be included in kernel >> even if not using XEN. Shouldn't all those devices be conditional? >> >> # sysctl -a | grep driver | tr ',' '\n' | grep 'x[be]*' >> ... >> [141 -1 xenevt] >> [142 142 xbd] >> [143 -1 xencons] > > I think this lists all the known major numbers for the $MACHINE, I don't think > it means that the driver is actually loaded. ... and a PVHVM guest can use the GENERIC kernel, but will want the Xen devices too. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: backward compatibility: how far can it reasonably go?
Jason Thorpe writes:
>> On Dec 8, 2021, at 10:52 AM, Greg A. Woods wrote:
>>
>> That's one bullet I've dodged entirely already since my oldest systems
>> are running netbsd-5 stable. (Though in theory isn't there supposed to
>> be COMPAT support for SA?)
>
> int
> compat_60_sys_sa_register(lwp_t *l,
> const struct compat_60_sys_sa_register_args *uap,
> register_t *retval)
> {
> return sys_nosys(l, uap, retval);
> }
>
> SA is one of those things that's REALLY hard to provide compatibility for.
>
> -- thorpej
Ya, that... I remember a diff that is probably sitting in a mailing
list message somewhere that provided an emulation I think at the lib
layer for this. However, in the case I mentioned, the lack of support
was not a show stopper. It really was the only thing that caused any
problems at all. Even a 7.x kernel, as an initial test, booted with a
4.x userland didn't really have any real trouble in that it ran all of
the /etc/rc stuff mostly just fine and I could log in. I would suspect
that going from 5.x forward would not be just too much of a problem.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: backward compatibility: how far can it reasonably go?
Brad Spencer writes: > You can also shut the DOMU down and present the disks to another DOMU in > real time using "xl block-attach " just use the "r" device if you > use devices or the file otherwise. I have done this for many years with > PV DOMUs on many different Xen versions. Once you have the disks > presented, mostly I just "tar -xpf " the sets or my own build of > userland on top of the old system. Use postinstall after that usually > once the system is booted up again with the newer kernel and newer > userland. You will need the etc sets for postinstall, but nothing has > been scary there. Messed that statement up... use the Not "r" device, the block device not the character device. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: backward compatibility: how far can it reasonably go?
"Greg A. Woods" writes: > At Tue, 7 Dec 2021 20:37:26 -0800 (PST), Paul Goyette > wrote: > Subject: Re: backward compatibility: how far can it reasonably go? >> >> Without looking at the details of your backtrace, the issue with >> ifconfig(8) could be related to PRs kern/54150 and/or kern/54151. > > Aw, damn, my memory is too short! Thanks for reminding me of those! > > The kernel crash was IPF-related, and in my test back then I was testing > on an i386 machine, which at the time did not, IIRC (and we know what > that might mean), was not running IPF. > > Anyway, the two machines I'm upgrading do need to run IPF, at least > until they are running a new OS with new pkgs. > > I'm beginning to think the only way to avoid that rabbit hole in order > to get these upgrades done in the next week will be to shut them down > and do the upgrades by mounting their filesystems in their dom0s. > > -- > Greg A. Woods > > Kelowna, BC +1 250 762-7675 RoboHack > Planix, Inc. Avoncote Farms You can also shut the DOMU down and present the disks to another DOMU in real time using "xl block-attach " just use the "r" device if you use devices or the file otherwise. I have done this for many years with PV DOMUs on many different Xen versions. Once you have the disks presented, mostly I just "tar -xpf " the sets or my own build of userland on top of the old system. Use postinstall after that usually once the system is booted up again with the newer kernel and newer userland. You will need the etc sets for postinstall, but nothing has been scary there. Although I have never seen this problem, there may be trouble if the block size that the DOMU sees of the disks is different from what DOM0 sees. There have been comments about this from time to time. I have also done other variations of this using a copy of /rescue/tar to extract the new userland onto a live DOMU system usually from the Xen console in single user mode. This has also generally worked out just fine and I use that in the case where I have made /usr a LVM disk inside of the DOMU and it is a pain to present that to another DOMU. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: backward compatibility: how far can it reasonably go?
"Greg A. Woods" writes: > So I've got a couple of old but important machines (Xen amd64 domUs) > running NetBSD-5, and I've finally decided that I'm reasonably well > enough prepared to try upgrading them. > > However it seems a "modern" (9.99.81, -current from about 2021-03-10) > kernel with COMPAT_40 isn't able to run some of the userland on those > systems. > > Is this something that should work? > > If it should I think it would make the upgrade much easier as I could > then plop down the new userland and run etcupdate. (there are of course > alternative ways to do the upgrade, eased by the fact they are domUs (*)) > > The most immediate problems I noticed are with networking. ifconfig -a > returns without printing anything, and trying to enable IPF crashes: When I took a system from 4.0 to 7.x some time ago, the only thing that I had problems with was anything that used scheduler activations since that had been removed. For me this only effected stuff from pkgsrc, as I also rolled in new userland at the same time. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: -current panic on boot on RPI3
Michael Cheponis writes: > Although likely a long-shot: can you try on a different RPi3? > > I have 3 of them, and I want all machines to boot from USB - but with even > low-level FW updated, only 2 of them work that way (they all should work > that way). I want to make sure I have 2 spares on hand for my application. > -- > Also, I keep one RPi3 up, and I run an ancient kernel (-current .17) > because I was unsuccessful with any new kernels, so I stopped building for > that machine. Some would run for a few weeks before crashing; but this > machine must work 24/7 (I should probably run 9.2). I stopped trying new > kernels maybe six months ago. I suppose I should try again with a newer > kernel. I was several RPIs that run 24x7x365 but with nothing later than 9.x_STABLE... I do have several spare RPI3 and will certainly try one or more of them.. I can also build a newer tree from -current. I was working on device drivers in -current when I noted the mentioned panic. > On Mon, Oct 18, 2021 at 3:42 AM Brad Spencer wrote: > >> Kimmo Suominen writes: >> >> > On Fri, Oct 01, 2021 at 05:03:01PM -0400, Brad Spencer wrote: >> >> >> >> I did a build of -current from source taken on 2021-09-30 and had it >> >> panic on my RPI 3. >> >> >> >> Just wondering if this a known problem. The panic is as follows: >> >> >> >> [ 1.2878232] kernel diagnostic assertion "l->l_stat == LSONPROC" >> failed: file "/usr/src/sys/kern/kern_sleepq.c", line 227 >> >> [ 1.2978253] cpu1: Begin traceback... >> >> [ 1.3478291] netbsd:vpanic+0x16c >> >> [ 1.3978354] netbsd:cv_enter+0xfc >> >> [ 1.4078352] netbsd:cv_wait+0x44 >> >> [ 1.4078352] 0x809aeeac: netbsd:xc_wait+0xa0 >> >> [ 1.4178637] 0x809aeee4: netbsd:pic_establish_intr+0x2a0 >> >> [ 1.4178637] 0x809aef14: netbsd:intr_establish_xname+0xa4 >> >> [ 1.4278385] 0x809aef2c: netbsd:intr_establish+0x20 >> >> [ 1.4278385] 0x809aef54: netbsd:bcm2836mp_intr_init+0x4c >> >> [ 1.4378399] 0x809aef6c: netbsd:arm_fdt_cpu_hatch+0x34 >> >> [ 1.4378399] 0x809aef94: netbsd:cpu_hatch+0x1bc >> >> [ 1.4478405] cpu1: End traceback... >> > >> > This looks like it could be related to PR 56264: >> > https://gnats.netbsd.org/56264 >> > >> > There is a workaround listed in that ticket, but you should already have >> > it as it was committed on 2021-09-01. But it might be worth checking if >> > you have src/sys/arch/arm/broadcom/bcm2835_intr.c r1.39 or later. >> > >> > Cheers, >> > + Kimmo >> >> Thanks... I have version 1.41 of bcm2835_intr.c. I noted that that the >> panic occurs with evbarm6, evbarm7 and aarch64 on the RPI3. I also >> found that compiling a kernel with only single processor support >> prevents the panic (disable MULTIPROCESSOR). I won't be able to test >> further for a couple of days, but I can try any suggestions anyone might >> have when I become more available again. Running SP is a little >> annoying and probably not reasonable for the long term unless the >> general thought is that the RPI3 is too long in the tooth to be well >> supported. I still suspect that there are a lot of them out there, but >> that is mostly just my opinion. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: -current panic on boot on RPI3
Kimmo Suominen writes: > On Fri, Oct 01, 2021 at 05:03:01PM -0400, Brad Spencer wrote: >> >> I did a build of -current from source taken on 2021-09-30 and had it >> panic on my RPI 3. >> >> Just wondering if this a known problem. The panic is as follows: >> >> [ 1.2878232] kernel diagnostic assertion "l->l_stat == LSONPROC" failed: >> file "/usr/src/sys/kern/kern_sleepq.c", line 227 >> [ 1.2978253] cpu1: Begin traceback... >> [ 1.3478291] netbsd:vpanic+0x16c >> [ 1.3978354] netbsd:cv_enter+0xfc >> [ 1.4078352] netbsd:cv_wait+0x44 >> [ 1.4078352] 0x809aeeac: netbsd:xc_wait+0xa0 >> [ 1.4178637] 0x809aeee4: netbsd:pic_establish_intr+0x2a0 >> [ 1.4178637] 0x809aef14: netbsd:intr_establish_xname+0xa4 >> [ 1.4278385] 0x809aef2c: netbsd:intr_establish+0x20 >> [ 1.4278385] 0x809aef54: netbsd:bcm2836mp_intr_init+0x4c >> [ 1.4378399] 0x809aef6c: netbsd:arm_fdt_cpu_hatch+0x34 >> [ 1.4378399] 0x809aef94: netbsd:cpu_hatch+0x1bc >> [ 1.4478405] cpu1: End traceback... > > This looks like it could be related to PR 56264: > https://gnats.netbsd.org/56264 > > There is a workaround listed in that ticket, but you should already have > it as it was committed on 2021-09-01. But it might be worth checking if > you have src/sys/arch/arm/broadcom/bcm2835_intr.c r1.39 or later. > > Cheers, > + Kimmo Thanks... I have version 1.41 of bcm2835_intr.c. I noted that that the panic occurs with evbarm6, evbarm7 and aarch64 on the RPI3. I also found that compiling a kernel with only single processor support prevents the panic (disable MULTIPROCESSOR). I won't be able to test further for a couple of days, but I can try any suggestions anyone might have when I become more available again. Running SP is a little annoying and probably not reasonable for the long term unless the general thought is that the RPI3 is too long in the tooth to be well supported. I still suspect that there are a lot of them out there, but that is mostly just my opinion. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
-current panic on boot on RPI3
[ 1.030] vchiq0 at simplebus1: BCM2835 VCHIQ
[ 1.030] /arm-pmu at simplebus0 not configured
[ 1.030] gpioleds0 at simplebus0:
[ 1.030] bcmrng0 at simplebus1: RNG
[ 1.030] entropy: ready
[ 1.030] cpu1: 600 MHz Cortex-A53 r0p4 (Cortex V8A core)
[ 1.030] cpu1: DC enabled IC enabled WB enabled EABT branch prediction
enabled
[ 1.2578383] cpu1: L1 32KB/64B 2-way (256 set) VIPT Instruction cache
[ 1.2578383] cpu1: L1 32KB/64B 4-way (128 set) write-back-locking-C PIPT Data
cache
[ 1.2678215] cpu1: L2 512KB/64B 16-way (512 set) write-through PIPT Unified
cache
[ 1.2778233] vfp1 at cpu1: NEON MPE (VFP 3.0+), rounding, NaN propagation,
denormals
[ 1.2878232] cpu3panic: : 600 MHz Cortex-A53 r0p4 (Cortex V8A core)
[ 1.2878232] kernel diagnostic assertion "l->l_stat == LSONPROC" failed: file
"/usr/src/sys/kern/kern_sleepq.c", line 227
[ 1.2978253] cpu1: Begin traceback...
[ 1.3078253] cpu3:0x809aedbc: DC enabled IC enabled WB enabled EABT branch
prediction enabled
[ 1.3178285] netbsd:db_paniccpu3: L1 32KB/64B 2-way (256 set) VIPT
Instruction cache
[ 1.3178285] cpu3: L1 32KB/64B 4-way (128 set) write-back-locking-C PIPT Data
cache
[ 1.3278278] cpu3: L2 512KB/64B 16-way (512 set) write-through PIPT Unified
cache
[ 1.3378291] +0x14
[ 1.3378291] 0x809aedd4: vfp3 at cpu3: NEON MPE (VFP 3.0+), rounding, NaN
propagation, denormals
[ 1.3478291] netbsd:vpanic+0x16c
[ 1.3478291] 0x809aedec: cpu2: 600 MHz Cortex-A53 r0p4 (Cortex V8A core)
[ 1.3578337] netbsd:__aeabi_uldivmodcpu2: DC enabled IC enabled WB enabled
EABT
[ 1.3678311] 0x809aee2c: branch prediction enabled
[ 1.3678311] netbsd:sleepq_enqueuecpu2: L1 32KB/64B 2-way (256 set) VIPT
Instruction cache
[ 1.3778336] +0x1a0
[ 1.3778336] 0x809aee5c: cpu2: L1 32KB/64B 4-way (128 set)
write-back-locking-C PIPT Data cache
[ 1.3878332] cpu2: L2 512KB/64B 16-way (512 set) write-through PIPT Unified
cache
[ 1.3978354] netbsd:cv_enter+0xfc
[ 1.3978354] 0x809aee84: vfp2 at cpu2: NEON MPE (VFP 3.0+), rounding, NaN
propagation, denormals
[ 1.4078352] netbsd:cv_wait+0x44
[ 1.4078352] 0x809aeeac: netbsd:xc_wait+0xa0
[ 1.4178637] 0x809aeee4: netbsd:pic_establish_intr+0x2a0
[ 1.4178637] 0x809aef14: netbsd:intr_establish_xname+0xa4
[ 1.4278385] 0x809aef2c: netbsd:intr_establish+0x20
[ 1.4278385] 0x809aef54: netbsd:bcm2836mp_intr_init+0x4c
[ 1.4378399] 0x809aef6c: netbsd:arm_fdt_cpu_hatch+0x34
[ 1.4378399] 0x809aef94: netbsd:cpu_hatch+0x1bc
[ 1.4478405] cpu1: End traceback...
Stopped in pid 0.16 (system) at netbsd:cpu_Debugger+0x4:bx r14
db{1}>
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
SPI on RPI?
I was wondering if anyone is using the built in spibus on a Raspberry PI (any version) lately. It is possible that I have outdated dts files, but I do not see the spibus device on my 9.x installs and was curious if it shows up for anyone in 9.x or -current and if anyone is currently using it. It seems like the dts files that are in the tree would contain a description for the SPI bus and the driver appears to have been modified for FDT support. Running ofctl appears to show a number of /spi@ lines but I don't know exactly what they indicate. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: ZFS quota performance issues?
r...@netbsd.org writes: > Sorry if this has already been discussed- > > I've been using ZFS on current happily for a few weeks now, and apart > from a little weirdness setting it up (wedge changing out from under me; > I think I've got that sorted now) it's been performing well. > > > However, I just put a quota on a filesystem I created, and any access > (ls, etc) to that file system causes the process to hang in zfscv for > many seconds. The operation eventually completes, but it doesn't seem > to be a caching issue, because another ls on the same directory makes it > happen reliably. When I remove the quota, all is well. > > > Known issue? Should I pr? This is in 9.99.87 circa end of July... > > +j Could you be specific by what you mean by "quota"? Just to be clear about it all... I use a number of filesystems that have the quota attribute (zfs get quota / zfs set quota=something ) set and do not observe this. However, there are user quotas too and I don't use them. I also am using ZFS with 9.x mostly and not -current. The latest build I have is 9.99.74 and also does not appear to have this problem. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: running xen on current
Manuel Bouyer writes: > On Thu, Apr 15, 2021 at 09:53:50AM +0100, Patrick Welche wrote: >> I have tried and failed to run xen on 3 -current/amd64 systems with >> 3 different failure modes: >> >> 1) laptop: xen.gz Building a PV Dom0 / ELF: not an ELF binary -> >> panic/reboot >> 2) desktop: XEN3_DOM0 panics including PR port-xen/55978 >> 3) server: Trampoline space cannot be allocated; will try fallback -> reboot >> >> They are all working NetBSD-current/amd64 systems. >> >> My conclusion was that xen is hopelessly broken, so was quite surprised >> by Greg Wood's thread about the finer points of running a guest OS, given >> that those systems won't even start the host OS. >> >> I dug out an old desktop, and to my pleasant surprise it booted XEN3_DOM0, >> and I have managed to run some XEN3_DOMUs. >> >> The difference between the working/broken setups seems to be that the >> working one is "BIOS" booting rather than EFI booting. >> >> Among all your xen success stories, are any of you EFI booting? > > AFAIK EFI is not yet supported by Xen (maybe this is supported by 4.15, > I've not had a chance to try yet). I have it running on fairly recent > Dell servers (in BIOS mode) There has been fiddling with Xen and EFI for quite some time. See: https://wiki.xenproject.org/wiki/Xen_EFI for example (might be old)... this indicates that Xen 4.3 or later could be built as a EFI binary and probably booted from the EFI firmware directly or with grub2 when grub2 is a EFI binary itself. Of course those instructions are all Linux-centric and I don't know if you created a Xen kernel like this if it would boot a NetBSD DOM0 kernel. I am in no position to try any tests with this right now personally, but it is tempting as I have a EFI only laptop that I could probably replace the hard drive temporarily. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: how to get nodev for zfs?
Greg Troxel writes: > I'm contemplating using zfs over NFS for domU package builders, and I'm > basically allergic to NFS for security reasons but it should be > confined. > > So I'm trying to reduce exposure, and have set setuid=off on > zfs filesystems. That successfully prevented a suid binary from working. > > The other usual thing is "nodev", and zfs has a devices property on or > off. So I went to set it to off and got an error that FreeBSD doesn't > support that. > > I made a device node (just with mknod) for wd0d and I was able to dd > from it. > > Is there any good approach to avoiding this? Why doesn't devices=off > just lead to the nodev mount option and work, similar to setuid=off > leads to nosuid? Not speaking for the total bug, but part of it exists in: src/external/cddl/osnet/dist/lib/libzfs/common/libzfs_dataset.c where there is a specific check to disallow the setting of the devices property and has the FreeBSD text. It would be a start to allow the property to be set. Once that is possible a "zfs set devices=off " should work (or panic??). Then the fileset may need to be umounted and remounted on NetBSD (zfs umount / zfs mount sort of thing). The second problem is that the mount may not pick up the property. That would be looking for zfs_mount(), zmount(), zfs_domount() and that ilk (probably zfs_main() in the zfs command itself too). I took a very brief look and did not see anything that would exclude the devices property, but did not have a great deal of time to sift though the code. There may be a translation missing somewhere too. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: zfs howto
Chavdar Ivanov writes: [snip] >> I am not sure of the complete context of the statement, but I do this >> all of the time with normal NetBSD NFS against a ZFS fileset. >> >> build% cat /etc/exports >> /usr/installed_src/PKGSRC_2018Q4 -alldirs -maproot=root >> anotherbuild.system.eldar.org >> >> build% zfs list /usr/installed_src/PKGSRC_2018Q4 >> NAME USED AVAIL REFER MOUNTPOINT >> tank/installed_src/PKGSRC_2018Q4 414M 250G 414M >> /usr/installed_src/PKGSRC_2018Q4 >> >> >> These are DOMUs running NetBSD 9.0_STABLE from around September. I have >> not tried this with -current, but there are no crashes for me with 9.x. [snip] > > I got it --- > > With the following entry in -etc-exports: > > /tank/t1 -maproot=0:10 -network 192.168.0/24 > > the NFS server crashes when /tank/t1 is zfs system. > > With the following one: > > /tank/t1 -maproot=root -network 192.168.0/24 > > it works fine. > > Mind you, '-maproot=0:10' is the first example from 'man exports' ... Glad to see that it isn't totally broken. I am by no means an expert in the ZFS code, and I am not in a position to take a lot of time looking at it right now, but if the trace back in the PR is correct, it makes it almost totally though the mkdir call and crashes in the log create step after the directory node is created. I am trying not to speculate too much here, but the code may fail to handle the group in the exports line. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: zfs howto
Chavdar Ivanov writes: [snip] >> If I want to export a zfs filesystem over NFS, what specifically do I >> need to do. >> >> Does the crash bug referenced in the NFS section still exist in >> current current? (It's still open) >> http://gnats.netbsd.org/55042 > > Yes, it still crashes, just checked creating a directory from a Linux > NFS client. I'll check if I got a dump in a moment ( I was rather > optimistic - testing this whilst doing pkg_rolling-replace... no > biggie, I'll restart it after the reboot...). > > Chavdar [snip] I am not sure of the complete context of the statement, but I do this all of the time with normal NetBSD NFS against a ZFS fileset. build% cat /etc/exports /usr/installed_src/PKGSRC_2018Q4 -alldirs -maproot=root anotherbuild.system.eldar.org build% zfs list /usr/installed_src/PKGSRC_2018Q4 NAME USED AVAIL REFER MOUNTPOINT tank/installed_src/PKGSRC_2018Q4 414M 250G 414M /usr/installed_src/PKGSRC_2018Q4 These are DOMUs running NetBSD 9.0_STABLE from around September. I have not tried this with -current, but there are no crashes for me with 9.x. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Using wg(4) with a commerical VPN provider
Jason Thorpe writes:
[snip]
>> wg_get_so_by_af(struct wg_softc *wg, const int af)
>> {
>>
>> +#if defined(INET) && defined(INET6)
>> return (af == AF_INET) ? wg->wg_so4 : wg->wg_so6;
>> +#else
>> +#ifdef INET
>> +return wg->wg_so4;
>> +#endif
>> +#ifdef INET6
>> +return wg->wg_so6;
>> +#endif
>> +#endif
>> }
>
> Seems ... not great to put #ifdefs like this in something that can be build
> as a module?
>
> -- thorpej
Ya, I noticed that too when I was looking at this. It is all over the
place, however, and currently if_wg does not build as a module anyway.
I think a lot more work will need to be done to get it there. I did not
look to see how any of the other if_xxx drivers deal with the lack of
INET or INET6 when modularized.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Using wg(4) with a commerical VPN provider
Roy Marples writes:
> On 11/11/2020 01:49, Brad Spencer wrote:
>> @@ -2352,6 +2361,7 @@
>> if (*af == AF_INET) {
>> packet_len = ntohs(ip->ip_len);
>> } else {
>> +#ifdef INET6
>> const struct ip6_hdr *ip6;
>>
>> if (__predict_false(decrypted_len < sizeof(struct ip6_hdr)))
>
>
> Might be better to roll it into case statement.
> Could wg one day work with a third address family?
>
> Roy
No earthly idea if that would ever be likely. In the short to medium
term, I suspect not. I am inclined to leave it as I have proposed
unless it functionally breaks something. I have zero ability to test
this path completely and the VPN provider I am using forcefully disables
IPv6 on their end. This topic can be left for a later time.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Using wg(4) with a commerical VPN provider
Hello... I would like to first thanks everyone who worked on and
contributed wg(4). For the past week or so I have been pushing lots of
bits attaching a 9.99.74 DOMU to a commercial VPN provider who provides
WireGuard as an option. After working out their rest API to get keys
set, it has been working just fine.
The only particular problem I encountered is that the if_wg driver will
not compile without INET6. Attached is a patch I have been running with
that allows if_wg.c to compile without INET6 and may help in the
opposite situation where INET is not defined. I have no way to test the
INET6 case. If there are no objections I will commit this in the next
few days.
--- sys/net/if_wg.c.DIST 2020-10-26 10:36:30.391354264 -0400
+++ sys/net/if_wg.c 2020-10-30 19:13:46.910323221 -0400
@@ -98,8 +98,8 @@
#include
#include
-#ifdef INET6
#include
+#ifdef INET6
#include
#include
#include
@@ -1611,7 +1611,16 @@
wg_get_so_by_af(struct wg_softc *wg, const int af)
{
+#if defined(INET) && defined(INET6)
return (af == AF_INET) ? wg->wg_so4 : wg->wg_so6;
+#else
+#ifdef INET
+ return wg->wg_so4;
+#endif
+#ifdef INET6
+ return wg->wg_so6;
+#endif
+#endif
}
static struct socket *
@@ -2352,6 +2361,7 @@
if (*af == AF_INET) {
packet_len = ntohs(ip->ip_len);
} else {
+#ifdef INET6
const struct ip6_hdr *ip6;
if (__predict_false(decrypted_len < sizeof(struct ip6_hdr)))
@@ -2359,6 +2369,9 @@
ip6 = (const struct ip6_hdr *)packet;
packet_len = sizeof(struct ip6_hdr) + ntohs(ip6->ip6_plen);
+#else
+ packet_len = decrypted_len + 1;
+#endif
}
WG_DLOG("packet_len=%u\n", packet_len);
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Is anyone using the wg(4) driver with a commerical VPN service??
Tom Ivar Helbekkmo writes: > Brad Spencer writes: > >> Just wondering if anyone has succeeded in getting the wg(4) network >> driver working with one of the commercial VPN providers? I attempted it >> with one in particular, with an admitted slightly older -current and did >> not succeed in getting it working. > > Haven't tried that, no - but I suspect you've configured something > wrong, as I know that our wg works reliably with the WireGuard(tm) > implementations in Linux and Android. (Well, either that, or you hit > one of the short windows of -current when something was wrong.) > > -tih Thanks for the infomation. The VPN service I am trying this with uses a REST style API call to get the needed bits of information in order for you to configure your side of the wg VPN. That API call partis working fine. It is completely possible that I messed something up on the local wg side. The system I am trying this on is a Xen amd64 PV DOMU from around September 1st or so. I had to compile a DOMU kernel with the wg pseudo device, but didn't really add anything else. I will try a newer tree and keep messing around. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Is anyone using the wg(4) driver with a commerical VPN service??
Just wondering if anyone has succeeded in getting the wg(4) network driver working with one of the commercial VPN providers? I attempted it with one in particular, with an admitted slightly older -current and did not succeed in getting it working. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: gpt type for zfs?
Sevan Janiyan writes: > On 28/03/2020 21:56, Thomas Klausner wrote: >> Are ZFS users on NetBSD using fbsd-zfs or avoiding gpt? :) > > ZFS doesn't actually care the filesystem time, it's happy to initialise > whatever device node you point at it. I actually partition things up in > sysinst as fat32 partitions. > > Regarding GPT, I connected up a pool which I created on MacOS some years > back to my laptop running 9.99.52, turns out I'd used a GPT scheme, I > couldn't actually import the pool because dk(4) hogs the device. (I > meant to try disabling dk support and retrying but haven't gotten around > to it yet). > > > Sevan I am a little suprised that the import didn't work. "zpool import" didn't find anything to import?? I currently use a gpt presented disk between NetBSD and FreeBSD without any particular problems. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: gpt type for zfs?
Thomas Klausner writes: > Hi! > > I just noticed that the gpt man page has a type fbsd-zfs, but none for > NetBSD ZFS or any other type. > > Are ZFS users on NetBSD using fbsd-zfs or avoiding gpt? :) > > Is there something we should do in gpt or the man page? > Thomas For gpt, I think anything will work, but I use fbsd-zfs especially if I think I might share the pool with FreeBSD. I have also put pools on raw devices and DOMU xbd* back stored by a LVM lvs. A lot of different combinations seem to work fine. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: xen & uefi
Patrick Welche writes: > Is booting into xen from uefi meant to work? > > I have a slightly unorthdox set up, but get: > >>> NetBSD/x86 EFI Boot (x64), Revision 1.1 (Tue Jan 28 13:49:42 UTC 2020) >>> (from) > ... > Start @ 0xce60 [1=0xce982000-0xce9820ec]... > Trampoline space cannot be allocated; will try fallback. > > > > Cheers, > > Patrick I didn't think that a DOM0 + UEFI worked anywhere very well at this point or at least was not a default... See https://wiki.xenproject.org/wiki/Xen_EFI for example... or ... https://ubuntuforums.org/showthread.php?t=2413434 -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: ZFS on root - almost there
Roy Marples writes: > So I thought I would have a go at setting up ZFS on root. > > Thanks to hannken@ it now boots :) > However, it panics at shutdown (or halt). Screen capture of the panic here: > http://www.netbsd.org/~roy/netbsd-zfs-panic.jpg > > Now, what I did during the initial setup was to adjust the mountpoint of > tank/ROOT/usr to /usr - ie relative to the chroot. > > The bootstrap phase is this in /etc/rc > > fsck -y / > zfs mount tank/ROOT > mount -t null /dev /tank/ROOT/dev > mount -t null / /tank/ROOT/altroot # this doesn't appear to work > sysctl -w init.root=/tank/ROOT > > This works fine, we enter the chroot > For the time being I've disabled fsck_root and adjusts zfs to load all mounts. > > We now get to the login with minimal errors and all appears to work. > You can see the mountlist inside the chroot at the top of the screen capture. > > If some kind person can fix this panic then I can copy across my live home > site > setup (web server, email, etc) and really test it out. > > Roy I have done the root-pivot thing before with cgd. I believe I noticed panics on shutdown there too. I also had lvm involved so I was never very sure what the problem was. This would have been in the late 6.x and early 7.x days. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: [Small Heads up] USE_SHLIBDIR=yes added some some library Makefiles
Andreas Gustafsson writes:
> Brad Spencer wrote:
>> I committed a change today to add USE_SHLIBDIR=yes to the libraries used
>> by /sbin/{zfs,mount_zfs,zpool}. The general effect will be to move the
>> libraries from /usr/lib to /lib and put compatibility links in place so
>> that things, say in /usr/pkg, continue to work as expected. Run tested
>> on amd64 and i386 and compile tested on evbarm. This will allow /usr
>> and /var to be mounted as a ZFS legacy filesystem and keeping with the
>> apparent pattern of having items in /sbin depend on items in /lib and
>> not /usr/lib.
>>
>> Sorry if this breaks anything.
>
> Looks like it broke MKDEBUG=yes builds:
>
> === 8 extra files in DESTDIR =
> Files in DESTDIR but missing from flist.
> File is obsolete or flist is out of date ?
> --
> ./usr/libdata/debug/lib/libavl.so.0.0.debug
> ./usr/libdata/debug/lib/libnvpair.so.0.0.debug
> ./usr/libdata/debug/lib/libpthread.so.1.4.debug
> ./usr/libdata/debug/lib/libumem.so.0.0.debug
> ./usr/libdata/debug/lib/libuutil.so.0.0.debug
> ./usr/libdata/debug/lib/libzfs.so.0.0.debug
> ./usr/libdata/debug/lib/libzfs_core.so.0.0.debug
> ./usr/libdata/debug/lib/libzpool.so.0.0.debug
> ===== end of 8 extra files ===
>
> *** [checkflist] Error code 1
I will look into this today.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
[Small Heads up] USE_SHLIBDIR=yes added some some library Makefiles
I committed a change today to add USE_SHLIBDIR=yes to the libraries used
by /sbin/{zfs,mount_zfs,zpool}. The general effect will be to move the
libraries from /usr/lib to /lib and put compatibility links in place so
that things, say in /usr/pkg, continue to work as expected. Run tested
on amd64 and i386 and compile tested on evbarm. This will allow /usr
and /var to be mounted as a ZFS legacy filesystem and keeping with the
apparent pattern of having items in /sbin depend on items in /lib and
not /usr/lib.
Sorry if this breaks anything.
If one happens to update /usr/lib without /lib there may be problems,
but hopefully that is very uncommon.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Automated report: NetBSD-current/i386 build failure
Martin Husemann writes: > On Mon, Sep 16, 2019 at 02:46:53PM +0300, Andreas Gustafsson wrote: >> On i386, MKZFS is not defined even though mk.conf(5) says it is. >> Which one is wrong, the code or the manpage? > > The man page is wrong, MKZFS does not make sense on anything with a 32bit > kernel. > > Martin Interesting... it appears we may still be building the solaris module. Is that for dtrace?? -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: etc/rc.d/zfs not installed
"K. Schreiner" writes:
> Hi,
>
> sorry for the last mail, patch was wrong!
>
> I hope, thats the right one now:
>
> Index: etc/rc.d/Makefile
> ===
> RCS file: /cvsroot/src/etc/rc.d/Makefile,v
> retrieving revision 1.105
> diff -u -r1.105 Makefile
> --- etc/rc.d/Makefile 15 Sep 2019 19:38:09 - 1.105
> +++ etc/rc.d/Makefile 16 Sep 2019 13:42:46 -
> @@ -54,14 +54,15 @@
> FILESBUILD_xfs=yes
> FILESBUILD_fccache= yes
>
> -.if ${MKZFS} != "no"
> -CONFIGFILES+= zfs
> -.endif
> -
> .SUFFIXES: .in
> .in:
> ${_MKTARGET_CREATE}
> ${TOOL_SED} -e 's,@X11ROOTDIR@,${X11ROOTDIR},g' < ${.IMPSRC} >
> ${.TARGET}
> .endif
>
> +.if ${MKZFS} != "no"
> +CONFIGFILES+= zfs
> +.endif
> +
> +
>
>
> Kurt
Thanks for seeing that.. my fault... I did not notice the .endif.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Automated report: NetBSD-current/i386 build failure
Andreas Gustafsson writes: > The build is still failing as of source date 2019.09.16.04.59.32, > now with a different error: > > == 2 missing files in DESTDIR > Files in flist but missing from DESTDIR. > File wasn't installed ? > -- > ./usr/share/man/html8/mount_zfs.html > ./usr/share/man/man8/mount_zfs.8 > end of 2 missing files == I will look at that, but won't be for a bit... apparently a "./build.sh .. release" does not show off that sort of thing. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Legacy ZFS mounts
Chavdar Ivanov writes: > Since yesterday I don't get my zfs fule systems automatically mounted, > I have to issue 'zfs mount -a'. > > Should I add some entry in /etc/fstab or place the above command in > one of the scripts? > > The zvols are accessible as usual. Add zfs=YES to your rc.conf. It works like LVM and raidframe now... This get the module started early and allows you to disable the support later (as opposed to by accident which is how it mostly worked before). It was in the commit message, but should have been mentioned more broadly. Sorry... -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Legacy ZFS mounts
m...@netbsd.org writes: [snip] >> The module has to load before zvols will show up and if that is ALL you >> were doing, I don't think anything else will prompt the loading of the >> module. That is, your /dev/zvol/* tree would not be there unless you >> execute the zfs (and probably the zpool) command prior to trying to use >> your devices (I think that it is the opening and use of /dev/zfs that >> does prompt the module load, but that isn't needed for pure zvol >> access). > > Would it make sense to make mount_zfs do that? The prompting of the module load would happen with mount_zfs for the ZFS filesystem/dataset case, so in that case you are very correct that a additional poke is not needed, but mount_zfs is not use when just accessing zvol devices. In fact, nothing is needed for raw zvol use, except the loading of the module. By example, lets suppose you are using LVM devices as back store to DOMU guests. You still have to do a /sbin/vgscan to get the devices created even if you are not going to use the logical volumes for anything else but back store. This might, have not actually looked, prompt the load of the device manager module hence our /etc/rc.d/lvm script (among other uses, of course, in this case). In the ZFS zvol case, it would be the same thing except that all that is needed is that the module load. The /etc/rc.d/zfs script I propose just does a "zfs list" (and checks the return code, as it is possible that /dev/zfs is missing or there were other errors and reports this... it is also possible to build a system without ZFS built, we have a make variable for that, and if you happen to do a zfs=YES we should not be dumb about that) which is what was already present in /etc/rc.d/mountall. I simply moved it sooner in the boot process and made it more literal and intentional with the variable as I wanted to use zvol devices before mountall ran. I would also need to do something to get the module loaded if I was going to present zvols to DOMU (as I do intend to do some day) but did not use a ZFS filesystem/dataset for anything in the DOM0. To cover the various cases, I don't see how one gets all the bits and pieces in place in really any other manor. As I said, this is all done more or less in this way for raidframe and LVM. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Legacy ZFS mounts
m...@netbsd.org writes: > when asking for reviews on diffs, please consider having them in an easy > to view format, rather than tar files. > > Having a 'zfs' script to load a module sounds wrong. We have module > autoloading for this purpose. Thanks for the comments... Ya, there are two purposes to the variable, one to make sure that the module loads early and the second for determining the desire to mount ZFS data sets in the mostly normal manor. If you have a corrupt ZFS cache, for example, you may want to not do the second one. The module has to load before zvols will show up and if that is ALL you were doing, I don't think anything else will prompt the loading of the module. That is, your /dev/zvol/* tree would not be there unless you execute the zfs (and probably the zpool) command prior to trying to use your devices (I think that it is the opening and use of /dev/zfs that does prompt the module load, but that isn't needed for pure zvol access). In addition, it is possible to put a FFS (or anything) in a zvol and use it in /etc/fstab, and the device would have to exist in that case too (technically the devices would still exist if you have used them once, but the module supporting them would not be present). I know that just trying to access /dev/zvol/* does not load the module. Hence the variable to prime the pump and get the module loaded early in the boot sequence. Off hand I do not know of any other way to get the desired behavior currently. Of course, in the root-filesystem-on-ZFS case this would not be needed (the module would have to be there already). In most of the ways that seem important the proposed behavior is not any different from what is done with LVM or raidframe currently. The zfs rc.d script may end up doing more in the future, like zpool import and the like. Make sense?? -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Legacy ZFS mounts
I would like to commit a change soon that will add support for legacy ZFS mounts. That is, those where the ZFS property mountpoint=legacy. This will allow one to specify ZFS mounts in /etc/fstab and will allow the use of critical_local_filesystems in /etc/rc.conf. Basically you could have a ZFS /var and /use if you wanted. The support for this is already pretty much present in the base code inherited from OpenSolaris. If the zfs command is linked to a command called "mount" in Solaris-world it will act like a classic mount command (mostly). In NetBSD-world, this can be changed to be linked to mount_zfs and placed in /sbin and it will do the same thing if the code is altered to recognize mount_zfs instead of mount. The change is in two parts, the first is to allow mount_zfs to be recognized instead of mount, hard link /sbin/zfs to /sbin/mount_zfs and provide for a man page. All of the stuff in distrib is dealt with. In addition, a new /etc/rc.d script comes into existence called zfs and a new variable for /etc/rc.conf called 'zfs' which must be set to YES to get zvols or use ZFS at all. Further, /etc/rc.d/mountall is altered to look for the zfs variable and do the usual "zfs mount" stuff if it is YES. The second part of the change has to do with the fact that /sbin/zfs and /sbin/zpool link libraries from /usr/lib, when they both probably should get libraries from /lib. So, a number of the Makefiles for the ZFS libraries got "USE_SHLIBDIR=yes" added to them. The only library outside of the ZFS support libraries that also had to have this is libpthread. In all cases compatibility links are automatically put in /usr/lib, so nothing that exists in the system or from pkgsrc breaks. This is all based upon prior art that is used already. All of this can be found at: https://www.netbsd.org/~brad/legacy_zfs_mounts as two tar files. One is the mount_zfs change and the other is the library move, or really just an adjustment. I have been running test -current systems with this change for a while without any apparent problems. It will also apply cleanly to 9.0. I won't get to committing this for a few days. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Recent XEN3_DOM0 begaviour
Chavdar Ivanov writes: > Hi, > > I recently restored my earlier NetBSD/XEN setup on an old(ish) HP > EliteBook laptop; it all went rather well using xen packages for 4.11. > I then added a couple of disks to the system (one - the internal > mSATA, the other - on the tray replacing the DVD) with the intention > to use the second one as storage for XEN guests. I formatted this one > as GPT and added a couple of disks: > . [snip] > It all works well, with the exception that when I reboot the host > under XEN using the usual > > menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc;multiboot /xen.gz dom0_mem=2048M > > the kernel decides that it has been booted from dk0, instead of wd0, > and I have to be present for the boot and point wd0 to be the boot > disk (BTW if I point at this moment to dk1/dk0 I am able to boot the > guest on bare metal using the same XEN3_DOM0 kernel... ). > > Is there any means to force XEN3_DOMU to use wd0 as boot device? This > doesn't happen if I use the normal /netbsd kernel, it can see dk0/dk1 > but proceeds to booot from wd0 by default. > > Or perhaps I have to modify XEN3_DOM0 ? > > > Chavdar Yes, this is possible. NetBSD is using the multiboot protocol to boot Xen DOM0. There is a man page at 'man 8 multiboot' or 'man 8 x86/multiboot'. I used to have to use the root= option on previous versions of my DOM0 systems, but have not had to do that recently due to hardware reorgs. If I remember things correctly, you should be able to change your menu as follows: menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc root=wd0a;multiboot /xen.gz dom0_mem=2048M You may also want to add dom0_max_vcpus=1 as well. The world will likely be different once PVHVM is available, but for now, a DOM0 can not use more than one CPU: menu=Xen:load /netbsd-XEN3_DOM0.gz console=pc root=wd0a;multiboot /xen.gz dom0_mem=2048M dom0_max_vcpus=1 -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Panic with recent -current with interrupt setup
Masanobu SAITOH writes: > On 2018/10/03 5:47, Brad Spencer wrote: >> m...@netbsd.org writes: >> >>> On Tue, Oct 02, 2018 at 06:55:48AM -0400, Brad Spencer wrote: >>>> [snip] >> Here is more information: >> >> Screen shot of the panic: >> http://www.netbsd.org/~brad/PANIC_1.jpg >> http://www.netbsd.org/~brad/PANIC_2.jpg >> >> Screen shot of the ddb bt command, sorry for the quality: >> http://www.netbsd.org/~brad/BT_1.jpg >> http://www.netbsd.org/~brad/BT_2.jpg >> http://www.netbsd.org/~brad/BT_3.jpg >> >> Hopefully I also managed to attach a couple of files that are of a >> working NetBSD dmesg and a working xl dmesg. The Xen version I am using >> is 4.8.3 built from source pulled from HEAD on Saturday or so. >> >> This system is pretty new. It has a 4 core Ryzen CPU, 16GB memory. A >> two port Intel NIC is also present, that would be wm0 and wm1 in the >> dmesg. The motherboard has a Intel NIC on it as well, which is wm2. >> >> If I do a dmesg in ddb I can get the reason for the panic: >> >> panic: kernel diagnostic assertion "irq2vect[irq] == 0" failed: >> file"/usr/src/sys/arch/xen/x86/pintr.c", line 202 >> >> It looks like this may have triggered on the onboard wm2 interface. >> > > Try to revert x86/pci/pci_intr_machdep.c rev. 1.45: > > http://mail-index.netbsd.org/source-changes/2018/09/23/msg099361.html Reverting to version 1.44 of x86/pci/pci_intr_machdep.c eliminated the panic and the system boots up. Version 1.45 is a revert itself, but there must be something in it that a Xen DOM0 needs. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Panic with recent -current with interrupt setup
m...@netbsd.org writes: > On Tue, Oct 02, 2018 at 06:55:48AM -0400, Brad Spencer wrote: >> >> Just wondering if anyone else has seen this, but I am getting panics on >> boot during probe with sources after 2018-09-23 [at some point, at least >> 2018-09-29 and 2018-10-01 panic, but 2018-09-23 doesn't]. This is with >> trying to use the stock XEN3_DOM0 kernel on a new system I am setting >> up. The panics seem related to setting up interrupts or printing >> interrupt information in the intel wm(4) driver. The system in question >> does not have a serial port on it in any form, but I can probably >> capture a screen shot of the panic. The keyboard works and ddb seems >> usable. > > I assume this is related to cherry's recent xen interrupt work. > If you're unable to type at the ddb prompt but can reproduce the crash, > but can see the output, it'd be interesting to see if it gives more > info with > > options DDB_COMMANDONENTER="bt" > > As a kernel option. > > But having the panic string would be nice too. Here is more information: Screen shot of the panic: http://www.netbsd.org/~brad/PANIC_1.jpg http://www.netbsd.org/~brad/PANIC_2.jpg Screen shot of the ddb bt command, sorry for the quality: http://www.netbsd.org/~brad/BT_1.jpg http://www.netbsd.org/~brad/BT_2.jpg http://www.netbsd.org/~brad/BT_3.jpg Hopefully I also managed to attach a couple of files that are of a working NetBSD dmesg and a working xl dmesg. The Xen version I am using is 4.8.3 built from source pulled from HEAD on Saturday or so. This system is pretty new. It has a 4 core Ryzen CPU, 16GB memory. A two port Intel NIC is also present, that would be wm0 and wm1 in the dmesg. The motherboard has a Intel NIC on it as well, which is wm2. If I do a dmesg in ddb I can get the reason for the panic: panic: kernel diagnostic assertion "irq2vect[irq] == 0" failed: file"/usr/src/sys/arch/xen/x86/pintr.c", line 202 It looks like this may have triggered on the onboard wm2 interface. [ 1.00] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, [ 1.00] 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, [ 1.00] 2018 The NetBSD Foundation, Inc. All rights reserved. [ 1.00] Copyright (c) 1982, 1986, 1989, 1991, 1993 [ 1.00] The Regents of the University of California. All rights reserved. [ 1.00] NetBSD 8.99.25 (XEN3_DOM0) #0: Sat Sep 22 05:24:22 UTC 2018 [ 1.00] mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/xen/compile/XEN3_DOM0 [ 1.00] total memory = 4096 MB [ 1.00] avail memory = 3948 MB [ 1.00] cpu_rng: RDSEED [ 1.00] timecounter: Timecounters tick every 10.000 msec [ 1.00] Kernelized RAIDframe activated [ 1.00] running cgd selftest aes-xts-256 aes-xts-512 done [ 1.00] To Be Filled By O.E.M. To Be Filled By O.E.M. (To Be Filled By O.E.M.) [ 1.00] mainbus0 (root) [ 1.00] ACPI: RSDP 0x000F05B0 24 (v02 ALASKA) [ 1.00] ACPI: XSDT 0xDC0210A0 BC (v01 ALASKA A M I 01072009 AMI 00010013) [ 1.00] ACPI: FACP 0xDC0278F0 000114 (v06 ALASKA A M I 01072009 AMI 00010013) [ 1.00] Firmware Warning (ACPI): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x/0x1 (20180810/tbfadt-693) [ 1.00] ACPI: DSDT 0xDC0211F0 0066FC (v02 ALASKA A M I 01072009 INTL 20120913) [ 1.00] ACPI: FACS 0xDC3D1E00 40 [ 1.00] ACPI: APIC 0xDC027A08 DE (v03 ALASKA A M I 01072009 AMI 00010013) [ 1.00] ACPI: FPDT 0xDC027AE8 44 (v01 ALASKA A M I 01072009 AMI 00010013) [ 1.00] ACPI: FIDT 0xDC027B30 9C (v01 ALASKA A M I 01072009 AMI 00010013) [ 1.00] ACPI: SSDT 0xDC027BD0 00533D (v02 AMDAmdTable 0002 MSFT 0202) [ 1.00] ACPI: SSDT 0xDC02CF10 0007DC (v01 AMDAMD CPU 0001 AMD 0001) [ 1.00] ACPI: CRAT 0xDC02D6F0 000810 (v01 AMDAMD CRAT 0001 AMD 0001) [ 1.00] ACPI: CDIT 0xDC02DF00 29 (v01 AMDAMD CDIT 0001 AMD 0001) [ 1.00] ACPI: SSDT 0xDC02DF30 002C9B (v01 AMDAMD AOD 0001 INTL 20120913) [ 1.00] ACPI: MCFG 0xDC030BD0 3C (v01 ALASKA A M I 01072009 MSFT 00010013) [ 1.00] ACPI: AAFT 0xDC030C10 000200 (v01 ALASKA OEMAAFT 01072009 MSFT 0097) [ 1.00] ACPI: HPET 0xDC030E10 38 (v01 ALASKA A M I 01072009 AMI 0005) [ 1.00] ACPI: UEFI 0xDC030E48 42 (v01 ) [ 1.00] ACPI: IVRS 0xDC030E90 D0 (v02 AMDAMD IVRS 0001 AMD ) [ 1.00] ACPI: SSDT 0x0
Panic with recent -current with interrupt setup
Just wondering if anyone else has seen this, but I am getting panics on boot during probe with sources after 2018-09-23 [at some point, at least 2018-09-29 and 2018-10-01 panic, but 2018-09-23 doesn't]. This is with trying to use the stock XEN3_DOM0 kernel on a new system I am setting up. The panics seem related to setting up interrupts or printing interrupt information in the intel wm(4) driver. The system in question does not have a serial port on it in any form, but I can probably capture a screen shot of the panic. The keyboard works and ddb seems usable. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: Is NetBSD ReFUSE layer capable to do MTP?
Rhialto <rhia...@falu.nl> writes: > On Sun 04 Sep 2016 at 19:15:07 +, Daniel Ölschlegel wrote: >> The library libmtp(in Pkgsrc) comes with a test program which should >> list some devices, so maybe its a problem with the library which i can >> debug if someone could establish a connection. > > Yes, I don't see my phone attached either with test programs like > mtp-connect or mtp-detect. > > -Olaf. Hello.. I was just messing with libmtp. I attached it to a Sandisk CLIP zip audio player in MTP mode, an old Android 4.2 tablet and a Android 6.0.1 Cyanogenmod 13 based phone. What I noted was that it can take a VERY long time for the mtp test programs to actually finish connecting... think many minutes. If mtp-detect just appears to hang, try waiting a while and see if it ever returns. The audio player worked the best, just a second or two to do most operations, but it probably is the simplest. The 4.2 Android tablet also worked and I moved files from it without any real problems. I didn't fiddle much with the Cyanogenmod 13 based phone much as it way too long for it to actually complete any operations. It was much simpler to just sftp files out of the device via 802.11. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS http://anduin.eldar.org - & - http://anduin.ipv6.eldar.org [IPv6 only]
Re: (almost) full disk encryption, cgdroot and firmware
Alexander Nasonov <al...@yandex.ru> writes: > This is my story on (almost) full disk encryption. > > I followed Pierre Proncher's instruction from Mar 2013. To my > surprise, it worked on the first boot. However, networking didn't > work because the kernel couldn't load iwm firmware. > > After a couple of attempts to fix firmware loading, I gave up on > cgdroot ramdisk and switched to a fake root on wd0a. It's similar > to cgdroot but with modules and firmware files. I hard-linked few > binaries including init from /rescue. > > My setup has not one but two cgd devices: cgd0 for a real root and > cgd1 for other partitions. cgd0's key is stored on unencrypted > wd0a, cgd1's key is stored on the real root cgd0a. I have to enter > two passwords instead of one but this setup gives me some protection > from an unsophisticated keylogger. Since wd0a is read-only, I can > add wd0a integrity check before running the second cgdconfig -C > command and abort before entering the second password if the check > fails. A real rootkit can easily fool the integrity checker, though. > > Alex I ran with a simular set up for quite a while including the use of a iwm based wireless card. I assume you used init.root and pointed it at the cgd after the password was given. If so, then I believe that the trick to getting the firmware to load is that it must be placed in the filesystem that the kernel actually was booted from and not the one that is chrooted to from init. It sound like in your case that would have been the ramdrive, in my version of this it was an external flash disk. I basically put a copy of /libdata on the boot media. Without doing this, the firmware wouldn't be found even if it was present on in the cgd filesystem that init.root chroot'ed to. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS http://anduin.eldar.org - & - http://anduin.ipv6.eldar.org [IPv6 only]
