Re: repeated failure to properly shutdown

[Eric Haszlakiewicz]

On 7/22/2016 2:41 PM, Martin Husemann wrote:

On Fri, Jul 22, 2016 at 11:37:56AM -0700, bch wrote:

How does that happen, how does one fix it ?

It is created by init if there is no /dev/console.

Boot some install media, mount your root file system (say on /mnt)
then:

cd /mnt/dev
sh MAKEDEV all

(hoping there is a MAKEDEV script there, if not: extract it from etc.tgz
from the install sets)


Another way to fix it is to use mount_null to gain access to the hidden 
/dev directory.
Up until a few minutes ago (when I checked in a change to mount_null.c), 
mount_null
would refuse to do this with a error about the paths not being distinct, 
but you can

build a copy that doesn't have this limitation with:

ftp 
'http://cvsweb.netbsd.org/bsdweb.cgi/~checkout~/src/sbin/mount_null/mount_null.c'

sed -e's/(subdir.*/(0)/' -I mount_null.c
gcc -o mount_null mount_null.c -lutil

Then:
./mount_null / /mnt
cd /mnt/dev
./MAKEDEV all

You'll still need to reboot to get rid of the tmpfs, but this way you 
only need to do that once

and you don't need separate media to boot off of.

Eric

Re: repeated failure to properly shutdown

[bch]

Iirc, where I *noticed* it was /etc/defaults/rc.d
On Jul 22, 2016 5:03 PM, "Robert Elz"  wrote:

> Date:Fri, 22 Jul 2016 16:27:19 -0700
> From:bch 
> Message-ID:  <
> cabfrot-2q4yoonz_d3vfjm8bgwknq3g7jxie8-fctc8w8v-...@mail.gmail.com>
>
>   | It could be that for some reason it's missing, and a first attempt to
> write
>   | to it just creates a regular file...
>
> Yes, that's a given - the question is just where is that first attempt.
> It has to be before init makes the tmpfs /dev (the /dev/null created in
> the tmpfs would have been the right thing, or you would have noticed that
> problem much sooner) and it has to be with the live system as root (not
> when it is mounted on /mnt) as nothing is likely to accidentally write
> to /mnt/dev/null ...
>
> A chroot to /mnt might do it I suppose...
>
> Michael Plass  said:
>
>   | Could it perhaps come from the ( set -o tabcomplete 2>/dev/null ) in
> /etc/
>   | shrc?
>
> Not in normal operation, /etc/shrc wouldn't normally be able to run until
> way after init has created the tmpfs /dev - init doesn't set ENV, so the
> sh it runs to execute MAKEDEV wouldn't source shrc - ENV set to /etc/shrc
> normally comes from /root/.profile which would be used only when root logs
> in.
>
> kre
>
>
>

Re: repeated failure to properly shutdown

[bch]

It could be that for some reason it's missing, and a first attempt to write
to it just creates a regular file...
On Jul 22, 2016 4:25 PM, "Robert Elz"  wrote:

> Date:Sat, 23 Jul 2016 04:38:42 +0700
> From:Robert Elz 
> Message-ID:  <20406.1469223...@andromeda.noi.kre.to>
>
>   | That /dev/null turned into a regular file is another bug [...]
>   | (This turns out to be a bug in MAKEDEV [...]
>
> Actually, not, it must be elsewhere, or as a result of something else.
>
> kre
>
>

Re: repeated failure to properly shutdown

[bch]

On Jul 22, 2016 2:39 PM, "Robert Elz"  wrote:
>
> Date:Fri, 22 Jul 2016 12:52:58 -0700
> From:bch 
> Message-ID:  <
cabfrot-qp37szwwfrj+zedjqxf5e2troxmh0lmqc+dckb0k...@mail.gmail.com>
>
>   | I think that biggest concern (unclean shutdown/reboot) is solved
(collision
>   | of /dev and a tmpfs mount, caused by default behavior of init in face
of
>   | missing /dev/console).
>
> Yes, and now we know what the cause is, we should be able to duplicate the
> problem, and work out what is really happening.   The system is supposed
to
> work with a tmpfs /dev, it should not panic during shutdown.   What's
more,
> this panic is probably not related to it being /dev - any tmpfs mounted
> with -o union on a mount point that is using WAPBL (-o log) will probably
> panic the same way.
>
>   | This disk was prepared remotely (I.e. from another running NetBSD
box) by
>   | partitioning the disk (disklabel), formatting (newfs), then mounting
all
>   | partitions appropriately under /mnt and running ./build.sh ...
install=/mnt
>
> That builds enough for the system to install, but it does not make a
> fully runnable system - there's more that sysinst  normally does (like
> populating /dev - but also making a basic rc.conf (including all network
> config, setting hostname) and fstab, that don't get built by build.sh
> either (and nor should they).   Those you must have done later.

I did, yes.

> Running
> MAKEDEV is just another of the steps that one needs to perform.

And that one I don't entirely recall what i did. There certainly was a
mostly-populated /dev... I wish I recalled specifically what I did there.

> That /dev/null turned into a regular file is another bug - it is being
> created before the tmpfs /dev is made, I have seen that before as well,
> but just corrected and ignored the problem until now.   (This turns out
> to be a bug in MAKEDEV which is run by init to make the tmpfs /dev when
> /dev/console is not present.)
>
> Your solution to that was correct.   (MAKEDEV did not fix it as it never
> replaces anything that already exists, only makes what does not - even if
> what exists is nonsense, and even if it created the nonsense itself.)
>
> It is good that your problems are overcome now - and thanks for bring it
to
> our attention, and for being willing to suffer through getting enough info
> to allow the problem to be better understood.

Really, my pleasure.

-bch

> kre
>

Re: repeated failure to properly shutdown

[bch]

mknod null c 2 2 did the trick, things boot/reboot alright now.

I think that biggest concern (unclean shutdown/reboot) is solved (collision
of /dev and a tmpfs mount, caused by default behavior of init in face of
missing /dev/console).

This disk was prepared remotely (I.e. from another running NetBSD box) by
partitioning the disk (disklabel), formatting (newfs), then mounting all
partitions appropriately under /mnt and running ./build.sh ... install=/mnt

Clearly some things didn't fully work. I've got some shutdown msgs that
seem new (though in situations like this (suspect system), under hard
scrutiny a lot of things start looking "new")... I'll post that in a new
thread, as I think it's out of scope for this threads intended purpose...

Thanks everyone for the help. I learned a bit more, and had fun.

-bch
On Jul 22, 2016 12:32 PM, "bch"  wrote:

> For some reason /dev/null is just a regular file... looking for its device
> special numbers to put it in by mknod...
>
> I see what you're saying, did it (and do have a console device), but now
> my etc/defaults/rc.d is barfing on: cannot create /dev/null: read only file
> system
> On Jul 22, 2016 11:41 AM, "Martin Husemann"  wrote:
>
>> On Fri, Jul 22, 2016 at 11:37:56AM -0700, bch wrote:
>> > How does that happen, how does one fix it ?
>>
>> It is created by init if there is no /dev/console.
>>
>> Boot some install media, mount your root file system (say on /mnt)
>> then:
>>
>> cd /mnt/dev
>> sh MAKEDEV all
>>
>> (hoping there is a MAKEDEV script there, if not: extract it from etc.tgz
>> from the install sets)
>>
>> Then reboot and check mount again.
>>
>> Martin
>>
>

Re: repeated failure to properly shutdown

[bch]

I see what you're saying, did it (and do have a console device), but now my
etc/defaults/rc.d is barfing on: cannot create /dev/null: read only file
system
On Jul 22, 2016 11:41 AM, "Martin Husemann"  wrote:

> On Fri, Jul 22, 2016 at 11:37:56AM -0700, bch wrote:
> > How does that happen, how does one fix it ?
>
> It is created by init if there is no /dev/console.
>
> Boot some install media, mount your root file system (say on /mnt)
> then:
>
> cd /mnt/dev
> sh MAKEDEV all
>
> (hoping there is a MAKEDEV script there, if not: extract it from etc.tgz
> from the install sets)
>
> Then reboot and check mount again.
>
> Martin
>

Re: repeated failure to properly shutdown

[Robert Elz]

Date:Fri, 22 Jul 2016 22:15:02 -0700
From:Michael Plass 
Message-ID:  <3c6b1155-00df-420d-82da-6227d62d7...@plass-family.net>


  | Hmm, it looks like doing "shutdown now" to get into single-user
  | will force-unmount the tmpfs file systems (/etc/rc.d/swap1),

Yes, but for Brad, that is what caused the panic that started all of
this discussion, so ...

  | so you could be left in a state where
  | creating a regular /dev/null becomes all too easy.

that possibility never arose.

  | As an aside, while testing out things I also noticed that MAKEDEV
  | tries to use $MKNOD in create_mfs_dev(), but this is only set if
  | the -m switch is supplied.

I'd need to analyse that a little more, but I am not sure it is
relevant.   Perhaps the MKNOD there should be TOOL_MKNOD ?

  | So the attempt to make a temporary console fails.

If the mknod of temp_console fails, the result would be that output
from MAKEDEV would be sent to a unlinked file ("temp_console") rather
than to the actual system console device, which is clearly the intent.
While that's not good, if it happens, nor is it a reason for anything else
to fail, and nor does it explain how /dev/null would get to be a file.

kre

Re: repeated failure to properly shutdown

[Michael Plass]

On Jul 22, 2016, at 5:02 PM, Robert Elz wrote:

> the question is just where is that first attempt.

Hmm, it looks like doing "shutdown now" to get into single-user will 
force-unmount
the tmpfs file systems (/etc/rc.d/swap1), so you could be left in a state where
creating a regular /dev/null becomes all too easy.


As an aside, while testing out things I also noticed that MAKEDEV tries to use 
$MKNOD
in create_mfs_dev(), but this is only set if the -m switch is supplied. So the 
attempt
to make a temporary console fails.

Thanks,
- Michael

Re: repeated failure to properly shutdown

[Robert Elz]

Date:Fri, 22 Jul 2016 17:09:30 -0700
From:bch 
Message-ID:  


  | Iirc, where I *noticed* it was /etc/defaults/rc.d

Yes, that (/etc/defaults/rc.conf I assume you mean) writes to /dev/null - but
init has made the tmpfs /dev (if it needs it) before it runs /etc/rc - and so
before /etc/defaults/rc.conf gets used.

There is essentially nothing possible from when the system boots before the
tmpfs /dev/ is made - when /dev/console is not there.  MAKEDEV is just about
the first thing init does in that case - which is why I initially assumed that
the problem had to be there (but MAKEDEV does literally nothing to /dev/null
except mknod it when needed.)

Given that I see two possibilities, and maybe you can remember which is more
likely?   Either /dev/null (the file in /dev on the root filesys) got created
before you initially booted the system, or it was created while you were
fixing the missing /dev/console just recently.

First, at any time did you have your new root filesystem mounted somewhere,
and chroot /to/it (that is if it were on /mnt and you did "chroot /mnt") ?

There was no need to do that to fix the missing /dev/console (and the missing
rest of /dev) and it was not what Martin suggested you do, so I am going to
guess that this did not happen today/yesterday when you were fixing things.
Sound right?

So, think back to when you first built the system.   Sometime then you would
have needed to do some configuration - did you boot first, and then configure
(stuff like the hostname, the network config, rc_configured=YES in rc.conf etc)
or did you set some of that up before you booted?  (It doesn't matter here
if it was the very first boot or not, just if you did setup only while running
the new system, or if you did some config from the system you used to run
build.sh).

If you did config using the older system - how did that happen?   Do you just
"cd /new-root/etc; edit; edit; edit ..." or did you "chroot /new-root" ... ?

kre

Re: repeated failure to properly shutdown

[Robert Elz]

Date:Fri, 22 Jul 2016 16:27:19 -0700
From:bch 
Message-ID:  


  | It could be that for some reason it's missing, and a first attempt to write
  | to it just creates a regular file...

Yes, that's a given - the question is just where is that first attempt.
It has to be before init makes the tmpfs /dev (the /dev/null created in
the tmpfs would have been the right thing, or you would have noticed that
problem much sooner) and it has to be with the live system as root (not
when it is mounted on /mnt) as nothing is likely to accidentally write
to /mnt/dev/null ...

A chroot to /mnt might do it I suppose...

Michael Plass  said:

  | Could it perhaps come from the ( set -o tabcomplete 2>/dev/null ) in /etc/
  | shrc? 

Not in normal operation, /etc/shrc wouldn't normally be able to run until
way after init has created the tmpfs /dev - init doesn't set ENV, so the
sh it runs to execute MAKEDEV wouldn't source shrc - ENV set to /etc/shrc
normally comes from /root/.profile which would be used only when root logs in.

kre

Re: repeated failure to properly shutdown

[Michael Plass]

On Jul 22, 2016, at 4:24 PM, Robert Elz wrote:

>Date:Sat, 23 Jul 2016 04:38:42 +0700
>From:Robert Elz 
>Message-ID:  <20406.1469223...@andromeda.noi.kre.to>
> 
>  | That /dev/null turned into a regular file is another bug [...]
>  | (This turns out to be a bug in MAKEDEV [...]
> 
> Actually, not, it must be elsewhere, or as a result of something else.
> 
> kre
> 
> 
> 


Could it perhaps come from the ( set -o tabcomplete 2>/dev/null ) in /etc/shrc?

- Michael

Re: repeated failure to properly shutdown

[Robert Elz]

Date:Sat, 23 Jul 2016 04:38:42 +0700
From:Robert Elz 
Message-ID:  <20406.1469223...@andromeda.noi.kre.to>

  | That /dev/null turned into a regular file is another bug [...]
  | (This turns out to be a bug in MAKEDEV [...]

Actually, not, it must be elsewhere, or as a result of something else.

kre

Re: repeated failure to properly shutdown

[Rhialto]

On Sat 23 Jul 2016 at 04:38:42 +0700, Robert Elz wrote:
> That /dev/null turned into a regular file is another bug - it is being
> created before the tmpfs /dev is made, I have seen that before as well,
> but just corrected and ignored the problem until now.   

Similarly, I noticed that if /var is a tmpfs (or any initially empty
directory really), then /etc/rc.d/mountcritlocal fails because it wants
to cd to /var/run and that has not been created (if that ever happens).

-Olaf.
-- 
___ Olaf 'Rhialto' Seibert  -- Wayland: Those who don't understand X
\X/ rhialto/at/xs4all.nl-- are condemned to reinvent it. Poorly.


signature.asc
Description: PGP signature

Re: repeated failure to properly shutdown

[Robert Elz]

Date:Fri, 22 Jul 2016 12:52:58 -0700
From:bch 
Message-ID:  


  | I think that biggest concern (unclean shutdown/reboot) is solved (collision
  | of /dev and a tmpfs mount, caused by default behavior of init in face of
  | missing /dev/console).

Yes, and now we know what the cause is, we should be able to duplicate the
problem, and work out what is really happening.   The system is supposed to
work with a tmpfs /dev, it should not panic during shutdown.   What's more,
this panic is probably not related to it being /dev - any tmpfs mounted
with -o union on a mount point that is using WAPBL (-o log) will probably
panic the same way.

  | This disk was prepared remotely (I.e. from another running NetBSD box) by
  | partitioning the disk (disklabel), formatting (newfs), then mounting all
  | partitions appropriately under /mnt and running ./build.sh ... install=/mnt

That builds enough for the system to install, but it does not make a
fully runnable system - there's more that sysinst  normally does (like
populating /dev - but also making a basic rc.conf (including all network
config, setting hostname) and fstab, that don't get built by build.sh
either (and nor should they).   Those you must have done later.  Running
MAKEDEV is just another of the steps that one needs to perform.

That /dev/null turned into a regular file is another bug - it is being
created before the tmpfs /dev is made, I have seen that before as well,
but just corrected and ignored the problem until now.   (This turns out
to be a bug in MAKEDEV which is run by init to make the tmpfs /dev when
/dev/console is not present.)

Your solution to that was correct.   (MAKEDEV did not fix it as it never
replaces anything that already exists, only makes what does not - even if
what exists is nonsense, and even if it created the nonsense itself.)

It is good that your problems are overcome now - and thanks for bring it to
our attention, and for being willing to suffer through getting enough info
to allow the problem to be better understood.

kre

Re: repeated failure to properly shutdown

[Martin Husemann]

On Fri, Jul 22, 2016 at 11:37:56AM -0700, bch wrote:
> How does that happen, how does one fix it ?

It is created by init if there is no /dev/console.

Boot some install media, mount your root file system (say on /mnt)
then: 

cd /mnt/dev
sh MAKEDEV all

(hoping there is a MAKEDEV script there, if not: extract it from etc.tgz
from the install sets)

Then reboot and check mount again.

Martin

Re: repeated failure to properly shutdown

[bch]

Wow -- there -is- a tmpfs on /dev

kamloops# mount
/dev/wd0a on / type ffs (log, local)
->  tmpfs on /dev type tmpfs (union, local)
/dev/wd0e on /var type ffs (log, local)
/dev/wd0f on /usr type ffs (log, local)
/dev/wd0g on /home type ffs (log, local)
kernfs on /kern type kernfs (local)
ptyfs on /dev/pts type ptyfs (local)
procfs on /proc type procfs (local)
tmpfs on /var/shm type tmpfs (local)


But no entry for it in fstab...

# NetBSD /etc/fstab
# See /usr/share/examples/fstab/ for more examples.
/dev/wd0a   /   ffs rw,log   1 1
/dev/wd0b   noneswapsw,dp0 0
/dev/wd0f   /usrffs rw,log   1 2
/dev/wd0e   /varffs rw,log   1 2
/dev/wd0g   /home   ffs rw,log   1 2
kernfs  /kern   kernfs  rw
ptyfs   /dev/ptsptyfs   rw
procfs  /proc   procfs  rw
/dev/cd0a   /cdrom  cd9660  ro,noauto

tmpfs   /var/shmtmpfs   rw,-m1777,-sram%25



How does that happen, how does one fix it ?




On 7/22/16, Ian D. Leroux  wrote:
>
>
> On Fri, Jul 22, 2016, at 14:00, Robert Elz wrote:
>> Date:Fri, 22 Jul 2016 07:11:50 -0400 From:"Ian D.
>> Leroux"  Message-ID:
>> <20160722071150.5248712b562feea8d5c89...@fastmail.fm>
>>
>>   | Might this be a good moment to test them out and commit them?
>>
>> Perhaps, but not really as a fix for the current problem -- we already
>> know, from what we have been told, that not doing the tmpfs umount
>> avoids the crash ... what I, at least, would like to find is why the
>> crash happens at all, rather than just work around it.
>
> Fair enough.
>
>> That won't make umounting a tmpfs /dev any more rational to do though
>> (but just a tmpfs that happens to contain a device node is perhaps not
>> the right test for what to avoid, and manual specification when that
>> fails to DTRT isn't a great alternative.)
>
> I'm not sure there *is* a truly correct test for what to avoid, given
> the nature of what's being done at swapoff, but there may well be better
> heuristics.  I don't want to derail this thread though, so we can take
> that up separately at a later date.
>
> Good luck fixing the crash!
>
> -- IDL
>

Re: repeated failure to properly shutdown

[Ian D. Leroux]

On Fri, Jul 22, 2016, at 14:00, Robert Elz wrote:
> Date:Fri, 22 Jul 2016 07:11:50 -0400 From:"Ian D.
> Leroux"  Message-ID:
> <20160722071150.5248712b562feea8d5c89...@fastmail.fm>
>
>   | Might this be a good moment to test them out and commit them?
>
> Perhaps, but not really as a fix for the current problem -- we already
> know, from what we have been told, that not doing the tmpfs umount
> avoids the crash ... what I, at least, would like to find is why the
> crash happens at all, rather than just work around it.

Fair enough.

> That won't make umounting a tmpfs /dev any more rational to do though
> (but just a tmpfs that happens to contain a device node is perhaps not
> the right test for what to avoid, and manual specification when that
> fails to DTRT isn't a great alternative.)

I'm not sure there *is* a truly correct test for what to avoid, given
the nature of what's being done at swapoff, but there may well be better
heuristics.  I don't want to derail this thread though, so we can take
that up separately at a later date.

Good luck fixing the crash!

-- IDL

Re: repeated failure to properly shutdown

[Robert Elz]

Date:Fri, 22 Jul 2016 07:11:50 -0400
From:"Ian D. Leroux" 
Message-ID:  <20160722071150.5248712b562feea8d5c89...@fastmail.fm>

  | Might this be a good moment to test them out and commit them?

Perhaps, but not really as a fix for the current problem -- we already
know, from what we have been told, that not doing the tmpfs umount
avoids the crash ... what I, at least, would like to find is why the
crash happens at all, rather than just work around it.

If it turns out that the tmpfs being umounted is /dev (and not /var/shm
which is the only tmpfs in Brad's fstab - but which is unlikely to have
any union mounts anywhere near it - and not something else that is being
mounted some other way) then we should be able to reproduce the exact
environment, and work out why the crash happens, and then fix it.

That won't make umounting a tmpfs /dev any more rational to do though
(but just a tmpfs that happens to contain a device node is perhaps not
the right test for what to avoid, and manual specification when that fails
to DTRT isn't a great alternative.)

kre

Re: repeated failure to properly shutdown

[Robert Elz]

Date:Fri, 22 Jul 2016 00:33:01 -0700
From:bch 
Message-ID:  


  | Confirm this stack frame is the/a one we care about?

It looks right, yes, though one level further up should give the
same results (vp is passed in as a param, so the same one exists up one level.)

kre

Re: repeated failure to properly shutdown

[Ian D. Leroux]

On Fri, 22 Jul 2016 16:57:08 +0700 Robert Elz  wrote:
> "J. Hannken-Illjes"  said:
> 
>   | No populated "/dev" so it uses dev on tmpfs? 
> 
> Ah yes, very possible - the output from mount will tell us that, but I
> remember earlier reports of problems from unmounting (or attempting
> to) an unmount of /dev (and hardly surprising really.)

Tangentially, I wrote (some of) those earlier reports of problems with
unmounting /dev at shutdown and my patches to fix the prolblem are
still sitting uncommitted in bin/51019.  Might this be a good moment to
test them out and commit them?  If it doesn't fix OP's problem, it
would at least rule out one cause.

-- IDL

Re: repeated failure to properly shutdown

[bch]

I can do that tomorrow, yes. Confirm this stack frame is the/a one we care
about?

Regards,

-bch
On Jul 21, 2016 11:42 PM, "Martin Husemann"  wrote:

On Thu, Jul 21, 2016 at 04:38:57PM -0700, bch wrote:
> and the v_mount refcounts and flags are:
>
> (gdb) print vp->v_mount
> $2 = (struct mount *) 0xfe81081c2008
> (gdb) print vp->v_mount->mnt_refcnt
> $3 = 2501
> (gdb) print vp->v_mount->mnt_flag
> $4 = 4128
> (gdb)

can you also show

 print *vp
 print *vp->v_mount

please?

Thanks,

Martin

Re: repeated failure to properly shutdown

[Robert Elz]

Date:Fri, 22 Jul 2016 08:45:44 +
From:co...@sdf.org
Message-ID:  <20160722084544.ga14...@sdf.org>

  | probably good to remember that it's also saying it's double freed.
  | is it garbage data because it was freed before?

Perhaps, we will get a better idea when we see the full struct mount that
Martin requested.   But I doubt that was garbage, it is too "reasonable"
a value, for some definition of reasonable.


"J. Hannken-Illjes"  said:

  | No populated "/dev" so it uses dev on tmpfs? 

Ah yes, very possible - the output from mount will tell us that, but I
remember earlier reports of problems from unmounting (or attempting to)
an unmount of /dev (and hardly surprising really.)   At least it looks as
if we might be getting closer to an understanding of the setup that causes
the problem so it can be duplicated, and debugged quicker that once a
day turnaround of these e-mail messages...

kre

Re: repeated failure to properly shutdown

[J. Hannken-Illjes]

> On 22 Jul 2016, at 10:39, Robert Elz  wrote:
> 
>Date:Thu, 21 Jul 2016 16:38:57 -0700
>From:bch 
>Message-ID:  
> 

Re: repeated failure to properly shutdown

[coypu]

On Fri, Jul 22, 2016 at 03:39:26PM +0700, Robert Elz wrote:
> Date:Thu, 21 Jul 2016 16:38:57 -0700
> From:bch 
> Message-ID:  
> 

Re: repeated failure to properly shutdown

[Robert Elz]

Date:Thu, 21 Jul 2016 16:38:57 -0700
From:bch 
Message-ID:  

Re: repeated failure to properly shutdown

[Robert Swindells]

bch  wrote:
>I have kernel cores, but I'm not as familiar w/ crash(8) as gdb.

You can use gdb instead of crash if you want:

Re: repeated failure to properly shutdown

[bch]

On 7/21/16, bch  wrote:
> Alright -- I'm in (recompiled kernel with proper options, induced
> crash, loaded w/ gdb). What frame and data are we interested in ?


OK -- I roughly reviewed the mail thread and it looks (to me) like frame 6:

> #6  0x808ca91d in vclean (vp=vp@entry=0xfe8108412008) at
> /usr/src/sys/kern/vfs_vnode.c:1032

could be interesting -- so I looked at it's vnode_t (original argument
to func()) and the v_tag for it is (transcribed):
VT_TMPFS

and the v_mount refcounts and flags are:

(gdb) print vp->v_mount
$2 = (struct mount *) 0xfe81081c2008
(gdb) print vp->v_mount->mnt_refcnt
$3 = 2501
(gdb) print vp->v_mount->mnt_flag
$4 = 4128
(gdb)


There's much more to look at in the v_mount -- I'll wait for direction
to find out what's "interesting", if this is even the stack frame we
care about...


Cheers,

-bch

>
> /usr/obj/sys/arch/amd64/compile/GENERIC
> kamloops# gdb ./netbsd.gdb
> GNU gdb (GDB) 7.10.1
> Copyright (C) 2015 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64--netbsd".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> .
> Find the GDB manual and other documentation resources online at:
> .
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from ./netbsd.gdb...done.
> (gdb) target kvm /var/crash/netbsd.15.core
> 0x80119935 in cpu_reboot (howto=howto@entry=260,
> bootstr=bootstr@entry=0x0) at
> /usr/src/sys/arch/amd64/amd64/machdep.c:672
> 672 dumpsys();
> (gdb) bt
> #0  0x80119935 in cpu_reboot (howto=howto@entry=260,
> bootstr=bootstr@entry=0x0) at
> /usr/src/sys/arch/amd64/amd64/machdep.c:672
> #1  0x80874aac in vpanic (fmt=0x80f85c20 "kernel
> %sassertion \"%s\" failed: file \"%s\", line %d double free of buffer?
> bp=%p, b_freelistindex=%d\n", ap=ap@entry=0xfe8041106b98)
> at /usr/src/sys/kern/subr_prf.c:342
> #2  0x80bb59f5 in kern_assert
> (fmt=fmt@entry=0x80f85c20 "kernel %sassertion \"%s\" failed:
> file \"%s\", line %d double free of buffer? bp=%p,
> b_freelistindex=%d\n")
> at /usr/src/sys/lib/libkern/kern_assert.c:51
> #3  0x808b2dcd in binstailfree (dp=0x8130c9c0
> , bp=0xfe811d4245a0) at /usr/src/sys/kern/vfs_bio.c:333
> #4  brelsel (bp=0xfe811d4245a0, set=) at
> /usr/src/sys/kern/vfs_bio.c:1092
> #5  0x808d0786 in wapbl_discard (wl=0xfe81081e2e08) at
> /usr/src/sys/kern/vfs_wapbl.c:661
> #6  0x808ca91d in vclean (vp=vp@entry=0xfe8108412008) at
> /usr/src/sys/kern/vfs_vnode.c:1032
> #7  0x808cc57d in vgone (vp=vp@entry=0xfe8108412008) at
> /usr/src/sys/kern/vfs_vnode.c:1152
> #8  0x808be525 in vflush (mp=mp@entry=0xfe81081c2008,
> skipvp=skipvp@entry=0x0, flags=flags@entry=2) at
> /usr/src/sys/kern/vfs_mount.c:531
> #9  0x80735844 in tmpfs_unmount (mp=0xfe81081c2008,
> mntflags=524288) at /usr/src/sys/fs/tmpfs/tmpfs_vfsops.c:243
> #10 0x808c1d95 in VFS_UNMOUNT (mp=mp@entry=0xfe81081c2008,
> a=a@entry=524288) at /usr/src/sys/kern/vfs_subr.c:1296
> #11 0x808be69d in dounmount (mp=mp@entry=0xfe81081c2008,
> flags=524288, l=l@entry=0xfe811ae34880) at
> /usr/src/sys/kern/vfs_mount.c:856
> #12 0x808c5155 in sys_unmount (l=0xfe811ae34880,
> uap=0xfe8041106f00, retval=) at
> /usr/src/sys/kern/vfs_syscalls.c:617
> #13 0x8013e6ac in sy_call (rval=0xfe8041106eb0,
> uap=0xfe8041106f00, l=0xfe811ae34880, sy=0x811ca9d0
> ) at /usr/src/sys/sys/syscallvar.h:65
> #14 sy_invoke (code=22, rval=0xfe8041106eb0,
> uap=0xfe8041106f00, l=0xfe811ae34880, sy=0x811ca9d0
> ) at /usr/src/sys/sys/syscallvar.h:94
> #15 syscall (frame=0xfe8041106f00) at
> /usr/src/sys/arch/x86/x86/syscall.c:156
> #16 0x80100761 in Xsyscall ()
>
>
> On 7/21/16, Robert Swindells  wrote:
>>
>> bch  wrote:
>>>I have kernel cores, but I'm not as familiar w/ crash(8) as gdb.
>>
>> You can use gdb instead of crash if you want:
>>
>> 
>>
>

Re: repeated failure to properly shutdown

[bch]

Alright -- I'm in (recompiled kernel with proper options, induced
crash, loaded w/ gdb). What frame and data are we interested in ?


-bch



/usr/obj/sys/arch/amd64/compile/GENERIC
kamloops# gdb ./netbsd.gdb
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64--netbsd".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./netbsd.gdb...done.
(gdb) target kvm /var/crash/netbsd.15.core
0x80119935 in cpu_reboot (howto=howto@entry=260,
bootstr=bootstr@entry=0x0) at
/usr/src/sys/arch/amd64/amd64/machdep.c:672
672 dumpsys();
(gdb) bt
#0  0x80119935 in cpu_reboot (howto=howto@entry=260,
bootstr=bootstr@entry=0x0) at
/usr/src/sys/arch/amd64/amd64/machdep.c:672
#1  0x80874aac in vpanic (fmt=0x80f85c20 "kernel
%sassertion \"%s\" failed: file \"%s\", line %d double free of buffer?
bp=%p, b_freelistindex=%d\n", ap=ap@entry=0xfe8041106b98)
at /usr/src/sys/kern/subr_prf.c:342
#2  0x80bb59f5 in kern_assert
(fmt=fmt@entry=0x80f85c20 "kernel %sassertion \"%s\" failed:
file \"%s\", line %d double free of buffer? bp=%p,
b_freelistindex=%d\n")
at /usr/src/sys/lib/libkern/kern_assert.c:51
#3  0x808b2dcd in binstailfree (dp=0x8130c9c0
, bp=0xfe811d4245a0) at /usr/src/sys/kern/vfs_bio.c:333
#4  brelsel (bp=0xfe811d4245a0, set=) at
/usr/src/sys/kern/vfs_bio.c:1092
#5  0x808d0786 in wapbl_discard (wl=0xfe81081e2e08) at
/usr/src/sys/kern/vfs_wapbl.c:661
#6  0x808ca91d in vclean (vp=vp@entry=0xfe8108412008) at
/usr/src/sys/kern/vfs_vnode.c:1032
#7  0x808cc57d in vgone (vp=vp@entry=0xfe8108412008) at
/usr/src/sys/kern/vfs_vnode.c:1152
#8  0x808be525 in vflush (mp=mp@entry=0xfe81081c2008,
skipvp=skipvp@entry=0x0, flags=flags@entry=2) at
/usr/src/sys/kern/vfs_mount.c:531
#9  0x80735844 in tmpfs_unmount (mp=0xfe81081c2008,
mntflags=524288) at /usr/src/sys/fs/tmpfs/tmpfs_vfsops.c:243
#10 0x808c1d95 in VFS_UNMOUNT (mp=mp@entry=0xfe81081c2008,
a=a@entry=524288) at /usr/src/sys/kern/vfs_subr.c:1296
#11 0x808be69d in dounmount (mp=mp@entry=0xfe81081c2008,
flags=524288, l=l@entry=0xfe811ae34880) at
/usr/src/sys/kern/vfs_mount.c:856
#12 0x808c5155 in sys_unmount (l=0xfe811ae34880,
uap=0xfe8041106f00, retval=) at
/usr/src/sys/kern/vfs_syscalls.c:617
#13 0x8013e6ac in sy_call (rval=0xfe8041106eb0,
uap=0xfe8041106f00, l=0xfe811ae34880, sy=0x811ca9d0
) at /usr/src/sys/sys/syscallvar.h:65
#14 sy_invoke (code=22, rval=0xfe8041106eb0,
uap=0xfe8041106f00, l=0xfe811ae34880, sy=0x811ca9d0
) at /usr/src/sys/sys/syscallvar.h:94
#15 syscall (frame=0xfe8041106f00) at
/usr/src/sys/arch/x86/x86/syscall.c:156
#16 0x80100761 in Xsyscall ()


On 7/21/16, Robert Swindells  wrote:
>
> bch  wrote:
>>I have kernel cores, but I'm not as familiar w/ crash(8) as gdb.
>
> You can use gdb instead of crash if you want:
>
> 
>

Re: repeated failure to properly shutdown

[bch]

On Jul 21, 2016 1:08 PM, "J. Hannken-Illjes"  wrote:
>
>
> > On 21 Jul 2016, at 19:26, co...@sdf.org wrote:
> >
> > On Thu, Jul 21, 2016 at 10:15:32AM -0700, bch wrote:
> >> Jul 20 23:55:59 kamloops /netbsd: wapbl_discard() at 
> >> netbsd:wapbl_discard+0x20c
> >> Jul 20 23:55:59 kamloops /netbsd: vclean() at netbsd:vclean+0x2ae
> > ...
> >> Jul 20 23:55:59 kamloops /netbsd: tmpfs_unmount() at 
> >> netbsd:tmpfs_unmount+0x2f
> >
> > For some reason this condition is met:
> > wapbl_vphaswapbl(vp)
> >
> > but why?
>
> The contents of this "struct vnode", especially its "v_tag"
> and "v_mount" could help.

I have kernel cores, but I'm not as familiar w/ crash(8) as gdb.

Initial backtrace looks like this:


Crash version 7.99.34, image version 7.99.34.
System panicked: kernel diagnostic assertion "bp->b_freelistindex ==
-1" failed: file "/usr/src/sys/kern/vfs_bio.c", line 334 double free
of buffer? bp=0xfe8108689120, b_freelistindex=0

Backtrace from time of crash is available.
_KERNEL_OPT_NARCNET() at 0
?() at fe8119892158
vpanic() at vpanic+0x149
cd_play_msf() at cd_play_msf
brelsel() at brelsel+0x5f0
wapbl_discard() at wapbl_discard+0x20c
vclean() at vclean+0x2ae
vgone() at vgone+0x4b
vflush() at vflush+0xcb
tmpfs_unmount() at tmpfs_unmount+0x2f
VFS_UNMOUNT() at VFS_UNMOUNT+0x24
dounmount() at dounmount+0xbc
sys_unmount() at sys_unmount+0x150
syscall() at syscall+0x15b
--- syscall (number 22) ---
7bfd4dc3ddda:



> --
> J. Hannken-Illjes - hann...@eis.cs.tu-bs.de - TU Braunschweig (Germany)
>

Re: repeated failure to properly shutdown

[J. Hannken-Illjes]

> On 21 Jul 2016, at 19:26, co...@sdf.org wrote:
> 
> On Thu, Jul 21, 2016 at 10:15:32AM -0700, bch wrote:
>> Jul 20 23:55:59 kamloops /netbsd: wapbl_discard() at 
>> netbsd:wapbl_discard+0x20c
>> Jul 20 23:55:59 kamloops /netbsd: vclean() at netbsd:vclean+0x2ae
> ...
>> Jul 20 23:55:59 kamloops /netbsd: tmpfs_unmount() at 
>> netbsd:tmpfs_unmount+0x2f
> 
> For some reason this condition is met:
> wapbl_vphaswapbl(vp)
> 
> but why?

The contents of this "struct vnode", especially its "v_tag"
and "v_mount" could help.

--
J. Hannken-Illjes - hann...@eis.cs.tu-bs.de - TU Braunschweig (Germany)

Re: repeated failure to properly shutdown

[bch]

Well -- that fixed the fault-on-exit (and therefore the dirty
filesystems on reboot) -- now to figure out why the tmpfs is so
problematic for me...

Thanks!

-bch



On 7/21/16, Robert Swindells  wrote:
>
> bch  wrote:
>>On 7/21/16, Robert Elz  wrote:
>>> Do you perhaps have /tmp mounted as a tmpfs with -o log ?
>>> (Can you show us /etc/fstab in its entirety?)
>>
>>For your viewing pleasure...
>>
>>===
>># NetBSD /etc/fstab
>># See /usr/share/examples/fstab/ for more examples.
>>/dev/wd0a /   ffs rw,log   1 1
>>/dev/wd0b noneswapsw,dp0 0
>>/dev/wd0f /usrffs rw,log   1 2
>>/dev/wd0e /varffs rw,log   1 2
>>/dev/wd0g /home   ffs rw,log   1 2
>>kernfs/kern   kernfs  rw
>>ptyfs /dev/ptsptyfs   rw
>>procfs/proc   procfs  rw
>>/dev/cd0a /cdrom  cd9660  ro,noauto
>>
>>tmpfs /var/shmtmpfs   rw,-m1777,-sram%25
>>===
>
> Maybe try the following in /etc/rc.d/swap1:
>
> --- /u1/src/etc/rc.d/swap1  2015-04-21 08:50:23.741934176 +0100
> +++ /etc/rc.d/swap1 2016-04-25 23:20:07.391864699 +0100
> @@ -31,8 +31,8 @@
>  swap1_stop()
>  {
> if checkyesno swapoff || [ -n "$rc_force" ]; then
> -   echo "Forcibly unmounting tmpfs filesystems"
> -   umount -aft tmpfs
> +   # echo "Forcibly unmounting tmpfs filesystems"
> +   # umount -aft tmpfs
> echo "Removing block-type swap devices"
> swapctl -U -t blk || [ $? = 2 ]
> fi
>
>
>
>

Re: repeated failure to properly shutdown

[Robert Swindells]

bch  wrote:
>On 7/21/16, Robert Elz  wrote:
>> Do you perhaps have /tmp mounted as a tmpfs with -o log ?
>> (Can you show us /etc/fstab in its entirety?)
>
>For your viewing pleasure...
>
>===
># NetBSD /etc/fstab
># See /usr/share/examples/fstab/ for more examples.
>/dev/wd0a  /   ffs rw,log   1 1
>/dev/wd0b  noneswapsw,dp0 0
>/dev/wd0f  /usrffs rw,log   1 2
>/dev/wd0e  /varffs rw,log   1 2
>/dev/wd0g  /home   ffs rw,log   1 2
>kernfs /kern   kernfs  rw
>ptyfs  /dev/ptsptyfs   rw
>procfs /proc   procfs  rw
>/dev/cd0a  /cdrom  cd9660  ro,noauto
>
>tmpfs  /var/shmtmpfs   rw,-m1777,-sram%25
>===

Maybe try the following in /etc/rc.d/swap1:

--- /u1/src/etc/rc.d/swap1  2015-04-21 08:50:23.741934176 +0100
+++ /etc/rc.d/swap1 2016-04-25 23:20:07.391864699 +0100
@@ -31,8 +31,8 @@
 swap1_stop()
 {
if checkyesno swapoff || [ -n "$rc_force" ]; then
-   echo "Forcibly unmounting tmpfs filesystems"
-   umount -aft tmpfs
+   # echo "Forcibly unmounting tmpfs filesystems"
+   # umount -aft tmpfs
echo "Removing block-type swap devices"
swapctl -U -t blk || [ $? = 2 ]
fi

Re: repeated failure to properly shutdown

[bch]

On 7/21/16, Robert Elz  wrote:
> Do you perhaps have /tmp mounted as a tmpfs with -o log ?
> (Can you show us /etc/fstab in its entirety?)


For your viewing pleasure...


===
# NetBSD /etc/fstab
# See /usr/share/examples/fstab/ for more examples.
/dev/wd0a   /   ffs rw,log   1 1
/dev/wd0b   noneswapsw,dp0 0
/dev/wd0f   /usrffs rw,log   1 2
/dev/wd0e   /varffs rw,log   1 2
/dev/wd0g   /home   ffs rw,log   1 2
kernfs  /kern   kernfs  rw
ptyfs   /dev/ptsptyfs   rw
procfs  /proc   procfs  rw
/dev/cd0a   /cdrom  cd9660  ro,noauto

tmpfs   /var/shmtmpfs   rw,-m1777,-sram%25
===



> kre
>
>

Re: repeated failure to properly shutdown

[coypu]

On Thu, Jul 21, 2016 at 10:15:32AM -0700, bch wrote:
> Jul 20 23:55:59 kamloops /netbsd: wapbl_discard() at 
> netbsd:wapbl_discard+0x20c
> Jul 20 23:55:59 kamloops /netbsd: vclean() at netbsd:vclean+0x2ae
...
> Jul 20 23:55:59 kamloops /netbsd: tmpfs_unmount() at netbsd:tmpfs_unmount+0x2f

For some reason this condition is met:
wapbl_vphaswapbl(vp)

but why?

Re: repeated failure to properly shutdown

[bch]

More information...


Jul 20 23:55:59 kamloops /netbsd: panic: kernel diagnostic assertion
"bp->b_freelistindex == -1" failed: file
"/usr/src/sys/kern/vfs_bio.c", line 334 double free of buffer?
bp=0xfe811ca935a0, b_freelistindex=0
Jul 20 23:55:59 kamloops /netbsd:
Jul 20 23:55:59 kamloops /netbsd: cpu3: Begin traceback...
Jul 20 23:55:59 kamloops /netbsd: vpanic() at netbsd:vpanic+0x140
Jul 20 23:55:59 kamloops /netbsd: cd_play_msf() at netbsd:cd_play_msf
Jul 20 23:55:59 kamloops /netbsd: brelsel() at netbsd:brelsel+0x5f0
Jul 20 23:55:59 kamloops /netbsd: wapbl_discard() at netbsd:wapbl_discard+0x20c
Jul 20 23:55:59 kamloops /netbsd: vclean() at netbsd:vclean+0x2ae
Jul 20 23:55:59 kamloops /netbsd: vgone() at netbsd:vgone+0x4b
Jul 20 23:55:59 kamloops /netbsd: vflush() at netbsd:vflush+0xcb
Jul 20 23:55:59 kamloops /netbsd: tmpfs_unmount() at netbsd:tmpfs_unmount+0x2f
Jul 20 23:55:59 kamloops /netbsd: VFS_UNMOUNT() at netbsd:VFS_UNMOUNT+0x24
Jul 20 23:55:59 kamloops /netbsd: dounmount() at netbsd:dounmount+0xbc
Jul 20 23:55:59 kamloops /netbsd: sys_unmount() at netbsd:sys_unmount+0x150
Jul 20 23:55:59 kamloops /netbsd: syscall() at netbsd:syscall+0x15b
Jul 20 23:55:59 kamloops /netbsd: --- syscall (number 22) ---
Jul 20 23:55:59 kamloops /netbsd: 76088863ddda:
Jul 20 23:55:59 kamloops /netbsd: cpu3: End traceback...
Jul 20 23:55:59 kamloops /netbsd:
Jul 20 23:55:59 kamloops /netbsd: dumping to dev 0,1 (offset=8, size=1018685):
Jul 20 23:55:59 kamloops /netbsd: dump Copyright (c) 1996, 1997, 1998,
1999, 2000, 2001, 2002, 2003, 2004, 2005,
Jul 20 23:55:59 kamloops /netbsd:2006, 2007, 2008, 2009, 2010,
2011, 2012, 2013, 2014, 2015, 2016
Jul 20 23:55:59 kamloops /netbsd:The NetBSD Foundation, Inc.  All
rights reserved.
Jul 20 23:55:59 kamloops /netbsd: Copyright (c) 1982, 1986, 1989, 1991, 1993
Jul 20 23:55:59 kamloops /netbsd:The Regents of the University of
California.  All rights reserved.


disklabel:

# /dev/rwd0:
type: SCSI
disk: 2105
label: fictitious
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 16
sectors/cylinder: 1008
cylinders: 16383
total sectors: 1953525168
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0   # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0

7 partitions:
#sizeoffset fstype [fsize bsize cpg/sgs]
 a:   1638400 0 4.2BSD  0 0 0  # (Cyl.  0 -   1625*)
 b:   4096000   1638400   swap # (Cyl.   1625*-   5688*)
 d: 1953525168 0 unused  0 0# (Cyl.  0
- 1938020)
 e:   3072000   5734400 4.2BSD  0 0 0  # (Cyl.   5688*-   8736*)
 f: 167772160   8806400 4.2BSD  0 0 0  # (Cyl.   8736*- 175177*)
 g: 1776946608 176578560 4.2BSD  0 0 0  # (Cyl.
175177*- 1938020)
# microseconds
track-to-track seek: 0  # microseconds
drivedata: 0

7 partitions:
#sizeoffset fstype [fsize bsize cpg/sgs]
 a:   1638400 0 4.2BSD  0 0 0  # (Cyl.  0 -   1625*)
 b:   4096000   1638400   swap # (Cyl.   1625*-   5688*)
 d: 1953525168 0 unused  0 0# (Cyl.  0
- 1938020)
 e:   3072000   5734400 4.2BSD  0 0 0  # (Cyl.   5688*-   8736*)
 f: 167772160   8806400 4.2BSD  0 0 0  # (Cyl.   8736*- 175177*)
 g: 1776946608 176578560 4.2BSD  0 0 0  # (Cyl.
175177*- 1938020)# microseconds
track-to-track seek: 0  # microseconds
drivedata: 0

7 partitions:
#sizeoffset fstype [fsize bsize cpg/sgs]
 a:   1638400 0 4.2BSD  0 0 0  # (Cyl.  0 -   1625*)
 b:   4096000   1638400   swap # (Cyl.   1625*-   5688*)
 d: 1953525168 0 unused  0 0# (Cyl.  0
- 1938020)
 e:   3072000   5734400 4.2BSD  0 0 0  # (Cyl.   5688*-   8736*)
 f: 167772160   8806400 4.2BSD  0   # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0

7 partitions:
#sizeoffset fstype [fsize bsize cpg/sgs]
 a:   1638400 0 4.2BSD  0 0 0  # (Cyl.  0 -   1625*)
 b:   4096000   1638400   swap # (Cyl.   1625*-   5688*)
 d: 1953525168 0 unused  0 0# (Cyl.  0
- 1938020)
 e:   3072000   5734400 4.2BSD  0 0 0  # (Cyl.   5688*-   8736*)
 f: 167772160   8806400 4.2BSD  0 0 0  # (Cyl.   8736*- 175177*)
 g: 1776946608 176578560 4.2BSD  0 0 0  # (Cyl.
175177*- 1938020)# microseconds
track-to-track seek: 0  # microseconds
drivedata: 0

7 partitions:
#sizeoffset fstype [fsize bsize cpg/sgs]
 a:   1638400 0 4.2BSD  0 0 0  # (Cyl.  0 -   1625*)
 b:   4096000   1638400   swap # (Cyl.   1625*-   5688*)
 d: 1953525168 0 unused  0 0# (Cyl.  0
- 1938020)
 

Re: repeated failure to properly shutdown

[Robert Elz]

Do you perhaps have /tmp mounted as a tmpfs with -o log ?
(Can you show us /etc/fstab in its entirety?)

kre

Re: repeated failure to properly shutdown

[bch]

Further, here is  a copy of my superblock info (for /, in this case)
for review if necessary. I'm running w/ the "log" option in fstab.

kamloops$ dumpfs -s /
file system: /dev/rwd0a
format  FFSv1
endian  little-endian
magic   11954   timeWed Jul 20 20:52:58 2016
superblock location 8192id  [ 578ec737 4f3e0a17 ]
cylgrp  dynamic inodes  4.4BSD  sblock  FFSv2   fslevel 4
nbfree  76636   ndir948 nifree  196627  nffree  545
ncg 18  size819200  blocks  793983
bsize   8192shift   13  mask0xe000
fsize   1024shift   10  mask0xfc00
frag8   shift   3   fsbtodb 1
bpg 5689fpg 45512   ipg 11072
minfree 5%  optim   timemaxcontig 8 maxbpg  2048
symlinklen 60   contigsumsize 8
maxfilesize 0x400801017fff
nindir  2048inopb   64
avgfilesize 16384   avgfpdir 64
sblkno  16  cblkno  24  iblkno  32  dblkno  1416
sbsize  2048cgsize  8192
csaddr  1416cssize  1024
cgrotor 0   fmod0   ronly   0   clean   0x02
wapbl version 0x1   location 2  flags 0x0
wapbl loc0 822192   loc1 2048   loc2 512loc3 5
flags   wapbl
fsmnt   /
volname swuid   0


On 7/20/16, bch  wrote:
> Jul 20 13:53:02 kamloops /netbsd: panic: kernel diagnostic assertion
> "bp->b_freelistindex == -1" failed: file
> "/usr/src/sys/kern/vfs_bio.c", line 334 double free of buffer?
> bp=0xfe811c4807e8, b_freelistindex=0
> Jul 20 13:53:02 kamloops /netbsd:
> Jul 20 13:53:02 kamloops /netbsd: cpu3: Begin traceback...
> Jul 20 13:53:02 kamloops /netbsd: vpanic() at netbsd:vpanic+0x140
> Jul 20 13:53:02 kamloops /netbsd: cd_play_msf() at netbsd:cd_play_msf
> Jul 20 13:53:02 kamloops /netbsd: brelsel() at netbsd:brelsel+0x5f0
> Jul 20 13:53:02 kamloops /netbsd: wapbl_discard() at
> netbsd:wapbl_discard+0x20c
> Jul 20 13:53:02 kamloops /netbsd: vclean() at netbsd:vclean+0x2ae
> Jul 20 13:53:02 kamloops /netbsd: vgone() at netbsd:vgone+0x4b
> Jul 20 13:53:02 kamloops /netbsd: vflush() at netbsd:vflush+0xcb
> Jul 20 13:53:02 kamloops /netbsd: tmpfs_unmount() at
> netbsd:tmpfs_unmount+0x2f
> Jul 20 13:53:02 kamloops /netbsd: VFS_UNMOUNT() at netbsd:VFS_UNMOUNT+0x24
> Jul 20 13:53:02 kamloops /netbsd: dounmount() at netbsd:dounmount+0xbc
> Jul 20 13:53:02 kamloops /netbsd: sys_unmount() at netbsd:sys_unmount+0x150
> Jul 20 13:53:02 kamloops /netbsd: syscall() at netbsd:syscall+0x15b
> Jul 20 13:53:02 kamloops /netbsd: --- syscall (number 22) ---
> Jul 20 13:53:02 kamloops /netbsd: 7b202543ddda:
> Jul 20 13:53:02 kamloops /netbsd: cpu3: End traceback...
>

repeated failure to properly shutdown

[bch]

Jul 20 13:53:02 kamloops /netbsd: panic: kernel diagnostic assertion
"bp->b_freelistindex == -1" failed: file
"/usr/src/sys/kern/vfs_bio.c", line 334 double free of buffer?
bp=0xfe811c4807e8, b_freelistindex=0
Jul 20 13:53:02 kamloops /netbsd:
Jul 20 13:53:02 kamloops /netbsd: cpu3: Begin traceback...
Jul 20 13:53:02 kamloops /netbsd: vpanic() at netbsd:vpanic+0x140
Jul 20 13:53:02 kamloops /netbsd: cd_play_msf() at netbsd:cd_play_msf
Jul 20 13:53:02 kamloops /netbsd: brelsel() at netbsd:brelsel+0x5f0
Jul 20 13:53:02 kamloops /netbsd: wapbl_discard() at netbsd:wapbl_discard+0x20c
Jul 20 13:53:02 kamloops /netbsd: vclean() at netbsd:vclean+0x2ae
Jul 20 13:53:02 kamloops /netbsd: vgone() at netbsd:vgone+0x4b
Jul 20 13:53:02 kamloops /netbsd: vflush() at netbsd:vflush+0xcb
Jul 20 13:53:02 kamloops /netbsd: tmpfs_unmount() at netbsd:tmpfs_unmount+0x2f
Jul 20 13:53:02 kamloops /netbsd: VFS_UNMOUNT() at netbsd:VFS_UNMOUNT+0x24
Jul 20 13:53:02 kamloops /netbsd: dounmount() at netbsd:dounmount+0xbc
Jul 20 13:53:02 kamloops /netbsd: sys_unmount() at netbsd:sys_unmount+0x150
Jul 20 13:53:02 kamloops /netbsd: syscall() at netbsd:syscall+0x15b
Jul 20 13:53:02 kamloops /netbsd: --- syscall (number 22) ---
Jul 20 13:53:02 kamloops /netbsd: 7b202543ddda:
Jul 20 13:53:02 kamloops /netbsd: cpu3: End traceback...