Re: Hanging at shutdown with mystery "file system full" error

[Robert Elz]

Date:Thu, 22 Oct 2015 07:52:06 +0200
From:"Ian D. Leroux" 
Message-ID:  <20151022075206.995f2caa63b153174b7f7...@fastmail.fm>

  | That last part I don't understand, or rather I don't understand
  | why /etc/rc.d/swap1 should worry about that case.

I'll explain what I think the issue is, and then others can correct it...

  | I would have
  | thought that any subsidiary mount point, tmpfs or not, would prevent an
  | unmount of (say) /var.

It would.

  | If the rc system is meant to unmount all
  | filesystems at shutdown time, then it already needs some logic to do so
  | in the right order.

It does (reverse order of mounts).

  | How are tmpfs filesystems special in this regard?

Well

The problem is that it is permitted to swap onto files.   If we did away
with that, there would be no problems to solve.

But since we allow swapping to files (as distinct from disc partitions)
that swapping has to be disabled, before the filesystems on which the
files live can be unmounted.

When that swapping is disabled, anything that was swapped out there needs
to be swapped in again (at this stage, during rc.shutdown processing, the
kernel is not aware that the system is about to terminate, this is all just
using the same facility as any other removal of a swap file/device).

One of the (potentially) big users of space on swap, is files on a tmpfs.
In fact, during shutdown, it is probably the only significant user.

On systems with limited ram, reading all those files into mem can cause
memory exhaustion, and hangs...   That's kind of unfortunate, especially when
you're in a situation where all you are really wanting to do is trash
everything.

So, to avoid that, tmpfs filesystems are forcibly unmounted first, before
swap is disabled, then that user of swap space is gone - if we are really
in the process of shutting down, most user processes should have already been
terminated, so there should be very limited (if any) process consumption of
swap space, so disabling swap should then function normally.

Once swap is disabled (at least swap on files) the filesystems can all be
unmounted (the question of swap on a file on a tmpfs and what would happen to
that I have no idea about, but it is such an absurd concept that I doubt
anyone cares ... it should just be prohibited, and perhaps is.)

At least that's the idea, but /dev on tmpfs kind of screws with this model...

  | My understanding of the not-so-recent change to /etc/rc.d/swap1 is that
  | it is meant to ensure that there is swap space left for swapctl to
  | function.  That seems to me to be orthogonal to the question of
  | ensuring that all filesystems can be unmounted.

It it perhaps a little clearer now (assuming my perceptions about the
problem are correct.)

  | That sounds to me like there are several possibly-unrelated shutdown
  | hangs,

Could be ... or it is also possible that the /dev problem and no space
message you're seeing are co-incidental. and while annoying, not actually
related to the hang you're seeing.

And, please treat my report of a hang as no more than a rumour .. I know
a hang happened to me (once) but I have not investigated at all, and almost
anything might have been the cause.

  | To the extent that read-only-/ with tmpfs-/dev is a reasonable
  | configuration (and at least Joerg and I seem to think so),

Yes, so do I (though devfs would be better) - I didn't mean to suggest 
otherwise, just that tmpfs /dev is not mandatory for a read only root.
(and that since /dev consists of just the directory, and a whole bunch
of inodes, and needs nothing that actually consumes space - or more
than the directory itself, mfs would actually work for it, perhaps
petter than tmpfs ... but that is a whole separate discussion.)

  | then /etc/rc.d/swap1 needs a more precise notion of which filesystems
  | it can safely unmount to make space on swap.

One notion I had was to perhaps add an "permanent" flag to mount.
If set, no umount would ever be able to remove it (with or without -f).
(Of course, kernel shutdown still happens, syncing and marking the
filesystem clean - just as happens with root, the one current filesystem
that cannot be unmounted.)   That would avoid embedding magic into weird
places about what can, and cannot, safely be unmounted.

  | On systems like mine, which have way more swap than they need, it seems
  | practical just to omit the 'umount -aft tmpfs' from swap1_stop(), but
  | that doesn't address the problem that the change was initially meant to
  | solve.

You could certainly do that.   Perhaps replace it with a "rm -rf /tmp/*"
(repeated for each tmpfs that you have - other than /dev, though even
it could probably be cleaned safely that way at shutdown time.)

The rm should be enough to remove anything consuming space (probably needs
to also deal with . files, since rm -fr /tmp itself is probably not safe,
just in case /tmp was not mounted) - there should be no 

Re: Hanging at shutdown with mystery "file system full" error

[Ian D. Leroux]

On Thu, 22 Oct 2015 13:57:25 +0700 Robert Elz  wrote:

> Date:Thu, 22 Oct 2015 07:52:06 +0200
> From:"Ian D. Leroux" 
> Message-ID:  <20151022075206.995f2caa63b153174b7f7...@fastmail.fm>
>   | How are tmpfs filesystems special in this regard?
> 
> Well
> 
> The problem is that it is permitted to swap onto files.   If we did
> away with that, there would be no problems to solve.
>
> [detailed and patient explanation of the interaction between tmpfs
> filesystems and file-backed swap]

Thanks!  That helped a lot.

>   | That sounds to me like there are several possibly-unrelated
>   | shutdown hangs,
> 
> Could be ... or it is also possible that the /dev problem and no space
> message you're seeing are co-incidental. and while annoying, not
> actually related to the hang you're seeing.

Ok, so let's separate the issues.

1- With tmpfs-mounted /dev, /etc/rc.d/swap1 attempts to umount -f /dev
before userland is finished at shutdown time.  Whether or not this
causes the hang and error message I see, this is clearly not a desirable
outcome.  I think we understand what the problem is, and there are
workarounds for my case and most others I can think of, although it's
not clear what the right general fix is.  I'll file a PR for this, and
further discussion can go there or in another thread.

2- The system will sometimes hang after printing a kernel error message
of the general form "on /var: file system full", despite the fact
that /var is not full and that the immediate trigger for the hang, at
least in my case and apparently also in the case of Robert Swindells,
is unmounting a tmpfs filesystem that has nothing to do with /var.

To investigate the second  issue we need a way of triggering the hang
that is not as manifestly crazy as removing /dev, which seems to be the
only way I've ever triggered it (Robert Swindells seems to have found
another way, though it's not clear what).  I'll try populating an
on-disk /dev and then seeing if I still get a hang at shutdown.

--
IDL

Re: Hanging at shutdown with mystery "file system full" error

[D'Arcy J.M. Cain]

On Thu, 22 Oct 2015 01:12:51 +0200
Joerg Sonnenberger  wrote:
> > Have you considered the possibility that having your devices on a
> > mounted FS is just asking for trouble?  I'm not sure but I think
> > that I would be surprised if I made all my devices suddenly
> > disappear and it didn't hang.  Why do you have it on a mounted FS
> > anyway?
> 
> As written, that's done automatically by init if /dev is not populated
> with devices. That is a common setup for ramdisks and read-only root
> installations. It must not be broken.

I see.  So how does the system deal with devices suddenly disappearing
during shutdown?  Also, the OP is not using a R/O system (it's a
laptop) so if /dev is on a tmpfs then he must be doing it manually in
his fstab, right?  That's not the same as init doing it automatically.

-- 
D'Arcy J.M. Cain 
http://www.NetBSD.org/ IM:da...@vex.net

Re: Hanging at shutdown with mystery "file system full" error

[Robert Swindells]

"D'Arcy J.M. Cain"  wrote:
>On Thu, 22 Oct 2015 01:12:51 +0200
>Joerg Sonnenberger  wrote:
>> > Have you considered the possibility that having your devices on a
>> > mounted FS is just asking for trouble?  I'm not sure but I think
>> > that I would be surprised if I made all my devices suddenly
>> > disappear and it didn't hang.  Why do you have it on a mounted FS
>> > anyway?
>> 
>> As written, that's done automatically by init if /dev is not populated
>> with devices. That is a common setup for ramdisks and read-only root
>> installations. It must not be broken.
>
>I see.  So how does the system deal with devices suddenly disappearing
>during shutdown?  Also, the OP is not using a R/O system (it's a
>laptop) so if /dev is on a tmpfs then he must be doing it manually in
>his fstab, right?  That's not the same as init doing it automatically.

The OP did state that the root filesystem is read-only.

It should be easy enough to mount root writable, populate /dev on this
filesystem then switch back to mounting it read-only.

There must be more than one way that a tmpfs can be active at the point
where it is forceably unmounted though. I don't have /dev on tmpfs and
get exactly the same error message at shutdown.

Robert Swindells

Re: Hanging at shutdown with mystery "file system full" error

[Ian D. Leroux]

On Wed, 21 Oct 2015 11:12:31 +0200 "Ian D. Leroux"
 wrote:
> Tonight I'll try shutting down with various combinations of tmpfs
> filesystems, to see if I can actually narrow down the problem to a
> specific mount point.

As it turns out, I don't need shutdown to trigger the hang, umount is
enough.  Specifically,

umount -f /dev

Causes an immediate hang with the same kernel error message
(..., on /var: file system full), attributed this time to the login
command.  (Without the -f, umount sensibly complains that /dev is busy).

I can unmount /var/shm and /tmp (the other two tmpfs filesystems on
this machine) without causing an immediate hang (but also without
preventing a subsequent hang at shutdown time), so at least on my
machine the problem seems to be specifically with unmounting /dev.

The options as I currently see them are:
- Unmounting /dev during shutdown (specifically during swap1_stop()) is
expected to work.  Then we have a problem with umount or perhaps with
the exact timing of the call to umount.  Maybe the hang has to do with
unmounting a *busy* tmpfs filesystem?  That would explain why I see
this as a /dev-related problem (because /dev is the only tmpfs
filesystem I have that is still busy at shutdown time) while others see
a similar error without having a tmpfs /dev.

- /dev should never be unmounted, and /dev is allowed to be on tmpfs.
Then swap1_stop() should not forcibly unmount all tmpfs filesystems.
Would it make sense to replace "umount -aft tmpfs" with
"umount -at tmpfs", so as to remove only filesystems that aren't busy?

- /dev is not allowed to be on tmpfs.  Then I'm not seeing a bug, I'm
just using a bogus configuration.  Note, however, that the automatic
creation of a tmpfs-mounted /dev is documented in init(8), and it's the
only way I know of running with a read-only root partition.

I'm out of time for tonight, but the next thing I want to try is to
artificially make /tmp or /var/shm busy at unmount or shutdown time and
see if I can trigger a hang that way.

--
IDL

Re: Hanging at shutdown with mystery "file system full" error

[Robert Elz]

Date:Wed, 21 Oct 2015 23:07:53 +0200
From:"Ian D. Leroux" 
Message-ID:  <20151021230753.d8eab7389a3df445751e1...@fastmail.fm>

  | umount -f /dev
  | 
  | Causes an immediate hang with the same kernel error message
  | (..., on /var: file system full), attributed this time to the login
  | command.

Not sure exactly why the full message, but in a way that makes sense, forcibly
unmounting will invalidate any vnodes from the filesystem, which would include
those for the mount devices for all of  the (normal) filesystems - there'd
be no way to access their vops for performing any i/o.

  | The options as I currently see them are:
  | Maybe the hang has to do with unmounting a *busy* tmpfs filesystem?

You'd be able to test/provoke that by just opening a file on /tmp and
then umount -f'ing it.   I doubt there'd be a problem.

  | - /dev should never be unmounted, and /dev is allowed to be on tmpfs.

Those are probably both true.

  | Then swap1_stop() should not forcibly unmount all tmpfs filesystems.

Well, perhaps not that one.

  | Would it make sense to replace "umount -aft tmpfs" with
  | "umount -at tmpfs", so as to remove only filesystems that aren't busy?

No, other busy tmpfs's (such as a command hung waiting on some device that
isn't responding with its cwd there) wouldn't get unmounted.   Not
unmounting tnpfs at all wouldn't be so bad (their state after shutdown is
hardly important!) but that could prevent other filesystems being unmounted
(a mounted /var/shm would block a normal unmount of /var for example).

  | - /dev is not allowed to be on tmpfs.  Then I'm not seeing a bug, I'm
  | just using a bogus configuration.  Note, however, that the automatic
  | creation of a tmpfs-mounted /dev is documented in init(8),

Yes, at least until we have a devfs having init build /dev on a tmpfs
(or an mfs would work I guess, and maybe a better choice?) is a good
way to handle /dev directories trhat are obviously not populated correctly.

  | and it's the only way I know of running with a read-only root partition.

But that's not true, which you can easily see by just booting single
user, and mounting /tmp (and if needed) /usr, /var, etc.   Most things
work fine.

A bunch of things don't work (like changing passwords, and chowning
/dev/console (etc) to the user who logs in) but none of that is actually
essential to operation.

For what it's worth I saw a shutdown hang on a fairly recent current
recently - but it was much later than you're describing, after the
"syncing disks 3 2 1 ..." message (never got to "Done" after 1).   I
waited a while (several minutes at least) and it just sat there - till
it needed to be powered off [that's why it was being shut down.] (Have not
yet restarted that system to see if there were any uncleaned filesystems
left over.)

That system has the "normal" tmpfs's (/tmp and /var/shm) neither of
which would have been busy - what's more, /etc/rc.shutdown had completed
I believe, so they should have been unmounted well before.

kre

Re: Hanging at shutdown with mystery "file system full" error

[D'Arcy J.M. Cain]

On Wed, 21 Oct 2015 23:07:53 +0200
"Ian D. Leroux"  wrote:
> As it turns out, I don't need shutdown to trigger the hang, umount is
> enough.  Specifically,
> 
> umount -f /dev
> 
> Causes an immediate hang with the same kernel error message

Have you considered the possibility that having your devices on a
mounted FS is just asking for trouble?  I'm not sure but I think that I
would be surprised if I made all my devices suddenly disappear and it
didn't hang.  Why do you have it on a mounted FS anyway?

-- 
D'Arcy J.M. Cain 
http://www.NetBSD.org/ IM:da...@vex.net

Re: Hanging at shutdown with mystery "file system full" error

[Joerg Sonnenberger]

On Wed, Oct 21, 2015 at 06:40:56PM -0400, D'Arcy J.M. Cain wrote:
> On Wed, 21 Oct 2015 23:07:53 +0200
> "Ian D. Leroux"  wrote:
> > As it turns out, I don't need shutdown to trigger the hang, umount is
> > enough.  Specifically,
> > 
> > umount -f /dev
> > 
> > Causes an immediate hang with the same kernel error message
> 
> Have you considered the possibility that having your devices on a
> mounted FS is just asking for trouble?  I'm not sure but I think that I
> would be surprised if I made all my devices suddenly disappear and it
> didn't hang.  Why do you have it on a mounted FS anyway?

As written, that's done automatically by init if /dev is not populated
with devices. That is a common setup for ramdisks and read-only root
installations. It must not be broken.

Joerg

Re: Hanging at shutdown with mystery "file system full" error

[Robert Elz]

Date:Wed, 21 Oct 2015 07:21:58 +0200
From:"Ian D. Leroux" 
Message-ID:  <20151021072158.633f85efa203b59dc6cb8...@fastmail.fm>

  | My guess is that swap1_stop() first forcibly
  | unmounts /dev, and then tries to remove block-type swap devices (and
  | generally carry on with shutdown) once /dev/wd0 no longer exists.  This
  | can't be sane.

Not sane perhaps, but most likely not as bad as it sounds, the kernel shouldn't
be using pathname lookups to find anything it is trying to remove, it has
vnodes for the devices already, which is all that's needed to close them.

So, while the problem you're seeing is probably all bound up in that
mess somewhere, I doubt it is quite as simple as "/dev/wd0x doesn't exist,
help"...

That said, I really don't understand the desire to cleanly disengage
everything.  As soon as userland is done (when init will no longer run again)
the kernel should just sync & mark clean all of the filesystems, force stopped
(or into a tight loop, whatever is appropriate for the hardware) all but one
of the cpus, then exit to the bios (or whatever) to reset/halt/reboot or
whatever is wanted.

kre

Re: Hanging at shutdown with mystery "file system full" error

[Robert Swindells]

"Ian D. Leroux"  wrote:
>Is /dev a tmpfs on the other machines that are seeing this hang?  If
>not, that would rule out my crude oversimplified guess completely, which
>would be good to know.

I don't have /dev on tmpfs, a normal installation should put /dev on
the root partition. 

It looks as if my /var/shm is empty too.

Robert Swindells

Re: Hanging at shutdown with mystery "file system full" error

[Ian D. Leroux]

On Wed, Oct 21, 2015, at 08:35, Robert Elz wrote:
> Not sane perhaps, but most likely not as bad as it sounds, the kernel
> shouldn't be using pathname lookups to find anything it is trying to
> remove, it has vnodes for the devices already, which is all that's
> needed to close them.
>
> So, while the problem you're seeing is probably all bound up in that
> mess somewhere, I doubt it is quite as simple as "/dev/wd0x doesn't
> exist, help"...

I agree that my guess (which is all it was) was a crude
oversimplication.

This implies, incidentally, that I could manually unmount all tmpfs
filesystems (including /dev) before shutting down.  I'll try that
tonight and see what happens.

Is /dev a tmpfs on the other machines that are seeing this hang?  If
not, that would rule out my crude oversimplified guess completely, which
would be good to know.

> That said, I really don't understand the desire to cleanly disengage
> everything.  As soon as userland is done (when init will no longer run
> again) the kernel should just sync & mark clean all of the
> filesystems, force stopped (or into a tight loop, whatever is
> appropriate for the hardware) all but one of the cpus, then exit to
> the bios (or whatever) to reset/halt/reboot or whatever is wanted.

I'm not sure if you're discussing what ought to be happening now or what
ought to be happening in a better design, but note that I'm pretty sure
that the current hang happens before "userland is done".  For one thing,
when I activate ddb it shows that I'm interrupting a userland daemon
(syslogd or estd).

--
IDL

Re: Hanging at shutdown with mystery "file system full" error

[Ian D. Leroux]

On Wed, Oct 21, 2015, at 09:57, Patrick Welche wrote:
> On Wed, Oct 21, 2015 at 07:21:58AM +0200, Ian D. Leroux wrote:
> > On Tue, 20 Oct 2015 21:18:51 +0100 (BST) Robert Swindells
> >  wrote:
> > > Are any filesystems using tmpfs ?
> >
> > /tmp, /dev and /var/shm are all tmpfs mounts.  If I manually unmount
> > /tmp and /var/shm before shutting down, I still see the hang. I
> > haven't tried unmounting /dev yet.
> >
> > > I saw the same problem on one of my systems, I fixed it by backing
> > > out the last change to /etc/rc.d/swap1:
> > >
> > > 
> >
> > I've commented out the relevant lines in swap1_stop() and now my
> > laptop reboots normally.  My guess is that swap1_stop() first
> > forcibly unmounts /dev, and then tries to remove block-type swap
> > devices (and generally carry on with shutdown) once /dev/wd0 no
> > longer exists.  This can't be sane.
>
> But you say you see the hang even when you unmount (/tmp and /var/shm)
> and not /dev.
>
> So the issue is unmounting /var/shm ?

I'm not sure how that follows.  Unmounting /var/shm by hand before
shutting down doesn't prevent the hang, so it's not the absence of
/var/shm in and of itself that causes a problem.  Also, I haven't tried
unmounting /tmp and /var/shm separately, so for now I can't say anything
about one that doesn't also apply to the other.

Tonight I'll try shutting down with various combinations of tmpfs
filesystems, to see if I can actually narrow down the problem to a
specific mount point.

> I am running 12 Oct code without hangs - is this a new problem since
> an update for you?

It's new since an update on Oct. 17th, but my last update was sometime
at the beginning of 2015, so that doesn't tell us much.

--
IDL

Re: Hanging at shutdown with mystery "file system full" error

[Ian D. Leroux]

On Wed, Oct 21, 2015, at 11:52, Robert Swindells wrote:
> I don't have /dev on tmpfs, a normal installation should put /dev on
> the root partition.

Ok, so this issue is probably not specifically about /dev.

(My /dev is a tmpfs mount created at boot-time because root is mounted
read-only and, at least back when I first set this system up, /dev
needed to be writeable.  I can't remember why at the moment.)

> It looks as if my /var/shm is empty too.

I'll check if there's anything in mine, and then try to identify which
tmpfs mounts are needed to trigger the hang.

--
IDL

Re: Hanging at shutdown with mystery "file system full" error

[Patrick Welche]

On Wed, Oct 21, 2015 at 07:21:58AM +0200, Ian D. Leroux wrote:
> On Tue, 20 Oct 2015 21:18:51 +0100 (BST) Robert Swindells
>  wrote:
> 
> > 
> > "Ian D. Leroux"  wrote:
> > >With sources from Oct. 17th I get a repeatable hang whenever I
> > >shutdown my laptop (NetBSD/amd64 current).  If X is running at
> > >shutdown time, then I can't switch back to the console (Ctrl-Alt-F1
> > >has no effect) and my only recourse is to kill the power.  If X is
> > >not running I can switch to the console and see that the last
> > >messages to the console are kernel (green) warnings of the form
> > >
> > >uid 0, pid 1, command init, on /var: file system full
> > 
> > Are any filesystems using tmpfs ?
> 
> /tmp, /dev and /var/shm are all tmpfs mounts.  If I manually
> unmount /tmp and /var/shm before shutting down, I still see the hang.
> I haven't tried unmounting /dev yet.
> 
> > I saw the same problem on one of my systems, I fixed it by backing out
> > the last change to /etc/rc.d/swap1:
> > 
> > 
> 
> I've commented out the relevant lines in swap1_stop() and now my laptop
> reboots normally.  My guess is that swap1_stop() first forcibly
> unmounts /dev, and then tries to remove block-type swap devices (and
> generally carry on with shutdown) once /dev/wd0 no longer exists.  This
> can't be sane.

But you say you see the hang even when you
unmount (/tmp and /var/shm) and not /dev.

So the issue is unmounting /var/shm ?

I am running 12 Oct code without hangs - is this a new problem since
an update for you?

Cheers,

Patrick

Re: Hanging at shutdown with mystery "file system full" error

[Robert Swindells]

"Ian D. Leroux"  wrote:
>With sources from Oct. 17th I get a repeatable hang whenever I shutdown
>my laptop (NetBSD/amd64 current).  If X is running at shutdown time,
>then I can't switch back to the console (Ctrl-Alt-F1 has no effect) and
>my only recourse is to kill the power.  If X is not running I can
>switch to the console and see that the last messages to the console are
>kernel (green) warnings of the form
>
>uid 0, pid 1, command init, on /var: file system full

Are any filesystems using tmpfs ?

I saw the same problem on one of my systems, I fixed it by backing out
the last change to /etc/rc.d/swap1:



All my other machines reboot fine.

Robert Swindells

Re: Hanging at shutdown with mystery "file system full" error

[Ottavio Caruso]

On 20 October 2015 at 19:57, Ian D. Leroux  wrote:
> With sources from Oct. 17th I get a repeatable hang whenever I shutdown
> my laptop (NetBSD/amd64 current).  If X is running at shutdown time,
> then I can't switch back to the console (Ctrl-Alt-F1 has no effect) and
> my only recourse is to kill the power.

I have a similar issue but I'm not running current, just 7.0-rc3. And
I do have /tmp on tmpfs, as pointed out by Robert Swindells.

Apart from the solution offered by Robert, I have fixed my issue by
calling the poweroff script from Fluxbox menu.

I don't know why, but if I issue poweroff either manually in xterm or
via fluxbox, I can poweroff fine; if I press the power button, the
system will just hang forever.

-- 
Ottavio

Re: Hanging at shutdown with mystery "file system full" error

[Robert Swindells]

Patrick Welche wrote:
>On Tue, Oct 20, 2015 at 09:18:51PM +0100, Robert Swindells wrote:
>> 
>> "Ian D. Leroux"  wrote:
>> >With sources from Oct. 17th I get a repeatable hang whenever I shutdown
>> >my laptop (NetBSD/amd64 current).  If X is running at shutdown time,
>> >then I can't switch back to the console (Ctrl-Alt-F1 has no effect) and
>> >my only recourse is to kill the power.  If X is not running I can
>> >switch to the console and see that the last messages to the console are
>> >kernel (green) warnings of the form
>> >
>> >uid 0, pid 1, command init, on /var: file system full
>> 
>> Are any filesystems using tmpfs ?
>> 
>> I saw the same problem on one of my systems, I fixed it by backing out
>> the last change to /etc/rc.d/swap1:
>> 
>> 
>
>How does that work? I made that change to *prevent* a reproducible hang...

I read the thread that lead to your change, it seemed reasonable to me.

I was trying to get a bit more of a feel for what was happening before
raising a PR.

>(/var isn't a tmpfs on your system is it? So what is the connection?)

No, /var is a partition on the single HD. There is a tmpfs on /var/shm
and one on /tmp.

The machine that hangs isn't running X11 but it is running a
postgresql server which creates a couple of files in /tmp.

Robert Swindells

Re: Hanging at shutdown with mystery "file system full" error

[Patrick Welche]

On Tue, Oct 20, 2015 at 09:18:51PM +0100, Robert Swindells wrote:
> 
> "Ian D. Leroux"  wrote:
> >With sources from Oct. 17th I get a repeatable hang whenever I shutdown
> >my laptop (NetBSD/amd64 current).  If X is running at shutdown time,
> >then I can't switch back to the console (Ctrl-Alt-F1 has no effect) and
> >my only recourse is to kill the power.  If X is not running I can
> >switch to the console and see that the last messages to the console are
> >kernel (green) warnings of the form
> >
> >uid 0, pid 1, command init, on /var: file system full
> 
> Are any filesystems using tmpfs ?
> 
> I saw the same problem on one of my systems, I fixed it by backing out
> the last change to /etc/rc.d/swap1:
> 
> 

How does that work? I made that change to *prevent* a reproducible hang...
(/var isn't a tmpfs on your system is it? So what is the connection?)

Cheers,

Patrick

Re: Hanging at shutdown with mystery "file system full" error

[Ian D. Leroux]

On Tue, 20 Oct 2015 21:18:51 +0100 (BST) Robert Swindells
 wrote:

> 
> "Ian D. Leroux"  wrote:
> >With sources from Oct. 17th I get a repeatable hang whenever I
> >shutdown my laptop (NetBSD/amd64 current).  If X is running at
> >shutdown time, then I can't switch back to the console (Ctrl-Alt-F1
> >has no effect) and my only recourse is to kill the power.  If X is
> >not running I can switch to the console and see that the last
> >messages to the console are kernel (green) warnings of the form
> >
> >uid 0, pid 1, command init, on /var: file system full
> 
> Are any filesystems using tmpfs ?

/tmp, /dev and /var/shm are all tmpfs mounts.  If I manually
unmount /tmp and /var/shm before shutting down, I still see the hang.
I haven't tried unmounting /dev yet.

> I saw the same problem on one of my systems, I fixed it by backing out
> the last change to /etc/rc.d/swap1:
> 
> 

I've commented out the relevant lines in swap1_stop() and now my laptop
reboots normally.  My guess is that swap1_stop() first forcibly
unmounts /dev, and then tries to remove block-type swap devices (and
generally carry on with shutdown) once /dev/wd0 no longer exists.  This
can't be sane.

I'm not sure what the right fix might be though, since I don't see how
to determine automatically and in general which tmpfs-mounted
filesystems are providing essential services (like /dev) and which ones
can be flushed to make space on swap (like /tmp).

--
IDL