On 6/19/2013 19:21, Jeremy Chadwick wrote:
On Wed, Jun 19, 2013 at 06:35:57PM +0700, Adam Strohl wrote:
Hello -STABLE@,
So I've seen this situation seemingly randomly on a number of both
physical 9.1 boxes as well as VMs for I would say 6-9 months at
least. I finally have a physical box here that reproduces it
consistently that I can reboot easily (ie; not a production/client
server).
No matter what I do:
reboot
shutdown -p
shutdown -r
This specific server will stop at "All buffers synced" and not
actually power down or reboot. KB input seems to be ignored. This
server is a ZFS NAS (with GMIRROR for boot blocks) but the other
boxes which show this are using GMIRRORs for root/swap/boot (no
ZFS).
Here is what happens on the console: http://i.imgur.com/1H8JMyB.jpg
When I reset the server it appears that disks were not dismounted
cleanly ... on this ZFS box it comes back quick because ZFS is good
like that but on the other servers with GMIRROR roots rebuilding the
GMIRROR and fscking at the same time is murder on the
disk/performance until it finishes.
1. You mention "as well as VMs". Anything under a "virtual machine" or
under a hypervisor is going to be very, very, **VERY** different than
bare metal. So I hope the issues you're talking about above are on bare
metal -- I will assume so.
Nope, I see basically the same thing sometimes under ESXi 5.0 Hypervisor
(and yes it worries me the implications of something so broad). Those
unites I just haven't been able to isolate on a server which isn't
critical. Lets focus on this server for now though per your suggestion
below.
2. We need to know what version of "9.1" you're using, i.e. 9.1-RELEASE.
If you use stable/9 (RELENG_9) we need to see uname -a output (you can
hide the machine name if you want).
Sorry, this ZFS box is 9.1-R P4 (kernel built today):
FreeBSD ilos.dsn 9.1-RELEASE-p4 FreeBSD 9.1-RELEASE-p4 #6: Wed Jun 19
15:31:12 ICT 2013 root@hostname:/usr/obj/usr/src/sys/ATEAMSYSTEMS amd64
3. Can we please have dmesg from this machine? The controller and some
other hardware details matter.
Sure take a look at the full log here: http://pastebin.com/k55gVVuU
This includes a boot, then a reboot as I describe (you can see it logs
the All Buffers Synced, etc) then powering back on.
4. Does "sysctl hw.usb.no_shutdown_wait=1" help you?
Weirdly this allowed it to reboot on the first try (without needing to
be reset), but not the second. The "Starting background file system
checks in 60 seconds" message appeared ... that only happens when
something is dirty, right?
So the second try with just this I could ctrl alt del it and it
responded .. kind of:
http://i.imgur.com/POAIaNg.jpg
Still had to reset it though.
5. Does "sysctl hw.acpi.handle_reboot=1" help you?
No change, still responded to a ctrl alt del like above, but like that
still needs to be reset and comes back dirty.
6. Does "sysctl hw.acpi.disable_on_reboot=1" help you?
No change. Same as above, ctrl alt del responds but needs a hard reset
still.
7. If none of the above helps, can you please boot verbose mode and then
when the system "locks up" on "shutdown -r now" take a picture of the
VGA console?
Lots of debug on boot obviously but not much different on shutdown/hang:
http://i.imgur.com/SgzSsoP.jpg
8. Does the machine run moused(8) (check the process list please, do not
rely on rc.conf) ?
ps -auxww | grep moused reveals nothing running (which is how I have
things set).
Another interesting thing is that this particular server runs slapd
(OpenLDAP) which, when it comes back up, has a "corrupted" DB
(easily fixed with db_recover, but still). This might be because FS
commits aren't happening at the end. I can even manually stop
slapd (service slapd stop) then run sync(8) (I assume this does
something for ZFS too) and it still comes back as hosed if I reboot
shortly after. If I start/stop slapd it's fine. So I feel like
there is an FS/dismount thing going on here.
sync(8) does not do what you think it does. Please read (not skim) this
entire thread starting here:
http://lists.freebsd.org/pipermail/freebsd-fs/2013-April/thread.html#16982
http://lists.freebsd.org/pipermail/freebsd-fs/2013-April/016982.html
Groking this now ..
Your problem is related to unclean shutdown; fix that and your issues go
away.
Yeah that is my feeling as well.
Additional information: I also have some boxes which will reboot
(ie; they don't freeze like some do at the end) but they don't
dismount cleanly either and have to rebuild both GMIRROR and fsck.
This might be a different issue, too.
Every issue needs to be handled/treated separately.
Sure, I just had run across some threads about that but will focus on
this ZFS box (and see if anything that fixes here does anything with
that once I can reliably reproduce it out of production).
--
Adam Strohl
http://www.ateamsystems.com/
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[email protected]"
