Re: freebsd-update and hang during reboot
On Thu, Apr 16, 2015 at 6:52 PM, Glen Barber g...@freebsd.org wrote: On Wed, Apr 15, 2015 at 02:44:44PM -0700, Nick Rogers wrote: On Mon, Mar 9, 2015 at 9:19 AM, Nick Rogers ncrog...@gmail.com wrote: Is anyone working on fixing this problem? It seems like this should have some kind of full court press as it is obviously affecting plenty of people, some of which have spoken up in the following PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195458 I realize its a tough problem to track down, and if I had the appropriate skills I would help. But so far all I've been able to do, like others, is replicate and complain about the problem. Its still affecting upgrading to 10.1-RELEASE-p6 from the official 10.1-RELEASE distribution, and from 10.1-RELEASE-p5. I just had another production server hang during reboot after updating to p6, and I don't see this changing for the inevitable p7 unless this problem gets more attention. Can someone with the right skill-set please help figure this out? Thank you. In case anyone is still dealing with this problem, the fix was MFC'd to stable/10 a few days. I am assuming this will not end up getting back ported to releng/10.1. An EN for 10.1-RELEASE is planned. Oh thats good to know. Thank you. I had asked that in the related bug a week or so ago and did not receive a response, so I figured it would not. Any idea the eta for the EN? Glen ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: freebsd-update and hang during reboot
On Fri, Apr 17, 2015 at 02:32:47PM -0700, Nick Rogers wrote: On Thu, Apr 16, 2015 at 6:52 PM, Glen Barber g...@freebsd.org wrote: On Wed, Apr 15, 2015 at 02:44:44PM -0700, Nick Rogers wrote: In case anyone is still dealing with this problem, the fix was MFC'd to stable/10 a few days. I am assuming this will not end up getting back ported to releng/10.1. An EN for 10.1-RELEASE is planned. Oh thats good to know. Thank you. I had asked that in the related bug a week or so ago and did not receive a response, so I figured it would not. Any idea the eta for the EN? I'm behind on email, and saw your email to this list first. No ETA yet, but it is being worked on. Glen pgp2XBjZ11Cx8.pgp Description: PGP signature
Re: freebsd-update and hang during reboot
On Wed, Apr 15, 2015 at 02:44:44PM -0700, Nick Rogers wrote: On Mon, Mar 9, 2015 at 9:19 AM, Nick Rogers ncrog...@gmail.com wrote: Is anyone working on fixing this problem? It seems like this should have some kind of full court press as it is obviously affecting plenty of people, some of which have spoken up in the following PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195458 I realize its a tough problem to track down, and if I had the appropriate skills I would help. But so far all I've been able to do, like others, is replicate and complain about the problem. Its still affecting upgrading to 10.1-RELEASE-p6 from the official 10.1-RELEASE distribution, and from 10.1-RELEASE-p5. I just had another production server hang during reboot after updating to p6, and I don't see this changing for the inevitable p7 unless this problem gets more attention. Can someone with the right skill-set please help figure this out? Thank you. In case anyone is still dealing with this problem, the fix was MFC'd to stable/10 a few days. I am assuming this will not end up getting back ported to releng/10.1. An EN for 10.1-RELEASE is planned. Glen pgphKjzTB_th4.pgp Description: PGP signature
Re: freebsd-update and hang during reboot
On Mon, Mar 9, 2015 at 9:19 AM, Nick Rogers ncrog...@gmail.com wrote: On Tue, Feb 10, 2015 at 1:37 PM, Nick Rogers ncrog...@gmail.com wrote: On Mon, Feb 9, 2015 at 9:08 AM, Ian Lepore i...@freebsd.org wrote: On Mon, 2015-02-09 at 11:41 -0500, Kurt Lidl wrote: Joel wrote: Hi, Just about every machine I have seems to hang after running freebsd-update and doing a reboot. The last message on the screen is All buffers synced” and it just freezes. This happens when doing a freebsd-update and going from 10.0 to 10.1, but also when doing a fresh 10.1 install and using freebsd-update to get the latest -pX security patches. As soon as I reboot the machine, it hangs. I’ve tried it on several different HP ProLiant models, on Intel NUCs and on VMware virtual machines. Same phenomenon everywhere. It’s really easy to trigger: just install 10.1, use default settings everywhere, freebsd-update fetch/install, shutdown -r now and BOOM. It hangs. I think I’ve seen it on 30 servers or so now. Everything works like it should after the initial hang tough - no matter how many times I reboot it completes the reboot cycle just fine. I’ve seen several people (mostly on IRC) mention this problem, but no solution. Is anyone working on fixing this? I ran into this problem in spades when upgrading a set of servers from FreeBSD 9.0 to 9.1. I happened consistently. Normal reboots worked, but when going from 9.0 to 9.1, it *ALWAYS* hung, and it always hung at the same place, after printing the All buffers synced message. I ultimately determined that if I did the following, rather than just a reboot or shutdown -r now 'FreeBSD 9.1-RELEASE upgrade', it would consistently AVOID the hang: sync ; sync ; sync ; shutdown -o -n -r now FreeBSD 9.1 install Your mileage may vary, but you don't have a lot to lose by trying it. -Kurt That is just bad advice. sync(1) does not g'tee that all data has been written, no matter how many times you type it. shutdown -n tells the system to abandon unwritten data. All in all, this is a recipe for silent filesystem corruption. Using it after an update is just asking to have a mix of old and new files on the system after the reboot. A more robust workaround would be to mount -r on all filesystems before invoking the shutdown (even a shutdown -n should be safe after everything has been remounted readonly). If the mount -r hangs on one of the filesystems, then you've probably got a clue as to where a normal shutdown is hanging. FWIW mount -r on the root filesystem hangs for me. If I disable softupdates-journaling on the root filesystem before the upgrade process, the system no longer hangs on the last reboot after userland upgrade. However, the root filesystem still comes up dirty with an incorrect free block count during fsck. Is anyone working on fixing this problem? It seems like this should have some kind of full court press as it is obviously affecting plenty of people, some of which have spoken up in the following PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195458 I realize its a tough problem to track down, and if I had the appropriate skills I would help. But so far all I've been able to do, like others, is replicate and complain about the problem. Its still affecting upgrading to 10.1-RELEASE-p6 from the official 10.1-RELEASE distribution, and from 10.1-RELEASE-p5. I just had another production server hang during reboot after updating to p6, and I don't see this changing for the inevitable p7 unless this problem gets more attention. Can someone with the right skill-set please help figure this out? Thank you. In case anyone is still dealing with this problem, the fix was MFC'd to stable/10 a few days. I am assuming this will not end up getting back ported to releng/10.1. I've compiled a patch with the fix that works against 10.1-RELEASE. Maybe it will be useful for any of you like me that don't run 10-stable, but are comfortable with custom kernels and are still dealing with this issue when running freebsd-update every time a new patch level is released. Diff is below. # Fix bug causing a hang while unmounting the root filesystem during # reboot after performing a freebsd-update. # # # Original commit to HEAD: # https://svnweb.freebsd.org/base?view=revisionrevision=280760 # MFC to stable: # https://svnweb.freebsd.org/base?view=revisionrevision=281350 # # The following commits were taken from stable/10/sys/ufs/ffs between # the release of 10.1-RELEASE (r272459) and MFC of the fix (r281350) # in order for the fix to cleanly apply to releng/10.1. The two # unrelated commits seem like reasonable fixes to include as well. # # https://svnweb.freebsd.org/base?view=revisionrevision=281350 # https://svnweb.freebsd.org/base?view=revisionrevision=278667 # https://svnweb.freebsd.org/base?view=revisionrevision=274305 # Index: ufs/ffs/ffs_vfsops.c
Re: freebsd-update and hang during reboot
Thanks Steve, for providing this patch, but please note that for this to work, you MUST copy a GENERIC kernel into /boot as /boot/GENERIC. If you run a GENERIC kernel, just cp /boot/kernel/kernel /boot/GENERIC. Details are in the Handbook at https://www.freebsd.org/doc/handbook/updating-upgrading-freebsdupdate.html. This means rebuilding a patched kernel after each update that touches the kernel. (Many do not!) Also be very careful with cutting and pasting this diff as whitespace can get messed up resulting in failures. (TAC NE SPACE+) Kevin Oberman, Network Engineer, Retired E-mail: rkober...@gmail.com On Wed, Apr 15, 2015 at 2:44 PM, Nick Rogers ncrog...@gmail.com wrote: On Mon, Mar 9, 2015 at 9:19 AM, Nick Rogers ncrog...@gmail.com wrote: On Tue, Feb 10, 2015 at 1:37 PM, Nick Rogers ncrog...@gmail.com wrote: On Mon, Feb 9, 2015 at 9:08 AM, Ian Lepore i...@freebsd.org wrote: On Mon, 2015-02-09 at 11:41 -0500, Kurt Lidl wrote: Joel wrote: Hi, Just about every machine I have seems to hang after running freebsd-update and doing a reboot. The last message on the screen is All buffers synced” and it just freezes. This happens when doing a freebsd-update and going from 10.0 to 10.1, but also when doing a fresh 10.1 install and using freebsd-update to get the latest -pX security patches. As soon as I reboot the machine, it hangs. I’ve tried it on several different HP ProLiant models, on Intel NUCs and on VMware virtual machines. Same phenomenon everywhere. It’s really easy to trigger: just install 10.1, use default settings everywhere, freebsd-update fetch/install, shutdown -r now and BOOM. It hangs. I think I’ve seen it on 30 servers or so now. Everything works like it should after the initial hang tough - no matter how many times I reboot it completes the reboot cycle just fine. I’ve seen several people (mostly on IRC) mention this problem, but no solution. Is anyone working on fixing this? I ran into this problem in spades when upgrading a set of servers from FreeBSD 9.0 to 9.1. I happened consistently. Normal reboots worked, but when going from 9.0 to 9.1, it *ALWAYS* hung, and it always hung at the same place, after printing the All buffers synced message. I ultimately determined that if I did the following, rather than just a reboot or shutdown -r now 'FreeBSD 9.1-RELEASE upgrade', it would consistently AVOID the hang: sync ; sync ; sync ; shutdown -o -n -r now FreeBSD 9.1 install Your mileage may vary, but you don't have a lot to lose by trying it. -Kurt That is just bad advice. sync(1) does not g'tee that all data has been written, no matter how many times you type it. shutdown -n tells the system to abandon unwritten data. All in all, this is a recipe for silent filesystem corruption. Using it after an update is just asking to have a mix of old and new files on the system after the reboot. A more robust workaround would be to mount -r on all filesystems before invoking the shutdown (even a shutdown -n should be safe after everything has been remounted readonly). If the mount -r hangs on one of the filesystems, then you've probably got a clue as to where a normal shutdown is hanging. FWIW mount -r on the root filesystem hangs for me. If I disable softupdates-journaling on the root filesystem before the upgrade process, the system no longer hangs on the last reboot after userland upgrade. However, the root filesystem still comes up dirty with an incorrect free block count during fsck. Is anyone working on fixing this problem? It seems like this should have some kind of full court press as it is obviously affecting plenty of people, some of which have spoken up in the following PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195458 I realize its a tough problem to track down, and if I had the appropriate skills I would help. But so far all I've been able to do, like others, is replicate and complain about the problem. Its still affecting upgrading to 10.1-RELEASE-p6 from the official 10.1-RELEASE distribution, and from 10.1-RELEASE-p5. I just had another production server hang during reboot after updating to p6, and I don't see this changing for the inevitable p7 unless this problem gets more attention. Can someone with the right skill-set please help figure this out? Thank you. In case anyone is still dealing with this problem, the fix was MFC'd to stable/10 a few days. I am assuming this will not end up getting back ported to releng/10.1. I've compiled a patch with the fix that works against 10.1-RELEASE. Maybe it will be useful for any of you like me that don't run 10-stable, but are comfortable with custom kernels and are still dealing with this issue when running freebsd-update every time a new patch level is released. Diff is
Re: freebsd-update and hang during reboot
On Tue, Feb 10, 2015 at 1:37 PM, Nick Rogers ncrog...@gmail.com wrote: On Mon, Feb 9, 2015 at 9:08 AM, Ian Lepore i...@freebsd.org wrote: On Mon, 2015-02-09 at 11:41 -0500, Kurt Lidl wrote: Joel wrote: Hi, Just about every machine I have seems to hang after running freebsd-update and doing a reboot. The last message on the screen is All buffers synced” and it just freezes. This happens when doing a freebsd-update and going from 10.0 to 10.1, but also when doing a fresh 10.1 install and using freebsd-update to get the latest -pX security patches. As soon as I reboot the machine, it hangs. I’ve tried it on several different HP ProLiant models, on Intel NUCs and on VMware virtual machines. Same phenomenon everywhere. It’s really easy to trigger: just install 10.1, use default settings everywhere, freebsd-update fetch/install, shutdown -r now and BOOM. It hangs. I think I’ve seen it on 30 servers or so now. Everything works like it should after the initial hang tough - no matter how many times I reboot it completes the reboot cycle just fine. I’ve seen several people (mostly on IRC) mention this problem, but no solution. Is anyone working on fixing this? I ran into this problem in spades when upgrading a set of servers from FreeBSD 9.0 to 9.1. I happened consistently. Normal reboots worked, but when going from 9.0 to 9.1, it *ALWAYS* hung, and it always hung at the same place, after printing the All buffers synced message. I ultimately determined that if I did the following, rather than just a reboot or shutdown -r now 'FreeBSD 9.1-RELEASE upgrade', it would consistently AVOID the hang: sync ; sync ; sync ; shutdown -o -n -r now FreeBSD 9.1 install Your mileage may vary, but you don't have a lot to lose by trying it. -Kurt That is just bad advice. sync(1) does not g'tee that all data has been written, no matter how many times you type it. shutdown -n tells the system to abandon unwritten data. All in all, this is a recipe for silent filesystem corruption. Using it after an update is just asking to have a mix of old and new files on the system after the reboot. A more robust workaround would be to mount -r on all filesystems before invoking the shutdown (even a shutdown -n should be safe after everything has been remounted readonly). If the mount -r hangs on one of the filesystems, then you've probably got a clue as to where a normal shutdown is hanging. FWIW mount -r on the root filesystem hangs for me. If I disable softupdates-journaling on the root filesystem before the upgrade process, the system no longer hangs on the last reboot after userland upgrade. However, the root filesystem still comes up dirty with an incorrect free block count during fsck. Is anyone working on fixing this problem? It seems like this should have some kind of full court press as it is obviously affecting plenty of people, some of which have spoken up in the following PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195458 I realize its a tough problem to track down, and if I had the appropriate skills I would help. But so far all I've been able to do, like others, is replicate and complain about the problem. Its still affecting upgrading to 10.1-RELEASE-p6 from the official 10.1-RELEASE distribution, and from 10.1-RELEASE-p5. I just had another production server hang during reboot after updating to p6, and I don't see this changing for the inevitable p7 unless this problem gets more attention. Can someone with the right skill-set please help figure this out? Thank you. -- Ian ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
