Re: ZFS Panic after freebsd-update
on 01/07/2013 21:50 Jeremy Chadwick said the following: The issue is that ZFS on FreeBSD is still young compared to other filesystems (specifically UFS). That's a fact. Nothing is perfect, but FFS/UFS tends to have a significantly larger number of bugs worked out of it to the point where people can use it without losing sleep (barring the SUJ stuff, don't get me started). That's subjective. I have the same concerns over other things, like ext2fs and fusefs for that matter -- but this thread is about a ZFS-related crash, and that's why I'm over-focused on it. I have an impression that you seem to state your (negative) opinion of ZFS in every other thread about ZFS problems. A heterogeneous (UFS+ZFS) setup, rather than homogeneous (ZFS-only), results in a system where an admin can upgrade + boot into single-user and perform some tasks to test/troubleshoot; if the ZFS layer is broken, it doesn't mean an essentially useless box. That isn't FUD, that's just the stage we're at right now. I'm aware lots of people have working ZFS-exclusive setups; like I said, works great until it doesn't. Yeah, a heterogeneous setup can have its benefits, but it can have its drawbacks too. This is true for heterogeneous vs monoculture in general. But the sword cuts both ways: what if something is broken in UFS layer or god forbid in VFS layer and you have only UFS? Besides, without mentioning specific classes of problems ZFS layer is broken is too vague. So, how do you kernel guys debug a problem in this environment: - ZFS-only - Running -RELEASE (i.e. no source, thus a kernel cannot be rebuilt with added debugging features, etc.) - No swap configured - No serial console I use boot environments and boot to a previous / known-good environment if I hit a loader bug, a kernel bug or a major userland problem in a new environment. I also use a mirrored setup and keep two copies of earlier boot chains. I am also not shy of live media in the case everything else fails. Now I wonder how you deal with the same kind of UFS-only environment. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS Panic after freebsd-update
On Tue, Jul 02, 2013 at 08:59:56AM +0300, Andriy Gapon wrote: on 01/07/2013 21:50 Jeremy Chadwick said the following: The issue is that ZFS on FreeBSD is still young compared to other filesystems (specifically UFS). That's a fact. Nothing is perfect, but FFS/UFS tends to have a significantly larger number of bugs worked out of it to the point where people can use it without losing sleep (barring the SUJ stuff, don't get me started). That's subjective. I have the same concerns over other things, like ext2fs and fusefs for that matter -- but this thread is about a ZFS-related crash, and that's why I'm over-focused on it. I have an impression that you seem to state your (negative) opinion of ZFS in every other thread about ZFS problems. The OP in question ended his post with the line Thoughts?, and I have given those thoughts. My thoughts/opinions/experience may differ from that of others. Diversity of thoughts/opinions/experiences is good. I'm not some kind of authoritative ZFS guru -- far from it. If I misunderstood what Thoughts? meant/implied, then draw and quarter me for it; my actions/words = my responsibility. I do not feel I have a negative opinion of ZFS. I still use it today on FreeBSD, donated money to Pawel when the project was originally announced (because I wanted to see something new and useful thrive on FreeBSD), and try my best to assist with issues pertaining to it where applicable. These are not the actions of someone with a negative opinion, these are the actions of someone who is supportive while simultaneously very cautious. Is ZFS better today than it was when it was introduced? By a long shot. For example, on my stable/9 system here I don't tune /boot/loader.conf any longer. But that doesn't change my viewpoint when it comes to using ZFS exclusively on a FreeBSD box. A heterogeneous (UFS+ZFS) setup, rather than homogeneous (ZFS-only), results in a system where an admin can upgrade + boot into single-user and perform some tasks to test/troubleshoot; if the ZFS layer is broken, it doesn't mean an essentially useless box. That isn't FUD, that's just the stage we're at right now. I'm aware lots of people have working ZFS-exclusive setups; like I said, works great until it doesn't. Yeah, a heterogeneous setup can have its benefits, but it can have its drawbacks too. This is true for heterogeneous vs monoculture in general. But the sword cuts both ways: what if something is broken in UFS layer or god forbid in VFS layer and you have only UFS? Besides, without mentioning specific classes of problems ZFS layer is broken is too vague. The likelihood of something being broken in UFS is significantly lower given its established history. I have to go off of experience, both personal and professional -- in my years of dealing with FreeBSD (1997-present), I have only encountered issues with UFS a few times (I can count them on one, maybe two hands), and I'm choosing to exclude SU+J from the picture for what should be obvious reasons. With ZFS, well... just look at the mailing lists and PR count. I don't want to be a jerk about it, but you really have to look at the quantity. It doesn't mean ZFS is crap, it just means that for me, I don't think we're quite there yet. And I will gladly admit -- because you are the one who taught me this -- that every incident need be treated unique. But one can't deny that a substantial percentage (I would say majority) of -fs and -stable posts relate somehow to ZFS; I'm often thrilled when it turns out to be something else. Playing a strange devil's advocate, let me give you an interesting example: softupdates. When SU was introduced to FreeBSD back in the late 90s, there were issues and concerns -- lots. As such, SU was chosen to be disabled by default on root filesystems given the importance of that filesystem (re: we do not want to risk losing as much data in the case of a crash -- see the official FAQ, section 8.3). All other filesystems defaulted to SU enabled. It's been like that up until 9.x where it now defaults to enabled. So that's what, 15 years? You could say that my example could also apply to ZFS, i.e. the reports are a part of its growth and maturity, and I'd agree. But I don't feel it's reached the point where I'm willing to risk going ZFS-only. Down the road, sure, but not now. That's just my take on it. Please make sure to also consider, politely, that a lot of people who have issues with ZFS have not been subscribed to the lists for long periods of time. They sign up/post when they have a problem. Meaning: they do not necessarily know of the history. If they did, I (again politely) believe they're likely to use a UFS+ZFS mix, or maybe a gmirror+UFS+ZFS mix (though the GPT/gmirror thing is... never mind...). So, how do you kernel guys debug a problem in this environment: - ZFS-only - Running -RELEASE (i.e. no source, thus a kernel cannot be
Re: ZFS Panic after freebsd-update
On Tue, Jul 02, 2013 at 12:57:16AM -0700, Jeremy Chadwick wrote: But in the OP's case, the situation sounds dire given the limitations -- limitations that someone (apparently not him) chose, which greatly hinder debugging/troubleshooting. Had a heterogeneous setup been chosen, the debugging/troubleshooting pains are less (IMO). When I see this, it makes me step back and ponder the decisions that lead to the ZFS-only setup. As an observer (though one who has used ZFS for some time, now), I might suggest that this can at least -seem- like FUD about ZFS because the limitations don't necessarily have anything to do with ZFS. That is, a situation in which one cannot recover, nor even effectively troubleshoot, if there is a problem, will be a dire one, regardless of what the problem might be or where its source might lie. -- greg byshenk - gbysh...@byshenk.net - Leiden, NL - Portland, OR USA ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
ZFS Panic after freebsd-update
Hello, I have not had much time to research this problem yet, so please let me know what further information I might be able to provide. This weekend I attempted to upgrade a computer from 8.2-RELEASE-p3 to 8.4 using freebsd-update. After I rebooted to test the new kernel, I got a panic. I had to take a picture of the screen. Here's a condensed version: panic: page fault cpuid = 1 KDB: stack backtrace: #0 #1 #2 #3 #4 #5 #6 #6 #6 #6 #6 #6 FreeBSD xeon.cap-press.com 8.2-RELEASE-p3 FreeBSD 8.2-RELEASE-p3 #0: Tue Sep 27 18:45:57 UTC 2011 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
ZFS Panic after freebsd-update
*** Sorry for partial first message! (gmail sent after multiple returns apparently?) *** Hello, I have not had much time to research this problem yet, so please let me know what further information I might be able to provide. This weekend I attempted to upgrade a computer from 8.2-RELEASE-p3 to 8.4 using freebsd-update. After I rebooted to test the new kernel, I got a panic. I had to take a picture of the screen. Here's a condensed version: panic: page fault cpuid = 1 KDB: stack backtrace: #0 kdb_backtrace #1 panic #2 trap_fatal #3 trap_pfault #4 trap #5 calltrap #6 vdev_mirror_child_select #7 ved_mirror_io_start #8 zio_vdev_io_start #9 zio_execute #10 arc_read #11 dbuf_read #12 dbuf_findbp #13 dbuf_hold_impl #14 dbuf_hold #15 dnode_hold_impl #16 dnu_buf_hold #17 zap_lockdir Uptime: 5s Cannot dump. Device not defined or unavailable. Automatic reboot in 15 seconds - press a key on the console to abort uname -a from before (and after) the reboot: FreeBSD xeon 8.2-RELEASE-p3 FreeBSD 8.2-RELEASE-p3 #0: Tue Sep 27 18:45:57 UTC 2011 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 dmesg is attached. I was able to reboot to the old kernel and am up and running back on 8.2 right now. Any thoughts? Thanks, Scott Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-RELEASE-p3 #0: Tue Sep 27 18:45:57 UTC 2011 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 Timecounter i8254 frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (2266.76-MHz K8-class CPU) Origin = GenuineIntel Id = 0x106a5 Family = 6 Model = 1a Stepping = 5 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2=0x9ce3bdSSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT AMD Features=0x28100800SYSCALL,NX,RDTSCP,LM AMD Features2=0x1LAHF TSC: P-state invariant real memory = 18253611008 (17408 MB) avail memory = 16513347584 (15748 MB) ACPI APIC Table: 031710 APIC1617 FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 16 cpu9 (AP): APIC ID: 17 cpu10 (AP): APIC ID: 18 cpu11 (AP): APIC ID: 19 cpu12 (AP): APIC ID: 20 cpu13 (AP): APIC ID: 21 cpu14 (AP): APIC ID: 22 cpu15 (AP): APIC ID: 23 ioapic0 Version 2.0 irqs 0-23 on motherboard ioapic1 Version 2.0 irqs 24-47 on motherboard kbd1 at kbdmux0 acpi0: 031710 XSDT1617 on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a (3) failed acpi0: reservation of 10, bff0 (3) failed Timecounter ACPI-fast frequency 3579545 Hz quality 1000 acpi_timer0: 24-bit timer at 3.579545MHz port 0x808-0x80b on acpi0 cpu0: ACPI CPU on acpi0 ACPI Warning: Incorrect checksum in table [OEMB] - 0xAD, should be 0xAA (20101013/tbutils-354) cpu1: ACPI CPU on acpi0 cpu2: ACPI CPU on acpi0 cpu3: ACPI CPU on acpi0 cpu4: ACPI CPU on acpi0 cpu5: ACPI CPU on acpi0 cpu6: ACPI CPU on acpi0 cpu7: ACPI CPU on acpi0 cpu8: ACPI CPU on acpi0 cpu9: ACPI CPU on acpi0 cpu10: ACPI CPU on acpi0 cpu11: ACPI CPU on acpi0 cpu12: ACPI CPU on acpi0 cpu13: ACPI CPU on acpi0 cpu14: ACPI CPU on acpi0 cpu15: ACPI CPU on acpi0 pcib0: ACPI Host-PCI bridge port 0xcf8-0xcff on acpi0 pci0: ACPI PCI bus on pcib0 pcib1: ACPI PCI-PCI bridge at device 1.0 on pci0 pci10: ACPI PCI bus on pcib1 pcib2: ACPI PCI-PCI bridge at device 3.0 on pci0 pci9: ACPI PCI bus on pcib2 pcib3: ACPI PCI-PCI bridge at device 7.0 on pci0 pci8: ACPI PCI bus on pcib3 pcib4: PCI-PCI bridge at device 8.0 on pci0 pci7: PCI bus on pcib4 pcib5: PCI-PCI bridge at device 9.0 on pci0 pci6: PCI bus on pcib5 pcib6: PCI-PCI bridge at device 10.0 on pci0 pci5: PCI bus on pcib6 pci0: base peripheral, interrupt controller at device 20.0 (no driver attached) pci0: base peripheral, interrupt controller at device 20.1 (no driver attached) pci0: base peripheral, interrupt controller at device 20.2 (no driver attached) pci0: base peripheral, interrupt controller at device 20.3 (no driver attached) pci0: base peripheral at device 22.0 (no driver attached) pci0: base peripheral at device 22.1 (no driver attached) pci0: base peripheral at device 22.2 (no driver attached) pci0: base peripheral at device 22.3 (no driver attached) pci0: base peripheral at device 22.4 (no driver attached) pci0: base peripheral at device 22.5 (no driver attached) pci0: base peripheral at device 22.6 (no driver attached) pci0: base peripheral at device 22.7 (no driver attached) uhci0:
Re: ZFS Panic after freebsd-update
On Mon, Jul 01, 2013 at 11:35:30AM -0400, Scott Sipe wrote: *** Sorry for partial first message! (gmail sent after multiple returns apparently?) *** Hello, I have not had much time to research this problem yet, so please let me know what further information I might be able to provide. This weekend I attempted to upgrade a computer from 8.2-RELEASE-p3 to 8.4 using freebsd-update. After I rebooted to test the new kernel, I got a panic. I had to take a picture of the screen. Here's a condensed version: panic: page fault cpuid = 1 KDB: stack backtrace: #0 kdb_backtrace #1 panic #2 trap_fatal #3 trap_pfault #4 trap #5 calltrap #6 vdev_mirror_child_select #7 ved_mirror_io_start #8 zio_vdev_io_start #9 zio_execute #10 arc_read #11 dbuf_read #12 dbuf_findbp #13 dbuf_hold_impl #14 dbuf_hold #15 dnode_hold_impl #16 dnu_buf_hold #17 zap_lockdir Uptime: 5s Cannot dump. Device not defined or unavailable. Automatic reboot in 15 seconds - press a key on the console to abort uname -a from before (and after) the reboot: FreeBSD xeon 8.2-RELEASE-p3 FreeBSD 8.2-RELEASE-p3 #0: Tue Sep 27 18:45:57 UTC 2011 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 dmesg is attached. I was able to reboot to the old kernel and am up and running back on 8.2 right now. Any thoughts? Thoughts: - All I see is an amd64 system with 16GB RAM and 4 disks driven by an ICH10 in AHCI mode. - Output from: zpool status - Output from: zpool get all - Output from: zfs get all - Output from: gpart show -p for every disk on the system - Output from: cat /etc/sysctl.conf - Output from: cat /boot/loader.conf - Is there a reason you do not have dumpdev defined in /etc/rc.conf (or alternately, no swap device defined in /etc/fstab (which will get used/honoured by the dumpdev=auto (the default)) ? Taking photos of the console and manually typing backtraces in is borderline worthless. Of course when I see lines like this: Trying to mount root from zfs:zroot ...this greatly diminishes any chances of live debugging on the system. It amazes me how often I see this come up on the lists -- people who have ZFS problems but use ZFS for their root/var/tmp/usr. I wish that behaviour would stop, as it makes debugging ZFS a serious PITA. This comes up on the list almost constantly, sad panda. - Get yourself stable/9 and try that: https://pub.allbsd.org/FreeBSD-snapshots/ - freebsd-fs is a better place for this discussion, especially since you're running a -RELEASE build, not a -STABLE build. -- | Jeremy Chadwick j...@koitsu.org | | UNIX Systems Administratorhttp://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS Panic after freebsd-update
On Mon, Jul 01, 2013 at 08:49:25AM -0700, Jeremy Chadwick wrote: - Is there a reason you do not have dumpdev defined in /etc/rc.conf (or alternately, no swap device defined in /etc/fstab (which will get used/honoured by the dumpdev=auto (the default)) ? This should have read or alternately, ***A*** swap device defined in /etc/fstab ... -- | Jeremy Chadwick j...@koitsu.org | | UNIX Systems Administratorhttp://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS Panic after freebsd-update
- Original Message - From: Jeremy Chadwick j...@koitsu.org To: Scott Sipe csco...@gmail.com Cc: freebsd-stable List freebsd-stable@freebsd.org Sent: Monday, July 01, 2013 4:49 PM Subject: Re: ZFS Panic after freebsd-update On Mon, Jul 01, 2013 at 11:35:30AM -0400, Scott Sipe wrote: *** Sorry for partial first message! (gmail sent after multiple returns apparently?) *** Hello, I have not had much time to research this problem yet, so please let me know what further information I might be able to provide. This weekend I attempted to upgrade a computer from 8.2-RELEASE-p3 to 8.4 using freebsd-update. After I rebooted to test the new kernel, I got a panic. I had to take a picture of the screen. Here's a condensed version: panic: page fault cpuid = 1 KDB: stack backtrace: #0 kdb_backtrace #1 panic #2 trap_fatal #3 trap_pfault #4 trap #5 calltrap #6 vdev_mirror_child_select #7 ved_mirror_io_start #8 zio_vdev_io_start #9 zio_execute #10 arc_read #11 dbuf_read #12 dbuf_findbp #13 dbuf_hold_impl #14 dbuf_hold #15 dnode_hold_impl #16 dnu_buf_hold #17 zap_lockdir Uptime: 5s Cannot dump. Device not defined or unavailable. Automatic reboot in 15 seconds - press a key on the console to abort uname -a from before (and after) the reboot: FreeBSD xeon 8.2-RELEASE-p3 FreeBSD 8.2-RELEASE-p3 #0: Tue Sep 27 18:45:57 UTC 2011 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 dmesg is attached. I was able to reboot to the old kernel and am up and running back on 8.2 right now. Any thoughts? This says your running a 8.2-RELEASE-p3 kernel not an 8.4-RELEASE kernel. Did the upgrade fail or is that dmesg / uname from your old kernel? Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS Panic after freebsd-update
On Jul 1, 2013, at 11:49 AM, Jeremy Chadwick j...@koitsu.org wrote: On Mon, Jul 01, 2013 at 11:35:30AM -0400, Scott Sipe wrote: *** Sorry for partial first message! (gmail sent after multiple returns apparently?) *** Hello, I have not had much time to research this problem yet, so please let me know what further information I might be able to provide. [[...]] Any thoughts? Thoughts: [[..]] Of course when I see lines like this: Trying to mount root from zfs:zroot ...this greatly diminishes any chances of live debugging on the system. It amazes me how often I see this come up on the lists -- people who have ZFS problems but use ZFS for their root/var/tmp/usr. I wish that behaviour would stop, as it makes debugging ZFS a serious PITA. This comes up on the list almost constantly, sad panda. I'm not sure why it amazes you that people are making widespread use of ZFS. You could make the same argument that people shouldn't use UFS2 journaling on their file systems because bugs in the implementation might make debugging journaled UFS2 file systems a serious PITA. The point is that there are VERY compelling reasons why people might want to use ZFS for root/var/tmp/usr/etc. (pooled storage; easy snapshots; etc.) and there should come a time when a given file system is generally regarded as safe. I'd say the time for ZFS came when they removed the big disclaimer from the boot messages. If ZFS is dangerous, they should reinstate the not ready for production warning. Until they do, I think it's unfair to castigate people for using ZFS universally. Isn't it a recurring theme on freebsd-current and freebsd-stable that more people need to use features so they can be debugged in realistic environments? If you're telling them, don't use that because it makes debugging harder, how are they supposed to get debugged and hence improved? :-) Cheers, Paul. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS Panic after freebsd-update
On Mon, Jul 01, 2013 at 12:23:45PM -0400, Paul Mather wrote: On Jul 1, 2013, at 11:49 AM, Jeremy Chadwick j...@koitsu.org wrote: On Mon, Jul 01, 2013 at 11:35:30AM -0400, Scott Sipe wrote: *** Sorry for partial first message! (gmail sent after multiple returns apparently?) *** Hello, I have not had much time to research this problem yet, so please let me know what further information I might be able to provide. [[...]] Any thoughts? Thoughts: [[..]] Of course when I see lines like this: Trying to mount root from zfs:zroot ...this greatly diminishes any chances of live debugging on the system. It amazes me how often I see this come up on the lists -- people who have ZFS problems but use ZFS for their root/var/tmp/usr. I wish that behaviour would stop, as it makes debugging ZFS a serious PITA. This comes up on the list almost constantly, sad panda. I'm not sure why it amazes you that people are making widespread use of ZFS. It's not widespread use of ZFS. It's widespread use of ZFS as their sole filesystem (specifically root/var/tmp/usr, or more specifically just root/usr). People are operating with the belief that ZFS just works, when reality shows it works until it doesn't. The mentality seems to be it's so rock solid it'll never break along with it can't happen to me. I tend to err on the side of caution, hence avoidance of ZFS for critical things like the aforementioned. It's different if you have a UFS root/var/tmp/usr and ZFS for everything else. You then have a system you can boot/use without issue even if ZFS is crapping the bed. You could make the same argument that people shouldn't use UFS2 journaling on their file systems because bugs in the implementation might make debugging journaled UFS2 file systems a serious PITA. Yup, and I do make that argument, quite regularly at that. There is even some evidence at this point in time that softupdates are broken: http://lists.freebsd.org/pipermail/freebsd-fs/2013-June/017424.html The point is that there are VERY compelling reasons why people might want to use ZFS for root/var/tmp/usr/etc. (pooled storage; easy snapshots; etc.) and there should come a time when a given file system is generally regarded as safe. While there may be compelling reasons, those reasons quickly get shot down when they realise they have a system they can't easily do troubleshooting with when the issue is with ZFS. I'd say the time for ZFS came when they removed the big disclaimer from the boot messages. If ZFS is dangerous, they should reinstate the not ready for production warning. Until they do, I think it's unfair to castigate people for using ZFS universally. The warning meant absolutely nothing at the time (it did not keep people away from it), and would mean nothing now if brought back. A single kernel printf() is not the right choice of action. Are we better off today than we were when ZFS was originally ported over? Yes, by far. Lots of improvements, in many great/good ways. No argument there. But there is no way I'd risk putting my root filesystem (or other key filesystems) on it -- still too new, still too many bugs, and users don't know about those problems until it's too late. Isn't it a recurring theme on freebsd-current and freebsd-stable that more people need to use features so they can be debugged in realistic environments? If you're telling them, don't use that because it makes debugging harder, how are they supposed to get debugged and hence improved? :-) 95% of FreeBSD users cannot debug kernel problems**. To debug a kernel problem, you need: a crash dump, a usable system with the exact kernel/world where the crash happened (i.e. you cannot crash 8.4 ZFS and boot into 8.2 and reliably debug it using that), and (most important of all) a developer who is familiar with kernel debugging *and* familiar with the bits which are crashing. Those who say what you're quoting are often the latter. Part of the need people to try this process you refer to is what stable/X is about, *without* the extra chaos of head. I'm one of those who for the past 15 years has advocated stable/X usage for a lot of reasons; I'll save the diatribe for some other time. But the OP is running -RELEASE, and chooses to run that, along with use of freebsd-update for binary updates. Their choices are limited: stick with 8.2, switch to stable/X, cease use of ZFS, or change OSes entirely. But even stable/X doesn't provide enough coverage at times (the recent fxp(4)/dhclient issue is proof of that). It's just too bad so many people have this broken mindset of what stability means on FreeBSD. ** = This number is probably more like 99%, especially when you consider what FreeNAS is catering to/trying to accomplish. -- | Jeremy Chadwick j...@koitsu.org | | UNIX Systems Administratorhttp://jdc.koitsu.org/ | | Making life hard for others
Re: ZFS Panic after freebsd-update
On Mon, Jul 1, 2013 at 1:04 PM, Jeremy Chadwick j...@koitsu.org wrote: On Mon, Jul 01, 2013 at 12:23:45PM -0400, Paul Mather wrote: On Jul 1, 2013, at 11:49 AM, Jeremy Chadwick j...@koitsu.org wrote: Of course when I see lines like this: Trying to mount root from zfs:zroot ...this greatly diminishes any chances of live debugging on the system. It amazes me how often I see this come up on the lists -- people who have ZFS problems but use ZFS for their root/var/tmp/usr. I wish that behaviour would stop, as it makes debugging ZFS a serious PITA. This comes up on the list almost constantly, sad panda. I'm not sure why it amazes you that people are making widespread use of ZFS. It's not widespread use of ZFS. It's widespread use of ZFS as their sole filesystem (specifically root/var/tmp/usr, or more specifically just root/usr). People are operating with the belief that ZFS just works, when reality shows it works until it doesn't. The mentality seems to be it's so rock solid it'll never break along with it can't happen to me. I tend to err on the side of caution, hence avoidance of ZFS for critical things like the aforementioned. It's different if you have a UFS root/var/tmp/usr and ZFS for everything else. You then have a system you can boot/use without issue even if ZFS is crapping the bed. ... 95% of FreeBSD users cannot debug kernel problems**. To debug a kernel problem, you need: a crash dump, a usable system with the exact kernel/world where the crash happened (i.e. you cannot crash 8.4 ZFS and boot into 8.2 and reliably debug it using that), and (most important of all) a developer who is familiar with kernel debugging *and* familiar with the bits which are crashing. Those who say what you're quoting are often the latter. ... But the OP is running -RELEASE, and chooses to run that, along with use of freebsd-update for binary updates. Their choices are limited: stick with 8.2, switch to stable/X, cease use of ZFS, or change OSes entirely. So I realize that neither 8.2-RELEASE or 8.4-RELEASE are stable, but I ultimately wasn't sure where the right place to go for discuss 8.4 is? Beyond the FS mailing list, was there a better place for my question? I'll provide the other requested information (zfs outputs, etc) to wherever would be best. This is a production machine (has been since late 2010) and after tweaking some ZFS settings initially has been totally stable. I wasn't incredibly closely involved in the initial configuration, but I've done at least one binary freebsd-update previously. Before this computer I had always done source upgrades. ZFS (and the thought of a panic like the one I saw this weekend!) made me leery of doing that. We're a small business--we have this server, an offsite backup server, and a firewall box. I understand that issues like this are are going to happen when I don't have a dedicated testing box, I just like to try to minimize them and keep them to weekends! It sounds like my best bet might be to add a new UFS disk, do a clean install of 9.1 onto that disk, and then import my existing ZFS pool? Thanks, Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS Panic after freebsd-update
on 01/07/2013 20:04 Jeremy Chadwick said the following: People are operating with the belief that ZFS just works, when reality shows it works until it doesn't That reality applies to everything that a man creates with a purpose to work. I am not sure why you are so over-focused on ZFS. Please stop spreading FUD. Thank you. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS Panic after freebsd-update
On Mon, Jul 01, 2013 at 02:04:24PM -0400, Scott Sipe wrote: On Mon, Jul 1, 2013 at 1:04 PM, Jeremy Chadwick j...@koitsu.org wrote: On Mon, Jul 01, 2013 at 12:23:45PM -0400, Paul Mather wrote: On Jul 1, 2013, at 11:49 AM, Jeremy Chadwick j...@koitsu.org wrote: Of course when I see lines like this: Trying to mount root from zfs:zroot ...this greatly diminishes any chances of live debugging on the system. It amazes me how often I see this come up on the lists -- people who have ZFS problems but use ZFS for their root/var/tmp/usr. I wish that behaviour would stop, as it makes debugging ZFS a serious PITA. This comes up on the list almost constantly, sad panda. I'm not sure why it amazes you that people are making widespread use of ZFS. It's not widespread use of ZFS. It's widespread use of ZFS as their sole filesystem (specifically root/var/tmp/usr, or more specifically just root/usr). People are operating with the belief that ZFS just works, when reality shows it works until it doesn't. The mentality seems to be it's so rock solid it'll never break along with it can't happen to me. I tend to err on the side of caution, hence avoidance of ZFS for critical things like the aforementioned. It's different if you have a UFS root/var/tmp/usr and ZFS for everything else. You then have a system you can boot/use without issue even if ZFS is crapping the bed. ... 95% of FreeBSD users cannot debug kernel problems**. To debug a kernel problem, you need: a crash dump, a usable system with the exact kernel/world where the crash happened (i.e. you cannot crash 8.4 ZFS and boot into 8.2 and reliably debug it using that), and (most important of all) a developer who is familiar with kernel debugging *and* familiar with the bits which are crashing. Those who say what you're quoting are often the latter. ... But the OP is running -RELEASE, and chooses to run that, along with use of freebsd-update for binary updates. Their choices are limited: stick with 8.2, switch to stable/X, cease use of ZFS, or change OSes entirely. So I realize that neither 8.2-RELEASE or 8.4-RELEASE are stable, but I ultimately wasn't sure where the right place to go for discuss 8.4 is? For filesystem issues, freebsd-fs@ is usually the best choice, because it discusses filesystem-related thing (regardless of stable vs. release, but knowing what version you have of course is mandatory). freebsd-stable@ is mainly for stable/X related discussions. Sorry to add pedanticism to an already difficult situation for you (and I sympathise, particularly since the purpose of the lists is often difficult to discern, even with their terse descriptions in mailman). Beyond the FS mailing list, was there a better place for my question? I'll provide the other requested information (zfs outputs, etc) to wherever would be best. Nope, not as far as I know. The only other place is send-pr(1), once you have an issue that can be reproduced. Keep in mind, however, that none of these options (mailing lists, send-pr, etc.) mandate a response from anyone. You/your business (see below) should be aware that there is always the possibility no one can help solve the actual problem; as such it's important that companies have proper upgrade/migration paths, rollback plans, and so on. This is a production machine (has been since late 2010) and after tweaking some ZFS settings initially has been totally stable. I wasn't incredibly closely involved in the initial configuration, but I've done at least one binary freebsd-update previously. Well regardless it sounds like moving from 8.2-RELEASE to 8.4-RELEASE causes ZFS to break for you, so that would classify as a regression. What the root cause is, however, is still unknown. Point: 8.2-RELEASE came out in February 2011, and 8.4-RELEASE came out in June 2013 -- that's almost 2.5 years of changes between versions. The number of changes between these two is major -- hundreds, maybe thousands. ZFS got worked on heavily during this time as well. I tend to tell anyone using ZFS that they should be running a stable/X (particularly stable/9) branch. I can expand on that justification if needed, as it's well-founded for a lot of reasons. Before this computer I had always done source upgrades. ZFS (and the thought of a panic like the one I saw this weekend!) made me leery of doing that. We're a small business--we have this server, an offsite backup server, and a firewall box. I understand that issues like this are are going to happen when I don't have a dedicated testing box, I just like to try to minimize them and keep them to weekends! Understood. It sounds like my best bet might be to add a new UFS disk, do a clean install of 9.1 onto that disk, and then import my existing ZFS pool? I would suggest starting with this: Get stable/9 from the place I mentioned, burn an
Re: ZFS Panic after freebsd-update
On Mon, Jul 01, 2013 at 09:10:45PM +0300, Andriy Gapon wrote: on 01/07/2013 20:04 Jeremy Chadwick said the following: People are operating with the belief that ZFS just works, when reality shows it works until it doesn't That reality applies to everything that a man creates with a purpose to work. I am not sure why you are so over-focused on ZFS. Please stop spreading FUD. Thank you. The issue is that ZFS on FreeBSD is still young compared to other filesystems (specifically UFS). Nothing is perfect, but FFS/UFS tends to have a significantly larger number of bugs worked out of it to the point where people can use it without losing sleep (barring the SUJ stuff, don't get me started). I have the same concerns over other things, like ext2fs and fusefs for that matter -- but this thread is about a ZFS-related crash, and that's why I'm over-focused on it. A heterogeneous (UFS+ZFS) setup, rather than homogeneous (ZFS-only), results in a system where an admin can upgrade + boot into single-user and perform some tasks to test/troubleshoot; if the ZFS layer is broken, it doesn't mean an essentially useless box. That isn't FUD, that's just the stage we're at right now. I'm aware lots of people have working ZFS-exclusive setups; like I said, works great until it doesn't. So, how do you kernel guys debug a problem in this environment: - ZFS-only - Running -RELEASE (i.e. no source, thus a kernel cannot be rebuilt with added debugging features, etc.) - No swap configured - No serial console -- | Jeremy Chadwick j...@koitsu.org | | UNIX Systems Administratorhttp://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS Panic after freebsd-update
- Original Message - From: Scott Sipe csco...@gmail.com So I realize that neither 8.2-RELEASE or 8.4-RELEASE are stable, but I ultimately wasn't sure where the right place to go for discuss 8.4 is? Beyond the FS mailing list, was there a better place for my question? I'll provide the other requested information (zfs outputs, etc) to wherever would be best. This is a production machine (has been since late 2010) and after tweaking some ZFS settings initially has been totally stable. I wasn't incredibly closely involved in the initial configuration, but I've done at least one binary freebsd-update previously. Before this computer I had always done source upgrades. ZFS (and the thought of a panic like the one I saw this weekend!) made me leery of doing that. We're a small business--we have this server, an offsite backup server, and a firewall box. I understand that issues like this are are going to happen when I don't have a dedicated testing box, I just like to try to minimize them and keep them to weekends! It sounds like my best bet might be to add a new UFS disk, do a clean install of 9.1 onto that disk, and then import my existing ZFS pool? There should be no reason why 8.4-RELEASE shouldn't work fine. Yes ZFS is continuously improving and these fixes / enhancements first hit head / current and are then MFC'ed back to stable/9 stable/8, but that doesn't mean the release branches should be avoided. If you can I would try booting from a 8.4-RELEASE cdrom / iso to see if it can successfully read the pool as this could eliminate out of sync kernel / world issues. Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS Panic after freebsd-update
Am 01.07.2013 um 20:56 schrieb Steven Hartland kill...@multiplay.co.uk: - Original Message - From: Scott Sipe csco...@gmail.com So I realize that neither 8.2-RELEASE or 8.4-RELEASE are stable, but I ultimately wasn't sure where the right place to go for discuss 8.4 is? Beyond the FS mailing list, was there a better place for my question? I'll provide the other requested information (zfs outputs, etc) to wherever would be best. This is a production machine (has been since late 2010) and after tweaking some ZFS settings initially has been totally stable. I wasn't incredibly closely involved in the initial configuration, but I've done at least one binary freebsd-update previously. Before this computer I had always done source upgrades. ZFS (and the thought of a panic like the one I saw this weekend!) made me leery of doing that. We're a small business--we have this server, an offsite backup server, and a firewall box. I understand that issues like this are are going to happen when I don't have a dedicated testing box, I just like to try to minimize them and keep them to weekends! It sounds like my best bet might be to add a new UFS disk, do a clean install of 9.1 onto that disk, and then import my existing ZFS pool? There should be no reason why 8.4-RELEASE shouldn't work fine. Yes ZFS is continuously improving and these fixes / enhancements first hit head / current and are then MFC'ed back to stable/9 stable/8, but that doesn't mean the release branches should be avoided. If you can I would try booting from a 8.4-RELEASE cdrom / iso to see if it can successfully read the pool as this could eliminate out of sync kernel / world issues. Personally, I find mfsbsd much more practical for booting up a rescue-environment. Also, if 8.4 does not work for some reason - maybe try 8.3? I have quite a lot of systems running 8.3 (and even more with 9.1) but none of them do zfsroot and none of them stresses ZFS very much. I've so far resisted the urge to update to 8.4. The reason why I would be interested to run zfs-root is that sometimes, you only have two hard drives and still want to do ZFS on it. Ideally, though, FreeBSD would be able to do something like SmartOS (one of the few features I kind of like about it…), where you boot from an USB-image (or ideally, via (i)PXE) but use all the available space for data and (3rd-party) software. That way, you always have something to boot from, but can maximize the usage of spindles and space. A basic FreeBSD install is, I think, less than 0.5G these days - I really hate wasting two 300 (or even 600) GB SAS hard disks just for that. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS Panic after freebsd-update
On Jul 1, 2013, at 19:04, Jeremy Chadwick j...@koitsu.org wrote: But even stable/X doesn't provide enough coverage at times (the recent fxp(4)/dhclient issue is proof of that). It's just too bad so many people have this broken mindset of what stability means on FreeBSD. As one of the few persons who have run into that issue I feel like I should speak up here and add that this issue was fixed within a very reasonable time span after raising the matter here on freebsd-stable@. You've personally been a great help in getting that fixed, so thank you for that. Apparently there was one earlier report of the issue very late in the pre-release process, which does imply that fxp hardware is fairly rarely in use among FreeBSD users these days (which was the excuse for how the issue passed testing for 8.4/9.1 RELEASE). I don't think the release engineering team can really be blamed for not catching bugs that go unreported that far into the release cycle; they have to make a decision when to release at some point and the later it gets into the cycle the harder it is to turn that decision around. I can completely understand that. That this happened was inconvenient, but it happens in stable. ISTR that stable doesn't mean stable in the sense that it won't crash, but rather that the API's won't change until the next release. I wish other OS companies were as reliable; both MS and Apple let a lot more slip by and they take a lot longer to release fixes as well. Of course nobody likes when their system behaves erratically due to some error outside their control, but until that point FreeBSD has been rock-solid for me for years. And even with this issue, the system was usable. To get back to the ZFS issue... ZFS has always seen a fairly large fraction of raised issues on this list. Often those were user mistakes, ranging from putting not enough memory into the system to not assigning enough to the ZIL (once that became usable). ZFS on FreeBSD has come a long way since then. I don't think it's in quite as usable a state on, for example, Linux. Yes, people are taking a risk when using ZFS for everything. The same goes for any FS. No matter which file system you use, if it breaks you're between a rock and a hard place. Depending on how badly broken it is, you may end up not being able to access your data and with some data that's not an option. That's what we have backups and test environments for, don't we? File system code can break. It shouldn't, and I think it's safe to say that in FreeBSD's history it has been very rare indeed, but it does happen. The problem is probably more that it's so rare that people don't take measures for the few times it does happen; like how many of us have an atomic shelter available to them? Or a rubber boat? How many nuclear incidents have there been versus how many serious file-system breakages in FreeBSD? How many of us first test an update to STABLE on an identical test system before upgrading our production servers? Jeremy, I know for a fact that you're a lot more on this list than I am and probably longer than I have been (I'm pretty sure you were around already back in the days when I started using FreeBSD 2.2.8), but in this case, as much respect as I have for you, I think you're overreacting a bit. And finally, we're having this whole discussion about how problematic FreeBSD's been (or not) recently WHILE THE OP HASNT EVEN GOTTEN BACK TO ANSWER DETAILS ABOUT HIS ISSUE YET. Perhaps it's a bit early for that? It's entirely possible that we're looking at some hardware issue here or a user error that triggered a corner case that wasn't handled or something like that. P.S: Personally, I don't use ZFS because I'm a bit of a database nut and feel like log-based file-systems aren't a good match for database write loads, but that's mostly just me being pedantic. Cheers, Alban Hertroys -- If you can't see the forest for the trees, cut the trees and you'll find there is no forest. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: ZFS Panic after freebsd-update
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 07/01/13 09:10, Steven Hartland wrote: [...] This says your running a 8.2-RELEASE-p3 kernel not an 8.4-RELEASE kernel. Did the upgrade fail or is that dmesg / uname from your old kernel? Looking at the context, he used freebsd-update to update 8.2-RELEASE to 8.4-RELEASE (which, the first step would be updating the kernel) and booted with that panic, and reverted to old kernel. It would be helpful if we have address of stack frame #6 as well as the tuning you he have done (in loader.conf), plus the actual panic message (looks like a kernel trap 12, but a glance at the code I didn't find a candidate line where this happens). Cheers, - -- Xin LI delp...@delphij.nethttps://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -BEGIN PGP SIGNATURE- iQEcBAEBCgAGBQJR0e91AAoJEG80Jeu8UPuz05MIAK21VdKOkVNISzrd9ZDKTpml EjKtrOUhXreI21XyuoVxGboIjNfBxbfPxu07Tj6ocY8LwwneMot9nW5d3xtsS71A ap9Ho3KFUKGv5RTHWO7mhbKhSXnKBl/SmyIeLx//I7vCfxQb0MWUT7bdRF56Eojj lUz6dnLDXt6q3p3TGC17mwETHbdvdrr4ptBANAXFaY763WFSW6pLWUr5KIxZ7f7i DqNKpShTC4LsVr6OZjq70E+1XFCM7E//ZKVbJWBNrGJd7kmk7raq7ERx8tJqcWu6 sdxWcjbG6bOlCmONcozohNsqRvpTKu1VK6JsWVBUq9Et2nY/2rKvu5lKyIvxPBg= =NmTM -END PGP SIGNATURE- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
