Re: [OmniOS-discuss] ZFS crash/reboot loop
On Jul 13, 2015, at 11:29 AM, Dan McDonald dan...@omniti.com wrote:
On Jul 13, 2015, at 11:25 AM, Derek Yarnell de...@umiacs.umd.edu wrote:
https://obj.umiacs.umd.edu/derek_support/vmdump.0
Yeah, that's what I'm seeking. Downloading it now to an r151014 box (you are
running r151014 according to the first mail). My normal '014 box is
otherwise indisposed at the moment, so this dump may take a bit longer to
analyze. I can forward it along to the ZFS folks once I've done my initial
analysis.
For bugs like these, I usually have to engage the illumos ZFS list. If
anyone here wants to follow along, I'll Cc: you on anything I report to them.
Okay, it's a VERIFY() failure in zio_buf_alloc(). It's passed a size of 0 by
its caller. Observe this MDB interaction:
$c
vpanic()
0xfba8b13d()
zio_buf_alloc+0x49(0)
arc_get_data_buf+0x12b(ff0d4071ca98)
arc_buf_alloc+0xd2(ff0d4dfec000, 0, 0, 1)
...SNIP!
0xff0d4071ca98 is an arc_buf_t, read off of disk. The code in
arc_get_data_buf starts with:
static void
arc_get_data_buf(arc_buf_t *buf)
{
arc_state_t *state = buf-b_hdr-b_l1hdr.b_state;
uint64_tsize = buf-b_hdr-b_size;
arc_buf_contents_t type = arc_buf_type(buf-b_hdr);
So let's look at that size:
ff0d4071ca98::print arc_buf_t b_hdr |::print arc_buf_hdr_t b_size
b_size = 0
Ouch. There's your zero.
I'm going to forward this very note to the illumos ZFS list. I see ONE
possible bugfix post-r151014 that might help:
commit 31c46cf23cd1cf4d66390a983dc5072d7d299ba2
Author: Alek Pinchuk a...@nexenta.com
Date: Tue Jun 30 09:44:11 2015 -0700
6033 arc_adjust() should search MFU lists for oldest buffer when adjusting
MFU size
Reviewed by: Saso Kiselkov saso.kisel...@nexenta.com
Reviewed by: Xin Li delp...@delphij.net
Reviewed by: Prakash Surya m...@prakashsurya.com
Approved by: Matthew Ahrens mahr...@delphix.com
It's a small bug, and I shudder to say this, even hot-patchable on a running
system if you're desperate. :)
Thanks,
Dan
d
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
On Jul 13, 2015, at 11:56 AM, Derek Yarnell de...@umiacs.umd.edu wrote: I don't need to hot patch (cold patch would be fine) so any update that I can apply and reboot would be fine. We have a second OmniOS r14 copy running that we are happy to patch in any way possible to get it mounted rw. IF (and only if) it's the bug I mentioned that's the problem. I want ZFS experts to take a look as well. It's on the ZFS list now, so we'll see what happens. If you're REALLY feeling brave, I can build a replacement ZFS module with 6033 in place for you to try, but I can't promise it'll work. Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
ff0d4071ca98::print arc_buf_t b_hdr |::print arc_buf_hdr_t b_size b_size = 0 Ouch. There's your zero. I'm going to forward this very note to the illumos ZFS list. I see ONE possible bugfix post-r151014 that might help: commit 31c46cf23cd1cf4d66390a983dc5072d7d299ba2 Author: Alek Pinchuk a...@nexenta.com Date: Tue Jun 30 09:44:11 2015 -0700 6033 arc_adjust() should search MFU lists for oldest buffer when adjusting MFU size Reviewed by: Saso Kiselkov saso.kisel...@nexenta.com Reviewed by: Xin Li delp...@delphij.net Reviewed by: Prakash Surya m...@prakashsurya.com Approved by: Matthew Ahrens mahr...@delphix.com It's a small bug, and I shudder to say this, even hot-patchable on a running system if you're desperate. :) Hi Dan, I don't need to hot patch (cold patch would be fine) so any update that I can apply and reboot would be fine. We have a second OmniOS r14 copy running that we are happy to patch in any way possible to get it mounted rw. Thanks, derek -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
Hi Dan, Sorry I have not dealt with dumpadm/savecore that much but it looks like this is what you want. https://obj.umiacs.umd.edu/derek_support/vmdump.0 Thanks, derek On 7/13/15 12:55 AM, Dan McDonald wrote: On Jul 12, 2015, at 9:18 PM, Richard Elling richard.ell...@richardelling.com wrote: Dan, if you're listening, Matt would be the best person to weigh-in on this. Yes he would be, Richard.. The panic in the arc_get_data_buf() paths is similar to older problems we'd seen in r151006. Derek, do you have a kernel coredump from these? I know you've been panic-and-reboot-and-panic-ing, but if you can get savecore(1M) to do its thing, having that dump would be useful. Thanks, Dan -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
On 7/13/15 12:02 PM, Dan McDonald wrote: On Jul 13, 2015, at 11:56 AM, Derek Yarnell de...@umiacs.umd.edu wrote: I don't need to hot patch (cold patch would be fine) so any update that I can apply and reboot would be fine. We have a second OmniOS r14 copy running that we are happy to patch in any way possible to get it mounted rw. IF (and only if) it's the bug I mentioned that's the problem. I want ZFS experts to take a look as well. It's on the ZFS list now, so we'll see what happens. If you're REALLY feeling brave, I can build a replacement ZFS module with 6033 in place for you to try, but I can't promise it'll work. Hi Dan, I would be happy to try to test a build with 6033 on it to see if that is my issue. We have secured all the critical data and only have scratch data left. So at this point I would happy to take a chance to see if this will fix the issue. Thanks, derek -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
On Sun, Jul 12, 2015 at 06:18:17PM -0700, Richard Elling wrote: Some additional block pointer verification code was added in changeset f63ab3d5a84a12b474655fc7e700db3efba6c4c9 and likely is the cause of this assertion. In general, assertion failures are almost always software problems -- the programmer didn't see what they expected. If this is something that might have been ignored prior to this code change, maybe they could set aok to avoid panicking when they import the pool to recover data? Not very familiar with that technique myself but I've seen it mentioned frequently in cases like this, unless things have changed since then. ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
First action: If you can mount the pool read-only, update your backup Then I would expect that a single bad disk is the reason of the problem on a write command. I would first check the system and fault log or smartvalues for hints about a bad disk. If there is a suspicious disk, remove that and retry a regular import. If there is no hint Next what I would try is a pool export. Then create a script that imports the pool followed by a scrub cancel. (Hope that the cancel is faster than the crash). Then check logs during some pool activity. If this does not help, I would remove all data disks and bootup. Then hot-plug disk by disk and check if its detected properly and check logs. Your pool remains offline until enough disks come back. Adding disk by disk and checking logs should help to find a bad disk that initiates a crash Next option is, try a pool import where always one or next disk is missing. Until there is no write, missing disks are not a problem with ZFS (you may need to clear errors). Last option: use another server where you try to import (mainboard, power, hba or backplane problem) remove all disks and do a nondestructive or smart test on another machine Gea On 12.07.2015 20:43, Derek Yarnell wrote: The on-going scrub automatically restarts, apparently even in read-only mode. You should 'zpool scrub -s poolname' ASAP after boot (if you can) to stop the ongoing scrub. We have tried to stop the scrub but it seems you can not cancel a scrub when the pool is mounted readonly. ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
On Jul 12, 2015, at 5:26 PM, Derek Yarnell de...@umiacs.umd.edu wrote: On 7/12/15 3:21 PM, Günther Alka wrote: First action: If you can mount the pool read-only, update your backup We are securing all the non-scratch data currently before messing with the pool any more. We had backups as recent as the night before but it is still going to be faster to pull the current data from the readonly pool than from backups. Then I would expect that a single bad disk is the reason of the problem on a write command. I would first check the system and fault log or smartvalues for hints about a bad disk. If there is a suspicious disk, remove that and retry a regular import. We have pulled all disks individually yesterday to test this exact theory. We have hit the mpt_sas disk failure panics before so we had already tried this. I don't believe this is a bad disk. Some additional block pointer verification code was added in changeset f63ab3d5a84a12b474655fc7e700db3efba6c4c9 and likely is the cause of this assertion. In general, assertion failures are almost always software problems -- the programmer didn't see what they expected. Dan, if you're listening, Matt would be the best person to weigh-in on this. -- richard If there is no hint Next what I would try is a pool export. Then create a script that imports the pool followed by a scrub cancel. (Hope that the cancel is faster than the crash). Then check logs during some pool activity. If I have not imported the pool RW can I export the pool? I thought we have tried this but I will have to confer. If this does not help, I would remove all data disks and bootup. Then hot-plug disk by disk and check if its detected properly and check logs. Your pool remains offline until enough disks come back. Adding disk by disk and checking logs should help to find a bad disk that initiates a crash This is interesting and we will try this once we secure the data. Next option is, try a pool import where always one or next disk is missing. Until there is no write, missing disks are not a problem with ZFS (you may need to clear errors). Wouldn't this be the same as above hot-plugging disk by disk? Last option: use another server where you try to import (mainboard, power, hba or backplane problem) remove all disks and do a nondestructive or smart test on another machine Sadly we do not have a spare chassis with 40 slots around to test this. I am so far unconvinced that this is a hardware problem though. We will most likely boot up into linux live CD to run smartctl and see if it has any information on the disks. -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
On 7/12/15 3:21 PM, Günther Alka wrote: First action: If you can mount the pool read-only, update your backup We are securing all the non-scratch data currently before messing with the pool any more. We had backups as recent as the night before but it is still going to be faster to pull the current data from the readonly pool than from backups. Then I would expect that a single bad disk is the reason of the problem on a write command. I would first check the system and fault log or smartvalues for hints about a bad disk. If there is a suspicious disk, remove that and retry a regular import. We have pulled all disks individually yesterday to test this exact theory. We have hit the mpt_sas disk failure panics before so we had already tried this. If there is no hint Next what I would try is a pool export. Then create a script that imports the pool followed by a scrub cancel. (Hope that the cancel is faster than the crash). Then check logs during some pool activity. If I have not imported the pool RW can I export the pool? I thought we have tried this but I will have to confer. If this does not help, I would remove all data disks and bootup. Then hot-plug disk by disk and check if its detected properly and check logs. Your pool remains offline until enough disks come back. Adding disk by disk and checking logs should help to find a bad disk that initiates a crash This is interesting and we will try this once we secure the data. Next option is, try a pool import where always one or next disk is missing. Until there is no write, missing disks are not a problem with ZFS (you may need to clear errors). Wouldn't this be the same as above hot-plugging disk by disk? Last option: use another server where you try to import (mainboard, power, hba or backplane problem) remove all disks and do a nondestructive or smart test on another machine Sadly we do not have a spare chassis with 40 slots around to test this. I am so far unconvinced that this is a hardware problem though. We will most likely boot up into linux live CD to run smartctl and see if it has any information on the disks. -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
On Jul 12, 2015, at 9:18 PM, Richard Elling richard.ell...@richardelling.com wrote: Dan, if you're listening, Matt would be the best person to weigh-in on this. Yes he would be, Richard.. The panic in the arc_get_data_buf() paths is similar to older problems we'd seen in r151006. Derek, do you have a kernel coredump from these? I know you've been panic-and-reboot-and-panic-ing, but if you can get savecore(1M) to do its thing, having that dump would be useful. Thanks, Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
On Sat, 11 Jul 2015, Derek Yarnell wrote: Hi, We just have had a catastrophic event on one of our OmniOS r14 file servers. In what seems to have been triggered by the weekly scrub of its one large zfs pool (~100T) it panics. This made it basically reboot continually and we have installed a second copy of OmniOS r14 in the mean time. We are able to mount the pool readonly and are currently securing the data as soon as possible. The on-going scrub automatically restarts, apparently even in read-only mode. You should 'zpool scrub -s poolname' ASAP after boot (if you can) to stop the ongoing scrub. ### After mounting in readonly mode pool: zvol00 state: ONLINE status: The pool is formatted using a legacy on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on software that does not support feature flags. scan: scrub in progress since Sat Jul 11 11:00:02 2015 2.24G scanned out of 69.5T at 1/s, (scan is slow, no estimated time) 0 repaired, 0.00% done Observe evidence of the re-started scrub. This may be tickling the problem which causes the panic. The underlying problem needs to be identified and fixed. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
The on-going scrub automatically restarts, apparently even in read-only mode. You should 'zpool scrub -s poolname' ASAP after boot (if you can) to stop the ongoing scrub. We have tried to stop the scrub but it seems you can not cancel a scrub when the pool is mounted readonly. -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] ZFS crash/reboot loop
Hi, We just have had a catastrophic event on one of our OmniOS r14 file servers. In what seems to have been triggered by the weekly scrub of its one large zfs pool (~100T) it panics. This made it basically reboot continually and we have installed a second copy of OmniOS r14 in the mean time. We are able to mount the pool readonly and are currently securing the data as soon as possible. The underlying zpool (zvol00) was a Nexenta that was imported on June 18th and had gotten through more than one weekly scrub since we had upgraded. We had not upgraded the pool from 28 yet and we are not using dedup. I have the the vmdump[0] file available if that is useful from when we tried running `zpool import zvol00`. The hardware should be fully compatible Supermicro X8DTH-i/6/iF/6F, Hitachi/HGST SAS drives, STEC ZeusRAM slogs, STEC l2arc drives with LSI 2008 controllers. [0] - https://obj.umiacs.umd.edu/derek_support/vmdump.0 TIME UUID SUNW-MSG-ID Jul 11 2015 21:23:40.015738000 f120cfb4-7e83-c576-ca89-e522daf796e7 SUNOS-8000-KL TIME CLASS ENA Jul 11 21:23:39.9681 ireport.os.sunos.panic.dump_available 0x Jul 11 21:23:24.1537 ireport.os.sunos.panic.dump_pending_on_device 0x nvlist version: 0 version = 0x0 class = list.suspect uuid = f120cfb4-7e83-c576-ca89-e522daf796e7 code = SUNOS-8000-KL diag-time = 1436664219 981033 de = fmd:///module/software-diagnosis fault-list-sz = 0x1 fault-list = (array of embedded nvlists) (start fault-list[0]) nvlist version: 0 version = 0x0 class = defect.sunos.kernel.panic certainty = 0x64 asru = sw:///:path=/var/crash/unknown/.f120cfb4-7e83-c576-ca89-e522daf796e7 resource = sw:///:path=/var/crash/unknown/.f120cfb4-7e83-c576-ca89-e522daf796e7 savecore-succcess = 1 dump-dir = /var/crash/unknown dump-files = vmdump.0 os-instance-uuid = f120cfb4-7e83-c576-ca89-e522daf796e7 panicstr = assertion failed: c (1ULL 24) 9 (0x7f 0x8000), file: ../../common/fs/zfs/zio.c, line: 221 panicstack = fba8b13d () | zfs:zio_buf_alloc+49 () | zfs:arc_get_data_buf+12b () | zfs:arc_buf_alloc+d2 () | zfs:arc_read+15b () | zfs:dsl_scan_prefetch+da () | zfs:dsl_scan_recurse+13c () | zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_visitdnode+b4 () | zfs:dsl_scan_recurse+435 () | zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_recurse+1b9 () | zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_recurse+1b9 () | zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_visitdnode+b4 () | zfs:dsl_scan_recurse+496 () | zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_visit_rootbp+5d () | zfs:dsl_scan_visit+273 () | zfs:dsl_scan_sync+247 () | zfs:spa_sync+2b3 () | zfs:txg_sync_thread+227 () | unix:thread_start+8 () | crashtime = 1436664019 panic-time = Sat Jul 11 21:20:19 2015 EDT (end fault-list[0]) fault-status = 0x1 severity = Major __ttl = 0x1 __tod = 0x55a1c19c 0xf02490 ### In readonly mode root@cbcbomni02:/root# zdb -e -bcsvL zvol00 assertion failed for thread 0xfd7fff162a40, thread-id 1: !claimed || !(zh-zh_flags ZIL_CLAIM_LR_SEQ_VALID) || (max_blk_seq == claim_blk_seq max_lr_seq == claim_lr_seq), file ../../../uts/common/fs/zfs/zil.c, line 367 Abort (core dumped) ### After mounting in readonly mode pool: zvol00 state: ONLINE status: The pool is formatted using a legacy on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on software that does not support feature flags. scan: scrub in progress since Sat Jul 11 11:00:02 2015 2.24G scanned out of 69.5T at 1/s, (scan is slow, no estimated time) 0 repaired, 0.00% done config: NAME STATE READ WRITE CKSUM zvol00 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 c4t5000CCA01A91D6F9d0 ONLINE 0 0 0 c4t5000CCA01A9D4B6Dd0 ONLINE 0 0 0 c4t5000CCA01A9ED1A9d0 ONLINE 0 0 0 c4t5000CCA01A9ED311d0 ONLINE 0 0 0 c4t5000CCA01A9ED345d0 ONLINE 0 0 0 c4t5000CCA01A9EDD21d0 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 c4t5000CCA01A9EE1D1d0 ONLINE 0 0 0 c4t5000CCA01A9EE3D9d0 ONLINE 0 0 0 c4t5000CCA01A9F6659d0 ONLINE 0 0 0 c4t5000CCA01A9F69C5d0 ONLINE 0 0 0 c4t5000CCA01A9F81A9d0 ONLINE
