Hi, I got the same problem this morning.
Thanks to Alasdair suggestion, I commented out the zfs "quota"
command from /etc/profile, so I
could enter the bash prompt and investigate.
As a summary of the problem:
- 3 zfs pools (rpool, areca, adpatec) each on a different
controller: rpool as a mirror of internal disks, areca as raidz
on 7 sata disks of the areca controller plus half space of SD as
zlog, adaptec as raidz on 8 disks of the adaptec controller plus
half space of SD as zlog.
- areca space is used for NFS sharing to unix servers, and always
responds.
- adaptec space is used for CIFS sharing and an iScsi volume for
the Windows PDC.
Then we have a vmware server with an iscsi resource store, given
to the virtualized PDC as a secondary disk for sqlserver data and
some more. PDC boots directly from the vmware server disks.
At once, both CIFS from the storage and PDC iscsi disk do not
respond.
CIFS fails probably because the PDC AD is not responding,
probably busy checking the iscsi disk, in a loop.
Going into the storage bash:
- zpool status shows everything find, every pool is correct.
- /var/adm/messages shows only smbd time outs with the PDC, no
hardware or zfs problem.
- at the time of failure, fmdump -eVvm showed same previously
found errors, 3 days earlier
- after rebooting all the infrastructure, fmdump -eVvm showed
same previously found errors around the time of rebooting the
storage, not at the time of the experienced failures. We find one
stated error for each disk of the adaptec controller (cut & paste
at the end)
- a zfs list areca/* showed all the areca filesystems
- a zfs list adaptec blocked never returning
- any access to the zfs structure of the adaptec would block
- as suggested by Alasdair, I ran savecore -L (I checked to have
the dump device and with enough space)
- the savecore command ran for sometime until reaching 100%, then
blocked never returning.
- I could not "init 5" the storage, never returning
- I tried sending the poweroff signal with the power button,
console showed its intention to power off, but never did it
- I forced powered off via the power button.
- Once everything was powered on again, everything ran fine.
- I looked for the dump into /var/crash, but I had no /var/crash
at all.
How can I investigate further this problem?
Do I have any chance to find the savecore output in the dump
device, even if I have not /var/crash?
Here is the fmdump output:
Nov 23 2012 09:25:04.422282821 ereport.io.scsi.cmd.disk.dev.uderr
nvlist version: 0
class = ereport.io.scsi.cmd.disk.dev.uderr
ena = 0x2b08d604c300001
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = dev
device-path =
/pci@0,0/pci8086,3595@2/pci8086,370@0/pci9005,2bc@e/disk@7,0
devid = id1,sd@TAdaptec_3805____________8366BAF3
(end detector)
devid = id1,sd@TAdaptec_3805____________8366BAF3
driver-assessment = fail
op-code = 0x1a
cdb = 0x1a 0x0 0x8 0x0 0x18 0x0
pkt-reason = 0x0
pkt-state = 0x1f
pkt-stats = 0x0
stat-code = 0x0
un-decode-info = sd_get_write_cache_enabled: Mode Sense caching
page code mismatch 0
__ttl = 0x1
----------------------------------------------------------------------------------
Da: Alasdair Lumsden <[email protected]>
A: [email protected]
Data: 10 novembre 2012 14.37.20 CET
Oggetto: Re: [discuss] illumos based ZFS storage failure
I haven't read the whole thread, but the next time it happens
you'll
want to invoke a panic and make the dump file available.
You'll want to
ensure that:
1. Multithreaded dump is disabled in /etc/system with:
* Disable MT dump
set dump_plat_mincpu=0
Without this there is a risk of your dump not saving correctly.
2. That you have a dump device and that it's big enough to
capture your
kernel size (zfs set volsize=X rpool/dump)
3. That dumpadm is happy and set to save cores etc:
dumpadm -y -z on -c kernel -d /dev/zvol/dsk/rpool/dump
There's lots of good info here:
http://wiki.illumos.org/display/illumos/How+To+Report+Problems
You can also inspect things with mdb while the system is up,
but if it's
a production system normally you want to get it rebooted and
into
production again ASAP. So in that situation, you can take a
dump of the
running system with:
savecore -L
One thing to keep in mind is /etc/profile runs
/usr/sbin/quota, which
can screw over logins when the zfs subsystem is unhappy. I
really think
it should be removed by default since on most systems quotas
aren't even
used. So comment it out - we do so on all our systems. This
will give
you a better chance of logging in when things go wrong.
I think there's a way to SSH in bypassing /etc/profile but I
can't
remember what it is - perhaps someone can chime in.
Good luck. Centralised storage is difficult to do and when it
goes wrong
everything that depends on it goes down. It's a "all your
eggs in one
giant failbasket". Doing it homebrew with ZFS is cost
effective and can
be fast, but it is also risky. This is why there are
companies like
Nexenta out there with certified combinations of hardware and
software
engineered to work together. This extends to validating firmware
combinations of disks/HBAs/etc.
Cheers,
Alasdair
-------------------------------------------
illumos-discuss
Archives: https://www.listbox.com/member/archive/182180/=now
RSS Feed:
https://www.listbox.com/member/archive/rss/182180/21175541-02f10c6f
Modify Your Subscription: https://www.listbox.com/member/?&;
Powered by Listbox: http://www.listbox.com
*illumos-discuss* | Archives
<https://www.listbox.com/member/archive/182180/=now>
<https://www.listbox.com/member/archive/rss/182180/21175546-6311bfb2>
| Modify
<https://www.listbox.com/member/?&;>
Your Subscription [Powered by Listbox] <http://www.listbox.com>
