comment below...
On May 22, 2011, at 8:31 AM, Jim Klimov wrote:
Hi all,
As I wrote before, I have a dpcool implemented as an iSCSI
device stored in a volume of my physical pool. When there
are many operations, such as attempts to destroy a dataset
(which leads to many small IOs in my config), the iSCSI
device is 100% busy for hours, latencies can grow to seconds
even with one task in queue max, and I believe this may lead
to the following problem - the device is considered timed out
and reportedly faulted:
===
# zpool status dcpool
pool: dcpool
state: ONLINE
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see: http://www.sun.com/msg/ZFS-8000-HC
scan: resilvered 111M in 0h5m with 0 errors on Wed May 18 04:03:02 2011
config:
NAME STATE READ WRITE CKSUM
dcpool ONLINE 7 129K 0
c0t600144F09844CF004D8376AE0002d0 ONLINE 0 397K 0
cache
c4t1d0p7 ONLINE 0 0 0
errors: 132410 data errors, use '-v' for a list
===
As we can see above, there are no devices actually marked as
FAULTED, except that the status says so. At times I had dmesg
reporting the failure as well, but not always, like this:
===
May 22 12:51:32 bofh-sol scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci
(scsi_vhci0):
May 22 12:51:32 bofh-sol
/scsi_vhci/disk@g600144f09844cf004d8376ae0002 (sd10): Command Timeout on
path
iscsi0/d...@iqn.1986-03.com.sun:02:380c2eb8-4a4a-6e78-e304-c8cd5e29a7a40001,0
May 22 12:51:32 bofh-sol iscsi: [ID 431120 kern.warning] WARNING: iscsi
connection(7/3f) closing connection - target requested reason:0x7
This means that the target closed the connection because there was already a
task in progress.
Likely this was the retry after the timeout. By default, these timeouts are
quite long, so by now
performance is already terrible.
I'm not sure if you can make the iscsi target any slower. I've seen some
well-known-manufacturer's
iscsi arrays take 2-3 minutes to respond, in a repeatable manner. The best
advice is to fix it or
replace it. I don't think there is anything you can do on the initiator side to
get to a state of
happiness.
-- richard
May 22 12:51:32 bofh-sol iscsi: [ID 732673 kern.info] NOTICE: iscsi
session(6) iqn.1986-03.com.sun:02:380c2eb8-4a4a-6e78-e304-c8cd5e29a7a4 offline
May 22 12:51:39 bofh-sol scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk@g600144f09844cf004d8376ae0002 (sd10):
May 22 12:51:39 bofh-soldrive offline
May 22 12:51:54 bofh-sol scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk@g600144f09844cf004d8376ae0002 (sd10):
May 22 12:51:54 bofh-soldrive offline
May 22 12:52:00 bofh-sol scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk@g600144f09844cf004d8376ae0002 (sd10):
...
May 22 13:04:35 bofh-sol scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk@g600144f09844cf004d8376ae0002 (sd10):
May 22 13:04:35 bofh-soldrive offline
May 22 13:04:45 bofh-sol iscsi: [ID 559844 kern.info] NOTICE: iscsi
session(6) iqn.1986-03.com.sun:02:380c2eb8-4a4a-6e78-e304-c8cd5e29a7a4 online
May 22 13:04:45 bofh-sol genunix: [ID 483743 kern.info]
/scsi_vhci/disk@g600144f09844cf004d8376ae0002 (sd10) multipath status:
degraded: path 1
iscsi0/d...@iqn.1986-03.com.sun:02:380c2eb8-4a4a-6e78-e304-c8cd5e29a7a40001,0
is online
May 22 13:04:46 bofh-sol fmd: [ID 377184 daemon.error] SUNW-MSG-ID:
ZFS-8000-HC, TYPE: Error, VER: 1, SEVERITY: Major
May 22 13:04:46 bofh-sol EVENT-TIME: Sun May 22 13:04:46 MSD 2011
May 22 13:04:46 bofh-sol PLATFORM: System-Product-Name, CSN:
System-Serial-Number, HOSTNAME: bofh-sol
May 22 13:04:46 bofh-sol SOURCE: zfs-diagnosis, REV: 1.0
May 22 13:04:46 bofh-sol EVENT-ID: 435a2ccd-e710-cd37-a18e-a46743002e56
May 22 13:04:46 bofh-sol DESC: The ZFS pool has experienced currently
unrecoverable I/O
May 22 13:04:46 bofh-solfailures. Refer to
http://sun.com/msg/ZFS-8000-HC for more information.
May 22 13:04:46 bofh-sol AUTO-RESPONSE: No automated response will be taken.
May 22 13:04:46 bofh-sol IMPACT: Read and write I/Os cannot be serviced.
May 22 13:04:46 bofh-sol REC-ACTION: Make sure the affected devices are
connected, then run
May 22 13:04:46 bofh-sol'zpool clear'.
===
The zillions of errors are in ZFS metadata as seen in zpool status -v,
and this condition can be cleared by forced reboot (uadmin 1 8) and
subsequent import of the dcpool, when after a few hours it comes
online with deferred-free blocks deleted.
Neither zpool export nor zpool clear help in this situation,
and may instead lead to a hang of the system (no IOs would
succeed, even to other pools).
I've tried zpool attributes