Hi all, As I wrote before, I have a "dpcool" implemented as an iSCSI device stored in a volume of my physical "pool". When there are many operations, such as attempts to destroy a dataset (which leads to many small IOs in my config), the iSCSI device is 100% busy for hours, latencies can grow to seconds even with one task in queue max, and I believe this may lead to the following problem - the device is considered timed out and reportedly faulted: === # zpool status dcpool pool: dcpool state: ONLINE status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scan: resilvered 111M in 0h5m with 0 errors on Wed May 18 04:03:02 2011 config: NAME STATE READ WRITE CKSUM dcpool ONLINE 7 129K 0 c0t600144F09844CF0000004D8376AE0002d0 ONLINE 0 397K 0 cache c4t1d0p7 ONLINE 0 0 0 errors: 132410 data errors, use '-v' for a list === As we can see above, there are no devices actually marked as FAULTED, except that the status says so. At times I had dmesg reporting the failure as well, but not always, like this: === May 22 12:51:32 bofh-sol scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0): May 22 12:51:32 bofh-sol /scsi_vhci/disk@g600144f09844cf0000004d8376ae0002 (sd10): Command Timeout on path iscsi0/d...@0000iqn.1986-03.com.sun:02:380c2eb8-4a4a-6e78-e304-c8cd5e29a7a40001,0 May 22 12:51:32 bofh-sol iscsi: [ID 431120 kern.warning] WARNING: iscsi connection(7/3f) closing connection - target requested reason:0x7 May 22 12:51:32 bofh-sol iscsi: [ID 732673 kern.info] NOTICE: iscsi session(6) iqn.1986-03.com.sun:02:380c2eb8-4a4a-6e78-e304-c8cd5e29a7a4 offline May 22 12:51:39 bofh-sol scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk@g600144f09844cf0000004d8376ae0002 (sd10): May 22 12:51:39 bofh-sol drive offline May 22 12:51:54 bofh-sol scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk@g600144f09844cf0000004d8376ae0002 (sd10): May 22 12:51:54 bofh-sol drive offline May 22 12:52:00 bofh-sol scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk@g600144f09844cf0000004d8376ae0002 (sd10): ... May 22 13:04:35 bofh-sol scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk@g600144f09844cf0000004d8376ae0002 (sd10): May 22 13:04:35 bofh-sol drive offline May 22 13:04:45 bofh-sol iscsi: [ID 559844 kern.info] NOTICE: iscsi session(6) iqn.1986-03.com.sun:02:380c2eb8-4a4a-6e78-e304-c8cd5e29a7a4 online May 22 13:04:45 bofh-sol genunix: [ID 483743 kern.info] /scsi_vhci/disk@g600144f09844cf0000004d8376ae0002 (sd10) multipath status: degraded: path 1 iscsi0/d...@0000iqn.1986-03.com.sun:02:380c2eb8-4a4a-6e78-e304-c8cd5e29a7a40001,0 is online May 22 13:04:46 bofh-sol fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-HC, TYPE: Error, VER: 1, SEVERITY: Major May 22 13:04:46 bofh-sol EVENT-TIME: Sun May 22 13:04:46 MSD 2011 May 22 13:04:46 bofh-sol PLATFORM: System-Product-Name, CSN: System-Serial-Number, HOSTNAME: bofh-sol May 22 13:04:46 bofh-sol SOURCE: zfs-diagnosis, REV: 1.0 May 22 13:04:46 bofh-sol EVENT-ID: 435a2ccd-e710-cd37-a18e-a46743002e56 May 22 13:04:46 bofh-sol DESC: The ZFS pool has experienced currently unrecoverable I/O May 22 13:04:46 bofh-sol failures. Refer to http://sun.com/msg/ZFS-8000-HC for more information. May 22 13:04:46 bofh-sol AUTO-RESPONSE: No automated response will be taken. May 22 13:04:46 bofh-sol IMPACT: Read and write I/Os cannot be serviced. May 22 13:04:46 bofh-sol REC-ACTION: Make sure the affected devices are connected, then run May 22 13:04:46 bofh-sol 'zpool clear'. === The zillions of errors are in ZFS metadata as seen in "zpool status -v", and this condition can be cleared by forced reboot (uadmin 1 8) and subsequent import of the "dcpool", when after a few hours it comes online with deferred-free blocks deleted. Neither "zpool export" nor "zpool clear" help in this situation, and may instead lead to a hang of the system (no IOs would succeed, even to other pools). I've tried zpool attributes failmode=wait and =continue, with same results. I am not sure what to make of the two reports (in dmesg and zpool status) - possibly when the system bottleneck clears, the iSCSI device becomes visible again - but IOs are no longer served. I did not manage to restart iSCSI target and/or initiator SMF services, because in my case importing/mounting of dcpool is wrapped in a service which depends on the iSCSI services, and umounting/exporting hangs - but I'd keep trying somehow (suggestions welcome, so far I think of breaking the dependency and/or disabling my dcpool-mount service). While this particular system is at home and I can afford to reboot it, I do wonder what should be properly done in production environments? I.e. is it possible to multipath iSCSI instances on different local IP addresses (i.e. localhost and NIC IP) - but that would probably fail at the same bottleneck moment - or to connect to the zvol/rdsk/... directly, without iSCSI? Thanks for ideas, //Jim Klimov
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
