Re: [zfs-discuss] Disk failure chokes all the disks attached to thefailingdisk HBA
seal.macc.unican.es scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc May 31 10:52:47 seal.macc.unican.es scsi: [ID 365881 kern.info] /pci@7a,0/pci8086,3410@9/pci1000,3140@0 (mpt2): May 31 10:52:47 seal.macc.unican.es Log info 0x3113 received for target 11. May 31 10:52:47 seal.macc.unican.es scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc May 31 10:52:51 seal.macc.unican.es scsi: [ID 243001 kern.warning] WARNING: /pci@7a,0/pci8086,3410@9/pci1000,3140@0 (mpt2): May 31 10:52:51 seal.macc.unican.es mpt_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x3000 May 31 10:52:51 seal.macc.unican.es scsi: [ID 243001 kern.warning] WARNING: /pci@7a,0/pci8086,3410@9/pci1000,3140@0 (mpt2): May 31 10:52:51 seal.macc.unican.es mpt_handle_event: IOCStatus=0x8000, IOCLogInfo=0x3000 May 31 10:52:53 seal.macc.unican.es scsi: [ID 365881 kern.info] /pci@7a,0/pci8086,3410@9/pci1000,3140@0 (mpt2): May 31 10:52:53 seal.macc.unican.es Log info 0x3000 received for target 11. May 31 10:52:53 seal.macc.unican.es scsi_status=0x0, ioc_status=0x804b, scsi_state=0xc May 31 10:52:56 seal.macc.unican.es scsi: [ID 243001 kern.warning] WARNING: /pci@7a,0/pci8086,3410@9/pci1000,3140@0 (mpt2): May 31 10:52:56 seal.macc.unican.es SAS Discovery Error on port 0. DiscoveryStatus is DiscoveryStatus is |Unaddressable device found| May 31 10:53:37 seal.macc.unican.es scsi: [ID 107833 kern.warning] WARNING: /pci@7a,0/pci8086,3410@9/pci1000,3140@0 (mpt2): May 31 10:53:37 seal.macc.unican.es passthrough command timeout May 31 10:53:37 seal.macc.unican.es scsi: [ID 365881 kern.info] /pci@7a,0/pci8086,3410@9/pci1000,3140@0 (mpt2): May 31 10:53:37 seal.macc.unican.es Rev. 8 LSI, Inc. 1068E found. May 31 10:53:37 seal.macc.unican.es scsi: [ID 365881 kern.info] /pci@7a,0/pci8086,3410@9/pci1000,3140@0 (mpt2): May 31 10:53:37 seal.macc.unican.es mpt2 supports power management. May 31 10:53:37 seal.macc.unican.es scsi: [ID 365881 kern.info] /pci@7a,0/pci8086,3410@9/pci1000,3140@0 (mpt2): May 31 10:53:37 seal.macc.unican.es mpt2: IOC Operational. May 31 10:54:10 seal.macc.unican.es fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major May 31 10:54:10 seal.macc.unican.es EVENT-TIME: Thu May 31 10:54:09 CEST 2012 May 31 10:54:10 seal.macc.unican.es PLATFORM: X8DTH-i-6-iF-6F, CSN: 1234567890, HOSTNAME: seal.macc.unican.es May 31 10:54:10 seal.macc.unican.es SOURCE: zfs-diagnosis, REV: 1.0 May 31 10:54:10 seal.macc.unican.es EVENT-ID: 5d33a13b-61e3-cf16-86a7-e9587d510170 May 31 10:54:10 seal.macc.unican.es DESC: The number of I/O errors associated with a ZFS device exceeded May 31 10:54:10 seal.macc.unican.es acceptable levels. Refer to http://sun.com/msg/ZFS-8000-FD for more information. May 31 10:54:10 seal.macc.unican.es AUTO-RESPONSE: The device has been offlined and marked as faulted. An attempt May 31 10:54:10 seal.macc.unican.es will be made to activate a hot spare if available. May 31 10:54:10 seal.macc.unican.es IMPACT: Fault tolerance of the pool may be compromised. May 31 10:54:10 seal.macc.unican.es REC-ACTION: Run 'zpool status -x' and replace the bad device. -- Antonio S. Cofiño Grupo de Meteorología de Santander Dep. de Matemática Aplicada y Ciencias de la Computación Universidad de Cantabria Escuela de Caminos Avenida de los Castros, 44 39005 Santander, Spain Tel: (+34) 942 20 1731 Fax: (+34) 942 20 1703 http://www.meteo.unican.es mailto:antonio.cof...@unican.es El 30/05/2012 18:52, Jim Klimov escribió: 2012-05-30 20:25, Antonio S. Cofiño wrote: Dear All, It may be this not the correct mailing list, but I'm having a ZFS issue when a disk is failing. I hope other users might help more on specific details, but while we're waiting for their answer - please search the list archives. Similar description of the problem comes up every few months, and it seems to be a fundamental flaw of (consumerish?) SATA drives with backplanes, leading to reset storms. I remember the mechanism being something like this: a problematic disk is detected and the system tries to have it reset so that it might stop causing problems. The SATA controller either ignores the command or takes too long to complete/respond, so the system goes up the stack and next resets the backplane or ultimately the controller. I am not qualified to comment whether this issue is fundamental (i.e. in SATA protocols) or incidental (cheap drives don't do advanced stuff, while expensive SATAs might be ok in this regard). There were discussions about using SATA-SAS interposers, but they might not fit mechanically, add latency and instability, and raise the system price to the point where native SAS disks would be better... Now, waiting for experts to chime in on whatever I missed ;) HTH, //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org
Re: [zfs-discuss] Disk failure chokes all the disks attached to thefailingdisk HBA
Markus, After Jim's answer I have started to read bout the well known issue. Is it just mpt causing the errors or also mpt_sas? Both drivers are causing the reset storm (See my answer to Jim's e-mail). General consensus from various people: don't use SATA drives on SAS back- planes. Some SATA drives might work better, but there seems to be no guarantee. And even for SAS-SAS, try to avoid SAS1 backplanes. In the Paul Kraus's answer it mentions that Oracle support says (among other things) 4. the problem happens with SAS as well as SATA drives, but is much less frequent That means, that using SAS drives it will reduce the probability of the issue but no guarantee exists. General consensus from various people: don't use SATA drives on SAS back- planes. Some SATA drives might work better, but there seems to be no guarantee. And even for SAS-SAS, try to avoid SAS1 backplanes. Yes, may be the 'general consensus' is right but 'general consensus' said me to use hardware based raid solutions. But I started to do 'risky business' (as some vendors told me) using ZFS and have ended discovering how robust is ZFS for this kind of protocol errors. From my complete naive point of view it appears more a issue with the HBA's FW than a issue with SATA drives. With you answers I have make a lot of re-search helping me to learn new things. Please more comments and help are welcome (from some SAS expert?). Antonio -- Antonio S. Cofiño El 31/05/2012 18:04, Weber, Markus escribió: Antonio S. Cofiño wrote: [...] The system is a supermicro motherboard X8DTH-6F in a 4U chassis (SC847E1-R1400LPB) and an external SAS2 JBOD (SC847E16-RJBOD1). It makes a system with a total of 4 backplanes (2x SAS + 2x SAS2) each of them connected to a 4 different HBA (2x LSI 3081E-R (1068 chip) + 2x LSI SAS9200-8e (2008 chip)). This system is has a total of 81 disk (2x SAS (SEAGATE ST3146356SS) + 34 SATA3 (Hitachi HDS722020ALA330) + 45 SATA6 (Hitachi HDS723020BLA642)) The issue arise when one of the disk starts to fail making long time accesses. After some time (minutes, but I'm not sure) all the disks, connected to the same HBA, start to report errors. This situation produce a general failure on the ZFS making the whole POOL unavailable. [...] Have been there and gave up at the end[1]. Could reproduce (even though it took a bit longer) under most Linux versions (incl. using latest LSI drivers) and LSI 3081E-R HBA. Is it just mpt causing the errors or also mpt_sas? In a lab environment the LSI 9200 HBA behaved better - I/O only dropped shortly and then continued on the other disks without generating errors. Had a lengthy Oracle case on this, but all proposed workarounds did not worked for me at all, which had been (some also from other forums) - disabling NCQ - allow-bus-device-reset=0; to /kernel/drv/sd.conf - set zfs:zfs_vdev_max_pending=1 - set mpt:mpt_enable_msi=0 - keep usage below 90% - no fmservices running and did temporarily did fmadm unload disk-transport or other disk access stuff (smartd?) - tried changing retries-timeout via sd-conf for the disks without any success and ended it doing via mdb At the end I knew the bad sector of the bad disk and by simply dd this sector once or twice to /dev/zero I could easily bring down the system/pool without any load on the disk system. General consensus from various people: don't use SATA drives on SAS back- planes. Some SATA drives might work better, but there seems to be no guarantee. And even for SAS-SAS, try to avoid SAS1 backplanes. Markus [1] Search for What's wrong with LSI 3081 (1068) + expander + (bad) SATA disk? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Disk failure chokes all the disks attached to the failing disk HBA
Dear All, It may be this not the correct mailing list, but I'm having a ZFS issue when a disk is failing. The system is a supermicro motherboard X8DTH-6F in a 4U chassis (SC847E1-R1400LPB) and an external SAS2 JBOD (SC847E16-RJBOD1). It makes a system with a total of 4 backplanes (2x SAS + 2x SAS2) each of them connected to a 4 different HBA (2x LSI 3081E-R (1068 chip) + 2x LSI SAS9200-8e (2008 chip)). This system is has a total of 81 disk (2x SAS (SEAGATE ST3146356SS) + 34 SATA3 (Hitachi HDS722020ALA330) + 45 SATA6 (Hitachi HDS723020BLA642)) The system is controlled by Opensolaris (snv_134) and it work normally. All the SATA disks are part of the same pool separate by raidz2 vdev composed by 11 (~) disks. The issue arise when on of the disk starts to fail making long time accesses. After some time (minutes, but I'm not sure) all the disks, connected to the same HBA, start to report errors. This situation produce a general failure on the ZFS making the whole POOL unavailable. Identifying the original failed disk producing access errors and removing it the pool starts to resilver with no problem, and all the spurious errors produced by the general error are recovered. My question is, there is anyway to anticipate this choking situation when a disk is failing, to avoid the general failure? Any help or suggestion is welcome. Regards Antonio -- -- Antonio S. Cofiño Grupo de Meteorología de Santander Dep. de Matemática Aplicada y Ciencias de la Computación Universidad de Cantabria Escuela de Caminos Avenida de los Castros, 44 39005 Santander, Spain Tel: (+34) 942 20 1731 Fax: (+34) 942 20 1703 http://www.meteo.unican.es mailto:antonio.cof...@unican.es ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
