Hi, On Thu, Jun 04, 2009 at 03:47:03PM +0100, [email protected] wrote: > Hello, > this is a little bit off topic but maybe anybody can help me. The physical > setup of our Debian etch heartbeat cluster looks like this: > > Cluster of two nodes connected with bonding to the network, with a crossover > connection to each other (interconnect) and with dual port Qlogic HBAs to our > SAN. The SAN consists of two independent (physical) SANs with different VSANs > (just to be more human readable) connected to some EMC clariions CX3-20f. > > We use multipathd, LVM2 and ocfs2. This is the multipath.conf: > > defaults { > rr_min_io 1000 > polling_interval 2 > no_path_retry 5 > user_friendly_names yes > } > > blacklist { > devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" > devnode "^hd[a-z][[0-9]*]" > devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]" > device { > vendor "DGC" > product "LUNZ" #EMC Clariion management LUN > } > device { > vendor "ATA" #We do not need mutlipathing for local drives > product "*" > } > device { > vendor "AMI" # No multipathing for SUN Virtual devices > product "*" > } > device { > vendor "HITACHI" # No multipathing for local scsi disks > product "H101414SCSUN146G" > } > } > > devices { > ## Device attributes for EMC CLARiiON > device { > vendor "DGC" > product "*" > path_grouping_policy group_by_prio > getuid_callout "/sbin/scsi_id -g -u -s /block/%n" > prio_callout "/sbin/mpath_prio_emc /dev/%n" > hardware_handler "1 emc" > features "1 queue_if_no_path" > no_path_retry fail > path_checker emc_clariion > path_selector "round-robin 0" > failback immediate > } > } > > multipaths { > multipath { > wwid 3600601603ac511001c7c92fec775dd11 > alias stosan01_lun070 > } > } > > > And we changed the "filter" of lvm2 to not scan /dev/sd* devices. After > booting up the system multipath -ll looks like this: > > stosan01_lun070 (3600601603ac511001c7c92fec775dd11) dm-7 DGC,RAID 5 > [size=133G][features=0][hwhandler=1 emc] > \_ round-robin 0 [prio=2][active] > \_ 0:0:1:1 sdd 8:48 [active][ready] > \_ 3:0:1:1 sdh 8:112 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:0:1 sdb 8:16 [active][ready] > \_ 3:0:0:1 sdf 8:80 [active][ready] > > If we now kill one of the connections by disable the zoning on one of our > SANs this changes to: > > stosan01_lun070 (3600601603ac511001c7c92fec775dd11) dm-7 DGC,RAID 5 > [size=133G][features=0][hwhandler=1 emc] > \_ round-robin 0 [prio=1][active] > \_ 0:0:1:1 sdd 8:48 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ 0:0:0:1 sdb 8:16 [active][ready] > > And /var/log/messages > > Jun 4 14:53:58 defr1elcbtd02 kernel: o2net: no longer connected to node > defr1elcbtd01 (num 0) at 192.168.0.101:7777 > Jun 4 14:54:01 defr1elcbtd02 kernel: (4741,0):dlm_wait_for_node_death:374 > 5EE89BC01EFC405E9197C198DEEAE678: waiting 5000ms for notification of death of > node 0 > Jun 4 14:54:16 defr1elcbtd02 last message repeated 3 times > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath: Failing path > 8:80. > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath: Failing path > 8:112. > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath emc: long > trespass command will be send > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath emc: honor > reservation bit will not be set (default) > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: ioctl: error adding > target to table
? > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath emc: long > trespass command will be send > Jun 4 14:54:21 defr1elcbtd02 kernel: device-mapper: multipath emc: honor > reservation bit will not be set (default) > Jun 4 14:54:21 defr1elcbtd02 kernel: (4741,0):dlm_wait_for_node_death:374 > 5EE89BC01EFC405E9197C198DEEAE678: waiting 5000ms for notification of death of > node 0 > Jun 4 14:54:26 defr1elcbtd02 kernel: scsi 1:0:1:1: SCSI error: return code = > 0x00020000 This looks like a multipath issue. AFAIK, the SCSI layer shouldn't perceive a change in path availability. Perhaps better to post to another list where such problems are discussed. Thanks, Dejan > Jun 4 14:54:26 defr1elcbtd02 kernel: end_request: I/O error, dev sdh, sector > 1672 > Jun 4 14:54:26 defr1elcbtd02 kernel: device-mapper: multipath emc: > emc_pg_init: sending switch-over command > Jun 4 14:54:26 defr1elcbtd02 kernel: (4741,0):dlm_wait_for_node_death:374 > 5EE89BC01EFC405E9197C198DEEAE678: waiting 5000ms for notification of death of > node 0 > Jun 4 14:54:57 defr1elcbtd02 last message repeated 6 times > > After 60 seconds the kernel panics because of ocfs2 is fencing this node: > > o2quo_make_decision:143 ERROR: fencing this node because it is connected to a > half-quorum of 1 out of 2 nodes which doesn't include the lowest active node 0 > > The o2cb settings are: > O2CB_HEARTBEAT_THRESHOLD=31 > O2CB_IDLE_TIMEOUT_MS=30000 > O2CB_KEEPALIVE_DELAY_MS=2000 > O2CB_RECONNECT_DELAY_MS=2000 > > If we just set the FC Port on one of the switches to shutdown everything > works fine. Qla2xxx detects port down, multipathd detects the missing path > and ocfs2 is not recognizing anything - just working fine like it should... > > Whats going on? Is this a bug? > > Regards Florian > > > Florian Engelmann | Unix System and Storage Administrator | IT Services > Germany | BT (Germany) GmbH & Co. oHG | Tel:+49 (0)89 2600 8173 | Mob:+49 > (0)176 1073 1081 | Fax:+49 (0)89 2600 4955 | Email: [email protected] > | Sitz und Registergericht der oHG: M?nchen, HRA 77639 | Pers?nlich haftende > Gesellschafterin der oHG: BT Deutschland GmbH | Sitz und Registergericht: > M?nchen, HRB 132307 | Gesch?ftsf?hrung: Karsten Lereuth, Joachim Piroth, Dr. > Sebastian Brandis, Martin Glock, Neil Smith, Judith Sasse | > www.bt.com/globalservices > > Diese E-Mail enth?lt Informationen von BT (Germany) GmbH & Co. oHG. Diese > sind m?glicherweise vertraulich und ausschlie?lich f?r den Adressaten > bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den > Adressaten ist untersagt. Sollten Sie nicht der f?r diese E-Mail bestimmte > Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder > Weitergabe untersagt. Sollten Sie diese elektronische Nachricht > irrt?mlicherweise erhalten haben, so informieren Sie uns bitte unverz?glich > telefonisch oder per E-Mail (Kontaktdaten obenstehend). Das E-Mail-System > wird von BT (Germany) GmbH & Co. oHG im Rahmen der gesetzlichen Bestimmungen > laufend ?berwacht, insbesondere um einen st?rungsfreien Betrieb zu > gew?hrleisten. Auch die vom System ?bertragenen Nachrichten werden im Rahmen > der gesetzlichen Bestimmungen durch BT (Germany) GmbH & Co. oHG ?berwacht und > k?nnen aufgezeichnet oder ausgewertet werden. > This electronic message contains information from BT (Germany) GmbH & Co. oHG > which may be privileged or confidential. The information is intended to be > for the use of the individual(s) or entity named above. If you are not the > intended recipient be aware that any disclosure, copying, distribution or use > of the contents of this information is prohibited. If you have received this > electronic message in error, please notify us by telephone or e-mail (to the > numbers or address above) immediately. Activity and use of the BT (Germany) > GmbH & Co. oHG e-mail system is monitored to secure its effective operation > and for other lawful business purposes. Communications using this system will > also be monitored and may be recorded to secure effective operation and for > other lawful business purposes. > > > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
