I'm also seeing this error using a Dell MD3800i array. The multipath errors shown in our logs are different however.
Feb 1 15:11:58 ovirt-node-production2 kernel: dd: sending ioctl 80306d02 to a partition! Feb 1 15:21:01 ovirt-node-production2 multipathd: dm-31: remove map (uevent) Feb 1 15:21:01 ovirt-node-production2 multipathd: dm-31: devmap not registered, can't remove Feb 1 15:21:01 ovirt-node-production2 multipathd: dm-31: remove map (uevent) The dd error seems to happen every time that SPM runs a test. On 01/31/2017 09:23 AM, Nathanaël Blanchet wrote: > > exactly the same issue by there with FC EMC domain storage... > > > Le 31/01/2017 à 15:20, Gianluca Cecchi a écrit : >> Hello, >> my test environment is composed by 2 old HP blades BL685c G1 >> (ovmsrv05 and ovmsrv06) and they are connected in a SAN with >> FC-switches to an old IBM DS4700 storage array. >> Apart from being old, they seem all ok from an hw point of view. >> I have configured oVirt 4.0.6 and an FCP storage domain. >> The hosts are plain CentOS 7.3 servers fully updated. >> It is not an hosted engine environment: the manager is a vm outside >> of the cluster. >> I have configured power mgmt on both and it works good. >> >> I have at the moment only one VM for test and it is doing quite nothing. >> >> Starting point: ovmsrv05 is in maintenance (since about 2 days) and >> the VM is running on ovmsrv06. >> I update qemu-kvm package on ovmsrv05 and then I restart it from web >> admin gui: >> Power Mgmt --> Restart >> >> Sequence of events in pane and the problem in subject: >> Jan 31, 2017 10:29:43 AM Host ovmsrv05 power management was verified >> successfully. >> Jan 31, 2017 10:29:43 AM Status of host ovmsrv05 was set to Up. >> Jan 31, 2017 10:29:38 AM Executing power management status on Host >> ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212. >> Jan 31, 2017 10:29:29 AM Activation of host ovmsrv05 initiated by >> admin@internal-authz. >> Jan 31, 2017 10:28:05 AM VM ol65 has recovered from paused back to up. >> Jan 31, 2017 10:27:55 AM VM ol65 has been paused due to storage I/O >> problem. >> Jan 31, 2017 10:27:55 AM VM ol65 has been paused. >> Jan 31, 2017 10:25:52 AM Host ovmsrv05 was restarted by >> admin@internal-authz. >> Jan 31, 2017 10:25:52 AM Host ovmsrv05 was started by >> admin@internal-authz. >> Jan 31, 2017 10:25:52 AM Power management start of Host ovmsrv05 >> succeeded. >> Jan 31, 2017 10:25:50 AM Executing power management status on Host >> ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212. >> Jan 31, 2017 10:25:37 AM Executing power management start on Host >> ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212. >> Jan 31, 2017 10:25:37 AM Power management start of Host ovmsrv05 >> initiated. >> Jan 31, 2017 10:25:37 AM Auto fence for host ovmsrv05 was started. >> Jan 31, 2017 10:25:37 AM All VMs' status on Non Responsive Host >> ovmsrv05 were changed to 'Down' by admin@internal-authz >> Jan 31, 2017 10:25:36 AM Host ovmsrv05 was stopped by >> admin@internal-authz. >> Jan 31, 2017 10:25:36 AM Power management stop of Host ovmsrv05 >> succeeded. >> Jan 31, 2017 10:25:34 AM Executing power management status on Host >> ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212. >> Jan 31, 2017 10:25:15 AM Executing power management stop on Host >> ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212. >> Jan 31, 2017 10:25:15 AM Power management stop of Host ovmsrv05 >> initiated. >> Jan 31, 2017 10:25:12 AM Executing power management status on Host >> ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212. >> >> Watching the timestamps, the culprit seems the reboot time of >> ovmsrv05 that detects some LUNs in owned state and other ones in unowned >> Full messages of both hosts here: >> https://drive.google.com/file/d/0BwoPbcrMv8mvekZQT1pjc0NMRlU/view?usp=sharing >> and >> https://drive.google.com/file/d/0BwoPbcrMv8mvcjBCYVdFZWdXTms/view?usp=sharing >> >> At this time there are 4 LUNs globally seen by the two hosts but only >> 1 of them is currently configured as the only storage domain in oVirt >> cluster. >> >> [root@ovmsrv05 ~]# multipath -l | grep ^36 >> 3600a0b8000299aa80000d08b55014119 dm-5 IBM ,1814 FAStT >> 3600a0b80002999020000cd3c5501458f dm-3 IBM ,1814 FAStT >> 3600a0b80002999020000ccf855011198 dm-2 IBM ,1814 FAStT >> 3600a0b8000299aa80000d08955014098 dm-4 IBM ,1814 FAStT >> >> the configured one: >> [root@ovmsrv05 ~]# multipath -l 3600a0b8000299aa80000d08b55014119 >> 3600a0b8000299aa80000d08b55014119 dm-5 IBM ,1814 FAStT >> size=4.0T features='0' hwhandler='1 rdac' wp=rw >> |-+- policy='service-time 0' prio=0 status=active >> | |- 0:0:1:3 sdl 8:176 active undef running >> | `- 2:0:1:3 sdp 8:240 active undef running >> `-+- policy='service-time 0' prio=0 status=enabled >> |- 0:0:0:3 sdd 8:48 active undef running >> `- 2:0:0:3 sdi 8:128 active undef running >> >> In mesages of booting node, arounf the problem registered by the storage: >> [root@ovmsrv05 ~]# grep owned /var/log/messages >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:0:1: rdac: LUN 1 (RDAC) (owned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:0:2: rdac: LUN 2 (RDAC) (owned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:0:3: rdac: LUN 3 (RDAC) >> (unowned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:0:1: rdac: LUN 1 (RDAC) (owned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:0:4: rdac: LUN 4 (RDAC) >> (unowned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:0:2: rdac: LUN 2 (RDAC) (owned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:1:1: rdac: LUN 1 (RDAC) >> (unowned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:0:3: rdac: LUN 3 (RDAC) >> (unowned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:0:4: rdac: LUN 4 (RDAC) >> (unowned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:1:2: rdac: LUN 2 (RDAC) >> (unowned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:1:1: rdac: LUN 1 (RDAC) >> (unowned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:1:3: rdac: LUN 3 (RDAC) (owned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:1:2: rdac: LUN 2 (RDAC) >> (unowned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:1:4: rdac: LUN 4 (RDAC) (owned) >> Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:1:3: rdac: LUN 3 (RDAC) (owned) >> Jan 31 10:27:39 ovmsrv05 kernel: scsi 2:0:1:4: rdac: LUN 4 (RDAC) (owned) >> >> I don't know exactly the meaning of owned/unowned in the output above.. >> Possibly it detects the 0:0:1:3 and 2:0:1:3 paths (those of the >> active group) as "owned" and this could have created problems with >> the active node? >> >> On active node strangely I don't loose all the paths, but the VM has >> been paused anyway >> >> [root@ovmsrv06 log]# grep "remaining active path" /var/log/messages >> Jan 31 10:27:48 ovmsrv06 multipathd: >> 3600a0b8000299aa80000d08b55014119: remaining active paths: 3 >> Jan 31 10:27:49 ovmsrv06 multipathd: >> 3600a0b8000299aa80000d08b55014119: remaining active paths: 2 >> Jan 31 10:27:56 ovmsrv06 multipathd: >> 3600a0b8000299aa80000d08b55014119: remaining active paths: 3 >> Jan 31 10:27:56 ovmsrv06 multipathd: >> 3600a0b8000299aa80000d08b55014119: remaining active paths: 2 >> Jan 31 10:27:56 ovmsrv06 multipathd: >> 3600a0b8000299aa80000d08b55014119: remaining active paths: 1 >> Jan 31 10:27:57 ovmsrv06 multipathd: >> 3600a0b8000299aa80000d08b55014119: remaining active paths: 2 >> Jan 31 10:28:01 ovmsrv06 multipathd: >> 3600a0b8000299aa80000d08b55014119: remaining active paths: 3 >> Jan 31 10:28:01 ovmsrv06 multipathd: >> 3600a0b8000299aa80000d08b55014119: remaining active paths: 4 >> >> I'm not an expert of this storage array in particular, and of the >> rdac hardware handler in general. >> >> What I see is that multipath.conf on both nodes: >> >> # VDSM REVISION 1.3 >> >> defaults { >> polling_interval 5 >> no_path_retry fail >> user_friendly_names no >> flush_on_last_del yes >> fast_io_fail_tmo 5 >> dev_loss_tmo 30 >> max_fds 4096 >> } >> >> >> devices { >> device { >> # These settings overrides built-in devices settings. It does >> not apply >> # to devices without built-in settings (these use the >> settings in the >> # "defaults" section), or to devices defined in the "devices" >> section. >> # Note: This is not available yet on Fedora 21. For more info see >> # https://bugzilla.redhat.com/1253799 >> all_devs yes >> no_path_retry fail >> } >> } >> >> >> beginning of /proc/scsi/scsi >> >> [root@ovmsrv06 ~]# cat /proc/scsi/scsi >> Attached devices: >> Host: scsi1 Channel: 01 Id: 00 Lun: 00 >> Vendor: HP Model: LOGICAL VOLUME Rev: 1.86 >> Type: Direct-Access ANSI SCSI revision: 05 >> Host: scsi0 Channel: 00 Id: 00 Lun: 01 >> Vendor: IBM Model: 1814 FAStT Rev: 0916 >> Type: Direct-Access ANSI SCSI revision: 05 >> ... >> >> To get default acquired config for this storage: >> >> multpathd -k >> > show config >> >> I can see: >> >> device { >> vendor "IBM" >> product "^1814" >> product_blacklist "Universal Xport" >> path_grouping_policy "group_by_prio" >> path_checker "rdac" >> features "0" >> hardware_handler "1 rdac" >> prio "rdac" >> failback immediate >> rr_weight "uniform" >> no_path_retry "fail" >> } >> >> >> and >> >> defaults { >> verbosity 2 >> polling_interval 5 >> max_polling_interval 20 >> reassign_maps "yes" >> multipath_dir "/lib64/multipath" >> path_selector "service-time 0" >> path_grouping_policy "failover" >> uid_attribute "ID_SERIAL" >> prio "const" >> prio_args "" >> features "0" >> path_checker "directio" >> alias_prefix "mpath" >> failback "manual" >> rr_min_io 1000 >> rr_min_io_rq 1 >> max_fds 4096 >> rr_weight "uniform" >> no_path_retry "fail" >> queue_without_daemon "no" >> flush_on_last_del "yes" >> user_friendly_names "no" >> fast_io_fail_tmo 5 >> dev_loss_tmo 30 >> bindings_file "/etc/multipath/bindings" >> wwids_file /etc/multipath/wwids >> log_checker_err always >> find_multipaths no >> retain_attached_hw_handler no >> detect_prio no >> hw_str_match no >> force_sync no >> deferred_remove no >> ignore_new_boot_devs no >> skip_kpartx no >> config_dir "/etc/multipath/conf.d" >> delay_watch_checks no >> delay_wait_checks no >> retrigger_tries 3 >> retrigger_delay 10 >> missing_uev_wait_timeout 30 >> new_bindings_in_boot no >> } >> >> Any hint on how to tune multipath.conf so that a powering on server >> doesn't create problems to running VMs? >> >> Thanks in advance, >> Gianluca >> >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > > -- > Nathanaël Blanchet > > Supervision réseau > Pôle Infrastrutures Informatiques > 227 avenue Professeur-Jean-Louis-Viala > 34193 MONTPELLIER CEDEX 5 > Tél. 33 (0)4 67 54 84 55 > Fax 33 (0)4 67 54 84 14 > blanc...@abes.fr > > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
