[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
This bug was fixed in the package multipath-tools - 0.7.4-2ubuntu3.1 --- multipath-tools (0.7.4-2ubuntu3.1) bionic; urgency=medium * d/p/lp-1891202-*: fix handling of sgio_get_vpd for e.g. long iscsi target names as generated by NetApp E-Series RDAC (LP: #1891202) -- Christian Ehrhardt Mon, 28 Sep 2020 16:14:24 +0200 ** Changed in: multipath-tools (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Hi
@fernando-ferraz asked me to proceed with the tests.
I used this /etc/multipath.conf file:
devices {
device {
vendor "SolidFir"
product "SSD SAN"
detect_checker "yes"
}
}
blacklist {
device {
vendor "SolidFir"
product "SSD SAN"
}
}
Without the correction patch I got the same error.
Then I installed the patch 0.7.4 (proposed enabled).
My steps were:
1. Set multipath config
2. Stop multipath-tools service
3. Create a solidfire volume
4. Attach volume to a existent instance
5. Start multipath-tools service
6. run sudo multipath -v6 -ll
Output:
Jun 24 12:11:56 | loading //lib/multipath/libchecktur.so checker
Jun 24 12:11:56 | loading //lib/multipath/libprioconst.so prioritizer
Jun 24 12:11:56 | Discover device
/sys/devices/pci:00/:00:06.0/virtio3/block/vda
Jun 24 12:11:56 | vda: dev not found in pathvec
Jun 24 12:11:56 | 252:0: dev_t not found in pathvec
Jun 24 12:11:56 | vda: blacklisted, udev property missing
Jun 24 12:11:56 | Discover device
/sys/devices/pci:00/:00:06.0/virtio3/block/vda/vda1
Jun 24 12:11:56 | Discover device
/sys/devices/pci:00/:00:06.0/virtio3/block/vda/vda14
Jun 24 12:11:56 | Discover device
/sys/devices/pci:00/:00:06.0/virtio3/block/vda/vda15
Jun 24 12:11:56 | Discover device
/sys/devices/platform/host2/session3/target2:0:0/2:0:0:0/block/sda
Jun 24 12:11:56 | sda: dev not found in pathvec
Jun 24 12:11:56 | 8:0: dev_t not found in pathvec
Jun 24 12:11:56 | sda: udev property ID_WWN whitelisted
Jun 24 12:11:56 | sda: mask = 0x7
Jun 24 12:11:56 | sda: dev_t = 8:0
Jun 24 12:11:56 | open
'/sys/devices/platform/host2/session3/target2:0:0/2:0:0:0/block/sda/size'
Jun 24 12:11:56 | sda: size = 2097152
Jun 24 12:11:56 | sda: vendor = SolidFir
Jun 24 12:11:56 | sda: product = SSD SAN
Jun 24 12:11:56 | sda: rev = 0001
Jun 24 12:11:56 | sda: h:b:t:l = 2:0:0:0
Jun 24 12:11:56 | sda: tgt_node_name =
iqn.2010-01.com.solidfire:xpsi.uuid-9a54f710-4104-4bd8-a53d-59df79a9ac1e.8441
Jun 24 12:11:56 | open
'/sys/devices/platform/host2/session3/target2:0:0/2:0:0:0/state'
Jun 24 12:11:56 | sda: path state = running
Jun 24 12:11:56 | sda: 1011 cyl, 34 heads, 61 sectors/track, start at 0
Jun 24 12:11:56 | open
'/sys/devices/platform/host2/session3/target2:0:0/2:0:0:0/vpd_pg80'
Jun 24 12:11:56 | sda: serial = 7870736920f9f47acc01
Jun 24 12:11:56 | sda: get_state
Jun 24 12:11:56 | sda: detect_checker = yes (setting: storage device
configuration)
Jun 24 12:11:56 | failed to issue vpd inquiry for pgc9
Jun 24 12:11:56 | sda: path_checker = tur (setting: multipath internal)
Jun 24 12:11:56 | sda: checker timeout = 30 s (setting: multipath internal)
Jun 24 12:11:56 | sda: tur state = up
Jun 24 12:11:56 | Discover device /sys/devices/virtual/block/loop0
Jun 24 12:11:56 | loop0: dev not found in pathvec
Jun 24 12:11:56 | 7:0: dev_t not found in pathvec
Jun 24 12:11:56 | loop0: blacklisted, udev property missing
Jun 24 12:11:56 | Discover device /sys/devices/virtual/block/loop1
Jun 24 12:11:56 | loop1: dev not found in pathvec
Jun 24 12:11:56 | 7:1: dev_t not found in pathvec
Jun 24 12:11:56 | loop1: blacklisted, udev property missing
Jun 24 12:11:56 | Discover device /sys/devices/virtual/block/loop2
Jun 24 12:11:56 | loop2: dev not found in pathvec
Jun 24 12:11:56 | 7:2: dev_t not found in pathvec
Jun 24 12:11:56 | loop2: blacklisted, udev property missing
Jun 24 12:11:56 | Discover device /sys/devices/virtual/block/loop3
Jun 24 12:11:56 | loop3: dev not found in pathvec
Jun 24 12:11:56 | 7:3: dev_t not found in pathvec
Jun 24 12:11:56 | loop3: blacklisted, udev property missing
Jun 24 12:11:56 | Discover device /sys/devices/virtual/block/loop4
Jun 24 12:11:56 | loop4: dev not found in pathvec
Jun 24 12:11:56 | 7:4: dev_t not found in pathvec
Jun 24 12:11:56 | loop4: blacklisted, udev property missing
Jun 24 12:11:56 | Discover device /sys/devices/virtual/block/loop5
Jun 24 12:11:56 | loop5: dev not found in pathvec
Jun 24 12:11:56 | 7:5: dev_t not found in pathvec
Jun 24 12:11:56 | loop5: blacklisted, udev property missing
Jun 24 12:11:56 | Discover device /sys/devices/virtual/block/loop6
Jun 24 12:11:56 | loop6: dev not found in pathvec
Jun 24 12:11:56 | 7:6: dev_t not found in pathvec
Jun 24 12:11:56 | loop6: blacklisted, udev property missing
Jun 24 12:11:56 | Discover device /sys/devices/virtual/block/loop7
Jun 24 12:11:56 | loop7: dev not found in pathvec
Jun 24 12:11:56 | 7:7: dev_t not found in pathvec
Jun 24 12:11:56 | loop7: blacklisted, udev property missing
Jun 24 12:11:56 | Discovered 1/10 paths
= paths list =
uuid hcildev dev_t pri dm_st chk_st vend/prod/revdev_st
2:0:0:0 sda 8:0 -1 undef undef SolidFir,SSD SAN unknown
Jun 24 12:11:56 | libdevmapper version 1.02.145 (2017-11-03)
Jun 24 12:11:56 | DM multipath kernel driver v1.13.0
Jun 24 12:11:56 | tur checker refcount 1
Jun 24 12:11:56 | unloading const prioritizer
Jun 24 12:11:56 | unloading tur checker
This patch seems to fix the issue.
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
** Tags removed: server-next -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
@fernando-ferraz I updated the bug description to try to make it more
general.
For the verification work needed by the SRU process you need to trigger
the bug, update the package to the version in -proposed, and then
demonstrate that the bug was fixed. If you can do this using your
SolidFire device that would be enough, just make sure you can trigger
the issue with this new hardware. TIA for your work!
** Description changed:
[Impact]
- * Users of a NetApp E-Series RDAC due to very long names increasing
-the VDP page and and cause multipathd to hang
+ * Users of some storage devices (for instance NetApp E-Series RDAC) due
+to very long names increasing the VDP page and and cause multipathd to
+hang
- * Introduced as side effect of other changes in 7.4 and fixed later this
-is backporting the HW exploitation to read an enhanced VDP in this case
-which in such cases works fine again.
+ * Introduced as side effect of other changes in 7.4 and fixed later this
+ is backporting the HW exploitation to read an enhanced VDP in this case
+ which in such cases works fine again.
[Test Case]
- * #1 Case: Test on a NetApp E-Series RDAC, the reporter has such HW and
-is able to do so.
+ * #1 Case: Test on affected hardware. The NetApp E-Series RDAC is an
+example, the reporter has such HW and is able to do so.
- * #2 regression tests: since this touches multipath discovery any such
-cases would be interesting tests. From installers using multipath on
-mainframes to multipath-iscsi disks on an raspi. The Range is huge and
-unclear how much exactly should be done. I can myself do said mainframe
-checks and utilize the pkgtests to run through iscsid.
-In addition we should keep this one a bit longer in -proposed IMHO.
+ * #2 regression tests: since this touches multipath discovery any such
+ cases would be interesting tests. From installers using multipath on
+ mainframes to multipath-iscsi disks on an raspi. The Range is huge and
+ unclear how much exactly should be done. I can myself do said mainframe
+ checks and utilize the pkgtests to run through iscsid.
+ In addition we should keep this one a bit longer in -proposed IMHO.
[Regression Potential]
- * All cases asking for vpd 0x83 should be unchanged and only those asking
-for an 0xc9 should get the new improved behavior. But I'm not a 100%
-sure on potential side effects yet. Changes and thereby regressions
-would be limited to the device discovery of multipathd, so that is what
-one needs to take an extra eye on e.g. pre/post list of multipath
-devices (and attributes as those are populated via vpd).
+ * All cases asking for vpd 0x83 should be unchanged and only those asking
+ for an 0xc9 should get the new improved behavior. But I'm not a 100%
+ sure on potential side effects yet. Changes and thereby regressions
+ would be limited to the device discovery of multipathd, so that is what
+ one needs to take an extra eye on e.g. pre/post list of multipath
+ devices (and attributes as those are populated via vpd).
[Other Info]
-
- * n/a
+
+ * n/a
---
I've notice recently that multipathd started hanging when trying to
handle long iscsi target names. In my case these targets are generated
by the OpenStack cinder driver for SolidFire.
My guess is that the problem is occurring due to a bug introduced in
multipath-tools 0.7.4, by commit "limbultipath: prefer RDAC checker with
detect_checker*"*[1].
This change adds a routine to check RDAC compatibility during the
detect_checker procedure, which runs into an infinite loop in the piece
of code that tries to gather SCSI VPD pages larger than 254 bytes (this
is my case due to long iSCSI Target names).
This issue was later addressed by several commits in 0.7.5 and 0.7.6:
"libmultipath: sgio_get_vpd: add page argument" [2], "libmultipath: Fix
sgio_get_vpd()" [3], "libmultipath: fix return code of sgio_get_vpd()"
[4], "libmultipath: get_vpd_sgio: support VPD 0xc9" [5].
I've found a temporary work around for this by using select_checker =
“no” in Ubuntu 18. This appears to avoid the problem by skipping RDAC
compatibility check code.
Also, I've also tested 0.8.3 in a RH environment and found no issues
running default select_checker = “yes” (Ubuntu 20 also delivers 0.8.3).
So, in short:
- Looks like Ubuntu multipath-tools is broken,
- There is a fix in newer versions that aren’t in the ubuntu 18 repo yet.
- The OpenStack Cinder driver for Solidfire just happen to hit this bug
because of the long names
- Setting select_checker = “no” is an possible work-around until Ubuntu
publishes 0.7.6
Bellow "multipath -r -v3" output from my env:
:/$ cat /etc/multipath.conf
devices {
device {
vendor "SolidFir" product "SSD SAN" detect_checker "no"
}
}
:/$ sudo multipath -r -v3 Jul
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Folks would like to suggest two changes to the description: 1) * Users of a NetApp E-Series RDAC due to very long names increasing the VDP page and and cause multipathd to hang The above is not exactly true, in fact any target (can be any storage vendor) with long names may hit if select_checker is enabled. 2) [Test Case] * #1 Case: Test on a NetApp E-Series RDAC, the reporter has such HW and is able to do so. Reporter has only SolidFire storage to test with. Would like to know for you if the test output with a SolidFire device (will hit the long names issue) is actually enough to validate the issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Hi folks, I will test that again using SolidFire hardware and get back soon with the results. We don't a have E-series storage to test but my understanding is that isn't the goal of the fix anyway. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Hello Fernando, or anyone else affected, Accepted multipath-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath- tools/0.7.4-2ubuntu3.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-bionic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: multipath-tools (Ubuntu Bionic) Status: Triaged => Fix Committed ** Tags added: verification-needed verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
While clearing out things for the new year I found the old initial upload rejected (as expected) in [1] but my re-upload seems to have been lost (I can't find it anywhere). So I re-uploaded it to bionic-unapproved again. I double checked - and at least right now it is in [2] as it should be. [1]: https://launchpad.net/ubuntu/bionic/+queue?queue_state=4_text=multipath-tools [2]: https://launchpad.net/ubuntu/bionic/+queue?queue_state=1_text=multipath-tools -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Improved upload tested from PPA [1] as ~ppa2 version. Then I uploaded the improved version to -unapproved for re-review. Again - thanks @Robie for the catch. [1]: https://launchpad.net/~ci-train-ppa- service/+archive/ubuntu/4282/+packages -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
So far: c6140952d 1f401b69e 9d0e6bffb d74a79f35 Unfortunately there are no direct Fixes mentioned, but I've tracked further related changes to sgio_get_vpd and get_vpd_sgio: Related but not needed: 3a309966 libmultipath: "fix sgio_get_vpd looping" fixes a different issue, not related to the SRU, is in 8.4 (>=Groovy) d1571b3f "libmultipath: add code to get vendor specific vpd data": 3e60ab1c "libmultipath: add vend_id to get_vpd_sgio" feature add, not related to the SRU, is in 8.4 (>=Groovy) Related and needed: 2c562423 "libmultipath: decrease log level for failed VPD c9" avoids log burst due to the change we intend to SRU Already in >=Focal 0f5e45b0 "libmultipath: fix length issues in get_vpd_sgio" fix buffer overwrite in the change we SRU Already in >=Focal ... Then the changes we already integrated follow. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
On Wed, Oct 28, 2020 at 7:11 PM Robie Basak <1891...@bugs.launchpad.net> wrote: > > + len = buff_len < 8 ? -ENODATA : > > + (buff_len <= maxlen ? buff_len : maxlen); > > + memcpy (str, buff, len); > > This looked concerning to me. There's a code path that will call memcpy > with a len of -ENODATA which seemed quite dangerous to me. > > Great catch Robie, I'll incorporate that. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
https://git.opensvc.com/gitweb.cgi?p=multipath- tools/.git;a=commitdiff;h=0f5e45b02508ca7ac93b63e07491edaf2e60cbc7 seems relevant. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
> + len = buff_len < 8 ? -ENODATA : > + (buff_len <= maxlen ? buff_len : maxlen); > + memcpy (str, buff, len); This looked concerning to me. There's a code path that will call memcpy with a len of -ENODATA which seemed quite dangerous to me. Upstream seems to have already fixed this here: https://git.opensvc.com/gitweb.cgi?p=multipath- tools/.git;a=blob;f=libmultipath/discovery.c;h=193d7127addbd3ad3feec30d1f67e20dabfb1293;hb=HEAD#l1303 Could you incorporate this please? Are there any other changes made upstream to this code since the cherry-picks which are also relevant? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
> > Problem is not related to users using NetApp E-Series. I've reproduced > this issue in a NetApp SolidFire system that has no relation with > E-Series. My understanding is that any storage system our platform that > make use of target names larger than 254 bytes will hit this problem. > Yeah - thanks for the info, that does not invalidate the SRU. It actually makes it slightly more important as more systems are affected. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Hi Christian, just to make it sure we are in sync: Problem is not related to users using NetApp E-Series. I've reproduced this issue in a NetApp SolidFire system that has no relation with E-Series. My understanding is that any storage system our platform that make use of target names larger than 254 bytes will hit this problem. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Uploaded to Bionic-unapproved to enter SRU processing. @Fernando - when this lands in -proposed we will need your help again to test it on an affected setup. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
** Description changed:
+ [Impact]
+
+ * Users of a NetApp E-Series RDAC due to very long names increasing
+the VDP page and and cause multipathd to hang
+
+ * Introduced as side effect of other changes in 7.4 and fixed later this
+is backporting the HW exploitation to read an enhanced VDP in this case
+which in such cases works fine again.
+
+ [Test Case]
+
+ * #1 Case: Test on a NetApp E-Series RDAC, the reporter has such HW and
+is able to do so.
+
+ * #2 regression tests: since this touches multipath discovery any such
+cases would be interesting tests. From installers using multipath on
+mainframes to multipath-iscsi disks on an raspi. The Range is huge and
+unclear how much exactly should be done. I can myself do said mainframe
+checks and utilize the pkgtests to run through iscsid.
+In addition we should keep this one a bit longer in -proposed IMHO.
+
+ [Regression Potential]
+
+ * All cases asking for vpd 0x83 should be unchanged and only those asking
+for an 0xc9 should get the new improved behavior. But I'm not a 100%
+sure on potential side effects yet. Changes and thereby regressions
+would be limited to the device discovery of multipathd, so that is what
+one needs to take an extra eye on e.g. pre/post list of multipath
+devices (and attributes as those are populated via vpd).
+
+ [Other Info]
+
+ * n/a
+
+ ---
+
I've notice recently that multipathd started hanging when trying to
handle long iscsi target names. In my case these targets are generated
by the OpenStack cinder driver for SolidFire.
My guess is that the problem is occurring due to a bug introduced in
multipath-tools 0.7.4, by commit "limbultipath: prefer RDAC checker with
detect_checker*"*[1].
This change adds a routine to check RDAC compatibility during the
detect_checker procedure, which runs into an infinite loop in the piece
of code that tries to gather SCSI VPD pages larger than 254 bytes (this
is my case due to long iSCSI Target names).
This issue was later addressed by several commits in 0.7.5 and 0.7.6:
"libmultipath: sgio_get_vpd: add page argument" [2], "libmultipath: Fix
sgio_get_vpd()" [3], "libmultipath: fix return code of sgio_get_vpd()"
[4], "libmultipath: get_vpd_sgio: support VPD 0xc9" [5].
I've found a temporary work around for this by using select_checker =
“no” in Ubuntu 18. This appears to avoid the problem by skipping RDAC
compatibility check code.
Also, I've also tested 0.8.3 in a RH environment and found no issues
running default select_checker = “yes” (Ubuntu 20 also delivers 0.8.3).
-
So, in short:
- - Looks like Ubuntu multipath-tools is broken,
- - There is a fix in newer versions that aren’t in the ubuntu 18 repo yet.
+ - Looks like Ubuntu multipath-tools is broken,
+ - There is a fix in newer versions that aren’t in the ubuntu 18 repo yet.
- The OpenStack Cinder driver for Solidfire just happen to hit this bug
because of the long names
- Setting select_checker = “no” is an possible work-around until Ubuntu
publishes 0.7.6
Bellow "multipath -r -v3" output from my env:
:/$ cat /etc/multipath.conf
- devices {
- device {
- vendor "SolidFir" product "SSD SAN" detect_checker "no"
- }
+ devices {
+ device {
+ vendor "SolidFir" product "SSD SAN" detect_checker "no"
+ }
}
:/$ sudo multipath -r -v3 Jul 09 19:06:24 | loading
//lib/multipath/libchecktur.so checker Jul 09 19:06:24 | loading
//lib/multipath/libprioconst.so prioritizer Jul 09 19:06:24 | vda:
blacklisted, udev property missing Jul 09 19:06:24 | sda: udev property
ID_WWN whitelisted Jul 09 19:06:24 | sda: mask = 0x1f Jul 09 19:06:24 |
sda: dev_t = 8:0 Jul 09 19:06:24 | sda: size = 10485760 Jul 09 19:06:24
| sda: vendor = SolidFir Jul 09 19:06:24 | sda: product = SSD SAN Jul 09
19:06:24 | sda: rev = 0001 Jul 09 19:06:24 | sda: h:b:t:l = 2:0:0:0 Jul
09 19:06:24 | sda: tgt_node_name = iqn.2010-01.com.solidfire:xpsi.uuid-
7b8a927e-4dbc-4a2b-b47a-1bde324895eb.53 Jul 09 19:06:24 | sda: path
state = running Jul 09 19:06:24 | sda: 1018 cyl, 166 heads, 62
sectors/track, start at 0 Jul 09 19:06:24 | sda: serial =
787073690035f47acc01 Jul 09 19:06:24 | sda: get_state Jul 09
19:06:24 | sda: detect_checker = no (setting: storage device
configuration) Jul 09 19:06:24 | sda: path_checker = tur (setting:
multipath internal) Jul 09 19:06:24 | sda: checker timeout = 30 s
(setting: multipath internal) Jul 09 19:06:24 | sda: tur state = up Jul
09 19:06:24 | sda: uid_attribute = ID_SERIAL (setting: multipath
internal) Jul 09 19:06:24 | sda: uid = 36f47acc1787073690035
(udev) Jul 09 19:06:24 | sda: detect_prio = yes (setting: multipath
internal) Jul 09 19:06:24 | sda: prio = const (setting: multipath
internal) Jul 09 19:06:24 | sda: prio args = "" (setting: multipath
internal) Jul 09 19:06:24 | sda: const prio = 1 Jul 09
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
** Merge proposal linked: https://code.launchpad.net/~paelzer/ubuntu/+source/multipath-tools/+git/multipath-tools/+merge/392693 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Hi Christian, sorry for the late reply. After applying the PPA, multipath no longer hangs when attaching a SolidFire volumes. The problem with long iscsi target names seems to be fixed now. See the output bellow: stack@60-sfernand-development:~$ sudo multipath -v6 -ll Oct 13 20:22:27 | loading //lib/multipath/libchecktur.so checker Oct 13 20:22:27 | loading //lib/multipath/libprioconst.so prioritizer Oct 13 20:22:27 | Discover device /sys/devices/pci:00/:00:06.0/virtio3/block/vda Oct 13 20:22:27 | vda: dev not found in pathvec Oct 13 20:22:27 | 252:0: dev_t not found in pathvec Oct 13 20:22:27 | vda: blacklisted, udev property missing Oct 13 20:22:27 | Discover device /sys/devices/pci:00/:00:06.0/virtio3/block/vda/vda1 Oct 13 20:22:27 | Discover device /sys/devices/pci:00/:00:06.0/virtio3/block/vda/vda14 Oct 13 20:22:27 | Discover device /sys/devices/pci:00/:00:06.0/virtio3/block/vda/vda15 Oct 13 20:22:27 | Discover device /sys/devices/platform/host4/session228/target4:0:0/4:0:0:0/block/sdc Oct 13 20:22:27 | sdc: dev not found in pathvec Oct 13 20:22:27 | 8:32: dev_t not found in pathvec Oct 13 20:22:27 | sdc: udev property ID_WWN whitelisted Oct 13 20:22:27 | sdc: mask = 0x7 Oct 13 20:22:27 | sdc: dev_t = 8:32 Oct 13 20:22:27 | open '/sys/devices/platform/host4/session228/target4:0:0/4:0:0:0/block/sdc/size' Oct 13 20:22:27 | sdc: size = 10485760 Oct 13 20:22:27 | sdc: vendor = SolidFir Oct 13 20:22:27 | sdc: product = SSD SAN Oct 13 20:22:27 | sdc: rev = 0001 Oct 13 20:22:27 | sdc: h:b:t:l = 4:0:0:0 Oct 13 20:22:27 | sdc: tgt_node_name = iqn.2010-01.com.solidfire:xpsi.uuid-e100a0c6-f8d6-4927-b6ea-213248083965.3409 Oct 13 20:22:27 | open '/sys/devices/platform/host4/session228/target4:0:0/4:0:0:0/state' Oct 13 20:22:27 | sdc: path state = running Oct 13 20:22:27 | sdc: 1018 cyl, 166 heads, 62 sectors/track, start at 0 Oct 13 20:22:27 | open '/sys/devices/platform/host4/session228/target4:0:0/4:0:0:0/vpd_pg80' Oct 13 20:22:27 | sdc: serial = 787073690d51f47acc01 Oct 13 20:22:27 | sdc: get_state Oct 13 20:22:27 | sdc: detect_checker = yes (setting: storage device configuration) Oct 13 20:22:27 | failed to issue vpd inquiry for pgc9 Oct 13 20:22:27 | sdc: path_checker = tur (setting: multipath internal) Oct 13 20:22:27 | sdc: checker timeout = 30 s (setting: multipath internal) Oct 13 20:22:27 | sdc: tur state = up Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop0 Oct 13 20:22:27 | loop0: dev not found in pathvec Oct 13 20:22:27 | 7:0: dev_t not found in pathvec Oct 13 20:22:27 | loop0: blacklisted, udev property missing Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop1 Oct 13 20:22:27 | loop1: dev not found in pathvec Oct 13 20:22:27 | 7:1: dev_t not found in pathvec Oct 13 20:22:27 | loop1: blacklisted, udev property missing Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop2 Oct 13 20:22:27 | loop2: dev not found in pathvec Oct 13 20:22:27 | 7:2: dev_t not found in pathvec Oct 13 20:22:27 | loop2: blacklisted, udev property missing Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop3 Oct 13 20:22:27 | loop3: dev not found in pathvec Oct 13 20:22:27 | 7:3: dev_t not found in pathvec Oct 13 20:22:27 | loop3: blacklisted, udev property missing Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop4 Oct 13 20:22:27 | loop4: dev not found in pathvec Oct 13 20:22:27 | 7:4: dev_t not found in pathvec Oct 13 20:22:27 | loop4: blacklisted, udev property missing Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop5 Oct 13 20:22:27 | loop5: dev not found in pathvec Oct 13 20:22:27 | 7:5: dev_t not found in pathvec Oct 13 20:22:27 | loop5: blacklisted, udev property missing Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop6 Oct 13 20:22:27 | loop6: dev not found in pathvec Oct 13 20:22:27 | 7:6: dev_t not found in pathvec Oct 13 20:22:27 | loop6: blacklisted, udev property missing Oct 13 20:22:27 | Discover device /sys/devices/virtual/block/loop7 Oct 13 20:22:27 | loop7: dev not found in pathvec Oct 13 20:22:27 | 7:7: dev_t not found in pathvec Oct 13 20:22:27 | loop7: blacklisted, udev property missing Oct 13 20:22:27 | Discovered 1/10 paths = paths list = uuid hcildev dev_t pri dm_st chk_st vend/prod/revdev_st 4:0:0:0 sdc 8:32 -1 undef undef SolidFir,SSD SAN unknown Oct 13 20:22:27 | libdevmapper version 1.02.145 (2017-11-03) Oct 13 20:22:27 | DM multipath kernel driver v1.13.0 Oct 13 20:22:27 | tur checker refcount 1 Oct 13 20:22:27 | unloading const prioritizer Oct 13 20:22:27 | unloading tur checker -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to:
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
@Fernando - I beg your pardon, but do not want to get this be lost. Are there any further updates on the PPA-test? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Hi Christian, I've tested the PPA and seems to work just fine. I will post the results here by the and of this week. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
7.6 1f401b69e libmultipath: Fix sgio_get_vpd() 7.5 9d0e6bffb libmultipath: get_vpd_sgio: support VPD 0xc9 7.5 d74a79f35 libmultipath: sgio_get_vpd: add page argument 7.5 c6140952d libmultipath: fix return code of sgio_get_vpd() All applied as-is without any noise and they also look small and reasonable. I already started to build it in: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4282/ @Fernando: If you could give those - once built - a try and let me know if/how they work that would be awesome. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Hi Christian, sounds good to me. I can test it for sure. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Hi Fernando, Bionic will stay at 0.7.4 for SRU Policy reasons [1] but we should check if we can backport just the fixes in a way that works for your case without regressing others. Thanks for your near instant reply BTW, If I get this backported would you be able to test a PPA to check if the fix would work for you? [1]: https://wiki.ubuntu.com/StableReleaseUpdates -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Hi Christian, you are correct. I Can't reproduce this issue in Focal (0.8.3). Could multipath-tools be upgraded to 0.8.3 for Bionic aswell? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Assuming the referred patches match they are all in >=7.5 which means >Bionic is fixed. ** Changed in: multipath-tools (Ubuntu Focal) Status: Triaged => Fix Released ** Changed in: multipath-tools (Ubuntu) Status: Triaged => Fix Released ** Changed in: multipath-tools (Ubuntu Eoan) Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
** Changed in: multipath-tools (Ubuntu) Assignee: Rafael David Tinoco (rafaeldtinoco) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1891202] Re: Multipathd hangs with long iscsi target names in Ubuntu 18.04
Thanks a lot for reporting this Fernando. Let me take a deeper look in what you provided and will get back to you soon. ** Also affects: multipath-tools (Ubuntu Eoan) Importance: Undecided Status: New ** Also affects: multipath-tools (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: multipath-tools (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: multipath-tools (Ubuntu Focal) Status: New => Triaged ** Changed in: multipath-tools (Ubuntu Eoan) Status: New => Triaged ** Changed in: multipath-tools (Ubuntu Bionic) Status: New => Triaged ** Changed in: multipath-tools (Ubuntu) Status: New => Triaged ** Changed in: multipath-tools (Ubuntu) Assignee: (unassigned) => Rafael David Tinoco (rafaeldtinoco) ** Tags added: server-next -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891202 Title: Multipathd hangs with long iscsi target names in Ubuntu 18.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1891202/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
