Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart
* Ritesh Raj Sarraf [180612 07:26]: > On Thu, 2018-05-31 at 15:29 +0200, Chris Hofstaedtler wrote: > > > For Debian, this could be a good candidate for a Stable update, > > > later after the > > > Stretch release. We should continue to keep this bug open until > > > that is done. > > > > Did you find time to possibly upload this to stable? > > Can I offer any help with this? > > No. I haven't done anything on this bug at all. Please feel free to > prepare/test the patch. I'll include it as is. Cool. I've put the patch into production a few days ago and it looks good. > FYI, I moved the multipath-tools packaging to salsa. You can also join > the team here. > https://salsa.debian.org/linux-blocks-team I've pushed a "stretch" branch, with an UNRELEASED changelog. If that looks good to you, I could also talk to -release about this. Cheers, Chris
Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart
On Thu, 2018-05-31 at 15:29 +0200, Chris Hofstaedtler wrote: > > For Debian, this could be a good candidate for a Stable update, > > later after the > > Stretch release. We should continue to keep this bug open until > > that is done. > > Did you find time to possibly upload this to stable? > Can I offer any help with this? No. I haven't done anything on this bug at all. Please feel free to prepare/test the patch. I'll include it as is. FYI, I moved the multipath-tools packaging to salsa. You can also join the team here. https://salsa.debian.org/linux-blocks-team Thanks, Ritesh -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart
Hey Ritesh, * Ritesh Raj Sarraf [180531 13:27]: > On Fri, 2017-04-07 at 15:45 +0200, Alban Browaeys wrote: > > upstream proposed fix : https://marc.info/?l=dm-devel=149154583809387 > > =2 . > > > > I tested a backport of this patch to release 0.6.4. > > Please find it attached. > > > > There are a few minor logged items left but nothing breaking boot or > > any locking issues anymore. > > This is good news. Thank you for following it up upstream. > Perhaps you'd want to follow it up, till it gets included into the master > repository. > > > For Debian, this could be a good candidate for a Stable update, later after > the > Stretch release. We should continue to keep this bug open until that is done. Did you find time to possibly upload this to stable? Can I offer any help with this? Thanks, Chris
Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart
Dear mantainer, I'm hitting this bug also. Simptoms are exactly the same described by Alban, altough the setup is quite different. The problem arise in a proxmox environment where a PV is configured on a shared iscsi LUN. The iscsi LUN is quite lazy to come online at boot, because the ethernet interface needs about 30 sec to become active. This is due to the spanning tree loop check performed by the cisco switch on the other side. After the boot, multipath -ll return no paths and multipathd goes stuck, as Alban said. I can confirm that recompiling multipath-tools package with Alban's patch solve the problem. So far, no regression for me. I think would be great to include the patch in a stretch update of the package, the bug could be not so hard to hit in many situations, IMHO. Many thanks to Alban for his analisys and for providing the patch. Thank you, -- Ciao, S.
Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart
On Fri, 2017-04-07 at 15:45 +0200, Alban Browaeys wrote: > upstream proposed fix : https://marc.info/?l=dm-devel=149154583809387 > =2 . > > I tested a backport of this patch to release 0.6.4. > Please find it attached. > > There are a few minor logged items left but nothing breaking boot or > any locking issues anymore. This is good news. Thank you for following it up upstream. Perhaps you'd want to follow it up, till it gets included into the master repository. For Debian, this could be a good candidate for a Stable update, later after the Stretch release. We should continue to keep this bug open until that is done. -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart
On Thu, 2017-04-06 at 21:37 +0200, Alban Browaeys wrote: > I pushed the issue upstream as you suggested (this afternoon :) > https://marc.info/?t=14914816591=1=2 > Thank you. > > If you agree, please go ahead and mark this closed. > > This bug is not a local issue even if unlikely to affect most users. > > From https://marc.info/?l=dm-devel=149142931410122=2 multipath > service starts after udevadm triggered, so most devices do not trigger > the udev codepath from multipath. > But bcache0 manages to. > > I have not managed to sort out why as of now. > I have only been able to see that bcache0 is only detected at boot. > If I use a "fixed" multipathd, bcache0 shows in the paths (even though > not functional as a multipath path I believe)... but does not if I > restart multipathd. > It could be that bcache device are not meant to work with multipath ... > if so could bcache devices get blacklisted ? > I wonder if another bug report (upstream) is of need. > Let's see what upstream has to say about your email. The thing with Linux storage is that independent subsystems are stacked upon one another. Such layers need close co-ordination in between the subsystems (SCSI, DM, Block, FS). Since bcache is a separate layer, let's see what others have to share. > > About this bug, would you include a patch for this non critical issue ? As long as the bug is acknowledged and a fix is committed in the upstream repo, it shouldn't be a problem. But given that currently Stretch is in freeze mode, it would highly depend on the timing. OTOH, we can always do a stable update later. > Else this bug could get close. > Next release will be fixed against this issue. Since there's no resolution (or root cause) to the issue yet, let's keep it open. -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart
Hello Alban, Since I haven't been able to reproduce this locally, can we close this ? I feel this issue may also be a side effect, because of the other issue you had reported. If you agree, please go ahead and mark this closed. Thanks, Ritesh On Sat, 2017-04-01 at 14:28 +0530, Ritesh Raj Sarraf wrote: > On Fri, 2017-03-31 at 17:33 +0200, Alban Browaeys wrote: > > Mind this is always the case when udev triggers an add (at boot). > > Well. For all block devices that get discovered, the trigger will happen. > > BTW, if you think there's a bug in the multipath code, it'd be much better to > take it on the upstream mailing list. -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart
On Fri, 2017-03-31 at 17:33 +0200, Alban Browaeys wrote: > Has this boxes been rebooted in between ? Sorry, I missed the 2nd part of your email. To answer your question, Yes. I double checked it. > The broken commit "fix INIT_REQUESTED_UDEV code" was in 0.6.4 http://gi > t.opensvc.com/gitweb.cgi?p=multipath- > tools/.git;a=shortlog;h=refs/tags/0.6.4 upstream but not in 0.6.3 http: > //git.opensvc.com/gitweb.cgi?p=multipath- > tools/.git;a=shortlog;h=refs/tags/0.6.3 . > The deadlock only shows up at boot. Only then udev triggers the add > events for the devices. It might be that this code path is only > triggered by virtual devices > (if without the fix I boot I get a deadlock of multipathd but it I kill > and start multipathd /dev/sda and /dev/sdb are availables -- but > bcache0 virtual device only do so with a fixed multipathd started at > boot. > Likely when I kill and restart multipathd , no add udev events > triggers) The block devices are generated independent of the multipathd daemon. -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart
On Fri, 2017-03-31 at 17:33 +0200, Alban Browaeys wrote: > Mind this is always the case when udev triggers an add (at boot). Well. For all block devices that get discovered, the trigger will happen. BTW, if you think there's a bug in the multipath code, it'd be much better to take it on the upstream mailing list. -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
Bug#859157: multipath-tools: after bootup multipathd timeout on commands - requires daemon restart
Control: tag -1 +moreinfo Hello, I cannot reproduce this locally in my setup. My guess is, from your other bug report, that you may have other components interfering with multipath but I cannot confirm that right now with the limited information available. On Fri, 2017-03-31 at 06:05 +0200, Alban Browaeys wrote: > Package: multipath-tools > Version: 0.6.4-5.1 > Severity: normal > > Dear Maintainer, > multipathd does not respond to commands : list paths or list maps > returns "timed out". > > The mutlipathd daemon when triggered by udev locks up in uev_add_path > per this locks is already helds by its caller uev_update_path. > > Here in thread 5 uev_update_path and uev_add_path chain > (note that in thread 4 checkerloop also waits for this lock). rrs@learner:~$ ssh 172.16.230.133 The authenticity of host '172.16.230.133 (172.16.230.133)' can't be established. ECDSA key fingerprint is SHA256:BrBphJSYS/93xpb/GxHgDPLTlHWGheloG7wlTepHQYk. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '172.16.230.133' (ECDSA) to the list of known hosts. rrs@172.16.230.133's password: Permission denied, please try again. rrs@172.16.230.133's password: The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Sat Nov 19 23:17:46 2016 from 172.16.20.1 rrs@debian-btrfs:~$ su - Password: su: Authentication failure rrs@debian-btrfs:~$ su - Password: root@debian-btrfs:~# multipathd -k multipathd> list paths hcildev dev_t pri dm_st chk_st dev_st next_check 2:0:0:0 sda 8:0 50 active ready running X. 20/40 3:0:0:0 sdb 8:16 50 active ready running .. 19/40 3:0:0:1 sdc 8:32 50 active ready running .. 19/40 2:0:0:1 sdd 8:48 50 active ready running X. 20/40 4:0:0:0 sde 8:64 50 active ready running X. 20/40 5:0:0:0 sdf 8:80 50 active ready running X. 20/40 5:0:0:1 sdg 8:96 50 active ready running X. 20/40 4:0:0:1 sdh 8:112 50 active ready running X. 20/40 multipathd> list maps name sysfs uuid mpatha dm-0 36001405c2d2d9a03751406691395e741 mpathb dm-1 36001405226c2409d98a4e35ba427b274 multipathd> root@debian-btrfs:~# apt policy multipath-tools multipath-tools: Installed: 0.6.4-5 Candidate: 0.6.4-5 Version table: *** 0.6.4-5 500 500 http://httpredir.debian.org/debian unstable/main amd64 Packages 100 /var/lib/dpkg/status 0.6.4-3 500 500 http://httpredir.debian.org/debian testing/main amd64 Packages -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
