[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Launchpad Bug Tracker]

This bug was fixed in the package multipath-tools - 0.4.9-3ubuntu7.15

---
multipath-tools (0.4.9-3ubuntu7.15) trusty; urgency=medium

  * d/p/0047-Add-existing-multipath-devices-to-wwids-file-on.patch:
Fix multipathd which does not update /etc/multipath/wwids file
when reconfigure is invoked. (LP: #1621835)

  [ Dragan Stancevic ]
  * d/p/0048-multipathd-delay-free-pathvec.patch :
Fix SEGV on multipathd shutdown (LP: #1616213)

  [ Nishanth Aravamudan ]
  * debian/patches/fix_use_after_free.patch: Fix use-after-free bugs.
Thanks to Christof Schmitt  and
Benjamin Marzinski .  Closes LP:  #1628723.

 -- Louis Bouchard   Mon, 12 Sep 2016
10:43:19 +0200

** Changed in: multipath-tools (Ubuntu Trusty)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Dragan S.]

User also verified issue as fixed with 0.4.9-3ubuntu7.15

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Dragan S.]

I pulled multipath-tools 0.4.9-3ubuntu7.15 and verified that the fix is
there.

** Tags removed: verification-needed
** Tags added: verification-done

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Brian Murray]

Hello Dragan, or anyone else affected,

Accepted multipath-tools into trusty-proposed. The package will build
now and be available at https://launchpad.net/ubuntu/+source/multipath-
tools/0.4.9-3ubuntu7.15 in a few hours, and then in the -proposed
repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
enable and use -proposed.  Your feedback will aid us getting this update
out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: multipath-tools (Ubuntu Trusty)
   Status: In Progress => Fix Committed

** Tags added: verification-needed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Louis Bouchard]

Marking dev-release as invalid, that code section was refactored and the
code of the patch is in the new code path

** Changed in: multipath-tools (Ubuntu)
   Status: In Progress => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Dragan S.]

PPA with the fix is available at:

https://launchpad.net/~dragan-s/+archive/ubuntu/lp1616213

** Description changed:

+ [Impact]
+ 
+  * During "service multipath-tools stop" multipath daemon
+is trying to cleanup and shut down several concurrent
+threads. At times depending on a race condition between
+two threads, one thread might free resources that are still
+used by another thread.
+ 
+This is causing the multipathd to dump crash core on
+stop events.
+ 
+  * Fix should be backported to trusty to avoid more support
+issues being filed.
+ 
+  * This change delays freeing resources that another thread is
+still using.
+ 
+ [Test Case]
+ 
+  * install multipath-tools, create a basic multipath.conf with
+devices under management. Run: "service multipath-tools start"
+run I/O on devices and keep the system CPU busy, then run
+"service multipath-tools stop".
+ 
+ [Regression Potential]
+ 
+  * There should be no regression potential with this change,
+this problem happens on the exit path and we are only delaying
+a free call.
+ 
+ [Original Description]
+ 
  On ubuntu trusty 14.04.4 in multipath-tools version 0.4.9-3ubuntu7.14
  there is bug in multipathd on shutdown.
  
  The code will access pathvec pointer which is a valid address:
  
  Reading symbols from /sbin/multipathd...Reading symbols from 
/usr/lib/debug//sbin/multipathd...done.
  done.
  [New LWP 41631]
  [New LWP 41584]
  [New LWP 41633]
  [New LWP 41632]
  [New LWP 41582]
  [New LWP 41583]
  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
  Core was generated by `/sbin/multipathd'.
  Program terminated with signal SIGSEGV, Segmentation fault.
- #0  0x004075db in checkerloop (ap=0x1b81040) at main.c:1150
+ #0 0x004075db in checkerloop (ap=0x1b81040) at main.c:1150
  
- 1150  vector_foreach_slot (vecs->pathvec, pp, i) {
+ 1150  vector_foreach_slot (vecs->pathvec, pp, i) {
  (gdb) list
- 1145  pthread_cleanup_push(cleanup_lock, >lock);
- 1146  lock(vecs->lock);
- 1147  condlog(4, "tick");
- 1148  
- 1149  if (vecs->pathvec) {
- 1150  vector_foreach_slot (vecs->pathvec, pp, i) {
- 1151  check_path(vecs, pp);
- 1152  }
- 1153  }
- 1154  if (vecs->mpvec) {
+ 1145  pthread_cleanup_push(cleanup_lock, >lock);
+ 1146  lock(vecs->lock);
+ 1147  condlog(4, "tick");
+ 1148
+ 1149  if (vecs->pathvec) {
+ 1150  vector_foreach_slot (vecs->pathvec, pp, i) {
+ 1151  check_path(vecs, pp);
+ 1152  }
+ 1153  }
+ 1154  if (vecs->mpvec) {
  
  Pathvec is a valid pointer:
  (gdb) p vecs->pathvec
  $1 = (vector) 0x1b81280
  
  But the contents of the structure are just garbage:
  
  (gdb) p *vecs->pathvec
  $2 = {allocated = 1651076143, slot = 0x756e696c2d34365f}
  (gdb)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Mathew Hodson]

** Tags added: patch

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Dragan S.]

** Patch added: "Fix debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+attachment/4752077/+files/fix-lp1616213.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Louis Bouchard]

** Changed in: multipath-tools (Ubuntu Trusty)
   Status: New => Incomplete

** Changed in: multipath-tools (Ubuntu Trusty)
   Status: Incomplete => In Progress

** Changed in: multipath-tools (Ubuntu Trusty)
 Assignee: (unassigned) => Dragan S. (dragan-s)

** Changed in: multipath-tools (Ubuntu)
   Importance: Undecided => Medium

** Changed in: multipath-tools (Ubuntu Trusty)
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Louis Bouchard]

** Also affects: multipath-tools (Ubuntu Trusty)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Dragan S.]

Thread 5 is on the way out:

(gdb) thread 5
[Switching to thread 5 (Thread 0x7f21ca11f840 (LWP 41582))]
#0  0x7f21c9111dfd in nanosleep () at ../sysdeps/unix/syscall-template.S:81
81  ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) up
#1  0x7f21c9111c94 in __sleep (seconds=0) at 
../sysdeps/unix/sysv/linux/sleep.c:137
137 ../sysdeps/unix/sysv/linux/sleep.c: No such file or directory.
(gdb) 
#2  0x004084e6 in child (param=0x0) at main.c:1539
1539sleep (1); /* This is weak. */
(gdb) list
1534free_polls();
1535
1536unlock(vecs->lock);
1537/* Now all the waitevent threads will start rushing in. */
1538while (vecs->lock.depth > 0) {
1539sleep (1); /* This is weak. */
1540condlog(3,"Have %d wait event checkers threads to 
de-alloc, waiting..\n", vecs->lock.depth);
1541}
1542pthread_mutex_destroy(vecs->lock.mutex);
1543FREE(vecs->lock.mutex);

The code in thread 5 before the sleep(1) is:
(gdb) list -
1524
1525pthread_cancel(check_thr);
1526pthread_cancel(uevent_thr);
1527pthread_cancel(uxlsnr_thr);
1528pthread_cancel(uevq_thr);
1529
1530free_keys(keys);
1531keys = NULL;
1532free_handlers(handlers);
1533handlers = NULL;
(gdb) list -
1514/*
1515 * exit path
1516 */
1517block_signal(SIGHUP, NULL);
1518lock(vecs->lock);
1519if (conf->queue_without_daemon == QUE_NO_DAEMON_OFF)
1520vector_foreach_slot(vecs->mpvec, mpp, i)
1521dm_queue_if_no_path(mpp->alias, 0);
1522remove_maps_and_stop_waiters(vecs);
1523free_pathvec(vecs->pathvec, FREE_PATHS);
(gdb) 

So as you can see at line 1523, thread 5 freed vecs->pathvec. Which thread 1 is 
accessing:
1150vector_foreach_slot (vecs->pathvec, pp, i) {

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1616213] Re: Core dump on multipathd shutdown - trusty 14.04.4

[Dragan S.]

Other threads on the way out:

(gdb) info threads
  Id   Target Id Frame 
  6Thread 0x7f21ca11e700 (LWP 41583) pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  5Thread 0x7f21ca11f840 (LWP 41582) 0x7f21c9111dfd in nanosleep () at 
../sysdeps/unix/syscall-template.S:81
  4Thread 0x7f21ca0b3700 (LWP 41632) (Exiting) 0x7f21c9145967 in 
madvise ()
at ../sysdeps/unix/syscall-template.S:81
  3Thread 0x7f21ca0a2700 (LWP 41633) (Exiting) 0x7f21c732a1d1 in ?? () 
from /lib/x86_64-linux-gnu/libgcc_s.so.1
  2Thread 0x7f21ca10d700 (LWP 41584) (Exiting) __lll_lock_wait ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
* 1Thread 0x7f21ca0c4700 (LWP 41631) 0x004075db in checkerloop 
(ap=0x1b81040) at main.c:1150
(gdb)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1616213

Title:
  Core dump on multipathd shutdown - trusty 14.04.4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1616213/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs