Hello Axel,
Am 21.10.2018 um 23:10 schrieb Axel Beckert:
> The system is about 2.5 years old.
...
> (I'd say this counts as a yes.)
Ok, will not do wild guesses next time ;-)
I think I were able to reproduce the issue in a buster amd64 qemu-VM,
by forwarding a real usb card reader with inserted SD-card into the VM.
There seems to be a problem with this forward and the virtual USB gets
reset - but is enough to get multipath called and crashing.
>From your backtrace already visible is that the first parameter to
find_multipaths_check_timeout is pp=0x0, that gets later
dereferenced without checking.
This looks like it got fixed upstream in patch [1].
At least a version 0.7.7-3 built with this patch does not crash like before.
Kind regards,
Bernhard
[1]
https://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=commit;h=ccfb9a38f5cb01a7af0f4e30d18e34d4fabc8b53
# backtrace from Alex
Core was generated by `/sbin/multipath -u sdi'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:62
62 ../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory.
(gdb) bt
#0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:62
#1 0x00007f294234e46c in _IO_vfprintf_internal (s=s@entry=0x7fff09131e20,
format=format@entry=0x561e8fda37bd "%s/%s", ap=ap@entry=0x7fff09131fa0) at
vfprintf.c:1643
#2 0x00007f2942404ae9 in ___vsnprintf_chk (s=0x7fff09132270
"/dev/shm/multipath/find_multipaths/", maxlen=<optimized out>, flags=1,
slen=<optimized out>, format=0x561e8fda37bd "%s/%s",
args=args@entry=0x7fff09131fa0) at vsnprintf_chk.c:63
#3 0x00007f2942404a15 in ___snprintf_chk (s=<optimized out>, maxlen=<optimized
out>, flags=<optimized out>, slen=<optimized out>, format=<optimized out>) at
snprintf_chk.c:34
#4 0x0000561e8fda2291 in snprintf (__fmt=0x561e8fda37bd "%s/%s", __n=4096,
__s=0x7fff09132270 "/dev/shm/multipath/find_multipaths/") at
/usr/include/x86_64-linux-gnu/bits/stdio2.h:64
#5 find_multipaths_check_timeout (pp=0x0, tmo=0, until=0x7fff091332f0) at
main.c:396
#6 0x0000561e8fda18a2 in print_cmd_valid (conf=0x561e90fdf970,
pathvec=0x561e9103d630, k=1) at main.c:487
#7 configure (devpath=0x561e9103c000 "sdi", dev_type=DEV_UEVENT,
cmd=CMD_VALID_PATH, conf=<optimized out>) at main.c:743
#8 main (argc=<optimized out>, argv=<optimized out>) at main.c:1130
##################
apt update
apt install devscripts dpkg-dev systemd-coredump gdb multipath-tools
multipath-tools-dbgsym
apt build-dep multipath-tools
mkdir multipath-tools/orig -p
cd multipath-tools/orig
apt source multipath-tools
cd ../..
(host) lsusb
Bus 001 Device 004: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer
(host) chmod 777 /dev/bus/usb/001/004
(qemu) device_add usb-host,vendorid=0x058f,productid=0x6362,id=cardreader
# the cardreader via usb seems not completely compatible, therefore produces
the "needed" resets and "capacity change ... to 0".
[Mo Okt 22 19:22:46 2018] usb 1-2.1: reset full-speed USB device number 4 using
uhci_hcd
...
[Mo Okt 22 19:22:48 2018] sdb: detected capacity change from 1030225920 to 0
[Mo Okt 22 19:22:48 2018] multipath[4740]: segfault at 100 ip 00007fc684bba136
sp 00007ffe36f352a8 error 4 in libc-2.27.so[7fc684b45000+146000]
[Mo Okt 22 19:22:48 2018] Code: 0f 1f 40 00 66 0f ef c0 66 0f ef c9 66 0f ef d2
66 0f ef db 48 89 f8 48 89 f9 48 81 e1 ff 0f 00 00 48 81 f9 cf 0f 00 00 77 6a
<f3> 0f 6f 20 66 0f 74 e0 66 0f d7 d4 85 d2 74 04 0f bc c2 c3 48 83
root@debian:~# coredumpctl gdb 4740
PID: 4740 (multipath)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Mon 2018-10-22 19:22:48 CEST (2min 0s ago)
Command Line: /sbin/multipath -u sdb
Executable: /sbin/multipath
Control Group: /system.slice/systemd-udevd.service
Unit: systemd-udevd.service
Slice: system.slice
Boot ID: b39ee631364643aaa54c9eb198153939
Machine ID: 32f43b50ac8c4b21941bc0b02f8e7811
Hostname: debian
Storage:
/var/lib/systemd/coredump/core.multipath.0.b39ee631364643aaa54c9eb198153939.4740.1540228968000000.lz4
Message: Process 4740 (multipath) of user 0 dumped core.
Stack trace of thread 4740:
#0 0x00007fc684bba136 n/a (libc.so.6)
#1 0x00007fc684b7346c _IO_vfprintf (libc.so.6)
#2 0x00007fc684c29ae9 __vsnprintf_chk (libc.so.6)
#3 0x00007fc684c29a15 __snprintf_chk (libc.so.6)
#4 0x0000556212df2316 n/a (multipath)
#5 0x0000556212df15bc n/a (multipath)
#6 0x00007fc684b45b17 __libc_start_main (libc.so.6)
#7 0x0000556212df18fa n/a (multipath)
GNU gdb (Debian 8.1-4+b1) 8.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /sbin/multipath...(no debugging symbols found)...done.
[New LWP 4740]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/sbin/multipath -u sdb'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
120 ../sysdeps/x86_64/multiarch/../strlen.S: Datei oder Verzeichnis nicht
gefunden.
(gdb) set height 0
(gdb) set width 0
(gdb) set pagination off
(gdb) bt
#0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
#1 0x00007fc684b7346c in _IO_vfprintf_internal (s=s@entry=0x7ffe36f35830,
format=format@entry=0x556212df3b12 "%s/%s", ap=ap@entry=0x7ffe36f359b0) at
vfprintf.c:1643
#2 0x00007fc684c29ae9 in ___vsnprintf_chk (s=0x7ffe36f35c80
"/dev/shm/multipath/find_multipaths/", maxlen=<optimized out>, flags=1,
slen=<optimized out>, format=0x556212df3b12 "%s/%s",
args=args@entry=0x7ffe36f359b0) at vsnprintf_chk.c:63
#3 0x00007fc684c29a15 in ___snprintf_chk (s=<optimized out>, maxlen=<optimized
out>, flags=<optimized out>, slen=<optimized out>, format=<optimized out>) at
snprintf_chk.c:34
#4 0x0000556212df2316 in ?? ()
#5 0x0000556212df15bc in ?? ()
#6 0x00007fc684b45b17 in __libc_start_main (main=0x556212df0700, argc=3,
argv=0x7ffe36f36eb8, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7ffe36f36ea8) at ../csu/libc-start.c:310
#7 0x0000556212df18fa in ?? ()
set height 0
set width 0
set pagination off
directory /home/benutzer/multipath-tools/orig/multipath-tools-0.7.7/multipath
display/i $pc
(gdb) bt
#0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
#1 0x00007fc684b7346c in _IO_vfprintf_internal (s=s@entry=0x7ffe36f35830,
format=format@entry=0x556212df3b12 "%s/%s", ap=ap@entry=0x7ffe36f359b0) at
vfprintf.c:1643
#2 0x00007fc684c29ae9 in ___vsnprintf_chk (s=0x7ffe36f35c80
"/dev/shm/multipath/find_multipaths/", maxlen=<optimized out>, flags=1,
slen=<optimized out>, format=0x556212df3b12 "%s/%s",
args=args@entry=0x7ffe36f359b0) at vsnprintf_chk.c:63
#3 0x00007fc684c29a15 in ___snprintf_chk (s=s@entry=0x7ffe36f35c80
"/dev/shm/multipath/find_multipaths/", maxlen=maxlen@entry=4096,
flags=flags@entry=1, slen=slen@entry=4096, format=format@entry=0x556212df3b12
"%s/%s") at snprintf_chk.c:34
#4 0x0000556212df2316 in snprintf (__fmt=0x556212df3b12 "%s/%s", __n=4096,
__s=0x7ffe36f35c80 "/dev/shm/multipath/find_multipaths/") at
/usr/include/x86_64-linux-gnu/bits/stdio2.h:64
#5 find_multipaths_check_timeout (pp=<optimized out>, tmo=tmo@entry=0,
until=<optimized out>) at main.c:396
#6 0x0000556212df15bc in print_cmd_valid (conf=0x556214624970,
pathvec=0x556214682630, k=1) at main.c:487
#7 configure (devpath=0x556214681000 "sdb", dev_type=DEV_UEVENT,
cmd=CMD_VALID_PATH, conf=0x556214624970) at main.c:743
#8 main () at main.c:1130
#9 0x00007fc684b45b17 in __libc_start_main (main=0x556212df0700 <main>,
argc=3, argv=0x7ffe36f36eb8, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7ffe36f36ea8) at ../csu/libc-start.c:310
#10 0x0000556212df18fa in _start () at main.c:481
(gdb) display/i $pc
1: x/i $pc
=> 0x7fc684bba136 <__strlen_sse2+38>: movdqu (%rax),%xmm4
(gdb) print/x $rax
$1 = 0x100
(gdb) up
#4 0x0000556212df2316 in snprintf (__fmt=0x556212df3b12 "%s/%s", __n=4096,
__s=0x7ffe36f35c80 "/dev/shm/multipath/find_multipaths/") at
/usr/include/x86_64-linux-gnu/bits/stdio2.h:64
64 return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
(gdb) up
#5 find_multipaths_check_timeout (pp=<optimized out>, tmo=tmo@entry=0,
until=<optimized out>) at main.c:396
396 if (snprintf(path, sizeof(path), "%s/%s", shm_find_mp_dir,
pp->dev_t)
(gdb) list main.c:371,400
371 /**
372 * find_multipaths_check_timeout(wwid, tmo)
373 * Helper for "find_multipaths smart"
374 *
375 * @param[in] pp: path to check / record
376 * @param[in] tmo: configured timeout for this WWID, or value <= 0 for
checking
377 * @param[out] until: timestamp until we must wait, CLOCK_REALTIME, if
return
378 * value is FIND_MULTIPATHS_WAITING
379 * @returns: FIND_MULTIPATHS_WAIT_DONE, if waiting has finished
380 * @returns: FIND_MULTIPATHS_ERROR, if internal error occurred
381 * @returns: FIND_MULTIPATHS_NEVER, if tmo is 0 and we didn't wait for
this
382 * device
383 * @returns: FIND_MULTIPATHS_WAITING, if timeout hasn't expired
384 */
385 static int find_multipaths_check_timeout(const struct path *pp, long
tmo,
386 struct timespec *until)
387 {
388 char path[PATH_MAX];
389 struct timespec now, ftimes[2], tdiff;
390 struct stat st;
391 long fd;
392 int r, err, retries = 0;
393
394 clock_gettime(CLOCK_REALTIME, &now);
395
396 if (snprintf(path, sizeof(path), "%s/%s", shm_find_mp_dir,
pp->dev_t) <--
397 >= sizeof(path)) {
398 condlog(1, "%s: path name overflow", __func__);
399 return FIND_MULTIPATHS_ERROR;
400 }
(gdb) print shm_find_mp_dir
$2 = "/dev/shm/multipath/find_multipaths"
(gdb) print pp->dev_t
value has been optimized out
(gdb) up
#6 0x0000556212df15bc in print_cmd_valid (conf=0x556214624970,
pathvec=0x556214682630, k=1) at main.c:487
487 wait = find_multipaths_check_timeout(pp, 0, &until);
(gdb) print pp->dev_t
(gdb) list 463,490
463 static int print_cmd_valid(int k, const vector pathvec,
464 struct config *conf)
465 {
466 static const int vals[] = { 1, 0, 2 };
467 int wait = FIND_MULTIPATHS_NEVER;
468 struct timespec until;
469 struct path *pp;
470
471 if (k < 0 || k >= sizeof(vals))
472 return 1;
473
474 if (k == 2) {
475 /*
476 * Caller ensures that pathvec[0] is the path to
477 * examine.
478 */
479 pp = VECTOR_SLOT(pathvec, 0);
480 select_find_multipaths_timeout(conf, pp);
481 wait = find_multipaths_check_timeout(
482 pp, pp->find_multipaths_timeout, &until);
483 if (wait != FIND_MULTIPATHS_WAITING)
484 k = 1;
485 } else if (pathvec != NULL) {
486 pp = VECTOR_SLOT(pathvec, 0);
487 wait = find_multipaths_check_timeout(pp, 0, &until);
<--
488 }
489 if (wait == FIND_MULTIPATHS_WAITING)
490 printf("FIND_MULTIPATHS_WAIT_UNTIL=\"%ld.%06ld\"\n",
(gdb) print pathvec
$7 = (const vector) 0x556214682630
(gdb) print *pathvec
$8 = {allocated = 0, slot = 0x0}
./libmultipath/vector.h:#define VECTOR_SIZE(V) ((V) ? ((V)->allocated) /
VECTOR_DEFAULT_SIZE : 0)
./libmultipath/vector.h:#define VECTOR_SLOT(V,E) (((V) && (E) < VECTOR_SIZE(V))
? (V)->slot[(E)] : NULL)
cd multipath-tools
cp orig try1 -a
cd try1/multipath-tools-0.7.7
wget
"https://git.opensvc.com/gitweb.cgi?p=multipath-tools/.git;a=patch;h=ccfb9a38f5cb01a7af0f4e30d18e34d4fabc8b53";
-O - | patch -p1
dpkg-buildpackage -b
su
cd /home/benutzer/multipath-tools/try1/
dpkg -i kpartx_0.7.7-3_amd64.deb multipath-tools_0.7.7-3_amd64.deb
multipath-tools-dbgsym_0.7.7-3_amd64.deb
