Re: [vpp-dev] vpp deadlock - syslog in signal handler

2017-10-17 Thread Dave Barach (dbarach)
In almost all cases, the glibc malloc heap will not be pickled since it's not 
used on a regular basis.

For some effort, one could replace the syslog library code, I guess.

Thanks... Dave

From: vpp-dev-boun...@lists.fd.io [mailto:vpp-dev-boun...@lists.fd.io] On 
Behalf Of Gabriel Ganne
Sent: Tuesday, October 17, 2017 4:18 AM
To: vpp-dev 
Subject: [vpp-dev] vpp deadlock - syslog in signal handler


Hi,



I have encountered a deadlock in vpp on the raising of a memory alloc exception.

The signal is caught by unix_signal_handler(), which determines this is a fatal 
error and then syslogs the error message.

The problem is that syslog() then tries to allocate a scratchpad memory, and 
deadlocks since allocation is the reason why I'm here in the first place.



clib_warning() functions should be safe because all the memory needed is 
alloc'ed at init, but I don't see how this syslog() call can succeed.

Should I just remove it ?

Or is there a way I don't know about to still make this work ?



Below is a backtrace of the problem:
#0  0xa42e2c0c in __lll_lock_wait_private 
(futex=futex@entry=0xa43869a0 ) at ./lowlevellock.c:33
#1  0xa426b6e8 in __GI___libc_malloc (bytes=bytes@entry=584) at 
malloc.c:2888
#2  0xa425ace8 in __GI___open_memstream (bufloc=0x655b4670, 
bufloc@entry=0x655b46d0, sizeloc=0x655b4678, 
sizeloc@entry=0x655b46d8) at memstream.c:76
#3  0xa42cef18 in __GI___vsyslog_chk (ap=..., fmt=0xa4be2990 "%s", 
flag=-1, pri=27) at ../misc/syslog.c:167
#4  __syslog (pri=pri@entry=27, fmt=fmt@entry=0xa4be2990 "%s") at 
../misc/syslog.c:117
#5  0xa4bd7ab4 in unix_signal_handler (signum=, 
si=, uc=) at 
/home/gannega/vpp/build-data/../src/vlib/unix/main.c:119
#6  
#7  0xa42654e0 in malloc_consolidate (av=av@entry=0xa43869a0 
) at malloc.c:4182
#8  0xa4269354 in malloc_consolidate (av=0xa43869a0 ) 
at malloc.c:4151
#9  _int_malloc (av=av@entry=0xa43869a0 , 
bytes=bytes@entry=32816) at malloc.c:3450
#10 0xa426b5b4 in __GI___libc_malloc (bytes=bytes@entry=32816) at 
malloc.c:2890
#11 0xa4299000 in __alloc_dir (statp=0x655b5d48, flags=0, 
close_fd=true, fd=5) at ../sysdeps/posix/opendir.c:247
#12 opendir_tail (fd=) at ../sysdeps/posix/opendir.c:145
#13 __opendir (name=name@entry=0xa4bdf258 "/sys/bus/pci/devices") at 
../sysdeps/posix/opendir.c:200
#14 0xa4bde088 in foreach_directory_file 
(dir_name=dir_name@entry=0xa4bdf258 "/sys/bus/pci/devices", 
f=f@entry=0xa4baf4a8 , arg=arg@entry=0xa4c0af30 
,
scan_dirs=scan_dirs@entry=0) at 
/home/gannega/vpp/build-data/../src/vlib/unix/util.c:59
#15 0xa4baed64 in linux_pci_init (vm=0xa4c0af30 ) 
at /home/gannega/vpp/build-data/../src/vlib/linux/pci.c:648
#16 0xa4bae504 in vlib_call_init_exit_functions (vm=0xa4c0af30 
, head=, call_once=call_once@entry=1) at 
/home/gannega/vpp/build-data/../src/vlib/init.c:57
#17 0xa4bae548 in vlib_call_all_init_functions (vm=) at 
/home/gannega/vpp/build-data/../src/vlib/init.c:75
#18 0xa4bb3838 in vlib_main (vm=, 
vm@entry=0xa4c0af30 , input=input@entry=0x655b5fc8) 
at /home/gannega/vpp/build-data/../src/vlib/main.c:1748
#19 0xa4bd7c0c in thread0 (arg=281473445834544) at 
/home/gannega/vpp/build-data/../src/vlib/unix/main.c:567
#20 0xa44f3e38 in clib_calljmp () at 
/home/gannega/vpp/build-data/../src/vppinfra/longjmp.S:676


Best regards,

--

Gabriel Ganne
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

[vpp-dev] vpp deadlock - syslog in signal handler

2017-10-17 Thread Gabriel Ganne
Hi,



I have encountered a deadlock in vpp on the raising of a memory alloc exception.

The signal is caught by unix_signal_handler(), which determines this is a fatal 
error and then syslogs the error message.

The problem is that syslog() then tries to allocate a scratchpad memory, and 
deadlocks since allocation is the reason why I'm here in the first place.



clib_warning() functions should be safe because all the memory needed is 
alloc'ed at init, but I don't see how this syslog() call can succeed.

Should I just remove it ?

Or is there a way I don't know about to still make this work ?



Below is a backtrace of the problem:
#0  0xa42e2c0c in __lll_lock_wait_private 
(futex=futex@entry=0xa43869a0 ) at ./lowlevellock.c:33
#1  0xa426b6e8 in __GI___libc_malloc (bytes=bytes@entry=584) at 
malloc.c:2888
#2  0xa425ace8 in __GI___open_memstream (bufloc=0x655b4670, 
bufloc@entry=0x655b46d0, sizeloc=0x655b4678, 
sizeloc@entry=0x655b46d8) at memstream.c:76
#3  0xa42cef18 in __GI___vsyslog_chk (ap=..., fmt=0xa4be2990 "%s", 
flag=-1, pri=27) at ../misc/syslog.c:167
#4  __syslog (pri=pri@entry=27, fmt=fmt@entry=0xa4be2990 "%s") at 
../misc/syslog.c:117
#5  0xa4bd7ab4 in unix_signal_handler (signum=, 
si=, uc=) at 
/home/gannega/vpp/build-data/../src/vlib/unix/main.c:119
#6  
#7  0xa42654e0 in malloc_consolidate (av=av@entry=0xa43869a0 
) at malloc.c:4182
#8  0xa4269354 in malloc_consolidate (av=0xa43869a0 ) 
at malloc.c:4151
#9  _int_malloc (av=av@entry=0xa43869a0 , 
bytes=bytes@entry=32816) at malloc.c:3450
#10 0xa426b5b4 in __GI___libc_malloc (bytes=bytes@entry=32816) at 
malloc.c:2890
#11 0xa4299000 in __alloc_dir (statp=0x655b5d48, flags=0, 
close_fd=true, fd=5) at ../sysdeps/posix/opendir.c:247
#12 opendir_tail (fd=) at ../sysdeps/posix/opendir.c:145
#13 __opendir (name=name@entry=0xa4bdf258 "/sys/bus/pci/devices") at 
../sysdeps/posix/opendir.c:200
#14 0xa4bde088 in foreach_directory_file 
(dir_name=dir_name@entry=0xa4bdf258 "/sys/bus/pci/devices", 
f=f@entry=0xa4baf4a8 , arg=arg@entry=0xa4c0af30 
,
scan_dirs=scan_dirs@entry=0) at 
/home/gannega/vpp/build-data/../src/vlib/unix/util.c:59
#15 0xa4baed64 in linux_pci_init (vm=0xa4c0af30 ) 
at /home/gannega/vpp/build-data/../src/vlib/linux/pci.c:648
#16 0xa4bae504 in vlib_call_init_exit_functions (vm=0xa4c0af30 
, head=, call_once=call_once@entry=1) at 
/home/gannega/vpp/build-data/../src/vlib/init.c:57
#17 0xa4bae548 in vlib_call_all_init_functions (vm=) at 
/home/gannega/vpp/build-data/../src/vlib/init.c:75
#18 0xa4bb3838 in vlib_main (vm=, 
vm@entry=0xa4c0af30 , input=input@entry=0x655b5fc8) 
at /home/gannega/vpp/build-data/../src/vlib/main.c:1748
#19 0xa4bd7c0c in thread0 (arg=281473445834544) at 
/home/gannega/vpp/build-data/../src/vlib/unix/main.c:567
#20 0xa44f3e38 in clib_calljmp () at 
/home/gannega/vpp/build-data/../src/vppinfra/longjmp.S:676


Best regards,

--

Gabriel Ganne
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev