Re: [vpp-dev] vpp main thread crashed at mspace_put

[Satya Murthy]

Thanks Sudhir for the quick inputs.
Will check if we have any leaks.

Also, as a side question, How are you finding leaks in VPP ?
Are you using address sanitizer (or) something else?

--
Thanks & Regards,
Murthy

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19833): https://lists.fd.io/g/vpp-dev/message/19833
Mute This Topic: https://lists.fd.io/mt/81600282/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp main thread crashed at mspace_put

[Sudhir CR via lists.fd.io]

Hi Murthy,
We observed this issue when memory is exhausted in our system (due to
memory leak in our application).
After solving the above mentioned issue we have not observed this issue.

Regards,
Sudhir

On Mon, Jul 19, 2021 at 4:46 PM Satya Murthy 
wrote:

> Hi Sudhir,
>
> Were you able to find a solution to this problem.
> We are also facing similar issue.
>
> Any inputs would be helpful.
>
> --
> Thanks & Regards,
> Murthy
> 
>
>

-- 
NOTICE TO
RECIPIENT This e-mail message and any attachments are 
confidential and may be
privileged. If you received this e-mail in error, 
any review, use,
dissemination, distribution, or copying of this e-mail is 
strictly
prohibited. Please notify us immediately of the error by return 
e-mail and
please delete this message from your system. For more 
information about Rtbrick, please visit us at www.rtbrick.com 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19832): https://lists.fd.io/g/vpp-dev/message/19832
Mute This Topic: https://lists.fd.io/mt/81600282/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp main thread crashed at mspace_put

[Satya Murthy]

Hi Sudhir,

Were you able to find a solution to this problem.
We are also facing similar issue.

Any inputs would be helpful.

--
Thanks & Regards,
Murthy

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19830): https://lists.fd.io/g/vpp-dev/message/19830
Mute This Topic: https://lists.fd.io/mt/81600282/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp main thread crashed at mspace_put

[Benoit Ganne (bganne) via lists.fd.io]

Hi Sudhir, this is probably a memory corruption which corrupted some heap 
metadata.
You can try running under ASan and see if it catches the error when it happens: 
https://fd.io/docs/vpp/master/troubleshooting/sanitizer.html
Otherwise, we'll need more info including coredump and api traces, see 
https://fd.io/docs/vpp/master/troubleshooting/reportingissues/index.html

Best
ben

> -Original Message-
> From: vpp-dev@lists.fd.io  On Behalf Of Sudhir CR via
> lists.fd.io
> Sent: vendredi 26 mars 2021 05:49
> To: vpp-dev@lists.fd.io
> Subject: Re: [vpp-dev] vpp main thread crashed at mspace_put
> 
> Hi All,
> The segmentation fault is happening at memset in the code below.
> #if CLIB_DEBUG <http://rajith/lxr/http/ident?sn=vpp-19-08;i=CLIB_DEBUG>  >
> 0 && !defined(CLIB_SANITIZE_ADDR)
>   /* Poison the object */
>   {
>  size_t psize = mspace_usable_size
> <http://rajith/lxr/http/ident?sn=vpp-19-08;i=mspace_usable_size>
> (object_header);
>  memset (object_header, 0x13, psize);
>}
> #endif
> Not sure how to proceed further to root cause the issue.
> 
> Thanks,
> Sudhir
> 
> 
> On Thu, Mar 25, 2021 at 5:31 PM Sudhir CR  <mailto:sud...@rtbrick.com> > wrote:
> 
> 
>   Hi All,
>   We have loaded our box with internet feed routes. Initially
> everything is good.
>   But after three hours we observed a vpp main thread crashed due to
> segmentation error at mspace_put function.
>   #28 0x7f0f802c0793 in unix_signal_handler (signum=11,
> si=0x7f0f33c086b0, uc=0x7f0f33c08580)
>   at /development/libvpp/src/vlib/unix/main.c:127
>   #29 
>   #30 __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-
> unaligned-erms.S:145
>   #31 0x7f0f7fa67735 in mspace_put (msp=0x130044010,
> p_arg=0x130089a48) at /development/libvpp/src/vppinfra/dlmalloc.c:4316
>   We are using 20.09 version and the complete backtrace is  pasted
> below. Any help in fixing the issue would be appreciated.
>   Thread 3 (Thread 0x7f0eccbfe700 (LWP 476)):
>   #0  0x7f0f8023cff7 in vlib_worker_thread_barrier_check () at
> /development/libvpp/src/vlib/threads.h:438
>   #1  0x7f0f8023751e in vlib_main_or_worker_loop
> (vm=0x7f0f652dd300, is_main=0) at /development/libvpp/src/vlib/main.c:1788
>   #2  0x7f0f80236d37 in vlib_worker_loop (vm=0x7f0f652dd300) at
> /development/libvpp/src/vlib/main.c:2008
>   #3  0x7f0f8028e91a in vlib_worker_thread_fn (arg=0x7f0f4a974940)
> at /development/libvpp/src/vlib/threads.c:1862
>   #4  0x7f0f7fab5c34 in clib_calljmp () at
> /development/libvpp/src/vppinfra/longjmp.S:123
>   #5  0x7f0eccbfdec0 in ?? ()
> [58/393]
>   #6  0x7f0f80286ac3 in vlib_worker_thread_bootstrap_fn
> (arg=0x7f0f4a974940) at /development/libvpp/src/vlib/threads.c:585
>   ---Type  to continue, or q  to quit---
>   Backtrace stopped: previous frame inner to this frame (corrupt
> stack?)
> 
>   Thread 2 (Thread 0x7f0ecd3ff700 (LWP 475)):
>   #0  0x7f0f8023cfec in vlib_worker_thread_barrier_check () at
> /development/libvpp/src/vlib/threads.h:438
>   #1  0x7f0f8023751e in vlib_main_or_worker_loop
> (vm=0x7f0f64d99ec0, is_main=0) at /development/libvpp/src/vlib/main.c:1788
>   #2  0x7f0f80236d37 in vlib_worker_loop (vm=0x7f0f64d99ec0) at
> /development/libvpp/src/vlib/main.c:2008
>   #3  0x7f0f8028e91a in vlib_worker_thread_fn (arg=0x7f0f4a974840)
> at /development/libvpp/src/vlib/threads.c:1862
>   #4  0x7f0f7fab5c34 in clib_calljmp () at
> /development/libvpp/src/vppinfra/longjmp.S:123
>   #5  0x7f0ecd3feec0 in ?? ()
>   #6  0x7f0f80286ac3 in vlib_worker_thread_bootstrap_fn
> (arg=0x7f0f4a974840) at /development/libvpp/src/vlib/threads.c:585
>   Backtrace stopped: previous frame inner to this frame (corrupt
> stack?)
> 
>   Thread 1 (Thread 0x7f0f8d076d00 (LWP 280)):
>   #0  __GI_raise (sig=sig@entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:51
>   #1  0x7f0f8c0d3921 in __GI_abort () at abort.c:79
>   #2  0x7f0f81cbd253 in os_panic () at
> /development/libvpp/src/vpp/vnet/main.c:572
>   #3  0x7f0f7fa91aa9 in debugger () at
> /development/libvpp/src/vppinfra/error.c:84
>   #4  0x7f0f7fa91827 in _clib_error (how_to_die=2,
> function_name=0x0, line_number=0, fmt=0x7f0f7fb613df "%s:%d (%s) assertion
> `%s' fails")
>   at /development/libvpp/src/vppinfra/error.c:143
>   #5  0x7f0f7fa98e61 in _vec_resize_inline (v=0x7f0f4a69fa90,
> length_increment=16, data_bytes=16, header_bytes=0, data_align=1,
>   numa_id=255) at /development/libvpp/src/vppinfra/vec.h:154
>   #

Re: [vpp-dev] vpp main thread crashed at mspace_put

[Sudhir CR via lists.fd.io]

Hi All,

The segmentation fault is happening at memset in the code below.

#if CLIB_DEBUG 
> 0 && !defined(CLIB_SANITIZE_ADDR)

  */* Poison the object */*
  {
 size_t psize = mspace_usable_size

(object_header);
 memset (object_header, 0x13, psize);
   }
#endif

Not sure how to proceed further to root cause the issue.


Thanks,

Sudhir



On Thu, Mar 25, 2021 at 5:31 PM Sudhir CR  wrote:

> Hi All,
> We have loaded our box with internet feed routes. Initially everything is
> good.
> But after *three hours* we observed a *vpp main thread crashed* due to 
> *segmentation
> error *at mspace_put function.
>
> #28 0x7f0f802c0793 in unix_signal_handler (signum=11, si=0x7f0f33c086b0, 
> uc=0x7f0f33c08580)
> at /development/libvpp/src/vlib/unix/main.c:127
> #29 
> #30 __memset_avx2_erms () at 
> ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:145
> #31 0x7f0f7fa67735 in mspace_put (msp=0x130044010, p_arg=0x130089a48) at 
> /development/libvpp/src/vppinfra/dlmalloc.c:4316
>
> We are using *20.09 version* and the complete* backtrace *is  pasted below. 
> Any help in fixing the issue would be appreciated.
>
> Thread 3 (Thread 0x7f0eccbfe700 (LWP 476)):
> #0  0x7f0f8023cff7 in vlib_worker_thread_barrier_check () at 
> /development/libvpp/src/vlib/threads.h:438
> #1  0x7f0f8023751e in vlib_main_or_worker_loop (vm=0x7f0f652dd300, 
> is_main=0) at /development/libvpp/src/vlib/main.c:1788
> #2  0x7f0f80236d37 in vlib_worker_loop (vm=0x7f0f652dd300) at 
> /development/libvpp/src/vlib/main.c:2008
> #3  0x7f0f8028e91a in vlib_worker_thread_fn (arg=0x7f0f4a974940) at 
> /development/libvpp/src/vlib/threads.c:1862
> #4  0x7f0f7fab5c34 in clib_calljmp () at 
> /development/libvpp/src/vppinfra/longjmp.S:123
> #5  0x7f0eccbfdec0 in ?? ()   
> [58/393]
> #6  0x7f0f80286ac3 in vlib_worker_thread_bootstrap_fn 
> (arg=0x7f0f4a974940) at /development/libvpp/src/vlib/threads.c:585
> ---Type  to continue, or q  to quit---
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>
> Thread 2 (Thread 0x7f0ecd3ff700 (LWP 475)):
> #0  0x7f0f8023cfec in vlib_worker_thread_barrier_check () at 
> /development/libvpp/src/vlib/threads.h:438
> #1  0x7f0f8023751e in vlib_main_or_worker_loop (vm=0x7f0f64d99ec0, 
> is_main=0) at /development/libvpp/src/vlib/main.c:1788
> #2  0x7f0f80236d37 in vlib_worker_loop (vm=0x7f0f64d99ec0) at 
> /development/libvpp/src/vlib/main.c:2008
> #3  0x7f0f8028e91a in vlib_worker_thread_fn (arg=0x7f0f4a974840) at 
> /development/libvpp/src/vlib/threads.c:1862
> #4  0x7f0f7fab5c34 in clib_calljmp () at 
> /development/libvpp/src/vppinfra/longjmp.S:123
> #5  0x7f0ecd3feec0 in ?? ()
> #6  0x7f0f80286ac3 in vlib_worker_thread_bootstrap_fn 
> (arg=0x7f0f4a974840) at /development/libvpp/src/vlib/threads.c:585
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>
> Thread 1 (Thread 0x7f0f8d076d00 (LWP 280)):
> #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> #1  0x7f0f8c0d3921 in __GI_abort () at abort.c:79
> #2  0x7f0f81cbd253 in os_panic () at 
> /development/libvpp/src/vpp/vnet/main.c:572
> #3  0x7f0f7fa91aa9 in debugger () at 
> /development/libvpp/src/vppinfra/error.c:84
> #4  0x7f0f7fa91827 in _clib_error (how_to_die=2, function_name=0x0, 
> line_number=0, fmt=0x7f0f7fb613df "%s:%d (%s) assertion `%s' fails")
> at /development/libvpp/src/vppinfra/error.c:143
> #5  0x7f0f7fa98e61 in _vec_resize_inline (v=0x7f0f4a69fa90, 
> length_increment=16, data_bytes=16, header_bytes=0, data_align=1,
> numa_id=255) at /development/libvpp/src/vppinfra/vec.h:154
> #6  0x7f0f7fa98b8b in va_format (s=0x7f0f4a69fa90 "", fmt=0x7f0f802dbf1b 
> "received signal %U, PC %U", va=0x7f0f33c065f0)
> at /development/libvpp/src/vppinfra/format.c:403
> #7  0x7f0f7faa03c6 in format (s=0x7f0f4a69fa90 "", fmt=0x7f0f802dbf1b 
> "received signal %U, PC %U")
> at /development/libvpp/src/vppinfra/format.c:428
> #8  0x7f0f802c0793 in unix_signal_handler (signum=6, si=0x7f0f33c068b0, 
> uc=0x7f0f33c06780)
> at /development/libvpp/src/vlib/unix/main.c:127
> #9  
> #10 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> #11 0x7f0f8c0d3921 in __GI_abort () at abort.c:79
> #12 0x7f0f81cbd253 in os_panic () at 
> /development/libvpp/src/vpp/vnet/main.c:572
> #13 0x7f0f7fa91aa9 in debugger () at 
> /development/libvpp/src/vppinfra/error.c:84
> #14 0x7f0f7fa91827 in _clib_error (how_to_die=2, function_name=0x0, 
> line_number=0, fmt=0x7f0f7fb613df "%s:%d (%s) assertion `%s' f[23/393]
> at /development/libvpp/src/vppinfra/error.c:143
> ---Type  to continue, or q  to quit---
> #15 0x7f0f7fa98e61 in 

[vpp-dev] vpp main thread crashed at mspace_put

[Sudhir CR via lists.fd.io]

Hi All,
We have loaded our box with internet feed routes. Initially everything is
good.
But after *three hours* we observed a *vpp main thread crashed* due to
*segmentation
error *at mspace_put function.

#28 0x7f0f802c0793 in unix_signal_handler (signum=11,
si=0x7f0f33c086b0, uc=0x7f0f33c08580)
at /development/libvpp/src/vlib/unix/main.c:127
#29 
#30 __memset_avx2_erms () at
../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:145
#31 0x7f0f7fa67735 in mspace_put (msp=0x130044010,
p_arg=0x130089a48) at /development/libvpp/src/vppinfra/dlmalloc.c:4316

We are using *20.09 version* and the complete* backtrace *is  pasted
below. Any help in fixing the issue would be appreciated.

Thread 3 (Thread 0x7f0eccbfe700 (LWP 476)):
#0  0x7f0f8023cff7 in vlib_worker_thread_barrier_check () at
/development/libvpp/src/vlib/threads.h:438
#1  0x7f0f8023751e in vlib_main_or_worker_loop (vm=0x7f0f652dd300,
is_main=0) at /development/libvpp/src/vlib/main.c:1788
#2  0x7f0f80236d37 in vlib_worker_loop (vm=0x7f0f652dd300) at
/development/libvpp/src/vlib/main.c:2008
#3  0x7f0f8028e91a in vlib_worker_thread_fn (arg=0x7f0f4a974940)
at /development/libvpp/src/vlib/threads.c:1862
#4  0x7f0f7fab5c34 in clib_calljmp () at
/development/libvpp/src/vppinfra/longjmp.S:123
#5  0x7f0eccbfdec0 in ?? ()

[58/393]
#6  0x7f0f80286ac3 in vlib_worker_thread_bootstrap_fn
(arg=0x7f0f4a974940) at /development/libvpp/src/vlib/threads.c:585
---Type  to continue, or q  to quit---
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 2 (Thread 0x7f0ecd3ff700 (LWP 475)):
#0  0x7f0f8023cfec in vlib_worker_thread_barrier_check () at
/development/libvpp/src/vlib/threads.h:438
#1  0x7f0f8023751e in vlib_main_or_worker_loop (vm=0x7f0f64d99ec0,
is_main=0) at /development/libvpp/src/vlib/main.c:1788
#2  0x7f0f80236d37 in vlib_worker_loop (vm=0x7f0f64d99ec0) at
/development/libvpp/src/vlib/main.c:2008
#3  0x7f0f8028e91a in vlib_worker_thread_fn (arg=0x7f0f4a974840)
at /development/libvpp/src/vlib/threads.c:1862
#4  0x7f0f7fab5c34 in clib_calljmp () at
/development/libvpp/src/vppinfra/longjmp.S:123
#5  0x7f0ecd3feec0 in ?? ()
#6  0x7f0f80286ac3 in vlib_worker_thread_bootstrap_fn
(arg=0x7f0f4a974840) at /development/libvpp/src/vlib/threads.c:585
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 1 (Thread 0x7f0f8d076d00 (LWP 280)):
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x7f0f8c0d3921 in __GI_abort () at abort.c:79
#2  0x7f0f81cbd253 in os_panic () at
/development/libvpp/src/vpp/vnet/main.c:572
#3  0x7f0f7fa91aa9 in debugger () at
/development/libvpp/src/vppinfra/error.c:84
#4  0x7f0f7fa91827 in _clib_error (how_to_die=2,
function_name=0x0, line_number=0, fmt=0x7f0f7fb613df "%s:%d (%s)
assertion `%s' fails")
at /development/libvpp/src/vppinfra/error.c:143
#5  0x7f0f7fa98e61 in _vec_resize_inline (v=0x7f0f4a69fa90,
length_increment=16, data_bytes=16, header_bytes=0, data_align=1,
numa_id=255) at /development/libvpp/src/vppinfra/vec.h:154
#6  0x7f0f7fa98b8b in va_format (s=0x7f0f4a69fa90 "",
fmt=0x7f0f802dbf1b "received signal %U, PC %U", va=0x7f0f33c065f0)
at /development/libvpp/src/vppinfra/format.c:403
#7  0x7f0f7faa03c6 in format (s=0x7f0f4a69fa90 "",
fmt=0x7f0f802dbf1b "received signal %U, PC %U")
at /development/libvpp/src/vppinfra/format.c:428
#8  0x7f0f802c0793 in unix_signal_handler (signum=6,
si=0x7f0f33c068b0, uc=0x7f0f33c06780)
at /development/libvpp/src/vlib/unix/main.c:127
#9  
#10 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#11 0x7f0f8c0d3921 in __GI_abort () at abort.c:79
#12 0x7f0f81cbd253 in os_panic () at
/development/libvpp/src/vpp/vnet/main.c:572
#13 0x7f0f7fa91aa9 in debugger () at
/development/libvpp/src/vppinfra/error.c:84
#14 0x7f0f7fa91827 in _clib_error (how_to_die=2,
function_name=0x0, line_number=0, fmt=0x7f0f7fb613df "%s:%d (%s)
assertion `%s' f[23/393]
at /development/libvpp/src/vppinfra/error.c:143
---Type  to continue, or q  to quit---
#15 0x7f0f7fa98e61 in _vec_resize_inline (v=0x7f0f4a69fa90,
length_increment=16, data_bytes=16, header_bytes=0, data_align=1,
numa_id=255) at /development/libvpp/src/vppinfra/vec.h:154
#16 0x7f0f7fa98b8b in va_format (s=0x7f0f4a69fa90 "",
fmt=0x7f0f802dbf1b "received signal %U, PC %U", va=0x7f0f33c074f0)
at /development/libvpp/src/vppinfra/format.c:403
#17 0x7f0f7faa03c6 in format (s=0x7f0f4a69fa90 "",
fmt=0x7f0f802dbf1b "received signal %U, PC %U")
at /development/libvpp/src/vppinfra/format.c:428
#18 0x7f0f802c0793 in unix_signal_handler (signum=6,
si=0x7f0f33c077b0, uc=0x7f0f33c07680)
at /development/libvpp/src/vlib/unix/main.c:127
#19 
#20 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#21 0x7f0f8c0d3921 in __GI_abort () at abort.c:79
#22 0x7f0f81cbd253 in os_panic () at