Re: [vpp-dev] deadlock issue in VPP during DHCP packet processing

2017-10-03 Thread Balaji Kn 
Hi John,

Applying the patch to 17.07 tree did not helped.  Are there any more
relevant fixes I need to apply on 17.07 tree for this issue?

Regards,
Balaji

On Tue, Sep 26, 2017 at 7:15 PM, John Lo (loj)  wrote:

> There was a patch recently merged in mater/17.10:
>
> https://gerrit.fd.io/r/#/c/8464/
>
>
>
> Can you try the latest image from master/17.10 or apply the patch it into
> your 17.07 tree and rebuild?
>
>
>
> Regards,
>
> John
>
>
>
> *From:* vpp-dev-boun...@lists.fd.io [mailto:vpp-dev-boun...@lists.fd.io] *On
> Behalf Of *Balaji Kn
> *Sent:* Tuesday, September 26, 2017 8:37 AM
> *To:* vpp-dev@lists.fd.io
> *Subject:* [vpp-dev] deadlock issue in VPP during DHCP packet processing
>
>
>
> Hello All,
>
>
>
> I am working on VPP 17.07 and using DHCP proxy functionality. CPU
> configuration provided as one main thread and one worker thread.
>
>
>
> cpu {
>
>   main-core 0
>
>   corelist-workers 1
>
> }
>
>
>
> Deadlock is observed while processing DHCP offer packet in VPP. However
> issue is not observed if i comment CPU configuration in startup.conf file
> (if running in single thread) and everything works smoothly.
>
>
>
> *Following message is displayed on console.*
>
> vlib_worker_thread_barrier_sync: worker thread deadlock
>
>
>
> *Backtrace from core file generated.*
>
> [New LWP 12792]
>
> [Thread debugging using libthread_db enabled]
>
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>
> Core was generated by `/usr/bin/vpp -c /etc/vpp/startup.conf'.
>
> Program terminated with signal SIGABRT, Aborted.
>
> #0  0x7f721ab0fc37 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>
> 56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or
> directory.
>
> (gdb) bt
>
> #0  0x7f721ab0fc37 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>
> #1  0x7f721ab13028 in __GI_abort () at abort.c:89
>
> #2  0x00407073 in os_panic () at /root/vfe/fe-vfe/datapath/vpp/
> build-data/../src/vpp/vnet/main.c:263
>
> #3  0x7f721c0b5d5d in vlib_worker_thread_barrier_sync
> (vm=0x7f721c2e12e0 )
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/threads.c:1192
>
> #4  0x7f721c2e973a in vl_msg_api_handler_with_vm_node 
> (am=am@entry=0x7f721c5063a0
> , the_msg=the_msg@entry=0x304bc6d4,
>
> vm=vm@entry=0x7f721c2e12e0 , node=node@entry=
> 0x7f71da6a8000)
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlibapi/api_
> shared.c:501
>
> #5  0x7f721c2f34be in memclnt_process (vm=,
> node=0x7f71da6a8000, f=)
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlibmemory/
> memory_vlib.c:544
>
> #6  0x7f721c08ec96 in vlib_process_bootstrap (_a=)
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1259
>
> #7  0x7f721b2ec858 in clib_calljmp () at /root/vfe/fe-vfe/datapath/vpp/
> build-data/../src/vppinfra/longjmp.S:110
>
> #8  0x7f71da9efe20 in ?? ()
>
> #9  0x7f721c090041 in vlib_process_startup (f=0x0, p=0x7f71da6a8000,
> vm=0x7f721c2e12e0 )
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1281
>
> #10 dispatch_process (vm=0x7f721c2e12e0 ,
> p=0x7f71da6a8000, last_time_stamp=58535483853222, f=0x0)
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1324
>
> #11 0x00d800d9 in ?? ()
>
>
>
> Any pointers would be appreciated.
>
>
>
> Regards,
>
> Balaji
>
>
>
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] deadlock issue in VPP during DHCP packet processing

2017-09-27 Thread Balaji Kn 
Hi John,

Applying the patch to 17.07 tree did not solved the issue.I am observing
many compilation issues with latest image of master and could not verify.

Am i missing anything?

Regards,
Balaji

On Tue, Sep 26, 2017 at 7:15 PM, John Lo (loj)  wrote:

> There was a patch recently merged in mater/17.10:
>
> https://gerrit.fd.io/r/#/c/8464/
>
>
>
> Can you try the latest image from master/17.10 or apply the patch it into
> your 17.07 tree and rebuild?
>
>
>
> Regards,
>
> John
>
>
>
> *From:* vpp-dev-boun...@lists.fd.io [mailto:vpp-dev-boun...@lists.fd.io] *On
> Behalf Of *Balaji Kn
> *Sent:* Tuesday, September 26, 2017 8:37 AM
> *To:* vpp-dev@lists.fd.io
> *Subject:* [vpp-dev] deadlock issue in VPP during DHCP packet processing
>
>
>
> Hello All,
>
>
>
> I am working on VPP 17.07 and using DHCP proxy functionality. CPU
> configuration provided as one main thread and one worker thread.
>
>
>
> cpu {
>
>   main-core 0
>
>   corelist-workers 1
>
> }
>
>
>
> Deadlock is observed while processing DHCP offer packet in VPP. However
> issue is not observed if i comment CPU configuration in startup.conf file
> (if running in single thread) and everything works smoothly.
>
>
>
> *Following message is displayed on console.*
>
> vlib_worker_thread_barrier_sync: worker thread deadlock
>
>
>
> *Backtrace from core file generated.*
>
> [New LWP 12792]
>
> [Thread debugging using libthread_db enabled]
>
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>
> Core was generated by `/usr/bin/vpp -c /etc/vpp/startup.conf'.
>
> Program terminated with signal SIGABRT, Aborted.
>
> #0  0x7f721ab0fc37 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>
> 56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or
> directory.
>
> (gdb) bt
>
> #0  0x7f721ab0fc37 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>
> #1  0x7f721ab13028 in __GI_abort () at abort.c:89
>
> #2  0x00407073 in os_panic () at /root/vfe/fe-vfe/datapath/vpp/
> build-data/../src/vpp/vnet/main.c:263
>
> #3  0x7f721c0b5d5d in vlib_worker_thread_barrier_sync
> (vm=0x7f721c2e12e0 )
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/threads.c:1192
>
> #4  0x7f721c2e973a in vl_msg_api_handler_with_vm_node 
> (am=am@entry=0x7f721c5063a0
> , the_msg=the_msg@entry=0x304bc6d4,
>
> vm=vm@entry=0x7f721c2e12e0 , node=node@entry=
> 0x7f71da6a8000)
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlibapi/api_
> shared.c:501
>
> #5  0x7f721c2f34be in memclnt_process (vm=,
> node=0x7f71da6a8000, f=)
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlibmemory/
> memory_vlib.c:544
>
> #6  0x7f721c08ec96 in vlib_process_bootstrap (_a=)
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1259
>
> #7  0x7f721b2ec858 in clib_calljmp () at /root/vfe/fe-vfe/datapath/vpp/
> build-data/../src/vppinfra/longjmp.S:110
>
> #8  0x7f71da9efe20 in ?? ()
>
> #9  0x7f721c090041 in vlib_process_startup (f=0x0, p=0x7f71da6a8000,
> vm=0x7f721c2e12e0 )
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1281
>
> #10 dispatch_process (vm=0x7f721c2e12e0 ,
> p=0x7f71da6a8000, last_time_stamp=58535483853222, f=0x0)
>
> at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1324
>
> #11 0x00d800d9 in ?? ()
>
>
>
> Any pointers would be appreciated.
>
>
>
> Regards,
>
> Balaji
>
>
>
___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] deadlock issue in VPP during DHCP packet processing

2017-09-26 Thread John Lo (loj) 
There was a patch recently merged in mater/17.10:
https://gerrit.fd.io/r/#/c/8464/

Can you try the latest image from master/17.10 or apply the patch it into your 
17.07 tree and rebuild?

Regards,
John

From: vpp-dev-boun...@lists.fd.io [mailto:vpp-dev-boun...@lists.fd.io] On 
Behalf Of Balaji Kn
Sent: Tuesday, September 26, 2017 8:37 AM
To: vpp-dev@lists.fd.io
Subject: [vpp-dev] deadlock issue in VPP during DHCP packet processing

Hello All,

I am working on VPP 17.07 and using DHCP proxy functionality. CPU configuration 
provided as one main thread and one worker thread.

cpu {
  main-core 0
  corelist-workers 1
}

Deadlock is observed while processing DHCP offer packet in VPP. However issue 
is not observed if i comment CPU configuration in startup.conf file (if running 
in single thread) and everything works smoothly.

Following message is displayed on console.
vlib_worker_thread_barrier_sync: worker thread deadlock

Backtrace from core file generated.
[New LWP 12792]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/vpp -c /etc/vpp/startup.conf'.
Program terminated with signal SIGABRT, Aborted.
#0  0x7f721ab0fc37 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x7f721ab0fc37 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x7f721ab13028 in __GI_abort () at abort.c:89
#2  0x00407073 in os_panic () at 
/root/vfe/fe-vfe/datapath/vpp/build-data/../src/vpp/vnet/main.c:263
#3  0x7f721c0b5d5d in vlib_worker_thread_barrier_sync (vm=0x7f721c2e12e0 
)
at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/threads.c:1192
#4  0x7f721c2e973a in vl_msg_api_handler_with_vm_node 
(am=am@entry=0x7f721c5063a0 , the_msg=the_msg@entry=0x304bc6d4,
vm=vm@entry=0x7f721c2e12e0 , 
node=node@entry=0x7f71da6a8000)
at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlibapi/api_shared.c:501
#5  0x7f721c2f34be in memclnt_process (vm=, 
node=0x7f71da6a8000, f=)
at 
/root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlibmemory/memory_vlib.c:544
#6  0x7f721c08ec96 in vlib_process_bootstrap (_a=)
at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1259
#7  0x7f721b2ec858 in clib_calljmp () at 
/root/vfe/fe-vfe/datapath/vpp/build-data/../src/vppinfra/longjmp.S:110
#8  0x7f71da9efe20 in ?? ()
#9  0x7f721c090041 in vlib_process_startup (f=0x0, p=0x7f71da6a8000, 
vm=0x7f721c2e12e0 )
at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1281
#10 dispatch_process (vm=0x7f721c2e12e0 , p=0x7f71da6a8000, 
last_time_stamp=58535483853222, f=0x0)
at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1324
#11 0x00d800d9 in ?? ()

Any pointers would be appreciated.

Regards,
Balaji

___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev

Re: [vpp-dev] deadlock issue in VPP during DHCP packet processing

2017-09-26 Thread Dave Barach (dbarach) 
Does this happen w/ master/latest? My guess: yes...

Florin and I are working on a patch to fix an obvious issue in this path right 
now, look for results shortly...

HTH... Dave


From: vpp-dev-boun...@lists.fd.io [mailto:vpp-dev-boun...@lists.fd.io] On 
Behalf Of Balaji Kn
Sent: Tuesday, September 26, 2017 8:37 AM
To: vpp-dev@lists.fd.io
Subject: [vpp-dev] deadlock issue in VPP during DHCP packet processing

Hello All,

I am working on VPP 17.07 and using DHCP proxy functionality. CPU configuration 
provided as one main thread and one worker thread.

cpu {
  main-core 0
  corelist-workers 1
}

Deadlock is observed while processing DHCP offer packet in VPP. However issue 
is not observed if i comment CPU configuration in startup.conf file (if running 
in single thread) and everything works smoothly.

Following message is displayed on console.
vlib_worker_thread_barrier_sync: worker thread deadlock

Backtrace from core file generated.
[New LWP 12792]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/vpp -c /etc/vpp/startup.conf'.
Program terminated with signal SIGABRT, Aborted.
#0  0x7f721ab0fc37 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x7f721ab0fc37 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x7f721ab13028 in __GI_abort () at abort.c:89
#2  0x00407073 in os_panic () at 
/root/vfe/fe-vfe/datapath/vpp/build-data/../src/vpp/vnet/main.c:263
#3  0x7f721c0b5d5d in vlib_worker_thread_barrier_sync (vm=0x7f721c2e12e0 
)
at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/threads.c:1192
#4  0x7f721c2e973a in vl_msg_api_handler_with_vm_node 
(am=am@entry=0x7f721c5063a0 , the_msg=the_msg@entry=0x304bc6d4,
vm=vm@entry=0x7f721c2e12e0 , 
node=node@entry=0x7f71da6a8000)
at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlibapi/api_shared.c:501
#5  0x7f721c2f34be in memclnt_process (vm=, 
node=0x7f71da6a8000, f=)
at 
/root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlibmemory/memory_vlib.c:544
#6  0x7f721c08ec96 in vlib_process_bootstrap (_a=)
at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1259
#7  0x7f721b2ec858 in clib_calljmp () at 
/root/vfe/fe-vfe/datapath/vpp/build-data/../src/vppinfra/longjmp.S:110
#8  0x7f71da9efe20 in ?? ()
#9  0x7f721c090041 in vlib_process_startup (f=0x0, p=0x7f71da6a8000, 
vm=0x7f721c2e12e0 )
at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1281
#10 dispatch_process (vm=0x7f721c2e12e0 , p=0x7f71da6a8000, 
last_time_stamp=58535483853222, f=0x0)
at /root/vfe/fe-vfe/datapath/vpp/build-data/../src/vlib/main.c:1324
#11 0x00d800d9 in ?? ()

Any pointers would be appreciated.

Regards,
Balaji

___
vpp-dev mailing list
vpp-dev@lists.fd.io
https://lists.fd.io/mailman/listinfo/vpp-dev