I'm seeing timeouts and coredumps...
e.g.
#6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440)
at /home/ksekera/vpp/src/svm/message_queue.h:299
299 return pthread_mutex_trylock (&mq->q->mutex);
(gdb) p mq
$1 = (svm_msg_q_t *) 0x204009440
(gdb) p mq->q
$2 = (svm_queue_t *) 0x0
which is part of
#4 <signal handler called>
#5 __pthread_mutex_trylock (mutex=0x0) at ../nptl/pthread_mutex_trylock.c:39
#6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440)
at /home/ksekera/vpp/src/svm/message_queue.h:299
#7 0x00007f9ba04055d5 in svm_msg_q_lock_and_alloc_msg_w_ring (mq=0x204009440,
ring_index=1, noblock=1 '\001', msg=0x7f9b5f7c2a80)
at /home/ksekera/vpp/src/svm/message_queue.c:121
#8 0x00007f9ba14be449 in mq_try_lock_and_alloc_msg (app_mq=0x204009440,
msg=0x7f9b5f7c2a80) at /home/ksekera/vpp/src/vnet/session/session_api.c:407
#9 0x00007f9ba14be509 in mq_send_session_accepted_cb (s=0x7f9b60351400)
at /home/ksekera/vpp/src/vnet/session/session_api.c:432
#10 0x00007f9ba1496ba0 in application_local_session_connect (
client_wrk=0x7f9b60805800, server_wrk=0x7f9b60805780, ll=0x7f9b5f4c9e40,
opaque=0) at /home/ksekera/vpp/src/vnet/session/application.c:1646
#11 0x00007f9ba14a5a62 in application_connect (a=0x7f9b5f7c2d30)
at /home/ksekera/vpp/src/vnet/session/application_interface.c:327
---Type <return> to continue, or q <return> to quit---
#12 0x00007f9ba14a69fd in vnet_connect (a=0x7f9b5f7c2d30)
at /home/ksekera/vpp/src/vnet/session/application_interface.c:673
#13 0x00007f9ba14c0f27 in vl_api_connect_sock_t_handler (mp=0x1300a6218)
at /home/ksekera/vpp/src/vnet/session/session_api.c:1305
#14 0x00007f9ba1b6cb25 in vl_msg_api_handler_with_vm_node (
am=0x7f9ba1d7dc60 <api_main>, the_msg=0x1300a6218,
vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000)
at /home/ksekera/vpp/src/vlibapi/api_shared.c:502
#15 0x00007f9ba1b39114 in void_mem_api_handle_msg_i (
am=0x7f9ba1d7dc60 <api_main>, vm=0x7f9ba08fc2c0 <vlib_global_main>,
node=0x7f9b5f7ba000, q=0x13004c440)
at /home/ksekera/vpp/src/vlibmemory/memory_api.c:700
#16 0x00007f9ba1b39183 in vl_mem_api_handle_msg_main (
vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000)
at /home/ksekera/vpp/src/vlibmemory/memory_api.c:710
#17 0x00007f9ba1b572dd in vl_api_clnt_process (
vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000, f=0x0)
at /home/ksekera/vpp/src/vlibmemory/vlib_api.c:350
#18 0x00007f9ba0674a11 in vlib_process_bootstrap (_a=140305300978672)
at /home/ksekera/vpp/src/vlib/main.c:1276
#19 0x00007f9b9fef4e74 in clib_calljmp ()
from
/home/ksekera/vpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.19.01
could this be the result of a timeout and the killing of the child
process?
Thanks,
Klement
Quoting Dave Wallace (2018-11-15 20:27:55)
> Klement,
>
> I just pulled the top-of-tree on master and ran only VCL tests on my 18.04
> box and they all passed (see below). Another strange thing about your
> failure is that the test that failed is NOT an extended test.
>
> I'm currently working on a patch ([1]https://gerrit.fd.io/r/#/c/13215/) to
> shorten the run time for the extended tests and convert them to regular
> tests. In the past, I have seen some unexplained failures of some of the
> extended tests. I'll let you know if I encounter any of them again.
>
> Thanks,
> -daw-
>
> ----- %< -----
> TEST=vcl.* make test-ext
> . . .
> make[2]: Leaving directory '/scratch/dwallacelf/lf/vpp/test/ext'
>
> ==============================================================================
> Sanity test case - verify if VPP is able to start
>
> ==============================================================================
> Running tests using custom test runner
> Active filters: file=test_vcl.py, class=None, function=None
> Adding tests from directory tree /scratch/dwallacelf/lf/vpp/test
> 28 out of 858 tests match specified filters
>
> ==============================================================================
> VCL Thru Host Stack Tests
>
> ==============================================================================
> run LDP thru host stack echo test
> OK
> run VCL thru host stack echo test
> OK
>
> ==============================================================================
> VCL Thru Host Stack Extended B Tests
>
> ==============================================================================
> run LDP thru host stack bi-directional (multiple sockets) test
> OK
>
> ==============================================================================
> VCL Thru Host Stack Iperf Tests
>
> ==============================================================================
> run LDP thru host stack iperf3 test
> OK
>
> ==============================================================================
> VCL Thru Host Stack Extended D Tests
>
> ==============================================================================
> run VCL thru host stack uni-directional (multiple sockets) test
> OK
>
> ==============================================================================
> VCL IPv6 Thru Host Stack Tests
>
> ==============================================================================
> run LDP IPv6 thru host stack echo test
> OK
> run VCL IPv6 thru host stack echo test
> OK
>
> ==============================================================================
> VCL Cut Thru Tests
>
> ==============================================================================
> run LDP cut thru bi-directional (multiple sockets) test
> OK
> run LDP cut thru echo test
> OK
> run LDP cut thru iperf3 test
> OK
> run LDP cut thru uni-directional (multiple sockets) test
> OK
> run VCL cut thru bi-directional (multiple sockets) test
> OK
> run VCL cut thru echo test
> OK
> run VCL cut thru uni-directional (multiple sockets) test
> OK
>
> ==============================================================================
> VCL Thru Host Stack Extended C Tests
>
> ==============================================================================
> run LDP thru host stack uni-directional (multiple sockets) test
> OK
>
> ==============================================================================
> VCL Thru Host Stack NSession Bidir Tests
>
> ==============================================================================
> run VCL thru host stack bi-directional (multiple sockets) test
> OK
>
> ==============================================================================
> VCL IPv6 Thru Host Stack Extended A Tests
>
> ==============================================================================
> run VCL thru host stack bi-directional (multiple sockets) test
> OK
>
> ==============================================================================
> VCL IPv6 Cut Thru Tests
>
> ==============================================================================
> run LDP IPv6 cut thru bi-directional (multiple sockets) test
> OK
> run LDP IPv6 cut thru echo test
> OK
> run LDP IPv6 cut thru iperf3 test
> OK
> run LDP IPv6 cut thru uni-directional (multiple sockets) test
> OK
> run VCL IPv6 cut thru bi-directional (multiple sockets) test
> OK
> run VCL IPv6 cut thru echo test
> OK
> run VCL IPv6 cut thru uni-directional (multiple sockets) test
> OK
>
> ==============================================================================
> VCL IPv6 Thru Host Stack Extended C Tests
>
> ==============================================================================
> run LDP thru host stack uni-directional (multiple sockets) test
> OK
>
> ==============================================================================
> VCL IPv6 Thru Host Stack Extended B Tests
>
> ==============================================================================
> run LDP thru host stack bi-directional (multiple sockets) test
> OK
>
> ==============================================================================
> VCL IPv6 Thru Host Stack Iperf Tests
>
> ==============================================================================
> run LDP thru host stack iperf3 test
> OK
>
> ==============================================================================
> VCL IPv6 Thru Host Stack Extended D Tests
>
> ==============================================================================
> run VCL thru host stack uni-directional (multiple sockets) test
> OK
>
> Ran 28 tests in 250.800s
>
> OK
>
>
> ==============================================================================
> TEST RESULTS:
> Scheduled tests: 28
> Executed tests: 28
> Passed tests: 28
>
> ==============================================================================
>
> Test run was successful
> ----- %< -----
> On 11/15/2018 5:47 AM, Klement Sekera via Lists.Fd.Io wrote:
>
> Hi all,
>
> I'm seeing failures on master branch on ubuntu 18.04 when invoking `make
> test-ext`
>
> FAILURES AND ERRORS IN TESTS:
> Testcase name: VCL Cut Thru Tests
> FAILURE: run VCL cut thru uni-directional (multiple sockets) test
> Testcase name: L2BD Test Case
> ERROR: L2BD MAC learning dual-loop test
> ERROR: L2BD MAC learning dual-loop test
> ERROR: L2BD MAC learning single-loop test
> Testcase name: Classifier PBR Test Case
> ERROR: IP PBR test
>
> digging a bit further, L2BD failure also occurs in `make test-debug`,
> while it doesn't appear in `make test`. This is a core due to assert.
>
> I think we should run both `make test` (release build) and `make
> test-debug` (debug build) as part of verify process. If it was up to me,
> I would run all the tests which we have in the verify job.
>
> Thoughts?
>
> Regards,
> Klement
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#11257): [2]https://lists.fd.io/g/vpp-dev/message/11257
> Mute This Topic: [3]https://lists.fd.io/mt/28144643/675079
> Group Owner: [4][email protected]
> Unsubscribe: [5]https://lists.fd.io/g/vpp-dev/unsub
> [[6][email protected]]
> -=-=-=-=-=-=-=-=-=-=-=-
>
> References
>
> Visible links
> 1. https://gerrit.fd.io/r/#/c/13215/
> 2. https://lists.fd.io/g/vpp-dev/message/11257
> 3. https://lists.fd.io/mt/28144643/675079
> 4. mailto:[email protected]
> 5. https://lists.fd.io/g/vpp-dev/unsub
> 6. mailto:[email protected]
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#11270): https://lists.fd.io/g/vpp-dev/message/11270
Mute This Topic: https://lists.fd.io/mt/28144643/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
