I've also noticed that quite often a binary called sock_test_client is left running after a vpp crash or test failure. What's worse, it eats 100% CPU.
Quoting Florin Coras (2018-11-16 00:40:06) > Thanks, Dave! > I’ll take a look at those as soon as I can. I’m running multiple > connections between 2 vpp hosts without issue, so it’s either a > cut-through session issue or it has to do with how we setup vpp for vrf > leaking. > Cheers, > Florin > > On Nov 15, 2018, at 3:00 PM, Dave Wallace <[1]dwallac...@gmail.com> > wrote: > Same here. However, in the same workspace where all tests passed, I can > get this test case to fail consistently: > > EXTENDED_TESTS=y TEST=vcl.VCLThruHostStackExtendedBTestCase.* make test > EXTENDED_TESTS=y TEST=vcl.VCLIpv6ThruHostStackExtendedBTestCase.* make > test > > In patch 13215, I discovered that making these test cases NOT run > multiple sockets in parallel the test passes. My latest patch to that > has the multiple sockets option commented out with "# ouch! Host Stack > Bug?" so that all tests pass. > > Thanks, > -daw- > On 11/15/2018 4:16 PM, Florin Coras wrote: > > That’s an interesting failure. Is the test machine running out of memory? > > The extended tests are unstable on my server, so I do see quite a number of > failures. However this: > > make test-ext TEST=vcl.VCLCutThruTestCase.test_vcl_cut_thru_uni_dir_nsock > > runs just fine. After the latest test framework changes, are we running > multiple tests/vpps in parallel? I suspect that may be a source of issues. > > Florin > > > On Nov 15, 2018, at 12:11 PM, Klement Sekera via Lists.Fd.Io > [2]<ksekera=cisco....@lists.fd.io> wrote: > > I'm seeing timeouts and coredumps... > > e.g. > > #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) > at /home/ksekera/vpp/src/svm/message_queue.h:299 > 299 return pthread_mutex_trylock (&mq->q->mutex); > (gdb) p mq > $1 = (svm_msg_q_t *) 0x204009440 > (gdb) p mq->q > $2 = (svm_queue_t *) 0x0 > > which is part of > > #4 <signal handler called> > #5 __pthread_mutex_trylock (mutex=0x0) at ../nptl/pthread_mutex_trylock.c:39 > #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) > at /home/ksekera/vpp/src/svm/message_queue.h:299 > #7 0x00007f9ba04055d5 in svm_msg_q_lock_and_alloc_msg_w_ring > (mq=0x204009440, > ring_index=1, noblock=1 '\001', msg=0x7f9b5f7c2a80) > at /home/ksekera/vpp/src/svm/message_queue.c:121 > #8 0x00007f9ba14be449 in mq_try_lock_and_alloc_msg (app_mq=0x204009440, > msg=0x7f9b5f7c2a80) at > /home/ksekera/vpp/src/vnet/session/session_api.c:407 > #9 0x00007f9ba14be509 in mq_send_session_accepted_cb (s=0x7f9b60351400) > at /home/ksekera/vpp/src/vnet/session/session_api.c:432 > #10 0x00007f9ba1496ba0 in application_local_session_connect ( > client_wrk=0x7f9b60805800, server_wrk=0x7f9b60805780, ll=0x7f9b5f4c9e40, > opaque=0) at /home/ksekera/vpp/src/vnet/session/application.c:1646 > #11 0x00007f9ba14a5a62 in application_connect (a=0x7f9b5f7c2d30) > at /home/ksekera/vpp/src/vnet/session/application_interface.c:327 > ---Type <return> to continue, or q <return> to quit--- > #12 0x00007f9ba14a69fd in vnet_connect (a=0x7f9b5f7c2d30) > at /home/ksekera/vpp/src/vnet/session/application_interface.c:673 > #13 0x00007f9ba14c0f27 in vl_api_connect_sock_t_handler (mp=0x1300a6218) > at /home/ksekera/vpp/src/vnet/session/session_api.c:1305 > #14 0x00007f9ba1b6cb25 in vl_msg_api_handler_with_vm_node ( > am=0x7f9ba1d7dc60 <api_main>, the_msg=0x1300a6218, > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) > at /home/ksekera/vpp/src/vlibapi/api_shared.c:502 > #15 0x00007f9ba1b39114 in void_mem_api_handle_msg_i ( > am=0x7f9ba1d7dc60 <api_main>, vm=0x7f9ba08fc2c0 <vlib_global_main>, > node=0x7f9b5f7ba000, q=0x13004c440) > at /home/ksekera/vpp/src/vlibmemory/memory_api.c:700 > #16 0x00007f9ba1b39183 in vl_mem_api_handle_msg_main ( > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) > at /home/ksekera/vpp/src/vlibmemory/memory_api.c:710 > #17 0x00007f9ba1b572dd in vl_api_clnt_process ( > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000, f=0x0) > at /home/ksekera/vpp/src/vlibmemory/vlib_api.c:350 > #18 0x00007f9ba0674a11 in vlib_process_bootstrap (_a=140305300978672) > at /home/ksekera/vpp/src/vlib/main.c:1276 > #19 0x00007f9b9fef4e74 in clib_calljmp () > from > /home/ksekera/vpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.19.01 > > could this be the result of a timeout and the killing of the child > process? > > Thanks, > Klement > > > Quoting Dave Wallace (2018-11-15 20:27:55) > > Klement, > > I just pulled the top-of-tree on master and ran only VCL tests on my 18.04 > box and they all passed (see below). Another strange thing about your > failure is that the test that failed is NOT an extended test. > > I'm currently working on a patch ([1][3]https://gerrit.fd.io/r/#/c/13215/) > to > shorten the run time for the extended tests and convert them to regular > tests. In the past, I have seen some unexplained failures of some of the > extended tests. I'll let you know if I encounter any of them again. > > Thanks, > -daw- > > ----- %< ----- > TEST=vcl.* make test-ext > . . . > make[2]: Leaving directory '/scratch/dwallacelf/lf/vpp/test/ext' > > ============================================================================== > Sanity test case - verify if VPP is able to start > > ============================================================================== > Running tests using custom test runner > Active filters: file=test_vcl.py, class=None, function=None > Adding tests from directory tree /scratch/dwallacelf/lf/vpp/test > 28 out of 858 tests match specified filters > > ============================================================================== > VCL Thru Host Stack Tests > > ============================================================================== > run LDP thru host stack echo test > OK > run VCL thru host stack echo test > OK > > ============================================================================== > VCL Thru Host Stack Extended B Tests > > ============================================================================== > run LDP thru host stack bi-directional (multiple sockets) test > OK > > ============================================================================== > VCL Thru Host Stack Iperf Tests > > ============================================================================== > run LDP thru host stack iperf3 test > OK > > ============================================================================== > VCL Thru Host Stack Extended D Tests > > ============================================================================== > run VCL thru host stack uni-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Tests > > ============================================================================== > run LDP IPv6 thru host stack echo test > OK > run VCL IPv6 thru host stack echo test > OK > > ============================================================================== > VCL Cut Thru Tests > > ============================================================================== > run LDP cut thru bi-directional (multiple sockets) test > OK > run LDP cut thru echo test > OK > run LDP cut thru iperf3 test > OK > run LDP cut thru uni-directional (multiple sockets) test > OK > run VCL cut thru bi-directional (multiple sockets) test > OK > run VCL cut thru echo test > OK > run VCL cut thru uni-directional (multiple sockets) test > OK > > ============================================================================== > VCL Thru Host Stack Extended C Tests > > ============================================================================== > run LDP thru host stack uni-directional (multiple sockets) test > OK > > ============================================================================== > VCL Thru Host Stack NSession Bidir Tests > > ============================================================================== > run VCL thru host stack bi-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Extended A Tests > > ============================================================================== > run VCL thru host stack bi-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Cut Thru Tests > > ============================================================================== > run LDP IPv6 cut thru bi-directional (multiple sockets) test > OK > run LDP IPv6 cut thru echo test > OK > run LDP IPv6 cut thru iperf3 test > OK > run LDP IPv6 cut thru uni-directional (multiple sockets) test > OK > run VCL IPv6 cut thru bi-directional (multiple sockets) test > OK > run VCL IPv6 cut thru echo test > OK > run VCL IPv6 cut thru uni-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Extended C Tests > > ============================================================================== > run LDP thru host stack uni-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Extended B Tests > > ============================================================================== > run LDP thru host stack bi-directional (multiple sockets) test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Iperf Tests > > ============================================================================== > run LDP thru host stack iperf3 test > OK > > ============================================================================== > VCL IPv6 Thru Host Stack Extended D Tests > > ============================================================================== > run VCL thru host stack uni-directional (multiple sockets) test > OK > > Ran 28 tests in 250.800s > > OK > > > ============================================================================== > TEST RESULTS: > Scheduled tests: 28 > Executed tests: 28 > Passed tests: 28 > > ============================================================================== > > Test run was successful > ----- %< ----- > On 11/15/2018 5:47 AM, Klement Sekera via Lists.Fd.Io wrote: > > Hi all, > > I'm seeing failures on master branch on ubuntu 18.04 when invoking `make > test-ext` > > FAILURES AND ERRORS IN TESTS: > Testcase name: VCL Cut Thru Tests > FAILURE: run VCL cut thru uni-directional (multiple sockets) test > Testcase name: L2BD Test Case > ERROR: L2BD MAC learning dual-loop test > ERROR: L2BD MAC learning dual-loop test > ERROR: L2BD MAC learning single-loop test > Testcase name: Classifier PBR Test Case > ERROR: IP PBR test > > digging a bit further, L2BD failure also occurs in `make test-debug`, > while it doesn't appear in `make test`. This is a core due to assert. > > I think we should run both `make test` (release build) and `make > test-debug` (debug build) as part of verify process. If it was up to me, > I would run all the tests which we have in the verify job. > > Thoughts? > > Regards, > Klement > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#11257): [2][4]https://lists.fd.io/g/vpp-dev/message/11257 > Mute This Topic: [3][5]https://lists.fd.io/mt/28144643/675079 > Group Owner: [[6]4]vpp-dev+ow...@lists.fd.io > Unsubscribe: [5][7]https://lists.fd.io/g/vpp-dev/unsub > [[[8]6]dwallac...@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > > References > > Visible links > 1. [9]https://gerrit.fd.io/r/#/c/13215/ > 2. [10]https://lists.fd.io/g/vpp-dev/message/11257 > 3. [11]https://lists.fd.io/mt/28144643/675079 > 4. [12]mailto:vpp-dev+ow...@lists.fd.io > 5. [13]https://lists.fd.io/g/vpp-dev/unsub > 6. [14]mailto:dwallac...@gmail.com > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#11270): [15]https://lists.fd.io/g/vpp-dev/message/11270 > Mute This Topic: [16]https://lists.fd.io/mt/28144643/675152 > Group Owner: [17]vpp-dev+ow...@lists.fd.io > Unsubscribe: [18]https://lists.fd.io/g/vpp-dev/unsub > [[19]fcoras.li...@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > > References > > Visible links > 1. mailto:dwallac...@gmail.com > 2. mailto:ksekera=cisco....@lists.fd.io > 3. https://gerrit.fd.io/r/#/c/13215/ > 4. https://lists.fd.io/g/vpp-dev/message/11257 > 5. https://lists.fd.io/mt/28144643/675079 > 6. mailto:4]vpp-dev+ow...@lists.fd.io > 7. https://lists.fd.io/g/vpp-dev/unsub > 8. mailto:6]dwallac...@gmail.com > 9. https://gerrit.fd.io/r/#/c/13215/ > 10. https://lists.fd.io/g/vpp-dev/message/11257 > 11. https://lists.fd.io/mt/28144643/675079 > 12. mailto:vpp-dev+ow...@lists.fd.io > 13. https://lists.fd.io/g/vpp-dev/unsub > 14. mailto:dwallac...@gmail.com > 15. https://lists.fd.io/g/vpp-dev/message/11270 > 16. https://lists.fd.io/mt/28144643/675152 > 17. mailto:vpp-dev+ow...@lists.fd.io > 18. https://lists.fd.io/g/vpp-dev/unsub > 19. mailto:fcoras.li...@gmail.com
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11285): https://lists.fd.io/g/vpp-dev/message/11285 Mute This Topic: https://lists.fd.io/mt/28144643/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
