Thanks, Dave! I’ll take a look at those as soon as I can. I’m running multiple connections between 2 vpp hosts without issue, so it’s either a cut-through session issue or it has to do with how we setup vpp for vrf leaking.
Cheers, Florin > On Nov 15, 2018, at 3:00 PM, Dave Wallace <dwallac...@gmail.com> wrote: > > Same here. However, in the same workspace where all tests passed, I can get > this test case to fail consistently: > > EXTENDED_TESTS=y TEST=vcl.VCLThruHostStackExtendedBTestCase.* make test > EXTENDED_TESTS=y TEST=vcl.VCLIpv6ThruHostStackExtendedBTestCase.* make test > > In patch 13215, I discovered that making these test cases NOT run multiple > sockets in parallel the test passes. My latest patch to that has the > multiple sockets option commented out with "# ouch! Host Stack Bug?" so that > all tests pass. > > Thanks, > -daw- > > On 11/15/2018 4:16 PM, Florin Coras wrote: >> That’s an interesting failure. Is the test machine running out of memory? >> >> The extended tests are unstable on my server, so I do see quite a number of >> failures. However this: >> >> make test-ext TEST=vcl.VCLCutThruTestCase.test_vcl_cut_thru_uni_dir_nsock >> >> runs just fine. After the latest test framework changes, are we running >> multiple tests/vpps in parallel? I suspect that may be a source of issues. >> >> Florin >> >>> On Nov 15, 2018, at 12:11 PM, Klement Sekera via Lists.Fd.Io >>> <ksekera=cisco....@lists.fd.io> <mailto:ksekera=cisco....@lists.fd.io> >>> wrote: >>> >>> I'm seeing timeouts and coredumps... >>> >>> e.g. >>> >>> #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) >>> at /home/ksekera/vpp/src/svm/message_queue.h:299 >>> 299 return pthread_mutex_trylock (&mq->q->mutex); >>> (gdb) p mq >>> $1 = (svm_msg_q_t *) 0x204009440 >>> (gdb) p mq->q >>> $2 = (svm_queue_t *) 0x0 >>> >>> which is part of >>> >>> #4 <signal handler called> >>> #5 __pthread_mutex_trylock (mutex=0x0) at >>> ../nptl/pthread_mutex_trylock.c:39 >>> #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) >>> at /home/ksekera/vpp/src/svm/message_queue.h:299 >>> #7 0x00007f9ba04055d5 in svm_msg_q_lock_and_alloc_msg_w_ring >>> (mq=0x204009440, >>> ring_index=1, noblock=1 '\001', msg=0x7f9b5f7c2a80) >>> at /home/ksekera/vpp/src/svm/message_queue.c:121 >>> #8 0x00007f9ba14be449 in mq_try_lock_and_alloc_msg (app_mq=0x204009440, >>> msg=0x7f9b5f7c2a80) at >>> /home/ksekera/vpp/src/vnet/session/session_api.c:407 >>> #9 0x00007f9ba14be509 in mq_send_session_accepted_cb (s=0x7f9b60351400) >>> at /home/ksekera/vpp/src/vnet/session/session_api.c:432 >>> #10 0x00007f9ba1496ba0 in application_local_session_connect ( >>> client_wrk=0x7f9b60805800, server_wrk=0x7f9b60805780, ll=0x7f9b5f4c9e40, >>> opaque=0) at /home/ksekera/vpp/src/vnet/session/application.c:1646 >>> #11 0x00007f9ba14a5a62 in application_connect (a=0x7f9b5f7c2d30) >>> at /home/ksekera/vpp/src/vnet/session/application_interface.c:327 >>> ---Type <return> to continue, or q <return> to quit--- >>> #12 0x00007f9ba14a69fd in vnet_connect (a=0x7f9b5f7c2d30) >>> at /home/ksekera/vpp/src/vnet/session/application_interface.c:673 >>> #13 0x00007f9ba14c0f27 in vl_api_connect_sock_t_handler (mp=0x1300a6218) >>> at /home/ksekera/vpp/src/vnet/session/session_api.c:1305 >>> #14 0x00007f9ba1b6cb25 in vl_msg_api_handler_with_vm_node ( >>> am=0x7f9ba1d7dc60 <api_main>, the_msg=0x1300a6218, >>> vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) >>> at /home/ksekera/vpp/src/vlibapi/api_shared.c:502 >>> #15 0x00007f9ba1b39114 in void_mem_api_handle_msg_i ( >>> am=0x7f9ba1d7dc60 <api_main>, vm=0x7f9ba08fc2c0 <vlib_global_main>, >>> node=0x7f9b5f7ba000, q=0x13004c440) >>> at /home/ksekera/vpp/src/vlibmemory/memory_api.c:700 >>> #16 0x00007f9ba1b39183 in vl_mem_api_handle_msg_main ( >>> vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) >>> at /home/ksekera/vpp/src/vlibmemory/memory_api.c:710 >>> #17 0x00007f9ba1b572dd in vl_api_clnt_process ( >>> vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000, f=0x0) >>> at /home/ksekera/vpp/src/vlibmemory/vlib_api.c:350 >>> #18 0x00007f9ba0674a11 in vlib_process_bootstrap (_a=140305300978672) >>> at /home/ksekera/vpp/src/vlib/main.c:1276 >>> #19 0x00007f9b9fef4e74 in clib_calljmp () >>> from >>> /home/ksekera/vpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.19.01 >>> >>> could this be the result of a timeout and the killing of the child >>> process? >>> >>> Thanks, >>> Klement >>> >>> >>> Quoting Dave Wallace (2018-11-15 20:27:55) >>>> Klement, >>>> >>>> I just pulled the top-of-tree on master and ran only VCL tests on my >>>> 18.04 >>>> box and they all passed (see below). Another strange thing about your >>>> failure is that the test that failed is NOT an extended test. >>>> >>>> I'm currently working on a patch ([1]https://gerrit.fd.io/r/#/c/13215/ >>>> <https://gerrit.fd.io/r/#/c/13215/>) to >>>> shorten the run time for the extended tests and convert them to regular >>>> tests. In the past, I have seen some unexplained failures of some of the >>>> extended tests. I'll let you know if I encounter any of them again. >>>> >>>> Thanks, >>>> -daw- >>>> >>>> ----- %< ----- >>>> TEST=vcl.* make test-ext >>>> . . . >>>> make[2]: Leaving directory '/scratch/dwallacelf/lf/vpp/test/ext' >>>> >>>> ============================================================================== >>>> Sanity test case - verify if VPP is able to start >>>> >>>> ============================================================================== >>>> Running tests using custom test runner >>>> Active filters: file=test_vcl.py, class=None, function=None >>>> Adding tests from directory tree /scratch/dwallacelf/lf/vpp/test >>>> 28 out of 858 tests match specified filters >>>> >>>> ============================================================================== >>>> VCL Thru Host Stack Tests >>>> >>>> ============================================================================== >>>> run LDP thru host stack echo test >>>> OK >>>> run VCL thru host stack echo test >>>> OK >>>> >>>> ============================================================================== >>>> VCL Thru Host Stack Extended B Tests >>>> >>>> ============================================================================== >>>> run LDP thru host stack bi-directional (multiple sockets) test >>>> OK >>>> >>>> ============================================================================== >>>> VCL Thru Host Stack Iperf Tests >>>> >>>> ============================================================================== >>>> run LDP thru host stack iperf3 test >>>> OK >>>> >>>> ============================================================================== >>>> VCL Thru Host Stack Extended D Tests >>>> >>>> ============================================================================== >>>> run VCL thru host stack uni-directional (multiple sockets) test >>>> OK >>>> >>>> ============================================================================== >>>> VCL IPv6 Thru Host Stack Tests >>>> >>>> ============================================================================== >>>> run LDP IPv6 thru host stack echo test >>>> OK >>>> run VCL IPv6 thru host stack echo test >>>> OK >>>> >>>> ============================================================================== >>>> VCL Cut Thru Tests >>>> >>>> ============================================================================== >>>> run LDP cut thru bi-directional (multiple sockets) test >>>> OK >>>> run LDP cut thru echo test >>>> OK >>>> run LDP cut thru iperf3 test >>>> OK >>>> run LDP cut thru uni-directional (multiple sockets) test >>>> OK >>>> run VCL cut thru bi-directional (multiple sockets) test >>>> OK >>>> run VCL cut thru echo test >>>> OK >>>> run VCL cut thru uni-directional (multiple sockets) test >>>> OK >>>> >>>> ============================================================================== >>>> VCL Thru Host Stack Extended C Tests >>>> >>>> ============================================================================== >>>> run LDP thru host stack uni-directional (multiple sockets) test >>>> OK >>>> >>>> ============================================================================== >>>> VCL Thru Host Stack NSession Bidir Tests >>>> >>>> ============================================================================== >>>> run VCL thru host stack bi-directional (multiple sockets) test >>>> OK >>>> >>>> ============================================================================== >>>> VCL IPv6 Thru Host Stack Extended A Tests >>>> >>>> ============================================================================== >>>> run VCL thru host stack bi-directional (multiple sockets) test >>>> OK >>>> >>>> ============================================================================== >>>> VCL IPv6 Cut Thru Tests >>>> >>>> ============================================================================== >>>> run LDP IPv6 cut thru bi-directional (multiple sockets) test >>>> OK >>>> run LDP IPv6 cut thru echo test >>>> OK >>>> run LDP IPv6 cut thru iperf3 test >>>> OK >>>> run LDP IPv6 cut thru uni-directional (multiple sockets) test >>>> OK >>>> run VCL IPv6 cut thru bi-directional (multiple sockets) test >>>> OK >>>> run VCL IPv6 cut thru echo test >>>> OK >>>> run VCL IPv6 cut thru uni-directional (multiple sockets) test >>>> OK >>>> >>>> ============================================================================== >>>> VCL IPv6 Thru Host Stack Extended C Tests >>>> >>>> ============================================================================== >>>> run LDP thru host stack uni-directional (multiple sockets) test >>>> OK >>>> >>>> ============================================================================== >>>> VCL IPv6 Thru Host Stack Extended B Tests >>>> >>>> ============================================================================== >>>> run LDP thru host stack bi-directional (multiple sockets) test >>>> OK >>>> >>>> ============================================================================== >>>> VCL IPv6 Thru Host Stack Iperf Tests >>>> >>>> ============================================================================== >>>> run LDP thru host stack iperf3 test >>>> OK >>>> >>>> ============================================================================== >>>> VCL IPv6 Thru Host Stack Extended D Tests >>>> >>>> ============================================================================== >>>> run VCL thru host stack uni-directional (multiple sockets) test >>>> OK >>>> >>>> Ran 28 tests in 250.800s >>>> >>>> OK >>>> >>>> >>>> ============================================================================== >>>> TEST RESULTS: >>>> Scheduled tests: 28 >>>> Executed tests: 28 >>>> Passed tests: 28 >>>> >>>> ============================================================================== >>>> >>>> Test run was successful >>>> ----- %< ----- >>>> On 11/15/2018 5:47 AM, Klement Sekera via Lists.Fd.Io wrote: >>>> >>>> Hi all, >>>> >>>> I'm seeing failures on master branch on ubuntu 18.04 when invoking `make >>>> test-ext` >>>> >>>> FAILURES AND ERRORS IN TESTS: >>>> Testcase name: VCL Cut Thru Tests >>>> FAILURE: run VCL cut thru uni-directional (multiple sockets) test >>>> Testcase name: L2BD Test Case >>>> ERROR: L2BD MAC learning dual-loop test >>>> ERROR: L2BD MAC learning dual-loop test >>>> ERROR: L2BD MAC learning single-loop test >>>> Testcase name: Classifier PBR Test Case >>>> ERROR: IP PBR test >>>> >>>> digging a bit further, L2BD failure also occurs in `make test-debug`, >>>> while it doesn't appear in `make test`. This is a core due to assert. >>>> >>>> I think we should run both `make test` (release build) and `make >>>> test-debug` (debug build) as part of verify process. If it was up to me, >>>> I would run all the tests which we have in the verify job. >>>> >>>> Thoughts? >>>> >>>> Regards, >>>> Klement >>>> >>>> >>>> -=-=-=-=-=-=-=-=-=-=-=- >>>> Links: You receive all messages sent to this group. >>>> >>>> View/Reply Online (#11257): [2]https://lists.fd.io/g/vpp-dev/message/11257 >>>> <https://lists.fd.io/g/vpp-dev/message/11257> >>>> Mute This Topic: [3]https://lists.fd.io/mt/28144643/675079 >>>> <https://lists.fd.io/mt/28144643/675079> >>>> Group Owner: [4]vpp-dev+ow...@lists.fd.io >>>> <mailto:4]vpp-dev+ow...@lists.fd.io> >>>> Unsubscribe: [5]https://lists.fd.io/g/vpp-dev/unsub >>>> <https://lists.fd.io/g/vpp-dev/unsub> [[6]dwallac...@gmail.com >>>> <mailto:6]dwallac...@gmail.com>] >>>> -=-=-=-=-=-=-=-=-=-=-=- >>>> >>>> References >>>> >>>> Visible links >>>> 1. https://gerrit.fd.io/r/#/c/13215/ <https://gerrit.fd.io/r/#/c/13215/> >>>> 2. https://lists.fd.io/g/vpp-dev/message/11257 >>>> <https://lists.fd.io/g/vpp-dev/message/11257> >>>> 3. https://lists.fd.io/mt/28144643/675079 >>>> <https://lists.fd.io/mt/28144643/675079> >>>> 4. mailto:vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> >>>> 5. https://lists.fd.io/g/vpp-dev/unsub >>>> <https://lists.fd.io/g/vpp-dev/unsub> >>>> 6. mailto:dwallac...@gmail.com <mailto:dwallac...@gmail.com> >>> -=-=-=-=-=-=-=-=-=-=-=- >>> Links: You receive all messages sent to this group. >>> >>> View/Reply Online (#11270): https://lists.fd.io/g/vpp-dev/message/11270 >>> <https://lists.fd.io/g/vpp-dev/message/11270> >>> Mute This Topic: https://lists.fd.io/mt/28144643/675152 >>> <https://lists.fd.io/mt/28144643/675152> >>> Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> >>> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub >>> <https://lists.fd.io/g/vpp-dev/unsub> [fcoras.li...@gmail.com >>> <mailto:fcoras.li...@gmail.com>] >>> -=-=-=-=-=-=-=-=-=-=-=- >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11273): https://lists.fd.io/g/vpp-dev/message/11273 Mute This Topic: https://lists.fd.io/mt/28144643/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
