No, it's on my laptop and out of 16g mem, 11g is caches so there shouldn't be any OOM issue there.
Unless you change TEST_JOBS option (setting it to either auto or <number of jobs>), the test framework still runs 1 job at a time. Klement Quoting Florin Coras (2018-11-15 22:16:05) > That’s an interesting failure. Is the test machine running out of memory? > > The extended tests are unstable on my server, so I do see quite a number of > failures. However this: > > make test-ext TEST=vcl.VCLCutThruTestCase.test_vcl_cut_thru_uni_dir_nsock > > runs just fine. After the latest test framework changes, are we running > multiple tests/vpps in parallel? I suspect that may be a source of issues. > > Florin > > > On Nov 15, 2018, at 12:11 PM, Klement Sekera via Lists.Fd.Io > > <ksekera=cisco....@lists.fd.io> wrote: > > > > I'm seeing timeouts and coredumps... > > > > e.g. > > > > #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) > > at /home/ksekera/vpp/src/svm/message_queue.h:299 > > 299 return pthread_mutex_trylock (&mq->q->mutex); > > (gdb) p mq > > $1 = (svm_msg_q_t *) 0x204009440 > > (gdb) p mq->q > > $2 = (svm_queue_t *) 0x0 > > > > which is part of > > > > #4 <signal handler called> > > #5 __pthread_mutex_trylock (mutex=0x0) at > > ../nptl/pthread_mutex_trylock.c:39 > > #6 0x00007f9ba0404eb6 in svm_msg_q_try_lock (mq=0x204009440) > > at /home/ksekera/vpp/src/svm/message_queue.h:299 > > #7 0x00007f9ba04055d5 in svm_msg_q_lock_and_alloc_msg_w_ring > > (mq=0x204009440, > > ring_index=1, noblock=1 '\001', msg=0x7f9b5f7c2a80) > > at /home/ksekera/vpp/src/svm/message_queue.c:121 > > #8 0x00007f9ba14be449 in mq_try_lock_and_alloc_msg (app_mq=0x204009440, > > msg=0x7f9b5f7c2a80) at > > /home/ksekera/vpp/src/vnet/session/session_api.c:407 > > #9 0x00007f9ba14be509 in mq_send_session_accepted_cb (s=0x7f9b60351400) > > at /home/ksekera/vpp/src/vnet/session/session_api.c:432 > > #10 0x00007f9ba1496ba0 in application_local_session_connect ( > > client_wrk=0x7f9b60805800, server_wrk=0x7f9b60805780, ll=0x7f9b5f4c9e40, > > opaque=0) at /home/ksekera/vpp/src/vnet/session/application.c:1646 > > #11 0x00007f9ba14a5a62 in application_connect (a=0x7f9b5f7c2d30) > > at /home/ksekera/vpp/src/vnet/session/application_interface.c:327 > > ---Type <return> to continue, or q <return> to quit--- > > #12 0x00007f9ba14a69fd in vnet_connect (a=0x7f9b5f7c2d30) > > at /home/ksekera/vpp/src/vnet/session/application_interface.c:673 > > #13 0x00007f9ba14c0f27 in vl_api_connect_sock_t_handler (mp=0x1300a6218) > > at /home/ksekera/vpp/src/vnet/session/session_api.c:1305 > > #14 0x00007f9ba1b6cb25 in vl_msg_api_handler_with_vm_node ( > > am=0x7f9ba1d7dc60 <api_main>, the_msg=0x1300a6218, > > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) > > at /home/ksekera/vpp/src/vlibapi/api_shared.c:502 > > #15 0x00007f9ba1b39114 in void_mem_api_handle_msg_i ( > > am=0x7f9ba1d7dc60 <api_main>, vm=0x7f9ba08fc2c0 <vlib_global_main>, > > node=0x7f9b5f7ba000, q=0x13004c440) > > at /home/ksekera/vpp/src/vlibmemory/memory_api.c:700 > > #16 0x00007f9ba1b39183 in vl_mem_api_handle_msg_main ( > > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000) > > at /home/ksekera/vpp/src/vlibmemory/memory_api.c:710 > > #17 0x00007f9ba1b572dd in vl_api_clnt_process ( > > vm=0x7f9ba08fc2c0 <vlib_global_main>, node=0x7f9b5f7ba000, f=0x0) > > at /home/ksekera/vpp/src/vlibmemory/vlib_api.c:350 > > #18 0x00007f9ba0674a11 in vlib_process_bootstrap (_a=140305300978672) > > at /home/ksekera/vpp/src/vlib/main.c:1276 > > #19 0x00007f9b9fef4e74 in clib_calljmp () > > from > > /home/ksekera/vpp/build-root/install-vpp_debug-native/vpp/lib/libvppinfra.so.19.01 > > > > could this be the result of a timeout and the killing of the child > > process? > > > > Thanks, > > Klement > > > > > > Quoting Dave Wallace (2018-11-15 20:27:55) > >> Klement, > >> > >> I just pulled the top-of-tree on master and ran only VCL tests on my > >> 18.04 > >> box and they all passed (see below). Another strange thing about your > >> failure is that the test that failed is NOT an extended test. > >> > >> I'm currently working on a patch ([1]https://gerrit.fd.io/r/#/c/13215/) > >> to > >> shorten the run time for the extended tests and convert them to regular > >> tests. In the past, I have seen some unexplained failures of some of the > >> extended tests. I'll let you know if I encounter any of them again. > >> > >> Thanks, > >> -daw- > >> > >> ----- %< ----- > >> TEST=vcl.* make test-ext > >> . . . > >> make[2]: Leaving directory '/scratch/dwallacelf/lf/vpp/test/ext' > >> > >> ============================================================================== > >> Sanity test case - verify if VPP is able to start > >> > >> ============================================================================== > >> Running tests using custom test runner > >> Active filters: file=test_vcl.py, class=None, function=None > >> Adding tests from directory tree /scratch/dwallacelf/lf/vpp/test > >> 28 out of 858 tests match specified filters > >> > >> ============================================================================== > >> VCL Thru Host Stack Tests > >> > >> ============================================================================== > >> run LDP thru host stack echo test > >> OK > >> run VCL thru host stack echo test > >> OK > >> > >> ============================================================================== > >> VCL Thru Host Stack Extended B Tests > >> > >> ============================================================================== > >> run LDP thru host stack bi-directional (multiple sockets) test > >> OK > >> > >> ============================================================================== > >> VCL Thru Host Stack Iperf Tests > >> > >> ============================================================================== > >> run LDP thru host stack iperf3 test > >> OK > >> > >> ============================================================================== > >> VCL Thru Host Stack Extended D Tests > >> > >> ============================================================================== > >> run VCL thru host stack uni-directional (multiple sockets) test > >> OK > >> > >> ============================================================================== > >> VCL IPv6 Thru Host Stack Tests > >> > >> ============================================================================== > >> run LDP IPv6 thru host stack echo test > >> OK > >> run VCL IPv6 thru host stack echo test > >> OK > >> > >> ============================================================================== > >> VCL Cut Thru Tests > >> > >> ============================================================================== > >> run LDP cut thru bi-directional (multiple sockets) test > >> OK > >> run LDP cut thru echo test > >> OK > >> run LDP cut thru iperf3 test > >> OK > >> run LDP cut thru uni-directional (multiple sockets) test > >> OK > >> run VCL cut thru bi-directional (multiple sockets) test > >> OK > >> run VCL cut thru echo test > >> OK > >> run VCL cut thru uni-directional (multiple sockets) test > >> OK > >> > >> ============================================================================== > >> VCL Thru Host Stack Extended C Tests > >> > >> ============================================================================== > >> run LDP thru host stack uni-directional (multiple sockets) test > >> OK > >> > >> ============================================================================== > >> VCL Thru Host Stack NSession Bidir Tests > >> > >> ============================================================================== > >> run VCL thru host stack bi-directional (multiple sockets) test > >> OK > >> > >> ============================================================================== > >> VCL IPv6 Thru Host Stack Extended A Tests > >> > >> ============================================================================== > >> run VCL thru host stack bi-directional (multiple sockets) test > >> OK > >> > >> ============================================================================== > >> VCL IPv6 Cut Thru Tests > >> > >> ============================================================================== > >> run LDP IPv6 cut thru bi-directional (multiple sockets) test > >> OK > >> run LDP IPv6 cut thru echo test > >> OK > >> run LDP IPv6 cut thru iperf3 test > >> OK > >> run LDP IPv6 cut thru uni-directional (multiple sockets) test > >> OK > >> run VCL IPv6 cut thru bi-directional (multiple sockets) test > >> OK > >> run VCL IPv6 cut thru echo test > >> OK > >> run VCL IPv6 cut thru uni-directional (multiple sockets) test > >> OK > >> > >> ============================================================================== > >> VCL IPv6 Thru Host Stack Extended C Tests > >> > >> ============================================================================== > >> run LDP thru host stack uni-directional (multiple sockets) test > >> OK > >> > >> ============================================================================== > >> VCL IPv6 Thru Host Stack Extended B Tests > >> > >> ============================================================================== > >> run LDP thru host stack bi-directional (multiple sockets) test > >> OK > >> > >> ============================================================================== > >> VCL IPv6 Thru Host Stack Iperf Tests > >> > >> ============================================================================== > >> run LDP thru host stack iperf3 test > >> OK > >> > >> ============================================================================== > >> VCL IPv6 Thru Host Stack Extended D Tests > >> > >> ============================================================================== > >> run VCL thru host stack uni-directional (multiple sockets) test > >> OK > >> > >> Ran 28 tests in 250.800s > >> > >> OK > >> > >> > >> ============================================================================== > >> TEST RESULTS: > >> Scheduled tests: 28 > >> Executed tests: 28 > >> Passed tests: 28 > >> > >> ============================================================================== > >> > >> Test run was successful > >> ----- %< ----- > >> On 11/15/2018 5:47 AM, Klement Sekera via Lists.Fd.Io wrote: > >> > >> Hi all, > >> > >> I'm seeing failures on master branch on ubuntu 18.04 when invoking `make > >> test-ext` > >> > >> FAILURES AND ERRORS IN TESTS: > >> Testcase name: VCL Cut Thru Tests > >> FAILURE: run VCL cut thru uni-directional (multiple sockets) test > >> Testcase name: L2BD Test Case > >> ERROR: L2BD MAC learning dual-loop test > >> ERROR: L2BD MAC learning dual-loop test > >> ERROR: L2BD MAC learning single-loop test > >> Testcase name: Classifier PBR Test Case > >> ERROR: IP PBR test > >> > >> digging a bit further, L2BD failure also occurs in `make test-debug`, > >> while it doesn't appear in `make test`. This is a core due to assert. > >> > >> I think we should run both `make test` (release build) and `make > >> test-debug` (debug build) as part of verify process. If it was up to me, > >> I would run all the tests which we have in the verify job. > >> > >> Thoughts? > >> > >> Regards, > >> Klement > >> > >> > >> -=-=-=-=-=-=-=-=-=-=-=- > >> Links: You receive all messages sent to this group. > >> > >> View/Reply Online (#11257): [2]https://lists.fd.io/g/vpp-dev/message/11257 > >> Mute This Topic: [3]https://lists.fd.io/mt/28144643/675079 > >> Group Owner: [4]vpp-dev+ow...@lists.fd.io > >> Unsubscribe: [5]https://lists.fd.io/g/vpp-dev/unsub > >> [[6]dwallac...@gmail.com] > >> -=-=-=-=-=-=-=-=-=-=-=- > >> > >> References > >> > >> Visible links > >> 1. https://gerrit.fd.io/r/#/c/13215/ > >> 2. https://lists.fd.io/g/vpp-dev/message/11257 > >> 3. https://lists.fd.io/mt/28144643/675079 > >> 4. mailto:vpp-dev+ow...@lists.fd.io > >> 5. https://lists.fd.io/g/vpp-dev/unsub > >> 6. mailto:dwallac...@gmail.com > > -=-=-=-=-=-=-=-=-=-=-=- > > Links: You receive all messages sent to this group. > > > > View/Reply Online (#11270): https://lists.fd.io/g/vpp-dev/message/11270 > > Mute This Topic: https://lists.fd.io/mt/28144643/675152 > > Group Owner: vpp-dev+ow...@lists.fd.io > > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [fcoras.li...@gmail.com] > > -=-=-=-=-=-=-=-=-=-=-=- >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11279): https://lists.fd.io/g/vpp-dev/message/11279 Mute This Topic: https://lists.fd.io/mt/28144643/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
