Roberto Bartzen Acosta <[email protected]> writes: > Hey Aaron,
Hi Roberto, > How are you doing? > How can I see the real reason for this error? the log message "echo '*** > ERROR: 255 ***'" > after the "podman cp" doesn't say much to me, sorry. > It seems to me that this has no relation to my patch. Can you help me > understand the > problem so I can fix it? Sure thing - in the link below there is a pointer to a build URL: https://github.com/ovsrobot/ovn/actions/runs/9897498819 And just looking at the logs, for example: main: clean up vswitch 8559 ../../../tests/ovn.at:36697: test -e $OVS_RUNDIR/ovs-vswitchd.pid 8560 ../../../tests/ovn.at:36697: ovs-appctl --timeout=10 -t ovs-vswitchd exit --cleanup 8561 ovn.at:36697: waiting while kill -0 $TMPPID 2>/dev/null... 8562 ovn.at:36697: wait succeeded quickly 8563 ../../../tests/ovn.at:36697: test -e $OVS_RUNDIR/ovsdb-server.pid 8564 ../../../tests/ovn.at:36697: ovs-appctl --timeout=10 -t ovsdb-server exit 8565 ovn.at:36697: waiting while kill -0 $TMPPID 2>/dev/null... 8566 ovn.at:36697: wait succeeded quickly 8567 Undefined Behavior Sanitizer or Address Sanitizer reported errors in: sanitizers.126090 8568 8569 ================================================================= 8570 ==126090==ERROR: LeakSanitizer: detected memory leaks 8571 8572 Direct leak of 108 byte(s) in 3 object(s) allocated from: 8573 #0 0x55b195ca1110 in realloc (/workspace/ovn-tmp/ovn-24.03.90/_build/sub/northd/ovn-northd+0x377110) (BuildId: 2796c1a70d6ad795706de27129fa1f63838eb036) 8574 #1 0x55b196100888 in xrealloc__ /workspace/ovn-tmp/ovs/lib/util.c:150:9 8575 #2 0x55b196100888 in xrealloc /workspace/ovn-tmp/ovs/lib/util.c:182:12 8576 #3 0x55b1960a4801 in ds_reserve /workspace/ovn-tmp/ovs/lib/dynamic-string.c:63:22 8577 #4 0x55b1960a4801 in ds_put_format_valist /workspace/ovn-tmp/ovs/lib/dynamic-string.c:164:9 8578 #5 0x55b1960a474a in ds_put_format /workspace/ovn-tmp/ovs/lib/dynamic-string.c:142:5 8579 #6 0x55b195d98baf in build_distr_lrouter_nat_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11079:5 8580 #7 0x55b195d98baf in build_lrouter_nat_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11302:17 8581 #8 0x55b195d18f26 in build_lrouter_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11440:9 8582 #9 0x55b195cfb804 in build_lflows_thread /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:16150:21 8583 #10 0x55b1960d88ce in ovsthread_wrapper /workspace/ovn-tmp/ovs/lib/ovs-thread.c:423:12 8584 #11 0x55b195c9e7dc in asan_thread_start(void*) asan_interceptors.cpp.o 8585 8586 Direct leak of 36 byte(s) in 1 object(s) allocated from: 8587 #0 0x55b195ca1110 in realloc (/workspace/ovn-tmp/ovn-24.03.90/_build/sub/northd/ovn-northd+0x377110) (BuildId: 2796c1a70d6ad795706de27129fa1f63838eb036) 8588 #1 0x55b196100888 in xrealloc__ /workspace/ovn-tmp/ovs/lib/util.c:150:9 8589 #2 0x55b196100888 in xrealloc /workspace/ovn-tmp/ovs/lib/util.c:182:12 8590 #3 0x55b1960a4801 in ds_reserve /workspace/ovn-tmp/ovs/lib/dynamic-string.c:63:22 8591 #4 0x55b1960a4801 in ds_put_format_valist /workspace/ovn-tmp/ovs/lib/dynamic-string.c:164:9 8592 #5 0x55b1960a474a in ds_put_format /workspace/ovn-tmp/ovs/lib/dynamic-string.c:142:5 8593 #6 0x55b195d98baf in build_distr_lrouter_nat_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11079:5 8594 #7 0x55b195d98baf in build_lrouter_nat_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11302:17 8595 #8 0x55b195d18f26 in build_lrouter_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11440:9 8596 #9 0x55b195d16a60 in lflow_handle_northd_lb_changes /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:16793:9 8597 #10 0x55b195dced9a in lflow_northd_handler /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/en-lflow.c:144:10 8598 #11 0x55b195e48bef in engine_compute /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../lib/inc-proc-eng.c:437:28 8599 #12 0x55b195e48bef in engine_run_node /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../lib/inc-proc-eng.c:499:14 8600 #13 0x55b195e48bef in engine_run /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../lib/inc-proc-eng.c:524:9 8601 #14 0x55b195e05da5 in inc_proc_northd_run /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/inc-proc-northd.c:414:5 8602 #15 0x55b195dc1bbc in main /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/ovn-northd.c:970:32 8603 #16 0x7fa4617251c9 (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3) 8604 #17 0x7fa46172528a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3) 8605 #18 0x55b195c05ea4 in _start (/workspace/ovn-tmp/ovn-24.03.90/_build/sub/northd/ovn-northd+0x2dbea4) (BuildId: 2796c1a70d6ad795706de27129fa1f63838eb036) 8606 8607 Direct leak of 27 byte(s) in 3 object(s) allocated from: 8608 #0 0x55b195ca1110 in realloc (/workspace/ovn-tmp/ovn-24.03.90/_build/sub/northd/ovn-northd+0x377110) (BuildId: 2796c1a70d6ad795706de27129fa1f63838eb036) 8609 #1 0x55b196100888 in xrealloc__ /workspace/ovn-tmp/ovs/lib/util.c:150:9 8610 #2 0x55b196100888 in xrealloc /workspace/ovn-tmp/ovs/lib/util.c:182:12 8611 #3 0x55b1960a4801 in ds_reserve /workspace/ovn-tmp/ovs/lib/dynamic-string.c:63:22 8612 #4 0x55b1960a4801 in ds_put_format_valist /workspace/ovn-tmp/ovs/lib/dynamic-string.c:164:9 8613 #5 0x55b1960a474a in ds_put_format /workspace/ovn-tmp/ovs/lib/dynamic-string.c:142:5 8614 #6 0x55b195d985e6 in build_distr_lrouter_nat_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11045:9 8615 #7 0x55b195d985e6 in build_lrouter_nat_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11302:17 8616 #8 0x55b195d18f26 in build_lrouter_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11440:9 8617 #9 0x55b195cfb804 in build_lflows_thread /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:16150:21 8618 #10 0x55b1960d88ce in ovsthread_wrapper /workspace/ovn-tmp/ovs/lib/ovs-thread.c:423:12 8619 #11 0x55b195c9e7dc in asan_thread_start(void*) asan_interceptors.cpp.o 8620 8621 Direct leak of 9 byte(s) in 1 object(s) allocated from: 8622 #0 0x55b195ca1110 in realloc (/workspace/ovn-tmp/ovn-24.03.90/_build/sub/northd/ovn-northd+0x377110) (BuildId: 2796c1a70d6ad795706de27129fa1f63838eb036) 8623 #1 0x55b196100888 in xrealloc__ /workspace/ovn-tmp/ovs/lib/util.c:150:9 8624 #2 0x55b196100888 in xrealloc /workspace/ovn-tmp/ovs/lib/util.c:182:12 8625 #3 0x55b1960a4801 in ds_reserve /workspace/ovn-tmp/ovs/lib/dynamic-string.c:63:22 8626 #4 0x55b1960a4801 in ds_put_format_valist /workspace/ovn-tmp/ovs/lib/dynamic-string.c:164:9 8627 #5 0x55b1960a474a in ds_put_format /workspace/ovn-tmp/ovs/lib/dynamic-string.c:142:5 8628 #6 0x55b195d985e6 in build_distr_lrouter_nat_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11045:9 8629 #7 0x55b195d985e6 in build_lrouter_nat_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11302:17 8630 #8 0x55b195d18f26 in build_lrouter_flows_for_lb /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:11440:9 8631 #9 0x55b195d16a60 in lflow_handle_northd_lb_changes /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/northd.c:16793:9 8632 #10 0x55b195dced9a in lflow_northd_handler /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/en-lflow.c:144:10 8633 #11 0x55b195e48bef in engine_compute /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../lib/inc-proc-eng.c:437:28 8634 #12 0x55b195e48bef in engine_run_node /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../lib/inc-proc-eng.c:499:14 8635 #13 0x55b195e48bef in engine_run /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../lib/inc-proc-eng.c:524:9 8636 #14 0x55b195e05da5 in inc_proc_northd_run /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/inc-proc-northd.c:414:5 8637 #15 0x55b195dc1bbc in main /workspace/ovn-tmp/ovn-24.03.90/_build/sub/../../northd/ovn-northd.c:970:32 8638 #16 0x7fa4617251c9 (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3) 8639 #17 0x7fa46172528a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 08134323d00289185684a4cd177d202f39c2a5f3) 8640 #18 0x55b195c05ea4 in _start (/workspace/ovn-tmp/ovn-24.03.90/_build/sub/northd/ovn-northd+0x2dbea4) (BuildId: 2796c1a70d6ad795706de27129fa1f63838eb036) 8641 8642 SUMMARY: AddressSanitizer: 180 byte(s) leaked in 8 allocation(s). 8643 ../../../tests/ovs-macros.at:231: hard failure 8644 500. ovn.at:36697: 500. DNAT_SNAT and LB traffic -- parallelization=yes -- ovn_monitor_all=no (ovn.at:36697): FAILED (ovs-macros.at:231) 8645 I guess that's because you added: struct ds undnat_action = DS_EMPTY_INITIALIZER;  struct ds dnat_action = DS_EMPTY_INITIALIZER;  struct ds snat_action = DS_EMPTY_INITIALIZER; BUT, it doesn't look like you destroy them when done, so you leak the memory. I think that's the cause for these, but just a quick review. If you do ever suspect an issue you can use the recheck feature of the robot to kick off another run and look for those. See the following for more details: https://mail.openvswitch.org/pipermail/ovs-dev/2023-December/410120.html Hope it helps! > Thanks! > > Kind regards, > Roberto > > Em qui., 11 de jul. de 2024 às 19:04, 0-day Robot <[email protected]> escreveu: > > From: [email protected] > > Test-Label: github-robot: Build and Test > Test-Status: fail > http://patchwork.ozlabs.org/api/patches/1959493/ > > _github build: failed_ > Build URL: https://github.com/ovsrobot/ovn/actions/runs/9897498819 > Build Logs: > -----------------------Summary of failed steps----------------------- > "linux clang test sanitizers 301-600" failed at step build > "linux clang test sanitizers 601- unstable" failed at step build > "linux clang system-test sanitizers -100" failed at step build > "linux clang system-test sanitizers 101- unstable" failed at step build > ----------------------End summary of failed steps-------------------- > > -------------------------------BEGIN LOGS---------------------------- > > #################################################################################### > > #### [Begin job log] "linux clang test sanitizers 301-600" at step build > > #################################################################################### > > /workspace/logs/testsuite.dir/499/ovn-nb/ > /workspace/logs/testsuite.dir/499/ovn-nb/ovn-nb.db > /workspace/logs/testsuite.dir/499/ovn-nb/.ovn-nb.db.~lock~ > /workspace/logs/testsuite.dir/499/ovn-nb/ovsdb-server.log > /workspace/logs/testsuite.dir/499/cleanup > /workspace/logs/testsuite.dir/499/main/ > /workspace/logs/testsuite.dir/499/main/hv2_br-phys-tx.pcap > /workspace/logs/testsuite.dir/499/main/hv2_br-phys-rx.pcap > /workspace/logs/testsuite.dir/499/main/hv1_br-phys-tx.pcap > /workspace/logs/testsuite.dir/499/main/hv1_br-phys-rx.pcap > /workspace/logs/testsuite.dir/499/main/conf.db > /workspace/logs/testsuite.dir/499/main/ovs-vswitchd.log > /workspace/logs/testsuite.dir/499/main/.conf.db.~lock~ > /workspace/logs/testsuite.dir/499/main/ovsdb-server.log > /workspace/logs/testsuite.log > /workspace/logs/config.log > /workspace/logs/system-userspace-testsuite.at > /workspace/logs/system-dpdk-testsuite.at > /workspace/logs/system-kmod-testsuite.at > + podman cp > > 5d6f68a7193409d86a8ab7c081c105496e9a3e4e994b119949e9709104b9c7a2://workspace/logs.tgz > logs.tgz > + echo '*** ERROR: 1 ***' > *** ERROR: 1 *** > + podman rm -f > 5d6f68a7193409d86a8ab7c081c105496e9a3e4e994b119949e9709104b9c7a2 > 5d6f68a7193409d86a8ab7c081c105496e9a3e4e994b119949e9709104b9c7a2 > ##[error]Process completed with exit code 1. > > #################################################################################### > > #### [End job log] "linux clang test sanitizers 301-600" at step build > > #################################################################################### > > > > #################################################################################### > > #### [Begin job log] "linux clang test sanitizers 601- unstable" at step > build > > #################################################################################### > > + '[' 255 -ne 0 ']' > + archive_logs > + '[' -z 1 ']' > + log_dir=/workspace/logs/ > + container_exec ' > mkdir /workspace/logs/ && cp > /workspace/ovn-tmp/config.log > /workspace/logs/ && cp -r > /workspace/ovn-tmp/*/_build/sub/tests/testsuite.* > /workspace/logs/ || true && cp -r > /workspace/ovn-tmp/tests/system-*-testsuite.* /workspace/logs/ || > true && > chmod -R +r /workspace/logs/ && > tar -czvf /workspace/logs.tgz /workspace/logs/ > ' > + podman exec -i > 3f432eb02e21a359ab371521889acf59a561caf0497569bf47b6aa8d0bfa0ee1 /bin/bash -c > ' > mkdir /workspace/logs/ && cp > /workspace/ovn-tmp/config.log > /workspace/logs/ && cp -r > /workspace/ovn-tmp/*/_build/sub/tests/testsuite.* > /workspace/logs/ || true && cp -r > /workspace/ovn-tmp/tests/system-*-testsuite.* /workspace/logs/ || > true && > chmod -R +r /workspace/logs/ && > tar -czvf /workspace/logs.tgz /workspace/logs/ > ' > cp: cannot stat '/workspace/ovn-tmp/*/_build/sub/tests/testsuite.*': No such > file or > directory > tar: Removing leading `/' from member names > | | #defi/workspace/logs/ > /workspace/logs/config.log > /workspace/logs/system-userspace-testsuite.at > /workspace/logs/system-dpdk-testsuite.at > /workspace/logs/system-kmod-testsuite.at > + podman cp > > 3f432eb02e21a359ab371521889acf59a561caf0497569bf47b6aa8d0bfa0ee1://workspace/logs.tgz > logs.tgz > + echo '*** ERROR: 255 ***' > *** ERROR: 255 *** > + podman rm -f > 3f432eb02e21a359ab371521889acf59a561caf0497569bf47b6aa8d0bfa0ee1 > 3f432eb02e21a359ab371521889acf59a561caf0497569bf47b6aa8d0bfa0ee1 > ##[error]Process completed with exit code 255. > > #################################################################################### > > #### [End job log] "linux clang test sanitizers 601- unstable" at step build > > #################################################################################### > > > > #################################################################################### > > #### [Begin job log] "linux clang system-test sanitizers -100" at step build > > #################################################################################### > > /workspace/logs/system-kmod-testsuite.dir/036/ovn-nb/ovn-nb.db > /workspace/logs/system-kmod-testsuite.dir/036/ovn-nb/.ovn-nb.db.~lock~ > /workspace/logs/system-kmod-testsuite.dir/036/ovn-nb/ovsdb-server.log > /workspace/logs/system-kmod-testsuite.dir/036/index.html.11 > /workspace/logs/system-kmod-testsuite.dir/036/ovsdb-server.log > /workspace/logs/system-kmod-testsuite.dir/036/cleanup > /workspace/logs/system-kmod-testsuite.dir/036/index.html.19 > /workspace/logs/system-kmod-testsuite.dir/036/index.html.15 > /workspace/logs/system-kmod-testsuite.dir/036/wget8.log > /workspace/logs/system-kmod-testsuite.dir/036/index.html.13 > /workspace/logs/system-kmod-testsuite.dir/036/index.html.2 > /workspace/logs/system-kmod-testsuite.dir/036/index.html.7 > /workspace/logs/system-kmod-testsuite.dir/036/index.html.16 > /workspace/logs/system-kmod-testsuite.dir/036/wget2.log > /workspace/logs/system-kmod-testsuite.dir/036/index.html.1 > /workspace/logs/system-kmod-testsuite.log > /workspace/logs/system-userspace-testsuite.at > /workspace/logs/system-dpdk-testsuite.at > /workspace/logs/system-kmod-testsuite.at > + podman cp > > 8de83105e9f78b7efbd81505be2ed39c2c78a061312a09510ec2d712c56ce049://workspace/logs.tgz > logs.tgz > + echo '*** ERROR: 1 ***' > *** ERROR: 1 *** > + podman rm -f > 8de83105e9f78b7efbd81505be2ed39c2c78a061312a09510ec2d712c56ce049 > 8de83105e9f78b7efbd81505be2ed39c2c78a061312a09510ec2d712c56ce049 > ##[error]Process completed with exit code 1. > > #################################################################################### > > #### [End job log] "linux clang system-test sanitizers -100" at step build > > #################################################################################### > > > > #################################################################################### > > #### [Begin job log] "linux clang system-test sanitizers 101- unstable" at > step build > > #################################################################################### > > /workspace/logs/system-kmod-testsuite.dir/181/conf.db > /workspace/logs/system-kmod-testsuite.dir/181/ovs-vswitchd.log > /workspace/logs/system-kmod-testsuite.dir/181/.conf.db.~lock~ > /workspace/logs/system-kmod-testsuite.dir/181/northd/ > /workspace/logs/system-kmod-testsuite.dir/181/northd/ovn-northd.log > /workspace/logs/system-kmod-testsuite.dir/181/ovn-controller.log > /workspace/logs/system-kmod-testsuite.dir/181/system-kmod-testsuite.log > /workspace/logs/system-kmod-testsuite.dir/181/nc1.pid > /workspace/logs/system-kmod-testsuite.dir/181/ovn-nb/ > /workspace/logs/system-kmod-testsuite.dir/181/ovn-nb/ovn-nb.db > /workspace/logs/system-kmod-testsuite.dir/181/ovn-nb/.ovn-nb.db.~lock~ > /workspace/logs/system-kmod-testsuite.dir/181/ovn-nb/ovsdb-server.log > /workspace/logs/system-kmod-testsuite.dir/181/ovsdb-server.log > /workspace/logs/system-kmod-testsuite.dir/181/nc0.pid > /workspace/logs/system-kmod-testsuite.dir/181/cleanup > /workspace/logs/system-kmod-testsuite.log > /workspace/logs/system-userspace-testsuite.at > /workspace/logs/system-dpdk-testsuite.at > /workspace/logs/system-kmod-testsuite.at > + podman cp > > fac1c98ecaa438027bf2301f9437907052e7cffac1a8854a76b201fab3d8c826://workspace/logs.tgz > logs.tgz > + echo '*** ERROR: 255 ***' > *** ERROR: 255 *** > + podman rm -f > fac1c98ecaa438027bf2301f9437907052e7cffac1a8854a76b201fab3d8c826 > fac1c98ecaa438027bf2301f9437907052e7cffac1a8854a76b201fab3d8c826 > ##[error]Process completed with exit code 255. > > #################################################################################### > > #### [End job log] "linux clang system-test sanitizers 101- unstable" at > step build > > #################################################################################### > > --------------------------------END LOGS----------------------------- > > ‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho > inicial. Se você > não está listado nos endereços constantes no cabeçalho, pedimos-lhe que > desconsidere > completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou > execução das > ações citadas estão imediatamente anuladas e proibidas’. > > ‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar > que nenhum vírus > esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade > por quaisquer > perdas ou danos causados por esse e-mail ou por seus anexos’. _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
