On 2019/09/02 08:25, Rafael Sadowski wrote:
> On Wed Aug 21, 2019 at 06:58:02PM +0200, Rafael Sadowski wrote:
> > The following diff switch the cmake test task to ctest(1). This enables us
> > to
> > easily debug/report the tests.
> >
> > In the normal state everything is as always. `make test` looks like:
> >
> > ===> Regression tests for cmocka-1.1.3
> > Test project /usr/ports/pobj/cmocka-1.1.3/build-amd64
> > Start 1: test_alloc
> > 1/44 Test #1: test_alloc ........................ Passed 0.00 sec
> > Start 2: test_group_setup_assert
> > 2/44 Test #2: test_group_setup_assert ........... Passed 0.00 sec
> >
> > ...
> >
> > 42/44 Test #42: assert_module_test ................ Passed 0.00 sec
> > Start 43: waiter_test_wrap
> > 43/44 Test #43: waiter_test_wrap .................. Passed 0.00 sec
> > Start 44: test_uptime
> > 44/44 Test #44: test_uptime ....................... Passed 0.00 sec
> >
> > 100% tests passed, 0 tests failed out of 44
> >
> > Total Test time (real) = 0.11 sec
> >
> >
> > ... but with the debug options it looks like:
> >
> > `env MODCMAKE_DEBUG=yes make test`
> >
> > 41/44 Test #41: assert_macro_test ................. Passed 0.00 sec
> > test 42
> > Start 42: assert_module_test
> >
> > 42: Test command:
> > /usr/ports/pobj/cmocka-1.1.3/build-amd64/example/assert_module_test
> > 42: Test timeout computed to be: 1500
> > 42: [==========] Running 3 test(s).
> > 42: [ RUN ] increment_value_fail
> > 42: [ ERROR ] --- ASSERT: value
> > 42: [ LINE ] ---
> > /usr/ports/pobj/cmocka-1.1.3/cmocka-1.1.3/example/assert_module.c:30:
> > error: Failure!
> > 42: [ FAILED ] increment_value_fail
> > 42: [ RUN ] increment_value_assert
> > 42: Expected assertion value occurred
> > 42: [ OK ] increment_value_assert
> > 42: [ RUN ] decrement_value_fail
> > 42: Expected assert in decrement_value(NULL)
> > 42: [ ERROR ] --- [ LINE ] ---
> > /usr/ports/pobj/cmocka-1.1.3/cmocka-1.1.3/example/assert_module_test.c:46:
> > error: Failure!
> > 42: [ FAILED ] decrement_value_fail
> > 42: [==========] 3 test(s) run.
> > 42: [ PASSED ] 1 test(s).
> > 42: [ FAILED ] 2 test(s), listed below:
> > 42: [ FAILED ] increment_value_fail
> > 42: [ FAILED ] decrement_value_fail
> > 42:
> > 42: 2 FAILED TEST(S)
> > 42/44 Test #42: assert_module_test ................ Passed 0.00 sec
> > test 43
> > Start 43: waiter_test_wrap
> >
> > 43: Test command:
> > /usr/ports/pobj/cmocka-1.1.3/build-amd64/example/mock/chef_wrap/waiter_test_wrap
> > 43: Test timeout computed to be: 1500
> > 43: [==========] Running 2 test(s).
> > 43: [ RUN ] test_order_hotdog
> > 43: [ OK ] test_order_hotdog
> > 43: [ RUN ] test_bad_dish
> > 43: [ OK ] test_bad_dish
> > 43: [==========] 2 test(s) run.
> > 43: [ PASSED ] 2 test(s).
> > 43/44 Test #43: waiter_test_wrap .................. Passed 0.00 sec
> >
> > That means that also stderr from the test can be seen. It's very helpful to
> > me.
> > You can also override CTEST_FLAGS and use a lot of ctest options to improve
> > the
> > output:
> >
> > https://cmake.org/cmake/help/latest/manual/ctest.1.html
> >
> > What do you say?
> >
> > Best Regards,
> >
> > RS
> >
> >
>
> Tweaked version blow, MODCMAKE_DEBUG was in the wrong scope.
> Objections?
I'm not so keen on MODCMAKE_DEBUG doing double-duty, both changing to
produce a debug build, and increasing verbosity in tests. Should this be
MODCMAKE_VERBOSE or something instead? Or perhaps just enabled by default?
If it's controlled by any variable, it needs a mention in port-modules(5).
> Index: cmake.port.mk
> ===================================================================
> RCS file: /cvs/ports/devel/cmake/cmake.port.mk,v
> retrieving revision 1.65
> diff -u -p -u -p -r1.65 cmake.port.mk
> --- cmake.port.mk 21 Aug 2019 05:47:40 -0000 1.65
> +++ cmake.port.mk 2 Sep 2019 06:22:45 -0000
> @@ -20,14 +20,21 @@ NINJA_FLAGS ?= -v -j ${MAKE_JOBS}
> CONFIGURE_ARGS += -DCMAKE_MAKE_PROGRAM=${NINJA}
> .endif
>
> +MODCMAKE_DEBUG ?= "no"
> +
> .if ${USE_NINJA:L} == "yes" || ${USE_NINJA:L} == "samurai"
> _MODCMAKE_GEN = Ninja
> MODCMAKE_BUILD_TARGET = cd ${WRKBUILD} && exec ${SETENV} ${MAKE_ENV} \
> ${NINJA} ${NINJA_FLAGS} ${ALL_TARGET}
> MODCMAKE_INSTALL_TARGET = cd ${WRKBUILD} && exec ${SETENV} ${MAKE_ENV} \
> ${FAKE_SETUP} ${NINJA} ${NINJA_FLAGS} ${FAKE_TARGET}
> +
> +.if ${MODCMAKE_DEBUG:L} == "yes"
> +CTEST_FLAGS ?= -VV
> +.endif
> +
> MODCMAKE_TEST_TARGET = cd ${WRKBUILD} && exec ${SETENV} ${ALL_TEST_ENV} \
> - ${NINJA} ${NINJA_FLAGS} ${TEST_FLAGS} ${TEST_TARGET}
> + ${LOCALBASE}/bin/ctest -j ${MAKE_JOBS} ${CTEST_FLAGS}
>
> .if !target(do-build)
> do-build:
> @@ -68,11 +75,10 @@ CONFIGURE_ENV += MODJAVA_VER=${MODJAVA_V
> MODTCL_LIB=${MODTCL_LIB} \
> MODTK_LIB=${MODTK_LIB}
>
> -MODCMAKE_DEBUG ?= No
> -
> .if empty(CONFIGURE_STYLE)
> CONFIGURE_STYLE= cmake
> .endif
> +
> MODCMAKE_configure= cd ${WRKBUILD} && ${SETENV} \
> CC="${CC}" CFLAGS="${CFLAGS}" \
> CXX="${CXX}" CXXFLAGS="${CXXFLAGS}" \
> @@ -92,8 +98,6 @@ MODCMAKE_BUILD_SUFFIX = -release.cmake
> SUBST_VARS += MODCMAKE_BUILD_SUFFIX
>
> SEPARATE_BUILD ?= Yes
> -
> -TEST_TARGET ?= test
>
> MODCMAKE_WANTCOLOR ?= No
> MODCMAKE_VERBOSE ?= Yes
>
