Re: [lttng-dev] babeltrace tests (master/2.0)

[Nathan Lynch]

Philippe Proulx  writes:

> On Fri, Sep 15, 2017 at 3:53 PM, Nathan Lynch  wrote:
>> Nathan Lynch  writes:
>>> Philippe Proulx  writes:

 I asked the Automake mailing list about this
  and
 we will most probably implement the suggested solution, which is to have
 individual top-level (in tests/) `check-X` targets which rerun Make with
 specific sets of tests.
>>
>> I should be able to implement this over the next week or so if you don't
>> anticipate conflicts with other ongoing work.  What do you say?
>
> It's been merged into master a few minutes ago. Here's the commit:
>
> 
> https://github.com/efficios/babeltrace/commit/cd1c5347f7583a20ece5a1a62fc0e0eddb82322b
>

Thanks!

Is this the desired result:

PASS: cli/test_trace_read
PASS: cli/test_packet_seq_num
PASS: cli/test_convert_args  
PASS: cli/intersection/test_intersection 
PASS: cli/test_trace_copy
PASS: cli/test_trimmer   
PASS: lib/test_bitfield  
PASS: lib/test_ctf_writer_complete   
PASS: lib/test_bt_values 
PASS: lib/test_ctf_ir_ref
PASS: lib/test_bt_ctf_field_type_validation  
PASS: lib/test_ir_visit  
PASS: lib/test_bt_notification_heap  
PASS: lib/test_graph_topo
PASS: lib/test_cc_prio_map   
PASS: lib/test_bt_notification_iterator  
PASS: lib/test_plugin_complete   
PASS: plugins/test-utils-muxer-complete  
PASS: plugins/test_dwarf_complete
PASS: plugins/test_bin_info_complete 

  
Testsuite summary for babeltrace 2.0.0-pre1  

  
# TOTAL: 20  
# PASS:  20  
# SKIP:  0   
# XFAIL: 0   
# FAIL:  0   
# XPASS: 0   
# ERROR: 0   


The granularity of results seems very much reduced; there had been
~1200 individual tests run before.
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] babeltrace tests (master/2.0)

[Nathan Lynch]

Nathan Lynch  writes:
> Philippe Proulx  writes:
>>
>> I asked the Automake mailing list about this
>>  and
>> we will most probably implement the suggested solution, which is to have
>> individual top-level (in tests/) `check-X` targets which rerun Make with
>> specific sets of tests.

I should be able to implement this over the next week or so if you don't
anticipate conflicts with other ongoing work.  What do you say?
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] babeltrace tests (master/2.0)

[Philippe Proulx]

On Fri, Sep 15, 2017 at 3:53 PM, Nathan Lynch  wrote:
> Nathan Lynch  writes:
>> Philippe Proulx  writes:
>>>
>>> I asked the Automake mailing list about this
>>>  and
>>> we will most probably implement the suggested solution, which is to have
>>> individual top-level (in tests/) `check-X` targets which rerun Make with
>>> specific sets of tests.
>
> I should be able to implement this over the next week or so if you don't
> anticipate conflicts with other ongoing work.  What do you say?

It's been merged into master a few minutes ago. Here's the commit:


https://github.com/efficios/babeltrace/commit/cd1c5347f7583a20ece5a1a62fc0e0eddb82322b

Philippe Proulx
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] babeltrace tests (master/2.0)

[Nathan Lynch]

Philippe Proulx  writes:

> On Thu, Aug 10, 2017 at 5:40 PM, Nathan Lynch  wrote:
>>
>> Ping?
>>
>
> Sorry. We looked into this. There seems to be no easy way with Automake
> to have individual `check` targets in subdirectories and make the
> top-level `check` target recurse without stopping when a subdirectory
> fails.
>
> As you wrote, it is possible with --keep-going, but this is not common
> practice and we don't expect our users to read make(1) to be able to run
> all the tests unconditionally.
>
> I asked the Automake mailing list about this
>  and
> we will most probably implement the suggested solution, which is to have
> individual top-level (in tests/) `check-X` targets which rerun Make with
> specific sets of tests.
>
> Thank you for reporting this by the way.


Thanks for following up.  A well-functioning test suite is a really
useful guide for occasional contributors like me :-)
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] babeltrace tests (master/2.0)

[Philippe Proulx]

On Thu, Aug 10, 2017 at 5:40 PM, Nathan Lynch  wrote:
>
> Ping?
>

Sorry. We looked into this. There seems to be no easy way with Automake
to have individual `check` targets in subdirectories and make the
top-level `check` target recurse without stopping when a subdirectory
fails.

As you wrote, it is possible with --keep-going, but this is not common
practice and we don't expect our users to read make(1) to be able to run
all the tests unconditionally.

I asked the Automake mailing list about this
 and
we will most probably implement the suggested solution, which is to have
individual top-level (in tests/) `check-X` targets which rerun Make with
specific sets of tests.

Thank you for reporting this by the way.

Phil

>
> Nathan Lynch  writes:
>> How is the babeltrace test suite on the master and stable-2.0 branches
>> intended to be used?
>>
>> Effectively there are four test suites, one for each definition of TESTS
>> in the tests/ directory.  If one of them has a failure, 'make check'
>> will not proceed to the next unless the -k/--keep-going flag is used.
>> Is this intended?  I unwittingly was running only the tests under
>> tests/cli (some of which always fail) until I discovered this.
>>
>> Defining multiple test suites this way seems to be unusual; other
>> projects I've checked just have one top-level definition of TESTS.
>>
>> I'm aware of the rationale for doing it this way:
>>
>> commit 6ca1931cb32ca2eb33252896d2a42a4c48af436a
>> Author: Philippe Proulx 
>> Date:   Fri May 5 16:45:11 2017 -0400
>>
>> tests: put TESTS list in each Makefile.am
>>
>> So that you can do `make check` in any subdirectory to run only those
>> tests.
>>
>> But it still feels non-idiomatic to me, and I'm wondering if there's a
>> better way.  There are other ways to limit the set of tests to run,
>> detailed here:
>>
>> https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html
>>
>> Here are my current results from running all test suites; let me know if
>> you want more detail and I'll follow up; there are likely several
>> different issues to investigate.
>>
>> $ find tests/ -name test-suite.log -exec head {} +
>> ==> tests/lib/test-suite.log <==
>> =
>>babeltrace 2.0.0-pre1: tests/lib/test-suite.log
>> =
>>
>> # TOTAL: 1200
>> # PASS:  1195
>> # SKIP:  0
>> # XFAIL: 0
>> # FAIL:  1
>> # XPASS: 0
>>
>> ==> tests/bindings/python/bt2/test-suite.log <==
>> =
>>babeltrace 2.0.0-pre1: tests/bindings/python/bt2/test-suite.log
>> =
>>
>> # TOTAL: 11152
>> # PASS:  0
>> # SKIP:  0
>> # XFAIL: 0
>> # FAIL:  11151
>> # XPASS: 0
>>
>> ==> tests/cli/test-suite.log <==
>> =
>>babeltrace 2.0.0-pre1: tests/cli/test-suite.log
>> =
>>
>> # TOTAL: 153
>> # PASS:  143
>> # SKIP:  0
>> # XFAIL: 0
>> # FAIL:  7
>> # XPASS: 0
>>
>> ==> tests/plugins/test-suite.log <==
>> =
>>babeltrace 2.0.0-pre1: tests/plugins/test-suite.log
>> =
>>
>> # TOTAL: 2
>> # PASS:  0
>> # SKIP:  0
>> # XFAIL: 0
>> # FAIL:  0
>> # XPASS: 0
>>
>> (BTW, since the tests don't pass/skip/xfail 100%, 'make distcheck' is 
>> failing.)
> ___
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] babeltrace tests (master/2.0)

[Nathan Lynch]

Ping?


Nathan Lynch  writes:
> How is the babeltrace test suite on the master and stable-2.0 branches
> intended to be used?
>
> Effectively there are four test suites, one for each definition of TESTS
> in the tests/ directory.  If one of them has a failure, 'make check'
> will not proceed to the next unless the -k/--keep-going flag is used.
> Is this intended?  I unwittingly was running only the tests under
> tests/cli (some of which always fail) until I discovered this.
>
> Defining multiple test suites this way seems to be unusual; other
> projects I've checked just have one top-level definition of TESTS.
>
> I'm aware of the rationale for doing it this way:
>
> commit 6ca1931cb32ca2eb33252896d2a42a4c48af436a
> Author: Philippe Proulx 
> Date:   Fri May 5 16:45:11 2017 -0400
>
> tests: put TESTS list in each Makefile.am
> 
> So that you can do `make check` in any subdirectory to run only those
> tests.
>
> But it still feels non-idiomatic to me, and I'm wondering if there's a
> better way.  There are other ways to limit the set of tests to run,
> detailed here:
>
> https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html
>
> Here are my current results from running all test suites; let me know if
> you want more detail and I'll follow up; there are likely several
> different issues to investigate.
>
> $ find tests/ -name test-suite.log -exec head {} +  
> ==> tests/lib/test-suite.log <==
> =
>babeltrace 2.0.0-pre1: tests/lib/test-suite.log
> =
>
> # TOTAL: 1200
> # PASS:  1195
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  1
> # XPASS: 0
>
> ==> tests/bindings/python/bt2/test-suite.log <==
> =
>babeltrace 2.0.0-pre1: tests/bindings/python/bt2/test-suite.log
> =
>
> # TOTAL: 11152
> # PASS:  0
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  11151
> # XPASS: 0
>
> ==> tests/cli/test-suite.log <==
> =
>babeltrace 2.0.0-pre1: tests/cli/test-suite.log
> =
>
> # TOTAL: 153
> # PASS:  143
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  7
> # XPASS: 0
>
> ==> tests/plugins/test-suite.log <==
> =
>babeltrace 2.0.0-pre1: tests/plugins/test-suite.log
> =
>
> # TOTAL: 2
> # PASS:  0
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  0
> # XPASS: 0
>
> (BTW, since the tests don't pass/skip/xfail 100%, 'make distcheck' is 
> failing.)
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev