On Tue, May 7, 2013 at 7:02 PM, Greg Von Kuster <g...@bx.psu.edu> wrote:
> Hi Peter,
>
> Missing test components implies a tool config that does not define a
> test (i.e, a missing test definition) or a tool config that defines a test,
> but the test's input or output files are missing from the repository.

This seems to be our point of confusion: I don't understand combining
these two categories - it seems unhelpful to me.

Tools missing a test definition clearly can't be tested - but since we'd
like every tool to have tests having this as an easily view listing is
useful both for authors and reviewers. It highlights tools which need
some work - or in some cases work on the Galaxy test framework itself.
They are neither passing nor failing tests - and it makes sense to list
them separately.

Tools with a test definition should be tested - if they are missing an
input or output file this is just a special case of a test failure (and can
be spotted without actually attempting to run the tool). This is clearly
a broken test and the tool author should be able to fix this easily (by
uploading the missing test data file)

> I don't see the benefit of the above where you place tools missing tests
> into a different category than tools with defined tests, but missing test 
> data.
> If any of the test components (test definition or required input or output 
> files)
> are missing, then the test cannot be executed, so defining it as a failing
> test in either case is a bit misleading.  It is actually a tool that is 
> missing
> test components that are required for execution which will result in a pass
> / fail status.

It is still a failing test (just for the trivial reason of missing a
test data file).

> It would be much simpler to change the filter for failing tests to include
> those that are missing test components so that the list of missing test
> components is a subset of the list of failing tests.

What I would like is three lists:

Latest revision: missing tool tests
 - repositories where at least 1 tool has no test defined

[The medium term TO-DO list for the Tool Author]

Latest revision: failing tool tests
 - repositories where at least 1 tool has a failing test (where I include
   tests missing their input or output test data files)

[The priority TO-DO list for the Tool Author]

Latest revision: all tool tests pass
 - repositories where every tool has tests and they all pass

[The good list, Tool Authors should aim to have everything here]

Right now http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus
would appear under both "missing tool tests" and "failing tool tests",
but I hope to fix this and have this under "missing tool tests" only
(until my current roadblocks with the Galaxy Test Framework are
resolved).

I hope I've managed a clearer explanation this time,

Thanks,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to