On Thu, Jan 8, 2015 at 3:27 AM, Jeltje van Baren
<jeltje.van.ba...@gmail.com> wrote:
> Hi,
>
> I'm trying to find out what the best way is to see if my functional tests
> work as expected, and I'm also wondering when these tests are (supposed to
> be) run. Here are the details:
>
> Following instructions at
> https://wiki.galaxyproject.org/Admin/Tools/AddToolTutorial, I created a
> little toolExample tool under galaxy-dist/tools/myTools. I also created test
> input and output in a test-data directory and added a test to the
> toolExample.xml file:
>   <tests>
>     <test>
>       <param name="input" value="testinput.fa"/>
>       <output name="out_file1" file="testoutput.txt"/>
>     </test>
>   </tests>
>
> And yes, the files are named correctly:
> tools/myTools/test-data/testinput.fa
> tools/myTools/test-data/testoutput.txt

If you are manually installing the tool directly under the folder
galaxy-dist/tools/myTools then you should put your test
files under galaxy-dist/test-data (this is how all the tools
used to work back before the ToolShed).

If you want to do the tool installation via a ToolShed,
things are different (more complicated for the tool
development side, but it becomes easier for the end
user to install things).

> I added the new tool to config/tools_conf.xml AND to
> config/tools_conf.xml.sample.
> Question #1: Why is a .sample file used by run_functional_tests.sh? I
> thought with .sample extensions were meant as examples for creating your own
> versions, I'm really surprised that any script would want to access these.

Historically (for reasons unclear to me), the tools for testing
were listed in galaxy-dist/tools_conf.xml.sample however
that was changed (about a year ago? I forget now) to simply
use galaxy-dist/tools_conf.xml (the master tool listing as used
before the ToolShed existed).

You might be using different settings in your main ini file,
but on our setup it is galaxy-dist/tools_conf.xml and not
galaxy-dist/config/tools_conf.xml (which seems to be in
use on your system given the new tool showed up on
restarting Galaxy):

> I restarted galaxy to verify that the new tool showed up,
> then checked that I could test it:
>
> ./run_functional_tests.sh -list | grep fa_gc_content_1
> It shows up in the list of tools. So then I tested it:
>
> ./run_functional_tests.sh -id fa_gc_content_1

Good - although run_functional_tests.sh is now obsolete
and just a backward compatible alias for run_tests.sh.

If you read this in some (out of date) documentation,
please say where so we can update it.

> This gives
> IOError: [Errno 2] No such file or directory:
> '/inside/home/jeltje/exp/varscan2/programs/galaxy-dist/test-data/testinput.fa'
>
> So it looks like at some point the testinput should have been copied to
> galaxy-dist/test-data but wasn't.

No, as noted above, you have to put the test files there
by hand (as long as you are deliberately doing all your
tool installation/development avoiding the ToolShed,
which is quite a common approach if you have a
separate Galaxy instance for development).

> Question # 2: Is there some setting that makes this test input (and output)
> get copied to the correct location, or do I have to do this manually?

No, unless you go down the ToolShed route, do this manually.

> I also tried testing by uploading the tool to the testtoolshed (as jtest1).
> I deliberatly put the wrong value in the testoutput.txt file because I'd
> like to see a test fail before I'll accept that it succeeds. It appears the
> testinput does not get tested on upload, because I got no automated test
> results, and no error messages anywhere. Is that expected behavior? I know
> the tools get tested every so often, I'm just surprised it doesn't happen on
> upload.
> The tool could then also be easily installed to my galaxy distribution,
> without errors in my log or on the browser pages.

The main and Test Tool Shed should now be running the tests
every 48 hours or so - this service had been unavailable over
the Christmas/New Year period but should be OK now.

> Question #3: At what point in creating or downloading a repository
> are these tests run?

Never, assuming I understand your question.

If you mean installing a tool from the ToolShed into a Galaxy
instance via the Galaxy administrator options, then I think
after installation any tests should be run automatically.

> And lastly, #4: is either of these the best way to test my tests, or did I
> overlook something?
>
> Thank you!
>
> -Jeltje

While developing a new tool (or improving one), I personally
use a development Galaxy server (not the one our users run
all their real jobs on), and install the tools manually (as above),
and then test via run_tests.sh -id my_tool_id.

This is complemented by a TravisCI setup, see:
http://blastedbio.blogspot.jp/2013/09/using-travis-ci-for-testing-galaxy-tools.html

If that works, I upload the tool to the TestToolShed, and wait
for those test results. If that all works and I am ready to make
a public release, then I upload to the main ToolShed.

I talked about this (in the context of the NCBI BLAST+
wrappers) at the GCC2014 conference, slides here:
http://www.slideshare.net/pjacock/galaxy-blast-gcc2014

There is even a video of my talk (and most of the others)
on the Galaxy website here:
https://wiki.galaxyproject.org/Events/GCC2014/Abstracts

I hope that helped. There is a work in progress
"Best Practices" being worked on here:
https://wiki.galaxyproject.org/Tools/BestPractices

Regards,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to