[galaxy-dev] Enabling iterative test-driven development of tools
Hello,
The current tool testing options provided by Galaxy are excellent for
verifying that Galaxy is functioning properly and that tools provide
reproducible results.
However, I believe there are two related shortcomings to the current
Galaxy approach, and I would like to propose a solution for these
shortcomings. The first is that the workflow for the tool developer is
very clunky, especially if the developer is building up a tool
incrementally with a test-driven development (TDD) approach. For each
parameter addition a whole new output file must be created externally,
manually verified by the developer, possibly converted to a regular
expression, and placed in the test data directory. Put another way, it
is easy to iteratively build up a tool, but not to iteratively build a
test case.
The second point is a bit pedantic. The current approach only verifies
that the output is the same as the supplied output, not that the
output is actually "correct". The typical workflow I described above
relies on manual inspection of the expected output file to verify it
is "correct". I believe it is better to programmatically state
assertions about what makes an output correct than to rely on manual
verification, this serves both to reduce human error and act as
documentation about what makes an output correct.
To address these two points, I propose an extensible addition to the
Galaxy tool syntax (and provide an implementation) to declaratively
make assertions about output files.
The syntax change is to the output child element of test elements.
With this change, the output element may have a child element named
assert_contents, which in turn may have any number of child elements
each of which describes an assertion about the contents of the
referenced output file. The file attribute on the output element is
still required if an assert_contents element is not found, but it is
optional if an assert_contents element is found. The whole file check
described by the file attribute will be executed in addition to the
listed assertions if both are present.
As an example, the following fragment assserts that an output file
contains the text 'chr7', does not contain the text 'chr8', and has a
line matching the regular expression '.*\s+127489808\s+127494553'.
Each potential child element of assert_contents corresponds to a
simple python function. These functions are broken out into modules
which are dynamically (mostly) loaded at test time. The extensibility
of this approach comes from how trivial it is to add new assertion
functions to a module and whole new modules of such functions.
I have started work on three modules of assertion functions, these are
for text, tabular, and XML output files respectively. has_text,
not_has_text, and has_line_matching above are examples of three such
assertion functions from the text module. To see how it works, here is
a function from the file test/base/asserts/text.py defining the
has_line_matching element:
def assert_has_line_matching(output, expression):
""" Asserts the specified output contains a line matching the
regular expression specified by the argument expression."""
match = re.search("^%s$" % expression, output, flags = re.MULTILINE)
assert match != None, "No line matching expression '%s' was found
in output file." % expression
As demonstrated, the function name corresponding to the element
element_name is just assert_element_name. The code that calls these
assertion functions, automatically matches XML attributes with
function arguments by names, and matches an argument named output with
a string containing the contents of the output file resulting from the
test run. Matching function arguments this way gracefully allows for
multiple arguments and optional arguments. There is additional
information about the implementation at the end of this e-mail.
This approach should really aide iterative development of tools. Each
new parameter you add to a tool is going to change the output in some
way, hopefully you will be able to describe how it affects the output
as an assertion. As you add new parameters, the previous parameters
will hopefully affect the output in the same way and the old assertion
will not need to change, you will just need to add new ones. Obviously
this won't always be the case, but hopefully changes to previous
assertions will be minimal over time. I believe this process will be
faster over time than repeatedly producing output files or interactive
GUI based testing, and the final product will be a richer test case.
I have attached two patches. The first patch (implementation.patch) is
the patch that I propose merging into galaxy-central. It modifies the
tool parser to parse these new elements, modifies twilltestcase.py to
perform the assertions, and includes the three modules of assertions
described above. The second patch (examples.patch) adds a data files
to the test-data directory and modifies
Re: [galaxy-dev] galaxy-cloud display data in browser broken
Uploaded data from Biomart and the data is displayed correctly in the middle pane. I deleted all datasets in the history except the first one (a fastq file), and it now displays correctly too! I upload a fasta file: it does not display. Could this be a bug with fasta data? Isabelle -- Isabelle Phan, DPhil Seattle Biomedical Research Institute +1(206)256 7113 > -Original Message- > From: Isabelle Phan > Sent: Wednesday, March 09, 2011 6:21 PM > To: 'galaxy-dev@lists.bx.psu.edu' > Cc: Andy Leonard; Gowthaman Ramasamy > Subject: galaxy-cloud display data in browser broken > > Hello, > > Running galaxy-cloud latest release on amazon EC2, configuration out of the > box > except for: > host = 0.0.0.0 > > I can upload data, run tools, but when I click on the 'eye' icon to view the > data , > the middle panel is empty (accessing galaxy cloud from Firefox and Chrome on > CentOS linux). > > I tried creating a simple fasta file. It shows fine on the right panel, but > won't display > in the middle panel. > View parameters displays correctly. Only the actual data is not displayed. > The log > shows no error: > > 146.129.249.238 - - [10/Mar/2011:02:19:52 +] "GET > /datasets/5969b1f7201f12ae/display/?preview=True HTTP/1.1" 200 - "http://ec2- > xx.compute-1.amazonaws.com:8080/history?show_deleted=False" > "Mozilla/5.0 (X11; Linux i686 (x86_64)) AppleWebKit/534.24 (KHTML, like Gecko) > Chrome/11.0.697.0 Safari/534.24" > > Our local instance of Galaxy does not have this issue, only the cloud > instance. > Since this is a total blocker, any hint is *very* gratefully received, > > Isabelle > > -- > Isabelle Phan, DPhil > Seattle Biomedical Research Institute > +1(206)256 7113 > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] galaxy-cloud display data in browser broken
Hello, Running galaxy-cloud latest release on amazon EC2, configuration out of the box except for: host = 0.0.0.0 I can upload data, run tools, but when I click on the 'eye' icon to view the data , the middle panel is empty (accessing galaxy cloud from Firefox and Chrome on CentOS linux). I tried creating a simple fasta file. It shows fine on the right panel, but won't display in the middle panel. View parameters displays correctly. Only the actual data is not displayed. The log shows no error: 146.129.249.238 - - [10/Mar/2011:02:19:52 +] "GET /datasets/5969b1f7201f12ae/display/?preview=True HTTP/1.1" 200 - "http://ec2-xx.compute-1.amazonaws.com:8080/history?show_deleted=False"; "Mozilla/5.0 (X11; Linux i686 (x86_64)) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.697.0 Safari/534.24" Our local instance of Galaxy does not have this issue, only the cloud instance. Since this is a total blocker, any hint is *very* gratefully received, Isabelle -- Isabelle Phan, DPhil Seattle Biomedical Research Institute +1(206)256 7113 ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-user] irc channel for galaxy
Hi George, Thanks to your suggestion and work by Nate Coraor, Galaxy now has an IRC channel: Server: irc.freenode.net Channel: #galaxyproject This is an informal online gathering place for the Galaxy community to post questions and help each other out. Note that while IRC is conducive to quick discussion, it doesn't work so well as an official support channel (and therefore IRC is not an official Galaxy support channel). If you have a question, bug, or feature suggestion that you want to make sure the Galaxy team knows about, please continue to send these to the mailing lists. Please reply to this thread if you have any questions, or you can post them on the IRC. Thanks, Dave C. On Mon, Mar 7, 2011 at 1:57 PM, George Marselis < george.marse...@kaust.edu.sa> wrote: > Hey guys, > > Is there an irc channel for galaxy? I googled for "galaxy irc" but all I > see in the first fifty results are entries for the samsung galaxy (which > is an excellent phone ;) > > Best Regards, > > George Marselis, systems administrator > Building #2, Level 4, room 4327 > Computational Bioscience Research Center, KAUST > Land: +966-2-808-2944, Mobile: +966-56-321-7714, Skype: project2501a > > > ___ > The Galaxy User list should be used for the discussion of > Galaxy analysis and other features on the public server > at usegalaxy.org. Please keep all replies on the list by > using "reply all" in your mail client. For discussion of > local Galaxy instances and the Galaxy source code, please > use the Galaxy Development list: > > http://lists.bx.psu.edu/listinfo/galaxy-dev > > To manage your subscriptions to this and other Galaxy lists, > please use the interface at: > > http://lists.bx.psu.edu/ > -- http://galaxy.psu.edu/gcc2011/ http://getgalaxy.org http://usegalaxy.org/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] setup.sh missing
Hi Curtis, setup.sh has been removed from the installation process, as discussed in this thread: http://lists.bx.psu.edu/pipermail/galaxy-dev/2011-March/004588.html On Wed, Mar 9, 2011 at 2:12 PM, Curt Palm wrote: > I just cloned the current release version of galaxy: > > hg clone http://bitbucket.org/galaxy/galaxy-dist > > and can not run setup.sh because it is not present in the /galaxy-dist/ > directory. > > setup.sh is also not listed on the > https://bitbucket.org/galaxy/galaxy-dist/src page, > > is this an error or has the installation process changed? > > thanks > > *** > Curtis J. Palm cp...@stanford.edu > Stanford Genome Technology Center > > MC: 8307 > office: 650-812-1994cell: 408 858-7849 > *** > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] setup.sh missing
I just cloned the current release version of galaxy: hg clone http://bitbucket.org/galaxy/galaxy-dist and can not run setup.sh because it is not present in the /galaxy-dist/ directory. setup.sh is also not listed on the https://bitbucket.org/galaxy/galaxy-dist/src page, is this an error or has the installation process changed? thanks *** Curtis J. Palm cp...@stanford.edu Stanford Genome Technology Center MC: 8307 office: 650-812-1994cell: 408 858-7849 *** ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] divide fq into 2
On Tue, Mar 08, 2011 at 11:15:44PM -0500, Musa A. Hassan wrote: > Yes I can't get the file into galaxy at all. Am uploading from a file > path. the file is 35mb. When you say "uploading from a filepath" are you using the administrator-only functionality explained here: https://bitbucket.org/galaxy/galaxy-central/wiki/DataLibraries/UploadingFiles _not_ the 'Get Data' -> 'Upload File' selection from the 'Tools' menu? People work with files _much_ larger than that in Galaxy all the time. > Musa > > From: Ry4an Brase [ry4an+gal...@msi.umn.edu] > Sent: Tuesday, March 08, 2011 11:06 PM > To: Musa A. Hassan > Subject: Re: [galaxy-dev] divide fq into 2 > > On Tue, Mar 08, 2011 at 10:44:51PM -0500, Musa A. Hassan wrote: > > Hi Ry4an, > > > > I'd like to do this in galaxy, but the problem is it wont load into > > galaxy. As for using split, the file generated from this returns a > > length mismatch in say Tophat, maybe in the process of splitting the > > file some changes happen to the format. > > So you can't get the file into galaxy at all? Are you trying to upload > it through your browser (suitable only for non-huge files) or are you > using 'upload from file path'? How big (bytes) is the file. > > Also, you should try to keep your replies on the mailing list so that > other searching in the future find the same help. > > -- > Ry4an Brase 612-626-6575 > Software Developer Application Development > University of Minnesota Supercomputing Institutehttp://www.msi.umn.edu -- Ry4an Brase 612-626-6575 Software Developer Application Development University of Minnesota Supercomputing Institutehttp://www.msi.umn.edu ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Info button
On Wed, Mar 9, 2011 at 10:52 AM, Brad Chapman wrote: > Shaun and Peter; > >> > Hi, I really like the new feature to show dataset information and >> > parameters >> > used to run the tool. Problem is that the link doesn't always work. It >> > seems >> > as though the first time I open a history and click the "i" the information >> > is displayed in the main screen, but after that none of the links work. >> > Hovering over the button gives all the info in a black tool tip but quite >> > often this lingers around even when you take the mouse away. Is there any >> > way to turn off these tips? > > I wrote up a patch that will display this information in the central > panel on a click: > > https://bitbucket.org/galaxy/galaxy-central/issue/466/view-run-details-in-central-panel-instead > > Based on Kanwei's feedback it uses the tooltip in the shared history > view and a click to display in the analysis frame. That sounds like a more robust solution - I've subscribed to the issue. Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Info button
Shaun and Peter; > > Hi, I really like the new feature to show dataset information and parameters > > used to run the tool. Problem is that the link doesn't always work. It seems > > as though the first time I open a history and click the "i" the information > > is displayed in the main screen, but after that none of the links work. > > Hovering over the button gives all the info in a black tool tip but quite > > often this lingers around even when you take the mouse away. Is there any > > way to turn off these tips? I wrote up a patch that will display this information in the central panel on a click: https://bitbucket.org/galaxy/galaxy-central/issue/466/view-run-details-in-central-panel-instead Based on Kanwei's feedback it uses the tooltip in the shared history view and a click to display in the analysis frame. Brad ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Info button
On Wed, Mar 9, 2011 at 10:19 AM, SHAUN WEBB wrote: > > Hi, I really like the new feature to show dataset information and parameters > used to run the tool. Problem is that the link doesn't always work. It seems > as though the first time I open a history and click the "i" the information > is displayed in the main screen, but after that none of the links work. > Hovering over the button gives all the info in a black tool tip but quite > often this lingers around even when you take the mouse away. Is there any > way to turn off these tips? > > I've experienced this in firefox and chrome, I haven't tried in ie. > > Shaun I've noticed similar glitches, mostly the black info popup staying on screen. Mainly testing with Firefox on Mac OS X. Maybe adding a close icon the the corner would be one work around for when it fails to auto-close? Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Info button
Hi, I really like the new feature to show dataset information and parameters used to run the tool. Problem is that the link doesn't always work. It seems as though the first time I open a history and click the "i" the information is displayed in the main screen, but after that none of the links work. Hovering over the button gives all the info in a black tool tip but quite often this lingers around even when you take the mouse away. Is there any way to turn off these tips? I've experienced this in firefox and chrome, I haven't tried in ie. Shaun -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Galaxy process
Hi,
since making the last update I have found some new warnings in my
paster.log, it also seems as though the galaxy process starts to
gather memory and eventually hang (35% of 64G memory).
I've posted the entries below.
If anyone could help me understand what is going on that would be great.
Thanks.
Shaun Webb
paste.httpserver.ThreadPool INFO 2011-03-09 09:49:49,962 No idle
tasks, and only 0 busy tasks; adding 5 more workers
paste.httpserver.ThreadPool INFO 2011-03-09 09:49:58,754 No idle
tasks, and only 4 busy tasks; adding 1 more workers
paste.httpserver.ThreadPool INFO 2011-03-09 09:51:47,301 Culling 6
extra workers (5 idle workers present)
paste.httpserver.ThreadPool INFO 2011-03-09 09:55:17,163 No idle
tasks, and only 0 busy tasks; adding 5 more workers
129.215.14.72 - - [09/Mar/2011:09:48:40 +0100] "GET /history HTTP/1.1"
500 - "http://bifx-core.bio.ed.ac.uk:8080/"; "Mozilla/5.0 (Windows; U;
Windows NT 5.1; en-GB; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13
(.NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET
CLR 3.5.30729)"
paste.httpserver.ThreadPool INFO 2011-03-09 10:13:16,956 Culling 5
extra workers (7 idle workers present)
212.183.140.59 - - [09/Mar/2011:10:13:17 +0100] "GET / HTTP/1.1" 200 -
"-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US)
AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.102 Safari/534.13"
paste.httpserver.ThreadPool INFO 2011-03-09 10:14:35,715 No idle
tasks, and only 2 busy tasks; adding 3 more workers
paste.httpserver.ThreadPool WARNING 2011-03-09 10:15:15,104 Thread
140283224094464 hung (working on task for 3096 seconds)
Exception happened during processing of request from ('212.183.140.59', 10871)
Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 281, in
_handle_request_noblock
self.process_request(request, client_address)
File
"/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 1037, in
process_request
lambda: self.process_request_in_thread(request, client_address))
File
"/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 617, in
add_task
self.kill_hung_threads()
File
"/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 778, in
kill_hung_threads
self.kill_worker(worker.thread_id)
File
"/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py", line 705, in
kill_worker
killthread.async_raise(thread_id, SystemExit)
File
"/storage/home/galaxy/galaxy_dist/eggs/Paste-1.6-py2.6.egg/paste/util/killthread.py", line 22, in
async_raise
raise ValueError("invalid thread id")
ValueError: invalid thread id
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
