Hello all,

After 7 enjoyable hours of cheetah debugging ...
for those (like me) who are not well versed in python or cheetah templates, 
here are couple of debugging tips that can help.
        
=1= RTFM.
It's here:
http://www.cheetahtemplate.org/docs/users_guide_html/users_guide.html


=2= printing something to STDERR, to show the progress of the cheetah template 
compilation

To run python one-liners in your cheetech template, use "#silent".
Assuming this is my Galaxy XML tool:
=============
<tool id="cshl_bowtie_se1" name="Bowtie" version="0.1.3" >
<command interpreter="sh">bowtie_se1.sh
"$output"
"$input1"

#if str($report_aligned_fastq) == "True"
        "$output_aligned_fastq_2"
#else
        "no-al"
#end if

#if str($report_nonmappers_fastq) == "True"
        "$output_nonmappers_fastq_2"
#else
        "no-un"
#end if

#if str($report_maxmappers_fastq) == "True"
        "$output_maxmappers_fastq_1"
#else
        "no-max"
#end if

#if $mismatches_strategy.choice == "endtoend"
        -v $mismatches_strategy.end_to_end_limit
#else if $mismatches_strategy.choice == "seed"
        --seedlen $mismatches_strategy.seedlen
        --seedmms $mismatches_strategy.seedmms
        $mismatches_strategy.best
        $mismatches_strategy.strata
#end if

#if $multi_mappers_strategy.choice == "m_and_k"
        -k $multi_mappers_strategy.max_mapping
        -m $multi_mappers_strategy.max_mapping
#else if $multi_mappers_strategy.choice == "unique"
        -k 1
        -m 1
#else if $multi_mappers_strategy.choice == "all"
        -a
#else if $multi_mappers_strategy.choice == "random_one":
        -k 1
#end if

#if $input1.ext == "fastqillumina":
        --solexa1.3-quals
#else if $input1.ext == "fasta":
        -f
#else if $input1.ext == "fastqsolexa":
        --solexa-qual
#end if

[ and so on and on and on ...]
=============

Then to do something directly in python, just put "#silent" followed a python 
command.
The most useful command is to print something to STDERR, so it will appear on 
the screen
when you execute the tool. The "#silent" means it will not be part of the 
generated command line.

Example:

==========
...
...
<command interpreter="sh">bowtie_se1.sh
"$output"
"$input1"

#silent  sys.stderr.write("Hello World from cheetah generated code\n");

#if str($report_aligned_fastq) == "True"
        "$output_aligned_fastq_2"
#else
        "no-al"
#end if

...
...
===========

When the tool is executed (from inside Galaxy), and the job is prepared, and 
the command line is built,
the cheetah template will be filled, and the message will be printed to STDERR.





=3= Printing the value of a galaxy parameter
Same "#silent" as before, but add the variable as a format parameter:

===
#silent  sys.stderr.write(" Execute by user: %s\n" % ( $userEmail ) )
===

NOTE:
there are several levels of parameters interpolation going on here. 
"$userEmail" is a cheetah variable (obviously, because "$" doesn't mean 
anything in python).
It is first replaced by cheetah, then executed by python, getting this to work 
is sometimes tricky.


If you have some elabrate conditional/repeat/group parameters, it could look 
like this:
=======
#silent  sys.stderr.write(" mismatch strategy parameters: %s\n" % ( str( 
$mismatch_strategy ) ) )
=======

And the output would look like this (on the terminal showing Galaxy's log 
output):
=========
mismatches_strategy = {'seedmms': <galaxy.tools.InputValueWrapper object at 
0x203adfd0>, 'strata': <galaxy.tools.InputValueWrapper object at 0x203ad150>, 
'choice': 'seed', 'seedlen': <galaxy.tools.InputValueWrapper object at 
0x203adc90>, '__current_case__': 1, 'best': <galaxy.tools.InputValueWrapper 
object at 0x203ad590>}
=========

This assumes that you have a valid XML <param> named "mismatch_strategry",
which in my case was defined as:
======
<conditional name="mismatches_strategy" >
<param name="choice" type="select" label="Mismatches strategy">
        <option value="seed">Seed/Extent</option>
        <option value="endtoend">End-to-End mismatches limit (-v)</option>
        <option value="custom">Use custom options (below)</option>
</param>

<when value="endtoend">
        <param name="end_to_end_limit" type="integer" size="2" value="0" 
label="Max. number of mismatches allowed in the entire read (-v)" help="" />
</when>

<when value="seed">
        <param name="seedlen" type="integer" size="4" value="28" label="Seed 
length (--seendlen)" help="Length of the seed fragment in the read. 28 is a 
good choice for smallRNAs. Use larger values for long reads of DNA/RNA." />
        <param name="seedmms" type="integer" size="4" value="3" label="Allows 
mismatches in the seed (--seedmms)" help="How many mismatches are allowed in 
the seed fragment of the read. Enter values between 0 and 3." />
        <param name="best" type="boolean" truevalue="--best" falsevalue="" 
checked="True" label="Report best stratum (--best)" help="hits guaranteed best 
stratum; ties broken by quality" />
        <param name="strata" type="boolean" truevalue="--strata" falsevalue="" 
checked="false" label="Supress hits in sub-optimal strata (--strata)" 
help="hits in sub-optimal strata aren't reported (requires --best, -m, -k"/>
</when>

<when value="custom">
</when>
</conditional>
======




=4= Show all available parameters in the cheetah template

The safe way that I've found is:
========
#silent sys.stderr.write("!!!! Cheetah Template Variables !!!!\n")
#silent sys.stderr.write(" searchList = '%s'\n" % (str($searchList)) )
#silent sys.stderr.write("!!!! end-of-list !!!!\n")
========
"$searchList" is the cheetah variable containing all the templated variables.

The output will be ugly, but should always work. example:
==============
!!!! Cheetah Template Variables !!!!
 searchList = '[{}, 
<cheetah_DynamicallyCompiledCheetahTemplate_1301603121_44_70792.DynamicallyCompiledCheetahTemplate
 object at 0x6b70c50>, {'input2': <galaxy.tools.DatasetFilenameWrapper object 
at 0x6b76fd0>, 'userEmail': 'gor...@cshl.edu', 'input1': 
<galaxy.tools.DatasetFilenameWrapper object at 0x69902d0>, 
'user_custom_options': <galaxy.tools.InputValueWrapper object at 0x6b768d0>, 
'userId': '2', 'output_maxmappers_fastq_1': 
<galaxy.tools.DatasetFilenameWrapper object at 0x2aaaacd33b90>, 'dbkey': 'mm9', 
'output_maxmappers_fastq_2': <galaxy.tools.DatasetFilenameWrapper object at 
0x62e5890>, 'multi_mappers_strategy': {'__current_case__': 2, 'choice': 
'unique'}, 'GALAXY_DATA_INDEX_DIR': 
'/localdata1/gordon/projects/galaxy_dev/tool-data', 
'GALAXY_DATATYPES_CONF_FILE': 
'/localdata1/gordon/projects/galaxy_dev/datatypes_conf.xml', 'min_insertion': 
<galaxy.tools.InputValueWrapper object at 0x62e1850>, 
'report_nonmappers_fastq': <galaxy.tools.InputValueWrapper object at 0x6b76490>
, 'mismatches_strategy': {'seedmms': <galaxy.tools.InputValueWrapper object at 
0x6b76710>, 'strata': <galaxy.tools.InputValueWrapper object at 0x6b764d0>, 
'choice': 'seed', 'seedlen': <galaxy.tools.InputValueWrapper object at 
0x6b76d90>, '__current_case__': 1, 'best': <galaxy.tools.InputValueWrapper 
object at 0x6b76d10>}, 'output_nonmappers_fastq_1': 
<galaxy.tools.DatasetFilenameWrapper object at 0x2aaaacd33b10>, 
'report_aligned_fastq': <galaxy.tools.InputValueWrapper object at 0x6b76a10>, 
'__new_file_path__': '/localdata1/gordon/projects/galaxy_dev/database/tmp', 
'output_nonmappers_fastq_2': <galaxy.tools.DatasetFilenameWrapper object at 
0x2aaaacd33610>, 'database': {'source': 'fromlist', 'databasepath': 
<galaxy.tools.SelectToolParameterWrapper object at 0x62e1c10>, 
'__current_case__': 1}, 'report_maxmappers_fastq': 
<galaxy.tools.InputValueWrapper object at 0x6b76790>, 'max_insertion': 
<galaxy.tools.InputValueWrapper object at 0x62e1450>, 'suppress_non_mappers': 
<galaxy.tool
s.InputValueWrapper object at 0x6b76410>, 'output_aligned_fastq_2': 
<galaxy.tools.DatasetFilenameWrapper object at 0x6b6fa10>, 
'output_aligned_fastq_1': <galaxy.tools.DatasetFilenameWrapper object at 
0x6ab15d0>, 'GALAXY_ROOT_DIR': '/localdata1/gordon/projects/galaxy_dev', 
'output': <galaxy.tools.DatasetFilenameWrapper object at 0x6b76a90>, 
'chromInfo': 
'/localdata1/gordon/projects/galaxy_dev/tool-data/shared/ucsc/chrom/mm9.len', 
'__app__': <galaxy.tools.RawObjectWrapper object at 0x699c1d0>}]'
!!!! end-of-list !!!!
==============


A better way (but I'm not sure it's generic enough, or if it's specific to 
Galaxy, python gurus please chime in):
=====
#silent sys.stderr.write("!!!! Cheetah Template Variables !!!!\n")
#for k,v in $searchList[2].items()
#silent   sys.stderr.write(" %s = %s\n" % (str(k), str(v) ))
#end for
#silent sys.stderr.write("!!!! end-of-list !!!!\n")
=====

The output in this case will be:
========
!!!! Cheetah Template Variables !!!!
 input2 = 
/localdata1/gordon/projects/galaxy_dev/database/files/000/dataset_251.dat
 userEmail = gor...@cshl.edu
 input1 = 
/localdata1/gordon/projects/galaxy_dev/database/files/000/dataset_183.dat
 user_custom_options = 
 userId = 2
 output_maxmappers_fastq_1 = 
/localdata1/gordon/projects/galaxy_dev/database/files/000/dataset_533.dat
 dbkey = mm9
 output_maxmappers_fastq_2 = 
/localdata1/gordon/projects/galaxy_dev/database/files/000/dataset_534.dat
 multi_mappers_strategy = {'__current_case__': 2, 'choice': 'unique'}
 GALAXY_DATA_INDEX_DIR = /localdata1/gordon/projects/galaxy_dev/tool-data
 GALAXY_DATATYPES_CONF_FILE = 
/localdata1/gordon/projects/galaxy_dev/datatypes_conf.xml
 min_insertion = 0
 report_nonmappers_fastq = True
 mismatches_strategy = {'seedmms': <galaxy.tools.InputValueWrapper object at 
0x6a95b10>, 'strata': <galaxy.tools.InputValueWrapper object at 0x6a95e90>, 
'choice': 'seed', 'seedlen': <galaxy.tools.InputValueWrapper object at 
0x6a95390>, '__current_case__': 1, 'best': <galaxy.tools.InputValueWrapper 
object at 0x6a95210>}
 output_nonmappers_fastq_1 = 
/localdata1/gordon/projects/galaxy_dev/database/files/000/dataset_531.dat
 report_aligned_fastq = True
 __new_file_path__ = /localdata1/gordon/projects/galaxy_dev/database/tmp
 output_nonmappers_fastq_2 = 
/localdata1/gordon/projects/galaxy_dev/database/files/000/dataset_532.dat
 database = {'source': 'fromlist', 'databasepath': 
<galaxy.tools.SelectToolParameterWrapper object at 0x6a95350>, 
'__current_case__': 1}
 report_maxmappers_fastq = True
 max_insertion = 250
 suppress_non_mappers = --quiet
 output_aligned_fastq_2 = 
/localdata1/gordon/projects/galaxy_dev/database/files/000/dataset_530.dat
 output_aligned_fastq_1 = 
/localdata1/gordon/projects/galaxy_dev/database/files/000/dataset_529.dat
 GALAXY_ROOT_DIR = /localdata1/gordon/projects/galaxy_dev
 output = 
/localdata1/gordon/projects/galaxy_dev/database/files/000/dataset_528.dat
 chromInfo = 
/localdata1/gordon/projects/galaxy_dev/tool-data/shared/ucsc/chrom/mm9.len
 __app__ = galaxy.app:UniverseApplication
!!!! end-of-list !!!!
========

Who knew one can access "GALAXY_ROOT_DIR" as a tool parameter ? (execpt for the 
galaxy developers, of course... :) )




=5= Divide-and-conquer a long template
If you have strange cheetah compilation problems and can't understand what's 
wrong (sorry, python/cheetah code makes no sense to me...),
use "#breakpoint" to stop the compilation at any given point in the template.

If the cheetah template compiled without error, then all the statements above 
your "#breakpoint" are valid,
and you can move the "#breakpoint" to a further point in the template until you 
pin-point the offending statement.

The tool will obviously not run successfully without the entire command line, 
but we're debugging the template here, not the tool.


=6= The hazards of an "$input" parameter

With most other parameters, a typo in a variable name will produce a helpful 
cheetah error message.
Exmaple:
=========
<tool id="cshl_dummy" name="dummy" description="" >
<command interpreter="sh">echo $bar</command>
        <inputs>
                <param type="integer" name="foo" size="2" value="42"/>
        </inputs>
        <outputs>
                <data format="txt" name="output" />
        </outputs>
</tool>
=========

Since there's no "$bar" parameter (the name is "$foo"), the cheetah error will 
be:
========
Traceback (most recent call last):
  File 
"/localdata1/gordon/projects/galaxy_dev/lib/galaxy/jobs/runners/drmaa.py", line 
127, in queue_job
    job_wrapper.prepare()
  File "/localdata1/gordon/projects/galaxy_dev/lib/galaxy/jobs/__init__.py", 
line 371, in prepare
    self.command_line = self.tool.build_command_line( param_dict )
  File "/localdata1/gordon/projects/galaxy_dev/lib/galaxy/tools/__init__.py", 
line 1527, in build_command_line
    command_line = fill_template( self.command, context=param_dict )
  File "/localdata1/gordon/projects/galaxy_dev/lib/galaxy/util/template.py", 
line 9, in fill_template
    return str( Template( source=template_text, searchList=[context] ) )
  File 
"/localdata1/gordon/projects/galaxy_dev/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg/Cheetah/Template.py",
 line 1004, in __str__
    return getattr(self, mainMethName)()
  File "cheetah_DynamicallyCompiledCheetahTemplate_1301604373_52_56153.py", 
line 83, in respond
NotFound: cannot find 'bar'
========
Helpful enough, even without a line number or code fragment.


But if your typo is "$input" (and your parameters are named "$input1" or 
"$input2" or similar), the error will completely useless.

Example: The following bug (using "$input" when there's no "$input" parameter 
defined):
==========
<tool id="cshl_dummy" name="dummy" description="" >
<command interpreter="sh">echo $input</command>
        <inputs>
                <param type="integer" name="foo" size="2" value="42"/>
        </inputs>
        <outputs>
                <data format="txt" name="output" />
        </outputs>
</tool>
===========

Will throw the following exception:
===========
Traceback (most recent call last):
  File 
"/localdata1/gordon/projects/galaxy_dev/lib/galaxy/jobs/runners/drmaa.py", line 
127, in queue_job
    job_wrapper.prepare()
  File "/localdata1/gordon/projects/galaxy_dev/lib/galaxy/jobs/__init__.py", 
line 371, in prepare
    self.command_line = self.tool.build_command_line( param_dict )
  File "/localdata1/gordon/projects/galaxy_dev/lib/galaxy/tools/__init__.py", 
line 1527, in build_command_line
    command_line = fill_template( self.command, context=param_dict )
  File "/localdata1/gordon/projects/galaxy_dev/lib/galaxy/util/template.py", 
line 9, in fill_template
    return str( Template( source=template_text, searchList=[context] ) )
  File 
"/localdata1/gordon/projects/galaxy_dev/eggs/Cheetah-2.2.2-py2.6-linux-x86_64-ucs4.egg/Cheetah/Template.py",
 line 1004, in __str__
    return getattr(self, mainMethName)()
  File "cheetah_DynamicallyCompiledCheetahTemplate_1301604408_95_71749.py", 
line 83, in respond
  File "<string>", line 0
     
    ^
 SyntaxError: unexpected EOF while parsing
=============

Probably because there is a hidden "input" parameter in galaxy code that is 
used in the output format detection code, but I haven't found the exact spot.

This 'bug' alone took me two hours to solve :(



Hope this will help to others in future debuggings...

-gordon
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Reply via email to