Re: [galaxy-dev] pipeline execution succeeds but galaxy shows failure
Hi Brian,
A couple of days ago, smcmanus https://bitbucket.org/smcmanus pushed the
following change to the repo:
Tools can now specify their own handling of stderr and stdout regular
expressions as well as exit code ranges.
https://bitbucket.org/galaxy/galaxy-central/issue/325/allow-tool-authors-to-decide-whether-to-use-return-codes-or-stderr-for-detecting-job
It looks like the documentation has yet to be written.
Hope this helps,
Nicole
On Fri, Jul 20, 2012 at 12:46 PM, Brad Chapman chapm...@50mail.com wrote:
Brian;
I wrote a pipeline (xml attached) that, from what I can gather,
succeeds, but galaxy shows it as an error and doesn't make the output
file accessible as a new data set.
Is it possible the software is writing to standard error? Galaxy doesn't
check status codes, but rather check for stderr and assumes that output
indicates a problem. You can wrap the problematic programs with a little
script to eat up stderr and check that everything is okay:
http://wiki.g2.bx.psu.edu/Future/Job%20Failure%20When%20stderr
Brad
From the server log, I can see that the command line is being
constructed correctly, and it even indicates that it's captured the
output, but in the display of the web browser, it just shows up in the
error state. The script being run exits (0) on success. Any ideas?
Here's what the output section of my xml file looks like:
outputs
data format=bam name=coordSortedBam label=${tool.name}
on ${on_string}: coord-sorted read alignments
from_work_dir=alignment/alignment.coordSorted.bam/
/outputs
and here's what the server log states:
galaxy.jobs.handler INFO 2012-07-20 09:52:05,240 (30) Job dispatched
galaxy.jobs.runners.local DEBUG 2012-07-20 09:52:05,453 executing:
alignReads.pl --target
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_26.dat
-o alignment --aligner bowtie --single
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_23.dat
--seqType fq
galaxy.jobs DEBUG 2012-07-20 09:52:16,673 finish(): Moved
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/job_working_directory/000/30/alignment/alignment.coordSorted.bam
to
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_50.dat
as directed by from_work_dir
Again, as far as I can tell, everything worked - but the browser
doesn't think so.
I've run the exact command above on the command-line, and it exits(0)
indicating success.
Also, I've verified that when run through my galaxy instance, the
galaxy-relocated output file is as expected.
Many thanks for your help. I'm still getting my feet wet with
galaxy, reading through all the documentation and searching the
mailing list for additional help.
best regards,
-brian
--
--
Brian J. Haas
Manager, Genome Annotation and Analysis, Research and Development
The Broad Institute
http://broad.mit.edu/~bhaas
tool id=alignreads name=alignReads version=0.0.1
descriptionalignReads: short read alignment tool
wrapper/description
requirements
requirement type=packagetrinity/requirement
/requirements
command
alignReads.pl --target $target -o alignment --aligner
$aligner_selection.aligner
## Inputs.
#if str($inputs.paired_or_single) == paired:
--left $inputs.left_input --right $inputs.right_input
#if $inputs.left_input.ext == 'fa':
--seqType fa
#else:
--seqType fq
#end if
#if str($inputs.library_type) != None:
--SS_lib_type $inputs.library_type
#end if
--max_dist_between_pairs
$inputs.max_dist_between_pairs
#else:
--single $inputs.input
#if str($inputs.input.ext) == 'fa':
--seqType fa
#else:
--seqType fq
#end if
#if str($inputs.library_type) != None:
--SS_lib_type $inputs.library_type
#end if
#end if
## Additional parameters.
##if str($inputs.use_additional) == yes:
## -- $inputs.additional_params
##end if
## direct to output
## $trinity_log 2amp;1
/command
inputs
param format=fasta name=target type=data
label=target help=Fasta sequences targeted for short-read alignment /
conditional name=inputs
param name=paired_or_single type=select
label=Paired or Single-end data?
option value=pairedPaired/option
option value=singleSingle/option
/param
when value=paired
param format=fasta,fastq name=left_input
type=data label=Left/Forward strand reads help=/
param format=fasta,fastq
[galaxy-dev] pipeline execution succeeds but galaxy shows failure
Hi,
I wrote a pipeline (xml attached) that, from what I can gather,
succeeds, but galaxy shows it as an error and doesn't make the output
file accessible as a new data set.
From the server log, I can see that the command line is being
constructed correctly, and it even indicates that it's captured the
output, but in the display of the web browser, it just shows up in the
error state. The script being run exits (0) on success. Any ideas?
Here's what the output section of my xml file looks like:
outputs
data format=bam name=coordSortedBam label=${tool.name}
on ${on_string}: coord-sorted read alignments
from_work_dir=alignment/alignment.coordSorted.bam/
/outputs
and here's what the server log states:
galaxy.jobs.handler INFO 2012-07-20 09:52:05,240 (30) Job dispatched
galaxy.jobs.runners.local DEBUG 2012-07-20 09:52:05,453 executing:
alignReads.pl --target
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_26.dat
-o alignment --aligner bowtie --single
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_23.dat
--seqType fq
galaxy.jobs DEBUG 2012-07-20 09:52:16,673 finish(): Moved
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/job_working_directory/000/30/alignment/alignment.coordSorted.bam
to /Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_50.dat
as directed by from_work_dir
Again, as far as I can tell, everything worked - but the browser
doesn't think so.
I've run the exact command above on the command-line, and it exits(0)
indicating success.
Also, I've verified that when run through my galaxy instance, the
galaxy-relocated output file is as expected.
Many thanks for your help. I'm still getting my feet wet with
galaxy, reading through all the documentation and searching the
mailing list for additional help.
best regards,
-brian
--
--
Brian J. Haas
Manager, Genome Annotation and Analysis, Research and Development
The Broad Institute
http://broad.mit.edu/~bhaas
tool id=alignreads name=alignReads version=0.0.1
descriptionalignReads: short read alignment tool wrapper/description
requirements
requirement type=packagetrinity/requirement
/requirements
command
alignReads.pl --target $target -o alignment --aligner $aligner_selection.aligner
## Inputs.
#if str($inputs.paired_or_single) == paired:
--left $inputs.left_input --right $inputs.right_input
#if $inputs.left_input.ext == 'fa':
--seqType fa
#else:
--seqType fq
#end if
#if str($inputs.library_type) != None:
--SS_lib_type $inputs.library_type
#end if
--max_dist_between_pairs $inputs.max_dist_between_pairs
#else:
--single $inputs.input
#if str($inputs.input.ext) == 'fa':
--seqType fa
#else:
--seqType fq
#end if
#if str($inputs.library_type) != None:
--SS_lib_type $inputs.library_type
#end if
#end if
## Additional parameters.
##if str($inputs.use_additional) == yes:
## -- $inputs.additional_params
##end if
## direct to output
## $trinity_log 2amp;1
/command
inputs
param format=fasta name=target type=data label=target help=Fasta sequences targeted for short-read alignment /
conditional name=inputs
param name=paired_or_single type=select label=Paired or Single-end data?
option value=pairedPaired/option
option value=singleSingle/option
/param
when value=paired
param format=fasta,fastq name=left_input type=data label=Left/Forward strand reads help=/
param format=fasta,fastq name=right_input type=data label=Right/Reverse strand reads help=/
param name=library_type type=select label=Strand-specific Library Type
option value=NoneNone/option
option value=FRFR/option
option value=RFRF/option
/param
param name=max_dist_between_pairs type=integer value=2000 min=1 label=max_dist_between_pairs help=Maximum length expected between fragment pairs as aligned to the target, including introns where relevant./
/when
when value=single
param format=fasta,fastq name=input type=data label=Single-end reads help=/
param name=library_type type=select label=Strand-specific Library Type
option value=NoneNone/option
option value=FF/option
option value=RR/option
/param
/when
/conditional
conditional name=aligner_selection
param name=aligner type=select label=Select alignment tool to run
option value=bowtiebowtie/option
option value=bwabwa/option
option
Re: [galaxy-dev] pipeline execution succeeds but galaxy shows failure
Brian;
I wrote a pipeline (xml attached) that, from what I can gather,
succeeds, but galaxy shows it as an error and doesn't make the output
file accessible as a new data set.
Is it possible the software is writing to standard error? Galaxy doesn't
check status codes, but rather check for stderr and assumes that output
indicates a problem. You can wrap the problematic programs with a little
script to eat up stderr and check that everything is okay:
http://wiki.g2.bx.psu.edu/Future/Job%20Failure%20When%20stderr
Brad
From the server log, I can see that the command line is being
constructed correctly, and it even indicates that it's captured the
output, but in the display of the web browser, it just shows up in the
error state. The script being run exits (0) on success. Any ideas?
Here's what the output section of my xml file looks like:
outputs
data format=bam name=coordSortedBam label=${tool.name}
on ${on_string}: coord-sorted read alignments
from_work_dir=alignment/alignment.coordSorted.bam/
/outputs
and here's what the server log states:
galaxy.jobs.handler INFO 2012-07-20 09:52:05,240 (30) Job dispatched
galaxy.jobs.runners.local DEBUG 2012-07-20 09:52:05,453 executing:
alignReads.pl --target
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_26.dat
-o alignment --aligner bowtie --single
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_23.dat
--seqType fq
galaxy.jobs DEBUG 2012-07-20 09:52:16,673 finish(): Moved
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/job_working_directory/000/30/alignment/alignment.coordSorted.bam
to /Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_50.dat
as directed by from_work_dir
Again, as far as I can tell, everything worked - but the browser
doesn't think so.
I've run the exact command above on the command-line, and it exits(0)
indicating success.
Also, I've verified that when run through my galaxy instance, the
galaxy-relocated output file is as expected.
Many thanks for your help. I'm still getting my feet wet with
galaxy, reading through all the documentation and searching the
mailing list for additional help.
best regards,
-brian
--
--
Brian J. Haas
Manager, Genome Annotation and Analysis, Research and Development
The Broad Institute
http://broad.mit.edu/~bhaas
tool id=alignreads name=alignReads version=0.0.1
descriptionalignReads: short read alignment tool wrapper/description
requirements
requirement type=packagetrinity/requirement
/requirements
command
alignReads.pl --target $target -o alignment --aligner
$aligner_selection.aligner
## Inputs.
#if str($inputs.paired_or_single) == paired:
--left $inputs.left_input --right $inputs.right_input
#if $inputs.left_input.ext == 'fa':
--seqType fa
#else:
--seqType fq
#end if
#if str($inputs.library_type) != None:
--SS_lib_type $inputs.library_type
#end if
--max_dist_between_pairs $inputs.max_dist_between_pairs
#else:
--single $inputs.input
#if str($inputs.input.ext) == 'fa':
--seqType fa
#else:
--seqType fq
#end if
#if str($inputs.library_type) != None:
--SS_lib_type $inputs.library_type
#end if
#end if
## Additional parameters.
##if str($inputs.use_additional) == yes:
## -- $inputs.additional_params
##end if
## direct to output
## $trinity_log 2amp;1
/command
inputs
param format=fasta name=target type=data label=target
help=Fasta sequences targeted for short-read alignment /
conditional name=inputs
param name=paired_or_single type=select label=Paired
or Single-end data?
option value=pairedPaired/option
option value=singleSingle/option
/param
when value=paired
param format=fasta,fastq name=left_input type=data
label=Left/Forward strand reads help=/
param format=fasta,fastq name=right_input type=data
label=Right/Reverse strand reads help=/
param name=library_type type=select
label=Strand-specific Library Type
option value=NoneNone/option
option value=FRFR/option
option value=RFRF/option
/param
param name=max_dist_between_pairs type=integer
value=2000 min=1 label=max_dist_between_pairs help=Maximum length
expected between fragment pairs as aligned to the target, including introns
where relevant./
/when
when value=single
param
Re: [galaxy-dev] pipeline execution succeeds but galaxy shows failure
Thanks Brad and Nicole! This definitely explains it. The stderr
(which almost all my tools generate for monitoring purposes) was
resulting in galaxy thinking the process failed.
All is well and good now. HUGE THANKS!
-brian
On Fri, Jul 20, 2012 at 1:53 PM, Nicole Rockweiler
n.rockwei...@gmail.com wrote:
Hi Brian,
A couple of days ago, smcmanus pushed the following change to the repo:
Tools can now specify their own handling of stderr and stdout regular
expressions as well as exit code ranges.
https://bitbucket.org/galaxy/galaxy-central/issue/325/allow-tool-authors-to-decide-whether-to-use-return-codes-or-stderr-for-detecting-job
It looks like the documentation has yet to be written.
Hope this helps,
Nicole
On Fri, Jul 20, 2012 at 12:46 PM, Brad Chapman chapm...@50mail.com wrote:
Brian;
I wrote a pipeline (xml attached) that, from what I can gather,
succeeds, but galaxy shows it as an error and doesn't make the output
file accessible as a new data set.
Is it possible the software is writing to standard error? Galaxy doesn't
check status codes, but rather check for stderr and assumes that output
indicates a problem. You can wrap the problematic programs with a little
script to eat up stderr and check that everything is okay:
http://wiki.g2.bx.psu.edu/Future/Job%20Failure%20When%20stderr
Brad
From the server log, I can see that the command line is being
constructed correctly, and it even indicates that it's captured the
output, but in the display of the web browser, it just shows up in the
error state. The script being run exits (0) on success. Any ideas?
Here's what the output section of my xml file looks like:
outputs
data format=bam name=coordSortedBam label=${tool.name}
on ${on_string}: coord-sorted read alignments
from_work_dir=alignment/alignment.coordSorted.bam/
/outputs
and here's what the server log states:
galaxy.jobs.handler INFO 2012-07-20 09:52:05,240 (30) Job dispatched
galaxy.jobs.runners.local DEBUG 2012-07-20 09:52:05,453 executing:
alignReads.pl --target
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_26.dat
-o alignment --aligner bowtie --single
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_23.dat
--seqType fq
galaxy.jobs DEBUG 2012-07-20 09:52:16,673 finish(): Moved
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/job_working_directory/000/30/alignment/alignment.coordSorted.bam
to
/Users/bhaas/BioIfx/Galaxy/galaxy-dist/database/files/000/dataset_50.dat
as directed by from_work_dir
Again, as far as I can tell, everything worked - but the browser
doesn't think so.
I've run the exact command above on the command-line, and it exits(0)
indicating success.
Also, I've verified that when run through my galaxy instance, the
galaxy-relocated output file is as expected.
Many thanks for your help. I'm still getting my feet wet with
galaxy, reading through all the documentation and searching the
mailing list for additional help.
best regards,
-brian
--
--
Brian J. Haas
Manager, Genome Annotation and Analysis, Research and Development
The Broad Institute
http://broad.mit.edu/~bhaas
tool id=alignreads name=alignReads version=0.0.1
descriptionalignReads: short read alignment tool
wrapper/description
requirements
requirement type=packagetrinity/requirement
/requirements
command
alignReads.pl --target $target -o alignment --aligner
$aligner_selection.aligner
## Inputs.
#if str($inputs.paired_or_single) == paired:
--left $inputs.left_input --right $inputs.right_input
#if $inputs.left_input.ext == 'fa':
--seqType fa
#else:
--seqType fq
#end if
#if str($inputs.library_type) != None:
--SS_lib_type $inputs.library_type
#end if
--max_dist_between_pairs
$inputs.max_dist_between_pairs
#else:
--single $inputs.input
#if str($inputs.input.ext) == 'fa':
--seqType fa
#else:
--seqType fq
#end if
#if str($inputs.library_type) != None:
--SS_lib_type $inputs.library_type
#end if
#end if
## Additional parameters.
##if str($inputs.use_additional) == yes:
## -- $inputs.additional_params
##end if
## direct to output
## $trinity_log 2amp;1
/command
inputs
param format=fasta name=target type=data
label=target help=Fasta sequences targeted for short-read alignment /
conditional name=inputs
param name=paired_or_single type=select
label=Paired or Single-end data?
