Hi Scott,

Following some failing hard drives, I'm rebuilding our Galaxy server.
Something isn't quite right with our cluster integration yet, but it has
exposed a problem in Galaxy's handling of task splitting - it can
sometimes attempt to merge zero files.

Here is my fix for the BLAST XML format (now in the ToolShed),

Here's an example using the text format:

galaxy.jobs.splitters.multi ERROR 2012-10-18 16:26:21,330 Error merging files
Traceback (most recent call last):
  File "/mnt/galaxy/galaxy-central/lib/galaxy/jobs/splitters/multi.py",
line 133, in do_merge
    output_type.merge(output_files, output_file_name)
  File "/mnt/galaxy/galaxy-central/lib/galaxy/datatypes/data.py", line
545, in merge
    raise Exception('Result %s from %s' % (result, cmd))
Exception: Result 2 from cat  >

The problem obviously is that while "cat file1 ... fileN > merged" will
work fine for one or more files, with no files it sits waiting for stdin
(and from a user perspective stalls).

This logic error is in lib/galaxy/datatypes/data.py method merge,
which could either treat zero files as an error, or a no-op:

        if len(split_files) == 1:
            cmd = 'mv -f %s %s' % ( split_files[0], output_file )
            cmd = 'cat %s > %s' % ( ' '.join(split_files), output_file )
        result = os.system(cmd)

I think this should be something like this:

        if not split_files:
            raise Exception('Asked to merge zero files')
        elif len(split_files) == 1:
            cmd = 'mv -f %s %s' % ( split_files[0], output_file )
            cmd = 'cat %s > %s' % ( ' '.join(split_files), output_file )
        result = os.system(cmd)

It might also make sense to check for zero files in the code which
calls the merge, i.e. lib/galaxy/jobs/splitters/multi.py function do_merge
I'm still investigating upstream how this comes about, one clue:

galaxy.jobs.runners.drmaa DEBUG 2012-10-18 16:25:01,930 (273/510)
state change: job is running
galaxy.jobs.runners.drmaa DEBUG 2012-10-18 16:25:03,040 (273/510)
state change: job finished, but failed
galaxy.jobs.runners.drmaa DEBUG 2012-10-18 16:25:03,074 Job output not
returned from cluster
galaxy.jobs DEBUG 2012-10-18 16:25:03,074 task 641 for job 273 ended;
exit code: 0
galaxy.jobs DEBUG 2012-10-18 16:25:03,148 task 641 ended
galaxy.jobs.runners.tasks DEBUG 2012-10-18 16:25:05,169 execution
finished - beginning merge: tblastx -query
"/mnt/galaxy/galaxy-central/database/files/000/dataset_127.dat"   -db
"/var/local/blast/ncbi/nt" -query_gencode 2 -evalue 0.001 -out
-outfmt 0 -num_threads 8
galaxy.jobs.splitters.multi DEBUG 2012-10-18 16:25:05,181 files []

If you would prefer that small suggestion as a pull request, let me know.


Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:


Reply via email to