I broke the TaskWrapper last week with my exit code handling "fix",
the double semi-colon thing you are seeing there. Your fix would break
non-task split jobs so that is probably the problem(?) Hopefully? Want
to revert 8e001dc9675c and pull in the changeset I just pushed out.

Otherwise, I will test out task splitting later today.

I am very sorry.

-John

On Thu, Oct 17, 2013 at 4:53 AM, Peter Cock <p.j.a.c...@googlemail.com> wrote:
> Hi John,
>
> Is all your semi-colon fixing on the trunk? I've found another bug in
> this area (patch below), which is showing up with task splitting under
> SGE.
>
> e.g. this job is meant to be running and merging 5 BLAST XML files, stderr:
>
> nothing to merge for
> /mnt/galaxy/galaxy-central/database/files/000/dataset_498.dat
> (expected 5 files)
>
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0:
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/galaxy_288_60.sh:
> line 13: syntax error near unexpected token `;'
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/galaxy_288_60.sh:
> line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0;
> blastn -query 
> "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/dataset_495.dat"
>   -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001
> -out 
> "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_0/dataset_498.dat"
>     -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs
> 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit
> $return_code"'
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1:
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/galaxy_288_61.sh:
> line 13: syntax error near unexpected token `;'
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/galaxy_288_61.sh:
> line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1;
> blastn -query 
> "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/dataset_495.dat"
>   -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001
> -out 
> "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_1/dataset_498.dat"
>     -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs
> 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit
> $return_code"'
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2:
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/galaxy_288_62.sh:
> line 13: syntax error near unexpected token `;'
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/galaxy_288_62.sh:
> line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2;
> blastn -query 
> "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/dataset_495.dat"
>   -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001
> -out 
> "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_2/dataset_498.dat"
>     -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs
> 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit
> $return_code"'
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3:
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/galaxy_288_63.sh:
> line 13: syntax error near unexpected token `;'
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/galaxy_288_63.sh:
> line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3;
> blastn -query 
> "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/dataset_495.dat"
>   -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001
> -out 
> "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_3/dataset_498.dat"
>     -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs
> 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit
> $return_code"'
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4:
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/galaxy_288_64.sh:
> line 13: syntax error near unexpected token `;'
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/galaxy_288_64.sh:
> line 13: `/mnt/galaxy/galaxy-central/extract_dataset_parts.sh
> /mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4;
> blastn -query 
> "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/dataset_495.dat"
>   -db "/mnt/scratch/local/blast/ncbi/nt" -task megablast -evalue 0.001
> -out 
> "/mnt/galaxy/galaxy-central/database/job_working_directory/000/288/task_4/dataset_498.dat"
>     -outfmt 5 -num_threads 8 -dust yes -strand both -max_target_seqs
> 3; return_code=$?; cd /mnt/galaxy/galaxy-central; ; sh -c "exit
> $return_code"'
>
> Note the repeated semi-colon, which causes the child jobs to fail (patch 
> below).
>
> There is however a second bug, the merge fails yet the history entry
> is still green (success). The merge method is raising a ValueError yet
> it is being ignored.
>
> Regards,
>
> Peter
>
> $ hg branch
> default
>
> $ hg tip
> changeset:   12028:8e001dc9675c
> tag:         tip
> user:        John Chilton <jmchil...@gmail.com>
> date:        Wed Oct 16 23:15:21 2013 -0500
> summary:     Dependencies: Do not attempt to install package
> dependencies in tool shed if the will be ignored.
>
> $ hg diff lib/galaxy/jobs/command_factory.py
> diff -r 8e001dc9675c lib/galaxy/jobs/command_factory.py
> --- a/lib/galaxy/jobs/command_factory.py    Wed Oct 16 23:15:21 2013 -0500
> +++ b/lib/galaxy/jobs/command_factory.py    Thu Oct 17 10:50:18 2013 +0100
> @@ -58,7 +58,7 @@
>          if not captured_return_code:
>              commands += capture_return_code_command
>              captured_return_code = True
> -        commands += "; cd %s; " % abspath( getcwd() )
> +        commands += "; cd %s " % abspath( getcwd() )
>          commands += job_wrapper.setup_external_metadata(
>              exec_dir=abspath( getcwd() ),
>              tmp_dir=job_wrapper.working_directory,
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to