Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi Ross,
I have a cloudman version of Galaxy working well. What I am doing now is trying
to install Galaxy on starcluster, which is a SGE cluster. I should followed the
install method for the normal cluster/SGE install, not for the cloud.
Thank you for your quick reply.
Have a nice day.
Cai
--- 12年4月18日，周三, Ross 写道：
> 发件人: Ross
> 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh:
> samtools: not found
> 收件人: "zhengqiu cai"
> 抄送: "Jennifer Jackson" , galaxy-dev@lists.bx.psu.edu
> 日期: 2012年4月18日,周三,下午11:46
> Hi, Cai.
> Apologies but I had not noticed any reference to the cloud
> distribution before - that changes everything - I was
> talking about a
> normal cluster/SGE install - you should definitely not need
> to do
> anything at all to the AMI - maybe someone with cloudman
> experience
> can be more helpful than I can.
>
> On Wed, Apr 18, 2012 at 11:39 AM, zhengqiu cai
> wrote:
> > Hi Ross,
> >
> > I did not compile samtools using the --prefix. The
> samtools I used comes with Galaxy AMI image, and I copied it
> to the same path on my EC2. How to compile samtools using
> --prefix? I just typed 'make', and there is not a
> configuration file. In the cloudman version of Galaxy (it is
> working well for me), only two executables (samtools and
> maq2sam-long) in the directory
> /mnt/galaxyTools/tools/samtools/0.1.12. I copies the same
> files to my EC2. Is the only compiled executable 'samtools'
> fully functional for the cluster(it is on the local
> machine)? or should I set some other libraries?
> >
> > Thank you,
> >
> > Cai
> >
> > --- 12年4月18日，周三, Ross
> 写道：
> >
> >> 发件人: Ross
> >> 主题: Re: [galaxy-dev] Exception: Error Setting
> BAM Metadata: /bin/sh: samtools: not found
> >> 收件人: "zhengqiu cai"
> >> 抄送: "Jennifer Jackson" ,
> galaxy-dev@lists.bx.psu.edu
> >> 日期: 2012年4月18日,周三,下午11:07
> >> Hi, Cai - sorry to hear it's still
> >> not working right.
> >>
> >> You've compiled/installed samtools using the
> --prefix
> >> configure option
> >> and tested that all the samtools based Galaxy
> wrappers work
> >> if you use
> >> the local job runner? That means that both the
> appropriate
> >> lib and bin
> >> directories are on the Galaxy user's path ready to
> be
> >> inherited by SGE
> >> with the -V SGE option.
> >>
> >> If the local job runner works, that seems to
> isolate the
> >> problem to
> >> SGE configuration and operation - the steps I've
> suggested
> >> seem to
> >> work for us so I've run out of ideas on why it's
> not working
> >> at your
> >> shop and hope somebody with more experience can
> help out.
> >>
> >>
> >> On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai
>
> >> wrote:
> >> > Hi Ross,
> >> >
> >> > Thank you for your reply.
> >> > It still did not work.
> >> > I think SGE knows the path of samtools. Here
> is a test
> >> case I added to Galaxy, and it creates the index
> for fasta
> >> file. I pasted the samtools faidx wrapper and xml
> files
> >> below:
> >> >
> >>
> ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
> >> more samtools_faidx.pl
> >> > #!/usr/bin/perl
> >> > use strict;
> >> > use warnings;
> >> >
> >> > if (@ARGV < 1) {
> >> > die "Usage:
> >> perl $0 \n";
> >> > }
> >> > `samtools faidx $ARGV[0]`;
> >> > `cp ${ARGV[0]}.fai $ARGV[1]`;
> >> > exit;
> >> >
> >> >
> >>
> ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
> >> more samtools_faidx.xml
> >> > >> version="1.0.3">
> >> >
> >> > >> type="package">samtools
> >> >
> >> > creates fasta
> >> index
> >> >
> >> > samtools_faidx.pl $input1
> $output1
> >> >
> >> >
> >> > type="data"
> >> format="fasta" label="fasta file to be indexed"
> />
> >> >
> >> >
> >> > name="output1"
> >> label="${tool.name} on ${on_string}: fasta" />
> >> >
> >> >
> >> >
> >> > It works, which
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi, Cai.
Apologies but I had not noticed any reference to the cloud
distribution before - that changes everything - I was talking about a
normal cluster/SGE install - you should definitely not need to do
anything at all to the AMI - maybe someone with cloudman experience
can be more helpful than I can.
On Wed, Apr 18, 2012 at 11:39 AM, zhengqiu cai wrote:
> Hi Ross,
>
> I did not compile samtools using the --prefix. The samtools I used comes with
> Galaxy AMI image, and I copied it to the same path on my EC2. How to compile
> samtools using --prefix? I just typed 'make', and there is not a
> configuration file. In the cloudman version of Galaxy (it is working well for
> me), only two executables (samtools and maq2sam-long) in the directory
> /mnt/galaxyTools/tools/samtools/0.1.12. I copies the same files to my EC2. Is
> the only compiled executable 'samtools' fully functional for the cluster(it
> is on the local machine)? or should I set some other libraries?
>
> Thank you,
>
> Cai
>
> --- 12年4月18日，周三, Ross 写道：
>
>> 发件人: Ross
>> 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh:
>> samtools: not found
>> 收件人: "zhengqiu cai"
>> 抄送: "Jennifer Jackson" , galaxy-dev@lists.bx.psu.edu
>> 日期: 2012年4月18日,周三,下午11:07
>> Hi, Cai - sorry to hear it's still
>> not working right.
>>
>> You've compiled/installed samtools using the --prefix
>> configure option
>> and tested that all the samtools based Galaxy wrappers work
>> if you use
>> the local job runner? That means that both the appropriate
>> lib and bin
>> directories are on the Galaxy user's path ready to be
>> inherited by SGE
>> with the -V SGE option.
>>
>> If the local job runner works, that seems to isolate the
>> problem to
>> SGE configuration and operation - the steps I've suggested
>> seem to
>> work for us so I've run out of ideas on why it's not working
>> at your
>> shop and hope somebody with more experience can help out.
>>
>>
>> On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai
>> wrote:
>> > Hi Ross,
>> >
>> > Thank you for your reply.
>> > It still did not work.
>> > I think SGE knows the path of samtools. Here is a test
>> case I added to Galaxy, and it creates the index for fasta
>> file. I pasted the samtools faidx wrapper and xml files
>> below:
>> >
>> ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
>> more samtools_faidx.pl
>> > #!/usr/bin/perl
>> > use strict;
>> > use warnings;
>> >
>> > if (@ARGV < 1) {
>> >die "Usage:
>>perl $0 \n";
>> > }
>> > `samtools faidx $ARGV[0]`;
>> > `cp ${ARGV[0]}.fai $ARGV[1]`;
>> > exit;
>> >
>> >
>> ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
>> more samtools_faidx.xml
>> > > version="1.0.3">
>> >
>> >> type="package">samtools
>> >
>> > creates fasta
>> index
>> >
>> >samtools_faidx.pl $input1 $output1
>> >
>> >
>> >> format="fasta" label="fasta file to be indexed" />
>> >
>> >
>> >> label="${tool.name} on ${on_string}: fasta" />
>> >
>> >
>> >
>> > It works, which shows that SGE knows the path of
>> samtools.
>> >
>> >
>> > Cai
>> >
>> > --- 12年4月18日，周三, Ross
>> 写道：
>> >
>> >> 发件人: Ross
>> >> 主题: Re: [galaxy-dev] Exception: Error Setting
>> BAM Metadata: /bin/sh: samtools: not found
>> >> 收件人: "zhengqiu cai"
>> >> 抄送: "Jennifer Jackson" ,
>> galaxy-dev@lists.bx.psu.edu
>> >> 日期: 2012年4月18日,周三,上午5:18
>> >> Ah. Then you probably have a library
>> >> path problem.
>> >> Here's how we do it FWIW:
>> >>
>> >> Assume your want your shared cluster apps in
>> >> /data/apps/bin
>> >>
>> >> To share the compiled libraries and binaries
>> together,
>> >> remember to
>> >> compile apps like samtools using the
>> >> --prefix /data/app
>> >> flag so then you can add the /data/app/lib path to
>> the
>> >> galaxy user's
>> >> path so the nodes running jobs inherit those paths
>> because
>> >> of the -V
>
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi Ross,
I did not compile samtools using the --prefix. The samtools I used comes with
Galaxy AMI image, and I copied it to the same path on my EC2. How to compile
samtools using --prefix? I just typed 'make', and there is not a configuration
file. In the cloudman version of Galaxy (it is working well for me), only two
executables (samtools and maq2sam-long) in the directory
/mnt/galaxyTools/tools/samtools/0.1.12. I copies the same files to my EC2. Is
the only compiled executable 'samtools' fully functional for the cluster(it is
on the local machine)? or should I set some other libraries?
Thank you,
Cai
--- 12年4月18日，周三, Ross 写道：
> 发件人: Ross
> 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh:
> samtools: not found
> 收件人: "zhengqiu cai"
> 抄送: "Jennifer Jackson" , galaxy-dev@lists.bx.psu.edu
> 日期: 2012年4月18日,周三,下午11:07
> Hi, Cai - sorry to hear it's still
> not working right.
>
> You've compiled/installed samtools using the --prefix
> configure option
> and tested that all the samtools based Galaxy wrappers work
> if you use
> the local job runner? That means that both the appropriate
> lib and bin
> directories are on the Galaxy user's path ready to be
> inherited by SGE
> with the -V SGE option.
>
> If the local job runner works, that seems to isolate the
> problem to
> SGE configuration and operation - the steps I've suggested
> seem to
> work for us so I've run out of ideas on why it's not working
> at your
> shop and hope somebody with more experience can help out.
>
>
> On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai
> wrote:
> > Hi Ross,
> >
> > Thank you for your reply.
> > It still did not work.
> > I think SGE knows the path of samtools. Here is a test
> case I added to Galaxy, and it creates the index for fasta
> file. I pasted the samtools faidx wrapper and xml files
> below:
> >
> ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
> more samtools_faidx.pl
> > #!/usr/bin/perl
> > use strict;
> > use warnings;
> >
> > if (@ARGV < 1) {
> > die "Usage:
> perl $0 \n";
> > }
> > `samtools faidx $ARGV[0]`;
> > `cp ${ARGV[0]}.fai $ARGV[1]`;
> > exit;
> >
> >
> ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
> more samtools_faidx.xml
> > version="1.0.3">
> >
> > type="package">samtools
> >
> > creates fasta
> index
> >
> > samtools_faidx.pl $input1 $output1
> >
> >
> > format="fasta" label="fasta file to be indexed" />
> >
> >
> > label="${tool.name} on ${on_string}: fasta" />
> >
> >
> >
> > It works, which shows that SGE knows the path of
> samtools.
> >
> >
> > Cai
> >
> > --- 12年4月18日，周三, Ross
> 写道：
> >
> >> 发件人: Ross
> >> 主题: Re: [galaxy-dev] Exception: Error Setting
> BAM Metadata: /bin/sh: samtools: not found
> >> 收件人: "zhengqiu cai"
> >> 抄送: "Jennifer Jackson" ,
> galaxy-dev@lists.bx.psu.edu
> >> 日期: 2012年4月18日,周三,上午5:18
> >> Ah. Then you probably have a library
> >> path problem.
> >> Here's how we do it FWIW:
> >>
> >> Assume your want your shared cluster apps in
> >> /data/apps/bin
> >>
> >> To share the compiled libraries and binaries
> together,
> >> remember to
> >> compile apps like samtools using the
> >> --prefix /data/app
> >> flag so then you can add the /data/app/lib path to
> the
> >> galaxy user's
> >> path so the nodes running jobs inherit those paths
> because
> >> of the -V
> >> switch.
> >>
> >> I hope this works!
> >>
> >>
> >>
> >> On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai
>
> >> wrote:
> >> > Hi Ross,
> >> >
> >> > Thank you for your reply.
> >> > I tried it, and it was still not working.
> >> > Only samtools has this problem, and the other
> tools
> >> works fine.
> >> >
> >> > samtools was compiled from source correctly.
> If I
> >> install samtools using "sudo apt-get install
> samtools", this
> >> problem is gone, but this is not a solution for
> since it
> >> does not work across the cluster.
> >> >
> >> >
> >> > Cai
> >> >
> >> > --- 12年4月18日，周三, Ross
> &g
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi, Cai - sorry to hear it's still not working right.
You've compiled/installed samtools using the --prefix configure option
and tested that all the samtools based Galaxy wrappers work if you use
the local job runner? That means that both the appropriate lib and bin
directories are on the Galaxy user's path ready to be inherited by SGE
with the -V SGE option.
If the local job runner works, that seems to isolate the problem to
SGE configuration and operation - the steps I've suggested seem to
work for us so I've run out of ideas on why it's not working at your
shop and hope somebody with more experience can help out.
On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai wrote:
> Hi Ross,
>
> Thank you for your reply.
> It still did not work.
> I think SGE knows the path of samtools. Here is a test case I added to
> Galaxy, and it creates the index for fasta file. I pasted the samtools faidx
> wrapper and xml files below:
> ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more
> samtools_faidx.pl
> #!/usr/bin/perl
> use strict;
> use warnings;
>
> if (@ARGV < 1) {
>die "Usage: perl $0 \n";
> }
> `samtools faidx $ARGV[0]`;
> `cp ${ARGV[0]}.fai $ARGV[1]`;
> exit;
>
> ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more
> samtools_faidx.xml
>
>
>samtools
>
> creates fasta index
>
>samtools_faidx.pl $input1 $output1
>
>
>
>
>
>
>
>
>
> It works, which shows that SGE knows the path of samtools.
>
>
> Cai
>
> --- 12年4月18日，周三, Ross 写道：
>
>> 发件人: Ross
>> 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh:
>> samtools: not found
>> 收件人: "zhengqiu cai"
>> 抄送: "Jennifer Jackson" , galaxy-dev@lists.bx.psu.edu
>> 日期: 2012年4月18日,周三,上午5:18
>> Ah. Then you probably have a library
>> path problem.
>> Here's how we do it FWIW:
>>
>> Assume your want your shared cluster apps in
>> /data/apps/bin
>>
>> To share the compiled libraries and binaries together,
>> remember to
>> compile apps like samtools using the
>> --prefix /data/app
>> flag so then you can add the /data/app/lib path to the
>> galaxy user's
>> path so the nodes running jobs inherit those paths because
>> of the -V
>> switch.
>>
>> I hope this works!
>>
>>
>>
>> On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai
>> wrote:
>> > Hi Ross,
>> >
>> > Thank you for your reply.
>> > I tried it, and it was still not working.
>> > Only samtools has this problem, and the other tools
>> works fine.
>> >
>> > samtools was compiled from source correctly. If I
>> install samtools using "sudo apt-get install samtools", this
>> problem is gone, but this is not a solution for since it
>> does not work across the cluster.
>> >
>> >
>> > Cai
>> >
>> > --- 12年4月18日，周三, Ross
>> 写道：
>> >
>> >> 发件人: Ross
>> >> 主题: Re: [galaxy-dev] Exception: Error Setting
>> BAM Metadata: /bin/sh: samtools: not found
>> >> 收件人: "zhengqiu cai"
>> >> 抄送: "Jennifer Jackson" ,
>> galaxy-dev@lists.bx.psu.edu
>> >> 日期: 2012年4月18日,周三,上午4:59
>> >> Cai - if you are using SGE you
>> >> probably want to pass the Galaxy user's
>> >> path to the job using the
>> >> -V
>> >> switch in the system wide (or Galaxy user's home
>> directory)
>> >> sge_request file
>> >>
>> >> eg I have
>> >> rlazarus@iaas1-int:~$ cat
>> >> /var/lib/gridengine/default/common/sge_request
>> >> -cwd
>> >> -V
>> >>
>> >> Without that switch, the job gets no path.
>> >> With -V, it inherits the job submitter's full
>> path.
>> >>
>> >> On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai
>>
>> >> wrote:
>> >> > Hi Jen,
>> >> >
>> >> > Thank you for your quick response.
>> >> > I changed the env.sh file to what you
>> seggested, and
>> >> the problem still existed.
>> >> > I am pasting more details below:
>> >> >
>> >> > I submitted a job to convert sam to bam, and
>> the job
>> >> was running forever without outputing the result. I
>> then
>> >> checked the log, and it read:
>> >> > Traceback (most recent call last):
>>
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi Ross,
Thank you for your reply.
It still did not work.
I think SGE knows the path of samtools. Here is a test case I added to Galaxy,
and it creates the index for fasta file. I pasted the samtools faidx wrapper
and xml files below:
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more
samtools_faidx.pl
#!/usr/bin/perl
use strict;
use warnings;
if (@ARGV < 1) {
die "Usage: perl $0 \n";
}
`samtools faidx $ARGV[0]`;
`cp ${ARGV[0]}.fai $ARGV[1]`;
exit;
ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$ more
samtools_faidx.xml
samtools
creates fasta index
samtools_faidx.pl $input1 $output1
It works, which shows that SGE knows the path of samtools.
Cai
--- 12年4月18日，周三, Ross 写道：
> 发件人: Ross
> 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh:
> samtools: not found
> 收件人: "zhengqiu cai"
> 抄送: "Jennifer Jackson" , galaxy-dev@lists.bx.psu.edu
> 日期: 2012年4月18日,周三,上午5:18
> Ah. Then you probably have a library
> path problem.
> Here's how we do it FWIW:
>
> Assume your want your shared cluster apps in
> /data/apps/bin
>
> To share the compiled libraries and binaries together,
> remember to
> compile apps like samtools using the
> --prefix /data/app
> flag so then you can add the /data/app/lib path to the
> galaxy user's
> path so the nodes running jobs inherit those paths because
> of the -V
> switch.
>
> I hope this works!
>
>
>
> On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai
> wrote:
> > Hi Ross,
> >
> > Thank you for your reply.
> > I tried it, and it was still not working.
> > Only samtools has this problem, and the other tools
> works fine.
> >
> > samtools was compiled from source correctly. If I
> install samtools using "sudo apt-get install samtools", this
> problem is gone, but this is not a solution for since it
> does not work across the cluster.
> >
> >
> > Cai
> >
> > --- 12年4月18日，周三, Ross
> 写道：
> >
> >> 发件人: Ross
> >> 主题: Re: [galaxy-dev] Exception: Error Setting
> BAM Metadata: /bin/sh: samtools: not found
> >> 收件人: "zhengqiu cai"
> >> 抄送: "Jennifer Jackson" ,
> galaxy-dev@lists.bx.psu.edu
> >> 日期: 2012年4月18日,周三,上午4:59
> >> Cai - if you are using SGE you
> >> probably want to pass the Galaxy user's
> >> path to the job using the
> >> -V
> >> switch in the system wide (or Galaxy user's home
> directory)
> >> sge_request file
> >>
> >> eg I have
> >> rlazarus@iaas1-int:~$ cat
> >> /var/lib/gridengine/default/common/sge_request
> >> -cwd
> >> -V
> >>
> >> Without that switch, the job gets no path.
> >> With -V, it inherits the job submitter's full
> path.
> >>
> >> On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai
>
> >> wrote:
> >> > Hi Jen,
> >> >
> >> > Thank you for your quick response.
> >> > I changed the env.sh file to what you
> seggested, and
> >> the problem still existed.
> >> > I am pasting more details below:
> >> >
> >> > I submitted a job to convert sam to bam, and
> the job
> >> was running forever without outputing the result. I
> then
> >> checked the log, and it read:
> >> > Traceback (most recent call last):
> >> > File
> >>
> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
> >> line 336, in finish_job
> >> > drm_job_state.job_wrapper.finish(
> stdout,
> >> stderr )
> >> > File
> >>
> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py",
> >> line 637, in finish
> >> > dataset.set_meta( overwrite =
> False )
> >> > File
> >>
> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py",
> >> line 875, in set_meta
> >> > return self.datatype.set_meta(
> self, **kwd
> >> )
> >> > File
> >>
> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py",
> >> line 179, in set_meta
> >> > raise Exception, "Error Setting
> BAM
> >> Metadata: %s" % stderr
> >> > Exception: Error Setting BAM Metadata:
> /bin/sh:
> >> samtools: not found
> >> >
> >> > It means that the samtools is not in the PATH.
> I tried
> >> to set the PATH in a couple of methods according
> the Galaxy
> >> documentation:
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Ah. Then you probably have a library path problem. Here's how we do it FWIW: Assume your want your shared cluster apps in /data/apps/bin To share the compiled libraries and binaries together, remember to compile apps like samtools using the --prefix /data/app flag so then you can add the /data/app/lib path to the galaxy user's path so the nodes running jobs inherit those paths because of the -V switch. I hope this works! On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu cai wrote: > Hi Ross, > > Thank you for your reply. > I tried it, and it was still not working. > Only samtools has this problem, and the other tools works fine. > > samtools was compiled from source correctly. If I install samtools using > "sudo apt-get install samtools", this problem is gone, but this is not a > solution for since it does not work across the cluster. > > > Cai > > --- 12年4月18日，周三, Ross 写道： > >> 发件人: Ross >> 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: >> samtools: not found >> 收件人: "zhengqiu cai" >> 抄送: "Jennifer Jackson" , galaxy-dev@lists.bx.psu.edu >> 日期: 2012年4月18日,周三,上午4:59 >> Cai - if you are using SGE you >> probably want to pass the Galaxy user's >> path to the job using the >> -V >> switch in the system wide (or Galaxy user's home directory) >> sge_request file >> >> eg I have >> rlazarus@iaas1-int:~$ cat >> /var/lib/gridengine/default/common/sge_request >> -cwd >> -V >> >> Without that switch, the job gets no path. >> With -V, it inherits the job submitter's full path. >> >> On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai >> wrote: >> > Hi Jen, >> > >> > Thank you for your quick response. >> > I changed the env.sh file to what you seggested, and >> the problem still existed. >> > I am pasting more details below: >> > >> > I submitted a job to convert sam to bam, and the job >> was running forever without outputing the result. I then >> checked the log, and it read: >> > Traceback (most recent call last): >> > File >> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", >> line 336, in finish_job >> >drm_job_state.job_wrapper.finish( stdout, >> stderr ) >> > File >> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py", >> line 637, in finish >> >dataset.set_meta( overwrite = False ) >> > File >> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py", >> line 875, in set_meta >> >return self.datatype.set_meta( self, **kwd >> ) >> > File >> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py", >> line 179, in set_meta >> >raise Exception, "Error Setting BAM >> Metadata: %s" % stderr >> > Exception: Error Setting BAM Metadata: /bin/sh: >> samtools: not found >> > >> > It means that the samtools is not in the PATH. I tried >> to set the PATH in a couple of methods according the Galaxy >> documentation: >> > 1. put the path in the env.sh in the tool directory and >> symbolink default to the tool directory, e.g. default -> >> =/mnt/galaxyTools/tools/samtools/0.1.18 >> > 2. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 >> in ~/.sge_request >> > 3. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 >> in /path/sge_request >> > >> > none of them worked, and I got the above same problem. >> > >> > Then I checked the job log file in the >> job_working_directory, and it read: >> > Samtools Version: 0.1.18 (r982:295) >> > SAM file converted to BAM >> > >> > which shows that sge knows the PATH of samtools. To >> double check it, I added samtools index to Galaxy, and it >> worked well. I am very confused why SGE knows the tool path >> but cannot run the job correctly. >> > >> > The system I am using is ubuntu on EC2 (the image id >> is: ami-999d49f0, which is the one used by StarCluster). I >> checked out the code from galaxy-dist on bitbucket. Other >> tools such as bwa and bowtie worked well using the same >> setting method(put env.sh in the tools directory to set the >> tool path) >> > >> > >> > Thanks you very much, >> > >> > Cai >> > --- 12年4月18日，周三, Jennifer Jackson >> 写道： >> > >> >> 发件人: Jennifer Jackson >> >> 主题: Re: [galaxy-dev] Exception: Error Setting >> BAM Metadata: /bin/
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi Ross, Thank you for your reply. I tried it, and it was still not working. Only samtools has this problem, and the other tools works fine. samtools was compiled from source correctly. If I install samtools using "sudo apt-get install samtools", this problem is gone, but this is not a solution for since it does not work across the cluster. Cai --- 12年4月18日，周三, Ross 写道： > 发件人: Ross > 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: > samtools: not found > 收件人: "zhengqiu cai" > 抄送: "Jennifer Jackson" , galaxy-dev@lists.bx.psu.edu > 日期: 2012年4月18日,周三,上午4:59 > Cai - if you are using SGE you > probably want to pass the Galaxy user's > path to the job using the > -V > switch in the system wide (or Galaxy user's home directory) > sge_request file > > eg I have > rlazarus@iaas1-int:~$ cat > /var/lib/gridengine/default/common/sge_request > -cwd > -V > > Without that switch, the job gets no path. > With -V, it inherits the job submitter's full path. > > On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai > wrote: > > Hi Jen, > > > > Thank you for your quick response. > > I changed the env.sh file to what you seggested, and > the problem still existed. > > I am pasting more details below: > > > > I submitted a job to convert sam to bam, and the job > was running forever without outputing the result. I then > checked the log, and it read: > > Traceback (most recent call last): > > File > "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", > line 336, in finish_job > > drm_job_state.job_wrapper.finish( stdout, > stderr ) > > File > "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py", > line 637, in finish > > dataset.set_meta( overwrite = False ) > > File > "/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py", > line 875, in set_meta > > return self.datatype.set_meta( self, **kwd > ) > > File > "/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py", > line 179, in set_meta > > raise Exception, "Error Setting BAM > Metadata: %s" % stderr > > Exception: Error Setting BAM Metadata: /bin/sh: > samtools: not found > > > > It means that the samtools is not in the PATH. I tried > to set the PATH in a couple of methods according the Galaxy > documentation: > > 1. put the path in the env.sh in the tool directory and > symbolink default to the tool directory, e.g. default -> > =/mnt/galaxyTools/tools/samtools/0.1.18 > > 2. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 > in ~/.sge_request > > 3. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 > in /path/sge_request > > > > none of them worked, and I got the above same problem. > > > > Then I checked the job log file in the > job_working_directory, and it read: > > Samtools Version: 0.1.18 (r982:295) > > SAM file converted to BAM > > > > which shows that sge knows the PATH of samtools. To > double check it, I added samtools index to Galaxy, and it > worked well. I am very confused why SGE knows the tool path > but cannot run the job correctly. > > > > The system I am using is ubuntu on EC2 (the image id > is: ami-999d49f0, which is the one used by StarCluster). I > checked out the code from galaxy-dist on bitbucket. Other > tools such as bwa and bowtie worked well using the same > setting method(put env.sh in the tools directory to set the > tool path) > > > > > > Thanks you very much, > > > > Cai > > --- 12年4月18日，周三, Jennifer Jackson > 写道： > > > >> 发件人: Jennifer Jackson > >> 主题: Re: [galaxy-dev] Exception: Error Setting > BAM Metadata: /bin/sh: samtools: not found > >> 收件人: "zhengqiu cai" > >> 抄送: galaxy-dev@lists.bx.psu.edu > >> 日期: 2012年4月18日,周三,上午2:12 > >> Hi Cai, > >> > >> Please double check that your env.sh file contains > the > >> following: > >> > >> > >> > >> > PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH" > >> export PATH > >> > >> > >> http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies > >> > >> > >> Hopefully this helps, > >> > >> Jen > >> Galaxy team > >> > >> On 4/16/12 12:43 PM, zhengqiu cai wrote: > >> > Hi All, > >> > > >> > I was trying to setup the Galaxy tool > dependencies, and > >> I met the following problem when running sam-to-bam > in > >> samtoo
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Cai - if you are using SGE you probably want to pass the Galaxy user's path to the job using the -V switch in the system wide (or Galaxy user's home directory) sge_request file eg I have rlazarus@iaas1-int:~$ cat /var/lib/gridengine/default/common/sge_request -cwd -V Without that switch, the job gets no path. With -V, it inherits the job submitter's full path. On Tue, Apr 17, 2012 at 4:26 PM, zhengqiu cai wrote: > Hi Jen, > > Thank you for your quick response. > I changed the env.sh file to what you seggested, and the problem still > existed. > I am pasting more details below: > > I submitted a job to convert sam to bam, and the job was running forever > without outputing the result. I then checked the log, and it read: > Traceback (most recent call last): > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line > 336, in finish_job >drm_job_state.job_wrapper.finish( stdout, stderr ) > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, > in finish >dataset.set_meta( overwrite = False ) > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py", line 875, > in set_meta >return self.datatype.set_meta( self, **kwd ) > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py", line > 179, in set_meta >raise Exception, "Error Setting BAM Metadata: %s" % stderr > Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found > > It means that the samtools is not in the PATH. I tried to set the PATH in a > couple of methods according the Galaxy documentation: > 1. put the path in the env.sh in the tool directory and symbolink default to > the tool directory, e.g. default -> =/mnt/galaxyTools/tools/samtools/0.1.18 > 2. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in ~/.sge_request > 3. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in /path/sge_request > > none of them worked, and I got the above same problem. > > Then I checked the job log file in the job_working_directory, and it read: > Samtools Version: 0.1.18 (r982:295) > SAM file converted to BAM > > which shows that sge knows the PATH of samtools. To double check it, I added > samtools index to Galaxy, and it worked well. I am very confused why SGE > knows the tool path but cannot run the job correctly. > > The system I am using is ubuntu on EC2 (the image id is: ami-999d49f0, which > is the one used by StarCluster). I checked out the code from galaxy-dist on > bitbucket. Other tools such as bwa and bowtie worked well using the same > setting method(put env.sh in the tools directory to set the tool path) > > > Thanks you very much, > > Cai > --- 12年4月18日，周三, Jennifer Jackson 写道： > >> 发件人: Jennifer Jackson >> 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: >> samtools: not found >> 收件人: "zhengqiu cai" >> 抄送: galaxy-dev@lists.bx.psu.edu >> 日期: 2012年4月18日,周三,上午2:12 >> Hi Cai, >> >> Please double check that your env.sh file contains the >> following: >> >> >> >>PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH" >> export PATH >> >> >> http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies >> >> >> Hopefully this helps, >> >> Jen >> Galaxy team >> >> On 4/16/12 12:43 PM, zhengqiu cai wrote: >> > Hi All, >> > >> > I was trying to setup the Galaxy tool dependencies, and >> I met the following problem when running sam-to-bam in >> samtools: >> > Traceback (most recent call last): >> >File >> "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", >> line 336, in finish_job >> > drm_job_state.job_wrapper.finish( >> stdout, stderr ) >> >File >> "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py", >> line 637, in finish >> > dataset.set_meta( overwrite = False >> ) >> >File >> "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py", >> line 875, in set_meta >> > return self.datatype.set_meta( >> self, **kwd ) >> >File >> "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py", >> line 179, in set_meta >> > raise Exception, "Error Setting BAM >> Metadata: %s" % stderr >> > Exception: Error Setting BAM Metadata: /bin/sh: >> samtools: not found >> > >> > >> > I already set the path of samtools >> > Below I pasted how I set the path: >> > ubuntu@master:/
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi Jen, Thank you for your quick response. I changed the env.sh file to what you seggested, and the problem still existed. I am pasting more details below: I submitted a job to convert sam to bam, and the job was running forever without outputing the result. I then checked the log, and it read: Traceback (most recent call last): File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 336, in finish_job drm_job_state.job_wrapper.finish( stdout, stderr ) File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, in finish dataset.set_meta( overwrite = False ) File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py", line 875, in set_meta return self.datatype.set_meta( self, **kwd ) File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py", line 179, in set_meta raise Exception, "Error Setting BAM Metadata: %s" % stderr Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found It means that the samtools is not in the PATH. I tried to set the PATH in a couple of methods according the Galaxy documentation: 1. put the path in the env.sh in the tool directory and symbolink default to the tool directory, e.g. default -> =/mnt/galaxyTools/tools/samtools/0.1.18 2. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in ~/.sge_request 3. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in /path/sge_request none of them worked, and I got the above same problem. Then I checked the job log file in the job_working_directory, and it read: Samtools Version: 0.1.18 (r982:295) SAM file converted to BAM which shows that sge knows the PATH of samtools. To double check it, I added samtools index to Galaxy, and it worked well. I am very confused why SGE knows the tool path but cannot run the job correctly. The system I am using is ubuntu on EC2 (the image id is: ami-999d49f0, which is the one used by StarCluster). I checked out the code from galaxy-dist on bitbucket. Other tools such as bwa and bowtie worked well using the same setting method(put env.sh in the tools directory to set the tool path) Thanks you very much, Cai --- 12年4月18日，周三, Jennifer Jackson 写道： > 发件人: Jennifer Jackson > 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: > samtools: not found > 收件人: "zhengqiu cai" > 抄送: galaxy-dev@lists.bx.psu.edu > 日期: 2012年4月18日,周三,上午2:12 > Hi Cai, > > Please double check that your env.sh file contains the > following: > > > > PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH" > export PATH > > > http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies > > > Hopefully this helps, > > Jen > Galaxy team > > On 4/16/12 12:43 PM, zhengqiu cai wrote: > > Hi All, > > > > I was trying to setup the Galaxy tool dependencies, and > I met the following problem when running sam-to-bam in > samtools: > > Traceback (most recent call last): > > File > "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", > line 336, in finish_job > > drm_job_state.job_wrapper.finish( > stdout, stderr ) > > File > "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py", > line 637, in finish > > dataset.set_meta( overwrite = False > ) > > File > "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py", > line 875, in set_meta > > return self.datatype.set_meta( > self, **kwd ) > > File > "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py", > line 179, in set_meta > > raise Exception, "Error Setting BAM > Metadata: %s" % stderr > > Exception: Error Setting BAM Metadata: /bin/sh: > samtools: not found > > > > > > I already set the path of samtools > > Below I pasted how I set the path: > > ubuntu@master:/mnt/galaxyTools/tools/samtools$ ll > > total 8 > > drwxr-xr-x 4 ubuntu ubuntu 65 > 2012-04-16 19:09 ./ > > drwxr-xr-x 38 ubuntu ubuntu 4096 2012-04-16 18:41 ../ > > drwxr-xr-x 2 ubuntu ubuntu 53 > 2012-04-16 19:08 0.1.12/ > > drwxr-xr-x 2 ubuntu ubuntu 53 > 2012-04-13 20:23 0.1.7/ > > lrwxrwxrwx 1 ubuntu ubuntu 38 > 2012-04-16 19:09 default -> > /mnt/galaxyTools/tools/samtools/0.1.12/ > > > > ubuntu@master:/mnt/galaxyTools/tools/samtools$ more > default/env.sh > > PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH > > > > I used the same way to set other programs, such as bwa > and bowtie, and they worked perfectly. Only samtools could > not work. > > I even used the full path of samtools in the wrapper > file sam_to_bam.py, and it was not working neither. >
Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found
Hi Cai, Please double check that your env.sh file contains the following: PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH" export PATH http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies Hopefully this helps, Jen Galaxy team On 4/16/12 12:43 PM, zhengqiu cai wrote: Hi All, I was trying to setup the Galaxy tool dependencies, and I met the following problem when running sam-to-bam in samtools: Traceback (most recent call last): File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 336, in finish_job drm_job_state.job_wrapper.finish( stdout, stderr ) File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, in finish dataset.set_meta( overwrite = False ) File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py", line 875, in set_meta return self.datatype.set_meta( self, **kwd ) File "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py", line 179, in set_meta raise Exception, "Error Setting BAM Metadata: %s" % stderr Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found I already set the path of samtools Below I pasted how I set the path: ubuntu@master:/mnt/galaxyTools/tools/samtools$ ll total 8 drwxr-xr-x 4 ubuntu ubuntu 65 2012-04-16 19:09 ./ drwxr-xr-x 38 ubuntu ubuntu 4096 2012-04-16 18:41 ../ drwxr-xr-x 2 ubuntu ubuntu 53 2012-04-16 19:08 0.1.12/ drwxr-xr-x 2 ubuntu ubuntu 53 2012-04-13 20:23 0.1.7/ lrwxrwxrwx 1 ubuntu ubuntu 38 2012-04-16 19:09 default -> /mnt/galaxyTools/tools/samtools/0.1.12/ ubuntu@master:/mnt/galaxyTools/tools/samtools$ more default/env.sh PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH I used the same way to set other programs, such as bwa and bowtie, and they worked perfectly. Only samtools could not work. I even used the full path of samtools in the wrapper file sam_to_bam.py, and it was not working neither. I am wondering if any one met the same problem, or some one can give me some hints. Thank you very much. Cai ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Jackson http://galaxyproject.org ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
