Hi Ross,
I have a cloudman version of Galaxy working well. What I am doing now is trying
to install Galaxy on starcluster, which is a SGE cluster. I should followed the
install method for the normal cluster/SGE install, not for the cloud.
Thank you for your quick reply.
Have a nice day.
Cai
--- 12年4月18日，周三, Ross <ross.laza...@gmail.com> 写道：
> 发件人: Ross <ross.laza...@gmail.com>
> 主题: Re: [galaxy-dev] Exception: Error Setting BAM Metadata: /bin/sh:
> samtools: not found
> 收件人: "zhengqiu cai" <caizhq2...@yahoo.com.cn>
> 抄送: "Jennifer Jackson" <j...@bx.psu.edu>, galaxy-dev@lists.bx.psu.edu
> 日期: 2012年4月18日,周三,下午11:46
> Hi, Cai.
> Apologies but I had not noticed any reference to the cloud
> distribution before - that changes everything - I was
> talking about a
> normal cluster/SGE install - you should definitely not need
> to do
> anything at all to the AMI - maybe someone with cloudman
> experience
> can be more helpful than I can.
>
> On Wed, Apr 18, 2012 at 11:39 AM, zhengqiu cai <caizhq2...@yahoo.com.cn>
> wrote:
> > Hi Ross,
> >
> > I did not compile samtools using the --prefix. The
> samtools I used comes with Galaxy AMI image, and I copied it
> to the same path on my EC2. How to compile samtools using
> --prefix? I just typed 'make', and there is not a
> configuration file. In the cloudman version of Galaxy (it is
> working well for me), only two executables (samtools and
> maq2sam-long) in the directory
> /mnt/galaxyTools/tools/samtools/0.1.12. I copies the same
> files to my EC2. Is the only compiled executable 'samtools'
> fully functional for the cluster(it is on the local
> machine)? or should I set some other libraries?
> >
> > Thank you,
> >
> > Cai
> >
> > --- 12年4月18日，周三, Ross <ross.laza...@gmail.com>
> 写道：
> >
> >> 发件人: Ross <ross.laza...@gmail.com>
> >> 主题: Re: [galaxy-dev] Exception: Error Setting
> BAM Metadata: /bin/sh: samtools: not found
> >> 收件人: "zhengqiu cai" <caizhq2...@yahoo.com.cn>
> >> 抄送: "Jennifer Jackson" <j...@bx.psu.edu>,
> galaxy-dev@lists.bx.psu.edu
> >> 日期: 2012年4月18日,周三,下午11:07
> >> Hi, Cai - sorry to hear it's still
> >> not working right.
> >>
> >> You've compiled/installed samtools using the
> --prefix
> >> configure option
> >> and tested that all the samtools based Galaxy
> wrappers work
> >> if you use
> >> the local job runner? That means that both the
> appropriate
> >> lib and bin
> >> directories are on the Galaxy user's path ready to
> be
> >> inherited by SGE
> >> with the -V SGE option.
> >>
> >> If the local job runner works, that seems to
> isolate the
> >> problem to
> >> SGE configuration and operation - the steps I've
> suggested
> >> seem to
> >> work for us so I've run out of ideas on why it's
> not working
> >> at your
> >> shop and hope somebody with more experience can
> help out.
> >>
> >>
> >> On Wed, Apr 18, 2012 at 10:50 AM, zhengqiu cai
> <caizhq2...@yahoo.com.cn>
> >> wrote:
> >> > Hi Ross,
> >> >
> >> > Thank you for your reply.
> >> > It still did not work.
> >> > I think SGE knows the path of samtools. Here
> is a test
> >> case I added to Galaxy, and it creates the index
> for fasta
> >> file. I pasted the samtools faidx wrapper and xml
> files
> >> below:
> >> >
> >>
> ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
> >> more samtools_faidx.pl
> >> > #!/usr/bin/perl
> >> > use strict;
> >> > use warnings;
> >> >
> >> > if (@ARGV < 1) {
> >> > die "Usage:
> >> perl $0 <input fasta file>\n";
> >> > }
> >> > `samtools faidx $ARGV[0]`;
> >> > `cp ${ARGV[0]}.fai $ARGV[1]`;
> >> > exit;
> >> >
> >> >
> >>
> ubuntu@master:/mnt/galaxyTools/galaxy-dist/tools/samtools$
> >> more samtools_faidx.xml
> >> > <tool id="samtools_faidx" name="samtools
> faidx"
> >> version="1.0.3">
> >> > <requirements>
> >> > <requirement
> >> type="package">samtools</requirement>
> >> > </requirements>
> >> > <description>creates fasta
> >> index</description>
> >> > <command interpreter="perl">
> >> > samtools_faidx.pl $input1
> $output1
> >> > </command>
> >> > <inputs>
> >> > <param name="input1"
> type="data"
> >> format="fasta" label="fasta file to be indexed"
> />
> >> > </inputs>
> >> > <outputs>
> >> > <data format="fai"
> name="output1"
> >> label="${tool.name} on ${on_string}: fasta" />
> >> > </outputs>
> >> > </tool>
> >> >
> >> > It works, which shows that SGE knows the path
> of
> >> samtools.
> >> >
> >> >
> >> > Cai
> >> >
> >> > --- 12年4月18日，周三, Ross <ross.laza...@gmail.com>
> >> 写道：
> >> >
> >> >> 发件人: Ross <ross.laza...@gmail.com>
> >> >> 主题: Re: [galaxy-dev] Exception: Error
> Setting
> >> BAM Metadata: /bin/sh: samtools: not found
> >> >> 收件人: "zhengqiu cai" <caizhq2...@yahoo.com.cn>
> >> >> 抄送: "Jennifer Jackson" <j...@bx.psu.edu>,
> >> galaxy-dev@lists.bx.psu.edu
> >> >> 日期:
> 2012年4月18日,周三,上午5:18
> >> >> Ah. Then you probably have a library
> >> >> path problem.
> >> >> Here's how we do it FWIW:
> >> >>
> >> >> Assume your want your shared cluster apps
> in
> >> >> /data/apps/bin
> >> >>
> >> >> To share the compiled libraries and
> binaries
> >> together,
> >> >> remember to
> >> >> compile apps like samtools using the
> >> >> --prefix /data/app
> >> >> flag so then you can add the /data/app/lib
> path to
> >> the
> >> >> galaxy user's
> >> >> path so the nodes running jobs inherit
> those paths
> >> because
> >> >> of the -V
> >> >> switch.
> >> >>
> >> >> I hope this works!
> >> >>
> >> >>
> >> >>
> >> >> On Tue, Apr 17, 2012 at 5:10 PM, zhengqiu
> cai
> >> <caizhq2...@yahoo.com.cn>
> >> >> wrote:
> >> >> > Hi Ross,
> >> >> >
> >> >> > Thank you for your reply.
> >> >> > I tried it, and it was still not
> working.
> >> >> > Only samtools has this problem, and
> the other
> >> tools
> >> >> works fine.
> >> >> >
> >> >> > samtools was compiled from source
> correctly.
> >> If I
> >> >> install samtools using "sudo apt-get
> install
> >> samtools", this
> >> >> problem is gone, but this is not a
> solution for
> >> since it
> >> >> does not work across the cluster.
> >> >> >
> >> >> >
> >> >> > Cai
> >> >> >
> >> >> > --- 12年4月18日，周三, Ross
> <ross.laza...@gmail.com>
> >> >> 写道：
> >> >> >
> >> >> >> 发件人: Ross <ross.laza...@gmail.com>
> >> >> >> 主题: Re: [galaxy-dev]
> Exception: Error
> >> Setting
> >> >> BAM Metadata: /bin/sh: samtools: not
> found
> >> >> >> 收件人: "zhengqiu cai" <caizhq2...@yahoo.com.cn>
> >> >> >> 抄送: "Jennifer Jackson" <j...@bx.psu.edu>,
> >> >> galaxy-dev@lists.bx.psu.edu
> >> >> >> 日期:
> >> 2012年4月18日,周三,上午4:59
> >> >> >> Cai - if you are using SGE you
> >> >> >> probably want to pass the Galaxy
> user's
> >> >> >> path to the job using the
> >> >> >> -V
> >> >> >> switch in the system wide (or
> Galaxy
> >> user's home
> >> >> directory)
> >> >> >> sge_request file
> >> >> >>
> >> >> >> eg I have
> >> >> >> rlazarus@iaas1-int:~$ cat
> >> >> >>
> >> /var/lib/gridengine/default/common/sge_request
> >> >> >> -cwd
> >> >> >> -V
> >> >> >>
> >> >> >> Without that switch, the job gets
> no
> >> path.
> >> >> >> With -V, it inherits the job
> submitter's
> >> full
> >> >> path.
> >> >> >>
> >> >> >> On Tue, Apr 17, 2012 at 4:26 PM,
> zhengqiu
> >> cai
> >> >> <caizhq2...@yahoo.com.cn>
> >> >> >> wrote:
> >> >> >> > Hi Jen,
> >> >> >> >
> >> >> >> > Thank you for your quick
> response.
> >> >> >> > I changed the env.sh file to
> what
> >> you
> >> >> seggested, and
> >> >> >> the problem still existed.
> >> >> >> > I am pasting more details
> below:
> >> >> >> >
> >> >> >> > I submitted a job to convert
> sam to
> >> bam, and
> >> >> the job
> >> >> >> was running forever without
> outputing the
> >> result. I
> >> >> then
> >> >> >> checked the log, and it read:
> >> >> >> > Traceback (most recent call
> last):
> >> >> >> > File
> >> >> >>
> >> >>
> >>
> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
> >> >> >> line 336, in finish_job
> >> >> >> >
> >> drm_job_state.job_wrapper.finish(
> >> >> stdout,
> >> >> >> stderr )
> >> >> >> > File
> >> >> >>
> >> >>
> >>
> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py",
> >> >> >> line 637, in finish
> >> >> >> >
> dataset.set_meta(
> >> overwrite =
> >> >> False )
> >> >> >> > File
> >> >> >>
> >> >>
> >>
> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py",
> >> >> >> line 875, in set_meta
> >> >> >> > return
> >> self.datatype.set_meta(
> >> >> self, **kwd
> >> >> >> )
> >> >> >> > File
> >> >> >>
> >> >>
> >>
> "/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py",
> >> >> >> line 179, in set_meta
> >> >> >> > raise
> Exception, "Error
> >> Setting
> >> >> BAM
> >> >> >> Metadata: %s" % stderr
> >> >> >> > Exception: Error Setting
> BAM
> >> Metadata:
> >> >> /bin/sh:
> >> >> >> samtools: not found
> >> >> >> >
> >> >> >> > It means that the samtools
> is not in
> >> the PATH.
> >> >> I tried
> >> >> >> to set the PATH in a couple of
> methods
> >> according
> >> >> the Galaxy
> >> >> >> documentation:
> >> >> >> > 1. put the path in the
> env.sh in the
> >> tool
> >> >> directory and
> >> >> >> symbolink default to the tool
> directory,
> >> e.g.
> >> >> default ->
> >> >> >>
> =/mnt/galaxyTools/tools/samtools/0.1.18
> >> >> >> > 2. put -v
> >> >>
> PATH=/mnt/galaxyTools/tools/samtools/0.1.18
> >> >> >> in ~/.sge_request
> >> >> >> > 3. put -v
> >> >>
> PATH=/mnt/galaxyTools/tools/samtools/0.1.18
> >> >> >> in /path/sge_request
> >> >> >> >
> >> >> >> > none of them worked, and I
> got the
> >> above same
> >> >> problem.
> >> >> >> >
> >> >> >> > Then I checked the job log
> file in
> >> the
> >> >> >> job_working_directory, and it
> read:
> >> >> >> > Samtools Version: 0.1.18
> (r982:295)
> >> >> >> > SAM file converted to BAM
> >> >> >> >
> >> >> >> > which shows that sge knows
> the PATH
> >> of
> >> >> samtools. To
> >> >> >> double check it, I added samtools
> index to
> >> Galaxy,
> >> >> and it
> >> >> >> worked well. I am very confused
> why SGE
> >> knows the
> >> >> tool path
> >> >> >> but cannot run the job
> correctly.
> >> >> >> >
> >> >> >> > The system I am using is
> ubuntu on
> >> EC2 (the
> >> >> image id
> >> >> >> is: ami-999d49f0, which is the
> one used
> >> by
> >> >> StarCluster). I
> >> >> >> checked out the code from
> galaxy-dist on
> >> bitbucket.
> >> >> Other
> >> >> >> tools such as bwa and bowtie
> worked well
> >> using the
> >> >> same
> >> >> >> setting method(put env.sh in the
> tools
> >> directory to
> >> >> set the
> >> >> >> tool path)
> >> >> >> >
> >> >> >> >
> >> >> >> > Thanks you very much,
> >> >> >> >
> >> >> >> > Cai
> >> >> >> > --- 12年4月18日，周三,
> Jennifer
> >> Jackson
> >> >> <j...@bx.psu.edu>
> >> >> >> 写道：
> >> >> >> >
> >> >> >> >> 发件人: Jennifer
> Jackson
> >> <j...@bx.psu.edu>
> >> >> >> >> 主题: Re:
> [galaxy-dev]
> >> Exception: Error
> >> >> Setting
> >> >> >> BAM Metadata: /bin/sh: samtools:
> not
> >> found
> >> >> >> >> 收件人: "zhengqiu
> cai" <caizhq2...@yahoo.com.cn>
> >> >> >> >> 抄送: galaxy-dev@lists.bx.psu.edu
> >> >> >> >> 日期:
> >> >> 2012年4月18日,周三,上午2:12
> >> >> >> >> Hi Cai,
> >> >> >> >>
> >> >> >> >> Please double check that
> your
> >> env.sh file
> >> >> contains
> >> >> >> the
> >> >> >> >> following:
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> PATH="/mnt/galaxyTools/tools/samtools/0.1.12:$PATH"
> >> >> >> >>
> export PATH
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Hopefully this helps,
> >> >> >> >>
> >> >> >> >> Jen
> >> >> >> >> Galaxy team
> >> >> >> >>
> >> >> >> >> On 4/16/12 12:43 PM,
> zhengqiu cai
> >> wrote:
> >> >> >> >> > Hi All,
> >> >> >> >> >
> >> >> >> >> > I was trying to
> setup the
> >> Galaxy
> >> >> tool
> >> >> >> dependencies, and
> >> >> >> >> I met the following
> problem when
> >> running
> >> >> sam-to-bam
> >> >> >> in
> >> >> >> >> samtools:
> >> >> >> >> > Traceback (most
> recent call
> >> last):
> >> >> >> >> > File
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/runners/drmaa.py",
> >> >> >> >> line 336, in finish_job
> >> >> >> >> >
> >> >> >>
> drm_job_state.job_wrapper.finish(
> >> >> >> >> stdout, stderr )
> >> >> >> >> > File
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/jobs/__init__.py",
> >> >> >> >> line 637, in finish
> >> >> >> >> >
> >> >> dataset.set_meta(
> >> >> >> overwrite = False
> >> >> >> >> )
> >> >> >> >> > File
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/model/__init__.py",
> >> >> >> >> line 875, in set_meta
> >> >> >> >> >
> return
> >> >> >> self.datatype.set_meta(
> >> >> >> >> self, **kwd )
> >> >> >> >> > File
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> "/mnt/galaxyTools/cai_galaxy-dist/lib/galaxy/datatypes/binary.py",
> >> >> >> >> line 179, in set_meta
> >> >> >> >> >
> raise
> >> Exception,
> >> >> "Error
> >> >> >> Setting BAM
> >> >> >> >> Metadata: %s" % stderr
> >> >> >> >> > Exception: Error
> Setting
> >> BAM
> >> >> Metadata:
> >> >> >> /bin/sh:
> >> >> >> >> samtools: not found
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > I already set the
> path of
> >> samtools
> >> >> >> >> > Below I pasted how
> I set the
> >> path:
> >> >> >> >> >
> >> >>
> ubuntu@master:/mnt/galaxyTools/tools/samtools$
> >> >> >> ll
> >> >> >> >> > total 8
> >> >> >> >> > drwxr-xr-x 4
> ubuntu
> >> >> >> ubuntu 65
> >> >> >> >> 2012-04-16 19:09 ./
> >> >> >> >> > drwxr-xr-x 38
> ubuntu ubuntu
> >> 4096
> >> >> 2012-04-16
> >> >> >> 18:41 ../
> >> >> >> >> > drwxr-xr-x 2
> ubuntu
> >> >> >> ubuntu 53
> >> >> >> >> 2012-04-16 19:08
> 0.1.12/
> >> >> >> >> > drwxr-xr-x 2
> ubuntu
> >> >> >> ubuntu 53
> >> >> >> >> 2012-04-13 20:23 0.1.7/
> >> >> >> >> > lrwxrwxrwx 1
> ubuntu
> >> >> >> ubuntu 38
> >> >> >> >> 2012-04-16 19:09 default
> ->
> >> >> >> >>
> >> /mnt/galaxyTools/tools/samtools/0.1.12/
> >> >> >> >> >
> >> >> >> >> >
> >> >>
> ubuntu@master:/mnt/galaxyTools/tools/samtools$
> >> >> >> more
> >> >> >> >> default/env.sh
> >> >> >> >> >
> >> >> >>
> >> PATH=/mnt/galaxyTools/tools/samtools/0.1.12:$PATH
> >> >> >> >> >
> >> >> >> >> > I used the same way
> to set
> >> other
> >> >> programs,
> >> >> >> such as bwa
> >> >> >> >> and bowtie, and they
> worked
> >> perfectly.
> >> >> Only
> >> >> >> samtools could
> >> >> >> >> not work.
> >> >> >> >> > I even used the
> full path of
> >> samtools
> >> >> in the
> >> >> >> wrapper
> >> >> >> >> file sam_to_bam.py, and
> it was
> >> not
> >> >> working
> >> >> >> neither.
> >> >> >> >> >
> >> >> >> >> > I am wondering if
> any one
> >> met the
> >> >> same
> >> >> >> problem, or some
> >> >> >> >> one can give me some
> hints.
> >> >> >> >> >
> >> >> >> >> > Thank you very
> much.
> >> >> >> >> >
> >> >> >> >> > Cai
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >>
> >> >> >>
> >> >>
> >>
> ___________________________________________________________
> >> >> >> >> > Please keep all
> replies on
> >> the list
> >> >> by using
> >> >> >> "reply
> >> >> >> >> all"
> >> >> >> >> > in your mail
> client.
> >> To manage
> >> >> your
> >> >> >> subscriptions
> >> >> >> >> to this
> >> >> >> >> > and other Galaxy
> lists,
> >> please use
> >> >> the
> >> >> >> interface at:
> >> >> >> >> >
> >> >> >> >> > http://lists.bx.psu.edu/
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> Jennifer Jackson
> >> >> >> >> http://galaxyproject.org
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >>
> >>
> ___________________________________________________________
> >> >> >> > Please keep all replies on
> the list
> >> by using
> >> >> "reply
> >> >> >> all"
> >> >> >> > in your mail client.
> To manage
> >> your
> >> >> subscriptions
> >> >> >> to this
> >> >> >> > and other Galaxy lists,
> please use
> >> the
> >> >> interface at:
> >> >> >> >
> >> >> >> > http://lists.bx.psu.edu/
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Ross Lazarus MBBS MPH;
> >> >> >> Associate Professor, Harvard
> Medical
> >> School;
> >> >> >> Head, Medical Bioinformatics,
> BakerIDI;
> >> Tel: +61
> >> >> 385321444;
> >> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Ross Lazarus MBBS MPH;
> >> >> Associate Professor, Harvard Medical
> School;
> >> >> Head, Medical Bioinformatics, BakerIDI;
> Tel: +61
> >> 385321444;
> >> >>
> >>
> >>
> >>
> >> --
> >> Ross Lazarus MBBS MPH;
> >> Associate Professor, Harvard Medical School;
> >> Head, Medical Bioinformatics, BakerIDI; Tel: +61
> 385321444;
> >>
>
>
>
> --
> Ross Lazarus MBBS MPH;
> Associate Professor, Harvard Medical School;
> Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
