Re: [galaxy-dev] /bin/sh: samtools: not found-->>WORKAROUND
Yeah, I was using ['printenv| mail && samtools'] inside the subprocess.Popen(['samtools'],... in upload.py. Now right before in upload.py before the subprocess was called, I had /usr/bin/ in my PATH, but inside the subprocess the story was different. I just used the symbolic link to access samtools from something that remained in the PATH. I have no idea how this could happen, but I have noticed that galaxy does things with input and output and seems to manipulate the environment heavily--but why should a child process have a different environment when none was invoked? I'll figure that out later. I am still trying to convert some software to run with galaxy. On Mon, Apr 30, 2012 at 8:57 AM, Nate Coraor wrote: > On Apr 24, 2012, at 8:36 PM, Michael Moore wrote: > > > There is apparently a persistent problem with samtools which normally > lives at /usr/bin/samtools. I encountered a similar problem in Python when > uploading BAM files. > > > > I did not resolve the problem. I hacked for a while on binary.py in a > lib/ subdirectory and used os.system to send myself mail describing the > effective path at various points, and I added a missing > > > > logging.basicConfig() > > > > statement and scattered some log.WARNING statements strategically. All > this told me nothing. So I made a few symlinks to samtools. The one that > got things working was > > > > ln -s /usr/bin/samtools /home/galaxy/bin/samtools > > > > so--worked around but not resolved. > > Hi Michael, > > For tools that output BAM, samtools needs to be in your $PATH, or has to > be set up via the tool dependencies system. See the following for details: > >http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies > > For SGE, you can modify the $PATH used on the cluster in ~/.sge_request or > the file specified in the 'environment_setup_file' galaxy config option. > > --nate > > > > > Michael > > > > On Tue, Apr 17, 2012 at 12:15 PM, zhengqiu cai > wrote: > > Hi All, > > > > I submitted a job to convert sam to bam, and the job was running forever > without outputing the result. I then checked the log, and it read: > > Traceback (most recent call last): > > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", > line 336, in finish_job > >drm_job_state.job_wrapper.finish( stdout, stderr ) > > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py", line > 637, in finish > >dataset.set_meta( overwrite = False ) > > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py", line > 875, in set_meta > >return self.datatype.set_meta( self, **kwd ) > > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py", > line 179, in set_meta > >raise Exception, "Error Setting BAM Metadata: %s" % stderr > > Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found > > > > It means that the samtools is not in the PATH. I tried to set the PATH > in a couple of methods according the Galaxy documentation: > > 1. put the path in the env.sh in the tool directory and symbolink > default to the tool directory, e.g. default -> > =/mnt/galaxyTools/tools/samtools/0.1.18 > > 2. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in ~/.sge_request > > 3. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in > /path/sge_request > > > > none of them worked, and I got the above same problem. > > > > Then I checked the job log file in the job_working_directory, and it > read: > > Samtools Version: 0.1.18 (r982:295) > > SAM file converted to BAM > > > > which shows that sge knows the PATH of samtools. To double check it, I > added samtools index to Galaxy, and it worked well. I am very confused why > SGE knows the tool path but cannot run the job correctly. > > > > The system I am using is ubuntu on EC2. I checked out the code from > galaxy-dist on bitbucket. Other tools such as bwa and bowtie worked well > using the same setting method(put env.sh in the tools directory to set the > tool path) > > > > Thank you very much for any help or hints. > > > > Cai > > > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > > > http://lists.bx.psu.edu/ > > > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > > > http://lists.bx.psu.edu/ > > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] /bin/sh: samtools: not found-->>WORKAROUND
On Apr 24, 2012, at 8:36 PM, Michael Moore wrote: > There is apparently a persistent problem with samtools which normally lives > at /usr/bin/samtools. I encountered a similar problem in Python when > uploading BAM files. > > I did not resolve the problem. I hacked for a while on binary.py in a lib/ > subdirectory and used os.system to send myself mail describing the effective > path at various points, and I added a missing > > logging.basicConfig() > > statement and scattered some log.WARNING statements strategically. All this > told me nothing. So I made a few symlinks to samtools. The one that got > things working was > > ln -s /usr/bin/samtools /home/galaxy/bin/samtools > > so--worked around but not resolved. Hi Michael, For tools that output BAM, samtools needs to be in your $PATH, or has to be set up via the tool dependencies system. See the following for details: http://wiki.g2.bx.psu.edu/Admin/Config/Tool%20Dependencies For SGE, you can modify the $PATH used on the cluster in ~/.sge_request or the file specified in the 'environment_setup_file' galaxy config option. --nate > > Michael > > On Tue, Apr 17, 2012 at 12:15 PM, zhengqiu cai > wrote: > Hi All, > > I submitted a job to convert sam to bam, and the job was running forever > without outputing the result. I then checked the log, and it read: > Traceback (most recent call last): > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line > 336, in finish_job >drm_job_state.job_wrapper.finish( stdout, stderr ) > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py", line 637, > in finish >dataset.set_meta( overwrite = False ) > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py", line 875, > in set_meta >return self.datatype.set_meta( self, **kwd ) > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py", line > 179, in set_meta >raise Exception, "Error Setting BAM Metadata: %s" % stderr > Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found > > It means that the samtools is not in the PATH. I tried to set the PATH in a > couple of methods according the Galaxy documentation: > 1. put the path in the env.sh in the tool directory and symbolink default to > the tool directory, e.g. default -> =/mnt/galaxyTools/tools/samtools/0.1.18 > 2. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in ~/.sge_request > 3. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in /path/sge_request > > none of them worked, and I got the above same problem. > > Then I checked the job log file in the job_working_directory, and it read: > Samtools Version: 0.1.18 (r982:295) > SAM file converted to BAM > > which shows that sge knows the PATH of samtools. To double check it, I added > samtools index to Galaxy, and it worked well. I am very confused why SGE > knows the tool path but cannot run the job correctly. > > The system I am using is ubuntu on EC2. I checked out the code from > galaxy-dist on bitbucket. Other tools such as bwa and bowtie worked well > using the same setting method(put env.sh in the tools directory to set the > tool path) > > Thank you very much for any help or hints. > > Cai > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] /bin/sh: samtools: not found-->>WORKAROUND
There is apparently a persistent problem with samtools which normally lives at /usr/bin/samtools. I encountered a similar problem in Python when uploading BAM files. I did not resolve the problem. I hacked for a while on binary.py in a lib/ subdirectory and used os.system to send myself mail describing the effective path at various points, and I added a missing logging.basicConfig() statement and scattered some log.WARNING statements strategically. All this told me nothing. So I made a few symlinks to samtools. The one that got things working was ln -s /usr/bin/samtools /home/galaxy/bin/samtools so--worked around but not resolved. Michael On Tue, Apr 17, 2012 at 12:15 PM, zhengqiu cai wrote: > Hi All, > > I submitted a job to convert sam to bam, and the job was running forever > without outputing the result. I then checked the log, and it read: > Traceback (most recent call last): > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", > line 336, in finish_job >drm_job_state.job_wrapper.finish( stdout, stderr ) > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/jobs/__init__.py", line > 637, in finish >dataset.set_meta( overwrite = False ) > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/model/__init__.py", line > 875, in set_meta >return self.datatype.set_meta( self, **kwd ) > File "/mnt/galaxyTools/galaxy-dist/lib/galaxy/datatypes/binary.py", line > 179, in set_meta >raise Exception, "Error Setting BAM Metadata: %s" % stderr > Exception: Error Setting BAM Metadata: /bin/sh: samtools: not found > > It means that the samtools is not in the PATH. I tried to set the PATH in > a couple of methods according the Galaxy documentation: > 1. put the path in the env.sh in the tool directory and symbolink default > to the tool directory, e.g. default -> > =/mnt/galaxyTools/tools/samtools/0.1.18 > 2. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in ~/.sge_request > 3. put -v PATH=/mnt/galaxyTools/tools/samtools/0.1.18 in /path/sge_request > > none of them worked, and I got the above same problem. > > Then I checked the job log file in the job_working_directory, and it read: > Samtools Version: 0.1.18 (r982:295) > SAM file converted to BAM > > which shows that sge knows the PATH of samtools. To double check it, I > added samtools index to Galaxy, and it worked well. I am very confused why > SGE knows the tool path but cannot run the job correctly. > > The system I am using is ubuntu on EC2. I checked out the code from > galaxy-dist on bitbucket. Other tools such as bwa and bowtie worked well > using the same setting method(put env.sh in the tools directory to set the > tool path) > > Thank you very much for any help or hints. > > Cai > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
