Re: [galaxy-dev] Samtools running on submit-node and as a slurm job

[Marius van den Beek]

Hi Christopher,

If your galaxy is configured to set metadata externally then you shouldn't
see samtools index being run on the node that runs galaxy
(On a side-note galaxy will use pysam starting from release 18.01).
One possibility is that the metadata setting failed, and it has been tried
again internally.
You can control this with the `retry_metadata_internally` option in your
galaxy.yml / galaxy.ini
file.

Best,
Marius

On 24 April 2018 at 15:55, Previti 
wrote:

> Dear all,
>
> We found something strange with our Galaxy instance...
>
> Our Galaxy server is configured as a submit node using slurm to submit to
> our cluster.
>
> This works fine for the most part, but today we saw a samtools index job
> running "locally" on the submit node instead of a  cluster node.
>
> Is there any way of stopping this behavior? The submit nodes are not
> configured for data-intensive computation and everything runs significantly
> slower.
>
> I was running bamtools (splitting a bam-file into mapped into unmapped) at
> the time and the output bam-files were being indexed.
>
> Thanks and best regards,
>
> Christopher
> --
> *Dr. Christopher Previti*
> Genomics and Proteomics Core Facility
> High Throughput Sequencing (W190)
> Bioinformatician
>
> German Cancer Research Center (DKFZ)
> Foundation under Public Law
> Im Neuenheimer Feld 580
> 69120 Heidelberg
> Germany
> Room: B2.102 (INF580/TP3)
> Phone: +49 6221 42-4661
>
> christopher.prev...@dkfz.de 
> www.dkfz.de
>
> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
> VAT-ID No.: DE143293537
>
> Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
> Personen bestimmt, an die sie adressiert ist.
> Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
> Informationen enthalten. Sollten Sie nicht
> der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender
> und löschen Sie die Mitteilung.
> Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
> untersagt.
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Samtools running on submit-node and as a slurm job

[Previti]

Dear all,

We found something strange with our Galaxy instance...

Our Galaxy server is configured as a submit node using slurm to submit
to our cluster.

This works fine for the most part, but today we saw a samtools index job
running "locally" on the submit node instead of a  cluster node.

Is there any way of stopping this behavior? The submit nodes are not
configured for data-intensive computation and everything runs
significantly slower.

I was running bamtools (splitting a bam-file into mapped into unmapped)
at the time and the output bam-files were being indexed. 

Thanks and best regards,

Christopher

-- 
*Dr. Christopher Previti*
Genomics and Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

christopher.prev...@dkfz.de 
www.dkfz.de 

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
untersagt.


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Samtools problem in Docker

[Björn Grüning]

Hi Deepak,

this is still strange. But as a general remark, don't try to install
package_samtools_1_2 or package_* these are deprecated in favor of Conda
packages.

Glad it is working now for you!
Ciao,
Bjoern

On 15.08.2017 16:31, Deepak Tanwar wrote:
> Hi Björn,
> 
> I tried again from scratch.
> 
> 1. I have installed galaxy-stable image via Kitematic.
> 2. I install samtools: *package_samtools_1_2 (2015-07-17) iuc*
> 3. I have uploaded the bam file. Error:
> 
> Traceback (most recent call last): File
> "/galaxy-central/tools/data_source/upload.py", line 425, in 
> __main__() File "/galaxy-central/tools/data_source/upload.py", line 413,
> in __main__ add_file( dataset, registry, json_file, output_path ) File
> "/galaxy-central/tools/data_source/upload.py", line 340, in add_file if
> link_data_only == 'copy_files' and
> datatype.dataset_content_needs_grooming( output_path ): File
> "/galaxy-central/lib/galaxy/datatypes/binary.py", line 295, in
> dataset_content_needs_grooming version = self._get_samtools_version()
> File "/galaxy-central/lib/galaxy/datatypes/binary.py", line 244, in
> _get_samtools_version raise Exception(message) Exception: Attempting to
> use functionality requiring samtools, but it cannot be located on
> Galaxy's PATH.
> 
> 4. I restarted the galaxy and the uploaded the bam file. Error:
> 
> Traceback (most recent call last): File
> "/galaxy-central/tools/data_source/upload.py", line 425, in 
> __main__() File "/galaxy-central/tools/data_source/upload.py", line 413,
> in __main__ add_file( dataset, registry, json_file, output_path ) File
> "/galaxy-central/tools/data_source/upload.py", line 340, in add_file if
> link_data_only == 'copy_files' and
> datatype.dataset_content_needs_grooming( output_path ): File
> "/galaxy-central/lib/galaxy/datatypes/binary.py", line 295, in
> dataset_content_needs_grooming version = self._get_samtools_version()
> File "/galaxy-central/lib/galaxy/datatypes/binary.py", line 244, in
> _get_samtools_version raise Exception(message) Exception: Attempting to
> use functionality requiring samtools, but it cannot be located on
> Galaxy's PATH.
> 
> 
> 5. Now, I installed  *suite_samtools_1_2 (2015-04-27) devteam* and
> updated the packages. File uploaded successfully!
> 
> 
> 
> 
> 
>> On Aug 14, 2017, at 16:58, Björn Grüning > > wrote:
>>
>> Hi Deepak,
>>
>> as Peter mentioned the Galaxy Docker container can be used as it is.
>> Can you provide the original error message from the TS?
>>
>> Cheers,
>> Bjoern
>>
>> On 14.08.2017 18:17, Deepak Tanwar wrote:
>>> I am using https://hub.docker.com/r/bgruening/galaxy-stable
>>>
>>> I aded path to the samtools because I was getting error after installing
>>> samtools from toolshed.
>>>
>>> Thank you,
>>> Deepak
>>>
 On Aug 14, 2017, at 12:10, Peter Cock 
 > wrote:

 Galaxy itself uses samtools for many operations, which is different
 from when samtools is called as a dependency of a tool run by the
 Galaxy user.

 However, the Docker image ought to take care of that - can you
 give more details of which Docker image you are using (URL,
 version, etc)?

 Thanks,

 Peter


 On Mon, Aug 14, 2017 at 5:08 PM, Deepak Tanwar 
 > wrote:
> Hello everyone,
>
> I am facing problem in uploading bam files on Galaxy. I am using Docker
> image of Galaxy.
>
> I have installed samtools version 0.1.19 and added it to the path
> /tool_deps/samtools/0.1.19/iuc/package_samtools_0_1_19/c9bd782f5342/bin/samtools
>
> Error: Exception: Attempting to use functionality requiring samtools,
> but it
> cannot be located on Galaxy's PATH.
>
> Thank you,
> Deepak
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/
>>>
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>  https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>  http://galaxyproject.org/search/
>>>
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface 

Re: [galaxy-dev] Samtools problem in Docker

[Deepak Tanwar]

Hi Björn,

I tried again from scratch.

1. I have installed galaxy-stable image via Kitematic.
2. I install samtools: package_samtools_1_2 (2015-07-17) iuc
3. I have uploaded the bam file. Error:


Traceback (most recent call last):
  File "/galaxy-central/tools/data_source/upload.py", line 425, in 
__main__()
  File "/galaxy-central/tools/data_source/upload.py", line 413, in __main__
add_file( dataset, registry, json_file, output_path )
  File "/galaxy-central/tools/data_source/upload.py", line 340, in add_file
if link_data_only == 'copy_files' and 
datatype.dataset_content_needs_grooming( output_path ):
  File "/galaxy-central/lib/galaxy/datatypes/binary.py", line 295, in 
dataset_content_needs_grooming
version = self._get_samtools_version()
  File "/galaxy-central/lib/galaxy/datatypes/binary.py", line 244, in 
_get_samtools_version
raise Exception(message)
Exception: Attempting to use functionality requiring samtools, but it cannot be 
located on Galaxy's PATH.

4. I restarted the galaxy and the uploaded the bam file. Error:


Traceback (most recent call last):
  File "/galaxy-central/tools/data_source/upload.py", line 425, in 
__main__()
  File "/galaxy-central/tools/data_source/upload.py", line 413, in __main__
add_file( dataset, registry, json_file, output_path )
  File "/galaxy-central/tools/data_source/upload.py", line 340, in add_file
if link_data_only == 'copy_files' and 
datatype.dataset_content_needs_grooming( output_path ):
  File "/galaxy-central/lib/galaxy/datatypes/binary.py", line 295, in 
dataset_content_needs_grooming
version = self._get_samtools_version()
  File "/galaxy-central/lib/galaxy/datatypes/binary.py", line 244, in 
_get_samtools_version
raise Exception(message)
Exception: Attempting to use functionality requiring samtools, but it cannot be 
located on Galaxy's PATH.

5. Now, I installed  suite_samtools_1_2 (2015-04-27) devteam and updated the 
packages. File uploaded successfully!





On Aug 14, 2017, at 16:58, Björn Grüning 
> wrote:

Hi Deepak,

as Peter mentioned the Galaxy Docker container can be used as it is.
Can you provide the original error message from the TS?

Cheers,
Bjoern

On 14.08.2017 18:17, Deepak Tanwar wrote:
I am using https://hub.docker.com/r/bgruening/galaxy-stable

I aded path to the samtools because I was getting error after installing
samtools from toolshed.

Thank you,
Deepak

On Aug 14, 2017, at 12:10, Peter Cock 

> wrote:

Galaxy itself uses samtools for many operations, which is different
from when samtools is called as a dependency of a tool run by the
Galaxy user.

However, the Docker image ought to take care of that - can you
give more details of which Docker image you are using (URL,
version, etc)?

Thanks,

Peter


On Mon, Aug 14, 2017 at 5:08 PM, Deepak Tanwar 

> wrote:
Hello everyone,

I am facing problem in uploading bam files on Galaxy. I am using Docker
image of Galaxy.

I have installed samtools version 0.1.19 and added it to the path
/tool_deps/samtools/0.1.19/iuc/package_samtools_0_1_19/c9bd782f5342/bin/samtools

Error: Exception: Attempting to use functionality requiring samtools,
but it
cannot be located on Galaxy's PATH.

Thank you,
Deepak

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Samtools problem in Docker

[Björn Grüning]

Hi Deepak,

as Peter mentioned the Galaxy Docker container can be used as it is.
Can you provide the original error message from the TS?

Cheers,
Bjoern

On 14.08.2017 18:17, Deepak Tanwar wrote:
> I am using https://hub.docker.com/r/bgruening/galaxy-stable
> 
> I aded path to the samtools because I was getting error after installing
> samtools from toolshed.
> 
> Thank you,
> Deepak
> 
>> On Aug 14, 2017, at 12:10, Peter Cock > > wrote:
>>
>> Galaxy itself uses samtools for many operations, which is different
>> from when samtools is called as a dependency of a tool run by the
>> Galaxy user.
>>
>> However, the Docker image ought to take care of that - can you
>> give more details of which Docker image you are using (URL,
>> version, etc)?
>>
>> Thanks,
>>
>> Peter
>>
>>
>> On Mon, Aug 14, 2017 at 5:08 PM, Deepak Tanwar > > wrote:
>>> Hello everyone,
>>>
>>> I am facing problem in uploading bam files on Galaxy. I am using Docker
>>> image of Galaxy.
>>>
>>> I have installed samtools version 0.1.19 and added it to the path
>>> /tool_deps/samtools/0.1.19/iuc/package_samtools_0_1_19/c9bd782f5342/bin/samtools
>>>
>>> Error: Exception: Attempting to use functionality requiring samtools,
>>> but it
>>> cannot be located on Galaxy's PATH.
>>>
>>> Thank you,
>>> Deepak
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>  https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>  http://galaxyproject.org/search/
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Samtools problem in Docker

[Deepak Tanwar]

I am using https://hub.docker.com/r/bgruening/galaxy-stable

I aded path to the samtools because I was getting error after installing 
samtools from toolshed.

Thank you,
Deepak

On Aug 14, 2017, at 12:10, Peter Cock 
> wrote:

Galaxy itself uses samtools for many operations, which is different
from when samtools is called as a dependency of a tool run by the
Galaxy user.

However, the Docker image ought to take care of that - can you
give more details of which Docker image you are using (URL,
version, etc)?

Thanks,

Peter


On Mon, Aug 14, 2017 at 5:08 PM, Deepak Tanwar 
> wrote:
Hello everyone,

I am facing problem in uploading bam files on Galaxy. I am using Docker
image of Galaxy.

I have installed samtools version 0.1.19 and added it to the path
/tool_deps/samtools/0.1.19/iuc/package_samtools_0_1_19/c9bd782f5342/bin/samtools

Error: Exception: Attempting to use functionality requiring samtools, but it
cannot be located on Galaxy's PATH.

Thank you,
Deepak

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Samtools problem in Docker

[Peter Cock]

Galaxy itself uses samtools for many operations, which is different
from when samtools is called as a dependency of a tool run by the
Galaxy user.

However, the Docker image ought to take care of that - can you
give more details of which Docker image you are using (URL,
version, etc)?

Thanks,

Peter


On Mon, Aug 14, 2017 at 5:08 PM, Deepak Tanwar  wrote:
> Hello everyone,
>
> I am facing problem in uploading bam files on Galaxy. I am using Docker
> image of Galaxy.
>
> I have installed samtools version 0.1.19 and added it to the path
> /tool_deps/samtools/0.1.19/iuc/package_samtools_0_1_19/c9bd782f5342/bin/samtools
>
> Error: Exception: Attempting to use functionality requiring samtools, but it
> cannot be located on Galaxy's PATH.
>
> Thank you,
> Deepak
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Samtools problem in Docker

[Deepak Tanwar]

Hello everyone,

I am facing problem in uploading bam files on Galaxy. I am using Docker image 
of Galaxy.

I have installed samtools version 0.1.19 and added it to the path 
/tool_deps/samtools/0.1.19/iuc/package_samtools_0_1_19/c9bd782f5342/bin/samtools

Error: Exception: Attempting to use functionality requiring samtools, but it 
cannot be located on Galaxy's PATH.

Thank you,
Deepak
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] samtools

[Beginner TI]

Hi All,

I compiled samtools from source, and just realized that I don't have those
tool xml files. Tried to google but failed to find any samtools wrapper.
This is quite surprising since the main instance has it. Does anyone happen
to know where we could obtain those files that are used by galaxy main
instance? I don't want to use tool_shed and prefer to install all tools
manually. In that case, how could I access the tool xml files that are used
by tool_shed installation process?

Thanks,
Rui
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] samtools

[Martin Čech]

Hi Rui,

you can install the following suite which includes about a dozen of wrapped
samtools.

https://toolshed.g2.bx.psu.edu/view/devteam/suite_samtools_1_2/5b673ccc8747

Martin

On Mon, Sep 26, 2016 at 11:05 AM Beginner TI  wrote:

> I did see that if I click each individual tool, and then in the options, I
> could find the tool xml file, but this would require manually clicking each
> of it. Is there any way we could get these in batch? I shouldn't be the
> first or last person that want to make an identical clone of main galaxy
> instance, so I assume that there are some good/easy ways I'm missing?
>
> thanks,
> Rui
>
> On Sun, Sep 25, 2016 at 1:05 AM, Beginner TI  wrote:
>
>> Hi All,
>>
>> I compiled samtools from source, and just realized that I don't have
>> those tool xml files. Tried to google but failed to find any samtools
>> wrapper. This is quite surprising since the main instance has it. Does
>> anyone happen to know where we could obtain those files that are used by
>> galaxy main instance? I don't want to use tool_shed and prefer to install
>> all tools manually. In that case, how could I access the tool xml files
>> that are used by tool_shed installation process?
>>
>> Thanks,
>> Rui
>>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] samtools

[Beginner TI]

I did see that if I click each individual tool, and then in the options, I
could find the tool xml file, but this would require manually clicking each
of it. Is there any way we could get these in batch? I shouldn't be the
first or last person that want to make an identical clone of main galaxy
instance, so I assume that there are some good/easy ways I'm missing?

thanks,
Rui

On Sun, Sep 25, 2016 at 1:05 AM, Beginner TI  wrote:

> Hi All,
>
> I compiled samtools from source, and just realized that I don't have those
> tool xml files. Tried to google but failed to find any samtools wrapper.
> This is quite surprising since the main instance has it. Does anyone happen
> to know where we could obtain those files that are used by galaxy main
> instance? I don't want to use tool_shed and prefer to install all tools
> manually. In that case, how could I access the tool xml files that are used
> by tool_shed installation process?
>
> Thanks,
> Rui
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] samtools view tools

[Christian Brenninkmeijer]

Hi All,

Has anyone written galaxy tools for the samtools view functions such as

samtools view -f 0x0002 -F 0x0100 -uo

and/or

samtools view -uSo

Thanks
Christian
University of Manchester
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] samtools bam2fq --> samtools fastq / samtools fasta

[Peter Cock]

Hello all,

About a year ago I stated wrapping the "samtools bam2fq" command
from samtools v1.1 for use in Galaxy:

https://github.com/peterjc/pico_galaxy/tree/master/tools/samtools_bam2fq

That uncovered a number of bugs in "samtools bam2fq" which have
resulted in two new commands "samtools fastq" and "samtools fasta"
for converting a SAM/BAM/CRAM file into FASTQ or FASTA reads
with the release of samtools 1.3.

When time permits, I had planned to update my wrapper to target
samtools 1.3, probably as two separate tools which could potentially
hook into the Galaxy datatype conversion framework?

Where would be the best place for them? Under tools-devteam?

https://github.com/galaxyproject/tools-devteam/tree/master/tool_collections/samtools

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] samtools dependency changes ?

[John Chilton]

So it wasn't in 15.01 - it was before that but the local job runner
was changed to set metadata as part of the job (I think probably
sometime in 2014). When it was setting metadata externally it might
have used slightly different dependency resolution strategies that
would result in samtools being sourced for every metadata collection
and this would have resulted in the behavior you described.

I think a bigger problem was that the local job runner behaved so
differently than other job runners and that has been resolved. So now
we have to fix this for all of the job runners in a uniform way. I
think the path forward is something like this...

1. Right now there is this option in galaxy.ini called
enable_beta_tool_command_isolation - this causes the command generated
by the tool be executed in an environment (script) separate from the
metadata code generation environment. This needs to be enabled by
default so that a tool's dependencies don't interfere with the
metadata generation code.

2. The metadata generation code uniformly needs to source samtools
dependencies - like the old local job runner code did. Cannot do this
before the separation because it would prevent us from supporting
multiple versions of samtools for instance.

That shouldn't be a particularly hard process - there will probably be
bumps in the road and so I haven't enabled it by default yet.

So in the meantime I would just ensure Galaxy sources the required
packages on all your nodes and/or destinations. You can modify
environment variables directly in your job_conf.xml file and this will
work for any runner so this is easier than ever.

https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced#L253

-John













On Tue, Jun 16, 2015 at 2:11 PM, Wolfgang Maier
wolfgang.ma...@biologie.uni-freiburg.de wrote:
 Dear all,

 with older Galaxies (prior to latest_15.03 I think), you could satisfy
 Galaxy's samtools dependence for indexing bam files by having a samtools
 executable in tool-dependencies/samtools/default/bin (with the
 tool-dependencies directory declared as tool_dependency_dir in galaxy.ini of
 course).

 Now (checked with latest_15.03 and .05), this is not working any more! The
 executable will still be used during bam uploads, but not when a bam file
 gets created by a tool.

 The reason is that before the job runner (tested this with the local job
 runner only) used to build the dependency shell command for dependency
 'samtools' before finishing a job, but now the job wrapper finish method
 fails because it naively expects to find samtools on $PATH.

 Best,
 Wolfgang

 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Samtools 1.1 Dependency

[Dave Bouvier]

Hans,

An actions tag with specified architecture and operating system will 
only be processed if both attributes match, so that one would not be 
executed on your OSX system. As regards the error in the sed command, 
I've created a pull request for the IUC's samtools 1.1 (and others with 
the same error) tool dependency definition, once that is accepted I'll 
be able to update the toolshed with the working code. Thank you for 
alerting me to that error.


-
Dave Bouvier
http://galaxyproject.org
http://usegalaxy.org

On 07/09/2015 05:22 PM, Hans Vasquez-Gross wrote:

Hi All,

I have a galaxy installation on an OSX system that has been running fine
with our custom tools.  However, now we want to start using for some
simple BAC mapping.

When I go to the main toolshed to install bwa, it lists the following
dependencies.

Repository *package_bwa_0_7_10_039ea20639* revision *5b9aca1e1c07* owned
by *devteam*
Repository *package_samtools_1_1* revision *43f2fbec5d52* owned by *iuc*


However, when running through this process, it fails at the
package_samtools_1_1 stage with an Error. Looking at the
INSTALLATION.LOG i see the following.

#
sed -i 's/-lcurses/-lncurses/' Makefile
STDOUT

#

#
sed -i 's/-lcurses/-lncurses/' Makefile
STDERR
sed: 1: Makefile: invalid command code M
#

However, looking at the tool_dependencies.xml for this particular
package, I see the following. I've highlighted in yellow a key
configuration.

?xml version=1.0?
tool_dependency
 package name=samtools version=1.1
 install version=1.0
 actions_group
 actions architecture=x86_64 os=linux
 action
type=download_by_urlhttp://depot.galaxyproject.org/package/linux/x86_64/samtools/samtools-1.1-Linux-x86_64.tgz/action
 action type=move_directory_files
 source_directory./source_directory

destination_directory$INSTALL_DIR/destination_directory
 /action
 /actions
 actions
 package name=ncurses version=5.9
 repository changeset_revision=5e1760c773ba
name=package_ncurses_5_9 owner=iuc
prior_installation_required=True
toolshed=https://toolshed.g2.bx.psu.edu; /
 /package
 package name=zlib version=1.2.8
 repository changeset_revision=63a4a902cda2
name=package_zlib_1_2_8 owner=iuc prior_installation_required=True
toolshed=https://toolshed.g2.bx.psu.edu; /
 /package
 action
type=download_by_urlhttp://downloads.sourceforge.net/project/samtools/samtools/1.1/samtools-1.1.tar.bz2/action
 action type=set_environment_for_install
 repository changeset_revision=5e1760c773ba
name=package_ncurses_5_9 owner=iuc
toolshed=https://toolshed.g2.bx.psu.edu;
 package name=ncurses version=5.9 /
 /repository
 repository changeset_revision=63a4a902cda2
name=package_zlib_1_2_8 owner=iuc
toolshed=https://toolshed.g2.bx.psu.edu;
 package name=zlib version=1.2.8 /
 /repository
 /action
 action type=shell_commandsed -i
's/-lcurses/-lncurses/' Makefile/action
 action type=shell_commandsed -i -e
s|CFLAGS=\s*-g\s*-Wall\s*-O2\s*|CFLAGS= -g -Wall -O2
-I$NCURSES_INCLUDE_PATH/ncurses/ -I$NCURSES_INCLUDE_PATH
-L$NCURSES_LIB_PATH|g Makefile
/action
 action type=shell_commandmake/action
 action type=move_file
 sourcesamtools/source
 destination$INSTALL_DIR/bin/destination
 /action
 /actions
 action type=set_environment
 environment_variable action=prepend_to
name=PATH$INSTALL_DIR/bin/environment_variable
 environment_variable action=set_to
name=SAMTOOLS_ROOT_PATH$INSTALL_DIR/environment_variable
 /action
 /actions_group
 /install


So, to me, it sounds like this is trying to compile/install a
distribution meant for linux, but I'm on OSX.  I was able to
successfully install samtools 1.2 with no errors.

Is there anyway to update, or modify the code for the BWA installation
recipe to use as a dependency samtools 1.2 rather then 1.1?  Or does
anyone have any suggestions on how to get samtools 1.1 to work?

I tried making a bin/ directory and moving a precompiled samtools binary
for 1.1 in the directory.  But galaxy didn't seem to recognize that the
dependency was satisfied manually.

Any advice would be greatly appreciated.

Thank you,
-Hans

Re: [galaxy-dev] samtools dependency changes ?

[Bjoern Gruening]

Hi Wolfgang,

I only can tell you that we also have problems with handling BAM files 
properly in Galaxy.
Our issue is more due to unsorted BAM files, but as far as I understood 
this is because the metadata creation changed from using samtools to 
using pysam. Maybe this helps you in finding a workaround.


Ciao,
Bjoern


hmm, no replies yet, so is anybody able to reproduce this behavior and 
would you not consider it a bug?


Best,
Wolfgang


On 06/16/2015 03:11 PM, Wolfgang Maier wrote:

Dear all,

with older Galaxies (prior to latest_15.03 I think), you could satisfy
Galaxy's samtools dependence for indexing bam files by having a samtools
executable in tool-dependencies/samtools/default/bin (with the
tool-dependencies directory declared as tool_dependency_dir in
galaxy.ini of course).

Now (checked with latest_15.03 and .05), this is not working any more!
The executable will still be used during bam uploads, but not when a bam
file gets created by a tool.

The reason is that before the job runner (tested this with the local job
runner only) used to build the dependency shell command for dependency
'samtools' before finishing a job, but now the job wrapper finish method
fails because it naively expects to find samtools on $PATH.

Best,
Wolfgang

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] samtools dependency changes ?

[Wolfgang Maier]

hmm, no replies yet, so is anybody able to reproduce this behavior and 
would you not consider it a bug?


Best,
Wolfgang


On 06/16/2015 03:11 PM, Wolfgang Maier wrote:

Dear all,

with older Galaxies (prior to latest_15.03 I think), you could satisfy
Galaxy's samtools dependence for indexing bam files by having a samtools
executable in tool-dependencies/samtools/default/bin (with the
tool-dependencies directory declared as tool_dependency_dir in
galaxy.ini of course).

Now (checked with latest_15.03 and .05), this is not working any more!
The executable will still be used during bam uploads, but not when a bam
file gets created by a tool.

The reason is that before the job runner (tested this with the local job
runner only) used to build the dependency shell command for dependency
'samtools' before finishing a job, but now the job wrapper finish method
fails because it naively expects to find samtools on $PATH.

Best,
Wolfgang

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] samtools index tool

[Peter Cock]

Galaxy itself will call samtools index on BAM files, so
they should be there on disk already, See for example:

https://github.com/peterjc/pico_galaxy/blob/master/tools/samtools_idxstats/samtools_idxstats.xml

https://github.com/galaxyproject/tools-devteam/blob/master/tool_collections/samtools/samtools_idxstats/samtools_idxstats.xml

These demonstrate some different ways to find the BAI file.

Peter


On Fri, May 22, 2015 at 3:16 AM, Eleanor Morgan e.mor...@curtin.edu.au
wrote:

  Hi  all,

  Excuse me for the newbie question.  I am developing a local instance of
 galaxy to use primarily for teaching purposes.
 I  would like to index bam files produced in galaxy in order to view them
 in IGV.  I can’t seem to find the samtools index tool in the toolshed.  I
 can run samtools on the command line and generate indexes, but can find no
 way to add this functionality to the tool panel so that I can do it in
 Galaxy.  I also can’t find this functionality in the tool panel  (under
 Samtools or BAM tools) on usegalaxy.org.  Am I missing something?  Do I
 need to write my own tool?

  Regards,
 Eleanor

   Eleanor Morgan
 Senior Scientific Officer | School of Biomedical Sciences | Faculty of
 Health Sciences


 Curtin University
 Tel| +61 8 9266 7516

 Mob  | +61 408 004 369
 Fax| +61 8 9266 2342
 Email  | e.mor...@curtin.edu.au
 Web| biomed.curtin.edu.au



  Curtin University is a trademark of Curtin University of Technology.
 CRICOS Provider Code 00301J (WA), 02637B (NSW)




 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] samtools index tool

[Eleanor Morgan]

Hi  all,

Excuse me for the newbie question.  I am developing a local instance of galaxy 
to use primarily for teaching purposes.
I  would like to index bam files produced in galaxy in order to view them in 
IGV.  I can't seem to find the samtools index tool in the toolshed.  I can run 
samtools on the command line and generate indexes, but can find no way to add 
this functionality to the tool panel so that I can do it in Galaxy.  I also 
can't find this functionality in the tool panel  (under Samtools or BAM tools) 
on usegalaxy.org.  Am I missing something?  Do I need to write my own tool?

Regards,
Eleanor

Eleanor Morgan
Senior Scientific Officer | School of Biomedical Sciences | Faculty of Health 
Sciences

Curtin University
Tel| +61 8 9266 7516
Mob  | +61 408 004 369
Fax| +61 8 9266 2342
Email  | e.mor...@curtin.edu.aumailto:e.mor...@curtin.edu.au
Web| biomed.curtin.edu.au

[cid:ED62B336-732F-4D99-ABF8-AFACA1E1C7E3]

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)



___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] samtools installation problems

[Björn Grüning]

Hi Nathan,

please try the following.

ln -s /path/to/your/shed_tools/samtools/0.1.19/devteam/what/ever/
/path/to/your/shed_tools/samttools/default

The important part is that it is samtools  1.0 and that the default
link is a link to the folder in which you find the env.sh file.

I hope this will tell Galaxy how to find samtools for you. Otherwise
simply put the samtools binary into your $PATH.

Cheers and happy new year!
Bjoern

Am 31.12.2014 um 19:49 schrieb Nathan Kipniss:
 Hi All,
 
 New to galaxy here. I'm having an issue uploading bam files into a local
 instance of galaxy. I understand that it requires samtools, and I installed
 the appropriate dependency from the toolshed. However, I get the following
 error:
 
 Traceback (most recent call last):
   File /home/galaxy/galaxy-dist/tools/data_source/upload.py, line
 407, in module
 __main__()
   File /home/galaxy/galaxy-dist/tools/data_source/upload.py, line
 396, in __main__
 add_file( dataset, registry, json_file, output_path )
   File /home/galaxy/galaxy-dist/tools/data_source/upload.py, line
 324, in add_file
 if link_data_only == 'copy_files' and
 datatype.dataset_content_needs_grooming( output_path ):
   File /home/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py, line
 147, in dataset_content_needs_grooming
 version = self._get_samtools_version()
   File /home/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py, line
 129, in _get_samtools_version
 output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE,
 stdout=subprocess.PIPE ).communicate()[1]
   File /usr/lib/python2.7/subprocess.py, line 710, in __init__
 errread, errwrite)
   File /usr/lib/python2.7/subprocess.py, line 1327, in _execute_child
 raise child_exception
 OSError: [Errno 2] No such file or directory
 
 
 Which leads me to believe something is wrong with the install (can't
 find samtools). The dependency directory is also set. Is there another
 step I'm missing, or is it better to not install dependencies via the
 toolshed?
 
 Best,
 Nathan Kipniss
 
 
 
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/
 
 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/
 
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] samtools: command not found in bowtie2

[Anna Terry]

is bash_profile the best place for this, or is somewhere else better?

Cheers
Anna

On 26 November 2014 at 16:29, bjoern.gruen...@googlemail.com 
bjoern.gruen...@gmail.com wrote:

 The problem is that bowtie and Galaxy (can) use a different samtools
 version. Galaxy needs samtools to create metadata for BAM files, so this
 process is actually running ofter bowtie has finished.
 You need a samtools version globally accessible for Galaxy. Make sure you
 have smatools in your $PATH and than restart Galaxy.

 Cheers,
 Bjoern

 2014-11-26 17:14 GMT+01:00 Anna Terry iseegl...@googlemail.com:

 I just tried still the same problem.
 I ran bowtie2 to test, again I saw samtools come up on top, but same
 error:

 Traceback (most recent call last):
   File /home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py, line 
 129, in queue_job
 job_wrapper.finish( stdout, stderr, exit_code )
   File /home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py, line 998, in 
 finish
 dataset.datatype.set_meta( dataset, overwrite=False )  # call 
 datatype.set_meta directly for the initial set_meta call during dataset 
 creation
   File /home/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py, line 251, 
 in set_meta
 raise Exception, Error Setting BAM Metadata: %s % stderr
 Exception: Error Setting BAM Metadata: /bin/sh: samtools: command not found



 On 26 November 2014 at 16:01, Eric Kuyt eric.ku...@wur.nl wrote:

 ​Could you try going to Admin - Manage Installed tool shed repositories

 uninstall - reinstall ​package_samtools_0_1_18


 On 26 November 2014 at 16:55, Anna Terry iseegl...@googlemail.com
 wrote:

 yway, I am also getting the problem with samtools filter sam or bam





 --
 Central Veterinary Institute of Wageningen UR (CVI)
 Department of Infection Biology
 PO box 65, 8200 AB Lelystad, NL
 Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad

 Tel:  +31-(0)320-293391
 Fax: +31-(0)320-238153
 E-mail: eric.ku...@wur.nl
 Web: http://www.cvi.wur.nl



 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/



___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] samtools: command not found in bowtie2

[Anna Terry]

great, thanks

On 26 November 2014 at 16:39, bjoern.gruen...@googlemail.com 
bjoern.gruen...@gmail.com wrote:

 I would use .bashrc. You can also try to symlib the default dir to the
 actual /bin directory. Maybe this works for you.

 2014-11-26 17:36 GMT+01:00 Anna Terry iseegl...@googlemail.com:

 is bash_profile the best place for this, or is somewhere else better?

 Cheers
 Anna

 On 26 November 2014 at 16:29, bjoern.gruen...@googlemail.com 
 bjoern.gruen...@gmail.com wrote:

 The problem is that bowtie and Galaxy (can) use a different samtools
 version. Galaxy needs samtools to create metadata for BAM files, so this
 process is actually running ofter bowtie has finished.
 You need a samtools version globally accessible for Galaxy. Make sure
 you have smatools in your $PATH and than restart Galaxy.

 Cheers,
 Bjoern

 2014-11-26 17:14 GMT+01:00 Anna Terry iseegl...@googlemail.com:

 I just tried still the same problem.
 I ran bowtie2 to test, again I saw samtools come up on top, but same
 error:

 Traceback (most recent call last):
   File /home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/local.py, line 
 129, in queue_job
 job_wrapper.finish( stdout, stderr, exit_code )
   File /home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py, line 998, 
 in finish
 dataset.datatype.set_meta( dataset, overwrite=False )  # call 
 datatype.set_meta directly for the initial set_meta call during dataset 
 creation
   File /home/galaxy/galaxy-dist/lib/galaxy/datatypes/binary.py, line 
 251, in set_meta
 raise Exception, Error Setting BAM Metadata: %s % stderr
 Exception: Error Setting BAM Metadata: /bin/sh: samtools: command not found



 On 26 November 2014 at 16:01, Eric Kuyt eric.ku...@wur.nl wrote:

 ​Could you try going to Admin - Manage Installed tool shed
 repositories

 uninstall - reinstall ​package_samtools_0_1_18


 On 26 November 2014 at 16:55, Anna Terry iseegl...@googlemail.com
 wrote:

 yway, I am also getting the problem with samtools filter sam or bam





 --
 Central Veterinary Institute of Wageningen UR (CVI)
 Department of Infection Biology
 PO box 65, 8200 AB Lelystad, NL
 Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad

 Tel:  +31-(0)320-293391
 Fax: +31-(0)320-238153
 E-mail: eric.ku...@wur.nl
 Web: http://www.cvi.wur.nl



 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/





___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/