subject:"\[galaxy\-dev\] Test Toolshed Biopython package dependency Atlas fails to install $Was\: Re\: UnboundLocalError\: local variable 'prior_installation_required' referenced before assignment$"

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-30 Thread John Chilton

On Sun, Sep 29, 2013 at 10:43 PM, Guest, Simon
simon.gu...@agresearch.co.nz wrote:
 At Fri, 27 Sep 2013 00:23:37 -0500,
 John Chilton wrote:

 Simon,

 What is the advantage of putting that XML definition in the tool shed?
 It is not 100% true because of prior_install_required dependencies,
 but for the most part sourcing/load the environment for tools is a
 Galaxy problem, not so much a tool shed one. What if we did this
 instead?

 Add an option to Galaxy's universe_wsgi.ini with the following default:

 tool_dependency_resolution_order = gx_package_manual, gx_package_toolshed

 Which essentially implements my idea above, with James' additional
 configuration. But which can be overridden as:

 tool_dependency_resolution_order = plugin_module, gx_package_manual,
 gx_package_toolshed

 If set this way then placing requirement
 package=0.5.9bwa/requirement in a tool will result in the module
 bwa/0.5.9 being loaded if it is 'avail'able, else it will check for a
 manually installed env.sh (which is where MSI is currently putting its
 module loads), and else it will fallback to source the tool shed
 installed dependency.

 I feel like this will give you everything you want without any extra
 XML or configuration. Let me know if I am wrong.

 Hi John,

 I think you're right.  Your scheme is neater than what I was
 proposing.  The extra flexibility I was aiming at via some toolshed
 XML stuff appears not to be necessary upon further reflection.  (I
 wanted to ensure a Galaxy admin could just install some RPMs, install
 a toolshed tool, and have everything resolve nicely.  You seem to have
 achieved that with your scheme.)

 I haven't had a chance to try your code yet, but as soon as I can I
 will do so, and get back to you.

 If I leave out gx_package_toolshed altogether from
 tool_dependency_resolution_order, will the tool installation in Galaxy
 simply fail with a nice error message if the environment module and/or
 env.sh files are not found?  (This is what I would like, as it would
 serve as a prompt to the Galaxy admin to install some extra RPMs or
 whatever.)


First tool_dependency_resolution_order is not how I landed up
implementing it. Each resolver may need parameters so I decided to
do an XML configuration kind of like Nate's job_conf.xml stuff. So to
leave the tool shed stuff off you can just add
dependency_resolvers_conf.xml to your Galaxy root.

dependency_resolvers
  galaxy_packages /
  modules /
  modules versionless=true /
/dependency

You can also just leave galaxy_packages off if you are not using any
manually installed env.sh files.

The first modules resolver will try to match each tag like this:

  requirement type=package version=3.0.1R/requirement

with a module load like:

  R/3.0.1

if that exact module is available. The second versionless form of the
tag will result in Galaxy just falling back to auto-load whatever the
default R module is if the exact version specified (3.0.1) is not
available.

This code doesn't touch the tool shed at all, so if you check to
install dependencies, they will still be installed and if you don't
they will not be installed. This configuration just tells the Galaxy
how to use the dependencies that happen to be installed. You will need
to review installed tools and make sure you have matching modules.
Additionally, some tools may use this set_environment requirement
type, I am not sure how to implement this or how prevalent its use is.

-John


 Will this also work for those toolshed packages which bundle their
 package definitions (to download, make and install the tool
 dependency) along with their wrappers?

Can you opt not to install packages for such repositories? Either way,
the answer is I guess the same as above, the tool shed is unchanged,
its just how Galaxy utilizes the installed dependencies that is being
modified here.


 Thanks for working on this.

 cheers,
 Simon

 ===
 Attention: The information contained in this message and/or attachments
 from AgResearch Limited is intended only for the persons or entities
 to which it is addressed and may contain confidential and/or privileged
 material. Any review, retransmission, dissemination or other use of, or
 taking of any action in reliance upon, this information by persons or
 entities other than the intended recipients is prohibited by AgResearch
 Limited. If you have received this message in error, please notify the
 sender immediately.
 ===
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-29 Thread John Chilton

I have issued a pull request with a specific implementation of these ideas:

https://bitbucket.org/galaxy/galaxy-central/pull-request/227/tool-dependency-resolver-plugins/diff

Please feel free to comment.

-John

On Fri, Sep 27, 2013 at 12:23 AM, John Chilton chil...@msi.umn.edu wrote:
Simon,

What is the advantage of putting that XML definition in the tool shed?
It is not 100% true because of prior_install_required dependencies,
but for the most part sourcing/load the environment for tools is a
Galaxy problem, not so much a tool shed one. What if we did this
instead?

Add an option to Galaxy's universe_wsgi.ini with the following default:

tool_dependency_resolution_order = gx_package_manual, gx_package_toolshed

Which essentially implements my idea above, with James' additional
configuration. But which can be overridden as:

tool_dependency_resolution_order = plugin_module, gx_package_manual,
gx_package_toolshed

If set this way then placing requirement
package=0.5.9bwa/requirement in a tool will result in the module
bwa/0.5.9 being loaded if it is 'avail'able, else it will check for a
manually installed env.sh (which is where MSI is currently putting its
module loads), and else it will fallback to source the tool shed
installed dependency.

I feel like this will give you everything you want without any extra
XML or configuration. Let me know if I am wrong.

-John

On Thu, Sep 26, 2013 at 11:39 PM, Guest, Simon
simon.gu...@agresearch.co.nz wrote:
At Thu, 26 Sep 2013 22:03:09 -0400,
Greg Von Kuster wrote:

Hi John,

On Sep 26, 2013, at 9:15 PM, John Chilton chil...@msi.umn.edu wrote:

I was not even thinking we needed to modify the tool shed to implement
this. I was hoping (?) you could just modify:

Nothing in the Tool Shed itself would be affected or require modification
for this new feature as this is completely on the Galaxy side.

lib/galaxy/tools/deps/__init__.py

to implement this. If some tool contains the tag

requirement type=package version=1.7.1numpy/requirement

then if there is a manually installed tool_dependency in
`tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence
over the tool shed installed version (would that be something like
`tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me
know if this is way off base.

This is a possibility perhaps, but there seems to be a potential weakness
in that it doesn't require the Tool Dependency object to exist since the
tool will function without a installed dependency from the Tool Shed. Or,
if the installed depndency s required, then it is meaningless because it
won't be used. If the former case, then the tool dependency cannot be
shared via the Tool Shed's dependency mechanism because none of the
relationships will be defined since nothing is installed. Wouldn't it be
better to allow the Galaxy admin to point the ToolDependency object to a
specified binary on disk? In this way, all relationships defined by Tool
Shed installations will work as expected, with all contained tools with
that dependency point to that same shared location on disk.

Hi Greg, John,

What you are discussing is pretty close to what I am after, and what I
am prepared to spend some time working on, if that can help. This is
what I have posted about a couple of times previously. The option to
use a pre-installed package rather than a Galaxy installed one I think
would be very useful in general. I think it can be done in a way that
doesn't break the tool dependency model.

I envisage providing access to existing programs on disk by loading
the relevant environment module. Platforms will differ greatly on
where they have programs installed (usr/local/bwa-0.5.9,
/usr/libexec/bwa-0.5.9, etc.). However, we could arrange to have a
conventionally named environment module available, so Galaxy just has
to be told somehow to do a module load bwa/0.5.9, say, prior to
trying to run that tool, which will then be found on the PATH.

Hooking this in without killing the dependency on named and versioned
toolshed repos could work like this. In parallel with the repo sets
named, e.g. package_bwa_0_5_9, which download and build the package,
we have a set named like native_package_bwa_0_5_9. This could have a
tool_dependencies.xml file like this:

?xml version=1.0?
tool_dependency
native_package name=bwa version=0.5.9
actions
action type=load_module
module name=bwa version=0.5.9/module
/action
/actions
readme
Uses native BWA via environment module bwa/0.5.9
/readme
/native_package
/tool_dependency

What this does is ensure that any repo which requires version 0.5.9 of
bwa can have this dependency resolved by either package_bwa_0_5_9, or
native_package_bwa_0_5_9.

I envisage a Galaxy configuration setting that enables native package
and/or Galaxy package

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-29 Thread Björn Grüning

Hi Peter and John,

thanks for your comments and thanks for working on the patch John.

 On Thu, Sep 26, 2013 at 10:27 PM, John Chilton chil...@msi.umn.edu wrote:
  My recommendation would be make the tool dependency install work on as
  many platforms as you can and not try to optimize in such a way that
  it is not going to work - i.e. favor reproduciblity over performance.
 
 Reproducibility versus speed is a particular issue with floating point
 libraries - NumPy using ATLAS vs OpenBLAST vs Intel MKL vs
 just plain NumPy will probably all give slightly different answers
 (on some tasks), and at different speed.
 
 In this case, for simplicity I would advocate plain NumPy, without
 worrying about needing ATLAS. For packages needing NumPy
 with ALTAS, perhaps a new Tool Shed entry could be created,
 package_numpy_1_7_with_atlas or similar (based on the
 current configuration)?

I updated R, numpy, scipy, scikit and removed the atlas dependency. It
seems to work fine for the ChemicalToolBox. I do not remove the lapack
dependency, because I did not get any complains until now. I also
created new repositories *_with_atlas in my galaxytools repository, if
anyone is interested in atlas dependent packages.

Lets concentrate on reproducibility and leave out the speed improvements
for now. I admit it was do ambitious/idealistic.

Have a nice weekend,
Bjoern

 Peter



___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-29 Thread Peter Cock

On Sun, Sep 29, 2013 at 1:17 PM, Björn Grüning
bjoern.gruen...@pharmazie.uni-freiburg.de wrote:
 Hi Peter and John,

 thanks for your comments and thanks for working on the patch John.

 Peter wrote:
 In this case, for simplicity I would advocate plain NumPy, without
 worrying about needing ATLAS. For packages needing NumPy
 with ALTAS, perhaps a new Tool Shed entry could be created,
 package_numpy_1_7_with_atlas or similar (based on the
 current configuration)?

 I updated R, numpy, scipy, scikit and removed the atlas dependency. It
 seems to work fine for the ChemicalToolBox. I do not remove the lapack
 dependency, because I did not get any complains until now.

Great. Hopefully all my tools with a NumPy dependency via
Biopython will pickup this change automatically when the next
nightly Tool Shed tests are run.

(That's what I hope will happen, otherwise I'll need to bump the
dependency revision in the Biopython package  all the tools
calling that).

 I also
 created new repositories *_with_atlas in my galaxytools repository, if
 anyone is interested in atlas dependent packages.

Sounds good :)

 Lets concentrate on reproducibility and leave out the speed
 improvements for now. I admit it was do ambitious/idealistic.

It is nice to aim high, but I think here practicality wins.

 Have a nice weekend,
 Bjoern

You too,

Peter

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-29 Thread Guest, Simon

At Fri, 27 Sep 2013 00:23:37 -0500,
John Chilton wrote:
 
 Simon,
 
 What is the advantage of putting that XML definition in the tool shed?
 It is not 100% true because of prior_install_required dependencies,
 but for the most part sourcing/load the environment for tools is a
 Galaxy problem, not so much a tool shed one. What if we did this
 instead?
 
 Add an option to Galaxy's universe_wsgi.ini with the following default:
 
 tool_dependency_resolution_order = gx_package_manual, gx_package_toolshed
 
 Which essentially implements my idea above, with James' additional
 configuration. But which can be overridden as:
 
 tool_dependency_resolution_order = plugin_module, gx_package_manual,
 gx_package_toolshed
 
 If set this way then placing requirement
 package=0.5.9bwa/requirement in a tool will result in the module
 bwa/0.5.9 being loaded if it is 'avail'able, else it will check for a
 manually installed env.sh (which is where MSI is currently putting its
 module loads), and else it will fallback to source the tool shed
 installed dependency.
 
 I feel like this will give you everything you want without any extra
 XML or configuration. Let me know if I am wrong.

Hi John,

I think you're right.  Your scheme is neater than what I was
proposing.  The extra flexibility I was aiming at via some toolshed
XML stuff appears not to be necessary upon further reflection.  (I
wanted to ensure a Galaxy admin could just install some RPMs, install
a toolshed tool, and have everything resolve nicely.  You seem to have
achieved that with your scheme.)

I haven't had a chance to try your code yet, but as soon as I can I
will do so, and get back to you.

If I leave out gx_package_toolshed altogether from
tool_dependency_resolution_order, will the tool installation in Galaxy
simply fail with a nice error message if the environment module and/or
env.sh files are not found?  (This is what I would like, as it would
serve as a prompt to the Galaxy admin to install some extra RPMs or
whatever.)

Will this also work for those toolshed packages which bundle their
package definitions (to download, make and install the tool
dependency) along with their wrappers?

Thanks for working on this.

cheers,
Simon

===
Attention: The information contained in this message and/or attachments
from AgResearch Limited is intended only for the persons or entities
to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipients is prohibited by AgResearch
Limited. If you have received this message in error, please notify the
sender immediately.
===
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-27 Thread Peter Cock

On Thu, Sep 26, 2013 at 10:27 PM, John Chilton chil...@msi.umn.edu wrote:
 My recommendation would be make the tool dependency install work on as
 many platforms as you can and not try to optimize in such a way that
 it is not going to work - i.e. favor reproduciblity over performance.

Reproducibility versus speed is a particular issue with floating point
libraries - NumPy using ATLAS vs OpenBLAST vs Intel MKL vs
just plain NumPy will probably all give slightly different answers
(on some tasks), and at different speed.

In this case, for simplicity I would advocate plain NumPy, without
worrying about needing ATLAS. For packages needing NumPy
with ALTAS, perhaps a new Tool Shed entry could be created,
package_numpy_1_7_with_atlas or similar (based on the
current configuration)?

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-27 Thread James Taylor

Make the precedence a config option. Otherwise I agree.

In addition, I still like the idea I suggested earlier of dependency provider 
plugins. Then you could (for example) have one that uses 'modules' and skips 
env.sh entirely.

On Sep 26, 2013, at 9:15 PM, John Chilton chil...@msi.umn.edu wrote:

 I was not even thinking we needed to modify the tool shed to implement
 this. I was hoping (?) you could just modify:
 
 lib/galaxy/tools/deps/__init__.py
 
 to implement this. If some tool contains the tag
 
  requirement type=package version=1.7.1numpy/requirement
 
 then if there is a manually installed tool_dependency in
 `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence
 over the tool shed installed version (would that be something like
 `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me
 know if this is way off base.
 
 There is a lot you could do to make this more complicated of course -
 an interface for mapping exact tool shed dependencies to manually
 installed ones, the ability to auto-compile tool shed dependencies
 against manually installed libraries, etc..., but I am not sure those
 complexities are buying you anything really.
 
 Thoughts?
 
 -John
 
 
 
 On Thu, Sep 26, 2013 at 5:47 PM, Greg Von Kuster g...@bx.psu.edu wrote:
 Hi John,
 
 On Sep 26, 2013, at 5:27 PM, John Chilton chil...@msi.umn.edu wrote:
 
 My recommendation would be make the tool dependency install work on as
 many platforms as you can and not try to optimize in such a way that
 it is not going to work - i.e. favor reproduciblity over performance.
 If a system administrator or institution want to sacrifice
 reproduciblity and optimize specific packages they should be able to
 do so manually. Its not just Atlas and CPU throttling right? Its
 vendor versions of MPI, GPGPU variants of code, variants of OpenMP,
 etc  Even if the tool shed provided some mechanism for determining
 if some particular package optimization is going to work, perhaps its
 better to just not enable it by default because frequently these cause
 slightly different results than the unoptimized version.
 
 The problem with this recommendation that is Galaxy currently provides
 no mechanism for doing so. Luckily this is easy to solve and the
 solution solves other problems. If the tool dependency resolution code
 would grab the manually configured dependency instead of the tool shed
 variant when available, instead of favoring the opposite, then it
 would be really easy to add in an optimized version of numpy or an MPI
 version of software X.
 
 How would you like this to happen?  Would it work to provide an admin the 
 ability to create a ToolDependency object and point it to a manually 
 configured dependency in whatever location on disk the admin chooses via a 
 new UI feature?  Or do you have a different idea?
 
 Thanks,
 
 Greg Von Kuster
 
 
 
 Whats great is this solves other problems as well. For instance, our
 genomics Galaxy web server runs Debian but the worker nodes run
 CentOS. This means many tool shed installed dependencies do not work.
 JJ being the patient guy he is goes in and manually updates the tool
 shed installed env.sh files to load modules. Even if you think not
 running the same version of the OS on your server and worker nodes is
 a bit crazy, there is the much more reasonable (common) case of just
 wanting to submit to multiple different clusters. When I was talking
 with the guys at NCGAS they were unsure how to do this, this one
 change would make that a lot more tenable.
 
 -John
 
 On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning
 bjoern.gruen...@pharmazie.uni-freiburg.de wrote:
 Hi,
 
 Hi Bjoern,
 
 Is there anything else we (the Galaxy community) can do to help
 sort out the ATLAS installation problems?
 
 Thanks for asking. I have indeed a few things I would like some
 comments.
 
 Another choice might be to use OpenBLAS instead of ATLAS, e.g.
 http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration
 
 I have no experience with it. Does it also need to turn off CPU
 throttling? I would assume so, or how is it optimizing itself?
 
 However, I think we build NumPy without using ATLAS or any
 BLAS library. That seems like the most pragmatic solution
 in the short term - which I think is what Dan tried here:
 http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7
 
 I can remove them if that is the consensus.
 
 A few points:
 - fixing the atlas issue can speed up numpy, scipy, R considerably (by
 400% in some cases)
 - as far as I understand that performance gain is due to optimizing
 itself on specific hardware, for atlas there is no way around to disable
 CPU throttling (How about OpenBlas?)
 - it seems to be complicated to deactivate CPU throttling on OS-X
 - binary installation does not make sense in that case, because ATLAS is
 self optimizing
 - Distribution shipped ATLAS packages are not really faster
 
 Current state:
 - Atlas tries

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-26 Thread Peter Cock

Hi Bjoern,

Is there anything else we (the Galaxy community) can do to help
sort out the ATLAS installation problems?

Another choice might be to use OpenBLAS instead of ATLAS, e.g.
http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration

However, I think we build NumPy without using ATLAS or any
BLAS library. That seems like the most pragmatic solution
in the short term - which I think is what Dan tried here:
http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7

Thanks,

Peter
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-26 Thread Greg Von Kuster

In case this will help, I have the framework implemented (and committed) for 
handling pre-compioled binaries for tool dependencies for a supported set of 
architectures.  Dave B has updated a lot of tool dependency definitions on both 
the test and main tool sheds to use this enhancement - those that he has 
updated are currently all owned by the devteam in preparation for a tool 
migration stage that he will soon be committiing.  Perhaps the atlas tool 
dependency can be updated to proved a pre-compiled binary installation.

Greg Von Kuster

On Sep 26, 2013, at 11:10 AM, Peter Cock p.j.a.c...@googlemail.com wrote:

 Hi Bjoern,
 
 Is there anything else we (the Galaxy community) can do to help
 sort out the ATLAS installation problems?
 
 Another choice might be to use OpenBLAS instead of ATLAS, e.g.
 http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration
 
 However, I think we build NumPy without using ATLAS or any
 BLAS library. That seems like the most pragmatic solution
 in the short term - which I think is what Dan tried here:
 http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7
 
 Thanks,
 
 Peter
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/
 
 To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-26 Thread Björn Grüning

Hi,

 Hi Bjoern,
 
 Is there anything else we (the Galaxy community) can do to help
 sort out the ATLAS installation problems?

Thanks for asking. I have indeed a few things I would like some
comments.

 Another choice might be to use OpenBLAS instead of ATLAS, e.g.
 http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration

I have no experience with it. Does it also need to turn off CPU
throttling? I would assume so, or how is it optimizing itself? 

 However, I think we build NumPy without using ATLAS or any
 BLAS library. That seems like the most pragmatic solution
 in the short term - which I think is what Dan tried here:
 http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7

I can remove them if that is the consensus.

A few points:
- fixing the atlas issue can speed up numpy, scipy, R considerably (by
400% in some cases)
- as far as I understand that performance gain is due to optimizing
itself on specific hardware, for atlas there is no way around to disable
CPU throttling (How about OpenBlas?)
- it seems to be complicated to deactivate CPU throttling on OS-X
- binary installation does not make sense in that case, because ATLAS is
self optimizing
- Distribution shipped ATLAS packages are not really faster

Current state:
- Atlas tries two different commands to deactivate CPU throttling. Afaik
that only works on some Ubuntu versions, where no root privileges are
necessary.
- If atlas fails for some reason, numpy/R/scipy installation should not
be affected (that's was at least the aim)

Questions:
- Is it worth the hassle for some speed improvements? pip install numpy,
would be so easy?

- If we want to support ATLAS, any better idea to how to implement it?
Any Tool Shed feature that can help? - interactive installation?
- can we flag a tool dependency as optional? So it can fail?

- Can anyone help with testing and fixing it?


Any opinions/comments?
Bjoern

 Thanks,
 
 Peter



___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-26 Thread John Chilton

My recommendation would be make the tool dependency install work on as
many platforms as you can and not try to optimize in such a way that
it is not going to work - i.e. favor reproduciblity over performance.
If a system administrator or institution want to sacrifice
reproduciblity and optimize specific packages they should be able to
do so manually. Its not just Atlas and CPU throttling right? Its
vendor versions of MPI, GPGPU variants of code, variants of OpenMP,
etc  Even if the tool shed provided some mechanism for determining
if some particular package optimization is going to work, perhaps its
better to just not enable it by default because frequently these cause
slightly different results than the unoptimized version.

The problem with this recommendation that is Galaxy currently provides
no mechanism for doing so. Luckily this is easy to solve and the
solution solves other problems. If the tool dependency resolution code
would grab the manually configured dependency instead of the tool shed
variant when available, instead of favoring the opposite, then it
would be really easy to add in an optimized version of numpy or an MPI
version of software X.

Whats great is this solves other problems as well. For instance, our
genomics Galaxy web server runs Debian but the worker nodes run
CentOS. This means many tool shed installed dependencies do not work.
JJ being the patient guy he is goes in and manually updates the tool
shed installed env.sh files to load modules. Even if you think not
running the same version of the OS on your server and worker nodes is
a bit crazy, there is the much more reasonable (common) case of just
wanting to submit to multiple different clusters. When I was talking
with the guys at NCGAS they were unsure how to do this, this one
change would make that a lot more tenable.

-John

On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning
bjoern.gruen...@pharmazie.uni-freiburg.de wrote:
 Hi,

 Hi Bjoern,

 Is there anything else we (the Galaxy community) can do to help
 sort out the ATLAS installation problems?

 Thanks for asking. I have indeed a few things I would like some
 comments.

 Another choice might be to use OpenBLAS instead of ATLAS, e.g.
 http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration

 I have no experience with it. Does it also need to turn off CPU
 throttling? I would assume so, or how is it optimizing itself?

 However, I think we build NumPy without using ATLAS or any
 BLAS library. That seems like the most pragmatic solution
 in the short term - which I think is what Dan tried here:
 http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7

 I can remove them if that is the consensus.

 A few points:
 - fixing the atlas issue can speed up numpy, scipy, R considerably (by
 400% in some cases)
 - as far as I understand that performance gain is due to optimizing
 itself on specific hardware, for atlas there is no way around to disable
 CPU throttling (How about OpenBlas?)
 - it seems to be complicated to deactivate CPU throttling on OS-X
 - binary installation does not make sense in that case, because ATLAS is
 self optimizing
 - Distribution shipped ATLAS packages are not really faster

 Current state:
 - Atlas tries two different commands to deactivate CPU throttling. Afaik
 that only works on some Ubuntu versions, where no root privileges are
 necessary.
 - If atlas fails for some reason, numpy/R/scipy installation should not
 be affected (that's was at least the aim)

 Questions:
 - Is it worth the hassle for some speed improvements? pip install numpy,
 would be so easy?

 - If we want to support ATLAS, any better idea to how to implement it?
 Any Tool Shed feature that can help? - interactive installation?
 - can we flag a tool dependency as optional? So it can fail?

 - Can anyone help with testing and fixing it?


 Any opinions/comments?
 Bjoern

 Thanks,

 Peter



 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

 To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-26 Thread Greg Von Kuster

Hi John,

On Sep 26, 2013, at 5:27 PM, John Chilton chil...@msi.umn.edu wrote:

 My recommendation would be make the tool dependency install work on as
 many platforms as you can and not try to optimize in such a way that
 it is not going to work - i.e. favor reproduciblity over performance.
 If a system administrator or institution want to sacrifice
 reproduciblity and optimize specific packages they should be able to
 do so manually. Its not just Atlas and CPU throttling right? Its
 vendor versions of MPI, GPGPU variants of code, variants of OpenMP,
 etc  Even if the tool shed provided some mechanism for determining
 if some particular package optimization is going to work, perhaps its
 better to just not enable it by default because frequently these cause
 slightly different results than the unoptimized version.
 
 The problem with this recommendation that is Galaxy currently provides
 no mechanism for doing so. Luckily this is easy to solve and the
 solution solves other problems. If the tool dependency resolution code
 would grab the manually configured dependency instead of the tool shed
 variant when available, instead of favoring the opposite, then it
 would be really easy to add in an optimized version of numpy or an MPI
 version of software X.

How would you like this to happen?  Would it work to provide an admin the 
ability to create a ToolDependency object and point it to a manually 
configured dependency in whatever location on disk the admin chooses via a new 
UI feature?  Or do you have a different idea?

Thanks,

Greg Von Kuster


 
 Whats great is this solves other problems as well. For instance, our
 genomics Galaxy web server runs Debian but the worker nodes run
 CentOS. This means many tool shed installed dependencies do not work.
 JJ being the patient guy he is goes in and manually updates the tool
 shed installed env.sh files to load modules. Even if you think not
 running the same version of the OS on your server and worker nodes is
 a bit crazy, there is the much more reasonable (common) case of just
 wanting to submit to multiple different clusters. When I was talking
 with the guys at NCGAS they were unsure how to do this, this one
 change would make that a lot more tenable.
 
 -John
 
 On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning
 bjoern.gruen...@pharmazie.uni-freiburg.de wrote:
 Hi,
 
 Hi Bjoern,
 
 Is there anything else we (the Galaxy community) can do to help
 sort out the ATLAS installation problems?
 
 Thanks for asking. I have indeed a few things I would like some
 comments.
 
 Another choice might be to use OpenBLAS instead of ATLAS, e.g.
 http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration
 
 I have no experience with it. Does it also need to turn off CPU
 throttling? I would assume so, or how is it optimizing itself?
 
 However, I think we build NumPy without using ATLAS or any
 BLAS library. That seems like the most pragmatic solution
 in the short term - which I think is what Dan tried here:
 http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7
 
 I can remove them if that is the consensus.
 
 A few points:
 - fixing the atlas issue can speed up numpy, scipy, R considerably (by
 400% in some cases)
 - as far as I understand that performance gain is due to optimizing
 itself on specific hardware, for atlas there is no way around to disable
 CPU throttling (How about OpenBlas?)
 - it seems to be complicated to deactivate CPU throttling on OS-X
 - binary installation does not make sense in that case, because ATLAS is
 self optimizing
 - Distribution shipped ATLAS packages are not really faster
 
 Current state:
 - Atlas tries two different commands to deactivate CPU throttling. Afaik
 that only works on some Ubuntu versions, where no root privileges are
 necessary.
 - If atlas fails for some reason, numpy/R/scipy installation should not
 be affected (that's was at least the aim)
 
 Questions:
 - Is it worth the hassle for some speed improvements? pip install numpy,
 would be so easy?
 
 - If we want to support ATLAS, any better idea to how to implement it?
 Any Tool Shed feature that can help? - interactive installation?
   - can we flag a tool dependency as optional? So it can fail?
 
 - Can anyone help with testing and fixing it?
 
 
 Any opinions/comments?
 Bjoern
 
 Thanks,
 
 Peter
 
 
 
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/
 
 To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/
 
 ___
 Please keep all replies on the list by using reply all
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/
 
 To

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-26 Thread John Chilton

I was not even thinking we needed to modify the tool shed to implement
this. I was hoping (?) you could just modify:

lib/galaxy/tools/deps/__init__.py

to implement this. If some tool contains the tag

  requirement type=package version=1.7.1numpy/requirement

then if there is a manually installed tool_dependency in
`tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence
over the tool shed installed version (would that be something like
`tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me
know if this is way off base.

There is a lot you could do to make this more complicated of course -
an interface for mapping exact tool shed dependencies to manually
installed ones, the ability to auto-compile tool shed dependencies
against manually installed libraries, etc..., but I am not sure those
complexities are buying you anything really.

Thoughts?

-John



On Thu, Sep 26, 2013 at 5:47 PM, Greg Von Kuster g...@bx.psu.edu wrote:
 Hi John,

 On Sep 26, 2013, at 5:27 PM, John Chilton chil...@msi.umn.edu wrote:

 My recommendation would be make the tool dependency install work on as
 many platforms as you can and not try to optimize in such a way that
 it is not going to work - i.e. favor reproduciblity over performance.
 If a system administrator or institution want to sacrifice
 reproduciblity and optimize specific packages they should be able to
 do so manually. Its not just Atlas and CPU throttling right? Its
 vendor versions of MPI, GPGPU variants of code, variants of OpenMP,
 etc  Even if the tool shed provided some mechanism for determining
 if some particular package optimization is going to work, perhaps its
 better to just not enable it by default because frequently these cause
 slightly different results than the unoptimized version.

 The problem with this recommendation that is Galaxy currently provides
 no mechanism for doing so. Luckily this is easy to solve and the
 solution solves other problems. If the tool dependency resolution code
 would grab the manually configured dependency instead of the tool shed
 variant when available, instead of favoring the opposite, then it
 would be really easy to add in an optimized version of numpy or an MPI
 version of software X.

 How would you like this to happen?  Would it work to provide an admin the 
 ability to create a ToolDependency object and point it to a manually 
 configured dependency in whatever location on disk the admin chooses via a 
 new UI feature?  Or do you have a different idea?

 Thanks,

 Greg Von Kuster



 Whats great is this solves other problems as well. For instance, our
 genomics Galaxy web server runs Debian but the worker nodes run
 CentOS. This means many tool shed installed dependencies do not work.
 JJ being the patient guy he is goes in and manually updates the tool
 shed installed env.sh files to load modules. Even if you think not
 running the same version of the OS on your server and worker nodes is
 a bit crazy, there is the much more reasonable (common) case of just
 wanting to submit to multiple different clusters. When I was talking
 with the guys at NCGAS they were unsure how to do this, this one
 change would make that a lot more tenable.

 -John

 On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning
 bjoern.gruen...@pharmazie.uni-freiburg.de wrote:
 Hi,

 Hi Bjoern,

 Is there anything else we (the Galaxy community) can do to help
 sort out the ATLAS installation problems?

 Thanks for asking. I have indeed a few things I would like some
 comments.

 Another choice might be to use OpenBLAS instead of ATLAS, e.g.
 http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration

 I have no experience with it. Does it also need to turn off CPU
 throttling? I would assume so, or how is it optimizing itself?

 However, I think we build NumPy without using ATLAS or any
 BLAS library. That seems like the most pragmatic solution
 in the short term - which I think is what Dan tried here:
 http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7

 I can remove them if that is the consensus.

 A few points:
 - fixing the atlas issue can speed up numpy, scipy, R considerably (by
 400% in some cases)
 - as far as I understand that performance gain is due to optimizing
 itself on specific hardware, for atlas there is no way around to disable
 CPU throttling (How about OpenBlas?)
 - it seems to be complicated to deactivate CPU throttling on OS-X
 - binary installation does not make sense in that case, because ATLAS is
 self optimizing
 - Distribution shipped ATLAS packages are not really faster

 Current state:
 - Atlas tries two different commands to deactivate CPU throttling. Afaik
 that only works on some Ubuntu versions, where no root privileges are
 necessary.
 - If atlas fails for some reason, numpy/R/scipy installation should not
 be affected (that's was at least the aim)

 Questions:
 - Is it worth the hassle for some speed improvements? pip install numpy,
 would

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-26 Thread Greg Von Kuster

James, it seems I was answering at the same time you were, so to highlight my 
comments to John, I'm just wondering how this will work for repositories in the 
tool shed that do not contain any tools, but just tool dependency definitions 
or complex repository dependency definitions.

Thanks,

Greg Von Kuster

On Sep 26, 2013, at 9:49 PM, James Taylor ja...@taylorlab.org wrote:

 Make the precedence a config option. Otherwise I agree.
 
 In addition, I still like the idea I suggested earlier of dependency provider 
 plugins. Then you could (for example) have one that uses 'modules' and skips 
 env.sh entirely.
 
 On Sep 26, 2013, at 9:15 PM, John Chilton chil...@msi.umn.edu wrote:
 
 I was not even thinking we needed to modify the tool shed to implement
 this. I was hoping (?) you could just modify:
 
 lib/galaxy/tools/deps/__init__.py
 
 to implement this. If some tool contains the tag
 
 requirement type=package version=1.7.1numpy/requirement
 
 then if there is a manually installed tool_dependency in
 `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence
 over the tool shed installed version (would that be something like
 `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me
 know if this is way off base.
 
 There is a lot you could do to make this more complicated of course -
 an interface for mapping exact tool shed dependencies to manually
 installed ones, the ability to auto-compile tool shed dependencies
 against manually installed libraries, etc..., but I am not sure those
 complexities are buying you anything really.
 
 Thoughts?
 
 -John
 
 
 
 On Thu, Sep 26, 2013 at 5:47 PM, Greg Von Kuster g...@bx.psu.edu wrote:
 Hi John,
 
 On Sep 26, 2013, at 5:27 PM, John Chilton chil...@msi.umn.edu wrote:
 
 My recommendation would be make the tool dependency install work on as
 many platforms as you can and not try to optimize in such a way that
 it is not going to work - i.e. favor reproduciblity over performance.
 If a system administrator or institution want to sacrifice
 reproduciblity and optimize specific packages they should be able to
 do so manually. Its not just Atlas and CPU throttling right? Its
 vendor versions of MPI, GPGPU variants of code, variants of OpenMP,
 etc  Even if the tool shed provided some mechanism for determining
 if some particular package optimization is going to work, perhaps its
 better to just not enable it by default because frequently these cause
 slightly different results than the unoptimized version.
 
 The problem with this recommendation that is Galaxy currently provides
 no mechanism for doing so. Luckily this is easy to solve and the
 solution solves other problems. If the tool dependency resolution code
 would grab the manually configured dependency instead of the tool shed
 variant when available, instead of favoring the opposite, then it
 would be really easy to add in an optimized version of numpy or an MPI
 version of software X.
 
 How would you like this to happen?  Would it work to provide an admin the 
 ability to create a ToolDependency object and point it to a manually 
 configured dependency in whatever location on disk the admin chooses via a 
 new UI feature?  Or do you have a different idea?
 
 Thanks,
 
 Greg Von Kuster
 
 
 
 Whats great is this solves other problems as well. For instance, our
 genomics Galaxy web server runs Debian but the worker nodes run
 CentOS. This means many tool shed installed dependencies do not work.
 JJ being the patient guy he is goes in and manually updates the tool
 shed installed env.sh files to load modules. Even if you think not
 running the same version of the OS on your server and worker nodes is
 a bit crazy, there is the much more reasonable (common) case of just
 wanting to submit to multiple different clusters. When I was talking
 with the guys at NCGAS they were unsure how to do this, this one
 change would make that a lot more tenable.
 
 -John
 
 On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning
 bjoern.gruen...@pharmazie.uni-freiburg.de wrote:
 Hi,
 
 Hi Bjoern,
 
 Is there anything else we (the Galaxy community) can do to help
 sort out the ATLAS installation problems?
 
 Thanks for asking. I have indeed a few things I would like some
 comments.
 
 Another choice might be to use OpenBLAS instead of ATLAS, e.g.
 http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration
 
 I have no experience with it. Does it also need to turn off CPU
 throttling? I would assume so, or how is it optimizing itself?
 
 However, I think we build NumPy without using ATLAS or any
 BLAS library. That seems like the most pragmatic solution
 in the short term - which I think is what Dan tried here:
 http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7
 
 I can remove them if that is the consensus.
 
 A few points:
 - fixing the atlas issue can speed up numpy, scipy, R considerably (by
 400% in some cases)
 - as far as I understand that performance

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-26 Thread Guest, Simon

At Thu, 26 Sep 2013 22:03:09 -0400,
Greg Von Kuster wrote:
 
 Hi John,
 
 On Sep 26, 2013, at 9:15 PM, John Chilton chil...@msi.umn.edu wrote:
 
  I was not even thinking we needed to modify the tool shed to implement
  this. I was hoping (?) you could just modify:
 
 Nothing in the Tool Shed itself would be affected or require modification for 
 this new feature as this is completely on the Galaxy side.
 
  
  lib/galaxy/tools/deps/__init__.py
  
  to implement this. If some tool contains the tag
  
   requirement type=package version=1.7.1numpy/requirement
  
  then if there is a manually installed tool_dependency in
  `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence
  over the tool shed installed version (would that be something like
  `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me
  know if this is way off base.
 
 This is a possibility perhaps, but there seems to be a potential weakness in 
 that it doesn't require the Tool Dependency object to exist since the tool 
 will function without a installed dependency from the Tool Shed.  Or, if the 
 installed depndency s required, then it is meaningless because it won't be 
 used.  If the former case, then the tool dependency cannot be shared via the 
 Tool Shed's dependency mechanism because none of the relationships will be 
 defined  since nothing is installed.  Wouldn't it be better to allow the 
 Galaxy admin to point the ToolDependency object to a specified binary on 
 disk?  In this way, all relationships defined by Tool Shed installations will 
 work as expected, with all contained tools with that dependency point to that 
 same shared location on disk.

Hi Greg, John,

What you are discussing is pretty close to what I am after, and what I
am prepared to spend some time working on, if that can help.  This is
what I have posted about a couple of times previously.  The option to
use a pre-installed package rather than a Galaxy installed one I think
would be very useful in general.  I think it can be done in a way that
doesn't break the tool dependency model.

I envisage providing access to existing programs on disk by loading
the relevant environment module.  Platforms will differ greatly on
where they have programs installed (usr/local/bwa-0.5.9,
/usr/libexec/bwa-0.5.9, etc.).  However, we could arrange to have a
conventionally named environment module available, so Galaxy just has
to be told somehow to do a module load bwa/0.5.9, say, prior to
trying to run that tool, which will then be found on the PATH.

Hooking this in without killing the dependency on named and versioned
toolshed repos could work like this.  In parallel with the repo sets
named, e.g. package_bwa_0_5_9, which download and build the package,
we have a set named like native_package_bwa_0_5_9.  This could have a
tool_dependencies.xml file like this:

?xml version=1.0?
tool_dependency
native_package name=bwa version=0.5.9
actions
action type=load_module
module name=bwa version=0.5.9/module
/action
/actions
readme
Uses native BWA via environment module bwa/0.5.9
/readme
/native_package
/tool_dependency

What this does is ensure that any repo which requires version 0.5.9 of
bwa can have this dependency resolved by either package_bwa_0_5_9, or
native_package_bwa_0_5_9. 

I envisage a Galaxy configuration setting that enables native package
and/or Galaxy package dependency resolution, and which one should be
preferred.  Of course, if one of these has been manually installed, it
will be used as is.

I would really like to be able to install tools from the toolshed, and
have them resolve their dependencies automatically using my
preinstalled application suite.  I see that would also meet the needs
being discussed here.

How does that sound?

cheers,
Simon

===
Attention: The information contained in this message and/or attachments
from AgResearch Limited is intended only for the persons or entities
to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipients is prohibited by AgResearch
Limited. If you have received this message in error, please notify the
sender immediately.
===
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-26 Thread John Chilton

Simon,

What is the advantage of putting that XML definition in the tool shed?
It is not 100% true because of prior_install_required dependencies,
but for the most part sourcing/load the environment for tools is a
Galaxy problem, not so much a tool shed one. What if we did this
instead?

Add an option to Galaxy's universe_wsgi.ini with the following default:

tool_dependency_resolution_order = gx_package_manual, gx_package_toolshed

Which essentially implements my idea above, with James' additional
configuration. But which can be overridden as:

tool_dependency_resolution_order = plugin_module, gx_package_manual,
gx_package_toolshed

If set this way then placing requirement
package=0.5.9bwa/requirement in a tool will result in the module
bwa/0.5.9 being loaded if it is 'avail'able, else it will check for a
manually installed env.sh (which is where MSI is currently putting its
module loads), and else it will fallback to source the tool shed
installed dependency.

I feel like this will give you everything you want without any extra
XML or configuration. Let me know if I am wrong.

-John

On Thu, Sep 26, 2013 at 11:39 PM, Guest, Simon
simon.gu...@agresearch.co.nz wrote:
 At Thu, 26 Sep 2013 22:03:09 -0400,
 Greg Von Kuster wrote:

 Hi John,

 On Sep 26, 2013, at 9:15 PM, John Chilton chil...@msi.umn.edu wrote:

  I was not even thinking we needed to modify the tool shed to implement
  this. I was hoping (?) you could just modify:

 Nothing in the Tool Shed itself would be affected or require modification 
 for this new feature as this is completely on the Galaxy side.

 
  lib/galaxy/tools/deps/__init__.py
 
  to implement this. If some tool contains the tag
 
   requirement type=package version=1.7.1numpy/requirement
 
  then if there is a manually installed tool_dependency in
  `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence
  over the tool shed installed version (would that be something like
  `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me
  know if this is way off base.

 This is a possibility perhaps, but there seems to be a potential weakness in 
 that it doesn't require the Tool Dependency object to exist since the tool 
 will function without a installed dependency from the Tool Shed.  Or, if the 
 installed depndency s required, then it is meaningless because it won't be 
 used.  If the former case, then the tool dependency cannot be shared via the 
 Tool Shed's dependency mechanism because none of the relationships will be 
 defined  since nothing is installed.  Wouldn't it be better to allow the 
 Galaxy admin to point the ToolDependency object to a specified binary on 
 disk?  In this way, all relationships defined by Tool Shed installations 
 will work as expected, with all contained tools with that dependency point 
 to that same shared location on disk.

 Hi Greg, John,

 What you are discussing is pretty close to what I am after, and what I
 am prepared to spend some time working on, if that can help.  This is
 what I have posted about a couple of times previously.  The option to
 use a pre-installed package rather than a Galaxy installed one I think
 would be very useful in general.  I think it can be done in a way that
 doesn't break the tool dependency model.

 I envisage providing access to existing programs on disk by loading
 the relevant environment module.  Platforms will differ greatly on
 where they have programs installed (usr/local/bwa-0.5.9,
 /usr/libexec/bwa-0.5.9, etc.).  However, we could arrange to have a
 conventionally named environment module available, so Galaxy just has
 to be told somehow to do a module load bwa/0.5.9, say, prior to
 trying to run that tool, which will then be found on the PATH.

 Hooking this in without killing the dependency on named and versioned
 toolshed repos could work like this.  In parallel with the repo sets
 named, e.g. package_bwa_0_5_9, which download and build the package,
 we have a set named like native_package_bwa_0_5_9.  This could have a
 tool_dependencies.xml file like this:

 ?xml version=1.0?
 tool_dependency
 native_package name=bwa version=0.5.9
 actions
 action type=load_module
 module name=bwa version=0.5.9/module
 /action
 /actions
 readme
 Uses native BWA via environment module bwa/0.5.9
 /readme
 /native_package
 /tool_dependency

 What this does is ensure that any repo which requires version 0.5.9 of
 bwa can have this dependency resolved by either package_bwa_0_5_9, or
 native_package_bwa_0_5_9.

 I envisage a Galaxy configuration setting that enables native package
 and/or Galaxy package dependency resolution, and which one should be
 preferred.  Of course, if one of these has been manually installed, it
 will be used as is.

 I would really like to be able to install tools from the toolshed, and
 have them resolve their dependencies automatically using my
 preinstalled application suite.  I see that

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-26 Thread John Chilton

On Thu, Sep 26, 2013 at 9:10 PM, Greg Von Kuster g...@bx.psu.edu wrote:
 James, it seems I was answering at the same time you were, so to highlight my 
 comments to John, I'm just wondering how this will work for repositories in 
 the tool shed that do not contain any tools, but just tool dependency 
 definitions or complex repository dependency definitions.

I am certain our European colleagues will have use cases to contribute
when they have had their coffee, but I would just assume not get the
tool shed involved. If you want to optimize a tool for Galaxy, the
best practice would be you optimize it by replacing the package the
corresponding tool sources not by manually compiling some library and
having the tool shed compile additional tools against it. Does that
answer your question?

-John


 Thanks,

 Greg Von Kuster

 On Sep 26, 2013, at 9:49 PM, James Taylor ja...@taylorlab.org wrote:

 Make the precedence a config option. Otherwise I agree.

 In addition, I still like the idea I suggested earlier of dependency 
 provider plugins. Then you could (for example) have one that uses 'modules' 
 and skips env.sh entirely.

 On Sep 26, 2013, at 9:15 PM, John Chilton chil...@msi.umn.edu wrote:

 I was not even thinking we needed to modify the tool shed to implement
 this. I was hoping (?) you could just modify:

 lib/galaxy/tools/deps/__init__.py

 to implement this. If some tool contains the tag

 requirement type=package version=1.7.1numpy/requirement

 then if there is a manually installed tool_dependency in
 `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence
 over the tool shed installed version (would that be something like
 `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me
 know if this is way off base.

 There is a lot you could do to make this more complicated of course -
 an interface for mapping exact tool shed dependencies to manually
 installed ones, the ability to auto-compile tool shed dependencies
 against manually installed libraries, etc..., but I am not sure those
 complexities are buying you anything really.

 Thoughts?

 -John



 On Thu, Sep 26, 2013 at 5:47 PM, Greg Von Kuster g...@bx.psu.edu wrote:
 Hi John,

 On Sep 26, 2013, at 5:27 PM, John Chilton chil...@msi.umn.edu wrote:

 My recommendation would be make the tool dependency install work on as
 many platforms as you can and not try to optimize in such a way that
 it is not going to work - i.e. favor reproduciblity over performance.
 If a system administrator or institution want to sacrifice
 reproduciblity and optimize specific packages they should be able to
 do so manually. Its not just Atlas and CPU throttling right? Its
 vendor versions of MPI, GPGPU variants of code, variants of OpenMP,
 etc  Even if the tool shed provided some mechanism for determining
 if some particular package optimization is going to work, perhaps its
 better to just not enable it by default because frequently these cause
 slightly different results than the unoptimized version.

 The problem with this recommendation that is Galaxy currently provides
 no mechanism for doing so. Luckily this is easy to solve and the
 solution solves other problems. If the tool dependency resolution code
 would grab the manually configured dependency instead of the tool shed
 variant when available, instead of favoring the opposite, then it
 would be really easy to add in an optimized version of numpy or an MPI
 version of software X.

 How would you like this to happen?  Would it work to provide an admin the 
 ability to create a ToolDependency object and point it to a manually 
 configured dependency in whatever location on disk the admin chooses via 
 a new UI feature?  Or do you have a different idea?

 Thanks,

 Greg Von Kuster



 Whats great is this solves other problems as well. For instance, our
 genomics Galaxy web server runs Debian but the worker nodes run
 CentOS. This means many tool shed installed dependencies do not work.
 JJ being the patient guy he is goes in and manually updates the tool
 shed installed env.sh files to load modules. Even if you think not
 running the same version of the OS on your server and worker nodes is
 a bit crazy, there is the much more reasonable (common) case of just
 wanting to submit to multiple different clusters. When I was talking
 with the guys at NCGAS they were unsure how to do this, this one
 change would make that a lot more tenable.

 -John

 On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning
 bjoern.gruen...@pharmazie.uni-freiburg.de wrote:
 Hi,

 Hi Bjoern,

 Is there anything else we (the Galaxy community) can do to help
 sort out the ATLAS installation problems?

 Thanks for asking. I have indeed a few things I would like some
 comments.

 Another choice might be to use OpenBLAS instead of ATLAS, e.g.
 http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration

 I have no experience with it. Does it also need to turn off CPU
 throttling? I would

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-23 Thread Carlos Borroto

Forgot to copy the list on this email.

On Mon, Sep 23, 2013 at 9:32 AM, Carlos Borroto
carlos.borr...@gmail.com wrote:
 On Fri, Sep 20, 2013 at 6:12 PM, Björn Grüning
 bjoern.gruen...@pharmazie.uni-freiburg.de wrote:
 Hi Carlos,

 Can you try again?
 Also the new unstable version if you can. Thanks for the help! ATLAS is
 a beast :(


 Sorry for the delayed response. Busy weekend.

 Trying to install package_atlas_3_10 on Ubuntu 13.04:

 STDERR
 It appears you have cpu throttling enabled, which makes timings
 unreliable and an ATLAS install nonsensical.  Aborting.
 See ATLAS/INSTALL.txt for further information
 #


 I think this is the other relevant part in the log:

 # try to disable cpu throttling
 if hash cpufreq-selector 2/dev/null; then
 cpufreq-selector -g performance
 elif hash cpupower 2/dev/null; then
 cpupower frequency-set -g performance
 else
 echo 'Please deactivate CPU throttling by your
 own, or install cpufreq-selector'
 exit
 fi

 STDERR
 Error calling SetGovernor: Caller is not authorized

 I had to install 'cpufreq-selector' to get here, not installed by
 default. I can confirm I get the same error when trying to run this
 command directly:
 $ cpufreq-selector -g performance
 Error calling SetGovernor: Caller is not authorized

 package_atlas_3_11 fails in exactly the same way in this box.

 Somehow this is also a silent failing and numpy, biopython and
 ngs-tools(my tool package depending on biopython) get
 Installed(Green)

 Yes that is correct. I designed it (in theory) in that way, that if
 ATLAS crashes (due to CPU throttling enabled) every other package can be
 installed without problem. Every other behaviour is a bug.

 while lapack, atlas and split_by_barcode(my actual
 tool wrapper depending on ngs-tools package) get Installed, missing
 tool dependencies(Grey). This means if I try to use my wrapper in
 this state I get this error:
 /bin/sh: 1: ngs-tools: not found

 On Ubuntu it gets installed without errors. Everything is green :)

 However, if I do a Repair repository on split_by_barcode it goes
 into Installed(Green) and everything seems to work from then on.

 Mh, thats seems to be a bug. I will try to reproduce on a computer where
 ATLAS is crashing.


 Thanks for testing this. I also believe this might be a bug.
 --Carlos

___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-20 Thread Bjoern Gruening

Hi Carlos,

Hi Peter and Carlos,

On Mon, Sep 16, 2013 at 8:57 PM, Carlos Borroto
carlos.borr...@gmail.com wrote:
I did an extra test. Started with a clean 'galaxy-dist'. This time
both repositories fail with the same error. I guess before something
was cached for the repository with version 0.1.4.

I used biopython repository as a guide to write my tool dependency
definition:
http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61

I can confirm biopython repository is failing to install for me with
exactly the same error. I wonder if a recently addition in the test
toolshed broke the treatment of prior_installation_required.

Thanks,
Carlos

Could be - note that currently Biopython isn't currently
installing properly on the Test Tool Shed due to ALTAS
failing (a requirement of NumPy which is a requirement
of Biopython). Dave and Bjoern are I think looking at this
already...

I can't do much I tested it again and for me its working fine on my
computers I have at hand ... sorry.

In case it helps, this is how the INSTALLATION.log file ends on OS X 10.8.4:
$ tail -n 3
~/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/3508de0ebae1/INSTALLATION.log
x ATLAS/tune/lapack/lanbsrch.c
tar: Error exit delayed from previous errors.
#

This is the relevant part I can find in Galaxy's log:
[localhost] local: tar xfvj atlas3.10.1.tar.bz2

Warning: local() encountered an error (return code 1) while executing
'tar xfvj atlas3.10.1.tar.bz2'

After noticing this I got what I'm guessing is the original file from
sourceforge:
http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10.1.tar.bz2

I can confirm that trying to untar this file fails with the exact same
error. However, on Ubuntu 13.04 untaring this file works just fine.

That is new to me. How can that happen? Anyone with an OS-X can confirm
that?

On Ubuntu 13.04 the error I see is:
STDERR
It appears you have cpu throttling enabled, which makes timings
unreliable and an ATLAS install nonsensical. Aborting.
See ATLAS/INSTALL.txt for further information
#

Björn, you said cpu throttling can be easily disable on Ubuntu. Can
you comment how? Do I need to disable it completely, or should the
install script do it when installing?

ATLAS (once untared ;-)) needs cpu-throttling to be disabled to
optimizes its library. If it is not disabled ATLAS compilation will
fail. For OS-X I found that:

http://apple.stackexchange.com/questions/41045/how-can-i-disable-cpu-throttling-and-cpu-disabling

Sorry, I never touched a OS-X. Nevertheless, If its not disabled, it is
supposed to fail silently and downstream packages will not be affected.
But if its crashing during untarring, I can't do much. What I can do is
to repack the tarball and host it elsewhere. What brings me to:

http://gmod.827538.n3.nabble.com/RFC-Storing-of-tarballs-and-patches-for-tool-dependencies-to-enable-reproducibility-td4036591.html

Bad news for a Friday morning, sorry :(
Bjoern

Thanks,
Carlos

___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)

2013-09-20 Thread Ido Tamir

Yes this tar is broken at least on OSX.
Other people have the same issue:
http://code.google.com/p/libarchive/issues/detail?id=299

On Sep 20, 2013, at 10:41 AM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: