Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
On Sun, Sep 29, 2013 at 10:43 PM, Guest, Simon wrote: > At Fri, 27 Sep 2013 00:23:37 -0500, > John Chilton wrote: >> >> Simon, >> >> What is the advantage of putting that XML definition in the tool shed? >> It is not 100% true because of prior_install_required dependencies, >> but for the most part sourcing/load the environment for tools is a >> Galaxy problem, not so much a tool shed one. What if we did this >> instead? >> >> Add an option to Galaxy's universe_wsgi.ini with the following default: >> >> tool_dependency_resolution_order = gx_package_manual, gx_package_toolshed >> >> Which essentially implements my idea above, with James' additional >> configuration. But which can be overridden as: >> >> tool_dependency_resolution_order = plugin_module, gx_package_manual, >> gx_package_toolshed >> >> If set this way then placing > package="0.5.9">bwa in a tool will result in the module >> bwa/0.5.9 being loaded if it is 'avail'able, else it will check for a >> manually installed env.sh (which is where MSI is currently putting its >> module loads), and else it will fallback to source the tool shed >> installed dependency. >> >> I feel like this will give you everything you want without any extra >> XML or configuration. Let me know if I am wrong. > > Hi John, > > I think you're right. Your scheme is neater than what I was > proposing. The extra flexibility I was aiming at via some toolshed > XML stuff appears not to be necessary upon further reflection. (I > wanted to ensure a Galaxy admin could just install some RPMs, install > a toolshed tool, and have everything resolve nicely. You seem to have > achieved that with your scheme.) > > I haven't had a chance to try your code yet, but as soon as I can I > will do so, and get back to you. > > If I leave out gx_package_toolshed altogether from > tool_dependency_resolution_order, will the tool installation in Galaxy > simply fail with a nice error message if the environment module and/or > env.sh files are not found? (This is what I would like, as it would > serve as a prompt to the Galaxy admin to install some extra RPMs or > whatever.) First tool_dependency_resolution_order is not how I landed up implementing it. Each "resolver" may need parameters so I decided to do an XML configuration kind of like Nate's job_conf.xml stuff. So to leave the tool shed stuff off you can just add dependency_resolvers_conf.xml to your Galaxy root. You can also just leave galaxy_packages off if you are not using any manually installed env.sh files. The first modules resolver will try to match each tag like this: R with a module load like: R/3.0.1 if that exact module is available. The second versionless form of the tag will result in Galaxy just falling back to auto-load whatever the default R module is if the exact version specified (3.0.1) is not available. This code doesn't touch the tool shed at all, so if you check to install dependencies, they will still be installed and if you don't they will not be installed. This configuration just tells the Galaxy how to use the dependencies that happen to be installed. You will need to review installed tools and make sure you have matching modules. Additionally, some tools may use this set_environment requirement type, I am not sure how to implement this or how prevalent its use is. -John > > Will this also work for those toolshed packages which bundle their > package definitions (to download, make and install the tool > dependency) along with their wrappers? Can you opt not to install packages for such repositories? Either way, the answer is I guess the same as above, the tool shed is unchanged, its just how Galaxy utilizes the installed dependencies that is being modified here. > > Thanks for working on this. > > cheers, > Simon > > === > Attention: The information contained in this message and/or attachments > from AgResearch Limited is intended only for the persons or entities > to which it is addressed and may contain confidential and/or privileged > material. Any review, retransmission, dissemination or other use of, or > taking of any action in reliance upon, this information by persons or > entities other than the intended recipients is prohibited by AgResearch > Limited. If you have received this message in error, please notify the > sender immediately. > === ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
At Fri, 27 Sep 2013 00:23:37 -0500, John Chilton wrote: > > Simon, > > What is the advantage of putting that XML definition in the tool shed? > It is not 100% true because of prior_install_required dependencies, > but for the most part sourcing/load the environment for tools is a > Galaxy problem, not so much a tool shed one. What if we did this > instead? > > Add an option to Galaxy's universe_wsgi.ini with the following default: > > tool_dependency_resolution_order = gx_package_manual, gx_package_toolshed > > Which essentially implements my idea above, with James' additional > configuration. But which can be overridden as: > > tool_dependency_resolution_order = plugin_module, gx_package_manual, > gx_package_toolshed > > If set this way then placing package="0.5.9">bwa in a tool will result in the module > bwa/0.5.9 being loaded if it is 'avail'able, else it will check for a > manually installed env.sh (which is where MSI is currently putting its > module loads), and else it will fallback to source the tool shed > installed dependency. > > I feel like this will give you everything you want without any extra > XML or configuration. Let me know if I am wrong. Hi John, I think you're right. Your scheme is neater than what I was proposing. The extra flexibility I was aiming at via some toolshed XML stuff appears not to be necessary upon further reflection. (I wanted to ensure a Galaxy admin could just install some RPMs, install a toolshed tool, and have everything resolve nicely. You seem to have achieved that with your scheme.) I haven't had a chance to try your code yet, but as soon as I can I will do so, and get back to you. If I leave out gx_package_toolshed altogether from tool_dependency_resolution_order, will the tool installation in Galaxy simply fail with a nice error message if the environment module and/or env.sh files are not found? (This is what I would like, as it would serve as a prompt to the Galaxy admin to install some extra RPMs or whatever.) Will this also work for those toolshed packages which bundle their package definitions (to download, make and install the tool dependency) along with their wrappers? Thanks for working on this. cheers, Simon === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. === ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
On Sun, Sep 29, 2013 at 1:17 PM, Björn Grüning wrote: > Hi Peter and John, > > thanks for your comments and thanks for working on the patch John. > > Peter wrote: >> In this case, for simplicity I would advocate plain NumPy, without >> worrying about needing ATLAS. For packages needing NumPy >> with ALTAS, perhaps a new Tool Shed entry could be created, >> package_numpy_1_7_with_atlas or similar (based on the >> current configuration)? > > I updated R, numpy, scipy, scikit and removed the atlas dependency. It > seems to work fine for the ChemicalToolBox. I do not remove the lapack > dependency, because I did not get any complains until now. Great. Hopefully all my tools with a NumPy dependency via Biopython will pickup this change automatically when the next nightly Tool Shed tests are run. (That's what I hope will happen, otherwise I'll need to bump the dependency revision in the Biopython package & all the tools calling that). > I also > created new repositories *_with_atlas in my galaxytools repository, if > anyone is interested in atlas dependent packages. Sounds good :) > Lets concentrate on reproducibility and leave out the speed > improvements for now. I admit it was do ambitious/idealistic. It is nice to aim high, but I think here practicality wins. > Have a nice weekend, > Bjoern You too, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Hi Peter and John, thanks for your comments and thanks for working on the patch John. > On Thu, Sep 26, 2013 at 10:27 PM, John Chilton wrote: > > My recommendation would be make the tool dependency install work on as > > many platforms as you can and not try to optimize in such a way that > > it is not going to work - i.e. favor reproduciblity over performance. > > Reproducibility versus speed is a particular issue with floating point > libraries - NumPy using ATLAS vs OpenBLAST vs Intel MKL vs > just plain NumPy will probably all give slightly different answers > (on some tasks), and at different speed. > > In this case, for simplicity I would advocate plain NumPy, without > worrying about needing ATLAS. For packages needing NumPy > with ALTAS, perhaps a new Tool Shed entry could be created, > package_numpy_1_7_with_atlas or similar (based on the > current configuration)? I updated R, numpy, scipy, scikit and removed the atlas dependency. It seems to work fine for the ChemicalToolBox. I do not remove the lapack dependency, because I did not get any complains until now. I also created new repositories *_with_atlas in my galaxytools repository, if anyone is interested in atlas dependent packages. Lets concentrate on reproducibility and leave out the speed improvements for now. I admit it was do ambitious/idealistic. Have a nice weekend, Bjoern > Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
I have issued a pull request with a specific implementation of these ideas: https://bitbucket.org/galaxy/galaxy-central/pull-request/227/tool-dependency-resolver-plugins/diff Please feel free to comment. -John On Fri, Sep 27, 2013 at 12:23 AM, John Chilton wrote: > Simon, > > What is the advantage of putting that XML definition in the tool shed? > It is not 100% true because of prior_install_required dependencies, > but for the most part sourcing/load the environment for tools is a > Galaxy problem, not so much a tool shed one. What if we did this > instead? > > Add an option to Galaxy's universe_wsgi.ini with the following default: > > tool_dependency_resolution_order = gx_package_manual, gx_package_toolshed > > Which essentially implements my idea above, with James' additional > configuration. But which can be overridden as: > > tool_dependency_resolution_order = plugin_module, gx_package_manual, > gx_package_toolshed > > If set this way then placing package="0.5.9">bwa in a tool will result in the module > bwa/0.5.9 being loaded if it is 'avail'able, else it will check for a > manually installed env.sh (which is where MSI is currently putting its > module loads), and else it will fallback to source the tool shed > installed dependency. > > I feel like this will give you everything you want without any extra > XML or configuration. Let me know if I am wrong. > > -John > > On Thu, Sep 26, 2013 at 11:39 PM, Guest, Simon > wrote: >> At Thu, 26 Sep 2013 22:03:09 -0400, >> Greg Von Kuster wrote: >>> >>> Hi John, >>> >>> On Sep 26, 2013, at 9:15 PM, John Chilton wrote: >>> >>> > I was not even thinking we needed to modify the tool shed to implement >>> > this. I was hoping (?) you could just modify: >>> >>> Nothing in the Tool Shed itself would be affected or require modification >>> for this new feature as this is completely on the Galaxy side. >>> >>> > >>> > lib/galaxy/tools/deps/__init__.py >>> > >>> > to implement this. If some tool contains the tag >>> > >>> > numpy >>> > >>> > then if there is a manually installed tool_dependency in >>> > `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence >>> > over the tool shed installed version (would that be something like >>> > `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me >>> > know if this is way off base. >>> >>> This is a possibility perhaps, but there seems to be a potential weakness >>> in that it doesn't require the Tool Dependency object to exist since the >>> tool will function without a installed dependency from the Tool Shed. Or, >>> if the installed depndency s required, then it is meaningless because it >>> won't be used. If the former case, then the tool dependency cannot be >>> shared via the Tool Shed's dependency mechanism because none of the >>> relationships will be defined since nothing is installed. Wouldn't it be >>> better to allow the Galaxy admin to point the ToolDependency object to a >>> specified binary on disk? In this way, all relationships defined by Tool >>> Shed installations will work as expected, with all contained tools with >>> that dependency point to that same shared location on disk. >> >> Hi Greg, John, >> >> What you are discussing is pretty close to what I am after, and what I >> am prepared to spend some time working on, if that can help. This is >> what I have posted about a couple of times previously. The option to >> use a pre-installed package rather than a Galaxy installed one I think >> would be very useful in general. I think it can be done in a way that >> doesn't break the tool dependency model. >> >> I envisage providing access to existing programs on disk by loading >> the relevant environment module. Platforms will differ greatly on >> where they have programs installed (usr/local/bwa-0.5.9, >> /usr/libexec/bwa-0.5.9, etc.). However, we could arrange to have a >> conventionally named environment module available, so Galaxy just has >> to be told somehow to do a module load bwa/0.5.9, say, prior to >> trying to run that tool, which will then be found on the PATH. >> >> Hooking this in without killing the dependency on named and versioned >> toolshed repos could work like this. In parallel with the repo sets >> named, e.g. package_bwa_0_5_9, which download and build the package, >> we have a set named like native_package_bwa_0_5_9. This could have a >> tool_dependencies.xml file like this: >> >> >> >> >> >> >> >> >> >> >> Uses native BWA via environment module bwa/0.5.9 >> >> >> >> >> What this does is ensure that any repo which requires version 0.5.9 of >> bwa can have this dependency resolved by either package_bwa_0_5_9, or >> native_package_bwa_0_5_9. >> >> I envisage a Galaxy configuration setting that enables native package >> and/or Galaxy package dependency resolution, and which one should be >> preferred. Of course, if one of these has bee
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Make the precedence a config option. Otherwise I agree. In addition, I still like the idea I suggested earlier of dependency provider plugins. Then you could (for example) have one that uses 'modules' and skips env.sh entirely. On Sep 26, 2013, at 9:15 PM, John Chilton wrote: > I was not even thinking we needed to modify the tool shed to implement > this. I was hoping (?) you could just modify: > > lib/galaxy/tools/deps/__init__.py > > to implement this. If some tool contains the tag > > numpy > > then if there is a manually installed tool_dependency in > `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence > over the tool shed installed version (would that be something like > `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me > know if this is way off base. > > There is a lot you could do to make this more complicated of course - > an interface for mapping exact tool shed dependencies to manually > installed ones, the ability to auto-compile tool shed dependencies > against manually installed libraries, etc..., but I am not sure those > complexities are buying you anything really. > > Thoughts? > > -John > > > > On Thu, Sep 26, 2013 at 5:47 PM, Greg Von Kuster wrote: >> Hi John, >> >> On Sep 26, 2013, at 5:27 PM, John Chilton wrote: >> >>> My recommendation would be make the tool dependency install work on as >>> many platforms as you can and not try to optimize in such a way that >>> it is not going to work - i.e. favor reproduciblity over performance. >>> If a system administrator or institution want to sacrifice >>> reproduciblity and optimize specific packages they should be able to >>> do so manually. Its not just Atlas and CPU throttling right? Its >>> vendor versions of MPI, GPGPU variants of code, variants of OpenMP, >>> etc Even if the tool shed provided some mechanism for determining >>> if some particular package optimization is going to work, perhaps its >>> better to just not enable it by default because frequently these cause >>> slightly different results than the unoptimized version. >>> >>> The problem with this recommendation that is Galaxy currently provides >>> no mechanism for doing so. Luckily this is easy to solve and the >>> solution solves other problems. If the tool dependency resolution code >>> would grab the manually configured dependency instead of the tool shed >>> variant when available, instead of favoring the opposite, then it >>> would be really easy to add in an optimized version of numpy or an MPI >>> version of software X. >> >> How would you like this to happen? Would it work to provide an admin the >> ability to create a ToolDependency object and point it to a "manually >> configured dependency" in whatever location on disk the admin chooses via a >> new UI feature? Or do you have a different idea? >> >> Thanks, >> >> Greg Von Kuster >> >> >>> >>> Whats great is this solves other problems as well. For instance, our >>> genomics Galaxy web server runs Debian but the worker nodes run >>> CentOS. This means many tool shed installed dependencies do not work. >>> JJ being the patient guy he is goes in and manually updates the tool >>> shed installed env.sh files to load modules. Even if you think not >>> running the same version of the OS on your server and worker nodes is >>> a bit crazy, there is the much more reasonable (common) case of just >>> wanting to submit to multiple different clusters. When I was talking >>> with the guys at NCGAS they were unsure how to do this, this one >>> change would make that a lot more tenable. >>> >>> -John >>> >>> On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning >>> wrote: Hi, > Hi Bjoern, > > Is there anything else we (the Galaxy community) can do to help > sort out the ATLAS installation problems? Thanks for asking. I have indeed a few things I would like some comments. > Another choice might be to use OpenBLAS instead of ATLAS, e.g. > http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration I have no experience with it. Does it also need to turn off CPU throttling? I would assume so, or how is it optimizing itself? > However, I think we build NumPy without using ATLAS or any > BLAS library. That seems like the most pragmatic solution > in the short term - which I think is what Dan tried here: > http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7 I can remove them if that is the consensus. A few points: - fixing the atlas issue can speed up numpy, scipy, R considerably (by 400% in some cases) - as far as I understand that performance gain is due to optimizing itself on specific hardware, for atlas there is no way around to disable CPU throttling (How about OpenBlas?) - it seems to be complicated to deactivate CPU throttling on OS-X - binary installation does not m
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
On Thu, Sep 26, 2013 at 10:27 PM, John Chilton wrote: > My recommendation would be make the tool dependency install work on as > many platforms as you can and not try to optimize in such a way that > it is not going to work - i.e. favor reproduciblity over performance. Reproducibility versus speed is a particular issue with floating point libraries - NumPy using ATLAS vs OpenBLAST vs Intel MKL vs just plain NumPy will probably all give slightly different answers (on some tasks), and at different speed. In this case, for simplicity I would advocate plain NumPy, without worrying about needing ATLAS. For packages needing NumPy with ALTAS, perhaps a new Tool Shed entry could be created, package_numpy_1_7_with_atlas or similar (based on the current configuration)? Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
On Thu, Sep 26, 2013 at 9:10 PM, Greg Von Kuster wrote: > James, it seems I was answering at the same time you were, so to highlight my > comments to John, I'm just wondering how this will work for repositories in > the tool shed that do not contain any tools, but just tool dependency > definitions or complex repository dependency definitions. I am certain our European colleagues will have use cases to contribute when they have had their coffee, but I would just assume not get the tool shed involved. If you want to optimize a tool for Galaxy, the best practice would be you optimize it by replacing the package the corresponding tool sources not by manually compiling some library and having the tool shed compile additional tools against it. Does that answer your question? -John > > Thanks, > > Greg Von Kuster > > On Sep 26, 2013, at 9:49 PM, James Taylor wrote: > >> Make the precedence a config option. Otherwise I agree. >> >> In addition, I still like the idea I suggested earlier of dependency >> provider plugins. Then you could (for example) have one that uses 'modules' >> and skips env.sh entirely. >> >> On Sep 26, 2013, at 9:15 PM, John Chilton wrote: >> >>> I was not even thinking we needed to modify the tool shed to implement >>> this. I was hoping (?) you could just modify: >>> >>> lib/galaxy/tools/deps/__init__.py >>> >>> to implement this. If some tool contains the tag >>> >>> numpy >>> >>> then if there is a manually installed tool_dependency in >>> `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence >>> over the tool shed installed version (would that be something like >>> `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me >>> know if this is way off base. >>> >>> There is a lot you could do to make this more complicated of course - >>> an interface for mapping exact tool shed dependencies to manually >>> installed ones, the ability to auto-compile tool shed dependencies >>> against manually installed libraries, etc..., but I am not sure those >>> complexities are buying you anything really. >>> >>> Thoughts? >>> >>> -John >>> >>> >>> >>> On Thu, Sep 26, 2013 at 5:47 PM, Greg Von Kuster wrote: Hi John, On Sep 26, 2013, at 5:27 PM, John Chilton wrote: > My recommendation would be make the tool dependency install work on as > many platforms as you can and not try to optimize in such a way that > it is not going to work - i.e. favor reproduciblity over performance. > If a system administrator or institution want to sacrifice > reproduciblity and optimize specific packages they should be able to > do so manually. Its not just Atlas and CPU throttling right? Its > vendor versions of MPI, GPGPU variants of code, variants of OpenMP, > etc Even if the tool shed provided some mechanism for determining > if some particular package optimization is going to work, perhaps its > better to just not enable it by default because frequently these cause > slightly different results than the unoptimized version. > > The problem with this recommendation that is Galaxy currently provides > no mechanism for doing so. Luckily this is easy to solve and the > solution solves other problems. If the tool dependency resolution code > would grab the manually configured dependency instead of the tool shed > variant when available, instead of favoring the opposite, then it > would be really easy to add in an optimized version of numpy or an MPI > version of software X. How would you like this to happen? Would it work to provide an admin the ability to create a ToolDependency object and point it to a "manually configured dependency" in whatever location on disk the admin chooses via a new UI feature? Or do you have a different idea? Thanks, Greg Von Kuster > > Whats great is this solves other problems as well. For instance, our > genomics Galaxy web server runs Debian but the worker nodes run > CentOS. This means many tool shed installed dependencies do not work. > JJ being the patient guy he is goes in and manually updates the tool > shed installed env.sh files to load modules. Even if you think not > running the same version of the OS on your server and worker nodes is > a bit crazy, there is the much more reasonable (common) case of just > wanting to submit to multiple different clusters. When I was talking > with the guys at NCGAS they were unsure how to do this, this one > change would make that a lot more tenable. > > -John > > On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning > wrote: >> Hi, >> >>> Hi Bjoern, >>> >>> Is there anything else we (the Galaxy community) can do to help >>> sort out the ATLAS installation problems? >> >> Thanks for asking. I have indeed a few things I would like some >> comments. >> >>>
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Simon, What is the advantage of putting that XML definition in the tool shed? It is not 100% true because of prior_install_required dependencies, but for the most part sourcing/load the environment for tools is a Galaxy problem, not so much a tool shed one. What if we did this instead? Add an option to Galaxy's universe_wsgi.ini with the following default: tool_dependency_resolution_order = gx_package_manual, gx_package_toolshed Which essentially implements my idea above, with James' additional configuration. But which can be overridden as: tool_dependency_resolution_order = plugin_module, gx_package_manual, gx_package_toolshed If set this way then placing bwa in a tool will result in the module bwa/0.5.9 being loaded if it is 'avail'able, else it will check for a manually installed env.sh (which is where MSI is currently putting its module loads), and else it will fallback to source the tool shed installed dependency. I feel like this will give you everything you want without any extra XML or configuration. Let me know if I am wrong. -John On Thu, Sep 26, 2013 at 11:39 PM, Guest, Simon wrote: > At Thu, 26 Sep 2013 22:03:09 -0400, > Greg Von Kuster wrote: >> >> Hi John, >> >> On Sep 26, 2013, at 9:15 PM, John Chilton wrote: >> >> > I was not even thinking we needed to modify the tool shed to implement >> > this. I was hoping (?) you could just modify: >> >> Nothing in the Tool Shed itself would be affected or require modification >> for this new feature as this is completely on the Galaxy side. >> >> > >> > lib/galaxy/tools/deps/__init__.py >> > >> > to implement this. If some tool contains the tag >> > >> > numpy >> > >> > then if there is a manually installed tool_dependency in >> > `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence >> > over the tool shed installed version (would that be something like >> > `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me >> > know if this is way off base. >> >> This is a possibility perhaps, but there seems to be a potential weakness in >> that it doesn't require the Tool Dependency object to exist since the tool >> will function without a installed dependency from the Tool Shed. Or, if the >> installed depndency s required, then it is meaningless because it won't be >> used. If the former case, then the tool dependency cannot be shared via the >> Tool Shed's dependency mechanism because none of the relationships will be >> defined since nothing is installed. Wouldn't it be better to allow the >> Galaxy admin to point the ToolDependency object to a specified binary on >> disk? In this way, all relationships defined by Tool Shed installations >> will work as expected, with all contained tools with that dependency point >> to that same shared location on disk. > > Hi Greg, John, > > What you are discussing is pretty close to what I am after, and what I > am prepared to spend some time working on, if that can help. This is > what I have posted about a couple of times previously. The option to > use a pre-installed package rather than a Galaxy installed one I think > would be very useful in general. I think it can be done in a way that > doesn't break the tool dependency model. > > I envisage providing access to existing programs on disk by loading > the relevant environment module. Platforms will differ greatly on > where they have programs installed (usr/local/bwa-0.5.9, > /usr/libexec/bwa-0.5.9, etc.). However, we could arrange to have a > conventionally named environment module available, so Galaxy just has > to be told somehow to do a module load bwa/0.5.9, say, prior to > trying to run that tool, which will then be found on the PATH. > > Hooking this in without killing the dependency on named and versioned > toolshed repos could work like this. In parallel with the repo sets > named, e.g. package_bwa_0_5_9, which download and build the package, > we have a set named like native_package_bwa_0_5_9. This could have a > tool_dependencies.xml file like this: > > > > > > > > > > > Uses native BWA via environment module bwa/0.5.9 > > > > > What this does is ensure that any repo which requires version 0.5.9 of > bwa can have this dependency resolved by either package_bwa_0_5_9, or > native_package_bwa_0_5_9. > > I envisage a Galaxy configuration setting that enables native package > and/or Galaxy package dependency resolution, and which one should be > preferred. Of course, if one of these has been manually installed, it > will be used as is. > > I would really like to be able to install tools from the toolshed, and > have them resolve their dependencies automatically using my > preinstalled application suite. I see that would also meet the needs > being discussed here. > > How does that sound? > > cheers, > Simon > > === > Attention: The info
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
At Thu, 26 Sep 2013 22:03:09 -0400, Greg Von Kuster wrote: > > Hi John, > > On Sep 26, 2013, at 9:15 PM, John Chilton wrote: > > > I was not even thinking we needed to modify the tool shed to implement > > this. I was hoping (?) you could just modify: > > Nothing in the Tool Shed itself would be affected or require modification for > this new feature as this is completely on the Galaxy side. > > > > > lib/galaxy/tools/deps/__init__.py > > > > to implement this. If some tool contains the tag > > > > numpy > > > > then if there is a manually installed tool_dependency in > > `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence > > over the tool shed installed version (would that be something like > > `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me > > know if this is way off base. > > This is a possibility perhaps, but there seems to be a potential weakness in > that it doesn't require the Tool Dependency object to exist since the tool > will function without a installed dependency from the Tool Shed. Or, if the > installed depndency s required, then it is meaningless because it won't be > used. If the former case, then the tool dependency cannot be shared via the > Tool Shed's dependency mechanism because none of the relationships will be > defined since nothing is installed. Wouldn't it be better to allow the > Galaxy admin to point the ToolDependency object to a specified binary on > disk? In this way, all relationships defined by Tool Shed installations will > work as expected, with all contained tools with that dependency point to that > same shared location on disk. Hi Greg, John, What you are discussing is pretty close to what I am after, and what I am prepared to spend some time working on, if that can help. This is what I have posted about a couple of times previously. The option to use a pre-installed package rather than a Galaxy installed one I think would be very useful in general. I think it can be done in a way that doesn't break the tool dependency model. I envisage providing access to existing programs on disk by loading the relevant environment module. Platforms will differ greatly on where they have programs installed (usr/local/bwa-0.5.9, /usr/libexec/bwa-0.5.9, etc.). However, we could arrange to have a conventionally named environment module available, so Galaxy just has to be told somehow to do a module load bwa/0.5.9, say, prior to trying to run that tool, which will then be found on the PATH. Hooking this in without killing the dependency on named and versioned toolshed repos could work like this. In parallel with the repo sets named, e.g. package_bwa_0_5_9, which download and build the package, we have a set named like native_package_bwa_0_5_9. This could have a tool_dependencies.xml file like this: Uses native BWA via environment module bwa/0.5.9 What this does is ensure that any repo which requires version 0.5.9 of bwa can have this dependency resolved by either package_bwa_0_5_9, or native_package_bwa_0_5_9. I envisage a Galaxy configuration setting that enables native package and/or Galaxy package dependency resolution, and which one should be preferred. Of course, if one of these has been manually installed, it will be used as is. I would really like to be able to install tools from the toolshed, and have them resolve their dependencies automatically using my preinstalled application suite. I see that would also meet the needs being discussed here. How does that sound? cheers, Simon === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. === ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
James, it seems I was answering at the same time you were, so to highlight my comments to John, I'm just wondering how this will work for repositories in the tool shed that do not contain any tools, but just tool dependency definitions or complex repository dependency definitions. Thanks, Greg Von Kuster On Sep 26, 2013, at 9:49 PM, James Taylor wrote: > Make the precedence a config option. Otherwise I agree. > > In addition, I still like the idea I suggested earlier of dependency provider > plugins. Then you could (for example) have one that uses 'modules' and skips > env.sh entirely. > > On Sep 26, 2013, at 9:15 PM, John Chilton wrote: > >> I was not even thinking we needed to modify the tool shed to implement >> this. I was hoping (?) you could just modify: >> >> lib/galaxy/tools/deps/__init__.py >> >> to implement this. If some tool contains the tag >> >> numpy >> >> then if there is a manually installed tool_dependency in >> `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence >> over the tool shed installed version (would that be something like >> `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me >> know if this is way off base. >> >> There is a lot you could do to make this more complicated of course - >> an interface for mapping exact tool shed dependencies to manually >> installed ones, the ability to auto-compile tool shed dependencies >> against manually installed libraries, etc..., but I am not sure those >> complexities are buying you anything really. >> >> Thoughts? >> >> -John >> >> >> >> On Thu, Sep 26, 2013 at 5:47 PM, Greg Von Kuster wrote: >>> Hi John, >>> >>> On Sep 26, 2013, at 5:27 PM, John Chilton wrote: >>> My recommendation would be make the tool dependency install work on as many platforms as you can and not try to optimize in such a way that it is not going to work - i.e. favor reproduciblity over performance. If a system administrator or institution want to sacrifice reproduciblity and optimize specific packages they should be able to do so manually. Its not just Atlas and CPU throttling right? Its vendor versions of MPI, GPGPU variants of code, variants of OpenMP, etc Even if the tool shed provided some mechanism for determining if some particular package optimization is going to work, perhaps its better to just not enable it by default because frequently these cause slightly different results than the unoptimized version. The problem with this recommendation that is Galaxy currently provides no mechanism for doing so. Luckily this is easy to solve and the solution solves other problems. If the tool dependency resolution code would grab the manually configured dependency instead of the tool shed variant when available, instead of favoring the opposite, then it would be really easy to add in an optimized version of numpy or an MPI version of software X. >>> >>> How would you like this to happen? Would it work to provide an admin the >>> ability to create a ToolDependency object and point it to a "manually >>> configured dependency" in whatever location on disk the admin chooses via a >>> new UI feature? Or do you have a different idea? >>> >>> Thanks, >>> >>> Greg Von Kuster >>> >>> Whats great is this solves other problems as well. For instance, our genomics Galaxy web server runs Debian but the worker nodes run CentOS. This means many tool shed installed dependencies do not work. JJ being the patient guy he is goes in and manually updates the tool shed installed env.sh files to load modules. Even if you think not running the same version of the OS on your server and worker nodes is a bit crazy, there is the much more reasonable (common) case of just wanting to submit to multiple different clusters. When I was talking with the guys at NCGAS they were unsure how to do this, this one change would make that a lot more tenable. -John On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning wrote: > Hi, > >> Hi Bjoern, >> >> Is there anything else we (the Galaxy community) can do to help >> sort out the ATLAS installation problems? > > Thanks for asking. I have indeed a few things I would like some > comments. > >> Another choice might be to use OpenBLAS instead of ATLAS, e.g. >> http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration > > I have no experience with it. Does it also need to turn off CPU > throttling? I would assume so, or how is it optimizing itself? > >> However, I think we build NumPy without using ATLAS or any >> BLAS library. That seems like the most pragmatic solution >> in the short term - which I think is what Dan tried here: >> http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7 >
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Hi John, On Sep 26, 2013, at 9:15 PM, John Chilton wrote: > I was not even thinking we needed to modify the tool shed to implement > this. I was hoping (?) you could just modify: Nothing in the Tool Shed itself would be affected or require modification for this new feature as this is completely on the Galaxy side. > > lib/galaxy/tools/deps/__init__.py > > to implement this. If some tool contains the tag > > numpy > > then if there is a manually installed tool_dependency in > `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence > over the tool shed installed version (would that be something like > `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me > know if this is way off base. This is a possibility perhaps, but there seems to be a potential weakness in that it doesn't require the Tool Dependency object to exist since the tool will function without a installed dependency from the Tool Shed. Or, if the installed depndency s required, then it is meaningless because it won't be used. If the former case, then the tool dependency cannot be shared via the Tool Shed's dependency mechanism because none of the relationships will be defined since nothing is installed. Wouldn't it be better to allow the Galaxy admin to point the ToolDependency object to a specified binary on disk? In this way, all relationships defined by Tool Shed installations will work as expected, with all contained tools with that dependency point to that same shared location on disk. > > There is a lot you could do to make this more complicated of course - > an interface for mapping exact tool shed dependencies to manually > installed ones, the ability to auto-compile tool shed dependencies > against manually installed libraries, etc..., but I am not sure those > complexities are buying you anything really. This is certainly a debatable topic, but I'm not seeing how my approach creates more complexity. The Galaxy admin is required to manually compile the binary dependency in either case. I'm just providing him an easy UI feature to enable a ToolDependency object that can be shared by any number of tools contained in any number of repositories installed from the Tool Shed to locate it. Using this approach, the Galaxy admin can either choose to install thje dependency from the Tool Shed or manually compile hte dependency and have the ToolDependency object point to it. In either case,all Tool Shed dependency definitions (both repository and tool depndencies) would all work as expected with additional repository installs. Greg Von Kuster > > Thoughts? > > -John > > > > On Thu, Sep 26, 2013 at 5:47 PM, Greg Von Kuster wrote: >> Hi John, >> >> On Sep 26, 2013, at 5:27 PM, John Chilton wrote: >> >>> My recommendation would be make the tool dependency install work on as >>> many platforms as you can and not try to optimize in such a way that >>> it is not going to work - i.e. favor reproduciblity over performance. >>> If a system administrator or institution want to sacrifice >>> reproduciblity and optimize specific packages they should be able to >>> do so manually. Its not just Atlas and CPU throttling right? Its >>> vendor versions of MPI, GPGPU variants of code, variants of OpenMP, >>> etc Even if the tool shed provided some mechanism for determining >>> if some particular package optimization is going to work, perhaps its >>> better to just not enable it by default because frequently these cause >>> slightly different results than the unoptimized version. >>> >>> The problem with this recommendation that is Galaxy currently provides >>> no mechanism for doing so. Luckily this is easy to solve and the >>> solution solves other problems. If the tool dependency resolution code >>> would grab the manually configured dependency instead of the tool shed >>> variant when available, instead of favoring the opposite, then it >>> would be really easy to add in an optimized version of numpy or an MPI >>> version of software X. >> >> How would you like this to happen? Would it work to provide an admin the >> ability to create a ToolDependency object and point it to a "manually >> configured dependency" in whatever location on disk the admin chooses via a >> new UI feature? Or do you have a different idea? >> >> Thanks, >> >> Greg Von Kuster >> >> >>> >>> Whats great is this solves other problems as well. For instance, our >>> genomics Galaxy web server runs Debian but the worker nodes run >>> CentOS. This means many tool shed installed dependencies do not work. >>> JJ being the patient guy he is goes in and manually updates the tool >>> shed installed env.sh files to load modules. Even if you think not >>> running the same version of the OS on your server and worker nodes is >>> a bit crazy, there is the much more reasonable (common) case of just >>> wanting to submit to multiple different clusters. When I was talking >>> with the guys at NCGAS they were
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
I was not even thinking we needed to modify the tool shed to implement this. I was hoping (?) you could just modify: lib/galaxy/tools/deps/__init__.py to implement this. If some tool contains the tag numpy then if there is a manually installed tool_dependency in `tool_dependency_dir/numpy/1.7.1/env.sh` that would take precedence over the tool shed installed version (would that be something like `tool_dependency_dir/numpy/1.7.1/owner/name/changeset/env.sh`)? Let me know if this is way off base. There is a lot you could do to make this more complicated of course - an interface for mapping exact tool shed dependencies to manually installed ones, the ability to auto-compile tool shed dependencies against manually installed libraries, etc..., but I am not sure those complexities are buying you anything really. Thoughts? -John On Thu, Sep 26, 2013 at 5:47 PM, Greg Von Kuster wrote: > Hi John, > > On Sep 26, 2013, at 5:27 PM, John Chilton wrote: > >> My recommendation would be make the tool dependency install work on as >> many platforms as you can and not try to optimize in such a way that >> it is not going to work - i.e. favor reproduciblity over performance. >> If a system administrator or institution want to sacrifice >> reproduciblity and optimize specific packages they should be able to >> do so manually. Its not just Atlas and CPU throttling right? Its >> vendor versions of MPI, GPGPU variants of code, variants of OpenMP, >> etc Even if the tool shed provided some mechanism for determining >> if some particular package optimization is going to work, perhaps its >> better to just not enable it by default because frequently these cause >> slightly different results than the unoptimized version. >> >> The problem with this recommendation that is Galaxy currently provides >> no mechanism for doing so. Luckily this is easy to solve and the >> solution solves other problems. If the tool dependency resolution code >> would grab the manually configured dependency instead of the tool shed >> variant when available, instead of favoring the opposite, then it >> would be really easy to add in an optimized version of numpy or an MPI >> version of software X. > > How would you like this to happen? Would it work to provide an admin the > ability to create a ToolDependency object and point it to a "manually > configured dependency" in whatever location on disk the admin chooses via a > new UI feature? Or do you have a different idea? > > Thanks, > > Greg Von Kuster > > >> >> Whats great is this solves other problems as well. For instance, our >> genomics Galaxy web server runs Debian but the worker nodes run >> CentOS. This means many tool shed installed dependencies do not work. >> JJ being the patient guy he is goes in and manually updates the tool >> shed installed env.sh files to load modules. Even if you think not >> running the same version of the OS on your server and worker nodes is >> a bit crazy, there is the much more reasonable (common) case of just >> wanting to submit to multiple different clusters. When I was talking >> with the guys at NCGAS they were unsure how to do this, this one >> change would make that a lot more tenable. >> >> -John >> >> On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning >> wrote: >>> Hi, >>> Hi Bjoern, Is there anything else we (the Galaxy community) can do to help sort out the ATLAS installation problems? >>> >>> Thanks for asking. I have indeed a few things I would like some >>> comments. >>> Another choice might be to use OpenBLAS instead of ATLAS, e.g. http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration >>> >>> I have no experience with it. Does it also need to turn off CPU >>> throttling? I would assume so, or how is it optimizing itself? >>> However, I think we build NumPy without using ATLAS or any BLAS library. That seems like the most pragmatic solution in the short term - which I think is what Dan tried here: http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7 >>> >>> I can remove them if that is the consensus. >>> >>> A few points: >>> - fixing the atlas issue can speed up numpy, scipy, R considerably (by >>> 400% in some cases) >>> - as far as I understand that performance gain is due to optimizing >>> itself on specific hardware, for atlas there is no way around to disable >>> CPU throttling (How about OpenBlas?) >>> - it seems to be complicated to deactivate CPU throttling on OS-X >>> - binary installation does not make sense in that case, because ATLAS is >>> self optimizing >>> - Distribution shipped ATLAS packages are not really faster >>> >>> Current state: >>> - Atlas tries two different commands to deactivate CPU throttling. Afaik >>> that only works on some Ubuntu versions, where no root privileges are >>> necessary. >>> - If atlas fails for some reason, numpy/R/scipy installation should not >>> be affected (that's was at least the aim) >>
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Hi John, On Sep 26, 2013, at 5:27 PM, John Chilton wrote: > My recommendation would be make the tool dependency install work on as > many platforms as you can and not try to optimize in such a way that > it is not going to work - i.e. favor reproduciblity over performance. > If a system administrator or institution want to sacrifice > reproduciblity and optimize specific packages they should be able to > do so manually. Its not just Atlas and CPU throttling right? Its > vendor versions of MPI, GPGPU variants of code, variants of OpenMP, > etc Even if the tool shed provided some mechanism for determining > if some particular package optimization is going to work, perhaps its > better to just not enable it by default because frequently these cause > slightly different results than the unoptimized version. > > The problem with this recommendation that is Galaxy currently provides > no mechanism for doing so. Luckily this is easy to solve and the > solution solves other problems. If the tool dependency resolution code > would grab the manually configured dependency instead of the tool shed > variant when available, instead of favoring the opposite, then it > would be really easy to add in an optimized version of numpy or an MPI > version of software X. How would you like this to happen? Would it work to provide an admin the ability to create a ToolDependency object and point it to a "manually configured dependency" in whatever location on disk the admin chooses via a new UI feature? Or do you have a different idea? Thanks, Greg Von Kuster > > Whats great is this solves other problems as well. For instance, our > genomics Galaxy web server runs Debian but the worker nodes run > CentOS. This means many tool shed installed dependencies do not work. > JJ being the patient guy he is goes in and manually updates the tool > shed installed env.sh files to load modules. Even if you think not > running the same version of the OS on your server and worker nodes is > a bit crazy, there is the much more reasonable (common) case of just > wanting to submit to multiple different clusters. When I was talking > with the guys at NCGAS they were unsure how to do this, this one > change would make that a lot more tenable. > > -John > > On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning > wrote: >> Hi, >> >>> Hi Bjoern, >>> >>> Is there anything else we (the Galaxy community) can do to help >>> sort out the ATLAS installation problems? >> >> Thanks for asking. I have indeed a few things I would like some >> comments. >> >>> Another choice might be to use OpenBLAS instead of ATLAS, e.g. >>> http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration >> >> I have no experience with it. Does it also need to turn off CPU >> throttling? I would assume so, or how is it optimizing itself? >> >>> However, I think we build NumPy without using ATLAS or any >>> BLAS library. That seems like the most pragmatic solution >>> in the short term - which I think is what Dan tried here: >>> http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7 >> >> I can remove them if that is the consensus. >> >> A few points: >> - fixing the atlas issue can speed up numpy, scipy, R considerably (by >> 400% in some cases) >> - as far as I understand that performance gain is due to optimizing >> itself on specific hardware, for atlas there is no way around to disable >> CPU throttling (How about OpenBlas?) >> - it seems to be complicated to deactivate CPU throttling on OS-X >> - binary installation does not make sense in that case, because ATLAS is >> self optimizing >> - Distribution shipped ATLAS packages are not really faster >> >> Current state: >> - Atlas tries two different commands to deactivate CPU throttling. Afaik >> that only works on some Ubuntu versions, where no root privileges are >> necessary. >> - If atlas fails for some reason, numpy/R/scipy installation should not >> be affected (that's was at least the aim) >> >> Questions: >> - Is it worth the hassle for some speed improvements? pip install numpy, >> would be so easy? >> >> - If we want to support ATLAS, any better idea to how to implement it? >> Any Tool Shed feature that can help? -> interactive installation? >> - can we flag a tool dependency as optional? So it can fail? >> >> - Can anyone help with testing and fixing it? >> >> >> Any opinions/comments? >> Bjoern >> >>> Thanks, >>> >>> Peter >> >> >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > > ___ > Please keep all replies on the list by using "reply all" > in your
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
My recommendation would be make the tool dependency install work on as many platforms as you can and not try to optimize in such a way that it is not going to work - i.e. favor reproduciblity over performance. If a system administrator or institution want to sacrifice reproduciblity and optimize specific packages they should be able to do so manually. Its not just Atlas and CPU throttling right? Its vendor versions of MPI, GPGPU variants of code, variants of OpenMP, etc Even if the tool shed provided some mechanism for determining if some particular package optimization is going to work, perhaps its better to just not enable it by default because frequently these cause slightly different results than the unoptimized version. The problem with this recommendation that is Galaxy currently provides no mechanism for doing so. Luckily this is easy to solve and the solution solves other problems. If the tool dependency resolution code would grab the manually configured dependency instead of the tool shed variant when available, instead of favoring the opposite, then it would be really easy to add in an optimized version of numpy or an MPI version of software X. Whats great is this solves other problems as well. For instance, our genomics Galaxy web server runs Debian but the worker nodes run CentOS. This means many tool shed installed dependencies do not work. JJ being the patient guy he is goes in and manually updates the tool shed installed env.sh files to load modules. Even if you think not running the same version of the OS on your server and worker nodes is a bit crazy, there is the much more reasonable (common) case of just wanting to submit to multiple different clusters. When I was talking with the guys at NCGAS they were unsure how to do this, this one change would make that a lot more tenable. -John On Thu, Sep 26, 2013 at 1:29 PM, Björn Grüning wrote: > Hi, > >> Hi Bjoern, >> >> Is there anything else we (the Galaxy community) can do to help >> sort out the ATLAS installation problems? > > Thanks for asking. I have indeed a few things I would like some > comments. > >> Another choice might be to use OpenBLAS instead of ATLAS, e.g. >> http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration > > I have no experience with it. Does it also need to turn off CPU > throttling? I would assume so, or how is it optimizing itself? > >> However, I think we build NumPy without using ATLAS or any >> BLAS library. That seems like the most pragmatic solution >> in the short term - which I think is what Dan tried here: >> http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7 > > I can remove them if that is the consensus. > > A few points: > - fixing the atlas issue can speed up numpy, scipy, R considerably (by > 400% in some cases) > - as far as I understand that performance gain is due to optimizing > itself on specific hardware, for atlas there is no way around to disable > CPU throttling (How about OpenBlas?) > - it seems to be complicated to deactivate CPU throttling on OS-X > - binary installation does not make sense in that case, because ATLAS is > self optimizing > - Distribution shipped ATLAS packages are not really faster > > Current state: > - Atlas tries two different commands to deactivate CPU throttling. Afaik > that only works on some Ubuntu versions, where no root privileges are > necessary. > - If atlas fails for some reason, numpy/R/scipy installation should not > be affected (that's was at least the aim) > > Questions: > - Is it worth the hassle for some speed improvements? pip install numpy, > would be so easy? > > - If we want to support ATLAS, any better idea to how to implement it? > Any Tool Shed feature that can help? -> interactive installation? > - can we flag a tool dependency as optional? So it can fail? > > - Can anyone help with testing and fixing it? > > > Any opinions/comments? > Bjoern > >> Thanks, >> >> Peter > > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Hi, > Hi Bjoern, > > Is there anything else we (the Galaxy community) can do to help > sort out the ATLAS installation problems? Thanks for asking. I have indeed a few things I would like some comments. > Another choice might be to use OpenBLAS instead of ATLAS, e.g. > http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration I have no experience with it. Does it also need to turn off CPU throttling? I would assume so, or how is it optimizing itself? > However, I think we build NumPy without using ATLAS or any > BLAS library. That seems like the most pragmatic solution > in the short term - which I think is what Dan tried here: > http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7 I can remove them if that is the consensus. A few points: - fixing the atlas issue can speed up numpy, scipy, R considerably (by 400% in some cases) - as far as I understand that performance gain is due to optimizing itself on specific hardware, for atlas there is no way around to disable CPU throttling (How about OpenBlas?) - it seems to be complicated to deactivate CPU throttling on OS-X - binary installation does not make sense in that case, because ATLAS is self optimizing - Distribution shipped ATLAS packages are not really faster Current state: - Atlas tries two different commands to deactivate CPU throttling. Afaik that only works on some Ubuntu versions, where no root privileges are necessary. - If atlas fails for some reason, numpy/R/scipy installation should not be affected (that's was at least the aim) Questions: - Is it worth the hassle for some speed improvements? pip install numpy, would be so easy? - If we want to support ATLAS, any better idea to how to implement it? Any Tool Shed feature that can help? -> interactive installation? - can we flag a tool dependency as optional? So it can fail? - Can anyone help with testing and fixing it? Any opinions/comments? Bjoern > Thanks, > > Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
In case this will help, I have the framework implemented (and committed) for handling pre-compioled binaries for tool dependencies for a supported set of architectures. Dave B has updated a lot of tool dependency definitions on both the test and main tool sheds to use this enhancement - those that he has updated are currently all owned by the devteam in preparation for a tool migration stage that he will soon be committiing. Perhaps the atlas tool dependency can be updated to proved a pre-compiled binary installation. Greg Von Kuster On Sep 26, 2013, at 11:10 AM, Peter Cock wrote: > Hi Bjoern, > > Is there anything else we (the Galaxy community) can do to help > sort out the ATLAS installation problems? > > Another choice might be to use OpenBLAS instead of ATLAS, e.g. > http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration > > However, I think we build NumPy without using ATLAS or any > BLAS library. That seems like the most pragmatic solution > in the short term - which I think is what Dan tried here: > http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7 > > Thanks, > > Peter > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Hi Bjoern, Is there anything else we (the Galaxy community) can do to help sort out the ATLAS installation problems? Another choice might be to use OpenBLAS instead of ATLAS, e.g. http://stackoverflow.com/questions/11443302/compiling-numpy-with-openblas-integration However, I think we build NumPy without using ATLAS or any BLAS library. That seems like the most pragmatic solution in the short term - which I think is what Dan tried here: http://testtoolshed.g2.bx.psu.edu/view/blankenberg/package_numpy_1_7 Thanks, Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Forgot to copy the list on this email. On Mon, Sep 23, 2013 at 9:32 AM, Carlos Borroto wrote: > On Fri, Sep 20, 2013 at 6:12 PM, Björn Grüning > wrote: >> Hi Carlos, >> >> Can you try again? >> Also the new unstable version if you can. Thanks for the help! ATLAS is >> a beast :( >> > > Sorry for the delayed response. Busy weekend. > > Trying to install package_atlas_3_10 on Ubuntu 13.04: > > STDERR > It appears you have cpu throttling enabled, which makes timings > unreliable and an ATLAS install nonsensical. Aborting. > See ATLAS/INSTALL.txt for further information > # > > > I think this is the other relevant part in the log: > > # try to disable cpu throttling > if hash cpufreq-selector 2>/dev/null; then > cpufreq-selector -g performance > elif hash cpupower 2>/dev/null; then > cpupower frequency-set -g performance > else > echo 'Please deactivate CPU throttling by your > own, or install cpufreq-selector' > exit > fi > > STDERR > Error calling SetGovernor: Caller is not authorized > > I had to install 'cpufreq-selector' to get here, not installed by > default. I can confirm I get the same error when trying to run this > command directly: > $ cpufreq-selector -g performance > Error calling SetGovernor: Caller is not authorized > > package_atlas_3_11 fails in exactly the same way in this box. > >>> Somehow this is also a silent failing and numpy, biopython and >>> ngs-tools(my tool package depending on biopython) get >>> "Installed"(Green) >> >> Yes that is correct. I designed it (in theory) in that way, that if >> ATLAS crashes (due to CPU throttling enabled) every other package can be >> installed without problem. Every other behaviour is a bug. >> >>> while lapack, atlas and split_by_barcode(my actual >>> tool wrapper depending on ngs-tools package) get "Installed, missing >>> tool dependencies"(Grey). This means if I try to use my wrapper in >>> this state I get this error: >>> /bin/sh: 1: ngs-tools: not found >> >> On Ubuntu it gets installed without errors. Everything is green :) >> >>> However, if I do a "Repair repository" on split_by_barcode it goes >>> into "Installed"(Green) and everything seems to work from then on. >> >> Mh, thats seems to be a bug. I will try to reproduce on a computer where >> ATLAS is crashing. >> > > Thanks for testing this. I also believe this might be a bug. > --Carlos ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
On Fri, Sep 20, 2013 at 8:48 AM, Bjoern Gruening wrote: > Hi, > > I tried to things to solve it. > > 1: I uploaded a new version to: > http://testtoolshed.g2.bx.psu.edu/view/iuc/package_atlas_3_10 this time > with target_filename="ATLAS.tar.bz2". That is a rather new feature, so > you will need a recent Galaxy version. The prosite is that we do not > need to call 'tar', we are using the python tarfile module. Hopefully > that will work, also on OS-X > > 2: I have created: > http://testtoolshed.g2.bx.psu.edu/view/iuc/package_atlas_3_11 > Peter was able to install it on OS-X, at least on commandline, so I have > hope that this one is working, well. The downside is, that it is a > unstable version and will disappear from the server. > > Would be great if you can give both a try. For me its working. That > means on ubuntu its installed and on Fedora its failing silently with > 'It appears you have cpu throttling enabled, which makes timings > unreliable and an ATLAS install nonsensical. Aborting. See > ATLAS/INSTALL.txt for further information'. > I don't have access to my OS X box until later tonight. On Ubuntu with an up-to-date instance of galaxy-central is failing now for me with a different error: STDOUT ./xconfig -d s /home/cborroto/src/galaxy-central/database/tmp/tmp-toolshed-mtdyWTv64/ATLAS/build/../ -d b /home/cborroto/src/galaxy-central/database/tmp/tmp-toolshed-mtdyWTv64/ATLAS/build -Fa alg -fPIC -Si lapa ckref 1 xconfig exited with 127 # # mkdir build && cd build && mkdir /home/cborroto/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/d32504e1aea8/atlas/ && ../configure -Fa alg -fPIC --prefix=/home/cborroto/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/d32504e1aea8/atlas/ --with-netlib-lapack-tarfile=../lapack-3.4.2.tgz STDERR cat: ..//CONFIG/src/Makefile: No such file or directory cp: cannot stat ‘..//CONFIG/src/atlcomp.txt’: No such file or directory make: *** No rule to make target `xconfig'. Stop. /bin/sh: 1: ./xconfig: not found Full INSTALLATION.log: http://pastebin.com/raw.php?i=hvqFWvgJ Somehow this is also a silent failing and numpy, biopython and ngs-tools(my tool package depending on biopython) get "Installed"(Green) while lapack, atlas and split_by_barcode(my actual tool wrapper depending on ngs-tools package) get "Installed, missing tool dependencies"(Grey). This means if I try to use my wrapper in this state I get this error: /bin/sh: 1: ngs-tools: not found However, if I do a "Repair repository" on split_by_barcode it goes into "Installed"(Green) and everything seems to work from then on. Thanks for working on this, Carlos ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
> On Fri, Sep 20, 2013 at 1:48 PM, Bjoern Gruening > wrote: > > Hi, > > > > I tried to things to solve it. > > Was this specifically to solve ATLAS under Mac OS X, > or more generally include the problem shown on the > Galaxy (Test) Tool Shed? Yes. My hope was to use tarfile from python and not 'tar', because bsdtar seems to crash with that package. > > 1: I uploaded a new version to: > > http://testtoolshed.g2.bx.psu.edu/view/iuc/package_atlas_3_10 this time > > with target_filename="ATLAS.tar.bz2". That is a rather new feature, so > > you will need a recent Galaxy version. The prosite is that we do not > > need to call 'tar', we are using the python tarfile module. Hopefully > > that will work, also on OS-X > > Not tried. > > > 2: I have created: > > http://testtoolshed.g2.bx.psu.edu/view/iuc/package_atlas_3_11 > > Peter was able to install it on OS-X, at least on commandline, so I have > > hope that this one is working, well. The downside is, that it is a > > unstable version and will disappear from the server. > > Tried on a CentOS machine (running galaxy-central), it appeared > to work but on closer inspection: > > Missing tool dependencies - click the name to install the missing dependency > Name Version Type Installation status > atlas 3.11.11 package Error > > $ cat > /opt/galaxy-dist-shed-tools/atlas/3.11.11/iuc/package_atlas_3_11/3e1635c33e26/INSTALLATION.log > > # > > # try to disable cpu throttling > if hash cpufreq-selector 2>/dev/null; then > cpufreq-selector -g performance > elif hash cpupower 2>/dev/null; then > cpupower frequency-set -g performance > else > echo 'Please deactivate CPU throttling by your > own, or install cpufreq-selector' > exit > fi > > STDOUT > > # > > # > > # try to disable cpu throttling > if hash cpufreq-selector 2>/dev/null; then > cpufreq-selector -g performance > elif hash cpupower 2>/dev/null; then > cpupower frequency-set -g performance > else > echo 'Please deactivate CPU throttling by your > own, or install cpufreq-selector' > exit > fi > > STDERR > You must be root > # > > > > Would be great if you can give both a try. For me its working. That > > means on ubuntu its installed and on Fedora its failing silently with > > 'It appears you have cpu throttling enabled, which makes timings > > unreliable and an ATLAS install nonsensical. Aborting. See > > ATLAS/INSTALL.txt for further information'. > > Like Fedora, on CentOS I am getting a silent failure :( For whatever reason I get the following in my log. Galaxy tries to continue with the installation, that fine. But in your case Galaxy aborts. Nevertheless, failing silently means, that biopython and numpy will not be affected (hopefully). I can also remove the manuell deactivation of cpu throttling, it was just an idea to support more systems out of the box without admin invention. # # try to disable cpu throttling if hash cpufreq-selector 2>/dev/null; then cpufreq-selector -g performance elif hash cpupower 2>/dev/null; then cpupower frequency-set -g performance else echo 'Please deactivate CPU throttling by your own, or install cpufreq-selector' exit fi STDOUT # # # try to disable cpu throttling if hash cpufreq-selector 2>/dev/null; then cpufreq-selector -g performance elif hash cpupower 2>/dev/null; then cpupower frequency-set -g performance else echo 'Please deactivate CPU throttling by your own, or install cpufreq-selector' exit fi STDERR Error calling SetGovernor: Caller is not authorized # # mkdir build && cd build && mkdir /home/gruening/projects/code/galaxy-central/tool_deps/atlas/3.11.11/iuc/package_atlas_3_11/3e1635c33e26/atlas/ && ../configure -Fa alg -fPIC --prefix=/home/gruening/projects/code/galaxy-central/tool_deps/atlas/3.11.11/iuc/package_atlas_3_11/3e1635c33e26/atlas/ --with-netlib-lapack-tarfile=../lapack-3.4.2.tgz STDOUT gcc -I/home/g
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
On Fri, Sep 20, 2013 at 1:48 PM, Bjoern Gruening wrote: > Hi, > > I tried to things to solve it. Was this specifically to solve ATLAS under Mac OS X, or more generally include the problem shown on the Galaxy (Test) Tool Shed? > 1: I uploaded a new version to: > http://testtoolshed.g2.bx.psu.edu/view/iuc/package_atlas_3_10 this time > with target_filename="ATLAS.tar.bz2". That is a rather new feature, so > you will need a recent Galaxy version. The prosite is that we do not > need to call 'tar', we are using the python tarfile module. Hopefully > that will work, also on OS-X Not tried. > 2: I have created: > http://testtoolshed.g2.bx.psu.edu/view/iuc/package_atlas_3_11 > Peter was able to install it on OS-X, at least on commandline, so I have > hope that this one is working, well. The downside is, that it is a > unstable version and will disappear from the server. Tried on a CentOS machine (running galaxy-central), it appeared to work but on closer inspection: Missing tool dependencies - click the name to install the missing dependency Name Version Type Installation status atlas 3.11.11 package Error $ cat /opt/galaxy-dist-shed-tools/atlas/3.11.11/iuc/package_atlas_3_11/3e1635c33e26/INSTALLATION.log # # try to disable cpu throttling if hash cpufreq-selector 2>/dev/null; then cpufreq-selector -g performance elif hash cpupower 2>/dev/null; then cpupower frequency-set -g performance else echo 'Please deactivate CPU throttling by your own, or install cpufreq-selector' exit fi STDOUT # # # try to disable cpu throttling if hash cpufreq-selector 2>/dev/null; then cpufreq-selector -g performance elif hash cpupower 2>/dev/null; then cpupower frequency-set -g performance else echo 'Please deactivate CPU throttling by your own, or install cpufreq-selector' exit fi STDERR You must be root # > Would be great if you can give both a try. For me its working. That > means on ubuntu its installed and on Fedora its failing silently with > 'It appears you have cpu throttling enabled, which makes timings > unreliable and an ATLAS install nonsensical. Aborting. See > ATLAS/INSTALL.txt for further information'. Like Fedora, on CentOS I am getting a silent failure :( Peter ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Hi, I tried to things to solve it. 1: I uploaded a new version to: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_atlas_3_10 this time with target_filename="ATLAS.tar.bz2". That is a rather new feature, so you will need a recent Galaxy version. The prosite is that we do not need to call 'tar', we are using the python tarfile module. Hopefully that will work, also on OS-X 2: I have created: http://testtoolshed.g2.bx.psu.edu/view/iuc/package_atlas_3_11 Peter was able to install it on OS-X, at least on commandline, so I have hope that this one is working, well. The downside is, that it is a unstable version and will disappear from the server. Would be great if you can give both a try. For me its working. That means on ubuntu its installed and on Fedora its failing silently with 'It appears you have cpu throttling enabled, which makes timings unreliable and an ATLAS install nonsensical. Aborting. See ATLAS/INSTALL.txt for further information'. Thanks for your help! Bjoern > On Thu, Sep 19, 2013 at 5:15 PM, Björn Grüning > wrote: > > Hi Peter and Carlos, > > > >> On Mon, Sep 16, 2013 at 8:57 PM, Carlos Borroto > >> wrote: > >> > I did an extra test. Started with a clean 'galaxy-dist'. This time > >> > both repositories fail with the same error. I guess before something > >> > was cached for the repository with version 0.1.4. > >> > > >> > I used biopython repository as a guide to write my tool dependency > >> > definition: > >> > http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61 > >> > > >> > I can confirm biopython repository is failing to install for me with > >> > exactly the same error. I wonder if a recently addition in the test > >> > toolshed broke the treatment of "prior_installation_required". > >> > > >> > Thanks, > >> > Carlos > >> > >> Could be - note that currently Biopython isn't currently > >> installing properly on the Test Tool Shed due to ALTAS > >> failing (a requirement of NumPy which is a requirement > >> of Biopython). Dave and Bjoern are I think looking at this > >> already... > > > > I can't do much I tested it again and for me its working fine on my > > computers I have at hand ... sorry. > > > > In case it helps, this is how the INSTALLATION.log file ends on OS X 10.8.4: > $ tail -n 3 > ~/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/3508de0ebae1/INSTALLATION.log > x ATLAS/tune/lapack/lanbsrch.c > tar: Error exit delayed from previous errors. > # > > This is the relevant part I can find in Galaxy's log: > [localhost] local: tar xfvj atlas3.10.1.tar.bz2 > > Warning: local() encountered an error (return code 1) while executing > 'tar xfvj atlas3.10.1.tar.bz2' > > After noticing this I got what I'm guessing is the original file from > sourceforge: > http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10.1.tar.bz2 > > I can confirm that trying to untar this file fails with the exact same > error. However, on Ubuntu 13.04 untaring this file works just fine. > > On Ubuntu 13.04 the error I see is: > STDERR > It appears you have cpu throttling enabled, which makes timings > unreliable and an ATLAS install nonsensical. Aborting. > See ATLAS/INSTALL.txt for further information > # > > Björn, you said cpu throttling can be easily disable on Ubuntu. Can > you comment how? Do I need to disable it completely, or should the > install script do it when installing? > > Thanks, > Carlos ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
On Fri, Sep 20, 2013 at 10:17 AM, Bjoern Gruening wrote: > Hi Ido and Carlos, > > can you check if that tarball is working? > > http://downloads.sourceforge.net/project/math-atlas/Developer%20%28unstable%29/3.11.11/atlas3.11.11.tar.bz2 > > The chance is low, but if its working for you I will consider to create a > new version for it. > Thanks, > Bjoern Hi Bjoern, That archive file works for me on Mac OS X 10.8.5, details below. Peter -- I was fooled for a moment, it is one of the annoying SourceForge URLs which doesn't work at the command line - their mirror system is horrid: $ curl -O http://downloads.sourceforge.net/project/math-atlas/Developer%20%28unstable%29/3.11.11/atlas3.11.11.tar.bz2 % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 So, getting a real link via my browser: $ curl -O http://kent.dl.sourceforge.net/project/math-atlas/Developer%20%28unstable%29/3.11.11/atlas3.11.11.tar.bz2 % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 5115k 100 5115k0 0 1267k 0 0:00:04 0:00:04 --:--:-- 1272k $ tar -jxvf atlas3.11.11.tar.bz2 ... $ cd ATLAS/ $ ./configure ATLAS can no longer be configured in the exact source directory, create a subdir such as MyObj. See ATLAS/INSTALL.txt for help. $ mkdir my_build_dir ; cd my_build_dir $ ../configure gcc -I/tmp/x/ATLAS/my_build_dir/..//CONFIG/include -g -w -c /tmp/x/ATLAS/my_build_dir/..//CONFIG/src/atlconf_misc.c gcc -I/tmp/x/ATLAS/my_build_dir/..//CONFIG/include -g -w -o xconfig /tmp/x/ATLAS/my_build_dir/..//CONFIG/src/config.c atlconf_misc.o ./xconfig -d s /tmp/x/ATLAS/my_build_dir/../ -d b /tmp/x/ATLAS/my_build_dir OS configured as OSX (12) Assembly configured as GAS_x8664 (2) Vector ISA Extension configured as SSE3 (6,448) Architecture configured as Core2 (25) Clock rate configured as 2800Mhz Maximum number of threads configured as 8 Parallel make command configured as '$(MAKE) -j 8' Cannot detect CPU throttling. ... $ time make make -f Make.top build cd bin/ ; make xatlas_build /usr/bin/gcc -DL2SIZE=33554432 -I/tmp/x/ATLAS/my_build_dir/include -I/tmp/x/ATLAS/my_build_dir/..//include -I/tmp/x/ATLAS/my_build_dir/..//include/contrib -DAdd_ -DF77_INTEGER=int -DStringSunStyle -DATL_OS_OSX -DATL_ARCH_Core2 -DATL_CPUMHZ=2800 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 -m64 -DATL_NCPU=8 -O -fomit-frame-pointer -m64 -c /tmp/x/ATLAS/my_build_dir/..//bin/atlas_tee.c /usr/bin/gcc -DL2SIZE=33554432 -I/tmp/x/ATLAS/my_build_dir/include -I/tmp/x/ATLAS/my_build_dir/..//include -I/tmp/x/ATLAS/my_build_dir/..//include/contrib -DAdd_ -DF77_INTEGER=int -DStringSunStyle -DATL_OS_OSX -DATL_ARCH_Core2 -DATL_CPUMHZ=2800 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 -m64 -DATL_NCPU=8 -O -fomit-frame-pointer -m64 -o xatlas_tee atlas_tee.o ... cc1: error: unrecognized command line option "-mcpu=ultrasparc" FlagCheck.c:1: error: bad value (ultrasparc) for -mtune= switch cc1: error: unrecognized command line option "-maltivec" cc1: error: unrecognized command line option "-mabi=altivec" cc1: error: unrecognized command line option "-mcpu=ultrasparc" cc1: error: unrecognized command line option "-mcpu=ultrasparc" cc1: error: unrecognized command line option "-mcpu=ultrasparc" cc1: error: unrecognized command line option "-mcpu=970" cc1: error: unrecognized command line option "-mips4" cc1: error: unrecognized command line option "-mips4" cc1: error: unrecognized command line option "-mips4" cc1: error: unrecognized command line option "-mips4" cc1: error: unrecognized command line option "-mvsx" cc1: error: unrecognized command line option "-mfpu=vfpv3" ... ATLAS install complete. Examine ATLAS/bin//INSTALL_LOG/SUMMARY.LOG for details. make clean rm -rf *.dSYM rm -f *.o x* config?.out *core* real8m35.518s user7m33.738s sys2m17.387s $ time make test ... 8 cases: 8 passed, 0 skipped, 0 failed 4 cases: 4 passed, 0 skipped, 0 failed 8 cases: 8 passed, 0 skipped, 0 failed 4 cases: 4 passed, 0 skipped, 0 failed 8 cases: 8 passed, 0 skipped, 0 failed 4 cases: 4 passed, 0 skipped, 0 failed 8 cases: 8 passed, 0 skipped, 0 failed 4 cases: 4 passed, 0 skipped, 0 failed DONE SCOPING FOR FAILURES IN CBLAS TESTS: fgrep -e fault -e FAULT -e error -e ERROR -e fail -e FAIL \ interfaces/blas/C/testing/sanity.out | \ fgrep -v PASSED make[1]: [sanity_test] Error 1 (ignored) DONE SCOPING FOR FAILURES IN F77BLAS TESTS: fgrep -e fault -e FAULT -e error -e ERROR -e fail -e FAIL \ interfaces/blas/F77/testing/sanity.out | \ fgrep -v PASSED make[1]: [sanity_test] Error 1 (ignored) DONE real0m38.065s user0m27.856s sys0m6.329s __
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
tar xvfj atlas3.11.11.tar.bz2 shows no errors on OSX and creates one ATLAS folder. best, ido On Sep 20, 2013, at 11:17 AM, Bjoern Gruening wrote: > Hi Ido and Carlos, > > can you check if that tarball is working? > > http://downloads.sourceforge.net/project/math-atlas/Developer%20%28unstable%29/3.11.11/atlas3.11.11.tar.bz2 > > The chance is low, but if its working for you I will consider to create a new > version for it. > Thanks, > Bjoern > >> Yes this tar is broken at least on OSX. >> Other people have the same issue: >> >> http://code.google.com/p/libarchive/issues/detail?id=299 >> >> >> >> >> On Sep 20, 2013, at 10:41 AM, Bjoern Gruening < >> bjoern.gruen...@gmail.com >> > wrote: >> >> >> > Hi Carlos, >> > >> >>> Hi Peter and Carlos, >> >>> >> On Mon, Sep 16, 2013 at 8:57 PM, Carlos Borroto >> wrote: >> > I did an extra test. Started with a clean 'galaxy-dist'. This time >> > both repositories fail with the same error. I guess before something >> > was cached for the repository with version 0.1.4. >> > >> > I used biopython repository as a guide to write my tool dependency >> > definition: >> > http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61 >> > >> > I can confirm biopython repository is failing to install for me with >> > exactly the same error. I wonder if a recently addition in the test >> > toolshed broke the treatment of "prior_installation_required". >> > >> > Thanks, >> > Carlos >> >> Could be - note that currently Biopython isn't currently >> installing properly on the Test Tool Shed due to ALTAS >> failing (a requirement of NumPy which is a requirement >> of Biopython). Dave and Bjoern are I think looking at this >> already... >> >>> >> >>> I can't do much I tested it again and for me its working fine on my >> >>> computers I have at hand ... sorry. >> >>> >> >> >> >> In case it helps, this is how the INSTALLATION.log file ends on OS X >> >> 10.8.4: >> >> $ tail -n 3 >> >> ~/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/3508de0ebae1/INSTALLATION.log >> >> x ATLAS/tune/lapack/lanbsrch.c >> >> tar: Error exit delayed from previous errors. >> >> # >> >> >> >> This is the relevant part I can find in Galaxy's log: >> >> [localhost] local: tar xfvj atlas3.10.1.tar.bz2 >> >> >> >> Warning: local() encountered an error (return code 1) while executing >> >> 'tar xfvj atlas3.10.1.tar.bz2' >> >> >> >> After noticing this I got what I'm guessing is the original file from >> >> sourceforge: >> >> http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10.1.tar.bz2 >> >> >> >> I can confirm that trying to untar this file fails with the exact same >> >> error. However, on Ubuntu 13.04 untaring this file works just fine. >> > >> > That is new to me. How can that happen? Anyone with an OS-X can confirm >> > that? >> > >> >> On Ubuntu 13.04 the error I see is: >> >> STDERR >> >> It appears you have cpu throttling enabled, which makes timings >> >> unreliable and an ATLAS install nonsensical. Aborting. >> >> See ATLAS/INSTALL.txt for further information >> >> # >> >> >> >> Björn, you said cpu throttling can be easily disable on Ubuntu. Can >> >> you comment how? Do I need to disable it completely, or should the >> >> install script do it when installing? >> > >> > ATLAS (once untared ;-)) needs cpu-throttling to be disabled to >> > optimizes its library. If it is not disabled ATLAS compilation will >> > fail. For OS-X I found that: >> > >> > http://apple.stackexchange.com/questions/41045/how-can-i-disable-cpu-throttling-and-cpu-disabling >> > >> > Sorry, I never touched a OS-X. Nevertheless, If its not disabled, it is >> > supposed to fail silently and downstream packages will not be affected. >> > But if its crashing during untarring, I can't do much. What I can do is >> > to repack the tarball and host it elsewhere. What brings me to: >> > >> > http://gmod.827538.n3.nabble.com/RFC-Storing-of-tarballs-and-patches-for-tool-dependencies-to-enable-reproducibility-td4036591.html >> > >> > Bad news for a Friday morning, sorry :( >> > Bjoern >> > >> > >> >> Thanks, >> >> Carlos >> >> >> >> ___ >> >> Please keep all replies on the list by using "reply all" >> >> in your mail client. To manage your subscriptions to this >> >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ >> >> >> >> To search Galaxy mailing lists use the unified search at: >> >> http://galaxyproject.org/search/mailinglists/ >> > >> > >> > >> > ___ >> > Please keep all replies on the list by using "reply all" >> > in your mail client. To manage your subscriptions to this >> > and other Galaxy lists, please
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Thanks for posting that Ido, Right now I would suggest manually installing these dependencies, rather than ticking the box for the Tool Shed to do it for you. If you are using the Apple provided Python to run your Galaxy on Mac OS X, it comes with NumPy anyway, so compiling Biopython should be simple (just install XCode and the command line tools first). Note that I have not yet updated my Biopython using tools on the main tool shed to include the automatic dependency on the Tool Shed Biopython package, so they require a manual install for now. As soon as the NumPy/ATLAS packaging issue is solved, I plan to push those updates from the Test Tool Shed though. Peter On Fri, Sep 20, 2013 at 10:04 AM, Ido Tamir wrote: > Yes this tar is broken at least on OSX. > Other people have the same issue: > http://code.google.com/p/libarchive/issues/detail?id=299 > > > > On Sep 20, 2013, at 10:41 AM, Bjoern Gruening > wrote: > >> Hi Carlos, >> Hi Peter and Carlos, > On Mon, Sep 16, 2013 at 8:57 PM, Carlos Borroto > wrote: >> I did an extra test. Started with a clean 'galaxy-dist'. This time >> both repositories fail with the same error. I guess before something >> was cached for the repository with version 0.1.4. >> >> I used biopython repository as a guide to write my tool dependency >> definition: >> http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61 >> >> I can confirm biopython repository is failing to install for me with >> exactly the same error. I wonder if a recently addition in the test >> toolshed broke the treatment of "prior_installation_required". >> >> Thanks, >> Carlos > > Could be - note that currently Biopython isn't currently > installing properly on the Test Tool Shed due to ALTAS > failing (a requirement of NumPy which is a requirement > of Biopython). Dave and Bjoern are I think looking at this > already... I can't do much I tested it again and for me its working fine on my computers I have at hand ... sorry. >>> >>> In case it helps, this is how the INSTALLATION.log file ends on OS X 10.8.4: >>> $ tail -n 3 >>> ~/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/3508de0ebae1/INSTALLATION.log >>> x ATLAS/tune/lapack/lanbsrch.c >>> tar: Error exit delayed from previous errors. >>> # >>> >>> This is the relevant part I can find in Galaxy's log: >>> [localhost] local: tar xfvj atlas3.10.1.tar.bz2 >>> >>> Warning: local() encountered an error (return code 1) while executing >>> 'tar xfvj atlas3.10.1.tar.bz2' >>> >>> After noticing this I got what I'm guessing is the original file from >>> sourceforge: >>> http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10.1.tar.bz2 >>> >>> I can confirm that trying to untar this file fails with the exact same >>> error. However, on Ubuntu 13.04 untaring this file works just fine. >> >> That is new to me. How can that happen? Anyone with an OS-X can confirm >> that? >> >>> On Ubuntu 13.04 the error I see is: >>> STDERR >>> It appears you have cpu throttling enabled, which makes timings >>> unreliable and an ATLAS install nonsensical. Aborting. >>> See ATLAS/INSTALL.txt for further information >>> # >>> >>> Björn, you said cpu throttling can be easily disable on Ubuntu. Can >>> you comment how? Do I need to disable it completely, or should the >>> install script do it when installing? >> >> ATLAS (once untared ;-)) needs cpu-throttling to be disabled to >> optimizes its library. If it is not disabled ATLAS compilation will >> fail. For OS-X I found that: >> >> http://apple.stackexchange.com/questions/41045/how-can-i-disable-cpu-throttling-and-cpu-disabling >> >> Sorry, I never touched a OS-X. Nevertheless, If its not disabled, it is >> supposed to fail silently and downstream packages will not be affected. >> But if its crashing during untarring, I can't do much. What I can do is >> to repack the tarball and host it elsewhere. What brings me to: >> >> http://gmod.827538.n3.nabble.com/RFC-Storing-of-tarballs-and-patches-for-tool-dependencies-to-enable-reproducibility-td4036591.html >> >> Bad news for a Friday morning, sorry :( >> Bjoern >> >> >>> Thanks, >>> Carlos >>> >>> ___ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> http://lists.bx.psu.edu/ >>> >>> To search Galaxy mailing lists use the unified search at: >>> http://galaxyproject.org/search/mailinglists/ >> >> >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interfac
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Hi Ido and Carlos, can you check if that tarball is working? http://downloads.sourceforge.net/project/math-atlas/Developer%20% 28unstable%29/3.11.11/atlas3.11.11.tar.bz2 The chance is low, but if its working for you I will consider to create a new version for it. Thanks, Bjoern > Yes this tar is broken at least on OSX. > Other people have the same issue: > http://code.google.com/p/libarchive/issues/detail?id=299 > > > > On Sep 20, 2013, at 10:41 AM, Bjoern Gruening > wrote: > > > Hi Carlos, > > > >>> Hi Peter and Carlos, > >>> > On Mon, Sep 16, 2013 at 8:57 PM, Carlos Borroto > wrote: > > I did an extra test. Started with a clean 'galaxy-dist'. This time > > both repositories fail with the same error. I guess before something > > was cached for the repository with version 0.1.4. > > > > I used biopython repository as a guide to write my tool dependency > > definition: > > http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61 > > > > I can confirm biopython repository is failing to install for me with > > exactly the same error. I wonder if a recently addition in the test > > toolshed broke the treatment of "prior_installation_required". > > > > Thanks, > > Carlos > > Could be - note that currently Biopython isn't currently > installing properly on the Test Tool Shed due to ALTAS > failing (a requirement of NumPy which is a requirement > of Biopython). Dave and Bjoern are I think looking at this > already... > >>> > >>> I can't do much I tested it again and for me its working fine on my > >>> computers I have at hand ... sorry. > >>> > >> > >> In case it helps, this is how the INSTALLATION.log file ends on OS X > >> 10.8.4: > >> $ tail -n 3 > >> ~/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/3508de0ebae1/INSTALLATION.log > >> x ATLAS/tune/lapack/lanbsrch.c > >> tar: Error exit delayed from previous errors. > >> # > >> > >> This is the relevant part I can find in Galaxy's log: > >> [localhost] local: tar xfvj atlas3.10.1.tar.bz2 > >> > >> Warning: local() encountered an error (return code 1) while executing > >> 'tar xfvj atlas3.10.1.tar.bz2' > >> > >> After noticing this I got what I'm guessing is the original file from > >> sourceforge: > >> http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10.1.tar.bz2 > >> > >> I can confirm that trying to untar this file fails with the exact same > >> error. However, on Ubuntu 13.04 untaring this file works just fine. > > > > That is new to me. How can that happen? Anyone with an OS-X can confirm > > that? > > > >> On Ubuntu 13.04 the error I see is: > >> STDERR > >> It appears you have cpu throttling enabled, which makes timings > >> unreliable and an ATLAS install nonsensical. Aborting. > >> See ATLAS/INSTALL.txt for further information > >> # > >> > >> Björn, you said cpu throttling can be easily disable on Ubuntu. Can > >> you comment how? Do I need to disable it completely, or should the > >> install script do it when installing? > > > > ATLAS (once untared ;-)) needs cpu-throttling to be disabled to > > optimizes its library. If it is not disabled ATLAS compilation will > > fail. For OS-X I found that: > > > > http://apple.stackexchange.com/questions/41045/how-can-i-disable-cpu-throttling-and-cpu-disabling > > > > Sorry, I never touched a OS-X. Nevertheless, If its not disabled, it is > > supposed to fail silently and downstream packages will not be affected. > > But if its crashing during untarring, I can't do much. What I can do is > > to repack the tarball and host it elsewhere. What brings me to: > > > > http://gmod.827538.n3.nabble.com/RFC-Storing-of-tarballs-and-patches-for-tool-dependencies-to-enable-reproducibility-td4036591.html > > > > Bad news for a Friday morning, sorry :( > > Bjoern > > > > > >> Thanks, > >> Carlos > >> > >> ___ > >> Please keep all replies on the list by using "reply all" > >> in your mail client. To manage your subscriptions to this > >> and other Galaxy lists, please use the interface at: > >> http://lists.bx.psu.edu/ > >> > >> To search Galaxy mailing lists use the unified search at: > >> http://galaxyproject.org/search/mailinglists/ > > > > > > > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manag
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Yes this tar is broken at least on OSX. Other people have the same issue: http://code.google.com/p/libarchive/issues/detail?id=299 On Sep 20, 2013, at 10:41 AM, Bjoern Gruening wrote: > Hi Carlos, > >>> Hi Peter and Carlos, >>> On Mon, Sep 16, 2013 at 8:57 PM, Carlos Borroto wrote: > I did an extra test. Started with a clean 'galaxy-dist'. This time > both repositories fail with the same error. I guess before something > was cached for the repository with version 0.1.4. > > I used biopython repository as a guide to write my tool dependency > definition: > http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61 > > I can confirm biopython repository is failing to install for me with > exactly the same error. I wonder if a recently addition in the test > toolshed broke the treatment of "prior_installation_required". > > Thanks, > Carlos Could be - note that currently Biopython isn't currently installing properly on the Test Tool Shed due to ALTAS failing (a requirement of NumPy which is a requirement of Biopython). Dave and Bjoern are I think looking at this already... >>> >>> I can't do much I tested it again and for me its working fine on my >>> computers I have at hand ... sorry. >>> >> >> In case it helps, this is how the INSTALLATION.log file ends on OS X 10.8.4: >> $ tail -n 3 >> ~/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/3508de0ebae1/INSTALLATION.log >> x ATLAS/tune/lapack/lanbsrch.c >> tar: Error exit delayed from previous errors. >> # >> >> This is the relevant part I can find in Galaxy's log: >> [localhost] local: tar xfvj atlas3.10.1.tar.bz2 >> >> Warning: local() encountered an error (return code 1) while executing >> 'tar xfvj atlas3.10.1.tar.bz2' >> >> After noticing this I got what I'm guessing is the original file from >> sourceforge: >> http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10.1.tar.bz2 >> >> I can confirm that trying to untar this file fails with the exact same >> error. However, on Ubuntu 13.04 untaring this file works just fine. > > That is new to me. How can that happen? Anyone with an OS-X can confirm > that? > >> On Ubuntu 13.04 the error I see is: >> STDERR >> It appears you have cpu throttling enabled, which makes timings >> unreliable and an ATLAS install nonsensical. Aborting. >> See ATLAS/INSTALL.txt for further information >> # >> >> Björn, you said cpu throttling can be easily disable on Ubuntu. Can >> you comment how? Do I need to disable it completely, or should the >> install script do it when installing? > > ATLAS (once untared ;-)) needs cpu-throttling to be disabled to > optimizes its library. If it is not disabled ATLAS compilation will > fail. For OS-X I found that: > > http://apple.stackexchange.com/questions/41045/how-can-i-disable-cpu-throttling-and-cpu-disabling > > Sorry, I never touched a OS-X. Nevertheless, If its not disabled, it is > supposed to fail silently and downstream packages will not be affected. > But if its crashing during untarring, I can't do much. What I can do is > to repack the tarball and host it elsewhere. What brings me to: > > http://gmod.827538.n3.nabble.com/RFC-Storing-of-tarballs-and-patches-for-tool-dependencies-to-enable-reproducibility-td4036591.html > > Bad news for a Friday morning, sorry :( > Bjoern > > >> Thanks, >> Carlos >> >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> http://lists.bx.psu.edu/ >> >> To search Galaxy mailing lists use the unified search at: >> http://galaxyproject.org/search/mailinglists/ > > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Hi Carlos, > > Hi Peter and Carlos, > > > >> On Mon, Sep 16, 2013 at 8:57 PM, Carlos Borroto > >> wrote: > >> > I did an extra test. Started with a clean 'galaxy-dist'. This time > >> > both repositories fail with the same error. I guess before something > >> > was cached for the repository with version 0.1.4. > >> > > >> > I used biopython repository as a guide to write my tool dependency > >> > definition: > >> > http://testtoolshed.g2.bx.psu.edu/view/biopython/package_biopython_1_61 > >> > > >> > I can confirm biopython repository is failing to install for me with > >> > exactly the same error. I wonder if a recently addition in the test > >> > toolshed broke the treatment of "prior_installation_required". > >> > > >> > Thanks, > >> > Carlos > >> > >> Could be - note that currently Biopython isn't currently > >> installing properly on the Test Tool Shed due to ALTAS > >> failing (a requirement of NumPy which is a requirement > >> of Biopython). Dave and Bjoern are I think looking at this > >> already... > > > > I can't do much I tested it again and for me its working fine on my > > computers I have at hand ... sorry. > > > > In case it helps, this is how the INSTALLATION.log file ends on OS X 10.8.4: > $ tail -n 3 > ~/src/shed_tools_dependencies.central/atlas/3.10.1/iuc/package_atlas_3_10/3508de0ebae1/INSTALLATION.log > x ATLAS/tune/lapack/lanbsrch.c > tar: Error exit delayed from previous errors. > # > > This is the relevant part I can find in Galaxy's log: > [localhost] local: tar xfvj atlas3.10.1.tar.bz2 > > Warning: local() encountered an error (return code 1) while executing > 'tar xfvj atlas3.10.1.tar.bz2' > > After noticing this I got what I'm guessing is the original file from > sourceforge: > http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10.1.tar.bz2 > > I can confirm that trying to untar this file fails with the exact same > error. However, on Ubuntu 13.04 untaring this file works just fine. That is new to me. How can that happen? Anyone with an OS-X can confirm that? > On Ubuntu 13.04 the error I see is: > STDERR > It appears you have cpu throttling enabled, which makes timings > unreliable and an ATLAS install nonsensical. Aborting. > See ATLAS/INSTALL.txt for further information > # > > Björn, you said cpu throttling can be easily disable on Ubuntu. Can > you comment how? Do I need to disable it completely, or should the > install script do it when installing? ATLAS (once untared ;-)) needs cpu-throttling to be disabled to optimizes its library. If it is not disabled ATLAS compilation will fail. For OS-X I found that: http://apple.stackexchange.com/questions/41045/how-can-i-disable-cpu-throttling-and-cpu-disabling Sorry, I never touched a OS-X. Nevertheless, If its not disabled, it is supposed to fail silently and downstream packages will not be affected. But if its crashing during untarring, I can't do much. What I can do is to repack the tarball and host it elsewhere. What brings me to: http://gmod.827538.n3.nabble.com/RFC-Storing-of-tarballs-and-patches-for-tool-dependencies-to-enable-reproducibility-td4036591.html Bad news for a Friday morning, sorry :( Bjoern > Thanks, > Carlos > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
