Re: [galaxy-dev] Missing test results on (Test) Tool Shed
Hello again, I've not have any missing test results for a while, on the main or test Tool Shed, so this was a bit of a surprise: http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr/09a68a90d552 This is listed under Latest revision: failing tool tests, but I see no test results at all :( Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Fri, Sep 20, 2013 at 11:10 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Wed, Sep 18, 2013 at 11:47 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, Aug 27, 2013 at 2:18 PM, Dave Bouvier d...@bx.psu.edu wrote: Peter, I also tried running the command that returns error code 64 on the same system that runs the automated tests, and it downloaded the correct file for that operating system and architecture. So I'm not sure why it's failing when run through buildbot, but I'll look into it and get back to you. I've added a trello card to track progress on this issue: https://trello.com/c/9ERDMc7j/1079-toolshed-investigate-cause-of-shell-commands-failing-through-buildbot-when-they-are-valid-commands --Dave B. Hi Dave, Something seems to have changed again on the Test Tool Shed, http://testtoolshed.g2.bx.psu.edu/view/peterjc/ncbi_blast_plus/4ae1bac976f3 Installation errors - no functional tests were run for any tools in this changeset revision Tool dependencies TypeNameVersion blast+ package 2.2.26+ Error /bin/sh: 2: [[: not found /bin/sh: 3: [[: not found /bin/sh: 4: [[: not found /bin/sh: 6: [[: not found /bin/sh: 7: [[: not found tar: option requires an argument -- 'f' Try `tar --help' or `tar --usage' for more information. Is the test cluster really missing /bin/sh? Can you check that and post the full installation log please? I really would like to push an update to ncbi_blast_plus to the main Tool Shed with an updated README file (using RST) and the new citation information - plus of course handling the binary dependency via the new package: http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_26 Once this is done there is a backlog of updates I want to tackle following our productive BoF meeting at GCC2013: http://wiki.galaxyproject.org/Events/GCC2013/BoF/GalaxyBlast Thanks, Peter A related bug for Greg (I think), despite this strange no /bin/sh error, this repository is not listed on the Test Tool Shed under Latest revision: installation errors. Regards, Peter Aside from the oddities reported via the Test Tool Shed testing, feedback for the package_blast_plus_2_2_26 dependency has been positive, so I have just updated the BLAST+ wrappers on the main Tool Shed to use it (v0.0.20): http://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/70e7dcbf6573 Fingers crossed this works 'in the wild', and that the curious missing /bin/sh issue on the test machines can be solved. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How to make a Tool Shed private?
It's a bit unclear what you mean by private here, but it's possible to at least force users to login to use a Tool Shed if you include the following setting in the [app:main] section your tool_shed_wsgi.ini file. # Force everyone to log in (disable anonymous access) require_login = True Greg Von Kuster On Sep 22, 2013, at 2:39 PM, Serrano Pereira serrano.pere...@gmail.com wrote: Hello, Is it possible to make a Galaxy Tool Shed private? The only method I can think of at this moment is to use Apache's configuration to restrict access to certain IP addresses. I was wondering if there are alternative methods. Regards, Serrano ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Test Toolshed Biopython package dependency Atlas fails to install (Was: Re: UnboundLocalError: local variable 'prior_installation_required' referenced before assignment)
Forgot to copy the list on this email. On Mon, Sep 23, 2013 at 9:32 AM, Carlos Borroto carlos.borr...@gmail.com wrote: On Fri, Sep 20, 2013 at 6:12 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Carlos, Can you try again? Also the new unstable version if you can. Thanks for the help! ATLAS is a beast :( Sorry for the delayed response. Busy weekend. Trying to install package_atlas_3_10 on Ubuntu 13.04: STDERR It appears you have cpu throttling enabled, which makes timings unreliable and an ATLAS install nonsensical. Aborting. See ATLAS/INSTALL.txt for further information # I think this is the other relevant part in the log: # try to disable cpu throttling if hash cpufreq-selector 2/dev/null; then cpufreq-selector -g performance elif hash cpupower 2/dev/null; then cpupower frequency-set -g performance else echo 'Please deactivate CPU throttling by your own, or install cpufreq-selector' exit fi STDERR Error calling SetGovernor: Caller is not authorized I had to install 'cpufreq-selector' to get here, not installed by default. I can confirm I get the same error when trying to run this command directly: $ cpufreq-selector -g performance Error calling SetGovernor: Caller is not authorized package_atlas_3_11 fails in exactly the same way in this box. Somehow this is also a silent failing and numpy, biopython and ngs-tools(my tool package depending on biopython) get Installed(Green) Yes that is correct. I designed it (in theory) in that way, that if ATLAS crashes (due to CPU throttling enabled) every other package can be installed without problem. Every other behaviour is a bug. while lapack, atlas and split_by_barcode(my actual tool wrapper depending on ngs-tools package) get Installed, missing tool dependencies(Grey). This means if I try to use my wrapper in this state I get this error: /bin/sh: 1: ngs-tools: not found On Ubuntu it gets installed without errors. Everything is green :) However, if I do a Repair repository on split_by_barcode it goes into Installed(Green) and everything seems to work from then on. Mh, thats seems to be a bug. I will try to reproduce on a computer where ATLAS is crashing. Thanks for testing this. I also believe this might be a bug. --Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Grant authority missing when create new repository on Tool Shed
Hi Peter, This behavior has been corrected in 10637:216c01ce6625. Thanks for reporting this. Greg Von Kuster On Sep 23, 2013, at 7:09 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, I think I've found a bug, or at least an area for improvement, 1. Goto http://testtoolshed.g2.bx.psu.edu/ 2. Login, e.g. as IUC 3. Create new repository, e.g. package_blast_plus_2_2_27 4. Fill in description and 5. Look for Grant authority to make changes Actual result: No Grant authority to make changes box present. Expected result: Grant authority to make changes box present. Workaround: 6. Click on My writable repositories 7. Click on new repository 8. Now there is a Grant authority to make changes box. Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Regarding Jellyfish or other k-mer analysis software
Hello! I hope that everybody has a great day! In a pre-assemble analysis step I want to perform a k-mer analysis of the fastq-file to visualize the k-mer coverage distribution (too estimate genome size, heterozygosity, repeats etc). This would preferably be done with Jellyfish. Is there an existing wrapper for jellyfish or any similar k-mer analysis software? I have been searching the toolshed and with Google without any success. I am farly new to Galaxy so any help/input would really be appreciated! Thanks, Johan Schedin ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed: get_readme_files bug
Greg, That's part of the issue. Please try the following: 1) Direct your browser to http://toolshed.g2.bx.psu.edu/view/miller- lab/genome_diversity 2) Choose 33:5064f618ec1c under Repository revision 3) View README under Repository README files - may contain important installation or license information README The Genome Diversity tools require the following software: ADMIXTURE (we used version 1.22) http://www.genetics.ucla.edu/ software/admixture/ KING (we used version 1.5) http://people.virginia.edu/ wc9c/KING/ 4) Choose 30:4188853b940b under Repository revision 5) View README under Repository README files - may contain important installation or license information README The Genome Diversity tools require the following software: ADMIXTURE (we used version 1.22) http://www.genetics.ucla.edu/ software/admixture/ KING (we used version 1.5) http://people.virginia.edu/ wc9c/KING/ The README in step #5 above is not the README for changeset revision 4188853b940b. It is showing the README for changeset revision 5064f618ec1c instead. This is because revision 5064f618ec1c is currently found on disk in .../galaxy_toolshed/database/ community_files/000/repo_200. I would expect that when viewing a specific changeset revision of a repository in the tool shed, I would see the README files for that changeset revision. The relevant code can be found in build_readme_files_dict() in lib/tool_shed/util/readme_util.py. It reads the files in the mercurial working directory without ensuring that the working directory is at the requested changeset revision. -rico On Sep 22, 2013, at 7:26 AM, Greg Von Kuster wrote: Hi Rico, It looks like the files on disk simply did not have the hg update applied when the last changeset was committed. g2cmnty@montana:~/galaxy_toolshed/database/community_files/000/ repo_200$ hg update 46 files updated, 0 files merged, 26 files removed, 0 files unresolved g2cmnty@montana:~/galaxy_toolshed/database/community_files/000/ repo_200$ cat README The Genome Diversity tools require the following software: ADMIXTURE (we used version 1.22) http://www.genetics.ucla.edu/ software/admixture/ KING (we used version 1.5) http://people.virginia.edu/ ~wc9c/KING/ g2cmnty@montana:~/galaxy_toolshed/database/community_files/000/ repo_200$ Since the issue was not related to the Tool Shed's README utility, is was easy to miss it. Is the content of the README file now what you expect? Greg Von Kuster On Sep 21, 2013, at 6:13 PM, Richard Burhans r...@bx.psu.edu wrote: Greg, On Sep 21, 2013, at 6:03 PM, Greg Von Kuster wrote: Hi Rico, On Sep 21, 2013, at 5:56 PM, Richard Burhans r...@bx.psu.edu wrote: Greg, To be more clear, montana has revision 4188853b940b on disk, which is not the tip revision, 5064f618ec1c. Unless I am missing something, montana has revisions 5064f618ec1c on disk: g2cmnty@montana:~/galaxy_toolshed/database/community_files/000/ repo_200$ hg heads changeset: 33:5064f618ec1c tag: tip user:Richard Burhans burh...@bx.psu.edu date:Fri Sep 20 14:01:30 2013 -0400 summary: remove munkres dependency g2cmnty@montana:~/galaxy_toolshed/database/community_files/000/ repo_200$ cat README Source code for the executables needed by these tools can be found in the genome_diversity directory. Additionally, you'll need the following python modules: matplotlib (we used version 1.1.0) http://pypi.python.org/ packages/source/m/matplotlib/ mechanize (we used version 0.2.5) http://pypi.python.org/ packages/source/m/mechanize/ networkx (we used version 1.6) http://pypi.python.org/ packages/source/n/networkx/ fisher (we used version 0.1.4) http://pypi.python.org/ packages/source/f/fisher/ And the following software: ADMIXTURE (we used version 1.22) http://www.genetics.ucla.edu/ software/admixture/ EIGENSOFT (we used version 3.0) http:// genepath.med.harvard.edu/~reich/Software.htm PHAST (we used version 1.2.1) http:// compgen.bscb.cornell.edu/phast/ QuickTree (we used version 1.1) http://www.sanger.ac.uk/ resources/software/quicktree/ Images used in the tools' documentation are located in the static/ images directory. Please copy these to the static/images directory in your Galaxy installation. I believe that you are missing something. Please try the following experiment: $ hg clone http://toolshed.g2.bx.psu.edu/repos/miller-lab/ genome_diversity $ cd genome_diversity $ hg update tip $ cat README The Genome Diversity tools require the following software: ADMIXTURE (we used version 1.22) http://www.genetics.ucla.edu/ software/admixture/ KING (we used version 1.5) http://people.virginia.edu/ ~wc9c/KING/ $ hg update 4188853b940b $ cat README Source code for the executables needed by these tools can be found in the genome_diversity directory. Additionally,
Re: [galaxy-dev] Repository tool dependencies and virtualenv install
On Mon, Sep 23, 2013 at 1:57 PM, John Chilton chil...@msi.umn.edu wrote: On Mon, Sep 23, 2013 at 12:51 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Carlos! Hi, I have created a tool that depends on biopython. I don't like the idea of having several copies of biopython around and also for best practice making workflows using my tool reproducible, I chose to make my package repository[1] dependent of package_biopython_1_62. [1]http://testtoolshed.g2.bx.psu.edu/view/cjav/package_ngs_tools_0_1_6 Cool thanks! I think that is best practise! I disagree strongly. This is not a good idea. My thought is if you are using a virtualenv, you shouldn't be trying to compose it with other python dependencies - to me the whole point of virtualenv is that it creates isolated little environments. I am not sure why duplicating the biopython install reduces reproduciblity. If you want to use individual Python packages using Galaxy's dependency mechanism, I think you should then package them up one at a time and hand modify PYTHONPATH and PATH - the way biopython is done. Galaxy should have a separate action to make this easy, say install_pip or something like that, as outlined by James Taylor at some point. Hi John, First let me tell you why I think duplicating the biopython install reduces reproducibility. I have this on my tool setup.py: install_requires=[ docopt, biopython, python-levenshtein ], While I could specify versions here(ex. biopython==1.62), I feel that is not a good thing outside of Galaxy. I think pip should be free to install the latest version of these packages until I found there is an issue otherwise. I think this is the most common approach, I might be wrong thou. This leaves me with the issue that then when Galaxy installs my tool using virtualenv, it will grab the most up-to-date version of these packages, hence reducing reproducibility. Did I explain myself well enough? I'll be happy to debate about any of this. I agree with you about breaking the original intent of virtualenv, but as you mention without 'install_pip' or similar, I'm left without the option of making my life easy by just installing my package with an approach using pypi. Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Repository tool dependencies and virtualenv install
On Mon, Sep 23, 2013 at 1:09 PM, Carlos Borroto carlos.borr...@gmail.com wrote: On Mon, Sep 23, 2013 at 1:57 PM, John Chilton chil...@msi.umn.edu wrote: On Mon, Sep 23, 2013 at 12:51 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Carlos! Hi, I have created a tool that depends on biopython. I don't like the idea of having several copies of biopython around and also for best practice making workflows using my tool reproducible, I chose to make my package repository[1] dependent of package_biopython_1_62. [1]http://testtoolshed.g2.bx.psu.edu/view/cjav/package_ngs_tools_0_1_6 Cool thanks! I think that is best practise! I disagree strongly. This is not a good idea. My thought is if you are using a virtualenv, you shouldn't be trying to compose it with other python dependencies - to me the whole point of virtualenv is that it creates isolated little environments. I am not sure why duplicating the biopython install reduces reproduciblity. If you want to use individual Python packages using Galaxy's dependency mechanism, I think you should then package them up one at a time and hand modify PYTHONPATH and PATH - the way biopython is done. Galaxy should have a separate action to make this easy, say install_pip or something like that, as outlined by James Taylor at some point. Hi John, First let me tell you why I think duplicating the biopython install reduces reproducibility. I have this on my tool setup.py: install_requires=[ docopt, biopython, python-levenshtein ], While I could specify versions here(ex. biopython==1.62), I feel that is not a good thing outside of Galaxy. I think pip should be free to install the latest version of these packages until I found there is an issue otherwise. I think this is the most common approach, I might be wrong thou. This leaves me with the issue that then when Galaxy installs my tool using virtualenv, it will grab the most up-to-date version of these packages, hence reducing reproducibility. Did I explain myself well enough? I'll be happy to debate about any of this. I understand what you are saying and I sympathize with you here. Still I think the better approach is going to be to copy these requirements into the setup_virtualenv block and specify hard-coded versions. This way you get reproduciblity across all packages, not just biopython. I think this slight duplication is a smaller problem then mixing dependency mechanisms you described in your approach. To me it is analogous to installing some python dependencies via os packages and other ones via sudo pip install into /usr, it is a recipe for confusion. I agree with you about breaking the original intent of virtualenv, but as you mention without 'install_pip' or similar, I'm left without the option of making my life easy by just installing my package with an approach using pypi. Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Repository tool dependencies and virtualenv install
On Mon, Sep 23, 2013 at 1:51 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: I'm installing my package using the virtualenv option. As you can see below I marked biopython as prior_installation_required=True. However, I can see that virtualenv is installing a copy of biopython in 'venv' under my tool install(see below). This doesn't sound right. I was expecting that by making my repository dependent of biopython's one that source would be used in my tool. Can you try to insert that (maybe adopt, its not checked): action type=set_environment_for_install repository name=package_biopython_1_62 owner=biopython package name=biopython version=1.62 / /repository /action Only with that the PYTHONPATH is populated. Hi Björn, No luck with this recommendation. See my full tool_dependencies.xml below in case I miss read you. Biopython still gets installed into this repository install 'venv'. I will try to move to what biopython is doing. Hopefully and probably better, as John mentioned something like 'install_pip' will come around in the future with support for automatically modifying PYTHONPATH accordingly based on the repository dependencies. ?xml version=1.0? tool_dependency package name=biopython version=1.62 repository changeset_revision=ac9cc2992b69 name=package_biopython_1_62 owner=biopython prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / /package package name=ngs-tools version=0.1.6 install version=1.0 action type=set_environment_for_install repository name=package_biopython_1_62 owner=biopython package name=biopython version=1.62 / /repository /action actions action type=setup_virtualenvngs-tools==0.1.6/action /actions /install /package /tool_dependency Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Repository tool dependencies and virtualenv install
On Mon, Sep 23, 2013 at 1:25 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Am Montag, den 23.09.2013, 12:57 -0500 schrieb John Chilton: On Mon, Sep 23, 2013 at 12:51 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Carlos! Hi, I have created a tool that depends on biopython. I don't like the idea of having several copies of biopython around and also for best practice making workflows using my tool reproducible, I chose to make my package repository[1] dependent of package_biopython_1_62. [1]http://testtoolshed.g2.bx.psu.edu/view/cjav/package_ngs_tools_0_1_6 Cool thanks! I think that is best practise! I disagree strongly. This is not a good idea. My thought is if you are using a virtualenv, you shouldn't be trying to compose it with other python dependencies - to me the whole point of virtualenv is that it creates isolated little environments. I am not sure why duplicating the biopython install reduces reproduciblity. If you want to use individual Python packages using Galaxy's dependency mechanism, I think you should then package them up one at a time and hand modify PYTHONPATH and PATH - the way biopython is done. Galaxy should have a separate action to make this easy, say install_pip or something like that, as outlined by James Taylor at some point. Hi John, interesting point. What is the specific difference between venv and the pip installation? In which case do you propose to use which method? I never thought about different 'isolation-levels' (venv vs. the rest of the toolshed). We need to document somehow the use cases if we have different methods for python installations. I would use the venv method for everything (unless there is some reason virtualenv cannot be used a certain case, I would not be surprised if I was over estimating its power) and it wouldn't be composable - let the broader Python community worry about Python dependencies, how to install them, how to package them, etc Galaxy should just be concerned with figuring out how to enable the environment before running a job. Let me know if I am being a jackass who is oversimplifying this and there is some reason it cannot be done. The use case for not using venv is not necessarily something I understand, but its clear you and Carlos and others want to have individual Python dependencies packaged in the tool shed the way Debian or CentOS do it. You will have to explain that use case to me :). While I don't like this approach, if it is something you guys want to do, you should have the tools to make it as easy as possible to do this. If that is something like setup_pip (i.e. wrapper for pip install) or setup_python (wrapper for python setup.py --prefix=$install_dir install), cool beans these should be pretty straight forward to implement and should save on some typing. I still think of them as anti-patterns though :). -John Ciao, Bjoern -John I'm installing my package using the virtualenv option. As you can see below I marked biopython as prior_installation_required=True. However, I can see that virtualenv is installing a copy of biopython in 'venv' under my tool install(see below). This doesn't sound right. I was expecting that by making my repository dependent of biopython's one that source would be used in my tool. Can you try to insert that (maybe adopt, its not checked): action type=set_environment_for_install repository name=package_biopython_1_62 owner=biopython package name=biopython version=1.62 / /repository /action Only with that the PYTHONPATH is populated. My plan is to create any non-existing Tool dependency definition repository for every package my tool requires. This way you can always count the same version of any of these packages is being used. I thought that was the point, right? Yes! Full ool_dependencies.xml: ?xml version=1.0? tool_dependency package name=biopython version=1.62 repository changeset_revision=ac9cc2992b69 name=package_biopython_1_62 owner=biopython prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / /package package name=ngs-tools version=0.1.6 install version=1.0 actions action type=setup_virtualenvngs-tools==0.1.6/action /actions /install /package /tool_dependency Contents of site-packages in my tool install dir: $ ls -1 shed_tools_dependencies.central/ngs-tools/0.1.6/cjav/package_ngs_tools_0_1_6/3c646b8328bb/venv/lib/python2.7/site-packages/ Bio biopython-1.62-py2.7.egg-info BioSQL docopt-0.6.1-py2.7.egg-info docopt.py docopt.pyc easy-install.pth Levenshtein.so ngs ngs_tools-0.1.6-py2.7.egg-info pip-1.3.1-py2.7.egg python_Levenshtein-0.10.2-py2.7.egg-info setuptools-0.6c11-py2.7.egg setuptools.pth Thanks, Carlos
Re: [galaxy-dev] Repository tool dependencies and virtualenv install
On Mon, Sep 23, 2013 at 2:30 PM, Carlos Borroto carlos.borr...@gmail.com wrote: On Mon, Sep 23, 2013 at 2:22 PM, John Chilton chil...@msi.umn.edu wrote: Hi John, First let me tell you why I think duplicating the biopython install reduces reproducibility. I have this on my tool setup.py: install_requires=[ docopt, biopython, python-levenshtein ], While I could specify versions here(ex. biopython==1.62), I feel that is not a good thing outside of Galaxy. I think pip should be free to install the latest version of these packages until I found there is an issue otherwise. I think this is the most common approach, I might be wrong thou. This leaves me with the issue that then when Galaxy installs my tool using virtualenv, it will grab the most up-to-date version of these packages, hence reducing reproducibility. Did I explain myself well enough? I'll be happy to debate about any of this. I understand what you are saying and I sympathize with you here. Still I think the better approach is going to be to copy these requirements into the setup_virtualenv block and specify hard-coded versions. This way you get reproduciblity across all packages, not just biopython. Hi John, Could you go a little further with this recommendation. How can I specify versions for required packages in setup_virtualenv. I now have this: install version=1.0 actions action type=setup_virtualenvngs-tools==0.1.6/action /actions /install I tried these two without luck: action type=setup_virtualenvdocopt==0.6.1 python-levenshtein==0.10.2 biopython==1.62 ngs-tools==0.1.6/action So the contents is treated like a requirements.txt file. So the whitespace becomes important (I have a plan to improve this and sort of synchronize the syntax used for Ruby, Python, and R, but for now its just a file). So you want this: action type=setup_virtualenvdocopt==0.6.1 python-levenshtein==0.10.2 biopython==1.62 ngs-tools==0.1.6 /action Newline between dependencies, and no whitespace to the left of each package. Someday the syntax will be: action type=setup_virtualenv packagedocopt==0.6.1/package packagepython-levenshtein==0.10.2/package biopython==1.62 ngs-tools==0.1.6 /action action type=setup_virtualenvdocopt==0.6.1, python-levenshtein==0.10.2, biopython==1.62, ngs-tools==0.1.6/action I think this slight duplication is a smaller problem then mixing dependency mechanisms you described in your approach. To me it is analogous to installing some python dependencies via os packages and other ones via sudo pip install into /usr, it is a recipe for confusion. While I'm getting convinced that maybe some duplication is not that bad after all, please notice that my plan is to install everything from the toolshed. I also don't like mixing install methods. In fact, I would like for 'install_pip' to have the best practice option of doing always 'pip install --no-deps'. This would force you to first upload everything your package needs to the toolshed. Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Repository tool dependencies and virtualenv install
Opps, sent that last mail out before I meant to. On Mon, Sep 23, 2013 at 2:30 PM, Carlos Borroto carlos.borr...@gmail.com wrote: On Mon, Sep 23, 2013 at 2:22 PM, John Chilton chil...@msi.umn.edu wrote: Hi John, First let me tell you why I think duplicating the biopython install reduces reproducibility. I have this on my tool setup.py: install_requires=[ docopt, biopython, python-levenshtein ], While I could specify versions here(ex. biopython==1.62), I feel that is not a good thing outside of Galaxy. I think pip should be free to install the latest version of these packages until I found there is an issue otherwise. I think this is the most common approach, I might be wrong thou. This leaves me with the issue that then when Galaxy installs my tool using virtualenv, it will grab the most up-to-date version of these packages, hence reducing reproducibility. Did I explain myself well enough? I'll be happy to debate about any of this. I understand what you are saying and I sympathize with you here. Still I think the better approach is going to be to copy these requirements into the setup_virtualenv block and specify hard-coded versions. This way you get reproduciblity across all packages, not just biopython. Hi John, Could you go a little further with this recommendation. How can I specify versions for required packages in setup_virtualenv. I now have this: install version=1.0 actions action type=setup_virtualenvngs-tools==0.1.6/action /actions /install I tried these two without luck: action type=setup_virtualenvdocopt==0.6.1 python-levenshtein==0.10.2 biopython==1.62 ngs-tools==0.1.6/action action type=setup_virtualenvdocopt==0.6.1, python-levenshtein==0.10.2, biopython==1.62, ngs-tools==0.1.6/action So the contents or text of the action is treated like a requirements.txt file. So the whitespace becomes important (I have a plan to improve this and sort of synchronize the syntax used for Ruby, Python, and R, but for now its just a file). So you want this: action type=setup_virtualenvdocopt==0.6.1 python-levenshtein==0.10.2 biopython==1.62 ngs-tools==0.1.6 /action Newline between dependencies, and no whitespace to the left of each package. Someday I would hope the syntax will be (with the older version supported as well for backward compatibility sake) and to support an even simpler YAML based version: action type=setup_virtualenv packagedocopt==0.6.1/package packagepython-levenshtein==0.10.2/package packagengs-tools==0.1.6/package /action Hope this helps. -John I think this slight duplication is a smaller problem then mixing dependency mechanisms you described in your approach. To me it is analogous to installing some python dependencies via os packages and other ones via sudo pip install into /usr, it is a recipe for confusion. While I'm getting convinced that maybe some duplication is not that bad after all, please notice that my plan is to install everything from the toolshed. I also don't like mixing install methods. In fact, I would like for 'install_pip' to have the best practice option of doing always 'pip install --no-deps'. This would force you to first upload everything your package needs to the toolshed. Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Repository tool dependencies and virtualenv install
On Mon, Sep 23, 2013 at 2:24 PM, Carlos Borroto carlos.borr...@gmail.com wrote: Can you try to insert that (maybe adopt, its not checked): action type=set_environment_for_install repository name=package_biopython_1_62 owner=biopython package name=biopython version=1.62 / /repository /action Only with that the PYTHONPATH is populated. Hi Björn, No luck with this recommendation. See my full tool_dependencies.xml below in case I miss read you. Biopython still gets installed into this repository install 'venv'. I will try to move to what biopython is doing. Hopefully and probably better, as John mentioned something like 'install_pip' will come around in the future with support for automatically modifying PYTHONPATH accordingly based on the repository dependencies. ?xml version=1.0? tool_dependency package name=biopython version=1.62 repository changeset_revision=ac9cc2992b69 name=package_biopython_1_62 owner=biopython prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / /package package name=ngs-tools version=0.1.6 install version=1.0 action type=set_environment_for_install repository name=package_biopython_1_62 owner=biopython package name=biopython version=1.62 / /repository /action actions action type=setup_virtualenvngs-tools==0.1.6/action /actions /install /package /tool_dependency I made a mistake here. After fixing it still didn't work. I placed action set_environment_for_install outside of actions. It should be: ?xml version=1.0? tool_dependency package name=biopython version=1.62 repository changeset_revision=ac9cc2992b69 name=package_biopython_1_62 owner=biopython prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / /package package name=ngs-tools version=0.1.6 install version=1.0 actions action type=set_environment_for_install repository name=package_biopython_1_62 owner=biopython package name=biopython version=1.62 / /repository /action action type=setup_virtualenvngs-tools==0.1.6/action /actions /install /package /tool_dependency Testing now John's recommendations of specifying versions in setup_virtualenv, which means I won't be using package_biopython_1_62 and its dependencies. It would be nice to see how package_biopython_* could be used in my case. It would also be nice to know if the consensus is to keep everything(dependencies included) inside the toolshed or not. Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Want faster dataset import
I want to frequently import many tens of thousands of datasets. The files are on the same sever as Galaxy. But the upload based mechanism is really really slow. It takes hours to load this many files, yet the data is not moving at all! What is the best strategy to go about making a faster bulk import? I can imagine a tight loop that is My datatypes are limited. foreach folder new LibraryFolder foreach file in each directory new LibraryDataset new LibraryDatasetDatasetAssociation flush once at the end. Thoughts? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How to make a Tool Shed private?
My bad, I should have been clearer. What I mean is that no one should be able to access it unless I give explicit permission. Browsing or installing tools shouldn't be possible either without permission. Serrano On 09/23/2013 02:54 PM, Greg Von Kuster wrote: It's a bit unclear what you mean by private here, but it's possible to at least force users to login to use a Tool Shed if you include the following setting in the [app:main] section your tool_shed_wsgi.ini file. # Force everyone to log in (disable anonymous access) require_login = True Greg Von Kuster On Sep 22, 2013, at 2:39 PM, Serrano Pereira serrano.pere...@gmail.com mailto:serrano.pere...@gmail.com wrote: Hello, Is it possible to make a Galaxy Tool Shed private? The only method I can think of at this moment is to use Apache's configuration to restrict access to certain IP addresses. I was wondering if there are alternative methods. Regards, Serrano ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] History will not load
Hello, I have been having trouble with Galaxy since yesterday. I have been trying to run cuffdiff which has been waiting to run since Friday afternoon and now my history will not even load (red error message shows up). Any incite would be much appreciated. Maria ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] (no subject)
Hello, Galaxy will not load my histories and as a result I can not run the analysis that I need to run. I have tried repeatedly since 7:00 AM EST to load them with no luck. I worked on this history yesterday and everything was fine. Maria ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Missing test results on (Test) Tool Shed
Peter, Thank you for reporting this. I've been able to determine that there was an installation error that should have been recorded, but was not. I hope to have a fix committed shortly. --Dave B. On 09/23/2013 05:59 AM, Peter Cock wrote: Hello again, I've not have any missing test results for a while, on the main or test Tool Shed, so this was a bit of a surprise: http://toolshed.g2.bx.psu.edu/view/peterjc/blastxml_to_top_descr/09a68a90d552 This is listed under Latest revision: failing tool tests, but I see no test results at all :( Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] my history not visible
On Mon, Sep 23, 2013 at 2:02 PM, Robert Jackman rais...@gmail.com wrote: Hi, For some reason my history has stopped being visible on Galaxy today. I was using Galaxy a lot last week and this morning I find only this message: An error occurred getting the history data from the server. Please contact a Galaxy administrator if the problem persists. Robert Jackman Hello Robert, Are you asking about the main public Galaxy instance run by Penn State http://usegalaxy.org - in which case try their galaxy-user mailing list, or are you running your own local Galaxy instance, in which case what version do you have? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Repository tool dependencies and virtualenv install
Hi Carlos! Hi, I have created a tool that depends on biopython. I don't like the idea of having several copies of biopython around and also for best practice making workflows using my tool reproducible, I chose to make my package repository[1] dependent of package_biopython_1_62. [1]http://testtoolshed.g2.bx.psu.edu/view/cjav/package_ngs_tools_0_1_6 Cool thanks! I think that is best practise! I'm installing my package using the virtualenv option. As you can see below I marked biopython as prior_installation_required=True. However, I can see that virtualenv is installing a copy of biopython in 'venv' under my tool install(see below). This doesn't sound right. I was expecting that by making my repository dependent of biopython's one that source would be used in my tool. Can you try to insert that (maybe adopt, its not checked): action type=set_environment_for_install repository name=package_biopython_1_62 owner=biopython package name=biopython version=1.62 / /repository /action Only with that the PYTHONPATH is populated. My plan is to create any non-existing Tool dependency definition repository for every package my tool requires. This way you can always count the same version of any of these packages is being used. I thought that was the point, right? Yes! Full ool_dependencies.xml: ?xml version=1.0? tool_dependency package name=biopython version=1.62 repository changeset_revision=ac9cc2992b69 name=package_biopython_1_62 owner=biopython prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / /package package name=ngs-tools version=0.1.6 install version=1.0 actions action type=setup_virtualenvngs-tools==0.1.6/action /actions /install /package /tool_dependency Contents of site-packages in my tool install dir: $ ls -1 shed_tools_dependencies.central/ngs-tools/0.1.6/cjav/package_ngs_tools_0_1_6/3c646b8328bb/venv/lib/python2.7/site-packages/ Bio biopython-1.62-py2.7.egg-info BioSQL docopt-0.6.1-py2.7.egg-info docopt.py docopt.pyc easy-install.pth Levenshtein.so ngs ngs_tools-0.1.6-py2.7.egg-info pip-1.3.1-py2.7.egg python_Levenshtein-0.10.2-py2.7.egg-info setuptools-0.6c11-py2.7.egg setuptools.pth Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Repository tool dependencies and virtualenv install
Am Montag, den 23.09.2013, 12:57 -0500 schrieb John Chilton: On Mon, Sep 23, 2013 at 12:51 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Carlos! Hi, I have created a tool that depends on biopython. I don't like the idea of having several copies of biopython around and also for best practice making workflows using my tool reproducible, I chose to make my package repository[1] dependent of package_biopython_1_62. [1]http://testtoolshed.g2.bx.psu.edu/view/cjav/package_ngs_tools_0_1_6 Cool thanks! I think that is best practise! I disagree strongly. This is not a good idea. My thought is if you are using a virtualenv, you shouldn't be trying to compose it with other python dependencies - to me the whole point of virtualenv is that it creates isolated little environments. I am not sure why duplicating the biopython install reduces reproduciblity. If you want to use individual Python packages using Galaxy's dependency mechanism, I think you should then package them up one at a time and hand modify PYTHONPATH and PATH - the way biopython is done. Galaxy should have a separate action to make this easy, say install_pip or something like that, as outlined by James Taylor at some point. Hi John, interesting point. What is the specific difference between venv and the pip installation? In which case do you propose to use which method? I never thought about different 'isolation-levels' (venv vs. the rest of the toolshed). We need to document somehow the use cases if we have different methods for python installations. Ciao, Bjoern -John I'm installing my package using the virtualenv option. As you can see below I marked biopython as prior_installation_required=True. However, I can see that virtualenv is installing a copy of biopython in 'venv' under my tool install(see below). This doesn't sound right. I was expecting that by making my repository dependent of biopython's one that source would be used in my tool. Can you try to insert that (maybe adopt, its not checked): action type=set_environment_for_install repository name=package_biopython_1_62 owner=biopython package name=biopython version=1.62 / /repository /action Only with that the PYTHONPATH is populated. My plan is to create any non-existing Tool dependency definition repository for every package my tool requires. This way you can always count the same version of any of these packages is being used. I thought that was the point, right? Yes! Full ool_dependencies.xml: ?xml version=1.0? tool_dependency package name=biopython version=1.62 repository changeset_revision=ac9cc2992b69 name=package_biopython_1_62 owner=biopython prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / /package package name=ngs-tools version=0.1.6 install version=1.0 actions action type=setup_virtualenvngs-tools==0.1.6/action /actions /install /package /tool_dependency Contents of site-packages in my tool install dir: $ ls -1 shed_tools_dependencies.central/ngs-tools/0.1.6/cjav/package_ngs_tools_0_1_6/3c646b8328bb/venv/lib/python2.7/site-packages/ Bio biopython-1.62-py2.7.egg-info BioSQL docopt-0.6.1-py2.7.egg-info docopt.py docopt.pyc easy-install.pth Levenshtein.so ngs ngs_tools-0.1.6-py2.7.egg-info pip-1.3.1-py2.7.egg python_Levenshtein-0.10.2-py2.7.egg-info setuptools-0.6c11-py2.7.egg setuptools.pth Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] error in 2013_08_12 DevNewsBriefs
Dear Galaxy Team I was upgrading our servers with the 8/12 release, and going over the DevNoteshttp://wiki.galaxyproject.org/DevNewsBriefs/2013_08_12. I noted item Pull Requets Merged #5: SAMtools indexes #188http://wiki.galaxyproject.org/DevNewsBriefs/2013_08_12#Pull_Requests_Merged. But the file created by the pull request didn't seem to exist. After looking in bitbucket at the history, it looks like the pull request was backed outhttps://bitbucket.org/galaxy/galaxy-central/commits/73d7a93e2f8d0bafbcb0b4dd9bf562d1fa6a88e0#general-comments before the release. Have I correctly understood the situation? Regards, Curtis ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Access to history_id from tool config XML
At line ~694: incoming['__user_name__'] = user_name + if job.history and job.history.id: + incoming['__history_id__'] = job.history.id + else: + incoming['__history_id__'] = 'unknown' I have tested this change and it appears to give me exactly what I want. My question: does this change appear correct Yes, the change is correct. and can it be incorporated into the main galaxy code-base? Can you provide a usage scenario for including history_id in the tool dict? Thanks, J. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] my history not visible
Hi, For some reason my history has stopped being visible on Galaxy today. I was using Galaxy a lot last week and this morning I find only this message: An error occurred getting the history data from the server. Please contact a Galaxy administrator if the problem persists. Robert Jackman Robert Jackman Research Associate Professor 635 Commonwealth Avenue/ Sar443 Boston, MA 02215 rjack...@bu.edu 617-353-2719 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Repository tool dependencies and virtualenv install
Hi, I have created a tool that depends on biopython. I don't like the idea of having several copies of biopython around and also for best practice making workflows using my tool reproducible, I chose to make my package repository[1] dependent of package_biopython_1_62. [1]http://testtoolshed.g2.bx.psu.edu/view/cjav/package_ngs_tools_0_1_6 I'm installing my package using the virtualenv option. As you can see below I marked biopython as prior_installation_required=True. However, I can see that virtualenv is installing a copy of biopython in 'venv' under my tool install(see below). This doesn't sound right. I was expecting that by making my repository dependent of biopython's one that source would be used in my tool. My plan is to create any non-existing Tool dependency definition repository for every package my tool requires. This way you can always count the same version of any of these packages is being used. I thought that was the point, right? Full ool_dependencies.xml: ?xml version=1.0? tool_dependency package name=biopython version=1.62 repository changeset_revision=ac9cc2992b69 name=package_biopython_1_62 owner=biopython prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / /package package name=ngs-tools version=0.1.6 install version=1.0 actions action type=setup_virtualenvngs-tools==0.1.6/action /actions /install /package /tool_dependency Contents of site-packages in my tool install dir: $ ls -1 shed_tools_dependencies.central/ngs-tools/0.1.6/cjav/package_ngs_tools_0_1_6/3c646b8328bb/venv/lib/python2.7/site-packages/ Bio biopython-1.62-py2.7.egg-info BioSQL docopt-0.6.1-py2.7.egg-info docopt.py docopt.pyc easy-install.pth Levenshtein.so ngs ngs_tools-0.1.6-py2.7.egg-info pip-1.3.1-py2.7.egg python_Levenshtein-0.10.2-py2.7.egg-info setuptools-0.6c11-py2.7.egg setuptools.pth Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Repository tool dependencies and virtualenv install
On Mon, Sep 23, 2013 at 12:51 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Hi Carlos! Hi, I have created a tool that depends on biopython. I don't like the idea of having several copies of biopython around and also for best practice making workflows using my tool reproducible, I chose to make my package repository[1] dependent of package_biopython_1_62. [1]http://testtoolshed.g2.bx.psu.edu/view/cjav/package_ngs_tools_0_1_6 Cool thanks! I think that is best practise! I disagree strongly. This is not a good idea. My thought is if you are using a virtualenv, you shouldn't be trying to compose it with other python dependencies - to me the whole point of virtualenv is that it creates isolated little environments. I am not sure why duplicating the biopython install reduces reproduciblity. If you want to use individual Python packages using Galaxy's dependency mechanism, I think you should then package them up one at a time and hand modify PYTHONPATH and PATH - the way biopython is done. Galaxy should have a separate action to make this easy, say install_pip or something like that, as outlined by James Taylor at some point. -John I'm installing my package using the virtualenv option. As you can see below I marked biopython as prior_installation_required=True. However, I can see that virtualenv is installing a copy of biopython in 'venv' under my tool install(see below). This doesn't sound right. I was expecting that by making my repository dependent of biopython's one that source would be used in my tool. Can you try to insert that (maybe adopt, its not checked): action type=set_environment_for_install repository name=package_biopython_1_62 owner=biopython package name=biopython version=1.62 / /repository /action Only with that the PYTHONPATH is populated. My plan is to create any non-existing Tool dependency definition repository for every package my tool requires. This way you can always count the same version of any of these packages is being used. I thought that was the point, right? Yes! Full ool_dependencies.xml: ?xml version=1.0? tool_dependency package name=biopython version=1.62 repository changeset_revision=ac9cc2992b69 name=package_biopython_1_62 owner=biopython prior_installation_required=True toolshed=http://testtoolshed.g2.bx.psu.edu; / /package package name=ngs-tools version=0.1.6 install version=1.0 actions action type=setup_virtualenvngs-tools==0.1.6/action /actions /install /package /tool_dependency Contents of site-packages in my tool install dir: $ ls -1 shed_tools_dependencies.central/ngs-tools/0.1.6/cjav/package_ngs_tools_0_1_6/3c646b8328bb/venv/lib/python2.7/site-packages/ Bio biopython-1.62-py2.7.egg-info BioSQL docopt-0.6.1-py2.7.egg-info docopt.py docopt.pyc easy-install.pth Levenshtein.so ngs ngs_tools-0.1.6-py2.7.egg-info pip-1.3.1-py2.7.egg python_Levenshtein-0.10.2-py2.7.egg-info setuptools-0.6c11-py2.7.egg setuptools.pth Thanks, Carlos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] cluster error
Hi guys, Since the last upgrade we have observed this error. When submitting a job it sometimes come back failed with the following error tool error An error occurred with this dataset: *Unable to run this job due to a cluster error, please retry it later* * * It is not reproducible as when you relaunch the job it works but still very annoying when demonstrating Galaxy. It looks to me that there is something wrong with torque/pbs and that when one submit a number of jobs, it reaches an internal limit of some kind, and then stops communicating with torque (the batch system). galaxy.jobs DEBUG 2013-09-24 14:05:08,692 (4264) Working directory for job is: /data/galaxyTools/galaxydev/database/job_working_directory/004/4264 galaxy.jobs.handler DEBUG 2013-09-24 14:05:08,700 (4264) Dispatching to pbs runner galaxy.jobs.runners.pbs DEBUG 2013-09-24 14:05:08,885 (4263/9034.galaxy-compute) PBS job state changed from N to R galaxy.jobs DEBUG 2013-09-24 14:05:08,925 (4264) Persisting job destination (destination id: pbs:///) galaxy.jobs.handler INFO 2013-09-24 14:05:09,014 (4264) Job dispatched galaxy.jobs.runners.pbs ERROR 2013-09-24 14:05:09,524 (4262) All attempts to submit job failed galaxy.jobs.runners.pbs DEBUG 2013-09-24 14:05:15,795 (4264) submitting file /data/galaxyTools/galaxydev/database/pbs/4264.sh galaxy.jobs.runners.pbs DEBUG 2013-09-24 14:05:15,795 (4264) command is: python /data/galaxyTools/galaxydev/tools/fastq/fastq_groomer.py '/data/galaxyTools/galaxydev/database/files/007/dataset_7546.dat' 'sanger' '/data/galaxyTools/galaxydev/database/files/007/dataset_7610.dat' 'sanger' 'ascii' 'summarize_input'; cd /data/galaxyTools/galaxydev; /data/galaxyTools/galaxydev/set_metadata.sh ./database/files /data/galaxyTools/galaxydev/database/job_working_directory/004/4264 . /data/galaxyTools/galaxydev/universe_wsgi.ini /data/galaxyTools/galaxydev/database/tmp/tmpcdfNIZ /data/galaxyTools/galaxydev/database/job_working_directory/004/4264/galaxy.json /data/galaxyTools/galaxydev/database/job_working_directory/004/4264/metadata_in_HistoryDatasetAssociation_9064_Zk1kgC,/data/galaxyTools/galaxydev/database/job_working_directory/004/4264/metadata_kwds_HistoryDatasetAssociation_9064_yBjKtV,/data/galaxyTools/galaxydev/database/job_working_directory/004/4264/metadata_out_HistoryDatasetAssociation_9064_RBQTv6,/data/galaxyTools/galaxydev/database/job_working_directory/004/4264/metadata_results_HistoryDatasetAssociation_9064_zBFTMS,,/data/galaxyTools/galaxydev/database/job_working_directory/004/4264/metadata_override_HistoryDatasetAssociation_9064_0W_zYn galaxy.jobs.runners.pbs WARNING 2013-09-24 14:05:15,796 (4264) pbs_submit failed (try 1/5), PBS error 15033: No free connections galaxy.jobs.runners.pbs WARNING 2013-09-24 14:05:17,798 (4264) pbs_submit failed (try 2/5), PBS error 15033: No free connections galaxy.jobs.runners.pbs WARNING 2013-09-24 14:05:19,800 (4264) pbs_submit failed (try 3/5), PBS error 15033: No free connections Did you guys have ever experienced the same problem and if so how did you solve it ? Regards, Philippe ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
