Re: [galaxy-dev] Public toolshed giving internal server error
Hi all, Sorry for the service interruption. The Tool Shed should be back now, and the underlying disk usage problem has been alleviated, so it shouldn't occur again. --nate On Wed, Nov 19, 2014 at 9:15 AM, Lukasse, Pieter pieter.luka...@wur.nl wrote: I am also having problems when using the option Search and browse toolsheds ! -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto: galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Peter Briggs Sent: woensdag 19 november 2014 14:01 To: Björn Grüning; Galaxy Dev Subject: Re: [galaxy-dev] Public toolshed giving internal server error Hello Bjoern Thanks, seems to be okay now (on the main toolshed) - I can log out, log back in, and create and populate my new repository now. Thanks, best wishes Peter On 19/11/14 10:57, Björn Grüning wrote: Hi Peter, can you try again? I'm able to browse the ToolShed and reset metatada for example. Cheers, Bjoern Am 19.11.2014 um 10:36 schrieb Peter Briggs: Hello I'm trying to make a new repository on the public toolshed at https://toolshed.g2.bx.psu.edu/ but I keep getting the internal server error page. I was also unable to log out, or even to see the front page when trying to access it from a different browser. Is anyone else having this problem? Thanks best wishes Peter -- Peter Briggs peter.bri...@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] MarkupSafe egg missing? e.args[1].key != e.args[0].key
Peter, Unless you've made modifications to Galaxy that depend on external libraries, switching to a virtualenv for the server itself should be pretty safe. Tools themselves can still run without using the/any virtualenv, if desired. --nate On Tue, Nov 18, 2014 at 8:24 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Nov 18, 2014 7:26 AM, John Chilton jmchil...@gmail.com wrote: I will admit to not actually understanding Galaxy's dependency management but I think virtualenv is exactly the advice people who do understand it give http://dev.list.galaxyproject.org/Local-installation-problem-td4662627.html. It is a widely used tool precisely designed to solve such problems - I think it is the best way to go. I don't know why it would not be appropriate for existing installations - I think it is in fact somethimg of a best practice for existing installations. Our existing installation is not using a virtual env, and I fear switching to that could be disruptive. Certainly that error message should be more helpful but I am not sure we should do anything to address this beyond that - do you have a particular idea in mind? Not show the IndexError exception? :P Here the user should be told something about a conflict between MarkupSafe and paramiko (assuming this is the real problem). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Galaxy Mailing Lists are moving
Hello Galaxy Community, *Galaxy Mailing Lists are moving!* During the week of November 17-21, 2014, Galaxy mailing lists (including this one) will be migrated from lists.bx.psu.edu to lists.galaxyproject.org . This transition should be largely transparent to you, but there are a few things to be aware of: - List sender addresses and headers will change to reflect the updated domain: from bx.psu.edu to galaxyproject.org. Existing email filters you have set up may require adjustments. - Posts from lists.galaxyproject.org could be categorized as spam until you train your filtering method. - Mailing list archives and posting functionality will be briefly inaccessible during the migration. - The prior bx.psu.edu list posting email addresses will continue to accept email, which will be forwarded to the new list addresses. As always, you can manage your subscription for any Galaxy mailing list by following the link instructions at the bottom of this announcement. If you should notice any problems after the migration, please do not hesitate to let us know, via email to both list-ow...@lists.galaxyproject.org (after the move) and to myself. Thanks, --nate, and the Galaxy Team ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] problems installing tool dependency
I've just looked at this as well, it appears that /pub4 is not exposed and does not allow you to change into it, but it does allow you to fetch if you request files under it via the full path. e.g.: nate@victory% ncftp ftp.sanger.ac.uk ... Logged in to ftp.sanger.ac.uk. ncftp / ls ls-lR.gz.ok_to_rsyncREADME .request-for-update.lock .messagepub/ .request-for-update ncftp / cd pub4 Could not chdir to pub4: server said: pub4: No such file or directory ncftp / cd /pub4 Could not chdir to /pub4: server said: pub4: No such file or directory ncftp / get /pub4/resources/software/quicktree/quicktree.tar.gz quicktree.tar.gz: 38.12 kB 124.19 kB/s ncftp / I'll contact some folks in the Miller Lab to see if the tool can be updated to the path under /pub. On Wed, Sep 24, 2014 at 2:09 PM, Will Holtz who...@lygos.com wrote: I also was able to download quicktree.tar.gz by clicking on the URL in your message. However, when I manually ftp into ftp.sanger.ac.uk, there is no /pub4 directory. I'm a bit confused about how those two statements are both true, but maybe it will help someone else figure this out. Removing the '4' does result in a valid path ( ftp://ftp.sanger.ac.uk/pub/resources/software/quicktree/quicktree.tar.gz). -Will On Wed, Sep 24, 2014 at 10:57 AM, Sajdak, Doris dj...@buffalo.edu wrote: Hi all, I am trying to install the Genome Diversity tools from the Miller Lab that are in the Galaxy toolshed. I’ve been able to get all dependencies installed except Quicktree. Whenever I try to install it, I get the error: Error downloading from URL ftp://ftp.sanger.ac.uk/pub4/resources/software/quicktree/quicktree.tar.gz : urlopen error ftp error: 550 pub4: No such file or directory This is the tool_dependencies.xml file for installing quicktree: ?xml version=1.0? tool_dependency package name=quicktree version=1.1 install version=1.0 actions !-- Download source code -- action type=download_by_url target_filename=quicktree_1.1.tar.gz ftp://ftp.sanger.ac.uk/pub4/resources/software/quicktree/quicktree.tar.gz /action !-- Build quicktree -- action type=shell_commandmake quicktree/action !-- Install quicktree -- action type=shell_commandcp -R bin $INSTALL_DIR/action !-- Set environment for dependent repositories -- action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install /package /tool_dependency I know this is not their file I’m downloading but the thing is, if I copy and paste the URL, I can download it with no problem. I’m not sure why it’s reporting that it can’t be found but the xml looks right to me. I’ve tried contacting the Miller Lab about this but haven’t heard from anyone so I’m hoping someone here can tell me how to move past this problem. Is there a way to install this tool manually and have the main Genome Diversity tools recognize it on the Galaxy server? Also, I noticed when looking at the Tool Dependency Definitions at https://toolshed.g2.bx.psu.edu/ there are errors going back many versions for this tool (see Test runs – Installation errors – Tool dependencies). Thank you for any help you can provide. Dori ** Dori Sajdak Senior Systems Administrator State University of NY at Buffalo Center for Computational Research 701 Ellicott St Buffalo, NY 14203 Phone: (716) 881-8934 Fax: (716) 849-6656 Web: http://ccr.buffalo.edu ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Missing configuration and location files automatic creation
Hi Julien, I agree with your idea. As you say, there are still a number of defaults that we will set up with common_startup.sh even if they are configured in galaxy.ini. I had always assumed that most deployers who are using custom locations are bypassing run.sh anyway. For usegalaxy.org we just call fetch_eggs.py prior to starting and invoke Galaxy directly (via uWSGI for the web processes and individual calls to `python ./scripts/paster.py serve /path/to/galaxy.ini` for the handlers). run.sh is intended as more of a bootstrap script for uncustomized copies of Galaxy, but I do realize people use it as the primary method for controlling their servers. We intend to address the remaining items in common_startup.sh in a similar fashion to the config files we have already moved to config/ - namely, attempting to use the .sample file if the non-.sample does not exist and the location is the default. The only case where this won't be possible is with the first four files, which Galaxy needs to be able to modify. So I am not trying to deter you, but I also don't want you to waste a lot of effort on this. Thanks, --nate On Sat, Sep 20, 2014 at 12:56 AM, Julien Seiler julien.sei...@gmail.com wrote: Hi Nate, Indeed, I have noticed some significant improvements in the config files layout. All config files have been moved to the config/ directory and the automatic creation of missing config/location files is now handled by the common_startup.sh script. This is great but it is not solving my main concern about creating « unused » config files at startup. To be more specific, I will give you a short description of our Galaxy instance layout. We are currently sharing the sources of our Galaxy instance because we want to share our Vagrant setup with Galaxy tools developers. However, we actually don’t want to share all our config files but still want to keep them under version control. That’s why we decided to move all our config/location files in a private repository and point to this specific location in our main galaxy.ini config. In my opinion, if a galaxy.ini file already existing and is pointing to non-default location for some config files ou tool-data path, the Galaxy startup script should not be creating the corresponding default config files to the default location. My idea was to refactor the common_startup.sh script in order to parse an eventually existing galaxy.ini and then create each default config files to the location pointed by galaxy.ini. -- Julien Seiler @julozi https://twitter.com/julozi Le 20 sept. 2014 à 03:10, Nate Coraor n...@bx.psu.edu a écrit : Hi Julien, What timing for this email - we have just made significant changes to the default config layout, and this includes no longer copying sample configs to their non-sample locations. The changes are in our default branch right now and will be part of the stable release scheduled for October. Have a look, I'd be interested to hear your feedback since you've already put some thought into this. --nate On Fri, Sep 19, 2014 at 3:10 PM, Julien Seiler julien.sei...@gmail.com wrote: Hi all, I have notice that the run.sh script is creating automatically all missing configuration and location files to make sure Galaxy can work properly at first launch. However, the script is not taking care of the actual settings of an existing universe_wsgi.ini file. As a result, if the universe_wsgi.ini file says that job_conf.xml is located at foo/bar location, the run.sh will create a job_conf.xml file at the root of the galaxy source code anyway. I'm thinking of improving the run.sh behavior to respect file path declared in an existing universe_wsgi.ini file in order not to create unused config or location files. I have currently started to implement this behavior in a small Python script (more handy than pure sh). What do you think about this concern ? Should I go for a pull request at any point ? Regards, -- Julien @julozi https://twitter.com/julozi ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Missing configuration and location files automatic creation
Hi Julien, What timing for this email - we have just made significant changes to the default config layout, and this includes no longer copying sample configs to their non-sample locations. The changes are in our default branch right now and will be part of the stable release scheduled for October. Have a look, I'd be interested to hear your feedback since you've already put some thought into this. --nate On Fri, Sep 19, 2014 at 3:10 PM, Julien Seiler julien.sei...@gmail.com wrote: Hi all, I have notice that the run.sh script is creating automatically all missing configuration and location files to make sure Galaxy can work properly at first launch. However, the script is not taking care of the actual settings of an existing universe_wsgi.ini file. As a result, if the universe_wsgi.ini file says that job_conf.xml is located at foo/bar location, the run.sh will create a job_conf.xml file at the root of the galaxy source code anyway. I'm thinking of improving the run.sh behavior to respect file path declared in an existing universe_wsgi.ini file in order not to create unused config or location files. I have currently started to implement this behavior in a small Python script (more handy than pure sh). What do you think about this concern ? Should I go for a pull request at any point ? Regards, -- Julien @julozi https://twitter.com/julozi ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] find bam index files
Hi Ulf, Unfortunately, metadata files are assigned an id that is unrelated to the dataset id. To link the files you'll need to use the database. The numerical id you see in the filename is the id of the dataset in the `dataset` table. From this you can go backward to the history_dataset_association (hda) table, where hda.dataset_id = dataset.id. Finally, from there, hda.id can be used to find the metadata_file via metadata_file.hda_id. More succinctly: dataset.id = history_dataset_association.dataset_id history_dataset_association.id = metadata_file.hda_id --nate On Mon, Sep 15, 2014 at 6:34 AM, Ulf Schaefer ulf.schae...@phe.gov.uk wrote: Dear all For each bam file in my history I can download the associated bai (bam index) file. I assume these files are stored somewhere under /mount/galaxy/database/files/_metadata_files. Correct? Is there an easy way to find the bam index file for a bam file, given only the internal file name of the bam (e.g. /mont/galaxy/database/files/089/dataset_89231.dat)? I am asking because I would like to use the files_to_ftp tool to automatically download bams together with their associated indices. Thanks Ulf ** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of Public Health England, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. http://www.gov.uk/PHE ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Queueing system and job parameters
Hi Prakash, At this point, the documentation is the examples in job_resource_params_conf.xml.sample and job_conf.xml.sample. This code was in the August 2014 release, so you'll need to upgrade to use it. Here is the commit, which shows the changes to job_conf.xml.sample: https://bitbucket.org/galaxy/galaxy-central/commits/31b2924315d0b48fa7648ab4bfd04ef754829f71 --nate On Sun, Sep 14, 2014 at 8:34 AM, Velayutham, Prakash (Prakash) prakash.velayut...@cchmc.org wrote: Hi Nate, Can you point me to the documentation that explains this in more detail? What version of Galaxy supports this? My build is at least 3 months old, probably even more. Thanks, Prakash On Sep 11, 2014, at 1:17 PM, Nate Coraor n...@bx.psu.edu wrote: Hi Prakash, You can use the relatively new job resource params feature to insert a standard set of params on any tool form. See the example at: https://bitbucket.org/galaxy/galaxy-central/src/tip/job_resource_params_conf.xml.sample Here is an example of how we're using it on the Galaxy Test server: https://github.com/galaxyproject/usegalaxy-playbook/blob/master/files/galaxy/test.galaxyproject.org/config/job_resource_params_conf.xml https://github.com/galaxyproject/usegalaxy-playbook/blob/master/files/galaxy/test.galaxyproject.org/dynamic_rules/roundup_multi_walltime.py --nate On Wed, Sep 10, 2014 at 8:57 PM, Velayutham, Prakash (Prakash) prakash.velayut...@cchmc.org wrote: Hi, I have a local Galaxy instance. I have tied the server to a local HPC and we use LSF for scheduling jobs. I was wondering how a user can submit a job from Galaxy while specifying a wall time and memory requirements for a job… Is this doable? Thanks, Prakash ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Keeping Galaxy up to date
Hi Sebastian, As for Galaxy Main (usegalaxy.org), our process is automated with Ansible, the playbook for which can be found here: https://github.com/galaxyproject/usegalaxy-playbook A bit of documentation on what you should look for (new versions of universe_wsgi.ini.sample, datatypes_conf.xml.sample, etc.) would certainly be helpful, so I'll expand the keeping up to date page on getgalaxy.org into a new page and add more details. On Tue, Sep 9, 2014 at 6:55 AM, Sebastian Schaaf sch...@ibe.med.uni-muenchen.de wrote: Hi Thomas, We had this topic at the GCC this year, especially within the Galaxy Admins BoF. The truth is (please correct me anyone if this is not the case anymore!) that you are touching an unresolved point. Keeping track of tools, settings etc. across the versions is not given. People have their 'home-brew' solutions for this, e.g. keeping a (maybe virtual) machine in spare in order to invest several hours up to some days to test (with or without participation of the users) applicability to local constraints/histories/workflow/tools. The more testing is automated and the less users (or non-default pieces) the faster the update procedure can be. But there are also instances which did not receive updates for months or even years. On our side we will be facing reality within the next two weeks as we really need the update. Although preconditions are pretty good (few users, not that deep modifications, high grade of automation) we expect some issues. happy to have a virtualized environment... To wrap up my answer in a nutshell: no, there is no best practice guide and no migration tool, but every single contribution is warmly welcome :). It would be *very* interesting how updates are handled at Galaxy Main...? Cheers, Sebastian (very interested in further feedback) Hello, I try to keep our Galaxy instance up to date. Very often, some tools disappear and other do not work anymore. This breaks some users workflows. I may miss something. Is there a best update pratice to keep a Galaxy instance and the tool-sheds fully functionnal ? Regards, Thomas -- Thomas Bellembois, Network and System Administrator, IGFL (France) http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67 (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Sebastian Schaaf, M.Sc. Bioinformatics Faculty Coordinator NGS Infrastructure Chair of Biometry and Bioinformatics Department of Medical Informatics, Biometry and Epidemiology (IBE) University of Munich Marchioninistr. 15, K U1 (postal) Marchioninistr. 17, U 006 (office) D-81377 Munich (Germany) Tel: +49 89 2180-78178 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] ToolShed test failure: NotFound: cannot find 'ucsc_display_sites' while searching for 'APP.config.ucsc_display_sites'
Hi Peter, This was due to a bug I introduced last week, which I've just fixed in d1f6d05. Sorry for the trouble. --nate On Tue, Sep 9, 2014 at 9:50 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi all, I'm wondering why my samtools_depad repository tests have failed, and since I have not changed this recently presume this is due to a Galaxy change or general TestToolShed problem not specific to my tool: https://testtoolshed.g2.bx.psu.edu/view/peterjc/samtools_depad Tests that failed Tool id: samtools_depad Tool version: samtools_depad Test: test_tool_00 ( functional.test_toolbox.TestForTool_testtoolshed.g2.bx.psu.edu/repos/peterjc/samtools_depad/samtools_depad/0.0.1 ) Stderr: Traceback: Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 114, in test_tool self.do_it( td ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 35, in do_it self._verify_outputs( testdef, test_history, shed_tool_id, data_list, galaxy_interactor ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/functional/test_toolbox.py, line 75, in _verify_outputs galaxy_interactor.verify_output( history, output_data, output_testdef=output_testdef, shed_tool_id=shed_tool_id, maxseconds=maxseconds ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py, line 82, in verify_output self._verify_metadata( history_id, hid, attributes ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/test/base/interactor.py, line 103, in _verify_metadata raise Exception( msg ) Exception: Dataset metadata verification for [file_ext] failed, expected [bam] but found [None]. Traceback (most recent call last): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/web/framework/decorators.py, line 243, in decorator rval = func( self, trans, *args, **kwargs) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/webapps/galaxy/api/history_contents.py, line 188, in show return self.__show_dataset( trans, id, **kwd ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/webapps/galaxy/api/history_contents.py, line 214, in __show_dataset hda_dict[ 'display_apps' ] = self.get_display_apps( trans, hda ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/web/base/controller.py, line 855, in get_display_apps for display_app in hda.get_display_applications( trans ).itervalues(): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/model/__init__.py, line 1754, in get_display_applications return self.datatype.get_display_applications_by_dataset( self, trans ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/datatypes/data.py, line 445, in get_display_applications_by_dataset value = value.filter_by_dataset( dataset, trans ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/datatypes/display_applications/application.py, line 200, in filter_by_dataset if link_value.filter_by_dataset( data, trans ): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/datatypes/display_applications/application.py, line 78, in filter_by_dataset if fill_template( filter_elem.text, context = context ) != filter_elem.get( 'value', 'True' ): File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/lib/galaxy/util/template.py, line 9, in fill_template return str( Template( source=template_text, searchList=[context] ) ) File /var/opt/buildslaves/buildslave-ec2-2/buildbot-install-test-test-tool-shed-py27/build/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg/Cheetah/Template.py, line 1004, in __str__ return getattr(self, mainMethName)() File cheetah_DynamicallyCompiledCheetahTemplate_1410263883_33_43576.py, line 82, in respond NotFound: cannot find 'ucsc_display_sites' while searching for 'APP.config.ucsc_display_sites' requests.packages.urllib3.connectionpool: DEBUG: GET /api/histories/993bad2fe35335db/contents/7fbe67cfae825002?key=edc04240db9605fb7edc7bab44d3404c HTTP/1.1 500 None requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): 127.0.0.1 requests.packages.urllib3.connectionpool: DEBUG: GET /api/histories/993bad2fe35335db/contents/7fbe67cfae825002/provenance?key=edc04240db9605fb7edc7bab44d3404c HTTP/1.1 200 None
Re: [galaxy-dev] Join tool failure
On Sep 4, 2014, at 2:44 PM, Iry Witham iry.wit...@jax.org wrote: Hi Team, I have upgraded to the latest galaxy distribution on my test server. Now I am running into issues and can't figure this one out. I am attempting to run the join.py tool to join two datasets at specified fields. I am getting the following error: Traceback (most recent call last): File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/tools/filters/join.py, line 18, in module from galaxy.util.bunch import Bunch File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/galaxy/__init__.py, line 95, in module import galaxy.eggs File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/galaxy/eggs/__init__.py, line 5, in module import os, sys, shutil, glob, urllib, urllib2, ConfigParser, HTMLParser, zipimport, zipfile File /usr/lib64/python2.6/urllib2.py, line 93, in module import hashlib File /usr/lib64/python2.6/hashlib.py, line 88, in module import _hashlib ImportError: /usr/lib64/libssl.so.10: symbol EC_KEY_get0_group, version OPENSSL_1.0.1_EC not defined in file libcrypto.so.10 with link time reference Traceback (most recent call last): File ./scripts/set_metadata.py, line 29, in module from galaxy import eggs File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/galaxy/__init__.py, line 95, in module import galaxy.eggs File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/galaxy/eggs/__init__.py, line 5, in module import os, sys, shutil, glob, urllib, urllib2, ConfigParser, HTMLParser, zipimport, zipfile File /usr/lib64/python2.6/urllib2.py, line 93, in module import hashlib File /usr/lib64/python2.6/hashlib.py, line 88, in module import _hashlib ImportError: /usr/lib64/libssl.so.10: symbol EC_KEY_get0_group, version OPENSSL_1.0.1_EC not defined in file libcrypto.so.10 with link time reference I am at a loss on the resolution for this. Thanks, Iry Hi Iry, It looks like your libssl and libcrypto versions are out of sync. This shouldn't happen as both are provided by the OpenSSL packages for your OS (RHEL-based 6.x?), but I see that others have encountered similar problems: https://www.centos.org/forums/viewtopic.php?f=14t=44075 You might want to double check that your OpenSSL packages have not been replaced by versions from another source other than the official RHEL/CentOS core repositories. --nate The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] problems with database migration 119 - 120
Hi Hans, I'll add to John's statement about the ordering - a different ordering than the model is entirely possible because alters to add columns always add them at the end. The tables for Galaxy Main are so old that they are in a very different order than most newer installations. As John says, SQLAlchemy always references columns by name, so the order does not matter. The reason we are using an old version of SQLAlchemy is due to it being the last version that can be used with sqlalchemy_migrate. Upgrading to newer versions will also require a significant amount of work to rewrite to use a new versioning system. This will be done eventually but hasn't reached the top of the pile yet. Finally, you should switch to PostgreSQL. ;D --nate On Aug 26, 2014, at 11:37 AM, Hans-Rudolf Hotz h...@fmi.ch wrote: Hi John Thanks for the insights Hi Iyad Yes, the user account has the ability to drop and alter tables. Hans-Rudolf On 08/26/2014 03:08 PM, John Chilton wrote: Well it looks like the migration file has these columns listed in a different order than the mapping Galaxy uses - and the order yours appeared in were the ones from Galaxy's mapping file. So somehow Galaxy is automatically creating those tables prior to running the migration based on the code in Galaxy proper. That is odd. Ultimately though - I don't think it is harmful that the order is wrong since Galaxy always references the columns by name instead of order. I would be more concerned that you aren't getting the right indices / foreign keys - but it looks like you are still on MyISAM so you aren't going to get a ton of forced integrity anyway (and maybe this is why these errors have been okay in the past?). -John On Tue, Aug 26, 2014 at 8:52 AM, Kandalaft, Iyad iyad.kandal...@agr.gc.ca wrote: I've ran into other mysql problems but never anything like that. This is just a hunch and not based on anything concrete, but using an outdated version of sqlalchemy is probably not helping things. We're talking 2 migrations. Are they even implementing fixes for 0.7 anymore? It is odd you would ever get Table already exists since the ORM's job is to ensure the model consistency in the database. Why it would try to create the table before it checks for its existence is beyond me. You might want to make sure that the database user account has the ability to drop and alter tables. It may be that it tried to revert a failed upgrade and it wasn't able to. Iyad Kandalaft -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Hans-Rudolf Hotz Sent: Tuesday, August 26, 2014 5:22 AM To: galaxy-...@bx.psu.edu Subject: [galaxy-dev] problems with database migration 119 - 120 Hi all I am in the process of updating our galaxy servers (from release_2014.04.14 to latest_2014.08.11). when I execute ~/lib/galaxy/model/migrate/versions/0120_dataset_collections.py as part of the 'manage_db.sh upgrade' I run into a bizarre error: First, it produces 10 Mysql 1050 Error 'Table already exists' , I have encountered this before, and usually everything is fine. The table gets created and for whatever reason, the command get's executed a second time - no big deal. However, this time for two of those ten table the situation has been different. As usual, I have checked all the tables (where I got the errors) with the MySQL command describe table. For two tables: history_dataset_collection_association library_dataset_collection_association the order of the columns was wrong (ie did not correspond to the order in the create statement) - see below for example. I have dropped the tables and executed the create statements manually, everything seems fine, eg mysql describe library_dataset_collection_association; +---+--+--+-+-++ | Field | Type | Null | Key | Default | Extra | +---+--+--+-+-++ | id| int(11) | NO | PRI | NULL| auto_increment | | collection_id | int(11) | YES | MUL | NULL|| | folder_id | int(11) | YES | MUL | NULL|| | name | varchar(255) | YES | | NULL|| | deleted | tinyint(1) | YES | | NULL|| +---+--+--+-+-++ 5 rows in set (0.00 sec) mysql drop table library_dataset_collection_association; Query OK, 0 rows affected (0.02 sec) mysql CREATE TABLE library_dataset_collection_association ( - id INTEGER NOT NULL AUTO_INCREMENT, - collection_id INTEGER, - name VARCHAR(255), - deleted BOOL, - folder_id INTEGER, - PRIMARY KEY (id), - FOREIGN
Re: [galaxy-dev] Galaxy and HTTPS User-authentification
On Aug 20, 2014, at 3:53 AM, Matthias Enders m.end...@german-seed-alliance.de wrote: Hi, I have a question regarding the user authentication of Galaxy. As to my knowledge galaxy uses http, also for the authentication, so the User Email and the password are send in clear text. As I like to use Galaxy for user-authentication and due several disadvantages not an external authentification like described here (https://wiki.galaxyproject.org/Admin/Config/ExternalUserDatbases): Is there any way of using https or an encryption-method for sending user email and password. Hi Matthias, You can still serve Galaxy over HTTPS by placing it behind a proxy, even if you do not intend to perform authentication in the proxy. You'll just need to set X-URL-SCHEME in the proxy, as documented: For nginx: https://wiki.galaxyproject.org/Admin/Config/nginxProxy For Apache: https://wiki.galaxyproject.org/Admin/Config/ApacheProxy --nate With kind regards, Matthias Enders ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Database integrity error when installing new tools from toolshed
On Aug 21, 2014, at 3:48 PM, Michael Ta m...@pacificdx.com wrote:
Hi,
I updated a local Galaxy installation a while back and I followed the steps
listed on one of the wiki pages. The update included changes to the database
schema and I was careful to backup and restore it according to the
directions. However, when I try to install new tools from the tool shed or
update a tool to a new revision it does not complete successfully. The
following error appears in the log file.
File '/home/galaxy/galaxy-dist/lib/tool_shed/util/tool_util.py', line 761 in
handle_tool_versions
context.flush()
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/scoping.py',
line 114 in do
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py',
line 1718 in flush
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/session.py',
line 1789 in _flush
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.p
y', line 331 in execute
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/unitofwork.p
y', line 475 in execute
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.
py', line 64 in save_obj
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/orm/persistence.
py', line 558 in _emit_insert_statements
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py',
line 1449 in execute
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py',
line 1584 in _execute_clauseelement
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py',
line 1698 in _execute_context
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py',
line 1691 in _execute_context
File
'/home/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.p
y', line 331 in do_execute
IntegrityError: (IntegrityError) duplicate key value violates unique
constraint tool_version_association_pk ey
DETAIL: Key (id)=(83) already exists.
'INSERT INTO tool_version_association (tool_id, parent_id) VALUES
(%(tool_id)s, %(parent_id)s) RETURNING to ol_version_association.id'
{'parent_id': 440, 'tool_id': 439}
A few of the keys appear to be colliding with values already in the database,
did I miss something when I updated and restored the database that is causing
this?
Hi Michael,
The documentation should say to back up, but a restore is not necessary unless
you encounter some problem during the migration. Could you point me at the
documentation in question so I can update it?
Could you check that your backup included the sequences and that when you
restored the database, you restored the sequences? It looks like the sequence
in question (tool_version_association_id_seq) is not actually set to the
max(id) on the tool_version_association table, which means that the nextval
selected from that sequence would conflict with an id already in the table.
--nate
Thanks,
--
Michael Ta
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] TypeError with 'dict'
Hi Martin,
If there's a full traceback instead of just this fragment, that'd certainly
help with determining exactly where the problem is coming from.
--nate
On Aug 12, 2014, at 4:17 AM, bjoern.gruen...@googlemail.com
bjoern.gruen...@gmail.com wrote:
Hi,
Galaxy stores all information about a tool and it's parameters in a database.
I suppose if something is wrong with your tool, under some circumstances, it
can't be stored in the database.
Cheers,
Bjoern
2014-08-12 9:23 GMT+02:00 Martin Christiansen
martinchristianse...@hotmail.com:
Hi Björn,
I'm using galaxy as a front-end to run a larger pipeline in the background.
Originally I implemented the pipeline which had the same wrapper and was
running fine.
I have now begun to break it down into steps where this is the first step.
The only thing I've changed is the output.
How would this cause an error in the python egg?
Martin
Date: Tue, 12 Aug 2014 09:06:51 +0200
From: bjoern.gruen...@gmail.com
To: martinchristianse...@hotmail.com; galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] TypeError with 'dict'
Hi Martin,
please keep galaxy-dev in the CC list.
Am 12.08.2014 um 08:51 schrieb Martin Christiansen:
Hi Björn,
Most certainly. I have posted it below.
tool id=screen_reads name=Screen Reads
- add here a version number version=0.1
descriptionagainst hg19/description
command interpreter=bashscreen_reads.sh $Project.input
$Project.samples $Project $Samples /command
inputs
conditional name=Project
param name=input type=select label=Select project
option value=galaxy_test1galaxy_test1/option
option value=Untitled FolderUntitled Folder/option
- is the white space really needed? If so $Project.input will be two
words. Use ${Project.input} to convert it to onw argument
/param
when value=galaxy_test1
param name=samples type=select label=Select samples
display=checkboxes multiple=True
option value=147406386-700171390147406386-700171390/option
option value=158256496-700097688158256496-700097688/option
option value=158337416-700013715158337416-700013715/option
option value=158337416-700097837158337416-700097837/option
option value=158357646-700035237158357646-700035237/option
option value=158458797-700014562158458797-700014562/option
option value=158479027-700014724158479027-700014724/option
option value=158479027-700097196158479027-700097196/option
option value=158499257-700014837158499257-700014837/option
option value=158499257-700098561158499257-700098561/option
option value=158742018-700015181158742018-700015181/option
option value=158802708-700015245158802708-700015245/option
option value=158802708-700015250158802708-700015250/option
option value=158802708-700099803158802708-700099803/option
option value=158802708-700119165158802708-700119165/option
option value=158822939-700014954158822939-700014954/option
option value=158883629-700015113158883629-700015113/option
option value=158883629-700100227158883629-700100227/option
option value=158883629-700112812158883629-700112812/option
option value=158924089-700099307158924089-700099307/option
/param
/when
when value=Untitled Folder
param name=samples type=select label=Select samples
display=checkboxes multiple=True
- you don't have any option here.
/param
/when
/conditional
/inputs
outputs
data format=tabular name=Project label=Project
from_work_dir=/eva/projects/smash/MOCAT/test/martin/$Project.input/$Project.input/
data format=tabular name=Samples label=Samples
from_work_dir=/eva/projects/smash/MOCAT/test/martin/$Project.input/$Project.samples/
/outputs
- I'm not sure this will work. workdir is a special dir Galaxy creates
for you. It will be the working directory where your program get's
called. So assume your program creates foo.sam files. You can specify it
like from_work_dir='foo.sam'
The input handling looks also a little bit strange. You do not specify
any input file in Galaxy. That is not really Galaxy like :)
Cheers,
Bjoern
help
Reads screened against hg19.
/help
/tool
Best,
Martin
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
Re: [galaxy-dev] Galaxy error
Hi Emily,
We might be able to track this down on our side if you provide the URL and HTTP
method that caused this error and the time it occurred at.
Thanks,
--nate
On Aug 11, 2014, at 9:20 AM, John Chilton jmchil...@gmail.com wrote:
Hello,
Sorry for the delay - please keep all correspondences on galaxy-dev to
maximize the likelihood someone will respond and respond promptly. I do not
know what would cause this error - does anyone else have any ideas?
-John
On Mon, Aug 4, 2014 at 2:07 PM, Beck, Emily A emily-b...@uiowa.edu wrote:
Sorry for my delayed response. The error is on the main public server, and
happens when I try to map with Bowtie for Illumina. Thank you.
~Emily
Emily Beck
PhD Candidate
Llopart Lab
469 Biology Building
Iowa City, IA 52242
Lab: (319)335-3430
From: John Chilton [jmchil...@gmail.com]
Sent: Tuesday, July 22, 2014 4:06 PM
To: Beck, Emily A
Cc: galaxy-...@bx.psu.edu
Subject: Re: [galaxy-dev] Galaxy error
Hello,
Thanks for the bug report. Is this on the main public server (usegalaxy.org)
or a local instance at the University of Iowa?
-John
On Wed, Jul 16, 2014 at 10:39 AM, Beck, Emily A emily-b...@uiowa.edu wrote:
Hello,
I have repeatedly gotten the following error message when attempting to use
both the mapping and pileup functions in Galaxy. I am using files that I
have previously used successfully with these functions, and cannot fix it.
Thank you.
~Emily Beck
user
username emilybeck
quota_percent 19
total_disk_usage 52086193970
nice_total_disk_usage 48.5 GB
email emily-b...@uiowa.edu
is_admin false
tags_used
model_class User
id03f0554e2314759e
sourceHDACollection(f313d1d65c4ee57e,451)
xhr
readyState4
responseText {err_msg: Uncaught exception in exposed API method:,
err_code: 0}
responseJSON
err_msg Uncaught exception in exposed API method:
err_code 0
status500
statusTextInternal Server Error
responseHeaders
Servernginx/1.4.7
Date Wed, 16 Jul 2014 15:19:52 GMT
Content-Type application/json
Transfer-Encoding chunked
Connectionkeep-alive
Cache-Control max-age=0,no-cache,no-store
options
data
parse true
emulateHTTP false
emulateJSON false
Emily Beck
PhD Candidate
Llopart Lab
469 Biology Building
Iowa City, IA 52242
Lab: (319)335-3430
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Problems uploading files to a data library
On Tue, Aug 12, 2014 at 3:03 AM, Juan Vladimir de la Rosa Medina jvdelar...@hotmail.com wrote: Hi everyone, I recently have installed a local instance of galaxy in my computer, everything worked fine until I changed the path for *library_import_dir* in the *universe_wsgi.ini* file, to upload files to data libraries. I followed the Galaxy documentation to setup this feature: This is the path in the *universe_wsgi.ini* file *library_import_dir = /media/New/Vladimir/My_RNA-seq/* The steps I followed are: *Admin Data Library Create new data library Add datasets Upload directory of files* file format was set to *auto-detect* server Directory was */media/New/Vladimir/My_RNA-seq/* and we chose the option to *link to files* instead of copying them When I try to upload files, galaxy displays the following error: *Error Traceback:* *⇝ AttributeError: 'NoneType' object has no attribute 'new_state'* Juan, A bit of a guess here - have you removed the upload tool from your tool_conf.xml? --nate *?xml version=1.0 ?tracebacksysinfolanguage version=2.7.3Python/language/sysinfostackframe moduleweberror.evalexception.middleware/module filename/home/labinmunologiaulpgc/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py/filename line364/linefunctionrespond/function operationapp_iter = self.application(environ, detect_start_response)/operationoperation_context try: __traceback_supplement__ = errormiddleware.Supplement, self, environapp_iter = self.application(environ, detect_start_response)try:return_iter = list(app_iter) /operation_context/frameframe modulepaste.recursive/module filename/home/labinmunologiaulpgc/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py/filename line84/linefunction__call__/function operationreturn self.application(environ, start_response)/operationoperation_context environ['paste.recursive.script_name'] = my_script_name try:return self.application(environ, start_response) except ForwardRequestException, e:middleware = CheckForRecursionMiddleware(/operation_context/frame framemodulepaste.httpexceptions/module filename/home/labinmunologiaulpgc/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py http://httpexceptions.py//filename line633/line function__call__/functionoperationreturn self.application(environ, start_response)/operation operation_context []).append(HTTPException) try:return self.application(environ, start_response)except HTTPException, exc:return exc(environ, start_response)/operation_context/frame framemodulegalaxy.web.framework.base/module filename/home/labinmunologiaulpgc/galaxy-dist/lib/galaxy/web/framework/base.py/filename line132/line function__call__/function operationreturn self.handle_request( environ, start_response )/operationoperation_contextself.trace( message=quot;Starting requestquot; ) try:return self.handle_request( environ, start_response )finally: self.trace( message=quot;Handle request finishedquot; )/operation_context/frame frame modulegalaxy.web.framework.base/module filename/home/labinmunologiaulpgc/galaxy-dist/lib/galaxy/web/framework/base.py/filename line190/line functionhandle_request/function operationbody = method( trans, **kwargs )/operation operation_contextkwargs.pop( '_', None ) try: body = method( trans, **kwargs )except Exception, e: body = self.handle_controller_exception( e, trans, **kwargs )/operation_context/frame frame modulegalaxy.webapps.galaxy.controllers.library_common/module filename/home/labinmunologiaulpgc/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/library_common.py/filename line927/line functionupload_library_dataset/functionoperation**kwd )/operation operation_context widgets=widgets, replace_dataset=replace_dataset, **kwd ) if created_outputs_dict:if cntrller == 'api':/operation_context/frameframe modulegalaxy.webapps.galaxy.controllers.library_common/module filename/home/labinmunologiaulpgc/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/library_common.py/filename line1049/linefunctionupload_dataset/function operationstate = tool.new_state( trans )/operation operation_contexttool_id = 'upload1'tool =
Re: [galaxy-dev] can't install numpy from toolshed
On Aug 12, 2014, at 9:59 AM, Sajdak, Doris dj...@buffalo.edu wrote:
I’m running Galaxy in a virtual environment as described here (OS is Debian
3.2.54-2):
https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Use_a_clean_environment
When I go to install package_numpy_1_7 from the tool shed, I get the
following errors:
Running from numpy source directory.
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/config.py:413:
DeprecationWarning:
+
Usage of get_output is deprecated: please do not
use it anymore, and avoid configuration checks
involving running executable on the target machine.
+
DeprecationWarning)
Traceback (most recent call last):
File setup.py, line 214, in module
setup_package()
File setup.py, line 207, in setup_package
configuration=configuration )
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/core.py,
line 186, in setup
return old_setup(**new_attr)
File /usr/lib/python2.6/distutils/core.py, line 152, in setup
dist.run_commands()
File /usr/lib/python2.6/distutils/dist.py, line 975, in run_commands
self.run_command(cmd)
File /usr/lib/python2.6/distutils/dist.py, line 995, in run_command
cmd_obj.run()
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/install.py,
line 55, in run
r = old_install.run(self)
File /usr/lib/python2.6/distutils/command/install.py, line 615, in run
self.run_command('build')
File /usr/lib/python2.6/distutils/cmd.py, line 333, in run_command
self.distribution.run_command(command)
File /usr/lib/python2.6/distutils/dist.py, line 995, in run_command
cmd_obj.run()
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/build.py,
line 37, in run
old_build.run(self)
File /usr/lib/python2.6/distutils/command/build.py, line 135, in run
self.run_command(cmd_name)
File /usr/lib/python2.6/distutils/cmd.py, line 333, in run_command
self.distribution.run_command(command)
File /usr/lib/python2.6/distutils/dist.py, line 995, in run_command
cmd_obj.run()
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/build_src.py,
line 152, in run
self.build_sources()
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/build_src.py,
line 169, in build_sources
self.build_extension_sources(ext)
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/build_src.py,
line 328, in build_extension_sources
sources = self.generate_sources(sources, ext)
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/build_src.py,
line 385, in generate_sources
source = func(extension, build_dir)
File numpy/core/setup.py, line 410, in generate_config_h
moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
File numpy/core/setup.py, line 41, in check_types
out = check_types(*a, **kw)
File numpy/core/setup.py, line 271, in check_types
Cannot compile 'Python.h'. Perhaps you need to \
SystemError: Cannot compile 'Python.h'. Perhaps you need to install
python-dev|python-devel.
python-dev is installed on the machine, though I don’t know if that matters
in the virtual environment. Any idea what’s causing this?
Hi Dori,
Can you check that python2.6-dev is also installed? On Debian Wheezy,
python-dev installs the development files for Python 2.7, but your default
python appears to be Python 2.6.
That said, unless you are using 2.6 intentionally, I would suggest switching to
2.7. Although the Galaxy server and most Galaxy tools written in Python will
run in 2.6, some tools now require Python 2.7.
--nate
THANKS!
Dori
**
Dori Sajdak
Senior Systems Administrator
State University of NY at Buffalo
Center for Computational Research
701 Ellicott St
Buffalo, NY 14203
Phone: (716) 881-8934
Fax: (716) 849-6656
Web: http://ccr.buffalo.edu
**
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
Re: [galaxy-dev] can't install numpy from toolshed
On Aug 12, 2014, at 10:15 AM, Sajdak, Doris dj...@buffalo.edu wrote:
Hi Nate,
Thanks for the fast response. You were right, python2-6-dev wasn’t
installed. I’m all for using 2.7 and that’s what’s installed on the system
but I don’t know how to do that in the virtual environment. Can you tell me
how to switch that? Sorry if that’s a ridiculously basic question!
Hi Dori,
You can have virtualenv use a specified Python interpreter with the --python
argument to virtualenv(.py), by using the VIRTUALENV_PYTHON environment
varaible, or by calling virtualenv.py with the desired python interpreter (e.g.
`/usr/bin/python2.7 /path/to/virtualenv.py /path/to/venvdir`). See the
documentation for more:
https://virtualenv.pypa.io/en/latest/virtualenv.html#environment-variables-and-configuration-files
--nate
Thanks,
Dori
From: Nate Coraor [mailto:n...@bx.psu.edu]
Sent: Tuesday, August 12, 2014 10:08 AM
To: Sajdak, Doris
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] can't install numpy from toolshed
On Aug 12, 2014, at 9:59 AM, Sajdak, Doris dj...@buffalo.edu wrote:
I’m running Galaxy in a virtual environment as described here (OS is Debian
3.2.54-2):
https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Use_a_clean_environment
When I go to install package_numpy_1_7 from the tool shed, I get the
following errors:
Running from numpy source directory.
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/config.py:413:
DeprecationWarning:
+
Usage of get_output is deprecated: please do not
use it anymore, and avoid configuration checks
involving running executable on the target machine.
+
DeprecationWarning)
Traceback (most recent call last):
File setup.py, line 214, in module
setup_package()
File setup.py, line 207, in setup_package
configuration=configuration )
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/core.py,
line 186, in setup
return old_setup(**new_attr)
File /usr/lib/python2.6/distutils/core.py, line 152, in setup
dist.run_commands()
File /usr/lib/python2.6/distutils/dist.py, line 975, in run_commands
self.run_command(cmd)
File /usr/lib/python2.6/distutils/dist.py, line 995, in run_command
cmd_obj.run()
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/install.py,
line 55, in run
r = old_install.run(self)
File /usr/lib/python2.6/distutils/command/install.py, line 615, in run
self.run_command('build')
File /usr/lib/python2.6/distutils/cmd.py, line 333, in run_command
self.distribution.run_command(command)
File /usr/lib/python2.6/distutils/dist.py, line 995, in run_command
cmd_obj.run()
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/build.py,
line 37, in run
old_build.run(self)
File /usr/lib/python2.6/distutils/command/build.py, line 135, in run
self.run_command(cmd_name)
File /usr/lib/python2.6/distutils/cmd.py, line 333, in run_command
self.distribution.run_command(command)
File /usr/lib/python2.6/distutils/dist.py, line 995, in run_command
cmd_obj.run()
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/build_src.py,
line 152, in run
self.build_sources()
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/build_src.py,
line 169, in build_sources
self.build_extension_sources(ext)
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/build_src.py,
line 328, in build_extension_sources
sources = self.generate_sources(sources, ext)
File
/ifs/projects/galaxy/database/tmp/tmp-toolshed-mtdaP2Rhs/numpy-1.7.1/numpy/distutils/command/build_src.py,
line 385, in generate_sources
source = func(extension, build_dir)
File numpy/core/setup.py, line 410, in generate_config_h
moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
File numpy/core/setup.py, line 41, in check_types
out = check_types(*a, **kw)
File numpy/core/setup.py, line 271, in check_types
Cannot compile 'Python.h'. Perhaps you need to \
SystemError: Cannot compile 'Python.h'. Perhaps you need to install
python-dev|python-devel.
python-dev is installed on the machine, though I don’t know if that matters
in the virtual environment. Any idea what’s causing this?
Hi Dori,
Can you check that python2.6-dev is also installed? On Debian Wheezy,
python-dev installs the development files for Python 2.7, but your default
python appears to be Python 2.6.
That said, unless you are using 2.6 intentionally, I would suggest
[galaxy-dev] Galaxy Security Vulnerability
A security vulnerability was recently discovered by Inge Alexander Raknes that would allow a malicious person to execute arbitrary code on a Galaxy server. The vulnerability was in a method that uses Python pickle functionality to decode state information from tool forms. Because pickles can be used to instantiate arbitrary Python objects, tool states could be constructed to exploit this vulnerability. Because this vulnerability allows for arbitrary code execution, administrators are strongly encouraged to apply this fix IMMEDIATELY. We have tried to make it as easy and quick as possible for server administrators to update their Galaxy instances. The fix has been applied to every stable release from 2013.01.13 until the tip, so it is possible to get this fix on older releases without updating to a newer feature release. You can do this by identifying your current release and updating to the `latest_release_date` tag corresponding to your release. For example, if you are running release_2013.11.04 (or a subsequent commit to the stable branch of Galaxy between release_2013.11.04 and release_2014.02.10), you can update with: % hg pull % hg update latest_2013.11.04 For the changes to take effect, YOU MUST RESTART ALL GALAXY SERVER PROCESSES. If you do not want to pull any upstream changes, we have also created a standalone patch that fixes this problem, with multiple versions depending on your current Galaxy release: - pickle-2013.11.04.patch - This patch should apply cleanly (with offset/fuzz) to releases from 2013.11.04 up to the current stable tip. Available at: https://depot.galaxyproject.org/patch/pickle-2013.11.04.patch - pickle-2013.01.13.patch - This patch should apply cleanly (with offset/fuzz) to releases from 2013.01.13 up to 2013.08.12, and possibly older versions of Galaxy as well. Available at: https://depot.galaxyproject.org/patch/pickle-2013.01.13.patch If you happen to be running a very recent revision on the default branch or the newly created next-stable branch, a pickle-default.patch file exists at the same place. For older releases or instances with conflicting local modifications, manual application of the patch should not be difficult as it only includes a few small changes. To apply the patch, navigate to the root of your Galaxy directory, then run (replacing url_to_patch with the URL above that is correct for your release): % wget -O pickle.patch url_to_patch or: % curl -o pickle.patch url_to_patch and then: % patch -p1 pickle.patch patching file lib/galaxy/util/__init__.py Hunk #1 succeeded at 575 with fuzz 2 (offset -113 lines). patching file lib/galaxy/webapps/galaxy/controllers/ucsc_proxy.py Again, for the changes to take effect, YOU MUST RESTART ALL GALAXY SERVER PROCESSES. The Galaxy Team would like to extend special thanks to Inge Alexander Raknes and colleagues, who privately disclosed the vulnerability, with a full analysis and proof of concept. Credit for the fix and subsequent testing goes to my fellow Galaxy Team members John Chilton and Dannon Baker. On behalf of the Galaxy Team, --nate___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How to configure galaxy with a cluster
Hi Ben, Most of the tools you are looking for should be in the Galaxy Tool Shed: https://toolshed.g2.bx.psu.edu/ As for the earlier question regarding the resource selector drop down menu which I assume you are seeing on Galaxy Test, this is a feature currently available in the development version of Galaxy, and will be part of the next stable release: https://bitbucket.org/galaxy/galaxy-central/commits/31b2924315d0b48fa7648ab4bfd04ef754829f71 --nate On Jul 24, 2014, at 6:08 AM, 王渭巍 wangw...@inspur.com wrote: Thank you, Bjoern It turns out that I forgot start the scheduler, and now galaxy runs smoothly with torque. I am also trying to add more tools, some .xml config of the added tools can be found in tools, but some of them are missing, I don't know where are they. I want to try do some customizing to the web interface of the tools. Is there any guidline for me? Thanks a lot. Ciao, Ben From: Björn Grüning Date: 2014-07-22 16:15 To: 王渭巍; Björn Grüning; galaxy-dev Subject: Re: [galaxy-dev] How to configure galaxy with a cluster Hi Ben, if the job is in waiting in the queue it's unlikely (not impossible) that it is Galaxy fault. Can you recheck your Torque setup and how many cores and memory your job has requested? Ciao, Bjoern Am 22.07.2014 10:09, schrieb 王渭巍: Hi, Bjoern, I've tried the latest galaxy version with Torque 4.1.7, and it seems all right. But torque version 4.2 won't work. And I tried to submit“fastqc readqc” jobs via torque (runner pbs), but the job is always in the queue waiting. I submited “fastqc readqc”local (runner local) , and the job finished successfully. So the question is , it seems not all the tools can be submitted via torque (or other resource manager), right? 王渭巍 From: Björn Grüning Date: 2014-07-21 01:23 To: 王渭巍; Björn Grüning; galaxy-dev Subject: Re: [galaxy-dev] How to configure galaxy with a cluster Hi Ben, sorry but we do not run a Torque setup. Do you have any concrete questions or error messages? Cheers, Bjoern Am 17.07.2014 04:10, schrieb 王渭巍: Hi, Bjoern Would you share your procedure to make some tools to run on a cluster. I have tried https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster using Torque, but got errors. I think maybe it's job_conf.xml. Would you share yours? Thanks a lot Ben From: Björn Grüning Date: 2014-07-16 16:34 To: 王渭巍; Thomas Bellembois; galaxy-dev Subject: Re: [galaxy-dev] How to configure galaxy with a cluster Hi Ben, that is not possible at the moment. The idea is to keep the user-inferface as easy as possible for the user. You, as admin, can decide which resource a specific tool with a specific input will use. You will never see any options like that in a tool, but you can write a tool by yourself if you like, or enhance the megablast tool. Cheers, Bjoern Am 16.07.2014 09:43, schrieb 王渭巍: Thanks a lot, Thomas! It really helps, I added tools section followed your suggestion... here is my job_conf.xml ( I am using Torque, I have 3 servers. One for galaxy server, two for cluster computing. ) ?xml version=1.0? job_conf plugins plugin id=pbs type=runner load=galaxy.jobs.runners.pbs:PBSJobRunner/ /plugins destinations default=pbs_default destination id=pbs_default runner=pbs/ /destination destination id=long_jobs runner=pbs param id=Resource_Listwalltime=72:00:00,nodes=1:ppn=8/param param id=-p128/param /destination /destinations tools tool id=megablast_wrapper destination=long_jobs/ /tools /job_conf and still no cluster options in megablast item. How can I see cluster options in the page, for example, the page will let me choose to use local server or a cluster. Ben From: Thomas Bellembois Date: 2014-07-15 17:41 To: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] How to configure galaxy with a cluster Hello Ben, you can configure your Galaxy instance to use your cluster in the job_conf.xml file: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster You can set up your instance to use your cluster by default for all jobs or only for specific jobs. Here is a part of my job_conf.xml for example: plugins !-- LOCAL JOBS -- plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner workers=4/ !-- SUN GRID ENGINE -- plugin id=sge type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner/ /plugins handlers default=handlers handler id=handler0 tags=handlers/ handler id=handler1 tags=handlers/ /handlers destinations default=sge_default destination id=local runner=local/ destination id=sge_default runner=sge param
Re: [galaxy-dev] Providing BLAST db in a data library
On Jul 23, 2014, at 6:42 AM, Peter Cock p.j.a.c...@googlemail.com wrote: Interesting hypothesis - you may well be right. Galaxy guys - who is the expert to talk to on this and/or where in the code should we be looking? Thanks, Peter I think there's a bit of a mixup here - Peter, I believe you were asking if other composite types with an html primary dataset could be imported from the history to library, but Ulf, your test was the other direction (library-history). I'd be interested in knowing the outcome of the history-library test as well. I am woefully ignorant about the blastdbn datatype. Is the primary file supposed to be html type but empty? --nate On Wed, Jul 23, 2014 at 11:22 AM, Ulf Schaefer ulf.schae...@phe.gov.uk wrote: Dear Peter Thanks for your reply. I can import an html report (e.g. FastQC output) successfully into a new history from a data library. But the .dat file for the html is not empty like the one for the blastdb. Makes me think that I could do this with a blast db as well, if only it would not check for size 0 at the time of importing it. Thanks Ulf ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Run Jobs as Real User - How to Configure it for TORQUE
On Jul 22, 2014, at 2:13 PM, John Chilton jmchil...@gmail.com wrote: Running jobs as the real user is not available with the PBS job runner - one has to use the DRMAA interface to submit jobs as the real user. That said, the DRMAA runner plugin is compatible with Torque, you just need to compile and use pbs-drmaa, rather than Torque's libdrmaa: http://apps.man.poznan.pl/trac/pbs-drmaa --nate I have created a Trello card to add this functionality: https://trello.com/c/OddS8bMP Would be happy to field pull requests to add this - because I doubt anyone on the core team will be able to get to this anytime soon. Sorry I don't have better news. -John On Sun, Jul 20, 2014 at 11:26 PM, Ping Luo luop0...@gmail.com wrote: I have installed pbs_python module on our cluster to interface Galaxy with TORQUE. I am able to submit and run jobs on our cluster as the Galaxy user. I need to run Galaxy jobs as the real user. The instruction in the user guide is for DRMAA interface. How can I configure running jobs as real user for TORQUE? thank, Ping ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Delete an account (Galaxy local instance)
On Jul 7, 2014, at 6:11 AM, Pat-74100 leonardsqual...@hotmail.com wrote: I've found it: I've modified it in the file universe_wgsi.ini and now I can delete user by the admin tool panel. The problem is: when I delete the account and purge it why it still remain in the list ? is it possible to delete it completely ?? Hi Pat, Because Galaxy needs to track data provenance, we do not ever actually remove metadata such as this from the database. --nate From: leonardsqual...@hotmail.com To: galaxy-dev@lists.bx.psu.edu Date: Mon, 7 Jul 2014 09:49:48 + Subject: [galaxy-dev] Delete an account (Galaxy local instance) Dear Galaxy developers. Is it possible to delete an account (user not active anymore) on a Galaxy local instance as Admin ? Regards, Pat ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Object-Store, setting filetypes crashes Galaxy
Hi Björn,
I believe the problem is most likely this bug:
https://trello.com/c/OHlQoCrx
If you symlink /usr/local/galaxy/galaxy-dist/database/tmp to somewhere
writeable (e.g. what you have new_files_path set to), that'd confirm it.
I'm going to take a look at fixing this today.
--nate
On Thu, Jun 12, 2014 at 8:43 AM, bjoern.gruen...@googlemail.com
bjoern.gruen...@gmail.com wrote:
Hi,
I have configured to use the hierarchical object store but as soon as I
try to reset the filetpye of a dataset Galaxy is crashing with:
galaxy.objectstore DEBUG 2014-06-12 14:39:21,180 Using preferred backend
'files3' for creation of MetadataFile 5963
132.230.153.57 - - [12/Jun/2014:14:39:20 +0200] POST
/datasets/966f24627ef70c12/edit HTTP/1.1 500 -
http://galaxy.uni-freiburg.de/datasets/966f24627ef70c12/edit;
Mozilla/5.0 (X11; Linux x86_64; rv:29.0) Gecko/20100101 Fire
fox/29.0
Error - type 'exceptions.OSError': [Errno 2] No such file or directory:
'database/tmp/metadata_temp_file_1xnGcE'
URL: http://galaxy.uni-freiburg.de/datasets/966f24627ef70c12/edit
File
'/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py',
line 149 in __call__
app_iter = self.application(environ, sr_checker)
File
'/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
line 84 in __call__
return self.application(environ, start_response)
File
'/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
line 633 in __call__
return self.application(environ, start_response)
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py',
line 132 in __call__
return self.handle_request( environ, start_response )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py',
line 190 in handle_request
body = method( trans, **kwargs )
File
'/usr/local/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/dataset.py',
line 295 in edit
trans.app.datatypes_registry.set_external_metadata_tool.tool_action.execute(
trans.app.datatypes_registry.set_external_metadata_tool, trans, incoming =
{ 'input1':data }, overwrite = False ) #overwrite is False as per existi
ng behavior
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/actions/metadata.py',
line 18 in execute
overwrite, history, job_params )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/actions/metadata.py',
line 79 in execute_via_app
kwds = { 'overwrite' : overwrite } )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/datatypes/metadata.py',
line 717 in setup_external_metadata
shutil.copy( dataset.metadata.get( meta_key, None ).file_name,
metadata_temp.file_name )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/datatypes/metadata.py',
line 575 in file_name
self._filename = abspath( tempfile.NamedTemporaryFile( dir =
self.tmp_dir, prefix = metadata_temp_file_ ).name )
File '/usr/local/python/2.7/lib/python2.7/tempfile.py', line 454 in
NamedTemporaryFile
(fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
File '/usr/local/python/2.7/lib/python2.7/tempfile.py', line 235 in
_mkstemp_inner
fd = _os.open(file, flags, 0600)
OSError: [Errno 2] No such file or directory:
'database/tmp/metadata_temp_file_1xnGcE'
I have attached my object_store_conf.xml file.
Thanks,
Bjoern
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Old Tool Shed URL http://community.g2.bx.psu.edu/ dead
Hi all, This has been fixed. Sorry for the trouble, I am not sure when it disappeared. --nate On Tue, May 27, 2014 at 11:19 AM, Greg Von Kuster g...@bx.psu.edu wrote: Peter, sorry, this one is out of my hands. I'm hoping Nate can answer this when he gets a chance. Greg Von Kuster On May 27, 2014, at 11:16 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Tue, May 27, 2014 at 3:39 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi peter, It seems that we have stopped redirecting the old Tool Shed URL http://community.g2.bx.psu.edu/ to the new Tool Shed URL http://toolshed.g2.bx.psu.edu/ . I'm not sure when this happened. Sorry for the inconvenience. Greg Von Kuster Can the old domain be restored to fix pre-existing 3rd party links? There could be some in published papers, certainly there are on the mailing list archives. Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] User information when creating a new user
Hi Neil, The address fields can be populated from User - Preferences - Manage your information - User Addresses. You can also create custom forms - see the Manage form definitions link in the admin section. I believe it used to be possible to make custom User Information forms required at registration, but unfortunately this isn't working for me in testing, so it may have broken somewhere along the way (this feature is very rarely used). --nate On Wed, May 14, 2014 at 1:45 AM, neil.burd...@csiro.au wrote: Hi, when a new user creates a new account they only need an email address and password. This can be quite tricky to identify the user/demographics etc at a alter point. Is there a form that a user may complete where he could add optional data such as name, institution, address etc ? There is a table (user_address) that stores this information just not sure how/if it's populated Thanks Neil ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] drmaa
Hi Donny, You should be able to specify the queue using the nativeSpecification field of drmaa requests, e.g. in your job_conf.xml: destination id=batch runner=pbs_drmaa param id=nativeSpecification-q batch/param /destination Documentation on job_conf.xml's syntax by runner can be found here: https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster --nate On Tue, May 6, 2014 at 5:53 PM, Shrum, Donald C dcsh...@admin.fsu.eduwrote: Hi all, I've configured galaxy with the drama python module. This is really more of a drama question... We have no default queue set in moab and I can't seem to find a way to specify a queue in the docs I've been looking at here - http://drmaa-python.readthedocs.org/en/latest/tutorials.html I'd like to be able to specify a queue based on various pieces of logic in my destinations.py script. Any suggestions would be appreciated. Donny FSU Research Computing Center ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Help with cluster setup
On Wed, Apr 30, 2014 at 4:20 AM, 沈维燕 shenw...@gmail.com wrote: Hi Nate, From your previous email,Job deletion in the pbs runner will be fixed in the next stable release Galaxy.So whether this bug has been fixed in the version of Galaxy( https://bitbucket.org/galaxy/galaxy-dist/get/3b3365a39194.zip)?Thank you very much for your help. Regards,weiyan Hi Weiyan, Yes, this fix is included in the April, 2014 stable release. However, I would strongly encourage you to use `hg clone` rather than downloading a static tarball. There have been a number of patches to the stable branch since its April release. In addition, the tarball linked would pull from the default branch of Galaxy, which includes unstable changesets. --nate 2013-08-08 22:58 GMT+08:00 Nate Coraor n...@bx.psu.edu: On Aug 7, 2013, at 9:23 PM, shenwiyn wrote: Yes,and I also have the same confuse about that.Actually when I set server:id in the universe_wsgi.ini as follows for a try,my Galaxy doesn't work with Cluster,if I remove server:id,it work . Hi Shenwiyn, Are you starting all of the servers that you have defined in universe_wsgi.ini? If using run.sh, setting GALAXY_RUN_ALL in the environment will do this for you: http://wiki.galaxyproject.org/Admin/Config/Performance/Scaling [server:node01] use = egg:Paste#http port = 8080 host = 0.0.0.0 use_threadpool = true threadpool_workers = 5 This is my job_conf.xml : ?xml version=1.0? job_conf plugins workers=4 plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner workers=4/ plugin id=pbs type=runner load=galaxy.jobs.runners.pbs:PBSJobRunner workers=8/ /plugins handlers default=batch handler id=node01 tags=batch/ handler id=node02 tags=batch/ /handlers destinations default=regularjobs destination id=local runner=local/ destination id=regularjobs runner=pbs tags=cluster param id=Resource_Listwalltime=24:00:00,nodes=1:ppn=4,mem=10G/param param id=galaxy_external_runjob_scriptscripts/drmaa_external_runner.py/param param id=galaxy_external_killjob_scriptscripts/drmaa_external_killer.py/param param id=galaxy_external_chown_scriptscripts/external_chown_script.py/param /destination /destinations /job_conf The galaxy_external_* options are only supported with the drmaa plugin, and actually only belong in the univese_wsgi.ini for the moment, they have not been migrated to the new-style job configuration. They should also only be used if you are attempting to set up run jobs as the real user job running capabilities. Further more when I want to kill my jobs by clicking Catch(08-08-09-12-39).jpg in galaxy web,the job keeps on running in my background.I do not know how to fix this. Any help on this would be grateful.Thank you very much. Job deletion in the pbs runner was recently broken, but a fix for this bug will be part of the next stable release (on Monday). --nate shenwiyn From: Jurgens de Bruin Date: 2013-08-07 19:55 To: galaxy-dev Subject: [galaxy-dev] Help with cluster setup Hi, This is my first Galaxy installation setup so apologies for stupid questions. I am setting up Galaxy on a Cluster running Torque as the resource manager. I am working through the documentation but I am unclear on some things: Firstly I am unable to find : start_job_runners within the universe_wsgi.ini and I dont want to just add this anywhere - any help on this would be create. Further more this is my job_conf.xml : ?xml version=1.0? !-- A sample job config that explicitly configures job running the way it is configured by default (if there is no explicit config). -- job_conf plugins plugin id=hpc type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner workers=4/ /plugins handlers !-- Additional job handlers - the id should match the name of a [server:id] in universe_wsgi.ini. handler id=cn01/ handler id=cn02/ /handlers destinations destination id=hpc runner=drmaa/ /destinations /job_conf Does this look meaning full, further more where to I set the additional server:id in the universe_wsgi.ini. As background the cluster has 13 compute nodes and a shared storage array that can be accessed by all nodes in the cluster. Thanks again -- Regards/Groete/Mit freundlichen Grüßen/recuerdos/meilleures salutations/ distinti saluti/siong/duì yú/привет Jurgens de Bruin ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists
Re: [galaxy-dev] version of SAM-to-BAM (CloudMap)
Hi Xiaofei, You should be able to install 1.1.3 by selecting revision 3 of sam_to_bam from the Preview and install page (Admin - Search and browse tool sheds - Galaxy main tool shed - sam_to_bam - Preview and install), and then clicking Install to Galaxy. [image: Inline image 1] You can tell what version of the tool it'll install by looking at the Valid tools section of the Contents of this repository box. --nate On Thu, Apr 3, 2014 at 6:16 PM, Wang, Xiaofei xfw...@ku.edu wrote: Dear there, When I run CloudMap EMS Variant Density Mapping workflow (takes VCF of heterozygous and homozygous variants to subtract) (imported from uploaded file) on local Galaxy instance, I got this , when I wanted to edit the workflow. Also, I got this The following tools are beinge executed with a different version from what was available when this workflow was last saved because the previous version is no longer available for use on this galaxy instance. To upgrade your workflow and dismiss this message simply edit the workflow and re-save it to update the stored tool version. - toolshed.g2.bx.psu.edu/repos/devteam/sam_to_bam/sam_to_bam/1.1.2: using version '1.1.2' instead of version '1.1.3' indicated in this workflow. - ,when I wanted to run the workflow. I know it is the problem with version of SAM-to-BAM. But, when I searched the Tool sheds Search and browse tool sheds, there is only version 1.1.4 for SAM-to-BAM. When I tried to edit the workflow from Workflow ... Edit, I did not find where could I change the version of SAM-to-BAM. When I clicked on SAM-to-BAM on the Workflow Canvas, it showed the version is 1.1.3 for it. Could you tell me which version do I need exactly, and how to figure out this? Thanks a lot! Best, Xiaofei ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ inline: sam_to_bam_3.png___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Depth of Coverage (local galaxy __ CloudMap)
Hi Xiaofei, SQLite is not suitable for complex tasks like running workflows. Please switch to PostgreSQL: https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Switching_to_a_database_server --nate On Fri, Apr 4, 2014 at 1:35 PM, Wang, Xiaofei xfw...@ku.edu wrote: Dear there, I got this error when I run CloudMap EMS Variant Density Mapping workflow for Depth of Coverage on the local Galaxy instance on Mac. In fact, it works fine for Depth of Coverage when I run another workflow (Unmapped Mutant workflow) locally. I really appreciate you guys for helping me to figure out so many problems! Thanks a lot! Best, Xiaofei Traceback (most recent call last): File /Users/xiaofeiwang/softwares/galaxy-dist/lib/galaxy/jobs/runners/__init__.py, line 152, in prepare_job job_wrapper.prepare() File /Users/xiaofeiwang/softwares/galaxy-dist/lib/galaxy/jobs/__init__.py, line 652, in prepare if job.user is None and job.galaxy_session is None: File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/orm/attributes.py, line 168, in __get__ return self.impl.get(instance_state(instance),dict_) File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/orm/attributes.py, line 453, in get value = self.callable_(state, passive) File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/orm/strategies.py, line 508, in _load_for_state return self._emit_lazyload(session, state, ident_key) File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/orm/strategies.py, line 552, in _emit_lazyload return q._load_on_ident(ident_key) File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/orm/query.py, line 2512, in _load_on_ident return q.one() File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/orm/query.py, line 2184, in one ret = list(self) File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/orm/query.py, line 2227, in __iter__ return self._execute_and_instances(context) File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/orm/query.py, line 2242, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/engine/base.py, line 1449, in execute params) File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/engine/base.py, line 1584, in _execute_clauseelement compiled_sql, distilled_params File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/engine/base.py, line 1698, in _execute_context context) File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/engine/base.py, line 1691, in _execute_context context) File /Users/xiaofeiwang/softwares/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-macosx-10.6-intel-ucs2.egg/sqlalchemy/engine/default.py, line 331, in do_execute cursor.execute(statement, parameters) OperationalError: (OperationalError) database is locked u'SELECT galaxy_user.id AS galaxy_user_id, galaxy_user.create_time AS galaxy_user_create_time, galaxy_user.update_time AS galaxy_user_update_time, galaxy_user.email AS galaxy_user_email, galaxy_user.username AS galaxy_user_username, galaxy_user.password AS galaxy_user_password, galaxy_user.external AS galaxy_user_external, galaxy_user.form_values_id AS galaxy_user_form_values_id, galaxy_user.deleted AS galaxy_user_deleted, galaxy_user.purged AS galaxy_user_purged, galaxy_user.disk_usage AS galaxy_user_disk_usage, galaxy_user.active AS galaxy_user_active, galaxy_user.activation_token AS galaxy_user_activation_token \nFROM galaxy_user \nWHERE galaxy_user.id = ?' (1,) ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists
Re: [galaxy-dev] [CONTENT] Re: Re: Re: Re: Unable to remove old datasets
Hi Ravi, I believe admin_cleanup_datasets.py only works on database times. The rest of your assumptions are likely correct, although without looking at more details of the database I can't confirm. --nate On Fri, Mar 28, 2014 at 5:12 PM, Sanka, Ravi rsa...@jcvi.org wrote: Hi Nate, I checked and there are 3 rows of dataset 301 in the history_dataset_association table (none in library_dataset_dataset_association): dataset_id create_time update_time deleted 301 2/14/14 18:49 3/25/14 20:27 TRUE 301 3/6/14 15:48 3/25/14 18:41 TRUE 301 3/6/14 20:11 3/6/14 20:11 FALSE The one with the most recent create_time has its deleted status set to false. The other 2, older ones are true. I would have guessed that the most recent create_time instance is still false due being created within 30 days, but the second most recent is only 5 hours older and is set to true. Perhaps that instance was deleted by its user. That would cause its deleted status to become true, correct? I assume that if I were to wait until all 3 instances' create_times are past 30 days, my process will work, as admin_cleanup_datasets.py will set all 3 instances to false. Perchance, is there any setting on admin_cleanup_datasets.py that would cause it to judge datasets by their physical file's timestamp instead? -- Ravi Sanka ICS - Sr. Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 -- From: Nate Coraor n...@bx.psu.edu Date: Friday, March 28, 2014 1:56 PM To: Ravi Sanka rsa...@jcvi.org Cc: Carl Eberhard carlfeberh...@gmail.com, Peter Cock p.j.a.c...@googlemail.com, galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu Subject: [CONTENT] Re: Re: [galaxy-dev] Re: Re: Unable to remove old datasets Hi Ravi, Can you check whether any other history_dataset_association or library_dataset_dataset_association rows exist which reference the dataset_id that you are attempting to remove? When you run admin_cleanup_datasets.py, it'll set history_dataset_association.deleted = true. After that is done, you need to run cleanup_datasets.py with the `-6 -d 0` option to mark dataset.deleted = true, followed by `-3 -d 0 -r ` to remove the dataset file from disk and set dataset.purged = true. Note that the latter two operations will not do anything until *all* associated history_dataset_association and library_dataset_dataset_association rows are set to deleted = true. --nate On Fri, Mar 28, 2014 at 1:52 PM, Sanka, Ravi rsa...@jcvi.org wrote: Hi Nate, I checked the dataset's entry in history_dataset_association, and the value in field deleted is true. But if this does not enable the cleanup scripts to remove the dataset from disk, then how can I accomplish that? As an admin, my intention is to completely remove datasets that are past a certain age from Galaxy, including all instances of the dataset that may exist, regardless of whether or not the various users who own said instances have deleted them from their histories. Can this be done with admin_cleanup_datasets.py? If so, how? -- Ravi Sanka ICS - Sr. Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 -- From: Nate Coraor n...@bx.psu.edu Date: Friday, March 28, 2014 9:59 AM To: Ravi Sanka rsa...@jcvi.org Cc: Carl Eberhard carlfeberh...@gmail.com, Peter Cock p.j.a.c...@googlemail.com, galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu Subject: [CONTENT] Re: [galaxy-dev] Re: Re: Unable to remove old datasets Hi Ravi, If you take a look at the dataset's entry in the history_dataset_association table, is that marked deleted? admin_cleanup_datasets.py only marks history_dataset_association rows deleted, not datasets. Running the cleanup_datasets.py flow with -d 0 should have then caused the dataset to be deleted and purged, but this may not be the case if there is more than one instance of the dataset you are trying to purge (either another copy in a history somewhere, or in a library). --nate On Tue, Mar 25, 2014 at 5:12 PM, Sanka, Ravi rsa...@jcvi.org wrote: I have now been able to successfully remove datasets from disk. After deleting the dataset or history from the front-end interface (as the user), I then run the cleanup scripts as admin: python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -1 $@ ./scripts/cleanup_datasets/delete_userless_histories.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -2 -r $@ ./scripts/cleanup_datasets/purge_histories.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -3 -r $@ ./scripts/cleanup_datasets/purge_datasets.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -5 -r $@ ./scripts/cleanup_datasets/purge_folders.log
Re: [galaxy-dev] Galaxy packages for Bio-Linux - an update + Apache proxying
Hi Tim, I agree the single tool_data_table_conf.xml is a problem and it would be nice to implement functionality for multiple. I've added a Trello card for it here: https://trello.com/c/rm2p8PpZ As to the location file collections, I can foresee a few problems, e.g. conflicting definitions, bad formatting of the metadata in location file comments, and still needing to specify multiple paths where these location files can exist. --nate On Mon, Mar 31, 2014 at 8:00 AM, Tim Booth tbo...@ceh.ac.uk wrote: Hi Nate, It's great to hear that my rearrangements for packaging Galaxy are somewhat in line with what you are doing. My package is a bit of a rats-nest of symlinks right now but I'll clean these up as things become more configurable. One thing I'm a bit stuck on is the tool_data_table_conf.xml and shed_tool_data_table_conf.xml files. If the former is installed and updated by my package and the latter is managed by the internal tool-shed logic then this leaves nowhere for the user to make manual edits (we can use the debian config-files system to share ownership but it's awkward). Also I've made a galaxy-tools-bl package that adds some standard tool definitions in a file called bl_tool_conf.xml. I can set things up so this is seen by Galaxy: [app:main] ... tool_config_file = tool_conf.xml,shed_tool_conf.xml,bl_tool_conf.xml So I can split out tool_conf.xml but currently I can't split out items in tool_data_table_conf.xml (unless I make a startup script that compiles it on the fly from XML fragments, in the style of build_universe_config.py). Now if it were me I'd try and eliminate this XML file altogether, and have Galaxy just collect up all the available .loc files on startup without the need for a central registry file. This would involve embedding the couple of bits of metadata from the tool_data_table_conf.xml XML file into the .loc files themselves, and maybe there is some reason I've not spotted why this would be a bad idea. Any thoughts on this? Is it something you have considered or would consider? Cheers, TIM On Fri, 2014-03-28 at 18:49 +, Nate Coraor wrote: Hi Tim, I have recently been working on getting Galaxy Main's configs and server-modified files and directories out of the galaxy-dist directory, so our goals are aligning. Not everything can be moved without some trickery (e.g. symlinks) but most paths, including the paths to shed_*.xml are configurable in universe_wsgi.ini (which itself need not live in the galaxy-dist directory). --nate On Mon, Mar 24, 2014 at 1:38 PM, Tim Booth tbo...@ceh.ac.uk wrote: Hi, This is mainly a message to Carlos and Eric who volunteered to get involved with my Debian packaging, but of course any other help will also be appreciated. If either of you can spare time now it is a good moment to do so. I've been working on my package stuff over the last couple of weeks and have uploaded the latest to Debian-Med ready to build: http://anonscm.debian.org/viewvc/debian-med/trunk/packages/galaxy/trunk/debian/?view=log Note that you need the re-packed tarball as generated by get-orig-source.sh and if you have a problem generating that you can grab a copy of it here: https://launchpad.net/~nebc/+archive/galaxy/+files/galaxy_1.bl.py27.20140210.orig.tar.xz I've only built this for Ubuntu, and I know that to get it working on Debian you'll at least need to replace the upstart job with an /etc/init.d script. After that I think you should have something working (see my commit notes). My latest efforts have been to try and get tool-shed installs working. Galaxy expects to be able to write to its own shed_tools_*_conf.xml files as well as to shed_tools and the tool-data directory. It looks like there is work to have a separate shed_tool-data folder but this is not fully working so I'm seeing if I can patch it. Either way, it's vital for packaging that the files managed by DPKG and the files (over)written by the Galaxy server are separated out into /var and /usr respectively. Cheers, TIM On Mon, 2013-12-16 at 16:27 +, Carlos Borroto wrote: Hi Tim, This sounds great. I'll be happy to help testing and hopefully find some time to help packaging once it gets into Debian Med(are you submitting all your packages there?). One question, for apache/nginx configuration why not use something ala phpMyAdmin which ask you if you want to preconfigure the package
Re: [galaxy-dev] Problems submitting using PBS Pro 12.1
Hi Luca, Could you send your job_conf.xml please? Also, is there any chance that you modified run.sh to set DRMAA_LIBRARY_PATH there (which would override what you set on the command line)? --nate On Thu, Apr 3, 2014 at 9:17 AM, Luca Toldo lucato...@gmail.com wrote: Dear All, after having discovered the mess in which I was, and the incomplete PBS client installation, I've done the following: *1) PBS Pro* a) removed the incomplete pbs installation that I had b) downloaded the latest version from vendor c) installed it in //usr/pbs d) configured e) tested it -- worked well *2) drma-pbs* a) downloaded version 1.0.17 b) compiled it (./configure --with-pbs=/usr/pbs ) without any problem c) installed it in /usr/local/lib 3) modified DRMAA_LIBRARY_PATH=/usr/local/lib/libdrmaa.so 4) sh run.sh --stop 5) sh run.sh --daemon Unfortunately, however when I launch a new job I always get the error message galaxy.jobs DEBUG 2014-04-03 15:13:44,306 (261) Working directory for job is: /opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261 galaxy.jobs.handler DEBUG 2014-04-03 15:13:44,321 (261) Dispatching to drmaa runner galaxy.jobs DEBUG 2014-04-03 15:13:44,434 (261) Persisting job destination (destination id: pbs_default) galaxy.jobs.handler INFO 2014-04-03 15:13:44,474 (261) Job dispatched galaxy.tools.deps DEBUG 2014-04-03 15:13:44,659 Building dependency shell command for dependency 'clustalw2' galaxy.tools.deps WARNING 2014-04-03 15:13:44,660 Failed to resolve dependency on 'clustalw2', ignoring galaxy.jobs.runners.drmaa DEBUG 2014-04-03 15:13:45,050 (261) submitting file /opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/galaxy_261.sh galaxy.jobs.runners.drmaa DEBUG 2014-04-03 15:13:45,050 (261) command is: python /opt/ngs/bin/galaxy-dist/tools/rgenetics/rgClustalw.py -i /data/imgt.fasta -o /opt/ngs/bin/galaxy-dist/database/files/000/dataset_340.dat -s ALIGNED -l /opt/ngs/bin/galaxy-dist/database/files/000/dataset_341.dat -t Clustal_run -d DNA -f CLUSTAL; return_code=$?; cd /opt/ngs/bin/galaxy-dist; /opt/ngs/bin/galaxy-dist/set_metadata.sh ./database/files /opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261 . /opt/ngs/bin/galaxy-dist/universe_wsgi.ini /opt/ngs/bin/galaxy-dist/database/tmp/tmpq4Rmdl /opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/galaxy.json /opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/metadata_in_HistoryDatasetAssociation_347_GsBRYB,/opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/metadata_kwds_HistoryDatasetAssociation_347_dFD5t0,/opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/metadata_out_HistoryDatasetAssociation_347_k0zCP6,/opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/metadata_results_HistoryDatasetAssociation_347_4LVIFs,,/opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/metadata_override_HistoryDatasetAssociation_347_jKfp0r /opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/metadata_in_HistoryDatasetAssociation_348_Fyy0tJ,/opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/metadata_kwds_HistoryDatasetAssociation_348_9vWiO3,/opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/metadata_out_HistoryDatasetAssociation_348_HL3EIj,/opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/metadata_results_HistoryDatasetAssociation_348_s1gqxk,,/opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/metadata_override_HistoryDatasetAssociation_348_hyQkDF; sh -c exit $return_code galaxy.jobs.runners ERROR 2014-04-03 15:13:45,051 (261) Unhandled exception calling queue_job Traceback (most recent call last): File /opt/ngs/bin/galaxy-dist/lib/galaxy/jobs/runners/__init__.py, line 62, in run_next method(arg) File /opt/ngs/bin/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py, line 150, in queue_job external_job_id = self.ds.runJob(jt) File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/__init__.py, line 331, in runJob _h.c(_w.drmaa_run_job, jid, _ct.sizeof(jid), jobTemplate) File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/helpers.py, line 213, in c return f(*(args + (error_buffer, sizeof(error_buffer File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/errors.py, line 90, in error_check raise _ERRORS[code-1](code %s: %s % (code, error_buffer.value)) InvalidAttributeValueException: code 14: Illegal attribute or resource value: Illegal attribute or resource value If from the commandline I do qsub -q NGSq /opt/ngs/bin/galaxy-dist/database/job_working_directory/000/261/galaxy_261.sh then I get 1695145.servername ... What can I do now ? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
Re: [galaxy-dev] Problems submitting using PBS Pro 12.1
Hi Luca, Are you passing any params to the runner for inclusion as PBS submit parameters (param id=nativeSpecification in job_conf.xml)? --nate On Wed, Apr 2, 2014 at 12:59 PM, Luca Toldo lucato...@gmail.com wrote: Dear all, SUSE SLE 11 SP 1, with PBS Pro 12.1 client delivers galaxy.jobs.runners ERROR 2014-04-02 18:28:43,740 (249) Unhandled exception calling queue_job Traceback (most recent call last): File /opt/ngs/bin/galaxy-dist/lib/galaxy/jobs/runners/__init__.py, line 62, in run_next method(arg) File /opt/ngs/bin/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py, line 151, in queue_job external_job_id = self.ds.runJob(jt) File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/__init__.py, line 331, in runJob _h.c(_w.drmaa_run_job, jid, _ct.sizeof(jid), jobTemplate) File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/helpers.py, line 213, in c return f(*(args + (error_buffer, sizeof(error_buffer File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/errors.py, line 90, in error_check raise _ERRORS[code-1](code %s: %s % (code, error_buffer.value)) InvalidAttributeValueException: code 14: Illegal attribute or resource value: Illegal attribute or resource value If I do qsub -q NGSq /opt/ngs/bin/galaxy-dist/database/job_working_directory/000/249/galaxy_249.sh however the qsub returns 12345.myserver and the corresponding qstat | grep -i galaxy 12345.myserver galaxy_249.shgalaxy00:00:00 E NGSq Is the above error message perhaps related with a) PBS 12.1 b) drmaa-0.7-py2.6.egg I am trying to compile pbs-drmaa-1.0.17 but there are problems ... any advice ? any alternative ? Thanks luca ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Problems submitting using PBS Pro 12.1
Thanks Björn. Luca, what version of pbs-drmaa are you using now? On Wed, Apr 2, 2014 at 2:40 PM, Björn Grüning bjoern.gruen...@gmail.comwrote: Hi Nate, I tried to help Luca today, but I failed totally ... He is not using param id=nativeSpecification at all. Or put it that way if he isn't using it, he gets the same error. Cheers, Bjoern Am 02.04.2014 20:34, schrieb Nate Coraor: Hi Luca, Are you passing any params to the runner for inclusion as PBS submit parameters (param id=nativeSpecification in job_conf.xml)? --nate On Wed, Apr 2, 2014 at 12:59 PM, Luca Toldo lucato...@gmail.com wrote: Dear all, SUSE SLE 11 SP 1, with PBS Pro 12.1 client delivers galaxy.jobs.runners ERROR 2014-04-02 18:28:43,740 (249) Unhandled exception calling queue_job Traceback (most recent call last): File /opt/ngs/bin/galaxy-dist/lib/galaxy/jobs/runners/__init__.py, line 62, in run_next method(arg) File /opt/ngs/bin/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py, line 151, in queue_job external_job_id = self.ds.runJob(jt) File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/__init__.py, line 331, in runJob _h.c(_w.drmaa_run_job, jid, _ct.sizeof(jid), jobTemplate) File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/helpers.py, line 213, in c return f(*(args + (error_buffer, sizeof(error_buffer File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/errors.py, line 90, in error_check raise _ERRORS[code-1](code %s: %s % (code, error_buffer.value)) InvalidAttributeValueException: code 14: Illegal attribute or resource value: Illegal attribute or resource value If I do qsub -q NGSq /opt/ngs/bin/galaxy-dist/database/job_working_ directory/000/249/galaxy_249.sh however the qsub returns 12345.myserver and the corresponding qstat | grep -i galaxy 12345.myserver galaxy_249.shgalaxy00:00:00 E NGSq Is the above error message perhaps related with a) PBS 12.1 b) drmaa-0.7-py2.6.egg I am trying to compile pbs-drmaa-1.0.17 but there are problems ... any advice ? any alternative ? Thanks luca ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Problems submitting using PBS Pro 12.1
Hi Luca, We generally have not provided detailed documentation on this portion since it's not really within the scope of Galaxy itself, and documentation would be likely to become out of date as these external libraries (like pbs-drmaa) are updated, and cluster configurations are very site-specific. We are happy to help, though. It might be useful to add some debugging about the generated jobTemplate in drmaa.py to see what params are being set. The value you've set as the path to the drmaa library ($DRMAA_LIBRARY_PATH) might provide some insight on the version, or the output of: strings $DRMAA_LIBRARY_PATH | grep 1\.0 What are the problems you're experiencing compiling pbs-drmaa 1.0.17? --nate On Wed, Apr 2, 2014 at 2:52 PM, Luca Toldo lucato...@gmail.com wrote: Dear Nate, and Björn thankyou for your help. As Björn said, indeed i get always the same error message regardless if I use the id=nativeSpecification or not. How can one figure out the version of pbs-drmaa that the system is using ? It would be really useful to have a step-by-step tutorial on how to configure this part of the system. I've spent already several days trying to get it working and it constantly fails. Unfortunately I also got serious problem trying to compile the pbs-drmaa-1.0.17 as mentioned in my posting. What can I try next ? I was thinking to modify /jobs/runners/drmaa.py adding more logging ... Thankyou for your advice ! 2014-04-02 20:43 GMT+02:00 Nate Coraor n...@bx.psu.edu: Thanks Björn. Luca, what version of pbs-drmaa are you using now? On Wed, Apr 2, 2014 at 2:40 PM, Björn Grüning bjoern.gruen...@gmail.comwrote: Hi Nate, I tried to help Luca today, but I failed totally ... He is not using param id=nativeSpecification at all. Or put it that way if he isn't using it, he gets the same error. Cheers, Bjoern Am 02.04.2014 20:34, schrieb Nate Coraor: Hi Luca, Are you passing any params to the runner for inclusion as PBS submit parameters (param id=nativeSpecification in job_conf.xml)? --nate On Wed, Apr 2, 2014 at 12:59 PM, Luca Toldo lucato...@gmail.com wrote: Dear all, SUSE SLE 11 SP 1, with PBS Pro 12.1 client delivers galaxy.jobs.runners ERROR 2014-04-02 18:28:43,740 (249) Unhandled exception calling queue_job Traceback (most recent call last): File /opt/ngs/bin/galaxy-dist/lib/galaxy/jobs/runners/__init__. py, line 62, in run_next method(arg) File /opt/ngs/bin/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py, line 151, in queue_job external_job_id = self.ds.runJob(jt) File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/__init__.py, line 331, in runJob _h.c(_w.drmaa_run_job, jid, _ct.sizeof(jid), jobTemplate) File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/helpers.py, line 213, in c return f(*(args + (error_buffer, sizeof(error_buffer File /opt/ngs/bin/galaxy-dist/eggs/drmaa-0.6-py2.6.egg/drmaa/errors.py, line 90, in error_check raise _ERRORS[code-1](code %s: %s % (code, error_buffer.value)) InvalidAttributeValueException: code 14: Illegal attribute or resource value: Illegal attribute or resource value If I do qsub -q NGSq /opt/ngs/bin/galaxy-dist/database/job_working_ directory/000/249/galaxy_249.sh however the qsub returns 12345.myserver and the corresponding qstat | grep -i galaxy 12345.myserver galaxy_249.shgalaxy00:00:00 E NGSq Is the above error message perhaps related with a) PBS 12.1 b) drmaa-0.7-py2.6.egg I am trying to compile pbs-drmaa-1.0.17 but there are problems ... any advice ? any alternative ? Thanks luca ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [CONTENT] Re: Re: Unable to remove old datasets
Hi Ravi, If you take a look at the dataset's entry in the history_dataset_association table, is that marked deleted? admin_cleanup_datasets.py only marks history_dataset_association rows deleted, not datasets. Running the cleanup_datasets.py flow with -d 0 should have then caused the dataset to be deleted and purged, but this may not be the case if there is more than one instance of the dataset you are trying to purge (either another copy in a history somewhere, or in a library). --nate On Tue, Mar 25, 2014 at 5:12 PM, Sanka, Ravi rsa...@jcvi.org wrote: I have now been able to successfully remove datasets from disk. After deleting the dataset or history from the front-end interface (as the user), I then run the cleanup scripts as admin: python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -1 $@ ./scripts/cleanup_datasets/delete_userless_histories.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -2 -r $@ ./scripts/cleanup_datasets/purge_histories.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -3 -r $@ ./scripts/cleanup_datasets/purge_datasets.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -5 -r $@ ./scripts/cleanup_datasets/purge_folders.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -4 -r $@ ./scripts/cleanup_datasets/purge_libraries.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -6 -r $@ ./scripts/cleanup_datasets/delete_datasets.log However, my final goal is to have a process that can remove old datasets from disk regardless of whether or not the users have deleted them at the front-end (and then automate said process via cronjob). This will be essentially in a situation where users are likely to leave datasets unattended and accumulating disk space. I found the following Galaxy thread: http://dev.list.galaxyproject.org/Re-Improving-Administrative-Data-Clean-Up-pgcleanup-py-vs-cleanup-datasets-py-td4659330.html And am trying to use the script it mentions: python ./scripts/cleanup_datasets/admin_cleanup_datasets.py universe_wsgi.ini -d 30 --smtp smtp server --fromaddr rsa...@jcvi.org I chose -d 30 to remove all datasets older than 30 days, which currently only targets one dataset. The resulting stdout indicates success: # 2014-03-25 16:27:47 - Handling stuff older than 30 days Marked HistoryDatasetAssociation id 301 as deleted From: rsa...@jcvi.org To: isi...@jcvi.org Subject: Galaxy Server Cleanup - 1 datasets DELETED -- Galaxy Server Cleanup - The following datasets you own on Galaxy are older than 30 days and have been DELETED: Small.fastq in history Unnamed history You may be able to undelete them by logging into Galaxy, navigating to the appropriate history, selecting Include Deleted Datasets from the history options menu, and clicking on the link to undelete each dataset that you want to keep. You can then download the datasets. Thank you for your understanding and cooporation in this necessary cleanup in order to keep the Galaxy resource available. Please don't hesitate to contact us if you have any questions. -- Galaxy Administrators Marked 1 dataset instances as deleted But when I check the database, the status of dataset 301 is unchanged (ok-false-false-true). I then run the same cleanup_datasets.py routine from above (but with -d 30), but dataset 301 is still present. I tried a second time, this time using -d 0, but still no deletion (which is not surprising since the dataset's deleted status is still false). If I run admin_cleanup_datasets.py again with the same parameters, the stdout says no datasets matched the criteria, so it seems to remember it's previous execution, but it's NOT actually updating the database. What am I doing wrong? -- Ravi Sanka ICS - Sr. Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 -- From: Carl Eberhard carlfeberh...@gmail.com Date: Tuesday, March 18, 2014 2:09 PM To: Peter Cock p.j.a.c...@googlemail.com Cc: Ravi Sanka rsa...@jcvi.org, galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu Subject: [CONTENT] Re: [galaxy-dev] Re: Unable to remove old datasets The cleanup scripts enforce a sort of lifetime for the datasets. The first time they're run, they may mark a dataset as deleted and also reset the update time and you'll have to wait N days for the next stage of the lifetime. The next time they're run, or if a dataset has already been marked as deleted, the actual file removal happens and purged is set to true (if it wasn't already). You can manually pass in '-d 0' to force removal of datasets recently marked as deleted. The purge scripts do not check 'allow_user_dataset_purge', of course. On Tue, Mar
Re: [galaxy-dev] [CONTENT] Re: Re: Re: Unable to remove old datasets
Hi Ravi, Can you check whether any other history_dataset_association or library_dataset_dataset_association rows exist which reference the dataset_id that you are attempting to remove? When you run admin_cleanup_datasets.py, it'll set history_dataset_association.deleted = true. After that is done, you need to run cleanup_datasets.py with the `-6 -d 0` option to mark dataset.deleted = true, followed by `-3 -d 0 -r ` to remove the dataset file from disk and set dataset.purged = true. Note that the latter two operations will not do anything until *all* associated history_dataset_association and library_dataset_dataset_association rows are set to deleted = true. --nate On Fri, Mar 28, 2014 at 1:52 PM, Sanka, Ravi rsa...@jcvi.org wrote: Hi Nate, I checked the dataset's entry in history_dataset_association, and the value in field deleted is true. But if this does not enable the cleanup scripts to remove the dataset from disk, then how can I accomplish that? As an admin, my intention is to completely remove datasets that are past a certain age from Galaxy, including all instances of the dataset that may exist, regardless of whether or not the various users who own said instances have deleted them from their histories. Can this be done with admin_cleanup_datasets.py? If so, how? -- Ravi Sanka ICS - Sr. Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 -- From: Nate Coraor n...@bx.psu.edu Date: Friday, March 28, 2014 9:59 AM To: Ravi Sanka rsa...@jcvi.org Cc: Carl Eberhard carlfeberh...@gmail.com, Peter Cock p.j.a.c...@googlemail.com, galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu Subject: [CONTENT] Re: [galaxy-dev] Re: Re: Unable to remove old datasets Hi Ravi, If you take a look at the dataset's entry in the history_dataset_association table, is that marked deleted? admin_cleanup_datasets.py only marks history_dataset_association rows deleted, not datasets. Running the cleanup_datasets.py flow with -d 0 should have then caused the dataset to be deleted and purged, but this may not be the case if there is more than one instance of the dataset you are trying to purge (either another copy in a history somewhere, or in a library). --nate On Tue, Mar 25, 2014 at 5:12 PM, Sanka, Ravi rsa...@jcvi.org wrote: I have now been able to successfully remove datasets from disk. After deleting the dataset or history from the front-end interface (as the user), I then run the cleanup scripts as admin: python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -1 $@ ./scripts/cleanup_datasets/delete_userless_histories.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -2 -r $@ ./scripts/cleanup_datasets/purge_histories.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -3 -r $@ ./scripts/cleanup_datasets/purge_datasets.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -5 -r $@ ./scripts/cleanup_datasets/purge_folders.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -4 -r $@ ./scripts/cleanup_datasets/purge_libraries.log python ./scripts/cleanup_datasets/cleanup_datasets.py ./universe_wsgi.ini -d 0 -6 -r $@ ./scripts/cleanup_datasets/delete_datasets.log However, my final goal is to have a process that can remove old datasets from disk regardless of whether or not the users have deleted them at the front-end (and then automate said process via cronjob). This will be essentially in a situation where users are likely to leave datasets unattended and accumulating disk space. I found the following Galaxy thread: http://dev.list.galaxyproject.org/Re-Improving-Administrative-Data-Clean-Up-pgcleanup-py-vs-cleanup-datasets-py-td4659330.html And am trying to use the script it mentions: python ./scripts/cleanup_datasets/admin_cleanup_datasets.py universe_wsgi.ini -d 30 --smtp smtp server --fromaddr rsa...@jcvi.org I chose -d 30 to remove all datasets older than 30 days, which currently only targets one dataset. The resulting stdout indicates success: # 2014-03-25 16:27:47 - Handling stuff older than 30 days Marked HistoryDatasetAssociation id 301 as deleted From: rsa...@jcvi.org To: isi...@jcvi.org Subject: Galaxy Server Cleanup - 1 datasets DELETED -- Galaxy Server Cleanup - The following datasets you own on Galaxy are older than 30 days and have been DELETED: Small.fastq in history Unnamed history You may be able to undelete them by logging into Galaxy, navigating to the appropriate history, selecting Include Deleted Datasets from the history options menu, and clicking on the link to undelete each dataset that you want to keep. You can then download the datasets. Thank you for your understanding and cooporation in this necessary cleanup
Re: [galaxy-dev] restarting Galaxy without affecting jobs
Hi David, Setting track_jobs_in_database = True should not be required, recovery is supposed to work either way. Does Galaxy lose all jobs, or just the ones that completed while Galaxy was restarting? Can you provide the output from the Galaxy log that shows an attempt to recover a job and all related messages? Thanks, --nate On Mon, Mar 24, 2014 at 11:13 AM, David Hoover hoove...@helix.nih.govwrote: What are the configuration steps required for allowing a local Galaxy installation to be restarted without affecting currently running jobs? I have Galaxy using DRMAA to submit jobs onto a backend cluster. I thought that enable_job_recovery = True should allow this, but in a few tests I have found that although the batch jobs completed, Galaxy lost track of the jobs and classified them as failed. Would track_jobs_in_database = True be required? This is currently set to the default 'None'. Our local Galaxy installation has become quite busy, and restarts are not possible without forcing users to restart their jobs. David Hoover Helix Systems Staff ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy packages for Bio-Linux - an update + Apache proxying
Hi Tim, I have recently been working on getting Galaxy Main's configs and server-modified files and directories out of the galaxy-dist directory, so our goals are aligning. Not everything can be moved without some trickery (e.g. symlinks) but most paths, including the paths to shed_*.xml are configurable in universe_wsgi.ini (which itself need not live in the galaxy-dist directory). --nate On Mon, Mar 24, 2014 at 1:38 PM, Tim Booth tbo...@ceh.ac.uk wrote: Hi, This is mainly a message to Carlos and Eric who volunteered to get involved with my Debian packaging, but of course any other help will also be appreciated. If either of you can spare time now it is a good moment to do so. I've been working on my package stuff over the last couple of weeks and have uploaded the latest to Debian-Med ready to build: http://anonscm.debian.org/viewvc/debian-med/trunk/packages/galaxy/trunk/debian/?view=log Note that you need the re-packed tarball as generated by get-orig-source.sh and if you have a problem generating that you can grab a copy of it here: https://launchpad.net/~nebc/+archive/galaxy/+files/galaxy_1.bl.py27.20140210.orig.tar.xz I've only built this for Ubuntu, and I know that to get it working on Debian you'll at least need to replace the upstart job with an /etc/init.d script. After that I think you should have something working (see my commit notes). My latest efforts have been to try and get tool-shed installs working. Galaxy expects to be able to write to its own shed_tools_*_conf.xml files as well as to shed_tools and the tool-data directory. It looks like there is work to have a separate shed_tool-data folder but this is not fully working so I'm seeing if I can patch it. Either way, it's vital for packaging that the files managed by DPKG and the files (over)written by the Galaxy server are separated out into /var and /usr respectively. Cheers, TIM On Mon, 2013-12-16 at 16:27 +, Carlos Borroto wrote: Hi Tim, This sounds great. I'll be happy to help testing and hopefully find some time to help packaging once it gets into Debian Med(are you submitting all your packages there?). One question, for apache/nginx configuration why not use something ala phpMyAdmin which ask you if you want to preconfigure the package with a webserver in particular. The name of the DEB packaging technology to ask these kind of questions is evading me now. I think using something like that could open many possibilities in the future, like database backend to use, home URL, admin user/password, etc... Thanks for your work on this, Carlos On Fri, Dec 13, 2013 at 7:03 AM, Tim Booth tbo...@ceh.ac.uk wrote: Hi All, As previously mentioned, I'm back working on packaging the Galaxy server as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and ultimately pushing towards something that could be Debian compliant. There's a way to go in that regard, but I do now have an updated package for Bio-Linux in final testing and it also has a new trick: doing apt-get install galaxy-server-apache-proxy will set up just that with no further configuration needed. The galaxy server appears at http://localhost/galaxy and users log in with their regular system username and password. Uploads are enabled via regular SFTP so no special FTP server configuration is needed. It's a little hacky in parts but I'm generally pleased with the result. If anyone want to take a look I'd welcome comments. It's not in the main BL repo yet but can be found here: https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra Cheers, TIM -- Tim Booth tbo...@ceh.ac.uk NERC Environmental Bioinformatics Centre Centre for Ecology and Hydrology Maclean Bldg, Benson Lane Crowmarsh Gifford Wallingford, England OX10 8BB http://nebc.nerc.ac.uk +44 1491 69 2705 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Tim Booth tbo...@ceh.ac.uk NERC Environmental Bioinformatics Centre Centre for Ecology and Hydrology Maclean Bldg, Benson Lane Crowmarsh Gifford Wallingford, England OX10 8BB http://nebc.nerc.ac.uk +44 1491 69 2705 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] homebrew python
Hi Joshua, You may be able to trick Galaxy into using existing versions of OS X eggs, they are built for both 32 and 64-bit Intel, but should work fine with a single-arch build. If the attached patch works, let me know and I'll commit it. If you'd rather not mess with the Galaxy source, you should be able to build the missing eggs using `python ./scripts/scramble.py -e egg_package_name`. Usually, the fetch_eggs.py script will inform you of this - is it not doing so in your case? --nate On Mon, Mar 24, 2014 at 9:54 PM, Joshua Udall jaud...@gmail.com wrote: For various reasons, I installed a Homebrew of python instead of the system version on OSX 10.9.2. Now, when galaxy initializes, it isn't looking in the right location for eggs (or they aren't placed in the right spot on my system). I was able to manually install several of the eggs and the galaxy startup would move to the next egg until here. Some eggs are out of date, attempting to fetch... Warning: MarkupSafe (a dependent egg of WebHelpers) cannot be fetched Traceback (most recent call last): File ./scripts/fetch_eggs.py, line 37, in module c.resolve() # Only fetch eggs required by the config File /Users/galaxy/galaxy-old3/lib/galaxy/eggs/__init__.py, line 345, in resolve egg.resolve() File /Users/galaxy/galaxy-old3/lib/galaxy/eggs/__init__.py, line 195, in resolve return self.version_conflict( e.args[0], e.args[1] ) File /Users/galaxy/galaxy-old3/lib/galaxy/eggs/__init__.py, line 226, in version_conflict r = pkg_resources.working_set.resolve( ( dist.as_requirement(), ), env, egg.fetch ) File build/bdist.macosx-10.9-x86_64/egg/pkg_resources.py, line 588, in resolve The `plugin_env` should be an ``Environment`` instance that contains pkg_resources.DistributionNotFound: numpy==1.6.0 Fetch failed. I know numpy, WebHelpers, MarkupSafe are installed and they are current (maybe too current?) ... what would be a good way to resolve the conflict? The manual download below fails on a dozen packages or so. sudo python ./scripts/make_egg_packager.py py2.7-macosx-10.9-x86_64-ucs2 Using Python interpreter at /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python, Version 2.7.6 (default, Mar 24 2014, 14:39:40) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.38)] This platform is 'py2.7-macosx-10.9-x86_64-ucs2' Override with: make_egg_packager.py forced-platform Completed packager is 'egg_packager-py2.7-macosx-10.9-x86_64-ucs2.py'. To fetch eggs, please copy this file to a system with internet access and run with: python egg_packager-py2.7-macosx-10.9-x86_64-ucs2.py -- Joshua Udall Assistant Professor 295 WIDB Plant and Wildlife Science Dept. Brigham Young University Provo, UT 84602 801-422-9307 Fax: 801-422-0008 USA ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ osx_intel_platform.patch Description: Binary data ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Question concerning the xml file for local tools with multiple output files.
Hi Lifeng, Another option would be the 'from_work_dir' option to the data tag. Have a look at the tophat repository in the Tool Shed for an example: http://toolshed.g2.bx.psu.edu/view/devteam/tophat --nate On Tue, Mar 25, 2014 at 11:29 AM, Hans-Rudolf Hotz hansrudolf.h...@fmi.chwrote: Hi Lifeng I am glad to hear it works WRT your thoughts about using a wrapper script for each tool: I agree it might help you to standardize your tools, however it also introduces an extra step, which needs to be taken care of if you want to change/upgrade your tool. Personally, I would only use a wrapper if it is necessary or it adds some benefits for the tool. Others on the list might have a different opinion? Hans-Rudolf On Mar 25, 2014, at 3:13 PM, Lifeng Lin wrote: works like a charm. Thank you! I start to wonder if i should use this wrapper approach on all scripts regardless of the original input-output format for standardized integration, and possible automated integrations in the future. On Tue, Mar 25, 2014 at 6:53 AM, Hans-Rudolf Hotz hansrudolf.h...@fmi.ch wrote: Hi Lifeng The easiest way to execute your script will be to provide a wrapper script (written in your preferred language, eg Python, perl, etc). Call the wrapper script like this: commandwrapper $input $output1 $output2/command and define $output1 $output2 according to your needs: format=fasta, format=txt the wrapper will call your script and rename/move the output. Hope this helps, Regards, Hans-Rudolf On Mar 25, 2014, at 3:23 AM, galaxy-user-boun...@lists.bx.psu.edu galaxy-user-boun...@lists.bx.psu.edu wrote: From: Lifeng Lin linlif...@gmail.com Date: March 25, 2014 12:58:52 AM GMT+01:00 To: galaxy-u...@lists.bx.psu.edu Subject: Question concerning the xml file for local tools with multiple output files. Hi folks, I am trying to integrate some of my local Perl scripts into a downloaded instance of Galaxy. So far the script with a simple in_file to out_file format worked great, but I am having problems understanding how to treat scripts with multiple output files that share the same input argv. for example: the script run like this in command line: script name input_name output_name_base and two files are generated from this script: output_name_base.fasta and output_name_base.txt. I am at a loss of how these parameter should be represented in the xml format, especially how the outputs data name= tag should be filled, since in the command tag, there is only one $output. Any suggestions? thanks! #GalaxyNoobie ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] jobs stuck in new state
Hi David, This is pretty common in the case of workflows. When a workflow step fails, the next job in the workflow will be set to the paused state and all jobs downstream of the paused job will remain in the new state until corrective action is taken. The current query for finding jobs-ready-to-run (if tracking jobs in the database, which is automatically enabled for multiprocess Galaxy configurations) ignores 'new' state jobs whose inputs are not ready, so these jobs sitting around should not cause any harm. --nate On Wed, Mar 26, 2014 at 12:25 PM, David Hoover hoove...@helix.nih.govwrote: I have many jobs stuck in the 'new' state on our local Galaxy instance. The jobs can't be stopped using the Admin-Manage jobs tool. First, does anyone know why a job would get stuck in the 'new' state for weeks? I have cleaned things up by manually setting their states to 'error' in the MySQL database. Is there a better way of dealing with 'new' jobs? BTW, our Galaxy instance was updated about two weeks ago. Wondering, David Hoover Helix Systems Staff ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] ERROR executing tool
On Thu, Mar 27, 2014 at 7:13 AM, virginia dalla via virdalla...@hotmail.com
wrote:
Hi,
I tried to GROOMER my fastq data in fastq data, and galaxy did not allowed
me:Error executing tool: objectstore, __call_method failed: get_filename on
, kwargs: {}
could you please help me?
Thank you
Hi,
If you're using the Galaxy Main server at usegalaxy.org, could you use the
green bug icon to report this directly through Galaxy?
Thanks,
--nate
viR
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] How to install bowtie2 tool in galaxy
On Wed, Mar 5, 2014 at 8:18 PM, Ravi Alla ravi.a...@berkeley.edu wrote: Hi Jennifer, Thank you for this information. I was able to troubleshoot the bowtie2_indices. Like Bjoern said the bowtie2 tool needed to be reinstalled because the tool_table does not load the correct indices. I am still trying to wrap my head around the different toolsheds. The main tool shed resides in the galaxy-dist folder and uses tool_conf.xml to load tools into the side bar and tool_data_table_conf.xml to load indices (and other data) for the default tools that come with galaxy. Hi Ravi, The tools in the galaxy-dist directory and controlled via tool_conf.xml are not a part of the tool shed. The inclusion of tools directly in Galaxy predates the existence of the tool shed. Other than that, what you have above is correct. The shed tools reside in the ../shed_tools/ directory and use shed_tools_conf.xml to load tools into the side bar and the shed_tools_data_table_conf.xml to load indices. The shed_tool xml is setup in the universe.ini file. Right. The .loc files for default main tools reside in galaxy-dist/tool-data and the .loc files for the shed_tools reside in ../shed_tools/.../tool-name/tool_id/tool-name/ By default they'll all be under galaxy-dist/tool-data/. The versions in ../shed_tools are the sample location files provided by tool authors. And when I uninstall tools and reinstall I would have to update the metadata for that tool. I am not sure what you mean by this. If ../shed_tools dir is not present then shed-tools get installed under galaxy-dist/tool-data/toolshed.g2.bx.psu.edu/ ../shed_tools should be created if it does not exist. Tools won't be installed in galaxy-dist/tool-data/ And it is always better to install tools as wrappers + dependencies when possible. I agree. --nate Am I on the right track with this tool organization? Thanks Ravi On Mar 5, 2014, at 11:06 AM, Jennifer Jackson j...@bx.psu.edu wrote: Hi Ravi, The directory structure for the installation of ToolShed tools changed, which is why you have three directories. You perhaps had bowtie2 installed once before, then reinstalled (without completely removing the older version and associated data)? Or updated without resetting the metadata? In either case, the ../shed_tools directory is the new one. Having this as the path in your .xml configuration files (as Bjoern suggested earlier) and moving all contents data to be under the same location will be the simplest global solution ongoing. Links near the end of my reply can help explain how-to. For the specific reason why bowtie2 indices are not working: I noticed that the reference genome .fa file is not linked from the directory containing the indexes. This is required. Adding it in, the same way that you did for the bowtie2 indexes, into this dir: /global/referenceData/databases/bowtie2/hg19 will probably solve that part of the problem. I didn't see this posted - but I apologize if I am duplicating advice already given. I also tend to advise keeping all data under the same master data directory - all indexes and sequence data - as symbolic links to additional file system paths that are unknown to the 'galaxy user' cause a different set of problems. However, that said, this doesn't seem to be an issue in your specific case: if the bowtie2 indexes are functioning correctly - then environment is set up so that the other dir hierarchy where the .fa files are kept must included in the 'galaxy' user's ENV. Symbolic links that go outside of the local dir structure are known to cause problems unless the ENV config is carefully set up, and to my knowledge are best avoided entirely for certain uses such as upload by file path into libraries. For reference: NGS data set-up is described in this wiki - including expected content for each type of index: https://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup For examples, you can rsync a genome or two and examine the contents. Or, our /location dir and have a look at the .loc files. https://wiki.galaxyproject.org/Admin/DataIntegration Tool Shed help (very detailed): https://wiki.galaxyproject.org/Tool%20Shed In particular, if you had previously installed repositories (this is not clear, just suspected from the duplications), updating the Metadata with certain distribution updates can be very important. This has been necessary for the last few releases to update to changes. The News Brief noted this, and included a link to this wiki page. Also see the Related Pages lower down on the wiki. https://wiki.galaxyproject.org/ResettingMetadataForInstalledRepositories This may also be useful: https://wiki.galaxyproject.org/RepairingInstalledRepositories Hopefully you have sorted most of this out by now, or this helps! Jen Galaxy team On 3/4/14 12:21 PM, Ravi Alla wrote: Bjoern, This is getting frustrating. There are three places bowtie2_indices.loc file is
Re: [galaxy-dev] Tool Shed install best practise : Precompiled binaries vs local compile
Hi Peter, I just wanted to add my $0.02 USD to say that I mostly agree with this - I have long used binaries precompiled by the tool author on Main, especially for cases where, as you say, the compile-time dependency list is large and painful. The only gotcha here is to make sure that binaries support the oldest possible version of glibc that might be installed on any of the Linux distributions and versions that we support, and that no non-standard preinstalled libraries have been pulled in during the build process. --nate On Thu, Mar 6, 2014 at 12:46 PM, Peter Cock p.j.a.c...@googlemail.com wrote: Hi Greg, I've retitled the thread, previously about a ToolShed nightly test failure. A brief recap, we're talking about the Galaxy ToolShed XML installation recipes for the NCBI BLAST+ packages and my MIRA4 wrapper in their tool_dependencies.xml files: http://toolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_29 http://testtoolshed.g2.bx.psu.edu/view/iuc/package_blast_plus_2_2_29 http://testtoolshed.g2.bx.psu.edu/view/peterjc/mira4_assembler These use the pattern of having os/arch specific action tags (which download and install the tool author's precompiled binaries) and a fall back default action which is to report an error with the os/arch combination and that there are no ready made binaries available. Greg is instead advocating the fall back action be to download the source code, and do a local compile. My reply is below... On Thu, Mar 6, 2014 at 5:24 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Mar 6, 2014 at 4:53 PM, Greg Von Kuster g...@bx.psu.edu wrote: As we briefly discussed earlier, your mira4 recipe is not currently following best practices. Although you uncovered a problem in the framework which has now been corrected, your recipe's fall back actions tag set should be the recipe for installing mira4 from source ( http://sourceforge.net/projects/mira-assembler/ ) since there is no licensing issues for doing so. This would be a more ideal approach than echoing the error messages. Thanks very much for helping us discover this problem though! Greg Von Kuster Hi Greg, No problem - I'm good at discovering problems ;) If the download approach failed, it it most likely due to a transient error (e.g. network issues with download). Here I would much prefer Galaxy aborted and reported this as an error (and does not attempt the default action). Is that what you just fixed? As to best practice for the fall back action, I think that needs a new thread. Regards, Peter As to best practice, I do not agree that in cases like this (MIRA4, NCBI BLAST+) where there are provided binaries for the major platforms that the fall back should be compiling from source. The NCBI BLAST+ provide binaries for 32 bit and 64 bit Linux and Mac OS X (which I believe covers all the mainstream platforms Galaxy runs on). Similarly, MIRA4 provides binaries for 64 bit Linux and Mac OS X. Note that 32 bit binaries are not provided, but would be very restricted in terms of the datasets they could be used on anyway - and I doubt many of the systems Galaxy runs on these days are 32 bits. If the os/arch combination is exotic enough that precompiled binaries are not available, then it is likely compilation will be tricky anyway - or not supported for that tool, or Galaxy itself. Essentially I am arguing that where the precompiled tool binaries cover any mainstream system Galaxy might be used on, a local compile fall back is not needed. Also, these are both complex tools which are relatively slow to compile, and have quite a large compile time dependency set (e.g. MIRA4 requires at least a quite recent GCC, BOOST, flex, expat, and strongly recommends TCmalloc). Here at least some of the dependencies have been packaged for the ToolShed (probably by Bjoern?) but in the case of MIRA4 and BLAST+ this is still a lot of effort for no practical gain. I also feel there is an argument that the Galaxy goal of reproducibility should favour using precompiled binaries if available: A locally compiled binary will generally mean a different compiler version, perhaps with different optimisation flags, and different library versions. It will not necessarily give the same results as the tool author's provided precompiled binary. (Wow, this ended up being a long email!) Regards, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists,
Re: [galaxy-dev] Tool Shed install best practise : Precompiled binaries vs local compile
On Fri, Mar 14, 2014 at 12:47 PM, Peter Cock p.j.a.c...@googlemail.com wrote: On Fri, Mar 14, 2014 at 4:41 PM, Nate Coraor n...@bx.psu.edu wrote: Hi Peter, I just wanted to add my $0.02 USD to say that I mostly agree with this - I have long used binaries precompiled by the tool author on Main, especially for cases where, as you say, the compile-time dependency list is large and painful. The only gotcha here is to make sure that binaries support the oldest possible version of glibc that might be installed on any of the Linux distributions and versions that we support, and that no non-standard preinstalled libraries have been pulled in during the build process. --nate Good point about potential problems with glibc, and I guess any run time linked libraries which might be a different version or missing? Peter Right, many things (especially using autoconf) will automatically link against libraries if present and disable functionality if not. Hopefully if we were returning to mostly vanilla tool test VMs in the future, any problems would be easily discoverable. --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Torque Exec Exception
Hello, Are you able to run `qstat -x` on the command line on the remote system? What's the output? --nate On Thu, Mar 6, 2014 at 1:01 PM, ngsf...@hygenomics.com ngsf...@hygenomics.com wrote: hi all: galaxy.jobs.runners.cli_job.torque WARNING 2014-03-07 11:23:33,764 No valid qstat XML return from `qstat -x`, got the following: galaxy.jobs.runners ERROR 2014-03-07 11:23:33,765 Unhandled exception checking active jobs Traceback (most recent call last): File /share/home/ldapuser/main/lib/galaxy/jobs/runners/__init__.py, line 339, in monitor self.check_watched_items() File /share/home/ldapuser/main/lib/galaxy/jobs/runners/cli.py, line 150, in check_watched_items job_states = self.__get_job_states() File /share/home/ldapuser/main/lib/galaxy/jobs/runners/cli.py, line 202, in __get_job_states job_states.update(job_interface.parse_status(cmd_out.stdout, job_ids)) TypeError: 'NoneType' object is not iterable #qmgr -c p q batch create queue batch set queue batch queue_type = Execution set queue batch enabled = True set queue batch started = True Finish the torque tasks ， throw exceptions in the log file。 ngsf...@hygenomics.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] FTP password and web interface password
Hi Yec'han, Could you check that the 'password' column for the user in question in the 'galaxy_user' table in the database does not begin with $PBKDF2$? If not, do you have any debug logs from the FTP session and server that provide details on the failure? --nate On Wed, Mar 5, 2014 at 10:36 AM, Yec'han Laizet ylai...@pierroton.inra.fr wrote: Hello, does anybody have any idea of what I can do to fix the problem? Maybe an update is required? I currently use the changeset: 11219:5c789ab4144a thanks Yec'han Dr. Yec'han LAIZET Ingenieur Bioinformatique Tel: +33 (0)5 57 12 27 75 _ INRA-UMR BIOGECO 1202 Equipe Genetique 69 route d'Arcachon 33612 CESTAS Le 18/02/2014 08:39, Yec'han Laizet a écrit : Hi Bjoern, I indeed followed the wiki tutorial to set up my FTP service some time ago. It seems, as you suggest, that newly created users cannot connect by SFTP. I followed the fix by putting the use_pbkdf2 = False line just below the [app:main] and restarted the galaxy server. I have reseted a newly created user password but it still does not work. Yec'han Dr. Yec'han LAIZET Ingenieur Bioinformatique Tel: +33 (0)5 57 12 27 75 _ INRA-UMR BIOGECO 1202 Equipe Genetique 69 route d'Arcachon 33612 CESTAS Le 17/02/2014 18:12, Björn Grüning a écrit : Hi Yec'han, please have a look at https://wiki.galaxyproject.org/Admin/Config/Upload%20via%20FTP If you are running postgres and you only newly created users can't access the server its probably due to encryption changes. Set use_pbkdf2 = False and reset all passwort for new users. Cheers, Bjoern Am 17.02.2014 17:27, schrieb Yec'han Laizet: Hello, I set up a FTP server with SFTP support on my galaxy instance. I have a strange behavior when trying to connect by SFTP. Some users cannot authentify (access denied) whereas other can. As all users can login to the web interface with their credentials, I wanted to check if the length of the password could be the problem with SFTP. To do so, I went to the admin interface to reset the password of a user who could connect by SFTP. Then, this user can connect to the galaxy web interface with the new password but not by SFTP ; if we use the old password, it still works for SFTP authenfication as if both passwords are independent. Could you help me to solve the problem? Yec'han Dr. Yec'han LAIZET Ingenieur Bioinformatique Tel: +33 (0)5 57 12 27 75 _ INRA-UMR BIOGECO 1202 Equipe Genetique 69 route d'Arcachon 33612 CESTAS ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Uploaded files not previewing at random
Hi Ravi,
Are your datasets stored on a shared filesystem (e.g. NFS)? If yes,
you could try setting `retry_job_output_collection` to something 0
in universe_wsgi.ini? NFS attribute caching can produce behavior like
this.
--nate
On Wed, Mar 5, 2014 at 11:46 AM, Sanka, Ravi rsa...@jcvi.org wrote:
Greetings,
Since yesterday, I have been experiencing an unusual problem with Uploading
files.
Seemingly at random, an uploaded will finish successfully (history item in
green) but no preview is available. Under the title of the item, the keyword
empty is found and preview box says no peek. Such an item cannot be
previewed or downloaded; attempts to do so result in this error displayed in
the browser:
OK
The requested URL /datasets/{API ID}/display/ was not found on this server.
Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips DAV/2 Server at our
galaxy URL Port 80
However, the Full Path of the item is still valid and, when viewed in
command line, shows all the data. Furthermore, the item can still be used in
tools and so forth.
I and other users have tried multiple attempts at uploading the exact, same
file (a small FASTA) and the results are the same: sometimes this issue is
present, other times it is not. There seems to be no pattern nor are there
any errors in the paster.log.
--
Ravi Sanka
ICS - Sr. Bioinformatics Engineer
J. Craig Venter Institute
301-795-7743
--
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Markup safe egg problem/fetching the wrong eggs?
Hi Joe, Something is automatically re-fetching the Python 2.6 eggs? Is the Galaxy server itself using 2.6 perhaps? --nate On Fri, Feb 28, 2014 at 3:42 AM, Joseph Ward j.x.w...@dundee.ac.uk wrote: Hi, I've got a strange problem with python paths/eggs. When I run any tool that produces html/text output I get the error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched. So if I close galaxy, delete the eggs folder, run fetch_eggs.py (on the cluster, sourcing the same files as galaxy does so the python path should be identical) it gathers all the eggs it needs for python 2.7. Running a job then ends with the proper results, no errors, but not with python 2.6 eggs present in the eggs folder alongside the 2.7 eggs. Re-running the same job, or running any other that produces html output, not throws up the error mentioned above. I can't see anywhere where I've missed a python path or anywhere where python 2.6 is mentioned. Any idea what I'm missing here? Cheers, Joe ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Mercurial wants to overwrite my job_conf.xml
Hi Carrie, .hgignore should not affect files that you have explicitly tracked (e.g. with `hg add job_conf.xml`) - if the file is tracked, Mercurial will attempt to update and patch the working copy if there are upstream changes. Also, it looks like your current tip includes neither .hgignore or job_conf.xml: https://bitbucket.org/cganote/iu-ncgas-galaxy/src/tip Your tip is also on the stable branch - do you mean to be working in two branches? --nate On Wed, Feb 26, 2014 at 6:31 PM, Ganote, Carrie L cgan...@iu.edu wrote: Hi Galaxy devs, I had an issue recently where I must have messed up my .hgignore file or done something terrible to my repository. I created a bitbucket and pushed my instance to it. I then pulled that onto my laptop and hooked it all up with Eclipse. After I was happy with my code, I pushed to bb and tried to pull to my instance, but the job_conf.xml file was overwritten. I had all of my tools set up there for the instance; on my laptop I had removed all that because it didn't apply. Would it have messed up if I deleted the file on my laptop and replaced it with a copy of job_conf.xml.simple? I took the .simple out of course and altered it to just what I needed. This is cross-posted with Stack Overflow: http://stackoverflow.com/questions/22055348/mercurial-update-seems-to-overwrite-local-file Here is my repo: https://bitbucket.org/cganote/iu-ncgas-galaxy Any help would be greatly appreciated. -Carrie ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Mercurial wants to overwrite my job_conf.xml
Hi Carrie, The easiest solution at this point may be to generate diffs (e.g. with `hg diff`) of the changes you want, and then apply those in clean commits. --nate On Thu, Feb 27, 2014 at 10:10 AM, Ganote, Carrie L cgan...@iu.edu wrote: Hi Nate, Thanks for the reply! To answer/clarify your points: .hgignore should not affect files that you have explicitly tracked (e.g. with `hg add job_conf.xml`) I have a suspicion that when I tried to add a directory using Eclipse, I might have added the whole repository, taking it all out of .hgignore's control. if the file is tracked, Mercurial will attempt to update and patch the working copy if there are upstream changes. It does seem like it isn't merging or patching, just choosing one over the other... it looks like your current tip includes neither .hgignore or job_conf.xml I did indeed try to remove those files from being tracked in the tip. This caused those files to be removed from my working copy when I updated. Your tip is also on the stable branch - do you mean to be working in two branches? I didn't know what I was doing (still dont!) when I set up the repository - it seemed like if I was pulling from stable automatically, I should do all of my work in stable. I was further confused when Eclipse pushed to default by default and then I ended up mangling both. What I'm looking for now is a way to merge the last good changeset from my remote server (bf3924e Added pull..) with my last good local repo (6dc34fa Refined cli runner..). Should I just pull 6dc and replace the files I need from bf3 manually? Thanks so much for the advice, Carrie Ganote From: Nate Coraor [n...@bx.psu.edu] Sent: Thursday, February 27, 2014 8:49 AM To: Ganote, Carrie L Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Mercurial wants to overwrite my job_conf.xml Hi Carrie, .hgignore should not affect files that you have explicitly tracked (e.g. with `hg add job_conf.xml`) - if the file is tracked, Mercurial will attempt to update and patch the working copy if there are upstream changes. Also, it looks like your current tip includes neither .hgignore or job_conf.xml: https://bitbucket.org/cganote/iu-ncgas-galaxy/src/tip Your tip is also on the stable branch - do you mean to be working in two branches? --nate On Wed, Feb 26, 2014 at 6:31 PM, Ganote, Carrie L cgan...@iu.edu wrote: Hi Galaxy devs, I had an issue recently where I must have messed up my .hgignore file or done something terrible to my repository. I created a bitbucket and pushed my instance to it. I then pulled that onto my laptop and hooked it all up with Eclipse. After I was happy with my code, I pushed to bb and tried to pull to my instance, but the job_conf.xml file was overwritten. I had all of my tools set up there for the instance; on my laptop I had removed all that because it didn't apply. Would it have messed up if I deleted the file on my laptop and replaced it with a copy of job_conf.xml.simple? I took the .simple out of course and altered it to just what I needed. This is cross-posted with Stack Overflow: http://stackoverflow.com/questions/22055348/mercurial-update-seems-to-overwrite-local-file Here is my repo: https://bitbucket.org/cganote/iu-ncgas-galaxy Any help would be greatly appreciated. -Carrie ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] local toolshed with remote-user=true and require-user=falase
Hi Brad, If you're using HTTP Auth, that will be the case since HTTP Auth has no notion of sessions, the auth credentials must be provided with every request. For a more robust solution, you probably want to use an auth filter that creates an authentication session. Penn State uses Cosign for this, but there are other options. --nate On Feb 25, 2014, at 17:09, Langhorst, Brad langho...@neb.com wrote: Has anyone set up a local toolshed with external authentication? Is this expected to work? I have external auth working, but tools cannot be installed (403 forbidden) unless I turn of authentication. If i turn on remote-auth, i have to configure the webserver to ask for credentials otherwise i get an error page. It would make sense to have the webserver request credentials only for requests to a login page, but I don’t see how to do that. For now I’ve just turned off remote-auth. Brad -- Brad Langhorst, Ph.D. Applications and Product Development Scientist ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Cufflinks doesn't stop
Hi Björn, Does stopping other, non-split jobs work correctly for you? --nate On Wed, Feb 26, 2014 at 7:30 AM, Björn Grüning bjoern.gruen...@gmail.comwrote: Hi Nate, I'm running changeset: 12641:cab3db6e1d59 and I can see the same behaviour. The blast job is not listed under jobs in the admin section and I can't kill it, also not via deleting the dataset. Anything I can do to track that down? In my case it's a large blast job with splitting enabled. Cheers, Bjoern Hi all, There was a regression introduced in the most recent stable release that was preventing jobs from being stopped, fixed here: https://bitbucket.org/galaxy/galaxy-central/commits/ 1298d3f6aca59825d0eb3d32afd5686c4b1b9294?at=stable You can pull from the stable branch of galaxy-central to get that changeset. If that doesn't resolve the problem, it sounds like it may be due to the crashing handlers, so we would need to figure out the cause of those crashes. --nate On Sat, Feb 22, 2014 at 12:24 PM, Björn Grüning bjoern.gruen...@gmail.comwrote: Hi, I have one running at the moment, that job is also not listed in the jobs monitor under the admin-panel. Cheers, Bjoern On Tue, Feb 18, 2014 at 2:20 PM, Federico Zambelli federico.zambe...@unimi.it wrote: Hi, something strange happens with Cufflinks in our Galaxy server. When a user deletes a running Cufflinks job in fact the associated Cufflinks process(es) are not terminated. Apart from unneccessary CPU usage, this prevents other jobs from starting if the max jobs limit has been already reached by the user. This happens only with Cufflinks, Cuffcompare for comparison behaves just fine. Best, F. What cluster system are you using? Can you try a few other long running tools for comparison? I've been meaning to double check if this still happens, but I had seen this for BLAST+ jobs under SGE (noticeable again when large zombie jobs are blocking the queue). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Difficulty dispatching toolshed tool jobs via job_conf.xml
Hi Björn, Please try it without the trailing slash on the tool ID. --nate On Wed, Feb 26, 2014 at 11:28 AM, Björn Grüning bjoern.gruen...@gmail.comwrote: Hi Nate, sorry to bother you again, but that issue is still not fixed for me. The following is supposed to work, or? tool id=toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_blastn_wrapper/ destination=24_cores_24G / tool id=toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_blastx_wrapper/ destination=24_cores_24G / tool id=toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_tblastn_wrapper/ destination=24_cores_24G / tool id=ncbi_blastp_wrapper destination=24_cores_24G / tool id=toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_rpsblast_wrapper/ destination=24_cores_24G / tool id=toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ ncbi_tblastx_wrapper/ destination=24_cores_24G / tool id=toolshed.g2.bx.psu.edu/repos/peterjc/blast2go/ blast2go/0.0.6 destination=20G_memory/ Thanks, Bjoern Am 14.02.2014 16:37, schrieb Nate Coraor: Hi Björn, It should be fixed, the problem was with tools with uppercase characters in the guid: https://bitbucket.org/galaxy/galaxy-central/commits/0b955e54451c/ --nate On Thu, Feb 13, 2014 at 2:38 AM, Björn Grüning bjoern.gruen...@gmail.com wrote: Hi Nate Simon, was that solved? I think it's still not working? Or is that a regression. In particular shed_host/repos/owner/repo/tool_id is not working, and that is what is 90% needed, or? Thanks, Bjoern From: Nate Coraor [mailto:n...@bx.psu.edu] This should work, it looks like there may be a bug with handling tool shed IDs when determining job destinations. You are actually supposed to be able to use any of: shed_host/repos/owner/repo/tool_id/version shed_host/repos/owner/repo/tool_id tool_id I'll take a look at this as soon as possible. One thing you might try in the short term is using the percent encoded tool id in the tool tag in job_conf.xml: Hi Nate, Using the percent encoded form was the first thing I tried (before emailing the list). It didn't make any difference. I think there's another reason it's not picking it up. Thanks for looking into this. cheers, Simon === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. === ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Cufflinks doesn't stop
Hi all, There was a regression introduced in the most recent stable release that was preventing jobs from being stopped, fixed here: https://bitbucket.org/galaxy/galaxy-central/commits/1298d3f6aca59825d0eb3d32afd5686c4b1b9294?at=stable You can pull from the stable branch of galaxy-central to get that changeset. If that doesn't resolve the problem, it sounds like it may be due to the crashing handlers, so we would need to figure out the cause of those crashes. --nate On Sat, Feb 22, 2014 at 12:24 PM, Björn Grüning bjoern.gruen...@gmail.comwrote: Hi, I have one running at the moment, that job is also not listed in the jobs monitor under the admin-panel. Cheers, Bjoern On Tue, Feb 18, 2014 at 2:20 PM, Federico Zambelli federico.zambe...@unimi.it wrote: Hi, something strange happens with Cufflinks in our Galaxy server. When a user deletes a running Cufflinks job in fact the associated Cufflinks process(es) are not terminated. Apart from unneccessary CPU usage, this prevents other jobs from starting if the max jobs limit has been already reached by the user. This happens only with Cufflinks, Cuffcompare for comparison behaves just fine. Best, F. What cluster system are you using? Can you try a few other long running tools for comparison? I've been meaning to double check if this still happens, but I had seen this for BLAST+ jobs under SGE (noticeable again when large zombie jobs are blocking the queue). Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Error when installing package_galaxy_utils
Hi Sarah, Could you include the installation stdout as well? The dependency installation log separates stdout and stderr into two sections, there might be more detail about exactly where it's failing in the stdout. --nate On Fri, Feb 21, 2014 at 4:21 AM, Sarah Diehl di...@ie-freiburg.mpg.dewrote: Hi Nate, I tried it multiple times and always got the same error. I did successfully install other tools before and after that. I didn't notice any issues with the filesystem. Is there a way to find out what exactly the installation tries to do at that point? Thanks, Sarah - Original Message - From: Nate Coraor n...@bx.psu.edu To: Sarah Diehl di...@ie-freiburg.mpg.de Cc: galaxy-dev@lists.bx.psu.edu List galaxy-dev@lists.bx.psu.edu Sent: Friday, February 21, 2014 5:10:46 AM Subject: Re: [galaxy-dev] Error when installing package_galaxy_utils Hi Sarah, This looks a lot like a transient filesystem error. If you attempt to reinstall the repository, do you get the same error? --nate On Thu, Feb 20, 2014 at 10:48 AM, Sarah Diehl di...@ie-freiburg.mpg.de wrote: Hi everyone, I just updated our local Galaxy installation to the latest dist version and also did the tool migration steps (./scripts/migrate_tools/0009_tools.sh). However, the package galaxy_utils gave me an error and I don't have the slightest idea what happened. So any help to figure out the issue is greatly appreciated! Thanks, Sarah # export PYTHONPATH=$PYTHONPATH:/galaxy/galaxy_data/tools/galaxy_sequence_utils/1.0.0/devteam/package_galaxy_utils_1_0/0643676ad5f7/lib/python python setup.py install --home /galaxy/galaxy_data/tools/galaxy_seq uence_utils/1.0.0/devteam/package_galaxy_utils_1_0/0643676ad5f7 STDERR Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.38.tar.gz Extracting in /tmp/tmpo9Zxdz Now working in /tmp/tmpo9Zxdz/distribute-0.6.38 Building a Distribute egg in /galaxy/galaxy_server/database/tmp/tmp-toolshed-mtd00x35_/galaxy_sequence_utils-1.0.0 /galaxy/galaxy_server/database/tmp/tmp-toolshed-mtd00x35_/galaxy_sequence_utils-1.0.0/distribute-0.6.38-py2.7.egg error: site.py: Resource temporarily unavailable # ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Error when installing package_galaxy_utils
Hi Sarah, This looks a lot like a transient filesystem error. If you attempt to reinstall the repository, do you get the same error? --nate On Thu, Feb 20, 2014 at 10:48 AM, Sarah Diehl di...@ie-freiburg.mpg.dewrote: Hi everyone, I just updated our local Galaxy installation to the latest dist version and also did the tool migration steps (./scripts/migrate_tools/0009_tools.sh). However, the package galaxy_utils gave me an error and I don't have the slightest idea what happened. So any help to figure out the issue is greatly appreciated! Thanks, Sarah # export PYTHONPATH=$PYTHONPATH:/galaxy/galaxy_data/tools/galaxy_sequence_utils/1.0.0/devteam/package_galaxy_utils_1_0/0643676ad5f7/lib/python python setup.py install --home /galaxy/galaxy_data/tools/galaxy_seq uence_utils/1.0.0/devteam/package_galaxy_utils_1_0/0643676ad5f7 STDERR Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.38.tar.gz Extracting in /tmp/tmpo9Zxdz Now working in /tmp/tmpo9Zxdz/distribute-0.6.38 Building a Distribute egg in /galaxy/galaxy_server/database/tmp/tmp-toolshed-mtd00x35_/galaxy_sequence_utils-1.0.0 /galaxy/galaxy_server/database/tmp/tmp-toolshed-mtd00x35_/galaxy_sequence_utils-1.0.0/distribute-0.6.38-py2.7.egg error: site.py: Resource temporarily unavailable # ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Database and FTP setup
Hi Turner, You should be connecting to the database that Galaxy is using, specified in the database_connection option in universe_wsgi.ini. If you have not yet started Galaxy at least once, then the tables will not have been created yet. Once you run Galaxy and the tables have been created, you should be able to connect to that database (galaxydb, or whatever you have set in database_connection) with psql and the GRANT should succeed. --nate On Thu, Feb 20, 2014 at 9:49 AM, Conrad, Turner Allen conr...@livemail.uthscsa.edu wrote: Hi, I'm not experienced with postgres and ftp and am thus having difficulties setting up an ftp server on my local Galaxy instance, but it is essential as our files are all over 2GB. The admin support page ( https://wiki.galaxyproject.org/Admin/Config/Upload%20via%20FTP) is a little fuzzy on the details. I'm currently running the server out of Ubuntu 13.10 desktop both physically and via SSH. So far, I've created a local account postgres with a psql database galaxydb and user galaxyftp, but I can't figure out whether the galaxy_user table is supposed to exist, needs to be created, nor how to add this information to the database_connection option in universe_wsgi.ini. ALTER ROLE has been completed successfully, but I get the error ERROR: relation galaxy_user does not exist upon attempting GRANT SELECT ON. Is there a simple explanation of how to complete this task and get the proftpd server up and running for Ubuntu? Please assume proftpd is installed, but not configured, the database is in the state described above, and I am starting with sudo su - postgres followed by psql galaxydb . Turner Conrad UTHSCSA ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Database and FTP setup
Hi Turner, In universe_wsgi.ini, set 'use_pbkdf2 = False' in the '[app:main]' section. You probably do not need to make any changes to PostgreSQL's configuration unless your database is running on a different host from your Galaxy server. For proftpd.conf, the most important bits are these: # Do not authenticate against real (system) usersAuthPAM off# Set up mod_sql_password - Galaxy passwords are stored as hex-encoded SHA1SQLPasswordEngine onSQLPasswordEncoding hex# Set up mod_sql to authenticate against the Galaxy databaseSQLEngine onSQLBackend postgresSQLConnectInfo galax...@dbserver.example.org dbuser dbpasswordSQLAuthTypes SHA1SQLAuthenticate users# An empty directory in case chroot failsSQLDefaultHomedir /var/opt/local/proftpd# Define a custom query for lookup that returns a passwd-like entry. UID and GID should match your Galaxy user.SQLUserInfo custom:/LookupGalaxyUserSQLNamedQuery LookupGalaxyUser SELECT email,password,'512','512','/home/nate/galaxy_dist/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U' In your instance you most likely want: SQLConnectInfo galaxydb@/var/run/postgresql galaxyftp galaxyftpUsersPassword In SQLNamedQuery, make sure that you change the first 512 to the uid of the system user id your Galaxy server is running as, and the second 512 to the system group id. These can be obtained with `id -u` and `id -g` on the command line. /home/nate/galaxy_dist/database/ftp should be changed to the directory on your server in which files uploaded via FTP should be placed (this location is not permanent - the files themselves will be copied into the Galaxy directory). Hope this helps, --nate On Thu, Feb 20, 2014 at 11:33 PM, Conrad, Turner Allen conr...@livemail.uthscsa.edu wrote: Thanks, Nate! Got it to run after playing with the postgresql url, table was created, and I began populating it with my admin account. I also prepared the database for ftp by creating a user for lookup. As for proftpd, I have it installed, but am not quite sure what do now. How must I edit the postgresql and proftpd configuration files before changing the galaxy ini? 1LT Turner A Conrad, USAR IRR Ph.D. Candidate Graduate School of Biomedical Sciences MED 4.017V Dept. of Microbiology and Immunology 7703 Floyd Curl Drive San Antonio, TX 78229 Email: conr...@livemail.uthscsa.edu Office: (210) 567-3947 Cell: (210) 954-0893 On Feb 20, 2014 10:23 PM, Nate Coraor n...@bx.psu.edu wrote: Hi Turner, You should be connecting to the database that Galaxy is using, specified in the database_connection option in universe_wsgi.ini. If you have not yet started Galaxy at least once, then the tables will not have been created yet. Once you run Galaxy and the tables have been created, you should be able to connect to that database (galaxydb, or whatever you have set in database_connection) with psql and the GRANT should succeed. --nate On Thu, Feb 20, 2014 at 9:49 AM, Conrad, Turner Allen conr...@livemail.uthscsa.edu wrote: Hi, I'm not experienced with postgres and ftp and am thus having difficulties setting up an ftp server on my local Galaxy instance, but it is essential as our files are all over 2GB. The admin support page ( https://wiki.galaxyproject.org/Admin/Config/Upload%20via%20FTP) is a little fuzzy on the details. I'm currently running the server out of Ubuntu 13.10 desktop both physically and via SSH. So far, I've created a local account postgres with a psql database galaxydb and user galaxyftp, but I can't figure out whether the galaxy_user table is supposed to exist, needs to be created, nor how to add this information to the database_connection option in universe_wsgi.ini. ALTER ROLE has been completed successfully, but I get the error ERROR: relation galaxy_user does not exist upon attempting GRANT SELECT ON. Is there a simple explanation of how to complete this task and get the proftpd server up and running for Ubuntu? Please assume proftpd is installed, but not configured, the database is in the state described above, and I am starting with sudo su - postgres followed by psql galaxydb . Turner Conrad UTHSCSA ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified
Re: [galaxy-dev] Acceleration Card Optimization
Hi Adam, This is going to depend entirely on the tools you're using. I'd suggest running your typical workload and examining the I/O patterns to determine what makes the most sense. I'd think the temp directory, input dataset, and output dataset filesystems to be the most utilized. --nate On Fri, Feb 14, 2014 at 12:31 PM, Panzer, Adam panze...@kids.wustl.eduwrote: Hello All, I recently was gifted an ioFusion ioFX acceleration card to help speed my bioinformatics work along and I am wondering how best to integrate it into my instance to make Galaxy more efficient. Is it sufficient to merely ensure that the datasets being operated on are on the flash storage (I guess by placing the main Galaxy directory there) or are there other elements that should also move lest they create bottlenecks elsewhere (tmp directory, files related to proxy/ftp server, tool shed, etc.)? Suggestions will be greatly appreciated. Thanks, Adam The materials in this email are private and may contain Protected Health Information. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying, distribution or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return email. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Difficulty dispatching toolshed tool jobs via job_conf.xml
Hi Björn, It should be fixed, the problem was with tools with uppercase characters in the guid: https://bitbucket.org/galaxy/galaxy-central/commits/0b955e54451c/ --nate On Thu, Feb 13, 2014 at 2:38 AM, Björn Grüning bjoern.gruen...@gmail.comwrote: Hi Nate Simon, was that solved? I think it's still not working? Or is that a regression. In particular shed_host/repos/owner/repo/tool_id is not working, and that is what is 90% needed, or? Thanks, Bjoern From: Nate Coraor [mailto:n...@bx.psu.edu] This should work, it looks like there may be a bug with handling tool shed IDs when determining job destinations. You are actually supposed to be able to use any of: shed_host/repos/owner/repo/tool_id/version shed_host/repos/owner/repo/tool_id tool_id I'll take a look at this as soon as possible. One thing you might try in the short term is using the percent encoded tool id in the tool tag in job_conf.xml: Hi Nate, Using the percent encoded form was the first thing I tried (before emailing the list). It didn't make any difference. I think there's another reason it's not picking it up. Thanks for looking into this. cheers, Simon === Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. === ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] sysadmin help
Hi Donny, Thanks for the heads up about the broken link, I've fixed it. Other than that, are you running in to problems connecting Galaxy to your existing cluster? --nate On Tue, Jan 28, 2014 at 9:29 PM, Shrum, Donald C dcsh...@admin.fsu.eduwrote: Hi all, I could use a little basic help. I'm setting up a proof of concept galaxy server that will run in conjunction with our HPC here at FSU. Ultimately this will serve as a template where we will allow some users to run galaxy on a VM and use it as a mechanism for submitting/managing jobs on our cluster. I've installed Galaxy on a VM that mounts our HPC file system and managed to import an ecoli .fa file. My plan was to run bowtie2 on our HPC. https://rcc.fsu.edu/software/bowtie2 Disclaimer... I'm a sysadmin and other than compiling software my knowledge in bioinformatics is sorely lacking ;) I'm looking at the docs here https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#PBS I get a 404 for details on Torque - http://www.clusterresources.com/pages/products/torque-resource-manager.php Any help would be greatly appreciated. Additionally, if my intended use for Galaxy seems out of the scope of what would be useful, please let me know. Thanks, Donny Shrum FSU Research Computing Center ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Can REMOTE_USER be changed to another header variable?
Hi Prakash, Could you send the output of `hg summary`? Thanks, --nate On Fri, Jan 3, 2014 at 1:41 PM, Velayutham, Prakash (Prakash) prakash.velayut...@cchmc.org wrote: Hi Nate, I just updated my copy and the changes you pushed are in. However, the auth part is not working still. I added remote_user_header = 'HTTP_AUTH_USER' to universe_wsgi.ini and restarted Galaxy. When I hit the site, after logging into the front end proxy server, I get this. Access to Galaxy is denied Galaxy is configured to authenticate users via an external method (such as HTTP authentication in Apache), but a username was not provided by the upstream (proxy) server. This is generally due to a misconfiguration in the upstream server. Please contact your local Galaxy administrator. I am capturing all the header variables in a file and this is what the contents of the file is after the above DENIED message. [srv-galaxy@bmigalaxyp1 galaxy-dist]$ cat file.py HTTP_X_FORWARDED_SERVER: galaxy.research.cchmc.org HTTP_COOKIE: galaxysession=c6ca0ddb55be603ac556311ffa6257cd21da46c2083580c93cee9aaaf9c0c67c8e80f388ebf98dff; BIGipServerbmigw-pool=626771722.20480.; ObSSOCookie=QF4kYG5VvhHej14EN4XRqPVEgJ7ukfSLFWTmDjibS5YUstElLeDIwcxFAgtZhGi3uJGhh4f6lFQcmAl2B1%2FM%2BptbBKwkCGNQGkJhKhu1Pz4x7bjDOaifC9t%2Fhgy%2FN3FAoXSQUFFg0cVkXnKKhoA5Hxkt%2BcvkQObSn7Mr1Vi0xPakNoRcEC7k%2BhhR3Vp8oGUEkODLotLSAvkPfj8xL0rfzgYuLI3aY8F77M2Sj7vcDiOB03VOiBddelvOqLTHfYwlktQ81MlQq%2BjQPMX5wo9g7DhD7nwtSBgvozJ0VvmNmMfn%2BKvkgEXo8YbyQakY5PXg2pJE6IjUJTF%2FpKOfO5W2IKYzkqbDgicaMjTKq1Q7zr%2BW0BQKzhsEIjhHkneH2NRiIUiriemEbJVVo9nrMsxviT8Hah7X5YZ5kVGjBpX5owA%3D HTTP_ACCEPT_LANGUAGE: en-us paste.recursive.include: paste.recursive.Includer from / SCRIPT_NAME: REQUEST_METHOD: GET PATH_INFO: / HTTP_ORIGIN: https://login.research.cchmc.org SERVER_PROTOCOL: HTTP/1.1 QUERY_STRING: paste.throw_errors: True CONTENT_LENGTH: 0 weberror.evalexception: weberror.evalexception.middleware.EvalException object at 0x8d02d50 HTTP_USER_AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/536.30.1 (KHTML, like Gecko) Version/6.0.5 Safari/536.30.1 HTTP_CONNECTION: Keep-Alive SERVER_NAME: 0.0.0.0 REMOTE_ADDR: 10.199.194.17 ORGINAL_REMOTE_ADDR: 10.199.92.37 wsgi.url_scheme: http SERVER_PORT: 8080 paste.recursive.forward: paste.recursive.Forwarder from / paste.recursive.script_name: paste.evalexception: weberror.evalexception.middleware.EvalException object at 0x8d02d50 wsgi.input: socket._fileobject object at 0x8d9eb50 length=0 HTTP_HOST: galaxy.research.cchmc.org paste.recursive.include_app_iter: paste.recursive.IncluderAppIter from / wsgi.multithread: True HTTP_CONFVER: 1 HTTP_CACHE_CONTROL: max-age=0 HTTP_ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 wsgi.version: (1, 0) HTTP_AUTH_USER: prakash.velayut...@cchmc.org wsgi.run_once: False wsgi.errors: galaxy.util.pastescript.serve.LazyWriter object at 0x239db10 wsgi.multiprocess: False HTTP_X_FORWARDED_HOST: galaxy.research.cchmc.org HTTP_X_FORWARDED_FOR: 10.199.194.17 CONTENT_TYPE: request_id: 34e3f63274a611e3aaf1005056a84587 paste.httpserver.thread_pool: paste.httpserver.ThreadPool object at 0x8da5750 ORGINAL_HTTP_HOST: bmigalaxyp1.chmcres.cchmc.org:8080 HTTP_UID: VELGE9 [srv-galaxy@bmigalaxyp1 galaxy-dist]$ Obviously, I am logging in using HTTP_AUTH_USER, which does exist in the file, but auth is not going forward. Please note that without the recent changes, I was able to change every instance of REMOTE_USER in the source code with AUTH_USER and that worked without issues. Thanks, Prakash On Jan 3, 2014, at 11:45 AM, Nate Coraor n...@bx.psu.edu wrote: Hi Prakash, This was not previously possible, but I have added a config option for it: https://bitbucket.org/galaxy/galaxy-central/commits/e92e13e9c103cc1f36dff65e1523479bf5cb17ed If you're running the stable branch, you can apply the changes from this commit manually. --nate On Thu, Jan 2, 2014 at 11:09 AM, Jennifer Jackson j...@bx.psu.edu wrote: Hello Prakash, I am going to move this over to the galaxy-...@bx.psu.edu mailing list where it will have greater visibility within our development community. Best, Jen Galaxy team https://wiki.galaxyproject.org/MailingLists#The_lists On 1/2/14 7:27 AM, Velayutham, Prakash (Prakash) wrote: Hi, We have a SSO environment provided by Oracle Fusion products and for some reason, they don't like to send over HTTP_REMOTE_USER as a header variable to downstream servers. I have seen it before with other web sites I have integrated with Oracle Access Manager. Is there a way Galaxy can accept another HEADER variable than REMOTE_USER for its external authentication? As an extension: With just enabling HTTP_REMOTE_USER as a header variable from an external authenticator, Galaxy works without any issues. I tried this with a default Apache/mod_ldap/mod_authnz_ldap setup. However, when I mix the Oracle gateways
Re: [galaxy-dev] Can REMOTE_USER be changed to another header variable?
In universe_wsgi.ini, remove the single quotes from the value of remote_user_header, e.g.: remote_user_header = HTTP_AUTH_USER If that doesn't fix it, please make sure you don't have local changes interfering, e.g. inspect `hg diff`. --nate On Fri, Jan 3, 2014 at 2:21 PM, Velayutham, Prakash (Prakash) prakash.velayut...@cchmc.org wrote: [srv-galaxy@bmigalaxyp1 galaxy-dist]$ hg summary parent: 11939:e92e13e9c103 tip Allow changing the header for remote user. branch: default commit: 1 modified, 1 unknown update: (current) [srv-galaxy@bmigalaxyp1 galaxy-dist]$ Prakash On Jan 3, 2014, at 2:11 PM, Nate Coraor n...@bx.psu.edu wrote: Hi Prakash, Could you send the output of `hg summary`? Thanks, --nate On Fri, Jan 3, 2014 at 1:41 PM, Velayutham, Prakash (Prakash) prakash.velayut...@cchmc.org wrote: Hi Nate, I just updated my copy and the changes you pushed are in. However, the auth part is not working still. I added remote_user_header = 'HTTP_AUTH_USER' to universe_wsgi.ini and restarted Galaxy. When I hit the site, after logging into the front end proxy server, I get this. Access to Galaxy is denied Galaxy is configured to authenticate users via an external method (such as HTTP authentication in Apache), but a username was not provided by the upstream (proxy) server. This is generally due to a misconfiguration in the upstream server. Please contact your local Galaxy administrator. I am capturing all the header variables in a file and this is what the contents of the file is after the above DENIED message. [srv-galaxy@bmigalaxyp1 galaxy-dist]$ cat file.py HTTP_X_FORWARDED_SERVER: galaxy.research.cchmc.org HTTP_COOKIE: galaxysession=c6ca0ddb55be603ac556311ffa6257cd21da46c2083580c93cee9aaaf9c0c67c8e80f388ebf98dff; BIGipServerbmigw-pool=626771722.20480.; ObSSOCookie=QF4kYG5VvhHej14EN4XRqPVEgJ7ukfSLFWTmDjibS5YUstElLeDIwcxFAgtZhGi3uJGhh4f6lFQcmAl2B1%2FM%2BptbBKwkCGNQGkJhKhu1Pz4x7bjDOaifC9t%2Fhgy%2FN3FAoXSQUFFg0cVkXnKKhoA5Hxkt%2BcvkQObSn7Mr1Vi0xPakNoRcEC7k%2BhhR3Vp8oGUEkODLotLSAvkPfj8xL0rfzgYuLI3aY8F77M2Sj7vcDiOB03VOiBddelvOqLTHfYwlktQ81MlQq%2BjQPMX5wo9g7DhD7nwtSBgvozJ0VvmNmMfn%2BKvkgEXo8YbyQakY5PXg2pJE6IjUJTF%2FpKOfO5W2IKYzkqbDgicaMjTKq1Q7zr%2BW0BQKzhsEIjhHkneH2NRiIUiriemEbJVVo9nrMsxviT8Hah7X5YZ5kVGjBpX5owA%3D HTTP_ACCEPT_LANGUAGE: en-us paste.recursive.include: paste.recursive.Includer from / SCRIPT_NAME: REQUEST_METHOD: GET PATH_INFO: / HTTP_ORIGIN: https://login.research.cchmc.org SERVER_PROTOCOL: HTTP/1.1 QUERY_STRING: paste.throw_errors: True CONTENT_LENGTH: 0 weberror.evalexception: weberror.evalexception.middleware.EvalException object at 0x8d02d50 HTTP_USER_AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/536.30.1 (KHTML, like Gecko) Version/6.0.5 Safari/536.30.1 HTTP_CONNECTION: Keep-Alive SERVER_NAME: 0.0.0.0 REMOTE_ADDR: 10.199.194.17 ORGINAL_REMOTE_ADDR: 10.199.92.37 wsgi.url_scheme: http SERVER_PORT: 8080 paste.recursive.forward: paste.recursive.Forwarder from / paste.recursive.script_name: paste.evalexception: weberror.evalexception.middleware.EvalException object at 0x8d02d50 wsgi.input: socket._fileobject object at 0x8d9eb50 length=0 HTTP_HOST: galaxy.research.cchmc.org paste.recursive.include_app_iter: paste.recursive.IncluderAppIter from / wsgi.multithread: True HTTP_CONFVER: 1 HTTP_CACHE_CONTROL: max-age=0 HTTP_ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 wsgi.version: (1, 0) HTTP_AUTH_USER: prakash.velayut...@cchmc.org wsgi.run_once: False wsgi.errors: galaxy.util.pastescript.serve.LazyWriter object at 0x239db10 wsgi.multiprocess: False HTTP_X_FORWARDED_HOST: galaxy.research.cchmc.org HTTP_X_FORWARDED_FOR: 10.199.194.17 CONTENT_TYPE: request_id: 34e3f63274a611e3aaf1005056a84587 paste.httpserver.thread_pool: paste.httpserver.ThreadPool object at 0x8da5750 ORGINAL_HTTP_HOST: bmigalaxyp1.chmcres.cchmc.org:8080 HTTP_UID: VELGE9 [srv-galaxy@bmigalaxyp1 galaxy-dist]$ Obviously, I am logging in using HTTP_AUTH_USER, which does exist in the file, but auth is not going forward. Please note that without the recent changes, I was able to change every instance of REMOTE_USER in the source code with AUTH_USER and that worked without issues. Thanks, Prakash On Jan 3, 2014, at 11:45 AM, Nate Coraor n...@bx.psu.edu wrote: Hi Prakash, This was not previously possible, but I have added a config option for it: https://bitbucket.org/galaxy/galaxy-central/commits/e92e13e9c103cc1f36dff65e1523479bf5cb17ed If you're running the stable branch, you can apply the changes from this commit manually. --nate On Thu, Jan 2, 2014 at 11:09 AM, Jennifer Jackson j...@bx.psu.edu wrote: Hello Prakash, I am going to move this over to the galaxy-...@bx.psu.edu mailing list where it will have greater visibility within our development community. Best, Jen Galaxy team https://wiki.galaxyproject.org/MailingLists#The_lists On 1/2/14 7:27 AM, Velayutham
Re: [galaxy-dev] Local installation problem
On Thu, Nov 21, 2013 at 10:10 PM, Dr. César Rodríguez Sánchez
cesar.rodriguezsanc...@ucr.ac.cr wrote:
Dear Nate,
I am trying to install galaxy on a different computer. This time it does not
run because of the following:
Traceback (most recent call last):
File ./scripts/paster.py, line 33, in module
serve.run()
File /Users/rodriguez/galaxy-dist/lib/galaxy/util/pastescript/serve.py,
line 1049, in run
File /Users/rodriguez/galaxy-dist/lib/galaxy/util/pastescript/serve.py,
line 1055, in invoke
File /Users/rodriguez/galaxy-dist/lib/galaxy/util/pastescript/serve.py,
line 220, in run
File /Users/rodriguez/galaxy-dist/lib/galaxy/util/pastescript/serve.py,
line 643, in command
File
/Users/rodriguez/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line
350, in loadapp
File
/Users/rodriguez/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line
374, in loadobj
File
/Users/rodriguez/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line
399, in loadcontext
File
/Users/rodriguez/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line
423, in _loadconfig
File
/Users/rodriguez/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line
561, in get_context
File
/Users/rodriguez/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line
620, in _context_from_explicit
File
/Users/rodriguez/galaxy-dist/lib/galaxy/util/pastescript/loadwsgi.py, line
125, in import_string
File /Users/rodriguez/galaxy-dist/lib/pkg_resources.py, line 1954, in
load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File /Users/rodriguez/galaxy-dist/lib/galaxy/web/__init__.py, line 4, in
module
File /Users/rodriguez/galaxy-dist/lib/galaxy/web/framework/__init__.py,
line 40, in module
File
/Users/rodriguez/galaxy-dist/eggs/Babel-0.9.4-py2.6.egg/babel/support.py,
line 29, in module
File
/Users/rodriguez/galaxy-dist/eggs/Babel-0.9.4-py2.6.egg/babel/dates.py,
line 34, in module
File
/Users/rodriguez/galaxy-dist/eggs/Babel-0.9.4-py2.6.egg/babel/core.py,
line 642, in default_locale
File
/Users/rodriguez/galaxy-dist/eggs/Babel-0.9.4-py2.6.egg/babel/core.py,
line 763, in parse_locale
ValueError: expected only letters, got 'utf-8'
How can I solve this locale issue?
Hi Cesar,
Sorry I didn't get back to you earlier. If you haven't yet discovered
the solution, this is occurring because of a bug in the version of the
Babel library Galaxy is using. To work around this, set the
environment variable `LC_ALL` to a valid locale, e.g. 'en_US.UTF-8' or
'C', before starting Galaxy:
% export LC_ALL='en_US.UTF-8'
% sh run.sh
--nate
Thanks
Cesar
De: Nate Coraor n...@bx.psu.edu
Fecha: miércoles 20 de noviembre de 2013 08:12
Para: César Rodríguez Sánchez cesar.rodriguezsanc...@ucr.ac.cr
CC: galaxy-...@bx.psu.edu
Asunto: Re: [galaxy-dev] Local installation problem
Hi Cesar,
The easiest way to avoid module version conflicts is to use virtualenv to
set up a self-contained environment:
http://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Use_a_clean_environment
--nate
On Tue, Nov 19, 2013 at 4:02 PM, Cesar Rodriguez
cesar.rodriguezsanc...@ucr.ac.cr wrote:
Hi there,
I am trying to install Galaxy locally on my Mac OSX 10.9, python 2.7.
Nonetheless, when I run it, it says that some eggs are out of date. I have
fetched them, but the following message appears:
Mac-Pro-Cesar-Rodriguez:scripts Rodriguez$ python fetch_eggs.py
Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched
Warning: pycrypto (a dependent egg of Fabric) cannot be fetched
Fetched http://eggs.galaxyproject.org/paramiko/paramiko-1.11.1-py2.7.egg
One of Galaxy's managed eggs depends on something which is missing, this
is almost certainly a bug in the egg distribution.
Dependency paramiko requires pycrypto=2.1,!=2.4
Traceback (most recent call last):
File fetch_eggs.py, line 37, in module
c.resolve() # Only fetch eggs required by the config
File /Users/Rodriguez/galaxy-dist/lib/galaxy/eggs/__init__.py, line
345, in resolve
egg.resolve()
File /Users/Rodriguez/galaxy-dist/lib/galaxy/eggs/__init__.py, line
168, in resolve
dists = pkg_resources.working_set.resolve( (
self.distribution.as_requirement(), ), env, self.fetch )
File /Users/Rodriguez/galaxy-dist/lib/pkg_resources.py, line 569, in
resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (paramiko 1.11.1
(/Users/Rodriguez/galaxy-dist/eggs/paramiko-1.11.1-py2.7.egg),
Requirement.parse('pycrypto=2.1,!=2.4’))
What should I do to fix this?
Thank you so much for your help.
Cesar
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists
Re: [galaxy-dev] Request: Option to reduce server data transfer for big workflow in cluster
Hi Ben, The job running code is in lib/galaxy/jobs/. Galaxy jobs get a wrapper which include the start/finish methods, that's in __init__.py. handler.py is what dispatches jobs out to various runner plugins, finds new jobs to run, and generally controls the operation. runners/*.py are the individual DRM plugins. This is an interesting solution and I'd like to see the implementation. --nate On Thu, Dec 19, 2013 at 6:33 PM, Ben Gift corn8b...@gmail.com wrote: You've been extremely helpful, I appreciate it. So we went ahead and decided that we need this feature. We're planning to have a lot of people running huge pipelines, ones that work best on one node, and there's no reason to do all this writing to any shared file system when it works best on one node using /tmp/ for intermediate step data. So I've been working on that. So far I've made the checkbox for using one node (in run.mako). In workflow.py I catch this and set a new variable in each step of the workflow called use_one_node, if checkbox is checked. Now I'm trying to find where jobs are run, so that I can put the logic in for getting a node to run on, and setting that as a variable on each step. Could you point me in the direction of the files/classes associated with running the history's jobs, and getting nodes (or sending jobs to condor?)? Thanks, and I'll be sure to push this upstream after it's done if you'd like it. Maybe as something you can turn on from the universal_wsgi. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Sysadmin Question
Hi Donny, Galaxy expects data to be copied in to its data store (by default, galaxy-dist/database/files/), and it's not designed to access data in the way you're suggesting. However, there are some workarounds. First, if the data is added to a Data Library (an admin feature, although once a library is created, non-admin users can be given write access to specific libraries), data can be uploaded to libraries without copying it in to Galaxy. This is documented here: https://wiki.galaxyproject.org/Admin/DataLibraries/UploadingLibraryFiles The Galaxy user must have read access to the data in question to be imported. Datasets in Galaxy can also have their path set explicitly, a feature which some Galaxy admins have used to create a Galaxy tool that allows users to browse their local filesystem space and import datasets without copying them. Hope this helps, --nate On Thu, Dec 19, 2013 at 3:23 PM, Shrum, Donald C dcsh...@admin.fsu.edu wrote: Hi, I'm at the Florida State University HPC and setting up an a Galaxy server that will be used to submit jobs to our HPC cluster. I'm using apache as a proxy for the Galaxy server and I'm in the process of setting up ldap authentication. I had planned to mount our HPC file system (with user home directories) on the Galaxy server so that users would have access to their data but I'm wondering if I have the wrong idea about how Galaxy works and if there is a way to map users to their home directories in Galaxy easily. Thanks for any pointers. Donny Shrum ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] a galaxy of issues; Job output not returned from cluster,drmaa getting wrong LSF job id and job.o files not being found and moving files occurring before upload completes.
Hi Starr, I'd suggest setting 'retry_job_output_collection' in universe_wsgi.ini to some value 0 (e.g. 5). You may also want to try remounting the filesystem on which your working directories are located with the `-noac` temporarily just to rule out that the problem is related to attribute caching. The -noac option is not a good idea to use in production due to the performance penalty of disabling it, but it'd be useful for debugging the problem. --nate On Fri, Dec 6, 2013 at 5:47 PM, Hazard, E. Starr haza...@musc.edu wrote: This is a stand alone instance of Galaxy built Nov 28 a small research cluster using LSF and built on RHEL6.2. galaxy.tools.actions.upload_common INFO 2013-12-05 14:33:01,106 tool upload1 created job id 10 Working directory for job is: /depot/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/10 galaxy.jobs.handler DEBUG 2013-12-05 14:33:01,290 (10) Dispatching to drmaa runner galaxy.jobs DEBUG 2013-12-05 14:33:01,441 (10) Persisting job destination (destination id: drmaa) galaxy.jobs.handler INFO 2013-12-05 14:33:01,471 (10) Job dispatched galaxy.tools.deps DEBUG 2013-12-05 14:33:01,693 Building dependency shell command for dependency 'samtools' galaxy.tools.deps WARNING 2013-12-05 14:33:01,693 Failed to resolve dependency on 'samtools', ignoring galaxy.jobs.runners.drmaa DEBUG 2013-12-05 14:33:02,352 (10) submitting file /depot/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/10/ galaxy_10.sh galaxy.jobs.runners.drmaa DEBUG 2013-12-05 14:33:02,352 (10) command is: pythonŠ. galaxy.jobs DEBUG 2013-12-05 14:33:02,379 (10) Changing ownership of working directory with: /usr/bin/sudo -E scripts/external_chown_script.py /depot/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/10 galaxy 50982 galaxy.jobs.runners.drmaa DEBUG 2013-12-05 14:33:02,528 (10) submitting with credentials: galaxy [uid: 50981] galaxy.jobs.runners.drmaa DEBUG 2013-12-05 14:33:02,574 (10) Job script for external submission is: /depot/shared/app/Galaxy/galaxy-dist/database/lsf/10.jt_json galaxy.jobs.runners.drmaa INFO 2013-12-05 14:33:02,772 (10) queued as Job 28526 is submitted to default queue medium_priority. 28526 galaxy.jobs DEBUG 2013-12-05 14:33:02,837 (10) Persisting job destination (destination id: drmaa) galaxy.jobs.runners.drmaa INFO 2013-12-05 14:33:03,237 (10/Job 28526 is submitted to default queue medium_priority. 28526) job left DRM queue with following message: code 18: invalid LSF job id: Job 28526 is submitted to default queue medium_priority. 28526 galaxy.jobs DEBUG 2013-12-05 14:33:03,303 (10) Changing ownership of working directory with: /usr/bin/sudo -E scripts/external_chown_script.py /depot/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/10 galaxy 50982 128.23.163.166 - - [05/Dec/2013:14:33:05 -0400] GET /api/histories/50a7a2e81473b416/contents HTTP/1.1 200 - http://hpcc3.musc.edu:8089/root; Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0 galaxy.jobs.runners ERROR 2013-12-05 14:33:08,661 (10/Job 28526 is submitted to default queue medium_priority. 28526) Job output not returned from cluster: [Errno 2] No such file or directory: '/depot/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/10 /galaxy_10.o' galaxy.jobs DEBUG 2013-12-05 14:33:08,701 (10) Changing ownership of working directory with: /usr/bin/sudo -E scripts/external_chown_script.py /depot/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/10 galaxy 50982 galaxy.jobs DEBUG 2013-12-05 14:33:09,049 finish(): Moved /depot/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/10/ galaxy_dataset_16.dat to /depot/shared/app/Galaxy/galaxy-dist/database/files/000/dataset_16.dat galaxy.jobs DEBUG 2013-12-05 14:33:09,049 finish(): Moved /depot/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/10/ galaxy_dataset_15.dat to /depot/shared/app/Galaxy/galaxy-dist/database/files/000/dataset_15.dat galaxy.jobs DEBUG 2013-12-05 14:33:09,105 setting dataset state to ERROR galaxy.jobs DEBUG 2013-12-05 14:33:09,126 setting dataset state to ERROR galaxy.jobs DEBUG 2013-12-05 14:33:09,214 job 10 ended Job output not returned from cluster² appears in History, BUT in fact files are being written to directories indicated in my universe_wsgi.ini I am having no success getting a solution to this error job left DRM queue with following message: code 18: invalid LSF job id This file No such file or directory: '/depot/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/10 /galaxy_10.o¹ ³ exists AFTER the job finishes. SO at 2013-12-05 14:33:03 the file had not been written but does appear later. This operation is completing before the file can be uploaded and so an empty file is moved Moved /depot/shared/app/Galaxy/galaxy-dist/database/job_working_directory/000/10/ galaxy_dataset_15.dat to
Re: [galaxy-dev] hierarchical ObjectStore - noatime
Hi Björn, atime is definitely not used by Galaxy, and we have generally disabled it on all filesystems we mount. --nate On Tue, Dec 3, 2013 at 8:10 AM, Dannon Baker dannon.ba...@gmail.com wrote: On Tue, Dec 3, 2013 at 5:49 AM, Bjoern Gruening bjoern.gruen...@gmail.com wrote: The new discs will be type=distributed id=primary and the old disc will become type=disk id=secondary. If I understood correctly the old disc is them in some read-only state and will not touched until the primary discs are full or not working ... Correct -- the default HierarchicalObjectStore always writes new files to the first ObjectStore but can read through to any of them. Probably worth noting that, with this base implementation (that could easily be extended to do whatever you'd prefer), when the first objectstore gets full it still won't attempt to write to the second. Is it save to mount the old discs or all discs with noatime, to get a small performance gain? Is Galaxy using noatime? I do believe we're using it, but Nate would be better able to comment on the actual performance gain we see. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local installation problem
Hi Cesar,
The easiest way to avoid module version conflicts is to use virtualenv to
set up a self-contained environment:
http://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer#Use_a_clean_environment
--nate
On Tue, Nov 19, 2013 at 4:02 PM, Cesar Rodriguez
cesar.rodriguezsanc...@ucr.ac.cr wrote:
Hi there,
I am trying to install Galaxy locally on my Mac OSX 10.9, python 2.7.
Nonetheless, when I run it, it says that some eggs are out of date. I have
fetched them, but the following message appears:
Mac-Pro-Cesar-Rodriguez:scripts Rodriguez$ python fetch_eggs.py
Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched
Warning: pycrypto (a dependent egg of Fabric) cannot be fetched
Fetched http://eggs.galaxyproject.org/paramiko/paramiko-1.11.1-py2.7.egg
One of Galaxy's managed eggs depends on something which is missing, this
is almost certainly a bug in the egg distribution.
Dependency paramiko requires pycrypto=2.1,!=2.4
Traceback (most recent call last):
File fetch_eggs.py, line 37, in module
c.resolve() # Only fetch eggs required by the config
File /Users/Rodriguez/galaxy-dist/lib/galaxy/eggs/__init__.py, line
345, in resolve
egg.resolve()
File /Users/Rodriguez/galaxy-dist/lib/galaxy/eggs/__init__.py, line
168, in resolve
dists = pkg_resources.working_set.resolve( (
self.distribution.as_requirement(), ), env, self.fetch )
File /Users/Rodriguez/galaxy-dist/lib/pkg_resources.py, line 569, in
resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (paramiko 1.11.1
(/Users/Rodriguez/galaxy-dist/eggs/paramiko-1.11.1-py2.7.egg),
Requirement.parse('pycrypto=2.1,!=2.4’))
What should I do to fix this?
Thank you so much for your help.
Cesar
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] secure Galaxy with SSL
On Tue, Nov 19, 2013 at 12:32 PM, Jingchao Zhang zh...@unl.edu wrote: Hi Peter, Thanks so much for your help! It is like you said a browser issue. I also noticed that the Galaxy main server (https://usegalaxy.org/) doesn't have this problem. And their USCS main table browser uses https instead of http. Does anyone know why this isn't included in the current Galaxy release? And how can I change my current http USCS main to https USCS main? Hi Jim, We did this for usegalaxy.org simply by locally changing the URL in the tool's config from http to https. A proper fix for this tool (and other external sites which open in the center iframe) is in the works and will be released to the stable branch once it's done. --nate Thanks, Jim Message: 8 Date: Tue, 19 Nov 2013 08:51:03 + From: Peter Briggs peter.bri...@manchester.ac.uk To: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] secure Galaxy with SSL Message-ID: 528b2677.9050...@manchester.ac.uk Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hello Jim Re your problem #1 (UCSC browser appears to be blocked), I've seen something similar with our local Galaxy instance, which is also served via https. In our case I believe this is actually a browser issue: the latest version of Firefox silently blocks mixed secure and insecure content: https://blog.mozilla.org/tanvi/2013/04/10/mixed-content-blocking-enabled-in-firefox-23/ (I think Chrome and IE do something similar, although IE at least gives a warning.) The workaround is either to disable mixed content blocking (not trivial in Firefox, and probably not a good idea in general), or to do something like e.g. right-click Open link in new tab on the Get data/UCSC main table browser link in Galaxy. Once UCSC has loaded in the new tab it can be used to send data back to Galaxy without any problems. HTH Best wishes, Peter On 18/11/13 23:03, Jingchao Zhang wrote: Dear all, Today I installed the SSL module for our local Galaxy instance and the https://; link is working fine. I added this Location/ RequestHeader set X-URL-SCHEME https /Location in our Apache configuration file as instructed in this webpage: http://wiki.galaxyproject.org/Admin/Config/Apache%20Proxy Here are my problems with SSL: 1. Some build in links like UCSC Main http://genome.ucsc.edu/cgi-bin/hgTables?GALAXY_URL=https%3A//hcc-galaxy.unl.edu/tool_runnertool_id=ucsc_table_direct1hgta_compressType=nonesendToGalaxy=1hgta_outputType=bed table browser and UCSC Test http://genome-test.cse.ucsc.edu/cgi-bin/hgTables?GALAXY_URL=https%3A//hcc-galaxy.unl.edu/tool_runnertool_id=ucsc_table_direct_test1hgta_compressType=nonesendToGalaxy=1hgta_outputType=bed table browser become invalid. If I click on them, nothing will happen, as if they are blocked. 2. The old http link still works, which I think shouldn't because I added the RequestHeader ... https line in Apache configuration. I really want to disable the http link because the new users could be easily led to the old one. Both httpd and Galaxy have been restarted after the changes are made. Since I didn't find any similar threads in the mailist, I hope someone here can help me out with this. Thanks, Jim ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Peter Briggs peter.bri...@manchester.ac.uk Bioinformatics Core Facility University of Manchester B.1083 Michael Smith Bldg Tel: (0161) 2751482 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] wigTobigwig error: broken pipe and len file not found
On Sat, Nov 16, 2013 at 12:27 PM, ruiwang.sz ruiwang...@gmail.com wrote: Hi All, I‘m seeing some weird error messages...I googled but didn't see anything useful: So, it is during the wigToBigwig conversion: Dataset generation errors Dataset 47: Wig/BedGraph-to-bigWig on data 43 Tool execution generated the following error message: grep: writing output: Broken pipe grep: writing output: Broken pipe grep: writing output: Broken pipe and many more lines of the same error The tool produced the following additional output: Couldn't open tool-data/shared/ucsc/chrom/mm10.len , No such file or directory but it is there: ls -l tool-data/shared/ucsc/chrom/mm10.len -rw-rw-r-- 1 bioinfoadmin bioinfoadmin 1405 Oct 9 11:33 tool-data/shared/ucsc/chrom/mm10.len Hi Rui, Jobs run in a working directory based on the job id under whatever is configured as job_working_directory in universe_wsgi.ini. Thus, from that directory, tool-data/shared/ ... does not exist. To fix this, set len_file_path in universe_wsgi.ini to an absolute path. --nate From the Galaxy server log, I found these: galaxy.tools WARNING 2013-11-15 17:46:43,200 Failed to resolve dependency on 'ucsc_tools', ignoring galaxy.jobs.runners.local DEBUG 2013-11-15 17:46:43,255 (216) executing: grep -v ^track /home/bioinfoadmin/app/galaxy-dist/database/files/000/dataset_458.dat | wigToBigWig stdin tool-data/shared/ucsc/chrom/mm10.len /home/bioinfoadmin/app/galaxy-dist/database/files/000/dataset_473.dat -blockSize=256 -itemsPerSlot=1024 -clip 21 || echo Error running wigToBigWig. 2 and then, setmetadata set the dataset state to ERROR. I did verify that the result dataset_473.dat has a size 0. However, when I run the command alone: grep -v ^track /home/bioinfoadmin/app/galaxy-dist/database/files/000/dataset_458.dat | wigToBigWig stdin tool-data/shared/ucsc/chrom/mm10.len /home/bioinfoadmin/app/galaxy-dist/database/files/000/dataset_473.dat -blockSize=256 -itemsPerSlot=1024 -clip it successfully generated the dataset_473.dat. also, I look into the wig_to_Bigwig.xml in tools/filters, the command is: grep -v ^track $input1 | wigToBigWig stdin $chromInfo $out_file1 however, I don't see where this $chromInfo was defined, yet it indeed found the the correct value: tool-data/shared/ucsc/chrom/mm10.len. how does that happen? it is very puzzling...I'm not sure if anyone has seen this before, please let me know! Thanks, Rui ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Setting a queue for torque submission
Xian and I were able to resolve this on IRC, the problem is is that no default queue is specified in the PBS client (intentionally) and the `-q` param was improperly handled in the pbs job runner (it expected param id=“destination” instead). This has been fixed in the stable branch in changeset 51b4282dce3a. --nate On Nov 14, 2013, at 4:50 PM, Jennifer Jackson j...@bx.psu.edu wrote: Hello, I am going to move your question over to the galaxt-...@bx.psu.edu mailing list to give it better visibility to the development/local install community. http://wiki.galaxyproject.org/MailingLists#The_lists I didn't find it over here already, but apologies if double posted. Also, since I don't know the best answer, will let others jump in! Best, Jen Galaxy team On 11/14/13 6:35 AM, Xian Su wrote: Greetings I'm having a hard time getting job submission to work via torque. Posted below is the relevant part of my universe file and job_conf.xml. The main cause comes from our torque server requiring a queue to be requested. I can submit jobs just fine via job .sub files via command line using qsub -q M30 Running jobs via webgui results in the following: galaxy.jobs.runners.pbs WARNING 2013-11-14 13:54:23,722 (28) pbs_submit failed (try 5/5), PBS error 15039: Route rejected by all destinations galaxy.jobs.runners.pbs ERROR 2013-11-14 13:54:25,724 (28) All attempts to submit job failed On the torque server's pbs logs, I see the corresponding 11/14/2013 13:54:23;0080;PBS_Server.66159;Req;req_reject;Reject reply code=15039(No default queue specified MSG=requested queue not found), aux=0, type=QueueJob, from xians@podmt1-100-62.novalocal In my universe_wsgi.ini, I'm currently using the settings: start_job_runners = pbs default_cluster_job_runner = pbs:///-q M30 -l nodes=1:ppn=12/ ^I mainly have no idea how to find the correct syntax for this pbs:/// But when I comment out default_cluster_job_runner and instead uncomment the line for the following job conf, I get the same error: ?xml version=1.0? job_conf plugins workers=4 plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner/ plugin id=pbs type=runner load=galaxy.jobs.runners.pbs:PBSJobRunner workers=2/ /plugins handlers default=handlers handler id=main tags=handlers/ /handlers destinations default=pbs destination id=local runner=local/ destination id=pbs runner=pbs tags=mycluster param id=Resource_Listwalltime=72:00:00,nodes=1:ppn=12/param param id=-qM30/param /destination /destinations tools tool id=main handler=main destination=pbs/ /tools limits /limits /job_conf Thank you in advance for any help. Regards, Xian ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Jennifer Hillman-Jackson http://galaxyproject.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
Bash is easily obtained on these systems and I think the extra functionality available in post-Bourne shells ought to be allowed. I also question how many people are going to run tools on *BSD since most underlying analysis tools tend to only target Linux. That said, shell code should be restricted to Bourne-compatible syntax whenever there’s no good reason to use non-Bourne features, e.g. if all you’re doing is `export FOO=foo`, you should not be forcing the use of bash. In cases where bash really is required (say, you’re using arrays), the script should explicitly specify '#!/bin/bash' (or '#!/usr/bin/env bash'?) rather than '#!/bin/sh'. —nate On Nov 11, 2013, at 11:04 AM, James Taylor ja...@jamestaylor.org wrote: This is not an objection, but do we need bash? Can we live with posix sh? We should ask this question about every requirement we introduce. (bash is not part of the default installation of FreeBSD or OpenBSD for example. bash is unfortunately licensed under GPLV3, so if you are trying to create an OS not polluted by viral licensing you don't get bash). On Mon, Oct 7, 2013 at 11:36 AM, John Chilton chil...@msi.umn.edu wrote: My own preference is that we specify at least /bin/sh and /bin/bash are available before utilizing the tool shed. Is there an objection to this from any corner? Is there realistically a system that Galaxy should support that will not have /bin/bash available? -- James Taylor, Associate Professor, Biology/CS, Emory University ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Tool Shed packages for BLAST+ binaries
On Nov 14, 2013, at 12:21 PM, Peter Cock p.j.a.c...@googlemail.com wrote:
On Thu, Nov 14, 2013 at 5:13 PM, Nate Coraor n...@bx.psu.edu wrote:
Bash is easily obtained on these systems and I think the extra functionality
available in post-Bourne shells ought to be allowed. I also question how
many people are going to run tools on *BSD since most underlying
analysis tools tend to only target Linux.
So could bash be declared an expected Galaxy dependency? i.e. The
core system libraries and tools which Tool Authors may assume, and
which Galaxy Administrators should install?
It’s my opinion that it could. I’ll start a discussion about this shortly so
we can hammer out the rest.
That said, shell code should be restricted to Bourne-compatible syntax
whenever there’s no good reason to use non-Bourne features, e.g. if
all you’re doing is `export FOO=foo`, you should not be forcing the use
of bash. In cases where bash really is required (say, you’re using
arrays), the script should explicitly specify '#!/bin/bash' (or
'#!/usr/bin/env
bash'?) rather than '#!/bin/sh'.
I agree that any shell script (e.g. a tool wrapper) which is bash specific
should say that in the hash-bang line, rather than '#!/bin/sh'.
What about command line magic like -num_threads \${GALAXY_SLOTS:-8}
in a command tag using bash specific environment variable default values?
${FOO:-default} is, surprisingly, Bourne-compatible.
What about bash specific if statements in action type=shell_command
sections of a tool_dependencies.xml file (which is what the BLAST+
packages currently use on the main Tool Shed, pending an update
to use arch/os specific tags as tested on the Test Tool Shed)?
This is a bit tricker, since there’s currently no way to specify installation
actions to run on a system other than the Galaxy application server. However,
if we are saying that bash should be available to tools, I’d think there is no
reason to say that it’s not expected to be available to tool installation.
Unfortunately, I believe the current installation methods use subprocesses’
default, which is sh, which is not going to be bash on some systems (on Debian,
it’s dash).
—nate
Peter
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Fw: No peek issue and datasets wrongly reported as Empty
On Nov 7, 2013, at 2:45 PM, Jean-Francois Payotte wrote: Dear Galaxy developers, I know I am not the only one with this issue, as over time I've stumbled on a few mailing-list threads with other users having the same problem. And I know the recommended solution is to use the -noac mount option. (http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster#Unified_Method%29) However, it is said that using this -noac mount option comes with a performance trade-off, so when we first ran into this issue (datasets showing Empty and No peek, even though the file on the hard drive is full of content), we implemented the hack found in this thread: http://dev.list.galaxyproject.org/What-s-causing-this-error-td4141958.html#a4141963 In this thread, John suggested to add a sleep() in the finish_job method of the galaxy_dist/lib/galaxy/jobs/runnersdrmaa.py file. It worked very well for us. Adding a sleep(30) made all the jobs waiting 30 seconds before finishing, but the No peek issue had at least disappear). However, since the latest Galaxy updates, this file (drmaa.py) has been dramastically changed and the finish_job method doesn't exist anymore. Hence, I had to remove this hack, hoping that this issue would have disappeared as well. Unfortunaley, this No peek issue is still there and causing many headaches to some of our workflows users. My question is then: Can I put this sleep(30) in some other place (method and/or file) in order to achieve the same result? I would really like to solve this No peek issue without resorting to the -noac mount option. Actually, I am not even sure our system administrator would allow it. Hi Jean-François, The job runners have been largely refactored into lib/galaxy/jobs/runners/__init__.py, which is where you'll find finish_job(). However, we also recently added some tricks to work around this issue that has solved the problem (for usegalaxy.org, at least) without needing -noac. This is available in Monday's distribution release. Here's the commit: https://bitbucket.org/galaxy/galaxy-central/commits/384240b8cd29963f302a0349476cf83734cfb5df?at=default To use, set retry_job_output_collection 0 in the Galaxy config. --nate Thanks again for your help! Jean-François ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Security vulnerability in Galaxy filtering tools
Hi Ido, Thanks for the feedback. Replies below. On Nov 5, 2013, at 9:54 AM, Ido Tamir wrote: This seems to happen often e.g. http://wiki.galaxyproject.org/DevNewsBriefs/2012_10_23#Compute_Tool_Security_Fix I'm not sure I'd agree that it's often - we've had 4 or 5 vulnerabilities over the life of the project. 2 allowed arbitrary code execution, the others were less severe. a) are there general guidelines in the wiki on how to avoid these problems when creating tools? The guidelines for writing a Galaxy tool are no different from best practices for writing secure code. In specific for this vulnerability, execution of user input should be handled with extreme care, and this tool had some gaps in its input validation and sanitization. For what it's worth, the filter tool (on which the other vulnerable tools were based) is one of the few tools surviving from the very early days of Galaxy, and would not be implemented the same way if written today. b) is there a way to check automatically if all input fields are correctly escaped in a tool? I am not sure how Galaxy could do this. Galaxy sanitizes the command line so that input fields passed to a tool as command line arguments cannot be crafted to exploit the shell's parsing rules. What the tool itself does with its inputs are out of Galaxy's control. --nate A search for security in the wiki brings up: • Admin/Data Libraries/Library Security 0.0k - rev: 1 (current) last modified: 2013-01-02 23:54:33 • Admin/DataLibraries/LibrarySecurity 19.2k - rev: 4 (current) last modified: 2013-01-03 00:12:36 • HelpOnConfiguration/SecurityPolicy 1.9k - rev: 1 (current) last modified: 0 • Learn/Security Features 7.0k - rev: 3 (current) last modified: 2011-09-13 16:52:08 • News/2013_04_08_Galaxy_Security_Release 1.3k - rev: 3 (current) last modified: 2013-04-08 16:56:41 escape does not bring up anything. thank you very much, ido On Nov 5, 2013, at 12:45 AM, Nate Coraor n...@bx.psu.edu wrote: A security vulnerability was recently discovered by John Chilton with Galaxy's Filter data on any column using simple expressions and Filter on ambiguities in polymorphism datasets tools that can allow for arbitrary execution of code on the command line. The fix for these tools has been committed to the Galaxy source. The timing of this commit coincides with the next Galaxy stable release (which has also been pushed out today). To apply the fix and simultaneously update to the new Galaxy stable release, ensure you are on the stable branch and upgrade to the latest changeset: % hg branch stable % hg pull -u For Galaxy installations that administrators are not yet ready to upgrade to the latest release, there are three workarounds. First, for Galaxy installations running on a relatively new version of the stable release (e.g. release_2013.08.12), Galaxy can be updated to the specific changeset that that contains the fix. This will include all of the stable (non-feature) commits that have been accumulated since the 8/12 release plus any new features included with (and prior to) the 8/12 release, but without all of the new features included in the 11/4 release. Ensure you are on the stable branch and then upgrade to the specific changeset: % hg pull -u -r e094c73fed4d Second, the patch can be downloaded and applied manually: % wget -o security.patch https://bitbucket.org/galaxy/galaxy-central/commits/e094c73fed4dc66b589932edb83412cb8b827cd3raw/ and then: % hg patch security.patch or: % patch -p1 security.patch Third, the tools can be completely disabled by removing them from the tool configuration file (by default, tool_conf.xml) and restarting all Galaxy server processes. The relevant lines in tool_conf.xml are: tool file=stats/dna_filtering.xml / tool file=stats/filtering.xml / The full 11/4 Galaxy Distribution News Brief will be available later today and will contain details of changes since the last release. --nate Galaxy Team ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Security vulnerability in Galaxy filtering tools
A security vulnerability was recently discovered by John Chilton with Galaxy's Filter data on any column using simple expressions and Filter on ambiguities in polymorphism datasets tools that can allow for arbitrary execution of code on the command line. The fix for these tools has been committed to the Galaxy source. The timing of this commit coincides with the next Galaxy stable release (which has also been pushed out today). To apply the fix and simultaneously update to the new Galaxy stable release, ensure you are on the stable branch and upgrade to the latest changeset: % hg branch stable % hg pull -u For Galaxy installations that administrators are not yet ready to upgrade to the latest release, there are three workarounds. First, for Galaxy installations running on a relatively new version of the stable release (e.g. release_2013.08.12), Galaxy can be updated to the specific changeset that that contains the fix. This will include all of the stable (non-feature) commits that have been accumulated since the 8/12 release plus any new features included with (and prior to) the 8/12 release, but without all of the new features included in the 11/4 release. Ensure you are on the stable branch and then upgrade to the specific changeset: % hg pull -u -r e094c73fed4d Second, the patch can be downloaded and applied manually: % wget -o security.patch https://bitbucket.org/galaxy/galaxy-central/commits/e094c73fed4dc66b589932edb83412cb8b827cd3raw/ and then: % hg patch security.patch or: % patch -p1 security.patch Third, the tools can be completely disabled by removing them from the tool configuration file (by default, tool_conf.xml) and restarting all Galaxy server processes. The relevant lines in tool_conf.xml are: tool file=stats/dna_filtering.xml / tool file=stats/filtering.xml / The full 11/4 Galaxy Distribution News Brief will be available later today and will contain details of changes since the last release. --nate Galaxy Team ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] jobid exception
Hi Rémy, Do you have some process that restarts Galaxy if it dies? It should not do this itself. Is there anything being logged right before the server restarts that indicates the problem? --nate On Oct 16, 2013, at 4:10 AM, Rémy Dernat wrote: Hi, About the last updates on DRMAA: now the job is submitted, it seems to work. However, galaxy is restarting constantly... So it became very unusable. I hope you will fix this. Regards. 2013/10/14 Rémy Dernat remy...@gmail.com Hi, After a big upgrade, I have an error when I submit job to SGE through drmaa Here is the error I get: galaxy.jobs.runners.drmaa WARNING 2013-10-14 11:26:08,075 (15615/68154) job check resulted in InvalidJobException: code 18: The job specified by the 'jobid' does not exist. The job is launch to sge because I see it with qstat SGE command. On the galaxy Web interface, the job is stuck in yellow (running state), although it is finished. When I submit job by using local destination it works fine. Here is my job_conf.xml file : ?xml version=1.0? !-- A sample job config that explicitly configures job running the way it is configured by default (if there is no explicit config). -- job_conf plugins plugin id=drmaa type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner/ plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner workers=4/ /plugins handlers handler id=main/ /handlers destinations default=sge_default !--destination id=big_jobs runner=drmaa param id=nativeSpecification-P bignodes -R y -pe threads 8/param /destination-- destination id=sge_default runner=drmaa param id=nativeSpecification-q all.q -V/param /destination destination id=local runner=local/ /destinations /job_conf Any help would be usefull. Regards, Rem ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] AttributeError: type object 'InvalidJobException' has no attribute 'name'
On Oct 14, 2013, at 6:07 AM, Peter Cock wrote: On Fri, Oct 11, 2013 at 9:12 AM, Peter Cock p.j.a.c...@googlemail.com wrote: On Thu, Oct 10, 2013 at 8:20 PM, Nate Coraor n...@bx.psu.edu wrote: Hi all, The recent changes to the DRMAA runner are for better handling of job-ending conditions under slurm, but it looks like SGE has different behavior when a job finishes. I'll provide a fix for this shortly, in the meantime, it's fine to use a slightly older version of drmaa.py. --nate Thanks Nate, So far I've only seen this once so it isn't urgent for me. Peter Hi Nate, I see you've fix the name attribute error: https://bitbucket.org/galaxy/galaxy-central/commits/ff76fd33b81cdde1fb270de688ec5e86488ba34d However it seems the underlying problem (job check resulted in: code 18: The job specified by the 'jobid' does not exist.) is affecting other people now: http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-October/017002.html Peter Hi Peter, I'm working on refactoring the DRMAA runner to allow for these different DRM behaviors without duplicating the code. In the interim, I've reverted the change to the DRMAA runner that resulted in the observed behavior in changeset d46b64f12c52. Thanks, --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] RFC: remove trailing semicolons from command line - Broken bowtie2_wrapper on some SGE systems
Hi Björn, Are you certain that SGE is responsible? I ran in to the same thing with bowtie2 with other runners. The double semicolon is invalid syntax in Bourne-compatible shells and so the tool ought to be fixed not to produce them (it happens under certain combinations of options). Have you noticed this happening with any other tools? --nate On Oct 11, 2013, at 8:57 AM, John Chilton wrote: This sounds like a good way to solve the problem, I guess I would modify On Thu, Oct 10, 2013 at 6:10 PM, Björn Grüning bjoern.gruen...@pharmazie.uni-freiburg.de wrote: Dear all, some SGE instances will add automatically an semicolon add the end of each command, resulting in a disrupted job, because ';;' are not allowed. The latest changes to the Bowtie2 wrapper resulting in such a case and a crash in our instance. An easy solution would be to fix the wrapper and omitting the trailing ';' but maybe its better to fix it once for all in the corresponding tools code, patched attached. I'm not familiar with other Job schedulers so I'm seeking for comments ... it there any disadvantage of removing trailing ';' from the command line? Ciao, Bjoern ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] UCSC Main table browser invalid link
On Oct 9, 2013, at 5:40 PM, Gonsky, Rivkah, Ph.D wrote: Link doesn’t work Hi Rivkah, If you're referring to the Galaxy instance at usegalaxy.org, this has been fixed. Thanks, --nate Rivkah Gonsky Inflammatory Bowel and Immunobiology Research Institute Davis 4063 8700 Beverly Blvd Los Angeles, CA 90048 310-423-7624 gon...@cshs.org IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is STRICTLY PROHIBITED. If you have received this message in error, please notify us immediately by calling (310) 423-6428 and destroy the related message. Thank You for your cooperation. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] AttributeError: type object 'InvalidJobException' has no attribute 'name'
Hi all, The recent changes to the DRMAA runner are for better handling of job-ending conditions under slurm, but it looks like SGE has different behavior when a job finishes. I'll provide a fix for this shortly, in the meantime, it's fine to use a slightly older version of drmaa.py. --nate On Oct 10, 2013, at 10:31 AM, Peter Cock wrote: On Tue, Oct 8, 2013 at 5:03 PM, Adhemar azn...@gmail.com wrote: Hi, After the last update I'm getting the following error. The job is submitted to SGE e executed, but galaxy doesn't get the result and keeps showing the job is executing (yellow box). Any clues? Thanks, Adhemar galaxy.jobs.runners ERROR 2013-10-08 13:01:18,488 Unhandled exception checking active jobs Traceback (most recent call last): File /opt/bioinformatics/share/galaxy20130410/lib/galaxy/jobs/runners/__init__.py, line 362, in monitor self.check_watched_items() File /opt/bioinformatics/share/galaxy20130410/lib/galaxy/jobs/runners/drmaa.py, line 217, in check_watched_items log.warning( (%s/%s) job check resulted in %s: %s, galaxy_id_tag, external_job_id, e.__class__.name, e ) AttributeError: type object 'InvalidJobException' has no attribute 'name' Same here, running galaxy-central with an SGE cluster (actually UGE but the same DRMAA wrapper etc) when cancelling several jobs via qdel at the command line: Galaxy.jobs.runners ERROR 2013-10-10 15:16:35,731 Unhandled exception checking active jobs Traceback (most recent call last): File /mnt/galaxy/galaxy-central/lib/galaxy/jobs/runners/__init__.py, line 362, in monitor self.check_watched_items() File /mnt/galaxy/galaxy-central/lib/galaxy/jobs/runners/drmaa.py, line 217, in check_watched_items log.warning( (%s/%s) job check resulted in %s: %s, galaxy_id_tag, external_job_id, e.__class__.name, e ) AttributeError: type object 'InvalidJobException' has no attribute 'name' $ hg branch default [galaxy@ppserver galaxy-central]$ hg heads | more changeset: 11871:c8b55344e779 tag: tip user:Ross Lazarus ross.laza...@gmail.com date:Tue Oct 08 16:30:54 2013 +1100 summary: Proper removal of rgenetics deprecated tool wrappers changeset: 11818:1f0e7ae9e324 branch: stable parent: 11761:a477486bf18e user:Daniel Blankenberg d...@bx.psu.edu date:Sun Sep 29 16:04:31 2013 +1000 summary: Add additional check and slice to _sniffnfix_pg9_hex(). Fixes issue seen when attempting to view saved visualizations. Further investigation may be needed. ... Killing Galaxy and restarting didn't fix this, the errors persist. I tried this fix to solve the attribute error in the logging call: $ hg diff /mnt/galaxy/galaxy-central/lib/galaxy/jobs/runners/drmaa.py diff -r c8b55344e779 lib/galaxy/jobs/runners/drmaa.py --- a/lib/galaxy/jobs/runners/drmaa.pyTue Oct 08 16:30:54 2013 +1100 +++ b/lib/galaxy/jobs/runners/drmaa.pyThu Oct 10 15:21:56 2013 +0100 @@ -214,7 +214,10 @@ state = self.ds.jobStatus( external_job_id ) # TODO: probably need to keep track of InvalidJobException count and remove after it exceeds some configurable except ( drmaa.DrmCommunicationException, drmaa.InternalException, drmaa.InvalidJobException ), e: -log.warning( (%s/%s) job check resulted in %s: %s, galaxy_id_tag, external_job_id, e.__class__.name, e ) +if hasattr(e.__class__, name): +log.warning( (%s/%s) job check resulted in %s: %s, galaxy_id_tag, external_job_id, e.__class__.name, e ) +else: +log.warning( (%s/%s) job check resulted in: %s, galaxy_id_tag, external_job_id, e ) new_watched.append( ajs ) continue except Exception, e: Now I get lots of these lines instead: galaxy.jobs.runners.drmaa WARNING 2013-10-10 15:22:16,489 (251/11372) job check resulted in: code 18: The job specified by the 'jobid' does not exist. galaxy.jobs.runners.drmaa WARNING 2013-10-10 15:22:16,533 (252/11373) job check resulted in: code 18: The job specified by the 'jobid' does not exist. galaxy.jobs.runners.drmaa WARNING 2013-10-10 15:22:17,580 (253/11374) job check resulted in: code 18: The job specified by the 'jobid' does not exist. galaxy.jobs.runners.drmaa WARNING 2013-10-10 15:22:17,624 (254/11375) job check resulted in: code 18: The job specified by the 'jobid' does not exist. galaxy.jobs.runners.drmaa WARNING 2013-10-10 15:22:17,668 (255/11376) job check resulted in: code 18: The job specified by the 'jobid' does not exist. galaxy.jobs.runners.drmaa WARNING 2013-10-10 15:22:17,712 (256/11377) job check resulted in: code 18: The job specified by the 'jobid' does not exist. (this seems to repeat, endlessly) I manually killed the jobs from the Galaxy history, and restarted Galaxy again. That seemed to fix this. If the DRMAA layer says the job was invalid
Re: [galaxy-dev] 'DECLARE CURSOR ... FOR UPDATE/SHARE is not supported'
Hi Sergei,
Your current Galaxy database migration revision (13) is extremely old. How old
was your previous Galaxy installation, and are you certain that the dump you
created was complete, and fully restored?
To solve the original 'DECLARE CURSOR ... FOR UPDATE' problem, you'd need to
update your version of PostgreSQL. 8.1.x is also very old, 9.1 or 9.2 are
recommended.
--nate
On Sep 11, 2013, at 7:58 AM, hogart wrote:
So,
I decided to completely drop the old galaxy psql database (I have a dump of
it) -
hogart# drop database galaxy;
and recreate it again -
[galaxy@cluster ~]$ creatdb
After the launching of the Galaxy it is terminated with the message with
requesting of upgrade the database, from 13 to 115. But the upgrading of
database scheme is also failed:
[galaxy@cluster galaxy-dist]$ sh manage_db.sh upgrade
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/url.py:105:
SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed
from 'postgres' to 'postgresql'. The new URL format is
postgresql[+driver]://user:pass@host/dbname
13 - 14...
Migration script to add support for Pages.
1) Creates Page and PageRevision tables
2) Adds username column to User table
Traceback (most recent call last):
File ./scripts/manage_db.py, line 64, in module
main( repository=repo, url=db_url )
File
/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/shell.py,
line 207, in main
ret = command_func(**kwargs)
File
/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/api.py,
line 186, in upgrade
return _migrate(url, repository, version, upgrade=True, err=err, **opts)
File string, line 2, in _migrate
File
/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/util/__init__.py,
line 159, in with_engine
return f(*a, **kw)
File
/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/api.py,
line 366, in _migrate
schema.runchange(ver, change, changeset.step)
File
/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/schema.py,
line 91, in runchange
change.run(self.engine, step)
File
/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/versioning/script/py.py,
line 145, in run
script_func(engine)
File lib/galaxy/model/migrate/versions/0014_pages.py, line 55, in upgrade
col.create( User_table, index_name='ix_user_username',
unique_name='username' )
File
/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/schema.py,
line 528, in create
engine._run_visitor(visitorcallable, self, connection, **kwargs)
File
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py,
line 2302, in _run_visitor
File
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py,
line 1972, in _run_visitor
File
/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py,
line 53, in traverse_single
ret = super(AlterTableVisitor, self).traverse_single(elem)
File
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/sql/visitors.py,
line 106, in traverse_single
File
/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py,
line 101, in visit_column
self.execute()
File
/export/apps/galaxy/galaxy-dist/eggs/sqlalchemy_migrate-0.7.2-py2.7.egg/migrate/changeset/ansisql.py,
line 42, in execute
return self.connection.execute(self.buffer.getvalue())
File
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py,
line 1449, in execute
File
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py,
line 1628, in _execute_text
File
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py,
line 1698, in _execute_context
File
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/base.py,
line 1691, in _execute_context
File
/export/apps/galaxy/galaxy-dist/eggs/SQLAlchemy-0.7.9-py2.7-linux-x86_64-ucs2.egg/sqlalchemy/engine/default.py,
line 331, in do_execute
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column username of
relation galaxy_user already exists
'\nALTER TABLE galaxy_user ADD username VARCHAR(255)' {}
Since, it was past release of Galaxy (release_2013.06.03), I pulled the
latest one and repeated the procedure, but the result was the same:
[galaxy@cluster galaxy-dist]$ sh run.sh
...
Re: [galaxy-dev] Difficulty dispatching toolshed tool jobs via job_conf.xml
Hi Simon,
This should work, it looks like there may be a bug with handling tool shed IDs
when determining job destinations. You are actually supposed to be able to use
any of:
shed_host/repos/owner/repo/tool_id/version
shed_host/repos/owner/repo/tool_id
tool_id
I'll take a look at this as soon as possible. One thing you might try in the
short term is using the percent encoded tool id in the tool tag in
job_conf.xml:
toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS%3A%20infoseq46/5.0.0
--nate
On Sep 10, 2013, at 10:35 PM, Guest, Simon wrote:
I am having trouble getting a toolshed tool to be dispatched to the
destination I list in the job_conf.xml file.
My job_conf.xml file looks like this:
?xml version=1.0?
job_conf
plugins
plugin id=local type=runner
load=galaxy.jobs.runners.local:LocalJobRunner workers=20/
plugin id=condor type=runner
load=galaxy.jobs.runners.condor:CondorJobRunner /
/plugins
handlers
handler id=main/
/handlers
destinations default=condor
destination id=local runner=local/
destination id=condor runner=condor
!-- With no params, jobs are submitted to the 'vanilla' universe
with:
notification = NEVER
getenv = true
Additional/override query ClassAd params can be specified
with
param tags, e.g
param id=request_cpus8/param
--
/destination
/destinations
tools
tool id=upload1 destination=local/
!-- Upload File --
tool id=ucsc_table_direct1 destination=local/
!-- UCSC Main --
... stuff omitted ...
tool
id=toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS:infoseq46/5.0.0
destination=local/
/tools
/job_conf
So, you can see I have a default destination of condor, but I'm trying to run
my toolshed EMBOSS infoseq tool on local. However, it is stubbornly running
on condor.
In lib/galaxy/tools/__init__.py:1132, I see this comment which got me
wondering:
# In the toolshed context, there is no job config.
Is it possible to define tool destinations for toolshed tools? Are there
some gotchas that I should know about? Any other ideas why my job is
ignoring the config in job_conf.xml? (By the way, I can change say the
upload1 tool to run on Condor by setting its destination in that file, so it
is doing something.) The other thing I saw in the source code is stuff about
old_id and toolshed guids. Do I need to understand this stuff?
The paster.log contains the following when I submit the infoseq job:
147.158.130.216 - - [11/Sep/2013:13:53:34 +1300] GET
/tool_runner?tool_id=toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS%3A%20infoseq46/5.0.0
HTTP/1.1 200 - http://galaxy-dev.agresearch.co.nz/; Mozilla/5.0 (X11;
Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
147.158.130.216 - - [11/Sep/2013:13:53:40 +1300] POST /tool_runner/index
HTTP/1.1 200 -
http://galaxy-dev.agresearch.co.nz/tool_runner?tool_id=toolshed-dev.agresearch.co.nz/toolshed/repos/guestsi/emboss_5_native/EMBOSS%3A%20infoseq46/5.0.0;
Mozilla/5.0 (X11; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
galaxy.jobs DEBUG 2013-09-11 13:53:40,886 (92) Working directory for job is:
/home/galaxy-dev/galaxy/database/job_working_directory/000/92
galaxy.tools DEBUG 2013-09-11 13:53:40,886 Tool::get_job_destination:
{'runner': 'condor', 'legacy': False, 'params': {}, 'tags': None, 'url':
None, 'converted': False, 'id': 'condor'}.
galaxy.jobs.handler DEBUG 2013-09-11 13:53:40,894 (92) Dispatching to condor
runner
(I added the debug output for Tool::get_job_destination to see what was going
on.)
Any ideas?
cheers,
Simon
===
Attention: The information contained in this message and/or attachments
from AgResearch Limited is intended only for the persons or entities
to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipients is prohibited by AgResearch
Limited. If you have received this message in error, please notify the
sender immediately.
===
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
Re: [galaxy-dev] 'DECLARE CURSOR ... FOR UPDATE/SHARE is not supported'
On Sep 11, 2013, at 9:16 AM, hogart wrote: Hi Nate, Yes, db schema is old, but the upgrading (manage_db.sh upgrade) to the current version resulted to the error, as mentioned above. I don't remember the release number of the previous version of Galaxy, but it was installed in the March this year. A didn't restored the dump yet, now I am playinig with the newly created database. In my newbie point of view, it seems like, that dropping of the old database (drop database galaxy; as superuser) was incomplete, - is it possible? Thanks for letting me know, I will try to upgrade the PostgeSQL (the current version is default version of CentOS 5.6). Sorry, I missed the bit where you were starting with an empty database and not reloading your dump of the old one. It does seem possible that your database is not empty on startup, as the username column should not already exist at the time that this migration script runs. When you drop and then create the database, Galaxy should create all the necessary tables and run through all of the migration scripts the first time that you start it. Is it doing this? If it thinks that the database is already at revision 13, I suspect the database is not empty when it starts up. Check your database_connection string in universe_wsgi.ini and make sure it is pointed at the correct database. --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Bismark error
On Sep 4, 2013, at 10:08 AM, Nikos Sidiropoulos wrote: Hi Nate Yes, it is a multiprocess setup. But is it because we have more than one web-servers, handlers or both? Because if it's just the web-servers I could just scale it down to one since it doesn't seem necessary with our number of users. It's would be nice not to have to restart even for the tool-shed tools. The problem in this case is that one of the web server processes loaded the shed tool but none of the handlers did. Scaling back to a single web process won't solve the problem since the handler(s) will still need to load the tool. Unfortunately you'll have to restart all processes (except, technically, the one web process which happened to install the tool). The only way to avoid the restart is to have a single process for all tasks (web serving and job handling), which comes at a performance penalty. --nate Bests, Nikos 2013/9/4 Nate Coraor n...@bx.psu.edu On Sep 4, 2013, at 9:17 AM, Nikos Sidiropoulos wrote: Hi Bjørn that does not look like a bismark error. Is it happen with other tools as well? No, I have only experienced it with Bismark. Not sure, sorry. But you should migrate to the new job configuration, better sooner than later. I am starting to get that idea. There are jobs still running on the server. When they're done I'll migrate it and get back to you if the problem is fixed or not. Hi Nikos, The old-style configuration is still supported for now. I suspect this may be a problem of the tool not loading in the handler application. Are you running a multiprocess Galaxy setup? If so, you will need to restart all Galaxy processes after installing tools from the tool shed. --nate Bests, Nikos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Bismark error
On Sep 4, 2013, at 9:17 AM, Nikos Sidiropoulos wrote: Hi Bjørn that does not look like a bismark error. Is it happen with other tools as well? No, I have only experienced it with Bismark. Not sure, sorry. But you should migrate to the new job configuration, better sooner than later. I am starting to get that idea. There are jobs still running on the server. When they're done I'll migrate it and get back to you if the problem is fixed or not. Hi Nikos, The old-style configuration is still supported for now. I suspect this may be a problem of the tool not loading in the handler application. Are you running a multiprocess Galaxy setup? If so, you will need to restart all Galaxy processes after installing tools from the tool shed. --nate Bests, Nikos ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local installation: Uploading fastq file takes ages - why?
On Aug 14, 2013, at 8:39 AM, Boaz Shaanan wrote: Hi, On my local galaxy installation (updated to the latest version yesterday), it takes ages (hours!) to upload a fastq files. I attach a report from the the computer on which galaxy is installed. The fastq file is uploaded from 132.72.88.186. Initially there is an error message (the attached file also includes the debug url at the end) and after a while the upload proceeds but takes very long time. Transferring the file by sftp between the two computers takes only seconds. Is there anytning wrong with my installation (i.e. something is not properly defined or something like this). Hi Boaz, Could you make sure that debug and use_interactive are both set to False in universe_wsgi.ini? Thanks, --nate I'd appreciate your help. Boaz Boaz Shaanan, Ph.D. Dept. of Life Sciences Ben-Gurion University of the Negev Beer-Sheva 84105 Israel E-mail: bshaa...@bgu.ac.il Phone: 972-8-647-2220 Skype: boaz.shaanan Fax: 972-8-647-2992 or 972-8-646-1710 galaxy_local_server_error.txt___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Help with cluster setup
On Aug 14, 2013, at 7:48 AM, Jurgens de Bruin wrote: Hi just to keep things up to date I have the the cluster up and running jobs are being submitted. Last problem I am facing is: 21: UCSC Main on Pig: refGene (chr18:1-61220071) error An error occurred with this dataset: The remote data source application may be off line, please try again later. Error: [Errno socket error] [Errno 101] Network is unreachable Hi Jurgens, Your cluster node will need the ability to connect to the Internet to connect to external data sources. Many clusters use private IP space and therefore must rely on NAT for a connection to the greater Internet. --nate On 13 August 2013 16:53, Jurgens de Bruin debrui...@gmail.com wrote: I am totally lost on what is happening now, I have Galaxy running but jobs are not being run: This is my setup: torque: qmgr -c 'p s' # # Create queues and set their attributes. # # # Create and define queue batch # create queue batch set queue batch queue_type = Execution set queue batch resources_default.nodes = 1 set queue batch resources_default.walltime = 01:00:00 set queue batch enabled = True set queue batch started = True # # Set server attributes. # set server scheduling = True set server acl_hosts = manager set server managers = root@* set server managers += jurgens@* set server operators = galaxy@* set server operators += jurgens@* set server operators += root@* set server default_queue = batch set server log_events = 511 set server mail_from = adm set server scheduler_iteration = 600 set server node_check_rate = 150 set server tcp_timeout = 300 set server job_stat_rate = 45 set server poll_jobs = True set server mom_job_sync = True set server keep_completed = 300 set server next_job_number = 17 set server moab_array_compatible = True This is my job_conf.xml ?xml version=1.0? !-- A sample job config that explicitly configures job running the way it is configured by default (if there is no explicit config). -- job_conf plugins !-- plugin id=drmaa type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner workers=4/ -- plugin id=drmaa type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner/ /plugins handlers default=batch handler id=cn01 tags=batch/ handler id=cn02 tags=batch/ /handlers destinations default=batch destination id=batch runner=drmaa tag=cluster,batch param id=nativeSpecfication-q batch/param /destination /destinations /job_conf This is parts of the universe_wsgi.ini # Configuration of the internal HTTP server. [server:main] # The internal HTTP server to use. Currently only Paste is provided. This # option is required. use = egg:Paste#http # The port on which to listen. port = 8989 # The address on which to listen. By default, only listen to localhost (Galaxy # will not be accessible over the network). Use '0.0.0.0' to listen on all # available network interfaces. #host = 127.0.0.1 host = 0.0.0.0 # Use a threadpool for the web server instead of creating a thread for each # request. use_threadpool = True # Number of threads in the web server thread pool. #threadpool_workers = 10 # Set the number of seconds a thread can work before you should kill it (assuming it will never finish) to 3 hours. threadpool_kill_thread_limit = 10800 [server:cn01] use = egg:Paste#http port = 8090 host = 127.0.0.1 use_threadpool = true threadpool_worker = 5 [server:cn02] use = egg:Paste#http port = 8091 host = 127.0.0.1 use_threadpool = true threadpool_worker = 5 Where cn01 and cn02 are cluster nodes echo $DRMAA_LIBRARY_PATH /usr/local/lib/libdrmaa.so On 8 August 2013 16:58, Nate Coraor n...@bx.psu.edu wrote: On Aug 7, 2013, at 9:23 PM, shenwiyn wrote: Yes,and I also have the same confuse about that.Actually when I set server:id in the universe_wsgi.ini as follows for a try,my Galaxy doesn't work with Cluster,if I remove server:id,it work . Hi Shenwiyn, Are you starting all of the servers that you have defined in universe_wsgi.ini? If using run.sh, setting GALAXY_RUN_ALL in the environment will do this for you: http://wiki.galaxyproject.org/Admin/Config/Performance/Scaling [server:node01] use = egg:Paste#http port = 8080 host = 0.0.0.0 use_threadpool = true threadpool_workers = 5 This is my job_conf.xml : ?xml version=1.0? job_conf plugins workers=4 plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner workers=4/ plugin id=pbs type=runner load=galaxy.jobs.runners.pbs:PBSJobRunner workers=8/ /plugins handlers default=batch handler id=node01 tags=batch/ handler id=node02 tags=batch/ /handlers destinations default=regularjobs destination id=local runner=local/ destination id
Re: [galaxy-dev] Postgresql database cleaning
On Aug 26, 2013, at 5:03 AM, Christophe Antoniewski wrote: Hi everybody, The python scripts to clean histories, datasets, users etc.. are fine... However, the records are not really removed from the postgresql database and as a result, this one gets bigger and bigger with unused records. Ours is above 1 To after 2 years of production. Is there a safe way to clean the database from unused records and their dependencies to reduce it size, without being a postgresql guru ? Hi Chris, The database maintains a permanent record of everything that was done, even though the underlying data can be removed. There are a lot of dependencies between objects in Galaxy and removing records, especially anything with a foreign key, could easily result in a lot of problems with all kinds of things, from the UI to running jobs. Because of this, records cannot be removed from the database. --nate Chris -- Christophe Antoniewski Drosophila Genetics and Epigenetics Laboratoire de Biolologie du Développement 9, Quai St Bernard, Boîte courrier 24 75252 Paris Cedex 05 Tel+33 1 44 27 34 39 Fax+33 1 44 27 34 45 Mobile+33 6 68 60 51 50 http://drosophile.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Postgresql database cleaning
On Aug 29, 2013, at 11:50 AM, Nate Coraor wrote: On Aug 26, 2013, at 5:03 AM, Christophe Antoniewski wrote: Hi everybody, The python scripts to clean histories, datasets, users etc.. are fine... However, the records are not really removed from the postgresql database and as a result, this one gets bigger and bigger with unused records. Ours is above 1 To after 2 years of production. Is there a safe way to clean the database from unused records and their dependencies to reduce it size, without being a postgresql guru ? Hi Chris, The database maintains a permanent record of everything that was done, even though the underlying data can be removed. There are a lot of dependencies between objects in Galaxy and removing records, especially anything with a foreign key, could easily result in a lot of problems with all kinds of things, from the UI to running jobs. Because of this, records cannot be removed from the database. Somehow I missed that you said your database is 1 TB - that should not be the case. Unless your database is not being vacuumed or you create objects at an extreme rate, it seems as though something has been stored in it that should not have. --nate --nate Chris -- Christophe Antoniewski Drosophila Genetics and Epigenetics Laboratoire de Biolologie du Développement 9, Quai St Bernard, Boîte courrier 24 75252 Paris Cedex 05 Tel +33 1 44 27 34 39 Fax +33 1 44 27 34 45 Mobile +33 6 68 60 51 50 http://drosophile.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] tool_dependencies.xml format
On Aug 26, 2013, at 11:59 AM, James Taylor wrote: On Mon, Aug 26, 2013 at 11:48 AM, John Chilton chil...@msi.umn.edu wrote: I think it is interesting that there was push back on providing infrastructure (tool actions) for obtaining CBL from github and performing installs based on it because it was not in the tool shed and therefore less reproducible, but the team believes infrastructure should be put in place to support pypi. Well, first, I'm not sure what the team believes, I'm stating what I believe and engaging in a discussion with the community. At some point this should evolve into what we are actually going to do and be codified in a spec as a Trello card, which is even then not set in stone. Second, I'm not suggesting we depend on PyPI. The nice thing about the second format I proposed on galaxy-dev is that we can easily parse out the URL and archive that file. Then someday we could provide a fallback repository where if the PyPI URL no longer works we still have it stored. I concur here, the experience and lessons learned by long-established package and dependency managers can provide some useful guidance for us going forward. APT has long relied on a model of archiving upstream source (as well as distro-generated binary (dpkg) packages), cataloging changes as a set of patches, and maintaining an understanding of installed files, even those meant to be user-edited. I think there is a strong advantage for us doing this as well. I think we all value reproduciblity here, but we make different calculations on what is reproducible. I think in terms of implementing the ideas James has laid out or similar things I have proposed, it might be beneficial to have some final answers on what external resources are allowed - both for obtaining a Galaxy IUC gold star and for the tool shed providing infrastructure to support their usage. My focus is ensuring that we can archive things that pass through the toolshed. Tarballs from *anywhere* are easy enough to deal with. External version control repositories are a bit more challenging, especially when you are pulling just a particular file out, so that's where things got a little hinky for me. Since we don't have the archival mechanism in place yet anyway, this is more a philosophical discussion and setting the right precedent. And yes, keeping an archive of all the software in the world is a scary prospect, though compared to the amount of data we currently keep for people it is a blip. And I'm not sure how else we can really achieve the level of reproducibility we desire. One additional step that will assist with long-term archival is generating static metadata and allowing the packaging and dependency systems to work outside of the Galaxy and Tool Shed applications. A package metadata catalog and package format that provided descriptions of packages on a generic webserver and installable without a running Galaxy instance are components that I believe are fairly important. As for user-edited files, the env.sh files, which are generated at install-time and then essentially untracked afterward scare me a bit. I think it'd be useful for the packaging system have a tighter concept of environment management. These are just my opinions, of course, and are going to be very APT/dpkg-biased simply due to my experience with and favor for Debian-based distros and dependency/package management, but I think there are useful concepts in this (and other systems) that we can draw from. Along those lines, one more idea I had thrown out a while ago was coming up with a way to incorporate (or at least automatically process so that we can convert to our format) the build definitions for other systems like MacPorts, BSD ports/pkgsrc, dpkg, rpm, etc. so that we can leverage the existing rules for building across our target platforms that have already been worked out by other package maintainers with more time. I think this aligns pretty well with Brad's thinking with CloudBioLinux, the difference in implementation being that we require multiple installable versions and platform independence. I am a bit worried that as we go down the repackage (almost) all dependencies path (which I do think is the right path), we also run the risk of most of our packages being out of date. That's almost a guaranteed outcome when even the huge packaging projects (Debian, Ubuntu, etc.) are rife with out-of-date packages. So being able to incorporate upstream build definitions may help us package dependencies quickly. --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] how to add settings of TMPDIR=/scratch for sge/drmaa in universe_wsgi.ini
On Aug 14, 2013, at 9:46 PM, tin h wrote:
Hello gurus,
I made some slight progress...
If I specify a handlers section into the job_conf.xml file with:
handlers
handler id=main/
/handlers
Then galaxy would at least start. However, it is not very functional, I
can't upload files or run job, it says:
An error occurred with this dataset:Unable to run job due to a
misconfiguration of the Galaxy job running system. Please contact a site
administrator.
My universe_wsgi.conf is fairly stock, the [server:main] section is certainly
there. The most relevant changes are:
start_job_runners = drmaa
default_cluster_job_runner = drmaa://-q default.q -V -v TMPDIR=/scratch/
If someone can point me to a job_conf.xml so that I can set the TMPDIR param
correctly for qsub (sge), that would be greatly appreciated!
Thanks
-Tin
Hi Tin,
The start_job_runners and default_cluster_job_runner parameters are ignored
once you switch to the new XML configuration. The problem is most likely that
the 'runner' attribute on your destination (drmaa) does not match the 'id'
attribute of the DRMAAJobRunner plugin (sge), so try changing one or the
other so that they match. More details regarding the error should be shown in
the log.
--nate
On Wed, Aug 14, 2013 at 8:04 AM, Jennifer Jackson j...@bx.psu.edu wrote:
Original Message
Subject: Re: how to add settings of TMPDIR=/scratch for sge/drmaa in
universe_wsgi.ini
Date: Tue, 13 Aug 2013 23:21:16 -0700
From: tin h tin6...@gmail.com
To: Jennifer Jackson j...@bx.psu.edu
CC: Galaxy Dev galaxy-...@bx.psu.edu
Thank you Jennifer.
I am not sure if my previous email got gabled up...
Just to be sure, I specified the following in universe_wsgi.ini
default_cluster_job_runner = drmaa://-q default.q -V -v TMPDIR=/scratch/
when i start galaxy, looking at the console log, this is what I see:
galaxy.jobs DEBUG 2013-08-14 14:08:14,885 Loaded job runner
'galaxy.jobs.runners.drmaa:DRMAAJobRunner' as 'drmaa'
galaxy.jobs.runners.drmaa DEBUG 2013-08-14 14:08:14,885 Converted URL
'drmaa://-q default.q -V -v TMPDIR=/scratch/' to destination runner=drmaa,
params={'nativeSpecification': '-q default.q -V -v TMPDIR='}
galaxy.jobs DEBUG 2013-08-14 14:08:14,885 Legacy destination with id
'drmaa://-q default.q -V -v TMPDIR=/scratch/', url 'drmaa://-q default.q -V
-v TMPDIR=/scratch/' converted, got params:
galaxy.jobs DEBUG 2013-08-14 14:08:14,885 nativeSpecification: -q
default.q -V -v TMPDIR=
The parser has stopped at the / before scratch and ignored the rest :(
I did refer to
http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster and tried to
define job_conf.xml as
?xml version=1.0?
job_conf
plugins
plugin id=local type=runner
load=galaxy.jobs.runners.local:LocalJobRunner workers=4/
plugin id=sge type=runner
load=galaxy.jobs.runners.drmaa:DRMAAJobRunner/
/plugins
destinations default=sge_default
destination id=sge_default runner=drmaa
param id=nativeSpecification-q default.q -V -v
TMPDIR=/scratch/param
/destination
/destinations
/job_conf
But then galaxy won't start, giving me the error of:
Traceback (most recent call last):
File /usr/prog/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py,
line 37, in app_factory
app = UniverseApplication( global_conf = global_conf, **kwargs )
File /usr/prog/galaxy/galaxy-dist/lib/galaxy/app.py, line 95, in __init__
self.job_config = jobs.JobConfiguration(self)
File /usr/prog/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py, line 107,
in __init__
self.__parse_job_conf_xml(tree)
File /usr/prog/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py, line 157,
in __parse_job_conf_xml
self.default_handler_id = self.__get_default(handlers,
self.handlers.keys())
File /usr/prog/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py, line 286,
in __get_default
rval = parent.get('default')
AttributeError: 'NoneType' object has no attribute 'get'
I am not sure what the default refers to, doesn't seems to be the
destinations default specified as I tried changing that name but the error
message remains the same.
Any help so that I can pass TMPDIR=/scratch into SGE qsub would be greatly
appreciated.
-Tin
--
I am riding to fundraise for the Cystic Fibrosis Foundation. Please consider
a tax deductible donation at
http://www.cff.org/LWC/dsp_DonationPage.cfm?idEvent=23815idUser=615989
~~~__oEngineering is the art of making
compromises.
~~~ _ _ Science is the reverse engineering of
the compromises made by nature.
~~~ (_)/(_)Medicine is the hacking of the
scientific knowledge base. - A comp sci student :-)
On 8/13/13 7:28 PM,
Re: [galaxy-dev] Jobs remain in queue until restart
On Aug 2, 2013, at 1:06 PM, Thon de Boer wrote: I did some more investigation of this issue I do notice that my 4 core, 8 slot VM machine has a load of 32, with only my 4 handler processes running (Plus my web server), but not even getting more than 10% of the CPU each. There seems to be some process in my handlers that takes an incredible amount of resources, even though TOP is not showing that (Show below) Has anyone have any idea how to figure out where the bottleneck is? Is there a way to turn on more detailed logging perhaps to see what each process is doing? My IT guy suggested there may be some “context Switching” going on due to the many threads that are running (I use a threadpool of 7 for each server), but not sure how to address that issue… Hi Thon, It looks like it's probably the memory use - if you restart the Galaxy processes, do you see any change? --nate Anyone? top - 10:00:53 up 37 days, 19:29, 8 users, load average: 32.10, 32.10, 32.09 Tasks: 181 total, 1 running, 180 sleeping, 0 stopped, 0 zombie Cpu(s): 4.8%us, 2.5%sy, 0.0%ni, 92.5%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 16334504k total, 16164084k used, 170420k free, 127720k buffers Swap: 4194296k total,15228k used, 4179068k free, 2460252k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 7190 svcgalax 20 0 2721m 284m 5976 S 9.9 1.8 142:53.84 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler3 --pid-file=handler3.pid --log-file=handler3.log --daemon 7183 svcgalax 20 0 2720m 286m 5984 S 6.4 1.8 135:52.63 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler2 --pid-file=handler2.pid --log-file=handler2.log --daemon 7175 svcgalax 20 0 2720m 287m 5976 S 5.6 1.8 117:59.40 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler1 --pid-file=handler1.pid --log-file=handler1.log --daemon 7166 svcgalax 20 0 3442m 2.7g 4884 S 4.6 17.5 74:31.66 python ./scripts/paster.py serve universe_wsgi.ini --server-name=web0 --pid-file=web0.pid --log-file=web0.log --daemon 7172 svcgalax 20 0 2720m 294m 5984 S 4.0 1.8 133:17.19 python ./scripts/paster.py serve universe_wsgi.ini --server-name=handler0 --pid-file=handler0.pid --log-file=handler0.log --daemon 1564 root 20 0 291m 13m 7552 S 0.3 0.1 1:49.65 /usr/sbin/httpd 7890 svcgalax 20 0 17216 1456 1036 S 0.3 0.0 2:15.73 top 10682 apache20 0 297m 11m 3516 S 0.3 0.1 0:02.23 /usr/sbin/httpd 11224 apache20 0 295m 11m 3236 S 0.3 0.1 0:00.29 /usr/sbin/httpd 11263 svcgalax 20 0 17248 1460 1036 R 0.3 0.0 0:00.06 top 1 root 20 0 21320 1040 784 S 0.0 0.0 0:00.95 /sbin/init 2 root 20 0 000 S 0.0 0.0 0:00.01 [kthreadd] 3 root RT 0 000 S 0.0 0.0 0:06.35 [migration/0] Regards, Thon Thon deBoer Ph.D., Bioinformatics Guru California, USA |p: +1 (650) 799-6839 |m: thondeb...@me.com From: galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Thon Deboer Sent: Wednesday, July 17, 2013 11:31 PM To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Jobs remain in queue until restart Hi, I have noticed that from time to time, the job queue seems to be “stuck” and can only be unstuck by restarting galaxy. The jobs seem to be in the queue state and the python job handler processes are hardly ticking over and the cluster is empty. When I restart, the startup procedure realizes all jobs are in the a “new state” and it then assigns a jobhandler after which the jobs start fine…. Any ideas? Thon P.S I am using the june version of galaxy and I DO set limits on my users in job_conf.xml as so: (Maybe it is related? Before it went into dormant mode, this user had started lots of jobs and may have hit the limit, but I assumed this limit was the number of running jobs at one time, right?) ?xml version=1.0? job_conf plugins workers=4 !-- workers is the number of threads for the runner's work queue. The default from plugins is used if not defined for a plugin. -- plugin id=local type=runner load=galaxy.jobs.runners.local:LocalJobRunner workers=2/ plugin id=drmaa type=runner load=galaxy.jobs.runners.drmaa:DRMAAJobRunner workers=8/ plugin id=cli type=runner load=galaxy.jobs.runners.cli:ShellJobRunner workers=2/ /plugins handlers default=handlers !-- Additional job handlers - the id should match the name of a [server:id] in universe_wsgi.ini. -- handler id=handler0 tags=handlers/ handler id=handler1 tags=handlers/ handler id=handler2 tags=handlers/ handler id=handler3 tags=handlers/ !-- handler id=handler10 tags=handlers/ handler id=handler11
Re: [galaxy-dev] Upstart script to manage a multi instance load balanced installation
On Aug 9, 2013, at 11:53 AM, Seth Sims wrote:
Dear Nate,
Adding su - galaxy as the first line of the pre-start script seems to
work reasonably well. Also it looks like the line that sets the egg cache is
not working properly. My egg cache ends up being /tmp/${SERVER_NAME}_egg/
but things still seem to be working so I've changed that part to use one
directory in /tmp/ for all instances.
Hi Seth,
Is your Galaxy user's home directory /srv/galaxy-dist? Otherwise, `su -
galaxy` would change the working directory to that user's home directory and
the rest of the script would fail.
I was thinking it might work to just use `su -c` for individual commands, e.g.:
pre-start script
echo checking python version
su - galaxy -c cd /srv/galaxy-dist ; python ./scripts/check_python.py
[ $? -ne 0 ] exit 1
echo pre-fetching tossing out expired eggs
su - galaxy -c cd /srv/galaxy-dist ; python ./scripts/check_eggs.py -q
if [ $? -eq 0 ]; then
echo Some eggs are out of date, attempting to fetch...
su - galaxy -c cd /srv/galaxy-dist ; python ./scripts/fetch_eggs.py
if [ $? -eq 0 ]; then
echo Fetch Successful.
else
echo Fetch failed.
fi
fi
echo starting servers
SERVERS=`sed -n 's/^\[server:\(.*\)\]/\1/ p' universe_wsgi.ini | xargs
echo`
for SERVER in ${SERVERS} ; do
echo starting server ${SERVER}
start galaxy-worker SERVER_NAME=${SERVER}
done
end script
Sincerely,
Seth Sims
*galaxy.conf*
author Seth Sims seth.s...@gmail.com
version 0.0.2
description galaxy master process. Fetches eggs and spawns all of the
servers it finds configured
start on started network-services
# put galaxy root directory here
chdir /srv/galaxy-dist/
pre-start script
su - galaxy
date
echo checking python version
python ./scripts/check_python.py
[ $? -ne 0 ] exit 1
echo pre-fetching tossing out expired eggs
python ./scripts/check_eggs.py -q
if [ $? -eq 0 ]; then
echo Some eggs are out of date, attempting to fetch...
python ./scripts/fetch_eggs.py
if [ $? -eq 0 ]; then
echo Fetch Successful.
else
echo Fetch failed.
fi
fi
echo starting servers
SERVERS=`sed -n 's/^\[server:\(.*\)\]/\1/ p' universe_wsgi.ini | xargs
echo`
for SERVER in ${SERVERS} ; do
echo starting server ${SERVER}
start galaxy-worker SERVER_NAME=${SERVER}
done
end script
post-stop script
SERVERS=`sed -n 's/^\[server:\(.*\)\]/\1/ p' universe_wsgi.ini | xargs
echo`
date
echo stopping galaxy servers.
for SERVER in ${SERVERS} ; do
echo stopping ${SERVER}
stop galaxy-worker SERVER_NAME=${SERVER}
done
end script
---
*galaxy-worker*
author Seth Sims seth.s...@gmail.com
version 0.0.2
description Starts a galaxy server instance. This is run from the
galaxy.conf file
instance $SERVER_NAME
#make sure we are running as the galaxy user
setuid galaxy
setgid galaxy
#put the galaxy root directory here
chdir /srv/galaxy-dist/
#system users don't have /home/ directories; so point the egg unpack to a
directory in /tmp/
env PYTHON_EGG_CACHE=/tmp/galaxy_eggs/
respawn
script
exec python ./scripts/paster.py serve universe_wsgi.ini
--server-name=${SERVER_NAME}
end script
On Thu, Aug 8, 2013 at 1:52 PM, Nate Coraor n...@bx.psu.edu wrote:
On Jul 15, 2013, at 6:29 PM, Seth Sims wrote:
After some work i've created an Upstart script which can manage a load
balanced galaxy configuration as described in the wiki. I thought that I
would put it on this list for other people to use. The script parses
universe_wsgi.ini just like run.sh and spawns all of the servers it finds.
It comes in two pieces galaxy.conf and galaxy-worker.conf. Once you place
them both in /etc/init and make the proper edits for the environment a
server can be started with sudo start galaxy. The configuration starts
the server at boot time and puts all of the instances under the management
of upstart which deals with pids, logging to syslog and respawning if an
instance crashes.
I have just gotten this working reasonably well but have done basically no
testing so there are bugs to be found. Any comments are welcome if anyone
knows a better way to do something here.
- Seth
Hi Seth,
Thanks for submitting these. I was about to commit them to the contrib/
directory along with the rest of the start scripts, but I was wondering if
you could avoid running the check/fetch scripts as root by just using `su -c`?
--nate
*galaxy.conf*
author Seth Sims seth.s...@gmail.com
version 0.0.1 test
description galaxy master process. Fetches eggs and spawns all of the
servers it finds
Re: [galaxy-dev] support pbkdf2 in proftpd 1.3.5rc3
On Aug 9, 2013, at 2:38 AM, Leon Mei wrote: Hi Nate, Thanks for the suggestion! Unfortunately, it still failed :( I got the following error message in proftp log: 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: enteringpostgres cmd_escapestring 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: enteringpostgres cmd_open 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: connection 'default' count is now 2 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: exiting postgres cmd_open 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: enteringpostgres cmd_close 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: connection 'default' count is now 1 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: exiting postgres cmd_close 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: exiting postgres cmd_escapestring 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: cache hit for user 'hailiang.m...@nbic.nl' 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: cmd_check 2013-08-09 08:32:41,777 mod_sql/4.3[32384]: checking password using SQLAuthType 'sha1' 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: 'sha1' SQLAuthType handler reports failure 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: checking password using SQLAuthType 'sha256' 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: 'sha256' SQLAuthType handler reports failure 2013-08-09 08:32:41,781 mod_sql/4.3[32384]: checking password using SQLAuthType 'pbkdf2' 2013-08-09 08:32:41,841 mod_sql/4.3[32384]: 'pbkdf2' SQLAuthType handler reports failure 2013-08-09 08:32:41,841 mod_sql/4.3[32384]: cmd_check 2013-08-09 08:32:41,841 mod_sql/4.3[32384]: cmd_auth The old user account generated before our code update still works. I wonder how it is configured at the Galaxy main server? Thanks, Leon It isn't in use on the Main server, but now that I'm aware that ProFTPD has PBKDF2 support, I will put this on my to-do list for next week to test. --nate On Thu, Aug 8, 2013 at 8:45 PM, Nate Coraor n...@bx.psu.edu wrote: On Jul 26, 2013, at 3:51 PM, Leon Mei wrote: Dear galaxy developers, We have tried today to upgrade our proftpd configuration to make uploading for our galaxy users possible again, both for users with old as well as new style hashed passwords. We upgraded proftpd on the server to 1.3.5rc3 and have the following SQL part in our configuration file based on the post of http://dev.list.galaxyproject.org/ProFTPD-integration-with-Galaxy-td4660295.html SQLEngine on SQLLogFile /var/log/proftpd-sql.log SQLBackend postgres SQLConnectInfo galaxy@localhost:5840 galaxyftp [ourpassword] SQLAuthTypesSHA1 SHA256 PBKDF2 SQLPasswordPBKDF2 SHA256 1000 24 SQLPasswordUserSalt sql:/GetUserSalt SQLAuthenticate users SQLDefaultUID 108 SQLDefaultGID 116 SQLDefaultHomedir /opt/cloudman/pkg/proftpd/var SQLUserInfo custom:/LookupGalaxyUser SQLNamedQuery LookupGalaxyUser SELECT email, (CASE WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password from 38 for 32) ELSE password END) AS password2,'108','116','/mnt/galaxyData/tmp/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U' SQLNamedQuery GetUserSalt SELECT (CASE WHEN SUBSTRING (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16) END) AS salt FROM galaxy_user WHERE email='%U' We have executed the LookupGalaxyUser and GetUserSalt commands manually, and the results look good. Now, old users can login via ftp, but for a new user, the authentication still fails: 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: cmd_check 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: checking password using SQLAuthType 'sha1' 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: 'sha1' SQLAuthType handler reports failure 2013-07-26 13:15:06,989 mod_sql/4.3[31761]: checking password using SQLAuthType 'pbkdf2' 2013-07-26 13:15:06,993 mod_sql/4.3[31761]: 'pbkdf2' SQLAuthType handler reports failure What are we missing? Thanks! Rob and Leon Hallo Leon and Rob, Thanks for working on this, when I'd looked a couple months ago I could not find an entirely-ProFTPD way to do this. I think it may have actually come about because I asked about it on their IRC channel. ;) This may work if you change SQLPasswordPBKDF2: SQLPasswordPBKDF2 SHA256 1 24 It'd be great if ProFTPD also supported pulling those values dynamically from the database, but Galaxy's PBKDF2 code currently has them hardcoded, so they will be static anyway. --nate -- Hailiang (Leon) Mei Netherlands Bioinformatics Center BioAssist NGS Taskforce - http://ngs.nbic.nl Skype: leon_meiMobile: +31 6 41709231 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions
