Re: [galaxy-dev] Serious start-up delay in next-stable
I had a similar issue a while ago. Greg mentioned the following setting in universe: # Enable use of an in-memory registry with bi-directional relationships # between repositories (i.e., in addition to lists of dependencies for a # repository, keep an in-memory registry of dependent items for each repository. #manage_dependency_relationships = True Setting this to false made the delay go away. I haven't had issues so far after disabling. See here for details: http://lists.bx.psu.edu/pipermail/galaxy-dev/2014-March/018821.html Best, Geert On 08/05/2014 09:01 PM, Dannon Baker wrote: There's definitely a request that will occasionally block at startup -- I've hit this before too, Martin. Looking into it now. On Tue, Aug 5, 2014 at 2:59 PM, Martin Čech mar...@bx.psu.edu mailto:mar...@bx.psu.edu wrote: I just freshly cloned galaxy-central, switched to next-stable and did initial startup - no problems detected and Galaxy booted in under 2m. On Tue, Aug 5, 2014 at 2:54 PM, Langhorst, Brad langho...@neb.com mailto:langho...@neb.com wrote: I’ve noticed that startup is very slow. It’s about 15 minutes for me to stop and restart my 4 daemons. It was much faster with galaxy on local storage. brad -- Bradley W. Langhorst, Ph.D. Applications and Product Development Scientist On Aug 5, 2014, at 2:41 PM, Björn Grüning bjoern.gruen...@gmail.com mailto:bjoern.gruen...@gmail.com wrote: Hi all, does anyone encountered a long start-up delay with the recent next-stable and starting a instance from scratch? For me its hanging at: tool_shed.galaxy_install.migrate.check DEBUG 2014-08-05 18:39:45,207 psycopg2 egg successfully loaded for postgres dialect for 3-5 minutes. Cheers, Bjoern ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandewe...@ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/in/geertvandeweyer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Issues with dependencies for toolshed packages
Hi Team, I have been trying to install package_snpEff_3_5 from the Galaxy main tool shed and package_snpEff_3_6 from the Galaxy test tool shed and have the same issue. Once the installer completes it shows that the dependency snpEff.x.x was not installed. When I try to install the dependency it reports that the directory /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/tmp/tmp-toolshed-mtdi48tQV/snpEff does not exist. I have checked to confirm that this is true and discovered that rather then a directory I find a null file named snpEff and the snpEff_v3.x_core.zip file. I checked my shedtool dependencies folder and the directory for these tools are empty. I have since uninstalled the package. I just tried again to install the snpEff tool and am getting the followig error message: Internal Server Error Galaxy was unable to successfully complete your request URL: http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413 Module galaxy.web.framework.middleware.error:149 in __call__ http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# app_iter = self.application(environ, sr_checker) Module paste.recursive:84 in __call__ http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# body = method( trans, **kwargs ) Module galaxy.web.framework:377 in decorator http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# return func( self, trans, *args, **kwargs ) Module galaxy.webapps.galaxy.controllers.admin_toolshed:1035 in prepare_for_install http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# raw_text = common_util.tool_shed_get( trans.app, tool_shed_url, url ) Module tool_shed.util.common_util:310 in tool_shed_get http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# response = urlopener.open( uri ) Module urllib2:395 in open http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# response = meth(req, response) Module urllib2:508 in http_response http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# 'http', request, response, code, msg, hdrs) Module urllib2:427 in error http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# result = self._call_chain(*args) Module urllib2:367 in _call_chain http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# result = func(*args) Module urllib2:603 in http_error_302 http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# return self.parent.open(new) Module urllib2:395 in open http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# response = meth(req, response) Module urllib2:508 in http_response http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# 'http', request, response, code, msg, hdrs) Module urllib2:433 in error
Re: [galaxy-dev] Main Galaxy Tool Shed is running the next-stable branch
Hi Eric, Sorry for the inconvenience. The tool shed server was out of space. I've corrected the problem and have successfully installed repositories into my local Galaxy instance. Please try again and let us know if you encounter problems. Greg Von Kuster On Aug 6, 2014, at 10:59 AM, Eric Kuyt erick...@gmail.com wrote: Hi Greg, Could there be some problems with https://toolshed.g2.bx.psu.edu/ I have some troubles installing toolshed tools in our galaxy. When I try installing for instance 'suite_samtools_0_1_19', I am presented with a 500 error. I tried then pulling a new stable galaxy from bitbucket, which seems to have the same problem. Maybe I'm doing something wrong both times or toolshed is having problems. I opt for the latter of course. -- part of galaxy traceback -- File '/usr/lib/python2.7/urllib2.py', line 527 in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error Thanks, Eric On 29 July 2014 23:35, Greg Von Kuster g...@bx.psu.edu wrote: Hello all, The main Galaxy Tool Shed is now running the next-stable branch ( e6813f244a2a (next-stable) tip ) in preparation for next upcoming Galaxy release. Please let us know if you encounter anything strange or broken and we'll get things fixed asap. The main Tool Shed will be updated to the stable branch when it is created for the upcoming Galaxy release. Thanks very much, Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Issues with dependencies for toolshed packages
Hi Iry, Sorry for the inconvenience. The tool shed server was out of space - I've corrected the problem and have successfully installed repositories. Please try again and let us know if you encounter problems. Thanks! Greg Von Kuster On Aug 6, 2014, at 12:20 PM, Iry Witham iry.wit...@jax.org wrote: Hi Team, I have been trying to install package_snpEff_3_5 from the Galaxy main tool shed and package_snpEff_3_6 from the Galaxy test tool shed and have the same issue. Once the installer completes it shows that the dependency snpEff.x.x was not installed. When I try to install the dependency it reports that the directory /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/tmp/tmp-toolshed-mtdi48tQV/snpEff does not exist. I have checked to confirm that this is true and discovered that rather then a directory I find a null file named snpEff and the snpEff_v3.x_core.zip file. I checked my shedtool dependencies folder and the directory for these tools are empty. I have since uninstalled the package. I just tried again to install the snpEff tool and am getting the followig error message: Internal Server Error Galaxy was unable to successfully complete your request URL: http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413 Module galaxy.web.framework.middleware.error:149 in __call__ app_iter = self.application(environ, sr_checker) Module paste.recursive:84 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:633 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:132 in __call__ return self.handle_request( environ, start_response ) Module galaxy.web.framework.base:190 in handle_request body = method( trans, **kwargs ) Module galaxy.web.framework:377 in decorator return func( self, trans, *args, **kwargs ) Module galaxy.webapps.galaxy.controllers.admin_toolshed:1035 in prepare_for_install raw_text = common_util.tool_shed_get( trans.app, tool_shed_url, url ) Module tool_shed.util.common_util:310 in tool_shed_get response = urlopener.open( uri ) Module urllib2:395 in open response = meth(req, response) Module urllib2:508 in http_response 'http', request, response, code, msg, hdrs) Module urllib2:427 in error result = self._call_chain(*args) Module urllib2:367 in _call_chain result = func(*args) Module urllib2:603 in http_error_302 return self.parent.open(new) Module urllib2:395 in open response = meth(req, response) Module urllib2:508 in http_response 'http', request, response, code, msg, hdrs) Module urllib2:433 in error return self._call_chain(*args) Module urllib2:367 in _call_chain result = func(*args) Module urllib2:516 in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error extra data full traceback text version This may be an intermittent problem due to load or other unpredictable factors, reloading the page may address the problem. I appreciate any possible assistance, Regards, __ Iry T. Witham Scientific Applications Administrator Scientific Computing Group Computational Sciences Dept. The Jackson Laboratory 600 Main Street Bar Harbor, ME 04609 Phone: 207-288-6744 email: iry.wit...@jax.org 372D007A-1B00-4668-BA6B-F0527C1F24BE[34][3][1].png The information in this email, including attachments, may be confidential and is intended solely for the addressee(s). If you believe you received this email by mistake, please notify the sender by return email as soon as possible. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Main Galaxy Tool Shed is running the next-stable branch
Hi Greg, Could there be some problems with https://toolshed.g2.bx.psu.edu/ I have some troubles installing toolshed tools in our galaxy. When I try installing for instance 'suite_samtools_0_1_19', I am presented with a 500 error. I tried then pulling a new stable galaxy from bitbucket, which seems to have the same problem. Maybe I'm doing something wrong both times or toolshed is having problems. I opt for the latter of course. -- part of galaxy traceback -- File '/usr/lib/python2.7/urllib2.py', line 527 in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 500: Internal Server Error Thanks, Eric On 29 July 2014 23:35, Greg Von Kuster g...@bx.psu.edu wrote: Hello all, The main Galaxy Tool Shed is now running the next-stable branch ( e6813f244a2a (next-stable) tip ) in preparation for next upcoming Galaxy release. Please let us know if you encounter anything strange or broken and we'll get things fixed asap. The main Tool Shed will be updated to the stable branch when it is created for the upcoming Galaxy release. Thanks very much, Greg Von Kuster ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] ToolShed error: samtools installation failure
Could someone please take a look? Thanks, Rui On Tue, Aug 5, 2014 at 8:27 PM, Beginner TI ruiwang...@gmail.com wrote: Dear Admin, Since the upgrade, I lost the samtools that I installed myself. Although it is still in the tool_conf.xml, it does not show up on the panel. I tried to install from toolshed, but failed. Error messages pasted below. Could someone take a look? Thanks, Rui ⇝ AttributeError: 'NoneType' object has no attribute 'strip' URL: http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977 Module weberror.evalexception.middleware:*364* in respond http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# app_iter *=* self*.*application*(*environ*,* detect_start_response*)* Module paste.recursive:*84* in __call__ http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# *return* self*.*application*(*environ*,* start_response*)* Module paste.httpexceptions:*633* in __call__ http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# *return* self*.*application*(*environ*,* start_response*)* Module galaxy.web.framework.base:*132* in __call__ http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# *return* self*.*handle_request*(* environ*,* start_response *)* Module galaxy.web.framework.base:*190* in handle_request http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# body *=* method*(* trans*,* kwargs *)* Module galaxy.web.framework:*369* in decorator http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# *return* func*(* self*,* trans*,* ***args*,* kwargs *)* Module galaxy.webapps.galaxy.controllers.admin_toolshed:*983* in prepare_for_install http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977#
Re: [galaxy-dev] Issues with dependencies for toolshed packages
Hi Greg, That resolved the internal server error issue. Now I need to figure out the issue with the dependencies. I was unable to post the actual error message previously, but am doing so now. Here is the error I get for the dependency installation: NameVersion TypeStatus Error snpEff http://galaxy2/admin_toolshed/manage_repository_tool_dependencies?repository_id=b1254334b8763c9coperation=browsetool_dependency_ids=81424a9b99396c5a 3.6 package Error File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py, line 110, in install_and_build_package_via_fabric tool_dependency = self.install_and_build_package( app, tool_shed_repository, tool_dependency, actions_dict ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py, line 103, in install_and_build_package dir = tmp_dir File /usr/lib64/python2.6/contextlib.py, line 34, in __exit__ self.gen.throw(type, value, traceback) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/Fabric-1.7.0-py2.6.egg/fabric/context_managers.py, line 142, in _setenv yield File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py, line 68, in install_and_build_package initial_download=True ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py, line 32, in execute_step initial_download=initial_download ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py, line 397, in execute_step dir = td_common_util.url_download( work_dir, downloaded_filename, url, extract=True ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py, line 560, in url_download extraction_path = archive.extract( install_dir ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py, line 60, in extract self.archive.extractall( os.path.join( path ) ) File /usr/lib64/python2.6/zipfile.py, line 940, in extractall self.extract(zipinfo, path, pwd) File /usr/lib64/python2.6/zipfile.py, line 928, in extract return self._extract_member(member, path, pwd) File /usr/lib64/python2.6/zipfile.py, line 965, in _extract_member target = file(targetpath, wb) [Errno 20] Not a directory: 'database/tmp/tmp-toolshed-mtdCAL19a/snpEff/snpEff.config' This current package is owned by jjohnson and is located in the test toolshed. Thanks, Iry From: Greg Von Kuster g...@bx.psu.edumailto:g...@bx.psu.edu Date: Wednesday, August 6, 2014 12:34 PM To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Issues with dependencies for toolshed packages Hi Iry, Sorry for the inconvenience. The tool shed server was out of space - I've corrected the problem and have successfully installed repositories. Please try again and let us know if you encounter problems. Thanks! Greg Von Kuster On Aug 6, 2014, at 12:20 PM, Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org wrote: Hi Team, I have been trying to install package_snpEff_3_5 from the Galaxy main tool shed and package_snpEff_3_6 from the Galaxy test tool shed and have the same issue. Once the installer completes it shows that the dependency snpEff.x.x was not installed. When I try to install the dependency it reports that the directory /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/tmp/tmp-toolshed-mtdi48tQV/snpEff does not exist. I have checked to confirm that this is true and discovered that rather then a directory I find a null file named snpEff and the snpEff_v3.x_core.zip file. I checked my shedtool dependencies folder and the directory for these tools are empty. I have since uninstalled the package. I just tried again to install the snpEff tool and am getting the followig error message: Internal Server Error Galaxy was unable to successfully complete your request URL: http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413 Module galaxy.web.framework.middleware.error:149 in __call__ http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# app_iter = self.application(environ,sr_checker) Module paste.recursive:84 in __call__ http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# return self.application(environ,start_response) Module paste.httpexceptions:633 in __call__
Re: [galaxy-dev] Issues with dependencies for toolshed packages
Iry, are you seeing the same error with: https://toolshed.g2.bx.psu.edu/view/iuc/snpeff Cheers, Bjoern 2014-08-06 19:12 GMT+02:00 Iry Witham iry.wit...@jax.org: Hi Greg, That resolved the internal server error issue. Now I need to figure out the issue with the dependencies. I was unable to post the actual error message previously, but am doing so now. Here is the error I get for the dependency installation: Name Version Type Status Error snpEff http://galaxy2/admin_toolshed/manage_repository_tool_dependencies?repository_id=b1254334b8763c9coperation=browsetool_dependency_ids=81424a9b99396c5a 3.6 package Error File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py, line 110, in install_and_build_package_via_fabric tool_dependency = self.install_and_build_package( app, tool_shed_repository, tool_dependency, actions_dict ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py, line 103, in install_and_build_package dir = tmp_dir File /usr/lib64/python2.6/contextlib.py, line 34, in __exit__ self.gen.throw(type, value, traceback) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/Fabric-1.7.0-py2.6.egg/fabric/context_managers.py, line 142, in _setenv yield File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py, line 68, in install_and_build_package initial_download=True ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py, line 32, in execute_step initial_download=initial_download ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py, line 397, in execute_step dir = td_common_util.url_download( work_dir, downloaded_filename, url, extract=True ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py, line 560, in url_download extraction_path = archive.extract( install_dir ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py, line 60, in extract self.archive.extractall( os.path.join( path ) ) File /usr/lib64/python2.6/zipfile.py, line 940, in extractall self.extract(zipinfo, path, pwd) File /usr/lib64/python2.6/zipfile.py, line 928, in extract return self._extract_member(member, path, pwd) File /usr/lib64/python2.6/zipfile.py, line 965, in _extract_member target = file(targetpath, wb) [Errno 20] Not a directory: 'database/tmp/tmp-toolshed-mtdCAL19a/snpEff/snpEff.config' This current package is owned by jjohnson and is located in the test toolshed. Thanks, Iry From: Greg Von Kuster g...@bx.psu.edu Date: Wednesday, August 6, 2014 12:34 PM To: Iry Witham iry.wit...@jax.org Cc: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Issues with dependencies for toolshed packages Hi Iry, Sorry for the inconvenience. The tool shed server was out of space - I've corrected the problem and have successfully installed repositories. Please try again and let us know if you encounter problems. Thanks! Greg Von Kuster On Aug 6, 2014, at 12:20 PM, Iry Witham iry.wit...@jax.org wrote: Hi Team, I have been trying to install package_snpEff_3_5 from the Galaxy main tool shed and package_snpEff_3_6 from the Galaxy test tool shed and have the same issue. Once the installer completes it shows that the dependency snpEff.x.x was not installed. When I try to install the dependency it reports that the directory /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/database/tmp/tmp-toolshed-mtdi48tQV/snpEff does not exist. I have checked to confirm that this is true and discovered that rather then a directory I find a null file named snpEff and the snpEff_v3.x_core.zip file. I checked my shedtool dependencies folder and the directory for these tools are empty. I have since uninstalled the package. I just tried again to install the snpEff tool and am getting the followig error message: Internal Server Error Galaxy was unable to successfully complete your request URL: http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413 Module galaxy.web.framework.middleware.error:*149* in __call__ http://galaxy2/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=76ad6cd032d7117echangeset_revisions=04d2a4f81413# app_iter *=* self*.*application*(*environ*,*sr_checker*)* Module paste.recursive:*84* in __call__
Re: [galaxy-dev] ToolShed error: samtools installation failure
Hi Rui, We'll definitely take a look. Please allow for a reasonable time period to pass before sending duplicate emails to the list asking for assistance. That said, does deleting your integrated_tool_panel.xml (this file is automatically regenerated by Galaxy, safe to remove it and restart) bring back your manually installed samtools? On Wed, Aug 6, 2014 at 1:08 PM, Beginner TI ruiwang...@gmail.com wrote: Could someone please take a look? Thanks, Rui On Tue, Aug 5, 2014 at 8:27 PM, Beginner TI ruiwang...@gmail.com wrote: Dear Admin, Since the upgrade, I lost the samtools that I installed myself. Although it is still in the tool_conf.xml, it does not show up on the panel. I tried to install from toolshed, but failed. Error messages pasted below. Could someone take a look? Thanks, Rui ⇝ AttributeError: 'NoneType' object has no attribute 'strip' URL: http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977 Module weberror.evalexception.middleware:*364* in respond http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# app_iter *=* self*.*application*(*environ*,* detect_start_response*)* Module paste.recursive:*84* in __call__ http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# *return* self*.*application*(*environ*,* start_response*)* Module paste.httpexceptions:*633* in __call__ http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# *return* self*.*application*(*environ*,* start_response*)* Module galaxy.web.framework.base:*132* in __call__ http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# *return* self*.*handle_request*(* environ*,* start_response *)* Module galaxy.web.framework.base:*190* in handle_request http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# body *=* method*(* trans*,* kwargs *)* Module galaxy.web.framework:*369* in decorator http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# *return* func*(* self*,* trans*,* ***args*,* kwargs *)* Module galaxy.webapps.galaxy.controllers.admin_toolshed:*983* in prepare_for_install
Re: [galaxy-dev] ToolShed error: samtools installation failure
Hi Dannon, Thanks for the reply. Sorry for the duplicate. I was wondering that the original subject sounded like a user problem so I updated it... Actually we are having a little down time right now for a file system check. I'll try your suggestions asap and let you know. At the mean time, could you please elaborate this a little bit? I noticed that in the tool_conf.xml, not every tool is shown on the panel although they are all defined there in the same format. I also noticed that all the tool xml files, such as section id=samtools name=NGS: SAM Tools tool file=samtools/sam_bitwise_flag_filter.xml / tool file=samtools/sam2interval.xml / tool file=samtools/sam_to_bam.xml / tool file=samtools/bam_to_sam.xml / tool file=samtools/sam_merge.xml / tool file=samtools/samtools_mpileup.xml / tool file=samtools/sam_pileup.xml / tool file=samtools/pileup_parser.xml / tool file=samtools/pileup_interval.xml / tool file=samtools/samtools_flagstat.xml / tool file=samtools/samtools_rmdup.xml / tool file=samtools/samtools_slice_bam.xml / /section the directory samtools and all these xml files are no longer in the tools directory. seems that upgrade deleted them. is this correct? or they are moved to somewhere else? I looked around but didn't see. Based on this, I think that even I removed the tool_panel.xml file you mentioned, I will still not be able to restore samtools even I could see it...could you please let me know how I could configure this? Thanks, Rui On Wed, Aug 6, 2014 at 10:23 AM, Dannon Baker dannon.ba...@gmail.com wrote: Hi Rui, We'll definitely take a look. Please allow for a reasonable time period to pass before sending duplicate emails to the list asking for assistance. That said, does deleting your integrated_tool_panel.xml (this file is automatically regenerated by Galaxy, safe to remove it and restart) bring back your manually installed samtools? On Wed, Aug 6, 2014 at 1:08 PM, Beginner TI ruiwang...@gmail.com wrote: Could someone please take a look? Thanks, Rui On Tue, Aug 5, 2014 at 8:27 PM, Beginner TI ruiwang...@gmail.com wrote: Dear Admin, Since the upgrade, I lost the samtools that I installed myself. Although it is still in the tool_conf.xml, it does not show up on the panel. I tried to install from toolshed, but failed. Error messages pasted below. Could someone take a look? Thanks, Rui ⇝ AttributeError: 'NoneType' object has no attribute 'strip' URL: http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977 Module weberror.evalexception.middleware:*364* in respond http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# app_iter *=* self*.*application*(*environ*,* detect_start_response*)* Module paste.recursive:*84* in __call__ http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# *return* self*.*application*(*environ*,* start_response*)* Module paste.httpexceptions:*633* in __call__ http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# *return* self*.*application*(*environ*,* start_response*)* Module galaxy.web.framework.base:*132* in __call__ http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977#
Re: [galaxy-dev] ToolShed error: samtools installation failure
Ahh, I think I see the issue. So, many galaxy tools are being migrated out of the distribution and to the toolshed -- you can find much more detailed information here: https://wiki.galaxyproject.org/MigratingToolsFromGalaxyDistribution The samtools wrappers were migrated in migration 8, I think, if that helps. Basically, there's a migration script that should have shown up in a terminal somewhere when you ran your galaxy instance after the update that will contain a reference to script(s) that you can run that'll take care of the migration. Let me know if you need more information, if the wiki page doesn't clear it up! -Dannon On Wed, Aug 6, 2014 at 1:36 PM, Beginner TI ruiwang...@gmail.com wrote: Hi Dannon, Thanks for the reply. Sorry for the duplicate. I was wondering that the original subject sounded like a user problem so I updated it... Actually we are having a little down time right now for a file system check. I'll try your suggestions asap and let you know. At the mean time, could you please elaborate this a little bit? I noticed that in the tool_conf.xml, not every tool is shown on the panel although they are all defined there in the same format. I also noticed that all the tool xml files, such as section id=samtools name=NGS: SAM Tools tool file=samtools/sam_bitwise_flag_filter.xml / tool file=samtools/sam2interval.xml / tool file=samtools/sam_to_bam.xml / tool file=samtools/bam_to_sam.xml / tool file=samtools/sam_merge.xml / tool file=samtools/samtools_mpileup.xml / tool file=samtools/sam_pileup.xml / tool file=samtools/pileup_parser.xml / tool file=samtools/pileup_interval.xml / tool file=samtools/samtools_flagstat.xml / tool file=samtools/samtools_rmdup.xml / tool file=samtools/samtools_slice_bam.xml / /section the directory samtools and all these xml files are no longer in the tools directory. seems that upgrade deleted them. is this correct? or they are moved to somewhere else? I looked around but didn't see. Based on this, I think that even I removed the tool_panel.xml file you mentioned, I will still not be able to restore samtools even I could see it...could you please let me know how I could configure this? Thanks, Rui On Wed, Aug 6, 2014 at 10:23 AM, Dannon Baker dannon.ba...@gmail.com wrote: Hi Rui, We'll definitely take a look. Please allow for a reasonable time period to pass before sending duplicate emails to the list asking for assistance. That said, does deleting your integrated_tool_panel.xml (this file is automatically regenerated by Galaxy, safe to remove it and restart) bring back your manually installed samtools? On Wed, Aug 6, 2014 at 1:08 PM, Beginner TI ruiwang...@gmail.com wrote: Could someone please take a look? Thanks, Rui On Tue, Aug 5, 2014 at 8:27 PM, Beginner TI ruiwang...@gmail.com wrote: Dear Admin, Since the upgrade, I lost the samtools that I installed myself. Although it is still in the tool_conf.xml, it does not show up on the panel. I tried to install from toolshed, but failed. Error messages pasted below. Could someone take a look? Thanks, Rui ⇝ AttributeError: 'NoneType' object has no attribute 'strip' URL: http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977 Module weberror.evalexception.middleware:*364* in respond http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# app_iter *=* self*.*application*(*environ*,* detect_start_response*)* Module paste.recursive:*84* in __call__ http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# *return* self*.*application*(*environ*,* start_response*)* Module paste.httpexceptions:*633* in __call__ http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977# view
Re: [galaxy-dev] Issues with dependencies for toolshed packages
Bjoern, I am not getting the same error. However, it is still failing. I get the following banner: This tool dependency is not installed correctly (see the Tool dependency installation error below). Choose Uninstall this tool dependency from the Repository Actions menu, correct problems if necessary, and try installing the dependency again. And the following information is listed: Tool shed repository: snpeff Tool shed repository changeset revision: e8adfc4c0a6b Tool dependency status: Error Tool dependency installation error: This tool dependency's required tool dependency snpEff version 3.4 has status Error. Tool dependency installation directory: ./../shed_tools/tool_dependencies/snpEff/3.4/iuc/snpeff/e8adfc4c0a6b Contents: snpEffhttp://galaxy2/admin_toolshed/browse_tool_dependency?repository_id=0dd27a44d4d97a53operation=browsetool_dependency_ids=53b036f027c29984# env.shhttp://galaxy2/admin_toolshed/browse_tool_dependency?repository_id=0dd27a44d4d97a53operation=browsetool_dependency_ids=53b036f027c29984# if [ -f /hpcdata/galaxy-test/galaxy-setup/shed_tools/tool_dependencies/snpEff/3.4/iuc/package_snpeff_3_4/af7ff8187d09/env.sh ] ; then . /hpcdata/galaxy-test/galaxy-setup/shed_tools/tool_dependencies/snpEff/3.4/iuc/package_snpeff_3_4/af7ff8187d09/env.sh ; fi Thanks, Iry From: bjoern.gruen...@googlemail.commailto:bjoern.gruen...@googlemail.com bjoern.gruen...@gmail.commailto:bjoern.gruen...@gmail.com Date: Wednesday, August 6, 2014 1:16 PM To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org Cc: Greg Von Kuster g...@bx.psu.edumailto:g...@bx.psu.edu, galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Issues with dependencies for toolshed packages Iry, are you seeing the same error with: https://toolshed.g2.bx.psu.edu/view/iuc/snpeff Cheers, Bjoern 2014-08-06 19:12 GMT+02:00 Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org: Hi Greg, That resolved the internal server error issue. Now I need to figure out the issue with the dependencies. I was unable to post the actual error message previously, but am doing so now. Here is the error I get for the dependency installation: NameVersion TypeStatus Error snpEff http://galaxy2/admin_toolshed/manage_repository_tool_dependencies?repository_id=b1254334b8763c9coperation=browsetool_dependency_ids=81424a9b99396c5a 3.6 package Error File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py, line 110, in install_and_build_package_via_fabric tool_dependency = self.install_and_build_package( app, tool_shed_repository, tool_dependency, actions_dict ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py, line 103, in install_and_build_package dir = tmp_dir File /usr/lib64/python2.6/contextlib.py, line 34, in __exit__ self.gen.throw(type, value, traceback) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/eggs/Fabric-1.7.0-py2.6.egg/fabric/context_managers.py, line 142, in _setenv yield File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/install_manager.py, line 68, in install_and_build_package initial_download=True ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/recipe_manager.py, line 32, in execute_step initial_download=initial_download ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/recipe/step_handler.py, line 397, in execute_step dir = td_common_util.url_download( work_dir, downloaded_filename, url, extract=True ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py, line 560, in url_download extraction_path = archive.extract( install_dir ) File /hpcdata/galaxy-test/galaxy-setup/galaxy-dist/lib/tool_shed/galaxy_install/tool_dependencies/td_common_util.py, line 60, in extract self.archive.extractall( os.path.join( path ) ) File /usr/lib64/python2.6/zipfile.py, line 940, in extractall self.extract(zipinfo, path, pwd) File /usr/lib64/python2.6/zipfile.py, line 928, in extract return self._extract_member(member, path, pwd) File /usr/lib64/python2.6/zipfile.py, line 965, in _extract_member target = file(targetpath, wb) [Errno 20] Not a directory: 'database/tmp/tmp-toolshed-mtdCAL19a/snpEff/snpEff.config' This current package is owned by jjohnson and is located in the test toolshed. Thanks, Iry From: Greg Von Kuster g...@bx.psu.edumailto:g...@bx.psu.edu Date: Wednesday, August 6, 2014 12:34 PM To: Iry Witham iry.wit...@jax.orgmailto:iry.wit...@jax.org Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Issues with
Re: [galaxy-dev] ToolShed error: samtools installation failure
Ah, I think that I had a wrong impression before...so by default, the migration scripts listed here will NOT be executed, am I correct? I think that 'done' at the bottom made me think that it was executed... So it's the user's responsibility to execute those scripts, correct? Thanks, Rui The Emboss 5.0.0 tools have been eliminated from the distribution and the Emboss datatypes have been removed fromdatatypes_conf.xml.sample. You should remove the Emboss datatypes from your version of datatypes_conf.xml. Therepositories named emboss_5 and emboss_datatypes from the main Galaxy Tool Shed at http://toolshed.g2.bx.psu.eduwill be installed into your local Galaxy instance at the location discussed above by running the following command.vsh ./scripts/migrate_tools/0002_tools.sh^doneAfter the installation process finishes, you can start your Galaxy server. As part of this installation process,entries for each of the following tool config files will be added to the file named ./migrated_tool_conf.xml, so thesetools will continue to be loaded into your tool panel. Because of this, existing entries for these files should beremoved from your file named ./tool_conf.xml, but only after the installation process finishes. On Wed, Aug 6, 2014 at 10:53 AM, Dannon Baker dannon.ba...@gmail.com wrote: Ahh, I think I see the issue. So, many galaxy tools are being migrated out of the distribution and to the toolshed -- you can find much more detailed information here: https://wiki.galaxyproject.org/MigratingToolsFromGalaxyDistribution The samtools wrappers were migrated in migration 8, I think, if that helps. Basically, there's a migration script that should have shown up in a terminal somewhere when you ran your galaxy instance after the update that will contain a reference to script(s) that you can run that'll take care of the migration. Let me know if you need more information, if the wiki page doesn't clear it up! -Dannon On Wed, Aug 6, 2014 at 1:36 PM, Beginner TI ruiwang...@gmail.com wrote: Hi Dannon, Thanks for the reply. Sorry for the duplicate. I was wondering that the original subject sounded like a user problem so I updated it... Actually we are having a little down time right now for a file system check. I'll try your suggestions asap and let you know. At the mean time, could you please elaborate this a little bit? I noticed that in the tool_conf.xml, not every tool is shown on the panel although they are all defined there in the same format. I also noticed that all the tool xml files, such as section id=samtools name=NGS: SAM Tools tool file=samtools/sam_bitwise_flag_filter.xml / tool file=samtools/sam2interval.xml / tool file=samtools/sam_to_bam.xml / tool file=samtools/bam_to_sam.xml / tool file=samtools/sam_merge.xml / tool file=samtools/samtools_mpileup.xml / tool file=samtools/sam_pileup.xml / tool file=samtools/pileup_parser.xml / tool file=samtools/pileup_interval.xml / tool file=samtools/samtools_flagstat.xml / tool file=samtools/samtools_rmdup.xml / tool file=samtools/samtools_slice_bam.xml / /section the directory samtools and all these xml files are no longer in the tools directory. seems that upgrade deleted them. is this correct? or they are moved to somewhere else? I looked around but didn't see. Based on this, I think that even I removed the tool_panel.xml file you mentioned, I will still not be able to restore samtools even I could see it...could you please let me know how I could configure this? Thanks, Rui On Wed, Aug 6, 2014 at 10:23 AM, Dannon Baker dannon.ba...@gmail.com wrote: Hi Rui, We'll definitely take a look. Please allow for a reasonable time period to pass before sending duplicate emails to the list asking for assistance. That said, does deleting your integrated_tool_panel.xml (this file is automatically regenerated by Galaxy, safe to remove it and restart) bring back your manually installed samtools? On Wed, Aug 6, 2014 at 1:08 PM, Beginner TI ruiwang...@gmail.com wrote: Could someone please take a look? Thanks, Rui On Tue, Aug 5, 2014 at 8:27 PM, Beginner TI ruiwang...@gmail.com wrote: Dear Admin, Since the upgrade, I lost the samtools that I installed myself. Although it is still in the tool_conf.xml, it does not show up on the panel. I tried to install from toolshed, but failed. Error messages pasted below. Could someone take a look? Thanks, Rui ⇝ AttributeError: 'NoneType' object has no attribute 'strip' URL: http://128.125.28.215/admin_toolshed/prepare_for_install?tool_shed_url=https://toolshed.g2.bx.psu.edu/repository_ids=d2c3034af88d998dchangeset_revisions=af65bfb54977 Module weberror.evalexception.middleware:*364* in respond
Re: [galaxy-dev] Packaging tools with language specific dependencies
Hi Renato, Am 05.08.2014 um 17:34 schrieb Renato Alves: Hi Bjoern, Thanks for the info and the fix. So far we had no issues but it's not extensive testing. I'll let you know if it fails for some reason. In addition to the shebang change we also started using perl script instead of simply script. Since perl is looked up in PATH, all else works as expected. Can you try to explain this a bit more? It does get a bit tricky if you need to guess where the script is located. $(which script) works but it's hacky. Still, it's the only way you can fix the shebang without changing it before. PS: There's shutil.which() but only python 3.3 onwards. Anything I can do to support you? I'm right that the shebang modifications work for you? Do you think that should go into Galaxy Core, so that every perl script under /bin/ will get a simple shebang? Thanks, Bjoern Cheers, Renato Quoting Björn Grüning on 02-08-2014 15:26: Hi Renato, cpanm is not the only problem it seems to be an general perl 'feature'. How annoying :( As I figured from: https://metacpan.org/source/RJBS/perl-5.20.0/Configure#L9162 we can use:-Dstartperl='#!/usr/bin/env perl' to change them via install. Now every perl script from the main perl has the correct shebang. Unfortunately, every other package installed with cpanm or Makefile.PL will have this annoying long shebang. I'm not sure there is a switch in cpan/cpanm to do that. Under cpan there are a few repositories, called change_shebang... so I assume we are not the only ones that have this problem. So I ended up with: action type=shell_commandsed -i 's|#!$INSTALL_DIR/bin/|#!/usr/bin/env |' $INSTALL_DIR/bin/*/action I have uploaded it to the testtoolshed and to galaxytools. If this is working for you, also with depending repositories, I will try to add this to into the Tool Shed core to make this sed command implicit for all perl tools. Cheers, Bjoern Am 01.08.2014 um 16:06 schrieb Renato Alves: Hi BJörn, We have run into some problems with the perl environment. This might also affect other tools that rely on a shebang with absolute paths to the interpreter. Details below. During the installation of the perl package (we tried package_perl_5_18 from main and testing toolshed), the cpanm script gets installed but when executed it is invalid/not found. The issue seems to be the length of the shebang in the script which is limited to 80 characters. In cpanm it's: !#/galaxy_dist/tool_dependencies/perl/5.18.1/bgruening/package_perl_5_18/e89824189ec6/bin/perl That is 92 characters long. When launching the script it gets truncated at ...perl_5_18/e898241, causing the command to fail with a bad interpreter error. I was able to workaround this problem by editing the script manually and replacing the long shebang by: #!/usr/bin/env perl As it stands it seems relying on the shebang to use the correct interpreter is a problem. Repositories with long (character wise) names, versions or owners will be more likely to suffer from this. So will installations that are not close to the system root. Renato Quoting Björn Grüning on 30-07-2014 12:04: Hi Renato, Am 30.07.2014 um 12:21 schrieb Renato Alves: Hi everyone, Is there any standard or commonly used way to package tools that have language specific dependencies. I know that with Python libraries one can use setup_virtualenv and with Java jars the JAVA_JAR_LIB strategy is used. Is there anything equivalent for R, Perl and Ruby libraries? please have a look at: https://github.com/bgruening/galaxytools/tree/master/test_repositories If you have any questions I'm happy to help you! Bjoern Thanks Renato ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] sql server timeout
On occasion I'm getting and 'Internal Server error' on login with my production galaxy server. If I refresh the page I'm able to login just fine. This instance of galaxy has an apache frontend and a mysql server on the back end. If I look at my log files I see: OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT galaxy_session.id AS galaxy_session_id. It looks like perhaps galaxy is trying to use a stale session or something like that. Has anyone else seen this problem and is there an easy solution aside from having users refresh the page? Thanks, Don FSU Research Computing Center ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] August 2014 Galaxy CloudMan release
*We just released an update to Galaxy CloudMan.* CloudMan offers an easy way to get a personal and completely functional instance of Galaxy in the cloud in just a few minutes, without any manual configuration. This is mostly an incremental bug fix release with the following summary of changes: - On AWS, updated *galaxyFS* snapshot (snap-e6e1c04a), which includes the June 2, 2014 Galaxy release with the July 30th security fix. All the tools installed via the Tool Shed have been updated and a number of new tools added, most notably: Tophat2, Bowtie2, FastQC, several FASTQ manipulation tools, several QC tools. - For AWS, added support for VPC - For OpenStack clouds, added the ability to automatically recover worker instances on cluster reboot - Added support for creating a file system based on a downloadable archive - Do not run Galaxy with multiple processes by default. This is because Tool Shed installs do not work properly in the multi-process mode. This feature can be enabled by setting user data option configure_multiple_galaxy_processes to True when launching an instance. - Set SGE slots in each queue to be equal to the number of cores on the instance - Set instance IP in the Galaxy's FTP data upload tool message - Added support for Nginx v1.4 and allow it (with the PAM module) to used as the authentication mechanism when accessing Galaxy Reports app - Fixed cluster deletion when performed via the API - No longer automatically start Hadoop and HTCondor services - On manually-invoked instance reboots, do not increment the instance reboot count that otherwise eventually leads to instance termination - Limit the size of the log message buffer used in the UI to 1000 lines. Long-running instances had issues with this log growing large and that led to poor UI performance. The complete log is still available from the Admin page (or the command line). - Automatically delete the bucket/container for Test type (*ie*, 'SGE only') clusters on cluster termination For complete details on implemented changes, please see the source code commits https://bitbucket.org/galaxy/cloudman/commits/all?search=835%3A903 . Enjoy and please let us know what you think, Enis Dannon The Galaxy Team https://wiki.galaxyproject.org/Cloud ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] condor compute cluster
Our galaxy server has an apache front end with ldap authentication.
I've added a condor destination...
plugin id=condor type=runner
load=galaxy.jobs.runners.condor:CondorJobRunner/
destination id=condor runner=condor/
There are some tools I'd like to point to condor but it appears all the jobs
will be sent as the user condor.
Is there an existing fix that will submit jobs to our condor cluster as the
logged in user?
If not could someone give me a small pointer for a hack fix?
Something along the lines of:
galaxy-dist/lib/galaxy/jobs/runners/util/condor/__init__.py
line 71:
submit = Popen(('condor_submit', submit_file), stdout=PIPE, stderr=STDOUT)
Could be changed to sudo -u xxx -c condor_submit...
Thanks all,
Don
FSU Research Computing Center
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] condor compute cluster
Jobs run as the condor user? That is surprising - Galaxy isn't running
as the condor user is it?
Is your intention to have the jobs run as the 'galaxy' user or to run
as the LDAP authenticated end-user submitting the job?
If it is to run as the Galaxy user - I think it would be best to have
condor reconfigured to run jobs as the submitting user (in this case
galaxy) - at most institutions that is the way these things are
usually configured.
If you would like to run the jobs as the actual LDAP authenticated
user - that is going to require a somewhat significant development
effort. In that case I would look at the drmaa runner and what it is
doing to run jobs as the actual user and try to do similar things in
the condor runner - I don't think it would be a couple lines of
hacking.
condor has a DRMAA interface - I have never heard of it being used
with Galaxy but that might be something worth trying as well. If you
can get that to work - you should be able to use the drmaa run as real
user mode.
-John
On Wed, Aug 6, 2014 at 4:40 PM, Shrum, Donald C dcsh...@admin.fsu.edu wrote:
Our galaxy server has an apache front end with ldap authentication.
I've added a condor destination...
plugin id=condor type=runner
load=galaxy.jobs.runners.condor:CondorJobRunner/
destination id=condor runner=condor/
There are some tools I'd like to point to condor but it appears all the jobs
will be sent as the user condor.
Is there an existing fix that will submit jobs to our condor cluster as the
logged in user?
If not could someone give me a small pointer for a hack fix?
Something along the lines of:
galaxy-dist/lib/galaxy/jobs/runners/util/condor/__init__.py
line 71:
submit = Popen(('condor_submit', submit_file), stdout=PIPE, stderr=STDOUT)
Could be changed to sudo -u xxx -c condor_submit...
Thanks all,
Don
FSU Research Computing Center
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
