[galaxy-dev] DRMAA Slurm error
Hi,
I have configured a new galaxy-project site with SLURM (version 14). I have one
server with a Galaxy-Project instance, one node with SLURM server and two SLURM
worker nodes. I have compile SLURM-DRMAA from source codes. When I run
“drmaa-run /bin/hostname” it’s work. But, when I try to run the server I got
the next error:
Traceback (most recent call last):
File /home/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py, line 39, in
app_factory
app = UniverseApplication( global_conf = global_conf, **kwargs )
File /home/galaxy-dist/lib/galaxy/app.py, line 141, in __init__
self.job_manager = manager.JobManager( self )
File /home/galaxy-dist/lib/galaxy/jobs/manager.py, line 23, in __init__
self.job_handler = handler.JobHandler( app )
File /home/galaxy-dist/lib/galaxy/jobs/handler.py, line 32, in __init__
self.dispatcher = DefaultJobDispatcher( app )
File /home/galaxy-dist/lib/galaxy/jobs/handler.py, line 704, in __init__
self.job_runners = self.app.job_config.get_job_runner_plugins(
self.app.config.server_name )
File /home/galaxy-dist/lib/galaxy/jobs/__init__.py, line 621, in
get_job_runner_plugins
rval[id] = runner_class( self.app, runner[ 'workers' ], **runner.get(
'kwds', {} ) )
File /home/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py, line 81, in
__init__
self.ds.initialize()
File /home/galaxy-dist/eggs/drmaa-0.7.6-py2.6.egg/drmaa/session.py, line
257, in initialize
py_drmaa_init(contactString)
File /home/galaxy-dist/eggs/drmaa-0.7.6-py2.6.egg/drmaa/wrappers.py, line
73, in py_drmaa_init
return _lib.drmaa_init(contact, error_buffer, sizeof(error_buffer))
File /home/galaxy-dist/eggs/drmaa-0.7.6-py2.6.egg/drmaa/errors.py, line
151, in error_check
raise _ERRORS[code - 1](error_string)
AlreadyActiveSessionException: code 11: DRMAA session already exist.
[root@galaxy-project galaxy-dist]#
This is my “job_conf.xml”:
job_conf
plugins workers=4
plugin id=local type=runner
load=galaxy.jobs.runners.local:LocalJobRunner/
plugin id=drmaa type=runner
load=galaxy.jobs.runners.drmaa:DRMAAJobRunner/
plugin id=cli type=runner
load=galaxy.jobs.runners.cli:ShellJobRunner /
plugin id=slurm type=runner
load=galaxy.jobs.runners.slurm:SlurmJobRunner
param id=drmaa_library_path/usr/local/lib/libdrmaa.so/param
/plugin
/plugins
handlers
handler id=main/
/handlers
destinations default=drmaa_slurm
destination id=local runner=local/
destination id=multicore_local runner=local
param id=local_slots4/param
param id=embed_metadata_in_jobTrue/param
job_metrics /
/destination
destination id=docker_local runner=local
param id=docker_enabledtrue/param
/destination
destination id=drmaa_slurm runner=drmaa
param
id=galaxy_external_runjob_scriptscripts/drmaa_external_runner.py/param
param
id=galaxy_external_killjob_scriptscripts/drmaa_external_killer.py/param
param
id=galaxy_external_chown_scriptscripts/external_chown_script.py/param
/destination
destination id=direct_slurm runner=slurm
param id=nativeSpecification--time=00:01:00/param
/destination
/destinations
resources default=default
group id=default/group
group id=memoryonlymemory/group
group id=allprocessors,memory,time,project/group
/resources
tools
tool id=foo handler=trackster_handler
param id=sourcetrackster/param
/tool
tool id=bar destination=dynamic/
tool id=longbar destination=dynamic resources=all /
tool id=baz handler=special_handlers destination=bigmem/
/tools
limits
limit type=registered_user_concurrent_jobs2/limit
limit type=anonymous_user_concurrent_jobs1/limit
limit type=destination_user_concurrent_jobs id=local1/limit
limit type=destination_user_concurrent_jobs tag=mycluster2/limit
limit type=destination_user_concurrent_jobs tag=longjobs1/limit
limit type=destination_total_concurrent_jobs id=local16/limit
limit type=destination_total_concurrent_jobs
tag=longjobs100/limit
limit type=walltime24:00:00/limit
limit type=output_size10GB/limit
/limits
/job_conf
Can you help me? I am newbie with “Galaxy Project” administration.
THANKS IN ADVANCE
Alfonso Pardo Diaz
System Administrator / Researcher
c/ Sola nº 1; 10200 Trujillo, ESPAÑA
Tel: +34 927 65 93 17 Fax: +34 927 32 32 37
[CETA-Ciemat logo]http://www.ceta-ciemat.es/
Confidencialidad:
Este mensaje y sus ficheros adjuntos se dirige exclusivamente a su destinatario
y puede contener información privilegiada o confidencial. Si no es vd. el
destinatario indicado, queda notificado de que la utilización, divulgación y/o
copia sin autorización está prohibida en virtud de la legislación vigente. Si
ha
Re: [galaxy-dev] problems installing tool dependency
Thank you very much Nate and thanks, Will, for helping with the troubleshooting. Dori From: Nate Coraor [mailto:n...@bx.psu.edu] Sent: Wednesday, September 24, 2014 2:58 PM To: Will Holtz Cc: Sajdak, Doris; galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] problems installing tool dependency I've just looked at this as well, it appears that /pub4 is not exposed and does not allow you to change into it, but it does allow you to fetch if you request files under it via the full path. e.g.: nate@victory% ncftp ftp.sanger.ac.ukhttp://ftp.sanger.ac.uk ... Logged in to ftp.sanger.ac.ukhttp://ftp.sanger.ac.uk. ncftp / ls ls-lR.gz.ok_to_rsyncREADME .request-for-update.lock .messagepub/ .request-for-update ncftp / cd pub4 Could not chdir to pub4: server said: pub4: No such file or directory ncftp / cd /pub4 Could not chdir to /pub4: server said: pub4: No such file or directory ncftp / get /pub4/resources/software/quicktree/quicktree.tar.gz quicktree.tar.gz: 38.12 kB 124.19 kB/s ncftp / I'll contact some folks in the Miller Lab to see if the tool can be updated to the path under /pub. On Wed, Sep 24, 2014 at 2:09 PM, Will Holtz who...@lygos.commailto:who...@lygos.com wrote: I also was able to download quicktree.tar.gz by clicking on the URL in your message. However, when I manually ftp into ftp.sanger.ac.ukhttp://ftp.sanger.ac.uk, there is no /pub4 directory. I'm a bit confused about how those two statements are both true, but maybe it will help someone else figure this out. Removing the '4' does result in a valid path (ftp://ftp.sanger.ac.uk/pub/resources/software/quicktree/quicktree.tar.gz). -Will On Wed, Sep 24, 2014 at 10:57 AM, Sajdak, Doris dj...@buffalo.edumailto:dj...@buffalo.edu wrote: Hi all, I am trying to install the Genome Diversity tools from the Miller Lab that are in the Galaxy toolshed. I’ve been able to get all dependencies installed except Quicktree. Whenever I try to install it, I get the error: Error downloading from URL ftp://ftp.sanger.ac.uk/pub4/resources/software/quicktree/quicktree.tar.gz: urlopen error ftp error: 550 pub4: No such file or directory This is the tool_dependencies.xml file for installing quicktree: ?xml version=1.0? tool_dependency package name=quicktree version=1.1 install version=1.0 actions !-- Download source code -- action type=download_by_url target_filename=quicktree_1.1.tar.gzftp://ftp.sanger.ac.uk/pub4/resources/software/quicktree/quicktree.tar.gz/action !-- Build quicktree -- action type=shell_commandmake quicktree/action !-- Install quicktree -- action type=shell_commandcp -R bin $INSTALL_DIR/action !-- Set environment for dependent repositories -- action type=set_environment environment_variable name=PATH action=prepend_to$INSTALL_DIR/bin/environment_variable /action /actions /install /package /tool_dependency I know this is not their file I’m downloading but the thing is, if I copy and paste the URL, I can download it with no problem. I’m not sure why it’s reporting that it can’t be found but the xml looks right to me. I’ve tried contacting the Miller Lab about this but haven’t heard from anyone so I’m hoping someone here can tell me how to move past this problem. Is there a way to install this tool manually and have the main Genome Diversity tools recognize it on the Galaxy server? Also, I noticed when looking at the Tool Dependency Definitions at https://toolshed.g2.bx.psu.edu/ there are errors going back many versions for this tool (see Test runs – Installation errors – Tool dependencies). Thank you for any help you can provide. Dori ** Dori Sajdak Senior Systems Administrator State University of NY at Buffalo Center for Computational Research 701 Ellicott St Buffalo, NY 14203 Phone: (716) 881-8934tel:%28716%29%20881-8934 Fax: (716) 849-6656tel:%28716%29%20849-6656 Web: http://ccr.buffalo.edu ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please
Re: [galaxy-dev] tool for STAR RNA-seq aligner
Ross, About the index files: It is way easier to have pre-built index files. However, when running a 2-pass STAR run, a user will need to generate their own reference index files based on the output SJ.tab.out file created in the first pass. Is this incorporated into your tool? About shared memory: I am under the impression that the latest version of STAR has deprecated this feature. I am unclear how this would help unless a single large-memory machine was dedicated to running all STAR jobs. Is this the case? Also, does the tool merge the SAM/BAM file with the output chimeric SAM file? David Hoover On 9/24/2014 7:03 PM, Ross wrote: Hi All, That (fubar in testtoolshed) star wrapper was derived from one originally written by Jeremy Goecks. I modified it for multiple inputs and added a few tweaks and it has been in production use in our group for about 6 months so I'm pretty sure it works reasonably well in our hands at least. I would really appreciate any available help getting it to a proven useful state - suggestions and code welcomed. I have not moved it to the main toolshed because aside from some encouragement, I've had no feedback to suggest it's working - or not. It is extremely fast - we regularly see 200-300M reads per minute in the logs! We regularly run a whole experiment worth (eg 12 - 24) fastq files simultaneously with the shared memory option working on our cluster - see the readme. Star index files made with a gene model (requires valid gff3) are huge - 20-30GB for hg19 - hence the need for shared memory if you run multiple jobs. That will eventually become a serious problem if you really want to allow users to make their own - we definitely do not. You need to be very careful about matching the gene model gff3 file to the reference and I had enough trouble getting it right for the few major genomes we use to make me think that I do not want users trying to do that generating 25GB of rubbish every time they get it wrong. There are challenges to do with needing different indexes for different length reads but we are seeing fairly consistent 60bp single ended reads for most of the incoming RNA seq experiments. A data manager would be a boon if anyone cares to write one... On Thu, Sep 25, 2014 at 6:55 AM, Curtis Hendrickson (Campus) curt...@uab.edu mailto:curt...@uab.edu wrote: Bjorn We'd be interested in this tool, as well. Any idea how close to functional it is? I see it's only on TEST toolshed, and not on production, at this point. I don't see any related Trello card when searching on star Regards, Curtis Galaxy Admin @ University of Alabama at Birmingham -Original Message- From: galaxy-dev-boun...@lists.bx.psu.edu mailto:galaxy-dev-boun...@lists.bx.psu.edu [mailto:galaxy-dev-boun...@lists.bx.psu.edu mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Björn Grüning Sent: Wednesday, September 24, 2014 3:15 PM To: galaxy-dev@lists.bx.psu.edu mailto:galaxy-dev@lists.bx.psu.edu; hoove...@helix.nih.gov mailto:hoove...@helix.nih.gov David Hoover Subject: Re: [galaxy-dev] tool for STAR RNA-seq aligner Hi David, yes there is inital code in the https://testtoolshed.g2.bx.psu.edu/. I think Ross has done some work on it. The main problem with Star is that is needs special indices (and a lot of it) and it would be great to offer data managers for it. Cheers, Bjoern Am 24.09.2014 um 22:05 schrieb David Hoover: Hi, I am developing a tool for STAR (https://code.google.com/p/rna-star/), and I realize I may be reinventing another wheel. Has anyone else created a tool for STAR? There's nothing else in the toolsheds for it yet. David David Hoover, PhD Helix Systems Staff SCB/DCSS/CIT/NIH 301-435-2986 http://helix.nih.gov ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy
Re: [galaxy-dev] DRMAA Slurm error
Solved!
The problem was I have configured the job_conf.xml two plugins entry with
DRMAA. I have deleted the entry:
plugin id=slurm type=runner
load=galaxy.jobs.runners.slurm:SlurmJobRunner
And now works!
Thanks
El 25/09/2014, a las 08:12, Pardo Diaz, Alfonso
alfonso.pa...@ciemat.esmailto:alfonso.pa...@ciemat.es escribió:
Hi,
I have configured a new galaxy-project site with SLURM (version 14). I have one
server with a Galaxy-Project instance, one node with SLURM server and two SLURM
worker nodes. I have compile SLURM-DRMAA from source codes. When I run
“drmaa-run /bin/hostname” it’s work. But, when I try to run the server I got
the next error:
Traceback (most recent call last):
File /home/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py, line 39, in
app_factory
app = UniverseApplication( global_conf = global_conf, **kwargs )
File /home/galaxy-dist/lib/galaxy/app.py, line 141, in __init__
self.job_manager = manager.JobManager( self )
File /home/galaxy-dist/lib/galaxy/jobs/manager.py, line 23, in __init__
self.job_handler = handler.JobHandler( app )
File /home/galaxy-dist/lib/galaxy/jobs/handler.py, line 32, in __init__
self.dispatcher = DefaultJobDispatcher( app )
File /home/galaxy-dist/lib/galaxy/jobs/handler.py, line 704, in __init__
self.job_runners = self.app.job_config.get_job_runner_plugins(
self.app.config.server_name )
File /home/galaxy-dist/lib/galaxy/jobs/__init__.py, line 621, in
get_job_runner_plugins
rval[id] = runner_class( self.app, runner[ 'workers' ], **runner.get(
'kwds', {} ) )
File /home/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py, line 81, in
__init__
self.ds.initialize()
File /home/galaxy-dist/eggs/drmaa-0.7.6-py2.6.egg/drmaa/session.py, line
257, in initialize
py_drmaa_init(contactString)
File /home/galaxy-dist/eggs/drmaa-0.7.6-py2.6.egg/drmaa/wrappers.py, line
73, in py_drmaa_init
return _lib.drmaa_init(contact, error_buffer, sizeof(error_buffer))
File /home/galaxy-dist/eggs/drmaa-0.7.6-py2.6.egg/drmaa/errors.py, line
151, in error_check
raise _ERRORS[code - 1](error_string)
AlreadyActiveSessionException: code 11: DRMAA session already exist.
[root@galaxy-project galaxy-dist]#
This is my “job_conf.xml”:
job_conf
plugins workers=4
plugin id=local type=runner
load=galaxy.jobs.runners.local:LocalJobRunner/
plugin id=drmaa type=runner
load=galaxy.jobs.runners.drmaa:DRMAAJobRunner/
plugin id=cli type=runner
load=galaxy.jobs.runners.cli:ShellJobRunner /
plugin id=slurm type=runner
load=galaxy.jobs.runners.slurm:SlurmJobRunner
param id=drmaa_library_path/usr/local/lib/libdrmaa.so/param
/plugin
/plugins
handlers
handler id=main/
/handlers
destinations default=drmaa_slurm
destination id=local runner=local/
destination id=multicore_local runner=local
param id=local_slots4/param
param id=embed_metadata_in_jobTrue/param
job_metrics /
/destination
destination id=docker_local runner=local
param id=docker_enabledtrue/param
/destination
destination id=drmaa_slurm runner=drmaa
param
id=galaxy_external_runjob_scriptscripts/drmaa_external_runner.py/param
param
id=galaxy_external_killjob_scriptscripts/drmaa_external_killer.py/param
param
id=galaxy_external_chown_scriptscripts/external_chown_script.py/param
/destination
destination id=direct_slurm runner=slurm
param id=nativeSpecification--time=00:01:00/param
/destination
/destinations
resources default=default
group id=default/group
group id=memoryonlymemory/group
group id=allprocessors,memory,time,project/group
/resources
tools
tool id=foo handler=trackster_handler
param id=sourcetrackster/param
/tool
tool id=bar destination=dynamic/
tool id=longbar destination=dynamic resources=all /
tool id=baz handler=special_handlers destination=bigmem/
/tools
limits
limit type=registered_user_concurrent_jobs2/limit
limit type=anonymous_user_concurrent_jobs1/limit
limit type=destination_user_concurrent_jobs id=local1/limit
limit type=destination_user_concurrent_jobs tag=mycluster2/limit
limit type=destination_user_concurrent_jobs tag=longjobs1/limit
limit type=destination_total_concurrent_jobs id=local16/limit
limit type=destination_total_concurrent_jobs
tag=longjobs100/limit
limit type=walltime24:00:00/limit
limit type=output_size10GB/limit
/limits
/job_conf
Can you help me? I am newbie with “Galaxy Project” administration.
THANKS IN ADVANCE
Alfonso Pardo Diaz
System Administrator / Researcher
c/ Sola nº 1; 10200 Trujillo, ESPAÑA
Tel: +34 927 65 93 17 Fax: +34 927 32 32 37
[CETA-Ciemat logo]http://www.ceta-ciemat.es/
[galaxy-dev] Error report via API?
Is there an API endpoint for sending the error report for a failed step in a history? I would like to run a workflow via API and email the first error that occurs. -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Simple standard for API use of a global user/key that all loaded tools can draw upon?
Do any of you see the need to establish a way for tools to use a global (current site) API url/key combination? From a security perspective I'm assuming this is basically ok since tools run under the galaxy user anyways (and can directly read and write files to their heart's content). I understand that a particular user's API key could help limit what they have access to. My tool however wants to do some prepatory form display and post-job work on behalf of the Galaxy system in addition to working with user-centric data; I presume others have similar use-case needs. I think it would amount to having a standard GALAXY_TOOL_API_URL and GALAXY_TOOL_API_KEY managed in universe_wsgi.ini (left blank if not desired by sys admin). The key would be generated by a galaxy admin using a real admin user id. These would be available as python or shell environment variables when it came time to execute a tool's command interpreter=python etc. work. The wee other detail is that I'm using code file=versioned_data_form.py / in my tool xml because there is API information I have to retrieve to set up the form with. So the url and key need to be accessible in that execution environment too?!! Am I missing the boat; alternately has this already been talked about?! Thanks for advice ... Damion Hsiao lab, BC Public Health Microbiology Reference Laboratory, BC Centre for Disease Control 655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Error report via API?
I don't believe this is currently available via the API - but we would like to move everything to the API so I don't see why this wouldn't be included (maybe with some rate limit-ting :)). I have created a Trello card: https://trello.com/c/W2PEpe8V This could potentially be fairly straight forward to move from a web controller to an API if you wanted to take this on and open a pull request :). Thanks for the suggestion, -John On Thu, Sep 25, 2014 at 3:14 PM, Evan Bollig boll0...@umn.edu wrote: Is there an API endpoint for sending the error report for a failed step in a history? I would like to run a workflow via API and email the first error that occurs. -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] sendmail from addr
Hey Evan, The following diff *should* add a new config option in universe_wsgi.ini called no_reply_email_from that can be used to set from when sending these activation emails. If you are feeling brave, try it out and let me know how it goes. I have not recreated this problem (read I have not tested this at all) - but if this works for you I would be happy to commit it to -central. https://github.com/jmchilton/galaxy-central/commit/9f023a72a56bce5efb6b1cef1609f3993bf2291b https://github.com/jmchilton/galaxy-central/commit/9f023a72a56bce5efb6b1cef1609f3993bf2291b.patch -John On Wed, Sep 24, 2014 at 5:56 PM, Evan Bollig boll0...@umn.edu wrote: On a local instance of Galaxy, I get this error when attempting to reset my password: Failed to reset password: (501, '5.1.7 Bad sender address syntax', 'galaxy-no-reply@10.32.16.121') I am told my mail server requires the from address to either have a domain name, or the IP address to be in brackets (e.g., [10.32.16.121]). Is there an option to configure the from address format, or override the address? If there are no configuration options, what files would I need to modify to add brackets? The domain name is not an option. Cheers, -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
