Re: [galaxy-dev] Galaxy updated botched?
On 06/12/2014 12:06 AM, John Chilton wrote: If anyone out there has tired using Galaxy with strictly MyISAM tables and found it untenable - by all means please chime in. Well, due to several reasons we are (still) using MySQL (version: 5.1.69) with only MyISAM tables. We have been running into troubles but they were never related to running MyISAM. However, I haven't done the latest Galaxy code upgrade, hence we are still on database version '118'. So, I am curious to see what will happen when I do the next upgrade (after GCC). Also, on my list of things to do for this year is actually changing all our Galaxy servers from MySQL to PostgreSQL. With my current workload, I doubt I will manage to do this and have to postpone this 'adventure' to next year. Regards, Hans-Rudolf ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] jobs submitted to a cluster
I've setup galaxy to submit jobs to our HPC cluster as the logged in user. I used the drama python module to submit the jobs to our moab server. It appears that the working directory for a submitted job is being removed by galaxy prior to the job completing on the cluster. I can see a working directory is created in the logs: galaxy.jobs DEBUG 2014-06-12 08:21:03,786 (15) Working directory for job is: /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15 I've confirmed the directory is created by watching on the file system and within about two seconds of the folder being created it is deleted. [root@admin 000]# watch -d ls -lR Every 2.0s: ls -lR Thu Jun 12 08:21:06 2014 total 64 drwxrwxrwx 2 dcshrum dcshrum 4096 Jun 12 08:21 15 I see the job sent via DRMAA: galaxy.jobs.handler DEBUG 2014-06-12 08:21:03,795 (15) Dispatching to drmaa runner galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) submitting file /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) native specification is: -N galaxyjob -l nodes=1,walltime=2:00 -q genacc_q galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,892 (15) submitting with credentials: dcshrum [uid: 232706] galaxy.jobs.runners.drmaa INFO 2014-06-12 08:21:06,196 (15) queued as 7570705.moab.local The job fails: galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:06,698 (15/7570705.moab.local) state change: job finished, but failed galaxy.jobs.runners DEBUG 2014-06-12 08:21:07,124 (15/7570705.moab.local) Unable to cleanup /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh: [Errno 2] No such file or directory: '/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh' I can see the same error in my moab log: *** error from copy /bin/cp: cannot create regular file `/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.o': No such file or directory *** end error output Any idea as to why galaxy removes the working directory? Is there a setting in the job_conf.xml that would resolve this? Thanks for any pointers. Donny FSU Research Computing Center ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Object-Store, setting filetypes crashes Galaxy
Hi,
I have configured to use the hierarchical object store but as soon as I try
to reset the filetpye of a dataset Galaxy is crashing with:
galaxy.objectstore DEBUG 2014-06-12 14:39:21,180 Using preferred backend
'files3' for creation of MetadataFile 5963
132.230.153.57 - - [12/Jun/2014:14:39:20 +0200] POST
/datasets/966f24627ef70c12/edit HTTP/1.1 500 -
http://galaxy.uni-freiburg.de/datasets/966f24627ef70c12/edit; Mozilla/5.0
(X11; Linux x86_64; rv:29.0) Gecko/20100101 Fire
fox/29.0
Error - type 'exceptions.OSError': [Errno 2] No such file or directory:
'database/tmp/metadata_temp_file_1xnGcE'
URL: http://galaxy.uni-freiburg.de/datasets/966f24627ef70c12/edit
File
'/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py',
line 149 in __call__
app_iter = self.application(environ, sr_checker)
File
'/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
line 84 in __call__
return self.application(environ, start_response)
File
'/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
line 633 in __call__
return self.application(environ, start_response)
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line
132 in __call__
return self.handle_request( environ, start_response )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py', line
190 in handle_request
body = method( trans, **kwargs )
File
'/usr/local/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/dataset.py',
line 295 in edit
trans.app.datatypes_registry.set_external_metadata_tool.tool_action.execute(
trans.app.datatypes_registry.set_external_metadata_tool, trans, incoming =
{ 'input1':data }, overwrite = False ) #overwrite is False as per existi
ng behavior
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/actions/metadata.py',
line 18 in execute
overwrite, history, job_params )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/actions/metadata.py',
line 79 in execute_via_app
kwds = { 'overwrite' : overwrite } )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/datatypes/metadata.py', line
717 in setup_external_metadata
shutil.copy( dataset.metadata.get( meta_key, None ).file_name,
metadata_temp.file_name )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/datatypes/metadata.py', line
575 in file_name
self._filename = abspath( tempfile.NamedTemporaryFile( dir =
self.tmp_dir, prefix = metadata_temp_file_ ).name )
File '/usr/local/python/2.7/lib/python2.7/tempfile.py', line 454 in
NamedTemporaryFile
(fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
File '/usr/local/python/2.7/lib/python2.7/tempfile.py', line 235 in
_mkstemp_inner
fd = _os.open(file, flags, 0600)
OSError: [Errno 2] No such file or directory:
'database/tmp/metadata_temp_file_1xnGcE'
I have attached my object_store_conf.xml file.
Thanks,
Bjoern
?xml version=1.0?
object_store type=hierarchical
backends
object_store type=distributed id=primary order=0
backends
backend id=files2 type=disk weight=1
files_dir path=/data/2/galaxy_db/files/
extra_dir type=temp path=/data/2/galaxy_db/tmp/
extra_dir type=job_work path=/data/2/galaxy_db/job_working_directory/
/backend
backend id=files3 type=disk weight=1
files_dir path=/data/3/galaxy_db/files/
extra_dir type=temp path=/data/3/galaxy_db/tmp/
extra_dir type=job_work path=/data/3/galaxy_db/job_working_directory/
/backend
/backends
/object_store
object_store type=disk id=secondary order=1
files_dir path=/data/0/galaxy_db/files/
extra_dir type=temp path=/data/0/galaxy_db/tmp/
extra_dir type=job_work path=/data/0/galaxy_db/job_working_directory/
/object_store
/backends
/object_store
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Missing navigation pane
Hey Stephen, Not a true solution, but I solved a similar issue (missing history pane) by clearing out the browser cache. Geert On 06/10/2014 04:39 PM, Dannon Baker wrote: Hey Stephen, Can you look in the server logs to see if there are any errors being reported? Or if there are any javascript errors in the browser window? You may also want to try clearing your browser cache. -Dannon On Tue, Jun 10, 2014 at 1:54 AM, Stephen E sedwards...@hotmail.com mailto:sedwards...@hotmail.com wrote: I installed a local instance of galaxy approximately a month ago (build 2014.04.14). I was having an issue with the Data Manager to fetch reference genomes and after checking to see if there was a newer version of galaxy, I decided to update and see if that fixed my problem. I ran hg pull and then hg update latest_2014.06.02. I tried to rerun galaxy but was instructed to run manage_db.sh so I did (sh manage_db.sh upgrade). When I ran galaxy after all this, it started but when opened in a web browser, the navagation pane at the top is missing (i.e. Analyze Data, Workflows, User, etc). There is a blue bar but nothing is on it. How do I get the missing navigation pane back? I can't do a lot of things without it (i.e. check help or change user settings). I can still get to the administrator section but only by appending /admin to the url. I need to know how to fix the new version or how to succesfully revert to a previous version without anything else breaking. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ -- Geert Vandeweyer, Ph.D. Department of Medical Genetics University of Antwerp Prins Boudewijnlaan 43 2650 Edegem Belgium Tel: +32 (0)3 275 97 56 E-mail: geert.vandewe...@ua.ac.be http://ua.ac.be/cognitivegenetics http://www.linkedin.com/in/geertvandeweyer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy updated botched?
Hey Hans (and anyone else using MySQL), Just to clarify -- MySQL doesn't have this (particular) error by default and as long as your server default table type has not and does not change you should be fine. I've just successfully tested MySQL using MyISAM all the way to the current tip revision and you should be able to upgrade safely when the time comes. -Dannon On Thu, Jun 12, 2014 at 2:01 AM, Hans-Rudolf Hotz h...@fmi.ch wrote: On 06/12/2014 12:06 AM, John Chilton wrote: If anyone out there has tired using Galaxy with strictly MyISAM tables and found it untenable - by all means please chime in. Well, due to several reasons we are (still) using MySQL (version: 5.1.69) with only MyISAM tables. We have been running into troubles but they were never related to running MyISAM. However, I haven't done the latest Galaxy code upgrade, hence we are still on database version '118'. So, I am curious to see what will happen when I do the next upgrade (after GCC). Also, on my list of things to do for this year is actually changing all our Galaxy servers from MySQL to PostgreSQL. With my current workload, I doubt I will manage to do this and have to postpone this 'adventure' to next year. Regards, Hans-Rudolf ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy updated botched?
Thanks to everyone for their input. John, your explanation was excellent. I completely agree with the implementation of galaxy as it stands now (with a minor documentation warning about default mysql engine switch in v5.5). Personally, I either place trust in the ORM to maintain referential integrity or I offload it to the DB Engine. Hence, I will keep the MyISAM tables going for now after I do some validation against an InnoDB database (please no MyISAM engine hate-mail LOL). With respect to dropping MySQL support, I will let that one go by for now LOL. Thanks again for your contributions and maintenance of this great platform. I feel that it will be a game changer in the bioinformatics community. Regards, Iyad Kandalaft Microbial Biodiversity Bioinformatics Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada 960 Carling Ave.| 960 Ave. Carling Ottawa, ON| Ottawa (ON) K1A 0C6 E-mail Address / Adresse courriel iyad.kandal...@agr.gc.ca Telephone | Téléphone 613-759-1228 Facsimile | Télécopieur 613-759-1701 Teletypewriter | Téléimprimeur 613-773-2600 Government of Canada | Gouvernement du Canada From: Dannon Baker [mailto:dannon.ba...@gmail.com] Sent: Thursday, June 12, 2014 9:16 AM To: Hans-Rudolf Hotz Cc: John Chilton; Kandalaft, Iyad; galaxy-...@bx.psu.edu Subject: Re: [galaxy-dev] Galaxy updated botched? Hey Hans (and anyone else using MySQL), Just to clarify -- MySQL doesn't have this (particular) error by default and as long as your server default table type has not and does not change you should be fine. I've just successfully tested MySQL using MyISAM all the way to the current tip revision and you should be able to upgrade safely when the time comes. -Dannon On Thu, Jun 12, 2014 at 2:01 AM, Hans-Rudolf Hotz h...@fmi.chmailto:h...@fmi.ch wrote: On 06/12/2014 12:06 AM, John Chilton wrote: If anyone out there has tired using Galaxy with strictly MyISAM tables and found it untenable - by all means please chime in. Well, due to several reasons we are (still) using MySQL (version: 5.1.69) with only MyISAM tables. We have been running into troubles but they were never related to running MyISAM. However, I haven't done the latest Galaxy code upgrade, hence we are still on database version '118'. So, I am curious to see what will happen when I do the next upgrade (after GCC). Also, on my list of things to do for this year is actually changing all our Galaxy servers from MySQL to PostgreSQL. With my current workload, I doubt I will manage to do this and have to postpone this 'adventure' to next year. Regards, Hans-Rudolf ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] jobs submitted to a cluster
Hey Donny, What is the value of keep_completed on your queue (from qmgr -c 'p s')? Could it be that your spool is flushing completed jobs immediately? I ran into issues the other day with libdrmaa requiring at least keep_complete = 60 seconds to properly detect completed jobs and clean up after itself. Cheers, -E -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu On Thu, Jun 12, 2014 at 7:36 AM, Shrum, Donald C dcsh...@admin.fsu.edu wrote: I've setup galaxy to submit jobs to our HPC cluster as the logged in user. I used the drama python module to submit the jobs to our moab server. It appears that the working directory for a submitted job is being removed by galaxy prior to the job completing on the cluster. I can see a working directory is created in the logs: galaxy.jobs DEBUG 2014-06-12 08:21:03,786 (15) Working directory for job is: /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15 I've confirmed the directory is created by watching on the file system and within about two seconds of the folder being created it is deleted. [root@admin 000]# watch -d ls -lR Every 2.0s: ls -lR Thu Jun 12 08:21:06 2014 total 64 drwxrwxrwx 2 dcshrum dcshrum 4096 Jun 12 08:21 15 I see the job sent via DRMAA: galaxy.jobs.handler DEBUG 2014-06-12 08:21:03,795 (15) Dispatching to drmaa runner galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) submitting file /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) native specification is: -N galaxyjob -l nodes=1,walltime=2:00 -q genacc_q galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,892 (15) submitting with credentials: dcshrum [uid: 232706] galaxy.jobs.runners.drmaa INFO 2014-06-12 08:21:06,196 (15) queued as 7570705.moab.local The job fails: galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:06,698 (15/7570705.moab.local) state change: job finished, but failed galaxy.jobs.runners DEBUG 2014-06-12 08:21:07,124 (15/7570705.moab.local) Unable to cleanup /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh: [Errno 2] No such file or directory: '/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh' I can see the same error in my moab log: *** error from copy /bin/cp: cannot create regular file `/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.o': No such file or directory *** end error output Any idea as to why galaxy removes the working directory? Is there a setting in the job_conf.xml that would resolve this? Thanks for any pointers. Donny FSU Research Computing Center ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] bug: API broken under remote_user
Hi Eric,
It's not broken per se but the documentation is lacking on this front. You
need to route your /galaxy/api folder to a proxy that does not require
authentication. Maybe this isn't the best way but it works for AAFC's
production galaxy. See my apache configuration file below:
--
# Function for LowerCase conversion used in rewriterule directive
RewriteMap lc int:tolower
# Setup the load balancer and force LDAP authentication with group file
authorization
Proxy balancer://galaxy-prod/*
BalancerMember http://localhost:6
BalancerMember http://localhost:60001
BalancerMember http://localhost:60002
# LDAP based authentication
AuthName Galaxy - Login with AAFC credentials
AuthType Basic
AuthBasicAuthoritative off
AuthBasicProvider ldap
AuthLDAPURL REDACTED
AuthLDAPBindDN 'REDACTED'
AuthLDAPBindPassword REDACTED
# File based authorization
AuthGroupFile /home/galaxy/permitted_users
Require group galaxy-users
RewriteEngine on
# Convert the sAMAccountName to lower case
RewriteRule ^ -
[E=AUTHENTICATE_sAMAccountName:${lc:%{ENV:AUTHENTICATE_sAMAccountName}}]
# Set the REMOTE_USER header to the contents of the LDAP query
response's sAMAccountName attribute
RequestHeader set REMOTE_USER %{AUTHENTICATE_sAMAccountName}e
/Proxy
Proxy balancer://galaxy-prod-noauth/*
BalancerMember http://localhost:6
BalancerMember http://localhost:60001
BalancerMember http://localhost:60002
# Required to allow unauthenticated access
# Not clear why this is so
Satisfy any
/Proxy
# Bypass authentication for the api endpoints when a key get variable is
provided by proxying directly to the galaxy web server
RewriteCond %{QUERY_STRING} key=
RewriteRule ^/galaxy/api/(.*) balancer://galaxy-prod-noauth/api/$1 [P]
# Bypass authentication for display servers
RewriteCond HTTP_HOST =hgw1.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw2.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw3.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw4.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw5.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw7.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw8.cse.ucsc.edu [NC]
RewriteRule ^/galaxy/root/display_as(.*)
balancer://galaxy-prod-noauth/root/display_as$1 [P]
# Serve static content directly from apache
RewriteRule ^/galaxy/static/style/(.*)
/home/galaxy/galaxy-dist/static/june_2007_style/blue/$1 [L]
RewriteRule ^/galaxy/static/scripts/(.*)
/home/galaxy/galaxy-dist/static/scripts/packed/$1 [L]
RewriteRule ^/galaxy/static/(.*) /home/galaxy/galaxy-dist/static/$1 [L]
RewriteRule ^/galaxy/favicon.ico /home/galaxy/galaxy-dist/static/favicon.ico [L]
RewriteRule ^/galaxy/robots.txt /home/galaxy/galaxy-dist/static/robots.txt [L]
# Route all other traffic through the load balancer
RewriteRule ^/galaxy/(.*)$ balancer://galaxy-prod/$1 [P]
--
Regards,
Iyad Kandalaft
Microbial Biodiversity Bioinformatics
Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada
960 Carling Ave.| 960 Ave. Carling
Ottawa, ON| Ottawa (ON) K1A 0C6
E-mail Address / Adresse courriel iyad.kandal...@agr.gc.ca
Telephone | Téléphone 613-759-1228
Facsimile | Télécopieur 613-759-1701
Teletypewriter | Téléimprimeur 613-773-2600
Government of Canada | Gouvernement du Canada
-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Eric Rasche
Sent: Wednesday, June 11, 2014 8:43 PM
To: galaxy-dev@lists.bx.psu.edu
Subject: [galaxy-dev] bug: API broken under remote_user
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
https://trello.com/c/AGKePuHZ/1630-expose-use-remote-user-via-configuration-api
I don't know if this is the correct card (it's been a long day and I may be
misreading it) but the API is completely broken under REMOTE_USER
authentication.
running ./scripts/api/display.py {key} http://localhost:8300 returns 403
forbidden.
running ./scripts/api/display.py {key} https://fqdn/galaxy/ returns 401
Authorization Required
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (GNU/Linux)
iQIcBAEBAgAGBQJTmPeFAAoJEMqDXdrsMcpVtmkP/1r36cFmedYJXnMFl2CG0e7P
VtSoMHYbtesc/IKnGwKEH1xtf2RNF7p8n0/0mC3QNAoI0n3A8XDoZRXaWzgmsZ4z
heGipPgotCyhFt/ud2W8eb0IlZ3AhtyWwvhG1hXcwQV4eXXHfuQCepPEbfyUUR2N
F+VCLMGMv3dnsv0ForC7iYesV4qqOsX48Wry6InVD41UpXOE0Bor5uB/HcohYjKS
sVhOmNSjLpME35Rka4vUW4oNIzeeBsQIQyrpCuZ2KTKImEkSoSEbTho9fhNUi0ll
EHlTu+qt4iaXZpqNOKU05p3TqdjxAmc2w/McmMzifwPZdmrxTZjaitFn+g1qXxYs
0ZVJtL/OZviFHuK+8qbbsCPIYYm/dVr5EPy821/F01dAGNw9k+lbk5kZdgC8zcBz
xFc5yk+QZGuc3KeDYLOidtn4c471Ez4LhgTd9wapYkSwva7aM5T1betpPZKufZbw
+SSC/KFi6J2rjVDOkHLCry185Bwha2hrn1tc2hd1+A/5zOuyqQ/u0isvRu+O7dxO
Re: [galaxy-dev] jobs submitted to a cluster
job_conf.xml is outside of my knowledge. Better wait to see what the others can tell us. -E -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu On Thu, Jun 12, 2014 at 8:31 AM, Shrum, Donald C dcsh...@admin.fsu.edu wrote: It's set to 600 seconds so I don't think that is the issue... Is there some sort of wait time to set in job_conf.xml -Original Message- From: Evan Bollig [mailto:boll0...@umn.edu] Sent: Thursday, June 12, 2014 9:27 AM To: Shrum, Donald C Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] jobs submitted to a cluster Hey Donny, What is the value of keep_completed on your queue (from qmgr -c 'p s')? Could it be that your spool is flushing completed jobs immediately? I ran into issues the other day with libdrmaa requiring at least keep_complete = 60 seconds to properly detect completed jobs and clean up after itself. Cheers, -E -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu On Thu, Jun 12, 2014 at 7:36 AM, Shrum, Donald C dcsh...@admin.fsu.edu wrote: I've setup galaxy to submit jobs to our HPC cluster as the logged in user. I used the drama python module to submit the jobs to our moab server. It appears that the working directory for a submitted job is being removed by galaxy prior to the job completing on the cluster. I can see a working directory is created in the logs: galaxy.jobs DEBUG 2014-06-12 08:21:03,786 (15) Working directory for job is: /panfs/storage.local/software/galaxy-dist/database/job_working_directo ry/000/15 I've confirmed the directory is created by watching on the file system and within about two seconds of the folder being created it is deleted. [root@admin 000]# watch -d ls -lR Every 2.0s: ls -lR Thu Jun 12 08:21:06 2014 total 64 drwxrwxrwx 2 dcshrum dcshrum 4096 Jun 12 08:21 15 I see the job sent via DRMAA: galaxy.jobs.handler DEBUG 2014-06-12 08:21:03,795 (15) Dispatching to drmaa runner galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) submitting file /panfs/storage.local/software/galaxy-dist/database/job_working_directo ry/000/15/galaxy_15.sh galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) native specification is: -N galaxyjob -l nodes=1,walltime=2:00 -q genacc_q galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,892 (15) submitting with credentials: dcshrum [uid: 232706] galaxy.jobs.runners.drmaa INFO 2014-06-12 08:21:06,196 (15) queued as 7570705.moab.local The job fails: galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:06,698 (15/7570705.moab.local) state change: job finished, but failed galaxy.jobs.runners DEBUG 2014-06-12 08:21:07,124 (15/7570705.moab.local) Unable to cleanup /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh: [Errno 2] No such file or directory: '/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh' I can see the same error in my moab log: *** error from copy /bin/cp: cannot create regular file `/panfs/storage.local/software/galaxy-dist/database/job_working_direct ory/000/15/galaxy_15.o': No such file or directory *** end error output Any idea as to why galaxy removes the working directory? Is there a setting in the job_conf.xml that would resolve this? Thanks for any pointers. Donny FSU Research Computing Center ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Toolshed upload error message
Vipin, I've attempted to duplicate this error locally, with no luck as yet. Would you be able to send me the specific tar.gz file that was causing problems? --Dave B. On Wed 11 Jun 2014 05:02:34 PM EDT, Vipin TS wrote: Hi Greg, Please find the attached screenshot from my side. The same screen message displayed yesterday. I can see that the new revision number updated at repository page. I tried to clone the repository and which returns a empty folder. When I click on the new revision number, I can see the commit message displayed and when click on the browse button from the left side which essentially gives a blank page. (screenshot attached) But when I view the content of this repository from the repository page it seems the files are updated. (screenshot attached) I selected the root folders correctly. I am not sure what is wrong here. Could you please have a look on this. Many thanks, Vipin | Rätsch Lab On Wed, Jun 11, 2014 at 9:03 AM, Greg Von Kuster g...@bx.psu.edu mailto:g...@bx.psu.edu wrote: Hi Vipin, Can you elaborate on the error you are seeing? Thanks, Greg Von Kuster On Jun 10, 2014, at 8:07 PM, Vipin TS vipin...@gmail.com mailto:vipin...@gmail.com wrote: Hi Greg, When I trying to upload a next release version o my https://toolshed.g2.bx.psu.edu/repos/vipints/fml_gff3togtf converter program to Community Toolshed, I am getting an internal error message. I am uploading a tar.gz file to the page https://toolshed.g2.bx.psu.edu/ but this fails. Could you please pass the error message or let me how I can add new files to the repository and delete the old version. Thanks, Vipin | Rätsch Lab ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ This body part will be downloaded on demand. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] bug: API broken under remote_user
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi Iyad,
Thanks for posting configs! I'm very curious how this actually works for
you.
I'm running the API scripts from the galaxy server itself, and I made
requests to http://fqdn/galaxy/, http://localhost:/, and
http://localhost:/galaxy/
The request to the fqdn unsurprisingly fails as apache answers it, and
apache requires authentication. Both of the localhost requests also fail
(403), indicating that galaxy was forbidding me due to lack of a
REMOTE_USER variable being set:
Galaxy is configured to authenticate users via an external method
(such as HTTP authentication in Apache), but a username was not
provided by the upstream (proxy) server.
- From my tests, I conclude that the routes under /api/ also have the
requirement on REMOTE_USER being set. (Maybe a dev can chime in on
this?) I'm fairly certain the only way to get the /api/ route to work
with the apache config presented is to statically set REMOTE_USER, such
that it's always a valid galaxy user. You don't set a static REMOTE_USER
variable in your apache config, so when galaxy is serving requests from
outside users, in my understanding, the REMOTE_USER var will be unset,
and galaxy should refuse to answer it.
Cheers,
Eric
On 06/12/2014 08:31 AM, Kandalaft, Iyad wrote:
Hi Eric,
It's not broken per se but the documentation is lacking on this front. You
need to route your /galaxy/api folder to a proxy that does not require
authentication. Maybe this isn't the best way but it works for AAFC's
production galaxy. See my apache configuration file below:
--
# Function for LowerCase conversion used in rewriterule directive
RewriteMap lc int:tolower
# Setup the load balancer and force LDAP authentication with group file
authorization
Proxy balancer://galaxy-prod/*
BalancerMember http://localhost:6
BalancerMember http://localhost:60001
BalancerMember http://localhost:60002
# LDAP based authentication
AuthName Galaxy - Login with AAFC credentials
AuthType Basic
AuthBasicAuthoritative off
AuthBasicProvider ldap
AuthLDAPURL REDACTED
AuthLDAPBindDN 'REDACTED'
AuthLDAPBindPassword REDACTED
# File based authorization
AuthGroupFile /home/galaxy/permitted_users
Require group galaxy-users
RewriteEngine on
# Convert the sAMAccountName to lower case
RewriteRule ^ -
[E=AUTHENTICATE_sAMAccountName:${lc:%{ENV:AUTHENTICATE_sAMAccountName}}]
# Set the REMOTE_USER header to the contents of the LDAP query
response's sAMAccountName attribute
RequestHeader set REMOTE_USER %{AUTHENTICATE_sAMAccountName}e
/Proxy
Proxy balancer://galaxy-prod-noauth/*
BalancerMember http://localhost:6
BalancerMember http://localhost:60001
BalancerMember http://localhost:60002
# Required to allow unauthenticated access
# Not clear why this is so
Satisfy any
/Proxy
# Bypass authentication for the api endpoints when a key get variable is
provided by proxying directly to the galaxy web server
RewriteCond %{QUERY_STRING} key=
RewriteRule ^/galaxy/api/(.*) balancer://galaxy-prod-noauth/api/$1 [P]
# Bypass authentication for display servers
RewriteCond HTTP_HOST =hgw1.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw2.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw3.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw4.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw5.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw7.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw8.cse.ucsc.edu [NC]
RewriteRule ^/galaxy/root/display_as(.*)
balancer://galaxy-prod-noauth/root/display_as$1 [P]
# Serve static content directly from apache
RewriteRule ^/galaxy/static/style/(.*)
/home/galaxy/galaxy-dist/static/june_2007_style/blue/$1 [L]
RewriteRule ^/galaxy/static/scripts/(.*)
/home/galaxy/galaxy-dist/static/scripts/packed/$1 [L]
RewriteRule ^/galaxy/static/(.*) /home/galaxy/galaxy-dist/static/$1 [L]
RewriteRule ^/galaxy/favicon.ico /home/galaxy/galaxy-dist/static/favicon.ico
[L]
RewriteRule ^/galaxy/robots.txt /home/galaxy/galaxy-dist/static/robots.txt [L]
# Route all other traffic through the load balancer
RewriteRule ^/galaxy/(.*)$ balancer://galaxy-prod/$1 [P]
--
Regards,
Iyad Kandalaft
Microbial Biodiversity Bioinformatics
Agriculture and Agri-Food Canada | Agriculture et Agroalimentaire Canada
960 Carling Ave.| 960 Ave. Carling
Ottawa, ON| Ottawa (ON) K1A 0C6
E-mail Address / Adresse courriel iyad.kandal...@agr.gc.ca
Telephone | Téléphone 613-759-1228
Facsimile | Télécopieur 613-759-1701
Teletypewriter | Téléimprimeur 613-773-2600
Government of Canada | Gouvernement du
Re: [galaxy-dev] Object-Store, setting filetypes crashes Galaxy
Hi Björn,
I believe the problem is most likely this bug:
https://trello.com/c/OHlQoCrx
If you symlink /usr/local/galaxy/galaxy-dist/database/tmp to somewhere
writeable (e.g. what you have new_files_path set to), that'd confirm it.
I'm going to take a look at fixing this today.
--nate
On Thu, Jun 12, 2014 at 8:43 AM, bjoern.gruen...@googlemail.com
bjoern.gruen...@gmail.com wrote:
Hi,
I have configured to use the hierarchical object store but as soon as I
try to reset the filetpye of a dataset Galaxy is crashing with:
galaxy.objectstore DEBUG 2014-06-12 14:39:21,180 Using preferred backend
'files3' for creation of MetadataFile 5963
132.230.153.57 - - [12/Jun/2014:14:39:20 +0200] POST
/datasets/966f24627ef70c12/edit HTTP/1.1 500 -
http://galaxy.uni-freiburg.de/datasets/966f24627ef70c12/edit;
Mozilla/5.0 (X11; Linux x86_64; rv:29.0) Gecko/20100101 Fire
fox/29.0
Error - type 'exceptions.OSError': [Errno 2] No such file or directory:
'database/tmp/metadata_temp_file_1xnGcE'
URL: http://galaxy.uni-freiburg.de/datasets/966f24627ef70c12/edit
File
'/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/middleware/error.py',
line 149 in __call__
app_iter = self.application(environ, sr_checker)
File
'/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py',
line 84 in __call__
return self.application(environ, start_response)
File
'/usr/local/galaxy/galaxy-dist/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py',
line 633 in __call__
return self.application(environ, start_response)
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py',
line 132 in __call__
return self.handle_request( environ, start_response )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/web/framework/base.py',
line 190 in handle_request
body = method( trans, **kwargs )
File
'/usr/local/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/dataset.py',
line 295 in edit
trans.app.datatypes_registry.set_external_metadata_tool.tool_action.execute(
trans.app.datatypes_registry.set_external_metadata_tool, trans, incoming =
{ 'input1':data }, overwrite = False ) #overwrite is False as per existi
ng behavior
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/actions/metadata.py',
line 18 in execute
overwrite, history, job_params )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/tools/actions/metadata.py',
line 79 in execute_via_app
kwds = { 'overwrite' : overwrite } )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/datatypes/metadata.py',
line 717 in setup_external_metadata
shutil.copy( dataset.metadata.get( meta_key, None ).file_name,
metadata_temp.file_name )
File '/usr/local/galaxy/galaxy-dist/lib/galaxy/datatypes/metadata.py',
line 575 in file_name
self._filename = abspath( tempfile.NamedTemporaryFile( dir =
self.tmp_dir, prefix = metadata_temp_file_ ).name )
File '/usr/local/python/2.7/lib/python2.7/tempfile.py', line 454 in
NamedTemporaryFile
(fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
File '/usr/local/python/2.7/lib/python2.7/tempfile.py', line 235 in
_mkstemp_inner
fd = _os.open(file, flags, 0600)
OSError: [Errno 2] No such file or directory:
'database/tmp/metadata_temp_file_1xnGcE'
I have attached my object_store_conf.xml file.
Thanks,
Bjoern
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] bug: API broken under remote_user
Hey Eric,
I am pretty sure Iyad's configuration is right - here is a similar
config for nginx with some discussion of the API problem
http://production-galaxy-instances-with-cloudman-and-cloudbiolinux.readthedocs.org/en/latest/#external-authentication-ldap.
It is fine if it is empty it just has to be set and in fact it
should use the real remote user if available so the UI elements that
target the API without a key continue to function properly (which I
guess is slowly becoming all of the UI). This is what that apache
config and my nginx config do I believe.
Does this answer your question, do you have lingering concerns?
-John
On Thu, Jun 12, 2014 at 9:24 AM, Eric Rasche rasche.e...@yandex.ru wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi Iyad,
Thanks for posting configs! I'm very curious how this actually works for
you.
I'm running the API scripts from the galaxy server itself, and I made
requests to http://fqdn/galaxy/, http://localhost:/, and
http://localhost:/galaxy/
The request to the fqdn unsurprisingly fails as apache answers it, and
apache requires authentication. Both of the localhost requests also fail
(403), indicating that galaxy was forbidding me due to lack of a
REMOTE_USER variable being set:
Galaxy is configured to authenticate users via an external method
(such as HTTP authentication in Apache), but a username was not
provided by the upstream (proxy) server.
- From my tests, I conclude that the routes under /api/ also have the
requirement on REMOTE_USER being set. (Maybe a dev can chime in on
this?) I'm fairly certain the only way to get the /api/ route to work
with the apache config presented is to statically set REMOTE_USER, such
that it's always a valid galaxy user. You don't set a static REMOTE_USER
variable in your apache config, so when galaxy is serving requests from
outside users, in my understanding, the REMOTE_USER var will be unset,
and galaxy should refuse to answer it.
Cheers,
Eric
On 06/12/2014 08:31 AM, Kandalaft, Iyad wrote:
Hi Eric,
It's not broken per se but the documentation is lacking on this front. You
need to route your /galaxy/api folder to a proxy that does not require
authentication. Maybe this isn't the best way but it works for AAFC's
production galaxy. See my apache configuration file below:
--
# Function for LowerCase conversion used in rewriterule directive
RewriteMap lc int:tolower
# Setup the load balancer and force LDAP authentication with group file
authorization
Proxy balancer://galaxy-prod/*
BalancerMember http://localhost:6
BalancerMember http://localhost:60001
BalancerMember http://localhost:60002
# LDAP based authentication
AuthName Galaxy - Login with AAFC credentials
AuthType Basic
AuthBasicAuthoritative off
AuthBasicProvider ldap
AuthLDAPURL REDACTED
AuthLDAPBindDN 'REDACTED'
AuthLDAPBindPassword REDACTED
# File based authorization
AuthGroupFile /home/galaxy/permitted_users
Require group galaxy-users
RewriteEngine on
# Convert the sAMAccountName to lower case
RewriteRule ^ -
[E=AUTHENTICATE_sAMAccountName:${lc:%{ENV:AUTHENTICATE_sAMAccountName}}]
# Set the REMOTE_USER header to the contents of the LDAP query
response's sAMAccountName attribute
RequestHeader set REMOTE_USER %{AUTHENTICATE_sAMAccountName}e
/Proxy
Proxy balancer://galaxy-prod-noauth/*
BalancerMember http://localhost:6
BalancerMember http://localhost:60001
BalancerMember http://localhost:60002
# Required to allow unauthenticated access
# Not clear why this is so
Satisfy any
/Proxy
# Bypass authentication for the api endpoints when a key get variable is
provided by proxying directly to the galaxy web server
RewriteCond %{QUERY_STRING} key=
RewriteRule ^/galaxy/api/(.*) balancer://galaxy-prod-noauth/api/$1 [P]
# Bypass authentication for display servers
RewriteCond HTTP_HOST =hgw1.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw2.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw3.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw4.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw5.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw7.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw8.cse.ucsc.edu [NC]
RewriteRule ^/galaxy/root/display_as(.*)
balancer://galaxy-prod-noauth/root/display_as$1 [P]
# Serve static content directly from apache
RewriteRule ^/galaxy/static/style/(.*)
/home/galaxy/galaxy-dist/static/june_2007_style/blue/$1 [L]
RewriteRule ^/galaxy/static/scripts/(.*)
/home/galaxy/galaxy-dist/static/scripts/packed/$1 [L]
RewriteRule ^/galaxy/static/(.*) /home/galaxy/galaxy-dist/static/$1 [L]
RewriteRule ^/galaxy/favicon.ico
Re: [galaxy-dev] Object-Store, setting filetypes crashes Galaxy
Hi Nate, yes that is fixing it for me! Thanks! Bjoern Am 12.06.2014 16:26, schrieb Nate Coraor: writeable (e.g. what you have new_files_path set to), that'd confirm it. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] bug: API broken under remote_user
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi John,
Seems like odd behaviour, but if that's how it works, then that'll
adequately fix my problem.
Thanks John/Iyad!
Cheers,
Eric
On 06/12/2014 09:40 AM, John Chilton wrote:
Hey Eric,
I am pretty sure Iyad's configuration is right - here is a similar
config for nginx with some discussion of the API problem
http://production-galaxy-instances-with-cloudman-and-cloudbiolinux.readthedocs.org/en/latest/#external-authentication-ldap.
It is fine if it is empty it just has to be set and in fact it
should use the real remote user if available so the UI elements that
target the API without a key continue to function properly (which I
guess is slowly becoming all of the UI). This is what that apache
config and my nginx config do I believe.
Does this answer your question, do you have lingering concerns?
-John
On Thu, Jun 12, 2014 at 9:24 AM, Eric Rasche rasche.e...@yandex.ru wrote:
Hi Iyad,
Thanks for posting configs! I'm very curious how this actually works for
you.
I'm running the API scripts from the galaxy server itself, and I made
requests to http://fqdn/galaxy/, http://localhost:/, and
http://localhost:/galaxy/
The request to the fqdn unsurprisingly fails as apache answers it, and
apache requires authentication. Both of the localhost requests also fail
(403), indicating that galaxy was forbidding me due to lack of a
REMOTE_USER variable being set:
Galaxy is configured to authenticate users via an external method
(such as HTTP authentication in Apache), but a username was not
provided by the upstream (proxy) server.
- From my tests, I conclude that the routes under /api/ also have the
requirement on REMOTE_USER being set. (Maybe a dev can chime in on
this?) I'm fairly certain the only way to get the /api/ route to work
with the apache config presented is to statically set REMOTE_USER, such
that it's always a valid galaxy user. You don't set a static REMOTE_USER
variable in your apache config, so when galaxy is serving requests from
outside users, in my understanding, the REMOTE_USER var will be unset,
and galaxy should refuse to answer it.
Cheers,
Eric
On 06/12/2014 08:31 AM, Kandalaft, Iyad wrote:
Hi Eric,
It's not broken per se but the documentation is lacking on this front.
You need to route your /galaxy/api folder to a proxy that does not require
authentication. Maybe this isn't the best way but it works for AAFC's
production galaxy. See my apache configuration file below:
--
# Function for LowerCase conversion used in rewriterule directive
RewriteMap lc int:tolower
# Setup the load balancer and force LDAP authentication with group file
authorization
Proxy balancer://galaxy-prod/*
BalancerMember http://localhost:6
BalancerMember http://localhost:60001
BalancerMember http://localhost:60002
# LDAP based authentication
AuthName Galaxy - Login with AAFC credentials
AuthType Basic
AuthBasicAuthoritative off
AuthBasicProvider ldap
AuthLDAPURL REDACTED
AuthLDAPBindDN 'REDACTED'
AuthLDAPBindPassword REDACTED
# File based authorization
AuthGroupFile /home/galaxy/permitted_users
Require group galaxy-users
RewriteEngine on
# Convert the sAMAccountName to lower case
RewriteRule ^ -
[E=AUTHENTICATE_sAMAccountName:${lc:%{ENV:AUTHENTICATE_sAMAccountName}}]
# Set the REMOTE_USER header to the contents of the LDAP query
response's sAMAccountName attribute
RequestHeader set REMOTE_USER %{AUTHENTICATE_sAMAccountName}e
/Proxy
Proxy balancer://galaxy-prod-noauth/*
BalancerMember http://localhost:6
BalancerMember http://localhost:60001
BalancerMember http://localhost:60002
# Required to allow unauthenticated access
# Not clear why this is so
Satisfy any
/Proxy
# Bypass authentication for the api endpoints when a key get variable is
provided by proxying directly to the galaxy web server
RewriteCond %{QUERY_STRING} key=
RewriteRule ^/galaxy/api/(.*) balancer://galaxy-prod-noauth/api/$1 [P]
# Bypass authentication for display servers
RewriteCond HTTP_HOST =hgw1.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw2.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw3.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw4.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw5.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw7.cse.ucsc.edu [OR,NC]
RewriteCond HTTP_HOST =hgw8.cse.ucsc.edu [NC]
RewriteRule ^/galaxy/root/display_as(.*)
balancer://galaxy-prod-noauth/root/display_as$1 [P]
# Serve static content directly from apache
RewriteRule ^/galaxy/static/style/(.*)
/home/galaxy/galaxy-dist/static/june_2007_style/blue/$1 [L]
RewriteRule
Re: [galaxy-dev] jobs submitted to a cluster
My guess is Galaxy is deleting the directory because it believe the job is in error because of some communication problem while polling your DRM via DRMAA - Galaxy thinks the job has failed before it has even been run. You can set cleanup_job = never in universe_wsgi.ini's app:main section to instruct Galaxy to not delete the working directory. I suspect this will allow the DRM to finish running your job - but Galaxy is still going to fail it since it cannot properly detect its status. Can you confirm? -John On Thu, Jun 12, 2014 at 8:36 AM, Evan Bollig boll0...@umn.edu wrote: job_conf.xml is outside of my knowledge. Better wait to see what the others can tell us. -E -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu On Thu, Jun 12, 2014 at 8:31 AM, Shrum, Donald C dcsh...@admin.fsu.edu wrote: It's set to 600 seconds so I don't think that is the issue... Is there some sort of wait time to set in job_conf.xml -Original Message- From: Evan Bollig [mailto:boll0...@umn.edu] Sent: Thursday, June 12, 2014 9:27 AM To: Shrum, Donald C Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] jobs submitted to a cluster Hey Donny, What is the value of keep_completed on your queue (from qmgr -c 'p s')? Could it be that your spool is flushing completed jobs immediately? I ran into issues the other day with libdrmaa requiring at least keep_complete = 60 seconds to properly detect completed jobs and clean up after itself. Cheers, -E -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu On Thu, Jun 12, 2014 at 7:36 AM, Shrum, Donald C dcsh...@admin.fsu.edu wrote: I've setup galaxy to submit jobs to our HPC cluster as the logged in user. I used the drama python module to submit the jobs to our moab server. It appears that the working directory for a submitted job is being removed by galaxy prior to the job completing on the cluster. I can see a working directory is created in the logs: galaxy.jobs DEBUG 2014-06-12 08:21:03,786 (15) Working directory for job is: /panfs/storage.local/software/galaxy-dist/database/job_working_directo ry/000/15 I've confirmed the directory is created by watching on the file system and within about two seconds of the folder being created it is deleted. [root@admin 000]# watch -d ls -lR Every 2.0s: ls -lR Thu Jun 12 08:21:06 2014 total 64 drwxrwxrwx 2 dcshrum dcshrum 4096 Jun 12 08:21 15 I see the job sent via DRMAA: galaxy.jobs.handler DEBUG 2014-06-12 08:21:03,795 (15) Dispatching to drmaa runner galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) submitting file /panfs/storage.local/software/galaxy-dist/database/job_working_directo ry/000/15/galaxy_15.sh galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) native specification is: -N galaxyjob -l nodes=1,walltime=2:00 -q genacc_q galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,892 (15) submitting with credentials: dcshrum [uid: 232706] galaxy.jobs.runners.drmaa INFO 2014-06-12 08:21:06,196 (15) queued as 7570705.moab.local The job fails: galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:06,698 (15/7570705.moab.local) state change: job finished, but failed galaxy.jobs.runners DEBUG 2014-06-12 08:21:07,124 (15/7570705.moab.local) Unable to cleanup /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh: [Errno 2] No such file or directory: '/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh' I can see the same error in my moab log: *** error from copy /bin/cp: cannot create regular file `/panfs/storage.local/software/galaxy-dist/database/job_working_direct ory/000/15/galaxy_15.o': No such file or directory *** end error output Any idea as to why galaxy removes the working directory? Is there a setting in the job_conf.xml that would resolve this? Thanks for any pointers. Donny FSU Research Computing Center ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To
Re: [galaxy-dev] jobs submitted to a cluster
Hi John, That did the trick. I have some other problem but I don't think it's galaxy from here. Thanks again for the reply. Donny -Original Message- From: John Chilton [mailto:jmchil...@gmail.com] Sent: Thursday, June 12, 2014 10:53 AM To: Evan Bollig Cc: Shrum, Donald C; galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] jobs submitted to a cluster My guess is Galaxy is deleting the directory because it believe the job is in error because of some communication problem while polling your DRM via DRMAA - Galaxy thinks the job has failed before it has even been run. You can set cleanup_job = never in universe_wsgi.ini's app:main section to instruct Galaxy to not delete the working directory. I suspect this will allow the DRM to finish running your job - but Galaxy is still going to fail it since it cannot properly detect its status. Can you confirm? -John On Thu, Jun 12, 2014 at 8:36 AM, Evan Bollig boll0...@umn.edu wrote: job_conf.xml is outside of my knowledge. Better wait to see what the others can tell us. -E -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu On Thu, Jun 12, 2014 at 8:31 AM, Shrum, Donald C dcsh...@admin.fsu.edu wrote: It's set to 600 seconds so I don't think that is the issue... Is there some sort of wait time to set in job_conf.xml -Original Message- From: Evan Bollig [mailto:boll0...@umn.edu] Sent: Thursday, June 12, 2014 9:27 AM To: Shrum, Donald C Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] jobs submitted to a cluster Hey Donny, What is the value of keep_completed on your queue (from qmgr -c 'p s')? Could it be that your spool is flushing completed jobs immediately? I ran into issues the other day with libdrmaa requiring at least keep_complete = 60 seconds to properly detect completed jobs and clean up after itself. Cheers, -E -Evan Bollig Research Associate | Application Developer | User Support Consultant Minnesota Supercomputing Institute 599 Walter Library 612 624 1447 e...@msi.umn.edu boll0...@umn.edu On Thu, Jun 12, 2014 at 7:36 AM, Shrum, Donald C dcsh...@admin.fsu.edu wrote: I've setup galaxy to submit jobs to our HPC cluster as the logged in user. I used the drama python module to submit the jobs to our moab server. It appears that the working directory for a submitted job is being removed by galaxy prior to the job completing on the cluster. I can see a working directory is created in the logs: galaxy.jobs DEBUG 2014-06-12 08:21:03,786 (15) Working directory for job is: /panfs/storage.local/software/galaxy-dist/database/job_working_direc to ry/000/15 I've confirmed the directory is created by watching on the file system and within about two seconds of the folder being created it is deleted. [root@admin 000]# watch -d ls -lR Every 2.0s: ls -lR Thu Jun 12 08:21:06 2014 total 64 drwxrwxrwx 2 dcshrum dcshrum 4096 Jun 12 08:21 15 I see the job sent via DRMAA: galaxy.jobs.handler DEBUG 2014-06-12 08:21:03,795 (15) Dispatching to drmaa runner galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) submitting file /panfs/storage.local/software/galaxy-dist/database/job_working_direc to ry/000/15/galaxy_15.sh galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,566 (15) native specification is: -N galaxyjob -l nodes=1,walltime=2:00 -q genacc_q galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:05,892 (15) submitting with credentials: dcshrum [uid: 232706] galaxy.jobs.runners.drmaa INFO 2014-06-12 08:21:06,196 (15) queued as 7570705.moab.local The job fails: galaxy.jobs.runners.drmaa DEBUG 2014-06-12 08:21:06,698 (15/7570705.moab.local) state change: job finished, but failed galaxy.jobs.runners DEBUG 2014-06-12 08:21:07,124 (15/7570705.moab.local) Unable to cleanup /panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh: [Errno 2] No such file or directory: '/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/15/galaxy_15.sh' I can see the same error in my moab log: *** error from copy /bin/cp: cannot create regular file `/panfs/storage.local/software/galaxy-dist/database/job_working_dire ct ory/000/15/galaxy_15.o': No such file or directory *** end error output Any idea as to why galaxy removes the working directory? Is there a setting in the job_conf.xml that would resolve this? Thanks for any pointers. Donny FSU Research Computing Center ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at:
[galaxy-dev] Error with fastx reverse compliment
Hello, I'm getting an error with fastx tools on a local install of galaxy. From this error I can't tell if it's a problem with the file, database, or tool. An error occurred with this dataset: zcat: /Volumes/Jose-Drobo/galaxy-dist/database/files/002/dataset_2875.dat.Z: No such file or directory fastx_reverse_complement: Premature End-Of-File (filename ='-') Ian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Browsing toolsheds - HTTP(s) and firewalls
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Today I tried to access my org's toolshed, only to realise that it wouldn't load as it was behind the firewall. Since I'm not in the mood to connect my VPN, I thought I'd posit the following: Perhaps the galaxy server should make the HTTP requests to the toolshed instead of the user's browser. This would solve the existing, frustrating issue of most major browsers refusing to load the main toolshed if you're running HTTPS (as galaxy would request over HTTP then serve over HTTPS, acting as a proxy), and it would solve the issue of not being able to load toolsheds which are behind firewalls. Thoughts? Cheers, Eric - -- Eric Rasche Programmer II Center for Phage Technology Texas AM University College Station, TX 77843 404-692-2048 e...@tamu.edu rasche.e...@yandex.ru -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) iQIcBAEBAgAGBQJTmhWZAAoJEMqDXdrsMcpV12wQAKMfVfyFoWjLpGV+102wSCLY 1TgSEBc8YI2mvJ59z1r6k2EobiBKnvWHYfi2rSkkmzAthn0lke56vSGLMmsjJ/k2 /XT844OmL4ybvL5OyseacZRNpThgeT7wz/KT/3FXtaBP6ssqqAgx2S3meqgzm7yQ T7F08JXzDv/0v+wnqRjWzupbQMpZdorMAc3P+A37e0mYFNKru/Nq1zOJEnYko+um tw3K+lgnhOHidM3s3XOFCsfqn5FizkIpduzQIbl7n4dILa5hGSkJm6KLqyykhR+C RNcBljJeULyOB/UDBvFE+hEqv/IqpqUccphwUHwEQoEnVBtl08MOeuEm50Epp6QT GopzYZ5fBgt38IaAlUV3tVHMGr+NjwUVWrIO8nTnlLoSfcDT3NsQuKlSbEN1JJ6z GjS+b6ZSjng7KF+rgTpU3hu/UGGT8tAzAlxK8ipRvZEFN6TkjSkcdSy9jSQrjYi/ MpTl//cpScPc6G0b/OMl1CbD7VJ6N2oPYx8m47b3dbfF5bUppugKYiPcI6Htx62z 4+FU4MrLkvUcj4O1t1hninRp7DfMag0kbLtHcdUmd1DU0h4TQRYQu+QGaBX9B53Q SU2FzCJXQI2skFC+vMPUNSoIYEKkYPRlBvkKQiMlBHMPA2jcHrBLY+VbLUd7AGJU lAAUgMNh5MXvvr6HM56m =cdqI -END PGP SIGNATURE- ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] multiple file selection for from_file
Hi all I gather that it is NOT allowed to assign multiple file to from_file attribute for options in select param as the expect value of from_file attribute is the name of A file. But would it be nice to allow this feature? For example, there are two loc file A and B, when I want to get the options from both files, instead of adding an extra file containing all options from two files, it will be more maintainable to use something like from_file = A.loc, B.loc, isn't it? BTW, could anyone help me on this http://dev.list.galaxyproject.org/jobs-don-t-get-killed-when-histories-deleted-tc4664678.html? Best regards! Jun ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] galaxy-dev Digest, Vol 96, Issue 14
Hello,
when i run my galaxy instance type:
$ sh run.sh
I get some error like this:
Traceback (most recent call last):
File /export/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py, line 38, in
app_factory
from galaxy.app import UniverseApplication
File /export/galaxy-dist/lib/galaxy/app.py, line 21, in module
from galaxy.queue_worker import GalaxyQueueWorker
File /export/galaxy-dist/lib/galaxy/queue_worker.py, line 10, in module
import galaxy.queues
File /export/galaxy-dist/lib/galaxy/queues.py, line 17, in module
from kombu import Exchange, Queue
File /export/galaxy-dist/eggs/kombu-3.0.13-py2.7.egg/kombu/__init__.py,
line 67, in __getattr__
module = __import__(object_origins[name], None, None, [name])
File /export/galaxy-dist/eggs/kombu-3.0.13-py2.7.egg/kombu/entity.py, line
10, in module
from .abstract import MaybeChannelBound
File /export/galaxy-dist/eggs/kombu-3.0.13-py2.7.egg/kombu/abstract.py,
line 12, in module
from .connection import maybe_channel
File /export/galaxy-dist/eggs/kombu-3.0.13-py2.7.egg/kombu/connection.py,
line 24, in module
from kombu import exceptions
File /export/galaxy-dist/eggs/kombu-3.0.13-py2.7.egg/kombu/exceptions.py,
line 12, in module
from amqp import ChannelError, ConnectionError, ResourceError
File /export/galaxy-dist/eggs/amqp-1.4.5-py2.7.egg/amqp/__init__.py, line
32, in module
from .basic_message import Message # noqa
File /export/galaxy-dist/eggs/amqp-1.4.5-py2.7.egg/amqp/basic_message.py,
line 19, in module
from .serialization import GenericContent
File /export/galaxy-dist/eggs/amqp-1.4.5-py2.7.egg/amqp/serialization.py,
line 33, in module
from .five import int_types, long_t, string, string_t, items
File /export/galaxy-dist/eggs/amqp-1.4.5-py2.7.egg/amqp/five.py, line 171,
in module
librt = ctypes.CDLL('librt.so.1', use_errno=True)
TypeError: __init__() got an unexpected keyword argument 'use_errno'
I have no idea why this happend. How can I fix it?___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
