[galaxy-dev] drmaa://native/ : native options are ignored
Hi, I'm working on an local installation of galaxy using torque with drmaa (the pbs-torque scramble failed). The torque-drmaa works fine so far, except for one issue. I'd like to specify some tool-dependent requirements from the tool_runners section in universe.wsgi.ini. For now I've been testing it with the setting below to have global native arguments : default_cluster_job_runner = drmaa://-l mem=4gb:nodes=1:ppn=6/ This should request 4gb of memory on a single node with 6 threads, but these requests are ignored. They are not listed on 'qstat -R' and more simultaneous jobs than possible are started if the requirements were taken into account. What am I missing here? Best regards, Geert Vandeweyer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] drmaa://native/ : native options are ignored
Hi default_cluster_job_runner = drmaa://-q srpipeline -P pipeline/ works for me on LSF, so your syntax seems to be correct. Assuming that -l mem=4gb:nodes=1:ppn=6 works teh way you expect when you start the jobs on your cluster from the shell, read on... Bearing in mind the the value of the option contains ':' , I'd check that it is read correctly by the parser that parses universe_wsgi.ini and it is passed correctly to drmaa. If it's passed to drmaa correctly, but does not produced desired effect, I'd look closely at the drmaa library. This might be a bug. It's possible to use the drmaa library from a C script - this way you can test if drmaa works the way you want. Also, It was rather straightforward to generate a perl swig wrapper for drmaa and write perl test scripts. Swig wrappers can be generated for any scripting language and also java. Marina On 20/06/2011 10:53, Geert Vandeweyer wrote: Hi, I'm working on an local installation of galaxy using torque with drmaa (the pbs-torque scramble failed). The torque-drmaa works fine so far, except for one issue. I'd like to specify some tool-dependent requirements from the tool_runners section in universe.wsgi.ini. For now I've been testing it with the setting below to have global native arguments : default_cluster_job_runner = drmaa://-l mem=4gb:nodes=1:ppn=6/ This should request 4gb of memory on a single node with 6 threads, but these requests are ignored. They are not listed on 'qstat -R' and more simultaneous jobs than possible are started if the requirements were taken into account. What am I missing here? Best regards, Geert Vandeweyer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] [API] Error message forwarding
Hi, I ran into a little something that is a bit annoying for debug when trying to upload files through the API with library_upload_from_import_dir.py. When the specified folder is wrong, python tries to process the error tuple like a dict, so the original error is hard to find. I modified a little the code to avoid that. It might not work in all cases but at least it solved my problem. Here is the original code: lib/galaxy/web/api/contents.py l.145-end else: rval = [] for k, v in output.items(): if type( v ) == trans.app.model.LibraryDatasetDatasetAssociation: v = v.library_dataset encoded_id = trans.security.encode_id( create_type + '.' + str( v.id ) ) rval.append( dict( id = encoded_id, name = v.name, url = url_for( 'content', library_id=library_id, id=encoded_id ) ) ) return rval Here is how I modified it: else: rval = [] try: for k, v in output.items(): if type( v ) == trans.app.model.LibraryDatasetDatasetAssociation: v = v.library_dataset encoded_id = trans.security.encode_id( create_type + '.' + str( v.id ) ) rval.append( dict( id = encoded_id, name = v.name, url = url_for( 'content', library_id=library_id, id=encoded_id ) ) ) return rval except: log.debug( 'HTTP Error %s: %s' % ( output[0], output[1] ) ) return None Since it saves me some time energy I just wanted to share it, so I hope it can help. Best L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [galaxy-bugs] tool updates for local installation
Helo Ravi, please send all questions to one of the Galaxy mail lists as sending to personal emails will usually not generate responses. I've forwarded your question to the dev list since I am not involved in the Galaxy cloud development efforts. On Jun 17, 2011, at 5:47 PM, Sanka, Ravi wrote: Hello Greg, Thank you for your quick response. I have another, related question. I understand that there is an Amazon-Cloud version of Galaxy called CloudMan, where tool updates are taken care of for the users. Does CloudMan take care of its own updates? If so, how often do the tools get updated? --- Ravi Sanka ICS - Bioinformatics Engineer J. Craig Venter Institute 301-795-7743 --- From: Greg Von Kuster [mailto:g...@bx.psu.edu] Sent: Friday, June 17, 2011 3:24 PM To: Sanka, Ravi Cc: galaxy-b...@bx.psu.edu Subject: Re: [galaxy-bugs] tool updates for local installation Hello Ravi, At the current time, it is up to the owner of the Galaxy instance to look at the Galaxy tool shed for updates, and pull them to their local instance. Regarding 3rd party dependencies, it is again the responsibility of the owner of the Galaxy instance to handle the dependencies. This will change at some point in the future, at which point we'll let you know of the changes. Thanks for your interest! Greg Von Kuster On Jun 17, 2011, at 3:02 PM, Sanka, Ravi wrote: Greetings, My name is Ravi Sanka. I have a question for maintaining tools on a local installation of Galaxy. What happens one or more updates are available for the tools that are part of the Galaxy community? Is it the client’s responsibility to search for updates of each tool on his install? Or do update alerts get sent by Galaxy to keep the tools at the most recent version? And what about those tools that are part of the Galaxy community but use third-party software (i.e. NCBI Blast+)? Is it the same for them, or does a different procedure apply? Thanks, - Ravi Sanka Greg Von Kuster Galaxy Development Team g...@bx.psu.edu Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Sharing authentication between Galaxy and other WSGI apps on the same web server (with custom UI)?
I'd like to have Galaxy and another application installed on the same Apache server and have the user authenticate only once. I think I understand how to do that by deferring authentication to Apache (instead of using Galaxy's built-in database). So far, so good, I think. What I'm wondering is if it is possible (in external user mode) to control the user experience of authentication versus being stuck with the one where the browser pops up the authentication dialog. Is it possible to implement a shared authentication mechanism that uses web pages for the UI? Or would we have to give up Apache-based security and snoop the Galaxy session cookie instead? Many thanks - John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.commailto:jdu...@illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Sharing authentication between Galaxy and other WSGI apps on the same web server (with custom UI)?
Hello John, I'm not an apache expert, but I can try to help with some info. your question involves two different issues, which are not dependent on one another. First, can one setup apache authentication that will affect both Galaxy and other things on your server ? The answer is yes. Example: We've setup our authentication on the root location of the server. Galaxy uses the prefix /galaxy, and other services use other prefixes, and since all of them are below the root location, the authentication applies to all. The user needs to login only once. === ## Root location of the server, protected with NTLM authentication Location / AuthName CSHL AuthType NTLM NTLMAuth on NTLMAuthoritative on ### couple of other authentication parameters... /Location ## ## Galaxy uses load-balancing and mod_rewrite and other things, ## but since it's below the root location, it will use the same authentication # Galaxy server Proxy balancer://galaxyprod BalancerMember http://localhost:8081 BalancerMember http://localhost:8082 /Proxy ReWriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy(.*) balancer://galaxyprod$1 [P] Location /galaxy require valid-user /Location ## ## Other services on the same server will use the same authentication, ## and can also limit user access with require statement. Alias /plans/ /home/gordon/projects/plans/ Location /plans require user gordon /Location === Second, Can apache use authentication which is not built-in in the browser, so instead of OS native ugly dialog, the user will see a custom web page? The answer is still yes, because authentication in Apache is modular. If you specify AuthType BASIC or AuthType Digest or AuthType NTLM (which are the only universally supported built-in authentication methods I'm aware of), then the client-side browser will display an OS native user/password dialog. If you install a custom authentication module, then you can use AuthType CUSTOMXXX (or sometimes a different command) and apache will use the module for custom authentication (which can involve custom webpages or anything else). As long as the custom module notifies apache that the user is authenticated, Apache doesn't care how it's done. There's one apache module called mod_auth_form ( http://httpd.apache.org/docs/trunk/mod/mod_auth_form.html ) which does exactly that, but I'm not sure if it's considered stable. There are other 3rd party solutions available, unfortunately those solution are usually quite complicated and laborious to install (I've read about them but never tried them myself): http://blog.ianbicking.org/more-on-single-signon.html https://neon1.net/mod_auth_pubtkt/ http://cosign.sourceforge.net/ http://mod-auth-script.sourceforge.net/ All of them claim to provide apache integration. And just as in the first question, once you change the AuthType in the root location to a custom authentication module, all the other sub-URLs will use that authentication. If you do get one of those to work, I'm interested in hearing about it, because I would like eventually to get rid of NTLM authentication. Regards, -gordon Duddy, John wrote, On 06/20/2011 02:04 PM: I’d like to have Galaxy and another application installed on the same Apache server and have the user authenticate only once. I think I understand how to do that by deferring authentication to Apache (instead of using Galaxy’s built-in database). So far, so good, I think. What I’m wondering is if it is possible (in external user mode) to control the user experience of authentication versus being stuck with the one where the browser pops up the authentication dialog. Is it possible to implement a shared authentication mechanism that uses web pages for the UI? Or would we have to give up Apache-based security and snoop the Galaxy session cookie instead? Many thanks - *John Duddy Sr. Staff Software Engineer Illumina, Inc. *9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com mailto:jdu...@illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] problems previewing certain files, downloading files and with login
Nate Coraor wrote: Matthew Conte wrote: Hi Nate, I was able to finally track down the login issue. It had to do with the following setting in my universe_wsgi.ini: * * *cookie_path = /galaxy* Removing this out fixed the problem and I should be fine leaving it out out since I don't need to run more than one instance of Galaxy. I probably shouldn't have had this setting in the first place, though I'm not exactly sure what this caused the problem. Anyways, thanks for all the help tracking these issues down. Hi Matt, I was about to get back to this now that the conference is over and I was also wondering if that option was related - weird though, I'm not sure why it should have an effect, but I'm glad you've figured out what's up. I guess there's a bug with the cookie path I'll need to check out. Okay, in testing I can only get this to happen if the browser has two cookies - one with the correct proxied path set and another set using just '/'. This arises from something like a bug in the library Galaxy uses to handle the request (WebOb) and the fact that the cookie headers don't include the cookie path along with the contents, so we'd have to choose one cookie or the other to use as your current session (probably the newer of the two?). Since this should really only happen in the case of a user getting a Galaxy cookie while running with cookie_path unset and then another once it's been set, it should be pretty rare and easily worked around by clearing cookies. If you're still interested in testing this and find that clearing cookies still doesn't fix the issue, please let me know. Thanks, --nate --nate -Matt On Mon, May 16, 2011 at 5:26 PM, Matthew Conte mco...@umd.edu wrote: Yep. On May 16, 2011 7:14 AM, Nate Coraor n...@bx.psu.edu wrote: Matthew Conte wrote: On Fri, May 13, 2011 at 11:18 AM, Nate Coraor n...@bx.psu.edu wrote: That's correct, although it should've been fixed. Can you remove the contents of galaxy-dist/database/compiled_templates, clear cache, reload, and see if you still get the same results? Thanks, --nate I've removed the contents of that folder, cleared the browser cache, reloaded and still get the same results (nothing different in the log either). I'm not sure if this will help, but maybe I should mention that when I try to log in I do get the following screen to flash for a brief second before automatically taking me back to the welcome screen: [image: Screen shot 2011-05-13 at 1.44.25 PM.png] I'm going to try to set up a test environment to replicate this shortly. Does this happen with both Apache and nginx? --nate ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] public datasets/histories with external auth
Hi,
We have a galaxy server setup using external shibboleth authentication. While
we would like to have site behind authentication realm, there are instances
when our galaxy datasets/histories need to be accessible publicly from other
websites. We tried adding an exception to auth rule for /datasets path using
Location directive in apache web server configuration, however galaxy server
returned an error as:
{{{
Access to Galaxy is denied
Galaxy is configured to authenticate users via an external method (such as HTTP
authentication in Apache), but a username was not provided by the upstream
(proxy) server. This is generally due to a misconfiguration in the upstream
server.
}}}
Is there any way to share public histories and datasets when galaxy is using
external authentication mechanism? I have thought about setting up (fake)
anonymous REMOTE_USER variable for /datasets path, but not sure whether this
is correct approach. Also, would it require any galaxy code changes? Any
thoughts?
Thanks,
Shantanu.
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Sharing authentication between Galaxy and other WSGI apps on the same web server (with custom UI)?
FWIW, Penn State uses Cosign for single sign-on, and I've successfully placed Galaxy behind Cosign before (Cosign relies on Kerberos, and so may not be a good fit at all sites). But as Assaf says, basically anything should work. --nate Duddy, John wrote: Thanks! That's perfect. John Duddy Sr. Staff Software Engineer Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com -Original Message- From: Assaf Gordon [mailto:gor...@cshl.edu] Sent: Monday, June 20, 2011 12:09 PM To: Duddy, John; galaxy-...@bx.psu.edu Subject: Re: [galaxy-dev] Sharing authentication between Galaxy and other WSGI apps on the same web server (with custom UI)? Hello John, I'm not an apache expert, but I can try to help with some info. your question involves two different issues, which are not dependent on one another. First, can one setup apache authentication that will affect both Galaxy and other things on your server ? The answer is yes. Example: We've setup our authentication on the root location of the server. Galaxy uses the prefix /galaxy, and other services use other prefixes, and since all of them are below the root location, the authentication applies to all. The user needs to login only once. === ## Root location of the server, protected with NTLM authentication Location / AuthName CSHL AuthType NTLM NTLMAuth on NTLMAuthoritative on ### couple of other authentication parameters... /Location ## ## Galaxy uses load-balancing and mod_rewrite and other things, ## but since it's below the root location, it will use the same authentication # Galaxy server Proxy balancer://galaxyprod BalancerMember http://localhost:8081 BalancerMember http://localhost:8082 /Proxy ReWriteRule ^/galaxy$ /galaxy/ [R] RewriteRule ^/galaxy(.*) balancer://galaxyprod$1 [P] Location /galaxy require valid-user /Location ## ## Other services on the same server will use the same authentication, ## and can also limit user access with require statement. Alias /plans/ /home/gordon/projects/plans/ Location /plans require user gordon /Location === Second, Can apache use authentication which is not built-in in the browser, so instead of OS native ugly dialog, the user will see a custom web page? The answer is still yes, because authentication in Apache is modular. If you specify AuthType BASIC or AuthType Digest or AuthType NTLM (which are the only universally supported built-in authentication methods I'm aware of), then the client-side browser will display an OS native user/password dialog. If you install a custom authentication module, then you can use AuthType CUSTOMXXX (or sometimes a different command) and apache will use the module for custom authentication (which can involve custom webpages or anything else). As long as the custom module notifies apache that the user is authenticated, Apache doesn't care how it's done. There's one apache module called mod_auth_form ( http://httpd.apache.org/docs/trunk/mod/mod_auth_form.html ) which does exactly that, but I'm not sure if it's considered stable. There are other 3rd party solutions available, unfortunately those solution are usually quite complicated and laborious to install (I've read about them but never tried them myself): http://blog.ianbicking.org/more-on-single-signon.html https://neon1.net/mod_auth_pubtkt/ http://cosign.sourceforge.net/ http://mod-auth-script.sourceforge.net/ All of them claim to provide apache integration. And just as in the first question, once you change the AuthType in the root location to a custom authentication module, all the other sub-URLs will use that authentication. If you do get one of those to work, I'm interested in hearing about it, because I would like eventually to get rid of NTLM authentication. Regards, -gordon Duddy, John wrote, On 06/20/2011 02:04 PM: I'd like to have Galaxy and another application installed on the same Apache server and have the user authenticate only once. I think I understand how to do that by deferring authentication to Apache (instead of using Galaxy's built-in database). So far, so good, I think. What I'm wondering is if it is possible (in external user mode) to control the user experience of authentication versus being stuck with the one where the browser pops up the authentication dialog. Is it possible to implement a shared authentication mechanism that uses web pages for the UI? Or would we have to give up Apache-based security and snoop the Galaxy session cookie instead? Many thanks - *John Duddy Sr. Staff Software Engineer Illumina, Inc. *9885 Towne Centre Drive San Diego, CA 92121 Tel: 858-736-3584 E-mail: jdu...@illumina.com mailto:jdu...@illumina.com
Re: [galaxy-dev] public datasets/histories with external auth
Shantanu Pavgi wrote:
Hi,
We have a galaxy server setup using external shibboleth authentication. While
we would like to have site behind authentication realm, there are instances
when our galaxy datasets/histories need to be accessible publicly from other
websites. We tried adding an exception to auth rule for /datasets path using
Location directive in apache web server configuration, however galaxy server
returned an error as:
{{{
Access to Galaxy is denied
Galaxy is configured to authenticate users via an external method (such as
HTTP authentication in Apache), but a username was not provided by the
upstream (proxy) server. This is generally due to a misconfiguration in the
upstream server.
}}}
Is there any way to share public histories and datasets when galaxy is using
external authentication mechanism? I have thought about setting up (fake)
anonymous REMOTE_USER variable for /datasets path, but not sure whether this
is correct approach. Also, would it require any galaxy code changes? Any
thoughts?
Hi Shantanu,
That's about all you can do, or modify
lib/galaxy/web/framework/middleware/remoteuser.py to let these
connections through. I would suggest the former solution of setting a
header in Apache, but only set it if the user is not authenticated.
--nate
Thanks,
Shantanu.
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] public datasets/histories with external auth
On Jun 20, 2011, at 2:40 PM, Nate Coraor wrote:
Shantanu Pavgi wrote:
Hi,
We have a galaxy server setup using external shibboleth authentication.
While we would like to have site behind authentication realm, there are
instances when our galaxy datasets/histories need to be accessible publicly
from other websites. We tried adding an exception to auth rule for /datasets
path using Location directive in apache web server configuration, however
galaxy server returned an error as:
{{{
Access to Galaxy is denied
Galaxy is configured to authenticate users via an external method (such as
HTTP authentication in Apache), but a username was not provided by the
upstream (proxy) server. This is generally due to a misconfiguration in the
upstream server.
}}}
Is there any way to share public histories and datasets when galaxy is using
external authentication mechanism? I have thought about setting up (fake)
anonymous REMOTE_USER variable for /datasets path, but not sure whether
this is correct approach. Also, would it require any galaxy code changes?
Any thoughts?
Hi Shantanu,
That's about all you can do, or modify
lib/galaxy/web/framework/middleware/remoteuser.py to let these
connections through. I would suggest the former solution of setting a
header in Apache, but only set it if the user is not authenticated.
--nate
Thanks for the reply Nate. That's helpful.
--
Shantanu.
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
[galaxy-dev] input security validation
Hello all, I am Russell, a student working with Dr. Victor Jin at the Dept. of Biomedical Informatics, OSU Medical Center. I am developing a Galaxy wrapper for our ChIP-seq peak-calling program BELT (PMID: 21138948), and I have a question about input validation. Does Galaxy filter for malicious field entries (such as XSS attacks) or is this our wrapper's responsibility (the MACS wrapper appears to not perform any explicit checks)? Thank you for your time. Sincerely, Russell Bonneville ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
