[galaxy-dev] Coverage for paired end data
Hi all. I am analysing paired end chip-seq samples. With single end reads I have seen people extend reads by an inferred insert size and then create coverage files in wig format. I was wondering if there are any tools available for paired end data that will create a coverage file by looking at properly mapped paired ends in a bam file and extending coverage across the distance between them. Thanks Shaun Webb -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] tool shed offline?
This was caused by a server crash which has been resolved. Sorry for the inconvenience. On Feb 1, 2012, at 12:44 AM, steve.mcma...@csiro.au wrote: Hi, This is my first post to this list. I’m working on a Galaxy service within CSIRO in Australia. I can’t seem to access http://toolshed.g2.bx.psu.edu/ at the moment. Is it offline or is there something else between here and there preventing the access? Regards. Steve McMahon Solutions architect senior systems administrator ASC Cluster Services Information Management Technology (IMT) CSIRO Phone: +61-2-62142968 | Mobile: +61-4-00779318 steve.mcma...@csiro.au | www.csiro.au PO Box 225, DICKSON ACT 2602 1 Wilf Crane Crescent, Yarralumla ACT 2600 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] generic sounding error when trying to upload files to data library
Hi Dhivya, Have you updated this instance or made any other changes recently? Do you see any errors in the logs? -Dannon On Jan 31, 2012, at 3:03 PM, dhivya arasappan wrote: Hi, We have our own galaxy instance and I'm trying to add datasets to a data library. It has always worked before, but today, when I choose any of the upload options, it gives the following error: This link may not be followed from within Galaxy. It seems like a very generic error. Any ideas of what could be going on? Thanks for the help. Dhivya ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] generic sounding error when trying to upload files to data library
Dhivya and I just got this fixed. We ended up restarting nginx and the upload pages appeared when selected. I believe the the wrong nginx process was running, because we had to compile nginx from source in order to enable the upload module, which was located in a different directory. Thanks for the reply Dannon. Joe On Wed, Feb 1, 2012 at 8:22 AM, Dannon Baker dannonba...@me.com wrote: Hi Dhivya, Have you updated this instance or made any other changes recently? Do you see any errors in the logs? -Dannon On Jan 31, 2012, at 3:03 PM, dhivya arasappan wrote: Hi, We have our own galaxy instance and I'm trying to add datasets to a data library. It has always worked before, but today, when I choose any of the upload options, it gives the following error: This link may not be followed from within Galaxy. It seems like a very generic error. Any ideas of what could be going on? Thanks for the help. Dhivya ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error viewing BAM files in IGV
On Dec 19, 2011, at 12:24 PM, Jim Johnson wrote: Hi, I'm seeing the same behavior. Galaxy is returning a web page rather than the requested .bai index file for the BAM file. In class WebApplication ( lib/galaxy/web/framework/base.py ) in __call__( self, environ, start_response ) line 133 # Setup the transaction trans = self.transaction_factory( environ ) Gets routed to the root controller and returns the galaxy server main page html. JJ Hi JJ, What URL is causing this? I figured this was a proxy prefix problem, but I tried setting it up here and the URL to the bai is still correct: http://example.org/proxy_prefix/display_application/encoded_dataset_id/igv_bam/local_default/None/data/galaxy_encoded_dataset_id.bam.bai --nate On Dec 14, 2011, at 4:22 PM, Alexander Graf wrote: Hi nate, I have tried it with several BAM files, resulting in the same error. If I download the BAM and bai files from Galaxy and load it into IGV manually, everything works like charm. Up to now I could not figure out why it is not working. Could I have better success switching to the nginx-server? Hi Alex, It should work with Apache as well. I don't have an environment set up here to test, but could you take a look at the Apache access and error logs to determine whether the file is being found and read properly? It's possible that the request is returning something other than a 200 code and the file data. --nate Alex Am 12.12.2011 um 16:11 schrieb Nate Coraor: On Dec 9, 2011, at 6:45 AM, Alexander Graf wrote: Hello, I have recently updated our Galaxy dist and I'm running into problems viewing BAM files in IGV (v2.0.22), saying: Invalid GZIP header. Hi Alex, Your config below looks okay at first glance. Can you verify that the file in question is a valid BAM? Or is this happening with all BAMs? --nate I have configured the Apache as explained in the Wiki using this http.conf: VirtualHost *:80 ServerName 127.0.0.1 RewriteEngine on RewriteRule ^ /galaxy$ /galaxy/ [R] RewriteRule ^ /galaxy/static/style/(.*) /opt/galaxy/static/june_2007_style/blue/ $1 [L] RewriteRule ^ /galaxy/static/scripts/(.*) /opt/galaxy/static/scripts/packed/ $1 [L] RewriteRule ^ /galaxy/static/(.*) /opt/galaxy/static/ $1 [L] RewriteRule ^/galaxy/favicon.ico /opt/galaxy/static/favicon.ico [L] RewriteRule ^/galaxy/robots.txt /opt/galaxy/static/robots.txt [L] RewriteRule ^/galaxy(.*) http://localhost:8081$1 [P] Proxy http://localhost:8081 Order deny,allow Allow from all /Proxy Location /galaxy # Define the authentication method XSendFile on XSendFilePath / # Compress all uncompressed content.http.conf SetOutputFilter DEFLATE SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary /Location Directory /galaxy/static ExpiresActive On ExpiresDefault access plus 6 hours /Directory /VirtualHost By having this parts changed in universe_wsgi.ini: [server:main] use = egg:Paste#http port = 8081 host = 0.0.0.0 use_threadpool = True [filter:gzip] use = egg:Paste#gzip [filter:proxy-prefix] use = egg:PasteDeploy#prefix prefix = /galaxy [app:main] paste.app_factory = galaxy.web.buildapp:app_factory filter-with = proxy-prefix cookie_path = /galaxy apache_xsendfile = True upstream_gzip = False The resulting Galaxy error log is: - galaxy.web.framework DEBUG 2011-12-09 12:32:08,825 Error: this request returned None from get_history(): http://10.153.182.203/galaxy/root 10.163.241.110 - - [09/Dec/2011:12:32:08 +0200] GET /galaxy/root?app_action=datauser_id=c9a3f3a19e75965dapp_name=igv_bamlink_name=local_defaultaction_param=galaxy_9b0f702d0207cd78.bam.baidataset_id=9b0f702d0207cd78 HTTP/1.1 200 - - IGV Version 2.0.22 (1360)11/29/2011 02:24 PM Java/1.6.0_22 10.163.241.110 - - [09/Dec/2011:12:32:08 +0200] HEAD /galaxy/display_application/9b0f702d0207cd78/igv_bam/local_default/c9a3f3a19e75965d/data/galaxy_9b0f702d0207cd78.bam HTTP/1.1 302 - - IGV Version 2.0.22 (1360)11/29/2011 02:24 PM Java/1.6.0_22 galaxy.web.framework
Re: [galaxy-dev] Error viewing BAM files in IGV
Nate, John Chilton got that fixed for us. As you suggest, it involved a fix in our proxy configuration. JJ On 2/1/12 9:01 AM, Nate Coraor wrote: On Dec 19, 2011, at 12:24 PM, Jim Johnson wrote: Hi, I'm seeing the same behavior. Galaxy is returning a web page rather than the requested .bai index file for the BAM file. In class WebApplication ( lib/galaxy/web/framework/base.py ) in __call__( self, environ, start_response ) line 133 # Setup the transaction trans = self.transaction_factory( environ ) Gets routed to the root controller and returns the galaxy server main page html. JJ Hi JJ, What URL is causing this? I figured this was a proxy prefix problem, but I tried setting it up here and the URL to the bai is still correct: http://example.org/proxy_prefix/display_application/encoded_dataset_id/igv_bam/local_default/None/data/galaxy_encoded_dataset_id.bam.bai --nate On Dec 14, 2011, at 4:22 PM, Alexander Graf wrote: Hi nate, I have tried it with several BAM files, resulting in the same error. If I download the BAM and bai files from Galaxy and load it into IGV manually, everything works like charm. Up to now I could not figure out why it is not working. Could I have better success switching to the nginx-server? Hi Alex, It should work with Apache as well. I don't have an environment set up here to test, but could you take a look at the Apache access and error logs to determine whether the file is being found and read properly? It's possible that the request is returning something other than a 200 code and the file data. --nate Alex Am 12.12.2011 um 16:11 schrieb Nate Coraor: On Dec 9, 2011, at 6:45 AM, Alexander Graf wrote: Hello, I have recently updated our Galaxy dist and I'm running into problems viewing BAM files in IGV (v2.0.22), saying: Invalid GZIP header. Hi Alex, Your config below looks okay at first glance. Can you verify that the file in question is a valid BAM? Or is this happening with all BAMs? --nate I have configured the Apache as explained in the Wiki using this http.conf: VirtualHost *:80 ServerName 127.0.0.1 RewriteEngine on RewriteRule ^ /galaxy$ /galaxy/ [R] RewriteRule ^ /galaxy/static/style/(.*) /opt/galaxy/static/june_2007_style/blue/ $1 [L] RewriteRule ^ /galaxy/static/scripts/(.*) /opt/galaxy/static/scripts/packed/ $1 [L] RewriteRule ^ /galaxy/static/(.*) /opt/galaxy/static/ $1 [L] RewriteRule ^/galaxy/favicon.ico /opt/galaxy/static/favicon.ico [L] RewriteRule ^/galaxy/robots.txt /opt/galaxy/static/robots.txt [L] RewriteRule ^/galaxy(.*) http://localhost:8081$1 [P] Proxy http://localhost:8081 Order deny,allow Allow from all /Proxy Location /galaxy # Define the authentication method XSendFile on XSendFilePath / # Compress all uncompressed content.http.conf SetOutputFilter DEFLATE SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary /Location Directory /galaxy/static ExpiresActive On ExpiresDefault access plus 6 hours /Directory /VirtualHost By having this parts changed in universe_wsgi.ini: [server:main] use = egg:Paste#http port = 8081 host = 0.0.0.0 use_threadpool = True [filter:gzip] use = egg:Paste#gzip [filter:proxy-prefix] use = egg:PasteDeploy#prefix prefix = /galaxy [app:main] paste.app_factory = galaxy.web.buildapp:app_factory filter-with = proxy-prefix cookie_path = /galaxy apache_xsendfile = True upstream_gzip = False The resulting Galaxy error log is: - galaxy.web.framework DEBUG 2011-12-09 12:32:08,825 Error: this request returned None from get_history(): http://10.153.182.203/galaxy/root 10.163.241.110 - - [09/Dec/2011:12:32:08 +0200] GET /galaxy/root?app_action=datauser_id=c9a3f3a19e75965dapp_name=igv_bamlink_name=local_defaultaction_param=galaxy_9b0f702d0207cd78.bam.baidataset_id=9b0f702d0207cd78 HTTP/1.1 200 - - IGV Version 2.0.22 (1360)11/29/2011 02:24 PM Java/1.6.0_22 10.163.241.110 - - [09/Dec/2011:12:32:08 +0200] HEAD /galaxy/display_application/9b0f702d0207cd78/igv_bam/local_default/c9a3f3a19e75965d/data/galaxy_9b0f702d0207cd78.bam HTTP/1.1 302 - - IGV Version 2.0.22 (1360)11/29/2011 02:24 PM Java/1.6.0_22
Re: [galaxy-dev] Error viewing BAM files in IGV
On Feb 1, 2012, at 10:05 AM, Jim Johnson wrote: Nate, John Chilton got that fixed for us. As you suggest, it involved a fix in our proxy configuration. JJ Great, thanks! --nate On 2/1/12 9:01 AM, Nate Coraor wrote: On Dec 19, 2011, at 12:24 PM, Jim Johnson wrote: Hi, I'm seeing the same behavior. Galaxy is returning a web page rather than the requested .bai index file for the BAM file. In class WebApplication ( lib/galaxy/web/framework/base.py ) in __call__( self, environ, start_response ) line 133 # Setup the transaction trans = self.transaction_factory( environ ) Gets routed to the root controller and returns the galaxy server main page html. JJ Hi JJ, What URL is causing this? I figured this was a proxy prefix problem, but I tried setting it up here and the URL to the bai is still correct: http://example.org/proxy_prefix/display_application/encoded_dataset_id/igv_bam/local_default/None/data/galaxy_encoded_dataset_id.bam.bai --nate On Dec 14, 2011, at 4:22 PM, Alexander Graf wrote: Hi nate, I have tried it with several BAM files, resulting in the same error. If I download the BAM and bai files from Galaxy and load it into IGV manually, everything works like charm. Up to now I could not figure out why it is not working. Could I have better success switching to the nginx-server? Hi Alex, It should work with Apache as well. I don't have an environment set up here to test, but could you take a look at the Apache access and error logs to determine whether the file is being found and read properly? It's possible that the request is returning something other than a 200 code and the file data. --nate Alex Am 12.12.2011 um 16:11 schrieb Nate Coraor: On Dec 9, 2011, at 6:45 AM, Alexander Graf wrote: Hello, I have recently updated our Galaxy dist and I'm running into problems viewing BAM files in IGV (v2.0.22), saying: Invalid GZIP header. Hi Alex, Your config below looks okay at first glance. Can you verify that the file in question is a valid BAM? Or is this happening with all BAMs? --nate I have configured the Apache as explained in the Wiki using this http.conf: VirtualHost *:80 ServerName 127.0.0.1 RewriteEngine on RewriteRule ^ /galaxy$ /galaxy/ [R] RewriteRule ^ /galaxy/static/style/(.*) /opt/galaxy/static/june_2007_style/blue/ $1 [L] RewriteRule ^ /galaxy/static/scripts/(.*) /opt/galaxy/static/scripts/packed/ $1 [L] RewriteRule ^ /galaxy/static/(.*) /opt/galaxy/static/ $1 [L] RewriteRule ^/galaxy/favicon.ico /opt/galaxy/static/favicon.ico [L] RewriteRule ^/galaxy/robots.txt /opt/galaxy/static/robots.txt [L] RewriteRule ^/galaxy(.*) http://localhost:8081$1 [P] Proxy http://localhost:8081 Order deny,allow Allow from all /Proxy Location /galaxy # Define the authentication method XSendFile on XSendFilePath / # Compress all uncompressed content.http.conf SetOutputFilter DEFLATE SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.(?:t?gz|zip|bz2)$ no-gzip dont-vary /Location Directory /galaxy/static ExpiresActive On ExpiresDefault access plus 6 hours /Directory /VirtualHost By having this parts changed in universe_wsgi.ini: [server:main] use = egg:Paste#http port = 8081 host = 0.0.0.0 use_threadpool = True [filter:gzip] use = egg:Paste#gzip [filter:proxy-prefix] use = egg:PasteDeploy#prefix prefix = /galaxy [app:main] paste.app_factory = galaxy.web.buildapp:app_factory filter-with = proxy-prefix cookie_path = /galaxy apache_xsendfile = True upstream_gzip = False The resulting Galaxy error log is: - galaxy.web.framework DEBUG 2011-12-09 12:32:08,825 Error: this request returned None from get_history(): http://10.153.182.203/galaxy/root 10.163.241.110 - - [09/Dec/2011:12:32:08 +0200] GET /galaxy/root?app_action=datauser_id=c9a3f3a19e75965dapp_name=igv_bamlink_name=local_defaultaction_param=galaxy_9b0f702d0207cd78.bam.baidataset_id=9b0f702d0207cd78 HTTP/1.1 200 - - IGV Version 2.0.22 (1360)11/29/2011 02:24 PM Java/1.6.0_22 10.163.241.110 - - [09/Dec/2011:12:32:08
Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)
Hi Nate, Yeah, looks like I have no choice... I'll try to negociate with the IT guys to run the runner on machine B. Thanks for all the information! BTW, out of curiosity, is submitting jobs across machines possible with Torque? Best, L-A Le 30/01/2012 18:20, Nate Coraor a écrit : On Jan 30, 2012, at 12:07 PM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the leads! But setting DRMAA_LIBRARY_PATH means I'm in trouble since the libraries are on machine B which is maintained by our IT dept. I cannot access them from machine A. Is it a desperate situation? Will it work if I have a copy of those libs somewhere? :/ Hi L-A, The Galaxy server will need to be a submission host, so I believe it will have to have PBS Pro installed. If it has this, then the FedStage DRMAA library should be installable on the same host. It may be possible copy the libraries, although I don't know whether you'd be able to configure the server address without access to the directories in which the library will look for its configuration. --nate Best, L-A Le 30/01/2012 17:18, Nate Coraor a écrit : On Jan 30, 2012, at 7:37 AM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the info! I'm trying to understand how the URL for DRMAA works but I don't understand how we can set it so it uses a different machine. Our Galaxy runner runs on machine A and the cluster is on machine B, where do I put B in the URL? In the wiki there is this example: drmaa://[native_options]/ I'm a bit confused, I would have expected something like: drmaa://[machine]/[native_options]/ like for TORQUE. Did I miss something? Hi L-A, Hrm, I've only used it with SGE, which uses an environment variable to define the cell location, and LSF, which I don't remember, but I assume it used the default. I think if you configure the PBS Pro client on the submission host and point DRMAA_LIBRARY_PATH at the correct libdrmaa, it will use your client configuration. There are other PBS Pro users on the list who can hopefully chime in with more details. --nate Best, L-A Le 19/01/2012 19:43, Nate Coraor a écrit : On Jan 16, 2012, at 5:22 AM, Louise-Amélie Schmitt wrote: Hello, We want to move Galaxy's jobs from our small TORQUE local install to a big cluster running PBS Pro. In the universe_wsgi.ini, I changed the cluster address as follows: default_cluster_job_runner = pbs:/// to: default_cluster_job_runner = pbs://sub-master/clng_new/ where sub-master is the name of the machine and clng_new is the queue. However, I get an error when trying to run any job: galaxy.jobs.runners.pbs ERROR 2012-01-16 11:10:00,894 Connection to PBS server for submit failed: 111: Could not find a text for this error, uhhh This corresponds to the qsub error 111 (Cannot connect to specified server host) which is, for some reason, caught by pbs_python as an error of its own (111 not corresponding to any pbs_python error code, hence the face-plant-message). Our guess is that we might need to re-scramble the pbs_python egg with PBS pro's libraries, is that correct? If it's the case, what do we have to set as LIBTORQUE_DIR? Hi L-A, pbs_python is only designed for TORQUE, I don't think it is compatible with the PBS Pro API. For that, you need to use the drmaa runner, which uses the FedStage libdrmaa for PBS Pro. --nate Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error viewing BAM files in IGV
On Jan 17, 2012, at 10:50 AM, Alexander Graf wrote:
Hi Nate,
sorry for my late reply, but I was on holiday last weeks.
I have looked in my apache error.log and access.log resulting into no new
insights - no output is given there.
So I gave nginx a try, getting the same error as before.
I think it is not a configuration issue of apache or nginx, but I don't have
any clue any more what's going on.
Hope you or somebody else can help me figure out the problem.
Hi Alex,
I thought this might be a problem with the way Galaxy was handling proxy
prefixes (/galaxy) but I don't believe that's the problem. It looks like your
XSendFile config is not working, as evidenced by the 'HEAD' HTTP commands in
the Galaxy log (these should be interrupted by the Apache/nginx server and not
passed through to Galaxy). Check your proxy server logs.
--nate
Alex
Below are the newly generated error logs:
If I try to view the BAM file locally, I get:
XX.XXX.XXX.XXX - - [17/Jan/2012:16:34:25 +0200] GET
/galaxy/display_application/8b4e4b7bbaf9bc68/igv_bam/local_default HTTP/1.0
302 - http://10.153.182.203/galaxy/history; Mozilla/5.0 (Ubuntu; X11; Linux
x86_64; rv:9.0.1) Gecko/20100101 Firefox/9.0.1
XX.XXX.XXX.XXX - - [17/Jan/2012:16:34:25 +0200] GET
/galaxy/display_application/8b4e4b7bbaf9bc68/igv_bam/local_default/5ce1ba9f488a6ce5/data/galaxy_8b4e4b7bbaf9bc68.bam.bai
HTTP/1.0 302 - - IGV Version 2.0.34 (1623)01/14/2012 12:45 AM
Java/1.6.0_22
galaxy.web.framework DEBUG 2012-01-17 16:34:25,389 Error: this request
returned None from get_history(): http://10.153.182.203/galaxy/root
XX.XXX.XXX.XXX - - [17/Jan/2012:16:34:25 +0200] GET
/galaxy/root?app_action=datauser_id=5ce1ba9f488a6ce5app_name=igv_bamlink_name=local_defaultaction_param=galaxy_8b4e4b7bbaf9bc68.bam.baidataset_id=8b4e4b7bbaf9bc68
HTTP/1.0 200 - - IGV Version 2.0.34 (1623)01/14/2012 12:45 AM
Java/1.6.0_22
XX.XXX.XXX.XXX - - [17/Jan/2012:16:34:25 +0200] HEAD
/galaxy/display_application/8b4e4b7bbaf9bc68/igv_bam/local_default/5ce1ba9f488a6ce5/data/galaxy_8b4e4b7bbaf9bc68.bam
HTTP/1.0 302 - - IGV Version 2.0.34 (1623)01/14/2012 12:45 AM
Java/1.6.0_22
galaxy.web.framework DEBUG 2012-01-17 16:34:25,470 Error: this request
returned None from get_history(): http://10.153.182.203/galaxy/root
XX.XXX.XXX.XXX - - [17/Jan/2012:16:34:25 +0200] HEAD
/galaxy/root?app_action=datauser_id=5ce1ba9f488a6ce5app_name=igv_bamlink_name=local_defaultaction_param=galaxy_8b4e4b7bbaf9bc68.bamdataset_id=8b4e4b7bbaf9bc68
HTTP/1.0 200 - - IGV Version 2.0.34 (1623)01/14/2012 12:45 AM
Java/1.6.0_22
XX.XXX.XXX.XXX - - [17/Jan/2012:16:34:25 +0200] GET
/galaxy/display_application/8b4e4b7bbaf9bc68/igv_bam/local_default/5ce1ba9f488a6ce5/data/galaxy_8b4e4b7bbaf9bc68.bam
HTTP/1.0 302 - - IGV Version 2.0.34 (1623)01/14/2012 12:45 AM
Java/1.6.0_22
galaxy.web.framework DEBUG 2012-01-17 16:34:25,639 Error: this request
returned None from get_history(): http://10.153.182.203/galaxy/root
XX.XXX.XXX.XXX - - [17/Jan/2012:16:34:25 +0200] GET
/galaxy/root?app_action=datauser_id=5ce1ba9f488a6ce5app_name=igv_bamlink_name=local_defaultaction_param=galaxy_8b4e4b7bbaf9bc68.bamdataset_id=8b4e4b7bbaf9bc68
HTTP/1.0 200 - - IGV Version 2.0.34 (1623)01/14/2012 12:45 AM
Java/1.6.0_22
and if I run it with the web current, I still get:
galaxy.web.framework DEBUG 2012-01-17 16:35:26,820 Error: this request
returned None from get_history(): http://10.153.182.203/galaxy/root
XX.XXX.XXX.XXX - - [17/Jan/2012:16:35:26 +0200] HEAD
/galaxy/root?app_action=datauser_id=5ce1ba9f488a6ce5app_name=igv_bamlink_name=web_link_mainaction_param=galaxy_8b4e4b7bbaf9bc68.bamdataset_id=8b4e4b7bbaf9bc68
HTTP/1.0 200 - - Java/1.6.0_22
Exception happened during processing of request from ('127.0.0.1', 33901)
Traceback (most recent call last):
File /opt/galaxy/eggs/Paste-1.6-py2.6.egg/paste/httpserver.py, line 1053,
in process_request_in_thread
self.finish_request(request, client_address)
File /usr/lib/python2.6/SocketServer.py, line 322, in finish_request
self.RequestHandlerClass(request, client_address, self)
File /usr/lib/python2.6/SocketServer.py, line 618, in __init__
self.finish()
File /usr/lib/python2.6/SocketServer.py, line 661, in finish
self.wfile.flush()
File /usr/lib/python2.6/socket.py, line 297, in flush
self._sock.sendall(buffer(data, write_offset, buffer_size))
error: [Errno 32] Broken pipe
XX.XXX.XXX.XXX - - [17/Jan/2012:16:35:26 +0200] GET
/galaxy/display_application/8b4e4b7bbaf9bc68/igv_bam/web_link_main/5ce1ba9f488a6ce5/data/galaxy_8b4e4b7bbaf9bc68.bam
HTTP/1.0 302 - - Java/1.6.0_22
galaxy.web.framework DEBUG 2012-01-17 16:35:26,910 Error: this request
returned None from get_history(): http://10.153.182.203/galaxy/root
2011/12/19 Nate Coraor n...@bx.psu.edu
On Dec 14, 2011, at 4:22 PM, Alexander
[galaxy-dev] Missing eggs for DRMMA?
Hi all, I'm working on a test setup using an SGE cluster. During start of run.sh up I see: galaxy.jobs ERROR 2012-02-01 15:03:14,164 Job runner is not loadable: galaxy.jobs.runners.drmma Traceback (most recent call last): File /data/galaxy/galaxy-central/lib/galaxy/jobs/__init__.py, line 1187, in _load_plugin module = __import__( module_name ) ImportError: No module named drmma According to this wiki page, Galaxy provides the DRMMA egg: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster However, Galaxy is not downloading a DRMAA egg, in fact there are none here: http://eggs.g2.bx.psu.edu/DRMAA_python/ Is this a documentation problem? Or a just missing egg? Should I scramble it myself locally? Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Missing eggs for DRMMA?
On Feb 1, 2012, at 10:14 AM, Peter Cock wrote: Hi all, I'm working on a test setup using an SGE cluster. During start of run.sh up I see: galaxy.jobs ERROR 2012-02-01 15:03:14,164 Job runner is not loadable: galaxy.jobs.runners.drmma Traceback (most recent call last): File /data/galaxy/galaxy-central/lib/galaxy/jobs/__init__.py, line 1187, in _load_plugin module = __import__( module_name ) ImportError: No module named drmma According to this wiki page, Galaxy provides the DRMMA egg: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Cluster However, Galaxy is not downloading a DRMAA egg, in fact there are none here: http://eggs.g2.bx.psu.edu/DRMAA_python/ Is this a documentation problem? Or a just missing egg? Should I scramble it myself locally? Hi Peter, I'm guessing it's a typo in your config. 'drmaa' vs. 'drmma'. --nate Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Missing eggs for DRMMA?
On Wed, Feb 1, 2012 at 3:28 PM, Nate Coraor n...@bx.psu.edu wrote: Hi Peter, I'm guessing it's a typo in your config. 'drmaa' vs. 'drmma'. --nate Indeed, $ sudo -u galaxy ./run.sh Some eggs are out of date, attempting to fetch... Fetched http://eggs.g2.bx.psu.edu/drmaa/drmaa-0.4b3-py2.6.egg Fetch successful. ... And I was confused by finding this folder but http://eggs.g2.bx.psu.edu/DRMAA_python/ not spotting this one later in the listing, http://eggs.g2.bx.psu.edu/drmaa/ That seems to do it, cheers! Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Missing eggs for DRMMA?
On Feb 1, 2012, at 10:33 AM, Peter Cock wrote: On Wed, Feb 1, 2012 at 3:28 PM, Nate Coraor n...@bx.psu.edu wrote: Hi Peter, I'm guessing it's a typo in your config. 'drmaa' vs. 'drmma'. --nate Indeed, $ sudo -u galaxy ./run.sh Some eggs are out of date, attempting to fetch... Fetched http://eggs.g2.bx.psu.edu/drmaa/drmaa-0.4b3-py2.6.egg Fetch successful. ... And I was confused by finding this folder but http://eggs.g2.bx.psu.edu/DRMAA_python/ not spotting this one later in the listing, http://eggs.g2.bx.psu.edu/drmaa/ The former is the old SGE-only C-extension DRMAA egg, not used any more but we leave all the eggs up in case someone pulls an old version of Galaxy for some reason. --nate That seems to do it, cheers! Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Switching TorquePBSpro: qsub error 111 (cannot connect to server)
On Feb 1, 2012, at 10:09 AM, Louise-Amélie Schmitt wrote: Hi Nate, Yeah, looks like I have no choice... I'll try to negociate with the IT guys to run the runner on machine B. You don't need a full PBS server on that machine, just the client libraries and configuration. i.e. if you can run 'qstat' from machine A and have it return the queue on machine B, that should be all you need. Thanks for all the information! BTW, out of curiosity, is submitting jobs across machines possible with Torque? Torque's syntax allows you to specify the server name right in the runner URL, but pbs_python links against libtorque, which is part of the torque client, which must be installed somewhere on the local system. Best, L-A Le 30/01/2012 18:20, Nate Coraor a écrit : On Jan 30, 2012, at 12:07 PM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the leads! But setting DRMAA_LIBRARY_PATH means I'm in trouble since the libraries are on machine B which is maintained by our IT dept. I cannot access them from machine A. Is it a desperate situation? Will it work if I have a copy of those libs somewhere? :/ Hi L-A, The Galaxy server will need to be a submission host, so I believe it will have to have PBS Pro installed. If it has this, then the FedStage DRMAA library should be installable on the same host. It may be possible copy the libraries, although I don't know whether you'd be able to configure the server address without access to the directories in which the library will look for its configuration. --nate Best, L-A Le 30/01/2012 17:18, Nate Coraor a écrit : On Jan 30, 2012, at 7:37 AM, Louise-Amélie Schmitt wrote: Hi Nate, Thanks for the info! I'm trying to understand how the URL for DRMAA works but I don't understand how we can set it so it uses a different machine. Our Galaxy runner runs on machine A and the cluster is on machine B, where do I put B in the URL? In the wiki there is this example: drmaa://[native_options]/ I'm a bit confused, I would have expected something like: drmaa://[machine]/[native_options]/ like for TORQUE. Did I miss something? Hi L-A, Hrm, I've only used it with SGE, which uses an environment variable to define the cell location, and LSF, which I don't remember, but I assume it used the default. I think if you configure the PBS Pro client on the submission host and point DRMAA_LIBRARY_PATH at the correct libdrmaa, it will use your client configuration. There are other PBS Pro users on the list who can hopefully chime in with more details. --nate Best, L-A Le 19/01/2012 19:43, Nate Coraor a écrit : On Jan 16, 2012, at 5:22 AM, Louise-Amélie Schmitt wrote: Hello, We want to move Galaxy's jobs from our small TORQUE local install to a big cluster running PBS Pro. In the universe_wsgi.ini, I changed the cluster address as follows: default_cluster_job_runner = pbs:/// to: default_cluster_job_runner = pbs://sub-master/clng_new/ where sub-master is the name of the machine and clng_new is the queue. However, I get an error when trying to run any job: galaxy.jobs.runners.pbs ERROR 2012-01-16 11:10:00,894 Connection to PBS server for submit failed: 111: Could not find a text for this error, uhhh This corresponds to the qsub error 111 (Cannot connect to specified server host) which is, for some reason, caught by pbs_python as an error of its own (111 not corresponding to any pbs_python error code, hence the face-plant-message). Our guess is that we might need to re-scramble the pbs_python egg with PBS pro's libraries, is that correct? If it's the case, what do we have to set as LIBTORQUE_DIR? Hi L-A, pbs_python is only designed for TORQUE, I don't think it is compatible with the PBS Pro API. For that, you need to use the drmaa runner, which uses the FedStage libdrmaa for PBS Pro. --nate Thanks, L-A ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Unix user account/connection and Galaxy connection
Hi all - I am working on transitioning our test galaxy - production and this is how I have decided to handle unix galaxy users. I have two groups created on our cluster. 1) galaxyadmin and 2) galaxyusers. galaxyadmin is a subset of galaxyusers. We have authentication occurring in an apache front to our campus id/passwd. I then place the campus id as well as campus email in the request header during the proxy to galaxy. The email is used as the galaxy id. I have augmented galaxy code such that when a new user logs onto galaxy and galaxy auto-creates the galaxy account, it also kicks off scripts that configure the unix account. It creates a new unix account if it does not exist (group galaxyusers), otherwise will augment the existing user's default group (keeping all other group relationships) to be galaxyusers. I have installed the latest version of openSSH. I have prevented all galaxyusers from sshing into the box, and instead set up a sftp jail. The sftp jail directory is not the user's home directory. The sshd_config also allows me to specify a sftp umask, so when a user puts a file into their sftp jail, it will be readable/writeable to anyone in the galaxyusers group (note the jail and preventing ssh does not expose the files to any other users). As part of the scripts that get kicked off in galaxy to create/mod a user - I also create the necessary ftp links to galaxy. I create a symlink from galaxy's ftp directory with the galaxy user id (email) to the user's jailed sftp directory (username). This allows galaxy runtime to import without problems since the files are galaxyusers group read/writeable. My galaxy runs under a galaxy userid (not root) but I have augmented the sudoers file to allow for just that id to have access to specific commands in order to adjust useraccounts. This has been all working in our production prototype testing and we will be having a security review on it shortly - but it should be sound. Hope this helps! Message: 11 Date: Mon, 30 Jan 2012 09:31:15 -0500 From: David Hoover hoove...@helix.nih.gov To: Sarah Maman sarah.ma...@toulouse.inra.fr Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Unix user account/connection and Galaxy connection Message-ID: a0ee343c-c9f5-4039-b684-dd370fb2f...@helix.nih.gov Content-Type: text/plain; charset=us-ascii I'd love to know answers to this question as well. So far, all I've done is create two tools for copying files between Galaxy and a user's personal directories using a rather dangerous setuid executable. I have mulled over two possibilities for this problem. One is to run Galaxy as root. The other is for each user to kick off their own personal instance of Galaxy as themselves, but accessing the same main database and file repository. The latter would require all files be read-write accesible to all users. Neither is good. On Jan 30, 2012, at 4:29 AM, Sarah Maman wrote: Dear all, There is currently no link between an user ssh connection on his own account (own space on Unix) and the Galaxy connection. How to run Galaxy on an Unix user account? How to link the data storage system and the Unix user account (symlink?)? The goal is to not need to copy or move data. PS : Thanks a lot to David, Gordon, Brad, Christophe and Nate for your help on LDAP authentification. I always try to find a solution (Apache configuration). I will inform Galaxy list when I will find a solution. Thanks in advance, Sarah ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Uploading metadata with file using Web API
On Jan 10, 2012, at 5:11 PM, Oren Livne wrote: Dear All, I have a file path that I'd like to link into Galaxy using a RESTful WEB API call with some additional metadata key-value pairs. How can I do that? I know the /api/libraries/ call respects JSON keys like folder_id, file_type, etc. But can I pass to it additional keys and it will know to store them with the file entry? Hi Oren, The API does not currently support setting any of the metadata, either on upload or afterward. Sorry for the inconvenience. Improvements would be welcome! --nate Thank you so much, Oren ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Moving Galaxy's installed folder
On Feb 1, 2012, at 12:49 PM, Peter Cock wrote: On Wed, Feb 1, 2012 at 5:44 PM, Nate Coraor n...@bx.psu.edu wrote: On Feb 1, 2012, at 12:16 PM, Peter Cock wrote: We've changed some things on the cluster to prepare for this - e.g. our Galaxy server is now a proper SGE submit host, so I'm not quite ready to move the production server... still a few little things to sort out first, e.g. https://bitbucket.org/galaxy/galaxy-central/issue/713 I suspect that you are going to encounter more problems with the local and cluster mountpoints being different. Is it not possible to mount it at /data/galaxy on the Galaxy server? It is mounted at /data/galaxy on the Galaxy server, but at /mnt/galaxy on the cluster, but I see what you mean. The two machines are administered by different people, but we can probably find a common mount point if trying to do this with symlinks is really not recommended. Symlinks should be fine, as well. I would make the Galaxy server's path real, however, and use symlinks on the cluster. --nate Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] changeset inconsistency
On Jan 23, 2012, at 8:01 AM, Hans-Rudolf Hotz wrote: Hi Nate Yes, upgrading one of our old servers with hg pull -u -r 63bc46cc73b7 and downloading a new clone with hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist, does not result in the same changeset number (ie '6528') Thank you very much Hans Hi Hans, So with the latest dist version where the changeset number is still off, is it still showing 2 heads? Is the older head still 70b8039a2eef? What version of Mercurial is this? How about the output of 'hg branches', 'hg branches -a', and 'hg branches -c'? Thanks, --nate On 01/20/2012 09:04 AM, Hans-Rudolf Hotz wrote: Hi Nate Thank you very much for your e-mail I am still testing and working on the last distribution, hopefully going live this Sunday. But I will look into downloading and upgrading (one of our dev servers) to the new distribution next week. Regards, Hans On 01/19/2012 07:39 PM, Nate Coraor wrote: Hi Hans, We're releasing a distribution today. If this is still the case with that distribution, let me know. There were branches that weren't showing up under newer versions of Mercurial. Hopefully this is all resolved in the latest. --nate On Jan 17, 2012, at 4:11 AM, Hans-Rudolf Hotz wrote: Hi I am in the process of upgrading all our Galaxy servers to the current changeset (b258de1e6cea, Nov. 18) and I have noticed an inconsistency: If I upgrade one of our old servers (which was on 720455407d1c, June 23) with 'hg pull -u -r b258de1e6cea' I get the following: haruhotz@silicon: hg heads changeset: 6297:b258de1e6cea tag: tip user: jeremy goecksjeremy.goe...@emory.edu date: Thu Nov 17 15:08:49 2011 -0500 summary: Pack js scripts. haruhotz@silicon: If I create a new installation with 'hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist' I get the following: haruhotz@silicon: hg heads changeset: 6298:b258de1e6cea tag: tip user: jeremy goecksjeremy.goe...@emory.edu date: Thu Nov 17 15:08:49 2011 -0500 summary: Pack js scripts. changeset: 5832:70b8039a2eef branch: feature/ws parent: 5826:6206a27bd2ae user: Nate Coraorn...@bx.psu.edu date: Wed Jul 27 10:48:03 2011 -0400 summary: Closed the feature/ws branch, changes from this branch were merged in 5827:f3a1086fac91. haruhotz@silicon: I am not worry about the merged branched, I am just a little bit surprised about the differences: 6298:b258de1e6cea versus 6297:b258de1e6cea Can anybody with more mercurial experience enlighten me what went wrong? and even better tell me that there is nothing to worry about... Regards, Hans ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy strips CSS from HTML files
Hi Cory, The new call to sanitize_html was introduced to more effectively prevent malicious content and possible XSS attacks, though I can't think off the top of my head why we couldn't allow style content. I'll see what I can do about relaxing the filter a little. Thanks! -Dannon On 01/30/2012 10:33 PM, Cory Spencer wrote: Hello all - One of the Galaxy tools I've been developing generates HTML output which I'd styled using astyle.../style tag in the HTML header. After updating to the latest Galaxy release earlier today, thehtml,head.../head,style andbody tags started to get stripped from the output, rendering previously CSS styled output rather unstylish. Delving into things, I noticed a change committed in December that sanitizes the output for HTML files via a call to sanitize_html: https://bitbucket.org/galaxy/galaxy-central/changeset/35fee32991ce#chg-lib/galaxy/web/controllers/dataset.py The added lines 381 - 383 in the new file appear to be causing this new behaviour. Is there any option for making this optional? What was the rational behind stripping out these tags on outputted HTML files? Thanks for any help! Cory Spencer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] naming of history steps
Hi Jeremy and Ross I agree that the current chaining mechanism would get very long after even a few steps. Ususally the most important information is the first and last step E.g. The TopHat run should be called TopHat on SOLiD 24A The alignment stats should be SAM/BAM Summary Metrics of Solid 24A With the rest of the tools in the chain identified in the more information box. This would also give graph generating tools a fighting chance to present something useful in any graphs generated. E.g. GC Bias Plot of Solid 24A could have a title of Solid 24A instead of dataset_234.dat What do you think of this first-last model? Brad -- Brad Langhorst New England Biolabs langho...@neb.com From: Jeremy Goecks jeremy.goe...@emory.edumailto:jeremy.goe...@emory.edu Date: Tue, 31 Jan 2012 09:00:38 -0500 To: Brad Langhorst langho...@neb.commailto:langho...@neb.com Cc: galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edumailto:galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] naming of history steps Brad, But I think it might be a lot easier to manage if step names were based on the titles of the history items instead of data 2 or whatever. Has this been tried and rejected for some reason? It's been tried and rejected because dataset names get very long and unwieldy. E.g. Sam/Bam Alignment Summary Metrics on Tophat on Filter FASTQ on my_rna_seq_reads Would a pull request implementing this change be welcomed? What we imagine would help is a way to easily show/find a dataset's analysis path -- its parents and its decendants -- so that it's possible to trace the datasets/tools used to create a dataset and the tools/datasets subsequently used. This is something we'd like to do but haven't put much effort into yet. Community contributions in this space would be great. Best, J. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] naming of history steps
+1 votes :-) This would also give graph generating tools a fighting chance to present something useful in any graphs generated. E.g. GC Bias Plot of Solid 24A could have a title of Solid 24A instead of dataset_234.dat What do you think of this first-last model? ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy strips CSS from HTML files
Hi Dannon and thanks for the response! I can see the need to sanitize incoming HTTP request parameters that may have malicious content. However, I'm unclear as to why this also needs to happen for HTML pages outputted by the Galaxy tools? If they have been generated with sanitized HTTP request parameters, is there still a risk of an XSS attack? If anything, would it be possible to make this sort of sanitization controllable via a configuration file option? Thanks! Cory On 2012-02-01, at 12:01 PM, Dannon Baker wrote: Hi Cory, The new call to sanitize_html was introduced to more effectively prevent malicious content and possible XSS attacks, though I can't think off the top of my head why we couldn't allow style content. I'll see what I can do about relaxing the filter a little. Thanks! -Dannon On 01/30/2012 10:33 PM, Cory Spencer wrote: Hello all - One of the Galaxy tools I've been developing generates HTML output which I'd styled using astyle.../style tag in the HTML header. After updating to the latest Galaxy release earlier today, thehtml,head.../head,style andbody tags started to get stripped from the output, rendering previously CSS styled output rather unstylish. Delving into things, I noticed a change committed in December that sanitizes the output for HTML files via a call to sanitize_html: https://bitbucket.org/galaxy/galaxy-central/changeset/35fee32991ce#chg-lib/galaxy/web/controllers/dataset.py The added lines 381 - 383 in the new file appear to be causing this new behaviour. Is there any option for making this optional? What was the rational behind stripping out these tags on outputted HTML files? Thanks for any help! Cory Spencer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy strips CSS from HTML files
On 02/01/2012 03:41 PM, Cory Spencer wrote:
Hi Dannon and thanks for the response!
I can see the need to sanitize incoming HTTP request parameters that may have
malicious content. However, I'm unclear as to why this also needs to happen
for HTML pages outputted by the Galaxy tools? If they have been generated with
sanitized HTTP request parameters, is there still a risk of an XSS
attack?toolbox
With Galaxy's toolbox at hand you could generate invalid HTML from plain
text components. A simple example, but consider the following:
Upload one plain text file with the content:
script
Another one with:
alert('oh no!');
/
And finally one with:
script
Taken individually they're meaningless, but run the concatenate datasets
tool to reassemble that into:
script
alert('oh no!');
/
script
Change the type of this dataset to html and there's your attack. If you
tried to upload this, we'd interpret it as malicious HTML and discard
it. As separate datasets, it's impossible to tell. Given Galaxy's
powerful text manipulation tools you could write just about whatever you
wanted using Galaxy itself and get it in the system as a (seemingly)
valid tool-generated dataset. Now, with the outbound sanitation on any
dataset served as text/html it doesn't matter and it gets handled
prior to serving.
Another option we discussed would be to trust all tool generated HTML,
disallow changing the datatype of anything *to* html, and so on, but
that approach comes with its own problems.
If anything, would it be possible to make this sort of sanitization
controllable via a configuration file option?
I'm rather hesitant to put in a disable option for a security feature,
though you're more than welcome to pop those two lines out of your
instance. I think the best path forward is probably relaxing the filter
a bit, the initial pass was somewhat draconian. Would relaxing the
filter to allow style content to pass through work for your needs?
-Dannon
Thanks!
Cory
On 2012-02-01, at 12:01 PM, Dannon Baker wrote:
Hi Cory,
The new call to sanitize_html was introduced to more effectively prevent
malicious content and possible XSS attacks, though I can't think off the top of
my head why we couldn't allow style content. I'll see what I can do about
relaxing the filter a little.
Thanks!
-Dannon
On 01/30/2012 10:33 PM, Cory Spencer wrote:
Hello all -
One of the Galaxy tools I've been developing generates HTML output which I'd styled using astyle.../style
tag in the HTML header. After updating to the latest Galaxy release earlier today,
thehtml,head.../head,style andbody tags started to get stripped from the
output, rendering previously CSS styled output rather unstylish.
Delving into things, I noticed a change committed in December that sanitizes the output
for HTML files via a call to sanitize_html:
https://bitbucket.org/galaxy/galaxy-central/changeset/35fee32991ce#chg-lib/galaxy/web/controllers/dataset.py
The added lines 381 - 383 in the new file appear to be causing this new
behaviour.
Is there any option for making this optional? What was the rational behind
stripping out these tags on outputted HTML files?
Thanks for any help!
Cory Spencer
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy Wiki
Hi John, (and every potential wiki contributor, which on the dev list is all of you :-) ** I apologize for missing your question back in December (and for not noticing it since, either). Thanks to Jen for refreshing the thread. 1. Thanks to your email, I have added your username and *107 others *to the Vetted Editors List (http://wiki.g2.bx.psu.edu/VettedEditorGroup). 2. *You (and 107 new friends!) will no longer have to answer textchas/captchas to update the wiki.* 3. Now that we have 140 vetted users on the wiki, I hope to see a flood of helpful updates there. :-) 4. For anyone who does not have an account on the wiki, and wants to update content, please create an account, send the username to me, and I will add you to vetted list. Again, I apologize for the delay. And, I would like to thank you for making me look at this. I had no idea we had that many logins on the wiki. Please let me know if you have any questions, Dave C. *Grinding Details, for those who care: * The Captcha questions are indeed darn irritating. We investigated all options currently available in MoinMoin, and unfortunately, they are the best choice. They also work very well at keeping out spam. They also apparently work very well at keeping out contributions. Some numbers: 198 Registered users on the wiki 32 of them were on the Vetted Editors List at the beginning of today 108 of them were added to the Vetted Editors List today, as a result of John's question. 58 of them are either blocked, known spammers, or I could not verify who they were. In the past, I have only added usernames to the Vetted Editor List, once the user had successfully made an edit. I am wondering if many of those 108 newly vetted users tried to update a page and were defeated by the captchas? In the future, I'll check the new user list, and update the vetted list once a month. On Tue, Jan 31, 2012 at 2:56 PM, Jennifer Jackson j...@bx.psu.edu wrote: Hi John, I am going to put you in touch with our wiki guru, Dave. Hopefully you two will be able to work out some sort of solution. I believe there is a super secret list he can add you to that will made the question burden much lighter (i.e. none). But with all sincerity, we really do appreciate the help, and look forward to your contributions, Best, Jen Galaxy team On 12/20/11 11:04 AM, John Major wrote: Galaxy team- I've tried a few times to add some content to the FAQ, but the captcha questions keep failing, or i'm submitting the wrong answer. I'd suggest a slightly less burdensome barrier to contributing- i'm certain this is dialing down your user contributions significantly. John __**_ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Jackson http://usegalaxy.org http://galaxyproject.org/wiki/**Supporthttp://galaxyproject.org/wiki/Support -- http://galaxyproject.org/ http://getgalaxy.org/ http://usegalaxy.org/ http://galaxyproject.org/wiki/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy strips CSS from HTML files
On 2012-02-01, at 1:33 PM, Dannon Baker wrote: With Galaxy's toolbox at hand you could generate invalid HTML from plain text components. A simple example, but consider the following: Upload one plain text file with the content: script Change the type of this dataset to html and there's your attack. If you tried to upload this, we'd interpret it as malicious HTML and discard it. As separate datasets, it's impossible to tell. Given Galaxy's powerful text manipulation tools you could write just about whatever you wanted using Galaxy itself and get it in the system as a (seemingly) valid tool-generated dataset. Now, with the outbound sanitation on any dataset served as text/html it doesn't matter and it gets handled prior to serving. Okay, I follow you there. That's a good example, thank you! Another option we discussed would be to trust all tool generated HTML, disallow changing the datatype of anything *to* html, and so on, but that approach comes with its own problems. In the case of the tool we're working on, this option is probably what would have worked best. If anything, would it be possible to make this sort of sanitization controllable via a configuration file option? I'm rather hesitant to put in a disable option for a security feature, though you're more than welcome to pop those two lines out of your instance. I think the best path forward is probably relaxing the filter a bit, the initial pass was somewhat draconian. Would relaxing the filter to allow style content to pass through work for your needs? Yes, we've already commented it out for the time being. :) Relaxing the filter would be a good improvement so far as we're concerned. I'd be happy to keep in contact with you during the process so that we can find the happy middle ground between security and usability. Thanks again! Cory ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Issue 710 in galaxy-central: An error occurred running this job- No alignments are available for sacCer3
Hi, when I try to fetch sequence for SacCer3 using UCSC Main table Browser, I received the following error: An error occurred running this job: No sequences are available for 'sacCer3', request them by reporting this error. I noticed that, It's possible to extract sequence for SacCer2, but again none of the SacCer2 or SacCer3 has allowed Fetch Alignments-Extract MAF Blocks Is it a bug or compatibility issue? Hope this can be fixed or else will be a problem for me. With Regards, Ambarish Biswas, University of Otago Department of Biochemistry, Dunedin, New Zealand, Tel: +64(22)0855647 Fax: +64(0)3 479 7866 On Thu, Feb 2, 2012 at 3:01 PM, Bitbucket issues-re...@bitbucket.orgwrote: --- you can reply above this line --- Issue 710: An error occurred running this job- No sequences are available for sacCer3 https://bitbucket.org/galaxy/galaxy-central/issue/710/an-error-occurred-running-this-job-no ambarishbiswas on Thu, 2 Feb 2012 03:01:34 +0100: Changes: component: nothing - tools -- This is an issue notification from bitbucket.org. You are receiving this either because you are the owner of the issue, or you are following the issue. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Running Galaxy on Rocks Cluster
Hi guys, Is there anyone has actually setup galaxy on a Rocks based cluster? I know that galaxy should be hosted on a submission host so that it can submit jobs to the cluster. Rocks provides preconfigured DRM such as SGE, so I start thinking if galaxy host should be deployed and managed by Rocks (e.g. as a Login Appliance in Rocks' term). Sorry if this is not a pure galaxy issue, but I guess people in here will be more appropriate to answer this. Regards, Derrick ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Running Galaxy on Rocks Cluster
I'm using Galaxy on a ROCKS cluster. I installed Galaxy on the headnode. I'm using PBS instead of SGE though. It shouldn't make a difference. I haven't thought about deploying Galaxy as a Roll if that's what you are referring to. Only because Galaxy doesn't need to run on all the nodes, it only needs a web server. You could make a roll and deploy it on the head node only, but that just seems to be more work than necessary. Its not be a bad idea if the roll is pre-configured and includes all the production tweaks...would have saved me 2-3 weeks of work, potentially. But now that its installed, I'm not going to go back and build a roll out of it. On Wed, Feb 1, 2012 at 10:13 PM, Derrick LIN klin...@gmail.com wrote: Hi guys, Is there anyone has actually setup galaxy on a Rocks based cluster? I know that galaxy should be hosted on a submission host so that it can submit jobs to the cluster. Rocks provides preconfigured DRM such as SGE, so I start thinking if galaxy host should be deployed and managed by Rocks (e.g. as a Login Appliance in Rocks' term). Sorry if this is not a pure galaxy issue, but I guess people in here will be more appropriate to answer this. Regards, Derrick ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] changeset inconsistency
Hi Nate Sorry, this was a typo, I meant to write: does noW result in the same changeset number (ie '6528') - so everything is fine now. I apologize for the confusion Hans On 02/01/2012 07:13 PM, Nate Coraor wrote: On Jan 23, 2012, at 8:01 AM, Hans-Rudolf Hotz wrote: Hi Nate Yes, upgrading one of our old servers with hg pull -u -r 63bc46cc73b7 and downloading a new clone with hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist, does not result in the same changeset number (ie '6528') Thank you very much Hans Hi Hans, So with the latest dist version where the changeset number is still off, is it still showing 2 heads? Is the older head still 70b8039a2eef? What version of Mercurial is this? How about the output of 'hg branches', 'hg branches -a', and 'hg branches -c'? Thanks, --nate On 01/20/2012 09:04 AM, Hans-Rudolf Hotz wrote: Hi Nate Thank you very much for your e-mail I am still testing and working on the last distribution, hopefully going live this Sunday. But I will look into downloading and upgrading (one of our dev servers) to the new distribution next week. Regards, Hans On 01/19/2012 07:39 PM, Nate Coraor wrote: Hi Hans, We're releasing a distribution today. If this is still the case with that distribution, let me know. There were branches that weren't showing up under newer versions of Mercurial. Hopefully this is all resolved in the latest. --nate On Jan 17, 2012, at 4:11 AM, Hans-Rudolf Hotz wrote: Hi I am in the process of upgrading all our Galaxy servers to the current changeset (b258de1e6cea, Nov. 18) and I have noticed an inconsistency: If I upgrade one of our old servers (which was on 720455407d1c, June 23) with 'hg pull -u -r b258de1e6cea' I get the following: haruhotz@silicon: hg heads changeset: 6297:b258de1e6cea tag: tip user: jeremy goecksjeremy.goe...@emory.edu date: Thu Nov 17 15:08:49 2011 -0500 summary: Pack js scripts. haruhotz@silicon: If I create a new installation with 'hg clone http://www.bx.psu.edu/hg/galaxy galaxy-dist' I get the following: haruhotz@silicon: hg heads changeset: 6298:b258de1e6cea tag: tip user: jeremy goecksjeremy.goe...@emory.edu date: Thu Nov 17 15:08:49 2011 -0500 summary: Pack js scripts. changeset: 5832:70b8039a2eef branch: feature/ws parent: 5826:6206a27bd2ae user: Nate Coraorn...@bx.psu.edu date: Wed Jul 27 10:48:03 2011 -0400 summary: Closed the feature/ws branch, changes from this branch were merged in 5827:f3a1086fac91. haruhotz@silicon: I am not worry about the merged branched, I am just a little bit surprised about the differences: 6298:b258de1e6cea versus 6297:b258de1e6cea Can anybody with more mercurial experience enlighten me what went wrong? and even better tell me that there is nothing to worry about... Regards, Hans ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
