[galaxy-dev] Local Galaxy Apache Config
Hello Galaxy Admins,
I am attempting to setup a local production-grade Galaxy server for
our department. Before I go ahead and turn on Galaxy for the first
time, would you be so kind as to please review my Apache2 config files
for errors? I want to be sure that things are in the correct place to
save me from headaches down the road.
The overall goals of the configuration are:
- always use HTTPS and secure FTP
- make maximum use of the machine (multiple Apache processes,
PostgreSQL, ProFTPD, etc.)
If it helps, Galaxy will be running on a:
- Dell Precision T7500
- 2 x Intel Xeon X5690 @ 3.46 GHz
- 96 GB RAM
- Ubuntu 14.04 64-bit
Thank you for your help!
FYI: My config files are based on the instructions at
https://wiki.galaxyproject.org/Admin/GetGalaxy and
https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer
##
#
# /etc/apache2/apache2.conf
#
##
# Global configuration
#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE! If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the Mutex documentation (available
# at URL:http://httpd.apache.org/docs/2.4/mod/core.html#mutex);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
#ServerRoot /etc/apache2
#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
Mutex file:${APACHE_LOCK_DIR} default
#
# PidFile: The file in which the server should record its process
# identification number when it starts.
# This needs to be set in /etc/apache2/envvars
#
PidFile ${APACHE_PID_FILE}
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300
#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to Off to deactivate.
#
KeepAlive On
#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100
#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 5
# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}
#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a VirtualHost
# container, error messages relating to that virtual host will be
# logged here. If you *do* define an error logfile for a VirtualHost
# container, that host's errors will be logged there and not here.
#
ErrorLog ${APACHE_LOG_DIR}/error.log
#
# LogLevel: Control the severity of messages logged to the error_log.
# Available values: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the log level for particular modules, e.g.
# LogLevel info ssl:warn
#
LogLevel warn
# Include module configuration:
IncludeOptional mods-enabled/*.load
IncludeOptional mods-enabled/*.conf
# Include list of ports to listen on
Include ports.conf
# Sets the default security model of the Apache2 HTTPD server. It does
# not allow access to the root filesystem outside of /usr/share and /var/www.
# The former is used by web applications packaged in Debian,
# the latter may be used for local directories served by the web server. If
# your system is serving content from a sub-directory in /srv you must allow
# access here, or in any related virtual host.
Directory /
Options FollowSymLinks
AllowOverride None
Require all denied
/Directory
Directory /usr/share
AllowOverride None
Require all granted
/Directory
Directory /var/www/
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
/Directory
#Directory /srv/
#Options Indexes FollowSymLinks
#AllowOverride None
#Require all granted
#/Directory
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives. See also the AllowOverride
# directive.
#
AccessFileName .htaccess
#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
FilesMatch ^\.ht
Require all denied
/FilesMatch
#
# The following directives define some format nicknames for use with
# a CustomLog directive.
#
# These deviate from the Common Log Format definitions in that they use %O
# (the actual bytes sent including headers) instead of %b
Re: [galaxy-dev] Local Galaxy web layout
Hi Shenwiyn, Can you specify what is not working? Did you find the file? Thanks, Sam On Wed, Feb 19, 2014 at 1:42 AM, shenw...@gmail.com shenw...@gmail.comwrote: Hi Sam and Hans-Rudolf， Thank you very much for your help.It seems that the static/scripts/galaxy.menu.js fail to work in my local Galaxy.As a result,would you please show me some more details about this? Regards, Shenwiyn *From:* sam guerler aysam.guer...@gmail.com *Date:* 2014-02-18 01:20 *To:* Hans-Rudolf Hotz h...@fmi.ch *CC:* shenw...@gmail.com; galaxy-dev galaxy-dev@lists.bx.psu.edu *Subject:* Re: [galaxy-dev] Local Galaxy web layout Hi Shenwiyn, This has changed recently. Please take a look at: static/scripts/galaxy.menu.js Using that file, you can add/remove/change labels and their position. It is recommended to run pack_scripts.py after changing the source file. Regards, Sam On Mon, Feb 17, 2014 at 11:33 AM, Hans-Rudolf Hotz h...@fmi.ch wrote: Hi Shenwiyn Have a look at: ~/galaxy_dist/database/compiled_templates/base_panels.mako.py Regards, Hans-Rudolf On 02/17/2014 08:04 AM, shenw...@gmail.com wrote: Galaxy developers: I want to change my local Galaxy web layout,for example,delete the top button of Visualization and Cloud or change their positions.Could anyone show me some details about this? Thank you very much. shenw...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Local Galaxy web layout
Galaxy developers:I want to change my local Galaxy web layout,for example,delete the top button of Visualization and Cloud or change their positions.Could anyone show me some details about this?Thank you very much. shenw...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local Galaxy web layout
Hi Shenwiyn Have a look at: ~/galaxy_dist/database/compiled_templates/base_panels.mako.py Regards, Hans-Rudolf On 02/17/2014 08:04 AM, shenw...@gmail.com wrote: Galaxy developers: I want to change my local Galaxy web layout,for example,delete the top button of Visualization and Cloud or change their positions.Could anyone show me some details about this? Thank you very much. shenw...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local Galaxy web layout
Hi Shenwiyn, This has changed recently. Please take a look at: static/scripts/galaxy.menu.js Using that file, you can add/remove/change labels and their position. It is recommended to run pack_scripts.py after changing the source file. Regards, Sam On Mon, Feb 17, 2014 at 11:33 AM, Hans-Rudolf Hotz h...@fmi.ch wrote: Hi Shenwiyn Have a look at: ~/galaxy_dist/database/compiled_templates/base_panels.mako.py Regards, Hans-Rudolf On 02/17/2014 08:04 AM, shenw...@gmail.com wrote: Galaxy developers: I want to change my local Galaxy web layout,for example,delete the top button of Visualization and Cloud or change their positions.Could anyone show me some details about this? Thank you very much. shenw...@gmail.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Local Galaxy Install - Debug Report FROM address as Admin and not User?
Howdy, On our local Galaxy install, everytime a user submits a bug report, it does email all the admins -- as defined in universe_wsgi.ini -- but the FROM address is not of the user, but rather the first email address from the admin list. Is this normal? May we have it changed so the FROM address is that of the user? Thanks, -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local Galaxy Install - Debug Report FROM address as Admin and not User?
Hi Adam, you are right about the behaviour, this is what Galaxy does. However it sends the email to the user too, so Reply all will reach the user's email address. This is how the feature was meant to be used. Martin On Tue, Oct 1, 2013 at 12:59 PM, Adam Brenner aebre...@uci.edu wrote: Howdy, On our local Galaxy install, everytime a user submits a bug report, it does email all the admins -- as defined in universe_wsgi.ini -- but the FROM address is not of the user, but rather the first email address from the admin list. Is this normal? May we have it changed so the FROM address is that of the user? Thanks, -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local Galaxy Install - Debug Report FROM address as Admin and not User?
Martin, Thanks for the response. Yes, I do notice the reply-to address does include the user, but wouldn't it be better if the FROM address was from the actual user reporting the error? When checking my email, for example, I do not want to see emails that are FROM me, to MYSELF when its really from our users. Do you know which of the python files contain the code that relates to this? Or point me in the right direction so I can change this? Thanks! -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu On Tue, Oct 1, 2013 at 10:52 AM, Martin Čech mar...@bx.psu.edu wrote: Hi Adam, you are right about the behaviour, this is what Galaxy does. However it sends the email to the user too, so Reply all will reach the user's email address. This is how the feature was meant to be used. Martin On Tue, Oct 1, 2013 at 12:59 PM, Adam Brenner aebre...@uci.edu wrote: Howdy, On our local Galaxy install, everytime a user submits a bug report, it does email all the admins -- as defined in universe_wsgi.ini -- but the FROM address is not of the user, but rather the first email address from the admin list. Is this normal? May we have it changed so the FROM address is that of the user? Thanks, -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local Galaxy Install - Debug Report FROM address as Admin and not User?
Adam, it is function *report_error* in /lib/galaxy/webapps/galaxy/controllers/dataset.py you need to set this line * frm = to_address * to * frm = email* and you should be all set. Martin On Tue, Oct 1, 2013 at 2:12 PM, Adam Brenner aebre...@uci.edu wrote: Martin, Thanks for the response. Yes, I do notice the reply-to address does include the user, but wouldn't it be better if the FROM address was from the actual user reporting the error? When checking my email, for example, I do not want to see emails that are FROM me, to MYSELF when its really from our users. Do you know which of the python files contain the code that relates to this? Or point me in the right direction so I can change this? Thanks! -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu On Tue, Oct 1, 2013 at 10:52 AM, Martin Čech mar...@bx.psu.edu wrote: Hi Adam, you are right about the behaviour, this is what Galaxy does. However it sends the email to the user too, so Reply all will reach the user's email address. This is how the feature was meant to be used. Martin On Tue, Oct 1, 2013 at 12:59 PM, Adam Brenner aebre...@uci.edu wrote: Howdy, On our local Galaxy install, everytime a user submits a bug report, it does email all the admins -- as defined in universe_wsgi.ini -- but the FROM address is not of the user, but rather the first email address from the admin list. Is this normal? May we have it changed so the FROM address is that of the user? Thanks, -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local Galaxy Install - Multiple Input Files for Workflow Missing?
Are you using Input Dataset steps in your workflows? The multiple inputs feature uses these to know how to distribute inputs -- other than that no other configuration steps are necessary. On Tue, Sep 24, 2013 at 2:43 PM, Adam Brenner aebre...@uci.edu wrote: Howdy, On our local galaxy cluster, I have gotten a report that the multiple input files selection on workflows is missing. How do I enable it? Looking at the past mailing list[1] this is the item that is missing from our setup -- the tooltip icon. We are running a version of galaxy that is roughly ~one month old: c42567f43aa7. [1]: http://dev.list.galaxyproject.org/Looking-for-recommendations-How-to-run-galaxy-workflows-in-batch-td4362836.html#a4362874 Thanks! -Adam -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local Galaxy Install - Multiple Job Handlers - Uploads waiting to run endlessly
So I finally figured it out (thanks to the help from the IRC folks, bag and natefoo). I incorrectly had my job_conf.xml setup where no job runners were set. All is working now. -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu On Sun, Apr 14, 2013 at 1:54 PM, Adam Brenner aebre...@uci.edu wrote: Anyone have any ideas? Still have not sorted this out -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu On Thu, Apr 11, 2013 at 2:24 PM, Adam Brenner aebre...@uci.edu wrote: FYI: Here are the entries in paster.log. If you notice no job_handlers are being sent. I performed tests on both nginx reverse proxy and direct localhost:8080 (limited to our private network on campus) 128.200.34.227 - - [11/Apr/2013:13:24:40 -0700] GET /tool_runner?tool_id=upload1 HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/root/tool_menu; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 galaxy.tools.actions.upload_common INFO 2013-04-11 13:24:44,173 tool upload1 created job id 8 128.200.34.227 - - [11/Apr/2013:13:24:43 -0700] POST /tool_runner/index HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/tool_runner?tool_id=upload1; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 128.200.34.227 - - [11/Apr/2013:13:24:44 -0700] GET /history HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/tool_runner/index; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 128.200.34.227 - - [11/Apr/2013:13:24:44 -0700] GET /api/histories/56959efe847bb360 HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/history; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu On Thu, Apr 11, 2013 at 12:39 PM, Adam Brenner aebre...@uci.edu wrote: Howdy Galaxy Admins, I am installing Galaxy on our HPC cluster and running into a strange issue that I can no longer debug further. I am running the latest galaxy build as of Tuesday April 9th. hg clone https://bitbucket.org/galaxy/galaxy-dist/ hg update stable When a user tries to upload a file via the web interface, nginx reverse proxy setup as described here http://wiki.galaxyproject.org/Admin/Config/Performance/nginx%20Proxy, the upload stays in a state of Job is waiting to run. The job stays like this for both FTP uploads, and uploads via browser. The dataset is a 319 bytes (both FTP and upload via browser) My universe_wsgi.ini variables are like so: .. under [app:main] new_file_path = /som/galaxy/tmp file_path = /som/galaxy/datasets/files track_jobs_in_database = True job_manager = manager job_handlers = handler0,handler1 ... before [app:main] in [server:main] [server:main] use = egg:Paste#http port = 8080 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:manager] use = egg:Paste#http port = 8079 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler0] use = egg:Paste#http port = 8090 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler1] use = egg:Paste#http port = 8091 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 Galaxy, for now, is running under the root user and is installed in /data/apps/galaxy (NFS mount) and the tmp files are stored on a GlusterFS system under /som/galaxy/. I have tried removing the extra job managers and handlers so I just have server:main and this still continues to happen. The paster.log files (with DEBUG = True) does not show any errors from python or issues writing to file. All POST and GET responses are recorded and nothing else. What I find strange is that if I look in new_file_path I do see my dataset as upload_file_data_Zx_1rK and cat'ing the file shows the correct contents. The items listed in my file_path (the dataset_*.dat) files are empty/blank..but are created. Any ideas as to why its failing? Is it nginx? I only have it
Re: [galaxy-dev] Local Galaxy Install - Multiple Job Handlers - Uploads waiting to run endlessly
Anyone have any ideas? Still have not sorted this out -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu On Thu, Apr 11, 2013 at 2:24 PM, Adam Brenner aebre...@uci.edu wrote: FYI: Here are the entries in paster.log. If you notice no job_handlers are being sent. I performed tests on both nginx reverse proxy and direct localhost:8080 (limited to our private network on campus) 128.200.34.227 - - [11/Apr/2013:13:24:40 -0700] GET /tool_runner?tool_id=upload1 HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/root/tool_menu; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 galaxy.tools.actions.upload_common INFO 2013-04-11 13:24:44,173 tool upload1 created job id 8 128.200.34.227 - - [11/Apr/2013:13:24:43 -0700] POST /tool_runner/index HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/tool_runner?tool_id=upload1; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 128.200.34.227 - - [11/Apr/2013:13:24:44 -0700] GET /history HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/tool_runner/index; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 128.200.34.227 - - [11/Apr/2013:13:24:44 -0700] GET /api/histories/56959efe847bb360 HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/history; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu On Thu, Apr 11, 2013 at 12:39 PM, Adam Brenner aebre...@uci.edu wrote: Howdy Galaxy Admins, I am installing Galaxy on our HPC cluster and running into a strange issue that I can no longer debug further. I am running the latest galaxy build as of Tuesday April 9th. hg clone https://bitbucket.org/galaxy/galaxy-dist/ hg update stable When a user tries to upload a file via the web interface, nginx reverse proxy setup as described here http://wiki.galaxyproject.org/Admin/Config/Performance/nginx%20Proxy, the upload stays in a state of Job is waiting to run. The job stays like this for both FTP uploads, and uploads via browser. The dataset is a 319 bytes (both FTP and upload via browser) My universe_wsgi.ini variables are like so: .. under [app:main] new_file_path = /som/galaxy/tmp file_path = /som/galaxy/datasets/files track_jobs_in_database = True job_manager = manager job_handlers = handler0,handler1 ... before [app:main] in [server:main] [server:main] use = egg:Paste#http port = 8080 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:manager] use = egg:Paste#http port = 8079 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler0] use = egg:Paste#http port = 8090 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler1] use = egg:Paste#http port = 8091 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 Galaxy, for now, is running under the root user and is installed in /data/apps/galaxy (NFS mount) and the tmp files are stored on a GlusterFS system under /som/galaxy/. I have tried removing the extra job managers and handlers so I just have server:main and this still continues to happen. The paster.log files (with DEBUG = True) does not show any errors from python or issues writing to file. All POST and GET responses are recorded and nothing else. What I find strange is that if I look in new_file_path I do see my dataset as upload_file_data_Zx_1rK and cat'ing the file shows the correct contents. The items listed in my file_path (the dataset_*.dat) files are empty/blank..but are created. Any ideas as to why its failing? Is it nginx? I only have it redirect 8080, and none of the other job managers / handlers...is that an issue? Do you think its permissions on the my file_path or new_file_path directories? A very stumped Adam. -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___
[galaxy-dev] Local Galaxy Install - Multiple Job Handlers - Uploads waiting to run endlessly
Howdy Galaxy Admins, I am installing Galaxy on our HPC cluster and running into a strange issue that I can no longer debug further. I am running the latest galaxy build as of Tuesday April 9th. hg clone https://bitbucket.org/galaxy/galaxy-dist/ hg update stable When a user tries to upload a file via the web interface, nginx reverse proxy setup as described here http://wiki.galaxyproject.org/Admin/Config/Performance/nginx%20Proxy, the upload stays in a state of Job is waiting to run. The job stays like this for both FTP uploads, and uploads via browser. The dataset is a 319 bytes (both FTP and upload via browser) My universe_wsgi.ini variables are like so: .. under [app:main] new_file_path = /som/galaxy/tmp file_path = /som/galaxy/datasets/files track_jobs_in_database = True job_manager = manager job_handlers = handler0,handler1 ... before [app:main] in [server:main] [server:main] use = egg:Paste#http port = 8080 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:manager] use = egg:Paste#http port = 8079 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler0] use = egg:Paste#http port = 8090 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler1] use = egg:Paste#http port = 8091 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 Galaxy, for now, is running under the root user and is installed in /data/apps/galaxy (NFS mount) and the tmp files are stored on a GlusterFS system under /som/galaxy/. I have tried removing the extra job managers and handlers so I just have server:main and this still continues to happen. The paster.log files (with DEBUG = True) does not show any errors from python or issues writing to file. All POST and GET responses are recorded and nothing else. What I find strange is that if I look in new_file_path I do see my dataset as upload_file_data_Zx_1rK and cat'ing the file shows the correct contents. The items listed in my file_path (the dataset_*.dat) files are empty/blank..but are created. Any ideas as to why its failing? Is it nginx? I only have it redirect 8080, and none of the other job managers / handlers...is that an issue? Do you think its permissions on the my file_path or new_file_path directories? A very stumped Adam. -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Local Galaxy Install - Multiple Job Handlers - Uploads waiting to run endlessly
FYI: Here are the entries in paster.log. If you notice no job_handlers are being sent. I performed tests on both nginx reverse proxy and direct localhost:8080 (limited to our private network on campus) 128.200.34.227 - - [11/Apr/2013:13:24:40 -0700] GET /tool_runner?tool_id=upload1 HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/root/tool_menu; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 galaxy.tools.actions.upload_common INFO 2013-04-11 13:24:44,173 tool upload1 created job id 8 128.200.34.227 - - [11/Apr/2013:13:24:43 -0700] POST /tool_runner/index HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/tool_runner?tool_id=upload1; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 128.200.34.227 - - [11/Apr/2013:13:24:44 -0700] GET /history HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/tool_runner/index; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 128.200.34.227 - - [11/Apr/2013:13:24:44 -0700] GET /api/histories/56959efe847bb360 HTTP/1.0 200 - http://ghtf-hpc.oit.uci.edu/history; Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.28.10 (KHTML, like Gecko) Version/6.0.3 Safari/536.28.10 -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu On Thu, Apr 11, 2013 at 12:39 PM, Adam Brenner aebre...@uci.edu wrote: Howdy Galaxy Admins, I am installing Galaxy on our HPC cluster and running into a strange issue that I can no longer debug further. I am running the latest galaxy build as of Tuesday April 9th. hg clone https://bitbucket.org/galaxy/galaxy-dist/ hg update stable When a user tries to upload a file via the web interface, nginx reverse proxy setup as described here http://wiki.galaxyproject.org/Admin/Config/Performance/nginx%20Proxy, the upload stays in a state of Job is waiting to run. The job stays like this for both FTP uploads, and uploads via browser. The dataset is a 319 bytes (both FTP and upload via browser) My universe_wsgi.ini variables are like so: .. under [app:main] new_file_path = /som/galaxy/tmp file_path = /som/galaxy/datasets/files track_jobs_in_database = True job_manager = manager job_handlers = handler0,handler1 ... before [app:main] in [server:main] [server:main] use = egg:Paste#http port = 8080 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:manager] use = egg:Paste#http port = 8079 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler0] use = egg:Paste#http port = 8090 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 [server:handler1] use = egg:Paste#http port = 8091 host = 127.0.0.1 use_threadpool = true threadpool_workers = 5 Galaxy, for now, is running under the root user and is installed in /data/apps/galaxy (NFS mount) and the tmp files are stored on a GlusterFS system under /som/galaxy/. I have tried removing the extra job managers and handlers so I just have server:main and this still continues to happen. The paster.log files (with DEBUG = True) does not show any errors from python or issues writing to file. All POST and GET responses are recorded and nothing else. What I find strange is that if I look in new_file_path I do see my dataset as upload_file_data_Zx_1rK and cat'ing the file shows the correct contents. The items listed in my file_path (the dataset_*.dat) files are empty/blank..but are created. Any ideas as to why its failing? Is it nginx? I only have it redirect 8080, and none of the other job managers / handlers...is that an issue? Do you think its permissions on the my file_path or new_file_path directories? A very stumped Adam. -- Adam Brenner Computer Science, Undergraduate Student Donald Bren School of Information and Computer Sciences Research Computing Support Office of Information Technology http://www.oit.uci.edu/rcs/ University of California, Irvine www.ics.uci.edu/~aebrenne/ aebre...@uci.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] local galaxy tool shed install
Having followed the directions in HostingALocalToolShed I ran run_community.sh with the default configurations in community_wsgi.ini. It executed without error but failed to start the tool shed as http://127.0.0.1:9009 returns Unable to Connect error Thanks for your help Mark This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] local galaxy tool shed install
Hi Mark, What version of Galaxy are you running, and what does your tool shed paster log show? Greg Von Kuster On Jan 15, 2013, at 5:49 PM, mark.r...@syngenta.com wrote: Having followed the directions in “HostingALocalToolShed” I ran run_community.sh with the default configurations in community_wsgi.ini. It executed without error but failed to start the tool shed as http://127.0.0.1:9009 returns “Unable to Connect” error Thanks for your help Mark This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited.___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Local galaxy
Repost to mailing list ___ For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev Hi I am doing NGS analysis in our local galaxy, I would like to access the output files for further analysis. I do have an account on the server where galaxy is installed, so I would like to access the output files without actually downloading them( aince they are huge). Can you please tell me where I can find my (local galaxy) output files on the server ? Is it stored under databases ? Thanks. Selvi Ramalingam Sent from my iPhone -- Jennifer Jackson http://galaxyproject.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local galaxy
Hi I am doing NGS analysis in our local galaxy, I would like to access the output files for further analysis. I do have an account on the server where galaxy is installed, so I would like to access the output files without actually downloading them( aince they are huge). Can you please tell me where I can find my (local galaxy) output files on the server ? Is it stored under databases ? Take a look at the following settings in your universe_wsgi.ini file: file_path - where datasets are stored expose_dataset_path - allows admins/users to see a dataset's path in the dataset information panel Best, J. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local Galaxy concept system: hardware spec
Some quick answers in the hopes that more qualified people will chip in: I have a couple of question around the topic hardware requirements for a server which is intended to be bought and used as concept machine for NGS-related jobs. First a comment - it sounds a bit like you are where we were 12 months ago in developing our Galaxy system and looking at similar needs. I think you'll almost always need more of everything, because people will always be analysing bigger datasets, building bigger assemblies, etc. 1. Using the described bioinformatics software: where are the potential system bottlenecks? (connections between CPUs, RAM, HDDs) While I/O is potentially a bottleneck (due to Galaxy copying and writing the datasets etc.), I wonder if in practice this is the case. Many of the NGS tasks are so long running that I/O issues may not be a significant hit. However, you may have a potential bottleneck in getting data onto the system. How does information get from the sequencer into the Galaxy instance? This may need some thinking about. 2. What is the expected relation of integer-based and floating point based calculations, which will be loading the CPU cores? I have no idea what this means. 3. Regarding the architectural differences (strengths, weaknesses): Would an AMD- or an Intel-System be more suitable? I don't think this will make any difference. If it's a question of the number of cores, that depends to some extent on how many concurrent users or tasks you'll have running. I suspect your number of concurrent users will be low (i.e. at any time, very few people and logged in and running stuff under Galaxy). 6. HDD access (R/W) is mainly in bigger blocks instead of masses of short operations - correct? That's my impression. -- Paul Agapow (paul-michael.aga...@hpa.org.uk) Bioinformatics, Health Protection Agency (UK) ** The information contained in the EMail and any attachments is confidential and intended solely and for the attention and use of the named addressee(s). It may not be disclosed to any other person without the express authority of the HPA, or the intended recipient, or both. If you are not the intended recipient, you must not disclose, copy, distribute or retain this message or any part of it. This footnote also confirms that this EMail has been swept for computer viruses by Symantec.Cloud, but please re-sweep any attachments before opening or saving. HTTP://www.HPA.org.uk ** ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local Galaxy concept system: hardware spec questions
Hey Scott, First of all thanks for the long reply - to keep it short I'll follow you with answering inline: Scott McManus wrote: Hey Sebastian- It may help to consider other pieces aside from compute nodes that you will need, such as nodes for proxies and databases, networking gear (such as switches and cables), and so on. http://usegalaxy.org/production has some details, and there are high-level pieces explained at http://wiki.g2.bx.psu.edu/Events/GDC2010?action=AttachFiledo=gettarget=GDC2010_building_scalable.pdf Thanks, I read through it, that is some evidence. You should also talk to your institution's IT folks about power requirements, how those costs passed on, off-site backup storage (though it sounds like you're counting on RAID 5/6), etc. One non-technical note regarding the organization (techies: skip that): This is right the point we're on currently - we had a first non-technical conversation ~1 month ago, and in the last days suddenly funding was released and led to zugzwang (as far as I read it also describes in English the force to (re)act). The structure is roughly as follows: there is the IT provider for the complete hospital campus (consisting of several clinics and some medical school institutions; we belong to the latter) and our own institute's IT, which serves internally science and research. We had hours of chats inside our institute and agreed that we are neither able nor willed to manage everything on our own (the system is intended for everyone doing NGS research at the campus). This main IT was not integrated in the announced non-technical conversation. Regarding the technical environment everything is on the way, today we'll have another meeting (the main IT folks are bothered by our targeted custom hardware). Backup is also part of conversations in September, we don't want to count on RAID6 alone. This topic is additionally very politics-driven (who pays for what?)... Technically the need is no question. It also may help if folks could share their experiences with benchmarking their own systems along with the tools that they've been using. The Galaxy Czars conference call could help - you could bring this up at the next meeting. Fortunately I joined the Czars group from the first meeting and also took part at the GCC2012 breakout session. You're absolutely right. Too bad that is of so short time until we have to act - that's the reason why I included the whole list, hoping that anyone did some benchmarking. We planed to, but our first server behaved quite moody... Sharing some experiences or hard fact values including system specs would be great for other people who are at the point to order hardware and are forced to state some reasons. I've answered inline, but in general I think that the bottleneck for your planned architecture will be I/O with respect to disk. The next bottleneck may be with respect to the network - if you have a disk farm with a 1 Gbps (125 MBps) connection, then it doesn't matter if your disks can write 400+ MBps. (Nate also included this in his presentation.) You may want to consider Infiniband over Ethernet - I think the Galaxy Czars call would be really helpful in this respect. Planned for the HDD connection is a RAID controller offering 1 GB/s - the array on our first server btw delivered 450 MB/s (measured). Network should not be the problem for the concept, it is intended to be relatively autarkic. Network load will only appear while loading data from an archive or the sequencer itself. A 10 Gbit/s connection is available. InfiniBand was considered for a short time but would exceed the current funding. A cluster is available, but the connection speed is quite low (due to usage more for statistical analyses). 1. Using the described bioinformatics software: where are the potential system bottlenecks? (connections between CPUs, RAM, HDDs) One way to get a better idea is to start with existing resources, create a sample workflow or two, and measure performance. Again, the Galaxy czars call could be a good bet. This is what we wanted to do (see above), but we did not get so far due to the announced technical issues (RAID controller, HDD crashes etc.) 2. What is the expected relation of integer-based and floating point based calculations, which will be loading the CPU cores? This also depends on the tools being used. This might be more relevant if your architecture were to use more specialized hardware (such as GPUs or FPGAs), but this should be a secondary concern. From plain theory I would expect the Needleman-Wunsch algorithm to be of high relevance, which should be integer calculation, basically. In the case of pairwise sequence alignment. MSAs may be different (may require floating point calcs). Unfortunately, GPU and/or FPGA usage are currently far out of range of this first concept, but in the back of my mind for a longer time :). In the standard CPU setting/environment I would suppose
Re: [galaxy-dev] Local Galaxy concept system: hardware spec questions
On Mon, Aug 13, 2012 at 11:23 AM, Sebastian Schaaf sch...@ibe.med.uni-muenchen.de wrote: Hi all, I have a couple of question around the topic hardware requirements for a server which is intended to be bought and used as concept machine for NGS-related jobs. It should be used for development of tools and workflows (using Galaxy, sure) as well as platform for some alpha users, who should learn to work on NGS data, which they just began to generate. ... Duplicate thread on the SEQanswers forum: http://seqanswers.com/forums/showthread.php?t=22456 Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local Galaxy concept system: hardware spec questions
Yes, thanks, I should have mentioned that. I posted in both forum and dev-list, because I don't expect the forum members and the dev-list subscribers to be a 100% identical... Sorry for any inconvenience... Peter Cock wrote: On Mon, Aug 13, 2012 at 11:23 AM, Sebastian Schaaf sch...@ibe.med.uni-muenchen.de wrote: Hi all, I have a couple of question around the topic hardware requirements for a server which is intended to be bought and used as concept machine for NGS-related jobs. It should be used for development of tools and workflows (using Galaxy, sure) as well as platform for some alpha users, who should learn to work on NGS data, which they just began to generate. ... Duplicate thread on the SEQanswers forum: http://seqanswers.com/forums/showthread.php?t=22456 Peter -- Sebastian Schaaf, M.Sc. Bioinformatics Chair of Biometry and Bioinformatics Department of Medical Information Sciences, Biometry and Epidemiology University of Munich Marchioninistr. 15, K U1 (postal) Marchioninistr. 17, U 006 (office) D-81377 Munich (Germany) Tel: +49 89 2180-78178 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local Galaxy concept system: hardware spec questions
Hey Sebastian- It may help to consider other pieces aside from compute nodes that you will need, such as nodes for proxies and databases, networking gear (such as switches and cables), and so on. http://usegalaxy.org/production has some details, and there are high-level pieces explained at http://wiki.g2.bx.psu.edu/Events/GDC2010?action=AttachFiledo=gettarget=GDC2010_building_scalable.pdf You should also talk to your institution's IT folks about power requirements, how those costs passed on, off-site backup storage (though it sounds like you're counting on RAID 5/6), etc. It also may help if folks could share their experiences with benchmarking their own systems along with the tools that they've been using. The Galaxy Czars conference call could help - you could bring this up at the next meeting. I've answered inline, but in general I think that the bottleneck for your planned architecture will be I/O with respect to disk. The next bottleneck may be with respect to the network - if you have a disk farm with a 1 Gbps (125 MBps) connection, then it doesn't matter if your disks can write 400+ MBps. (Nate also included this in his presentation.) You may want to consider Infiniband over Ethernet - I think the Galaxy Czars call would be really helpful in this respect. 1. Using the described bioinformatics software: where are the potential system bottlenecks? (connections between CPUs, RAM, HDDs) One way to get a better idea is to start with existing resources, create a sample workflow or two, and measure performance. Again, the Galaxy czars call could be a good bet. 2. What is the expected relation of integer-based and floating point based calculations, which will be loading the CPU cores? This also depends on the tools being used. This might be more relevant if your architecture were to use more specialized hardware (such as GPUs or FPGAs), but this should be a secondary concern. 3. Regarding the architectural differences (strengths, weaknesses): Would an AMD- or an Intel-System be more suitable? I really can't answer which processor line is more suitable, but I think that having enough RAM per core is more important. Nate shows that main.g2.bx.psu.edu has 4 GB RAM per core. 4. How much I/O (read and write) can be expected at the memory controllers? Which tasks are most I/O intensive (regarding RAM and/or HDDs)? Workflows currently write all output to disk and read all input from disk. This gets back to previous questions on benchmarking. 5. Roughly separated in mapping and clustering jobs: which amounts of main memory can be expected to be required by a single job (given e.g. Illumina exome data, 50x coverage)? As far as I know mapping should be around 4 GB, clustering much more (may reach high double digits). Nate's presentation shows that main.g2.bx.psu.edu has 24 to 48 GB per 8 core reservation, and as before it shows that there is 4 GB per core. 6. HDD access (R/W) is mainly in bigger blocks instead of masses of short operations - correct? Again, this all depends on the tool being used and could help with some benchmarks. This question sounds like it's mostly related to choosing the filesystem - is that right? If so, then you may want to consider a compressing file system such as ZFS or BtrFS. You may also want to consider filesystems like Ceph or Gluster (now Red Hat). I know that Ceph can run on top of XFS and BtrFS, but you should look into BtrFS's churn rate - it might still be evolving quickly. Again, a ping to the Galaxy Czars call may help on any and possibly all of these questions. Good luck! -Scott ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Local galaxy cannot connect to local toolshed
Hi, I managed to run a local Tool Shed and add a repository. On the same machine, a local galaxy is running, and configured for using this local toolshed. In the Admin menu of Galaxy, I can see my local Toolshed listed under 'Search and Browse tool sheds'. However, when I click on my local toolshed entry, following connection error appears in the main window: This webpage is not available The webpage at*http://localhost:9009/repository/browse_valid_repositories?galaxy_url=http://192.168.10.24/webapp=galaxy*might be temporarily down or it may have moved permanently to a new web address. Here are some suggestions: * Reload http://localhost:9009/repository/browse_valid_repositories?galaxy_url=http://192.168.10.24/webapp=galaxythis web page later. Error 102 (net::ERR_CONNECTION_REFUSED): The server refused the connection. Thanks for the help, Joachim Bioinformatics Training and Services (BITS) http://www.bits.vib.be @bitsatvib ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local galaxy Cufflinks problem
Chris, The problem is likely a bug in the Cufflinks wrapper that was recently fixed in galaxy-central: https://bitbucket.org/galaxy/galaxy-central/changeset/ff7c8c0bef60 Try updating your Galaxy instance to include this changeset; if this doesn't fix the problem, let us know. Good luck, J. On Nov 11, 2011, at 10:44 AM, Bidwell, Christopher A. wrote: Colleagues, We have set up a local Galaxy but I am having trouble running Cufflinks with the annotation gtf file. We have the iGenomes btau4.2 genome in the database and use the iGenomes btau4.2 gtf file in the history. However the –G isn’t showing up in the command line (below) and the output is the same as when I run cufflinks without annotation. Is something not set up correctly? Info: cufflinks v1.0.3 cufflinks -q --no-update-check -I 5 -F 0.05 -j 0.05 -p 4 -N -b /phillip/hiscansq/run00085-09-16-11_D0D1KACXX/110916_H179_0062_AD0D1KACXX/hr00206_OAR13-24/References/Bos_taurus/genome.fa The details page shows the gtf file being supplied from the history Input Parameter Value SAM or BAM file of aligned RNA-Seq reads 26: Tophat for Illumina on data 2 and data 1: accepted_hits Max Intron Length 5 Min Isoform Fraction 0.05 Pre MRNA Fraction 0.05 Perform quartile normalization Yes Conditional (reference_annotation) 1 Reference Aonnotation24: Btau42_iGenomes_annot.gtf Conditional (bias_correction) 0 Conditional (seq_source) 0 Conditional (singlePaired)0 Thanks for your help. Chris ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Local galaxy Cufflinks problem
Colleagues, We have set up a local Galaxy but I am having trouble running Cufflinks with the annotation gtf file. We have the iGenomes btau4.2 genome in the database and use the iGenomes btau4.2 gtf file in the history. However the –G isn’t showing up in the command line (below) and the output is the same as when I run cufflinks without annotation. Is something not set up correctly? Info: cufflinks v1.0.3 cufflinks -q --no-update-check -I 5 -F 0.05 -j 0.05 -p 4 -N -b /phillip/hiscansq/run00085-09-16-11_D0D1KACXX/110916_H179_0062_AD0D1KACXX/hr00206_OAR13-24/References/Bos_taurus/genome.fa The details page shows the gtf file being supplied from the history Input Parameter Value SAM or BAM file of aligned RNA-Seq reads 26: Tophat for Illumina on data 2 and data 1: accepted_hits Max Intron Length 5 Min Isoform Fraction 0.05 Pre MRNA Fraction 0.05 Perform quartile normalization Yes Conditional (reference_annotation) 1 Reference Aonnotation24: Btau42_iGenomes_annot.gtf Conditional (bias_correction) 0 Conditional (seq_source) 0 Conditional (singlePaired)0 Thanks for your help. Chris ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local Galaxy Instance MarkupSafe error
Hi Nate, It must be the issue here, I just learnt the head node is running with redhat 5 while Galaxy and all the dependencies are running on a redhat 6 server so yes the python version is not the same. I'll see what I can do with my IS department then. Cheers for the help :) Micka On 2 November 2011 17:18, Nate Coraor n...@bx.psu.edu wrote: Jerico Nico De Leon Revote wrote: It's the same case as what I'm getting. I can see the output via eye icon on the history panel and able to download the files as well. Do your cluster nodes have internet access? If so, log into a node and run the command again from there. Your nodes may have a different Python version or Unicode byte order encoding scheme than your Galaxy application server. --nate On 1 November 2011 03:37, Mickael ESCUDERO mickael.escud...@gmail.com wrote: Hi there, I'm getting exactly the same problem with any job running on a TORQUE/PBS cluster. The jobs actually run fine as I can see the output and download it, but it's marked as failed in the galaxy history, with the following message: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The command `python -ES ./scripts/fetch_eggs.py` gives nothing as output. If I run the same tools locally there is no problem. Cheers Micka Message: 5 Date: Thu, 27 Oct 2011 15:08:17 +1100 From: Jerico Nico De Leon Revote jerico.rev...@monash.edu To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Local Galaxy Instance MarkupSafe error Message-ID: cap9ulhyipxyb2cwtqosm3mquuu55arxy5por1dbahej1uco...@mail.gmail.com Content-Type: text/plain; charset=iso-8859-1 Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Thanks, Jerico -- next part -- An HTML attachment was scrubbed... URL: http://lists.bx.psu.edu/pipermail/galaxy-dev/attachments/20111027/cd777ff3/attachment-0001.html -- Message: 6 Date: Thu, 27 Oct 2011 16:58:13 +1100 From: Jerico Nico De Leon Revote jerico.rev...@monash.edu To: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Local Galaxy Instance MarkupSafe error Message-ID: cap9ulhbenenkqc50uzyxoca4w-zucdy8obs4i6b3e2qyfmy...@mail.gmail.com Content-Type: text/plain; charset=iso-8859-1 Just to follow-up on this. The MarkupSafe egg is definitely present on the eggs directory and the servers are ran through virtualenv. On 27 October 2011 15:08, Jerico Nico De Leon Revote jerico.rev...@monash.edu wrote: Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Thanks, Jerico -- next part -- An HTML attachment was scrubbed... URL: http://lists.bx.psu.edu/pipermail/galaxy-dev/attachments/20111027/f8baadf6/attachment-0001.html -- Message: 7 Date: Thu, 27 Oct 2011 02:40:51 -0400 From: Nate Coraor n...@bx.psu.edu To: Jerico Nico De Leon Revote jerico.rev...@monash.edu Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Local Galaxy Instance MarkupSafe error Message-ID: 20111027064051.gg2...@bx.psu.edu Content-Type: text/plain; charset=us-ascii Jerico Nico De Leon Revote wrote: Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Hi Jerico, Are you using a cluster? If not, could you run: % python -ES ./scripts/fetch_eggs.py --nate Thanks, Jerico ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu
Re: [galaxy-dev] Local Galaxy Instance MarkupSafe error
Hi Nate, Yes, that was the issue. I just made sure that the cluster nodes are using the same Python as the application server and it works now. Thanks for the support. Regards, Jerico On 3 November 2011 04:18, Nate Coraor n...@bx.psu.edu wrote: Jerico Nico De Leon Revote wrote: It's the same case as what I'm getting. I can see the output via eye icon on the history panel and able to download the files as well. Do your cluster nodes have internet access? If so, log into a node and run the command again from there. Your nodes may have a different Python version or Unicode byte order encoding scheme than your Galaxy application server. --nate On 1 November 2011 03:37, Mickael ESCUDERO mickael.escud...@gmail.com wrote: Hi there, I'm getting exactly the same problem with any job running on a TORQUE/PBS cluster. The jobs actually run fine as I can see the output and download it, but it's marked as failed in the galaxy history, with the following message: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The command `python -ES ./scripts/fetch_eggs.py` gives nothing as output. If I run the same tools locally there is no problem. Cheers Micka Message: 5 Date: Thu, 27 Oct 2011 15:08:17 +1100 From: Jerico Nico De Leon Revote jerico.rev...@monash.edu To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Local Galaxy Instance MarkupSafe error Message-ID: cap9ulhyipxyb2cwtqosm3mquuu55arxy5por1dbahej1uco...@mail.gmail.com Content-Type: text/plain; charset=iso-8859-1 Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Thanks, Jerico -- next part -- An HTML attachment was scrubbed... URL: http://lists.bx.psu.edu/pipermail/galaxy-dev/attachments/20111027/cd777ff3/attachment-0001.html -- Message: 6 Date: Thu, 27 Oct 2011 16:58:13 +1100 From: Jerico Nico De Leon Revote jerico.rev...@monash.edu To: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Local Galaxy Instance MarkupSafe error Message-ID: cap9ulhbenenkqc50uzyxoca4w-zucdy8obs4i6b3e2qyfmy...@mail.gmail.com Content-Type: text/plain; charset=iso-8859-1 Just to follow-up on this. The MarkupSafe egg is definitely present on the eggs directory and the servers are ran through virtualenv. On 27 October 2011 15:08, Jerico Nico De Leon Revote jerico.rev...@monash.edu wrote: Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Thanks, Jerico -- next part -- An HTML attachment was scrubbed... URL: http://lists.bx.psu.edu/pipermail/galaxy-dev/attachments/20111027/f8baadf6/attachment-0001.html -- Message: 7 Date: Thu, 27 Oct 2011 02:40:51 -0400 From: Nate Coraor n...@bx.psu.edu To: Jerico Nico De Leon Revote jerico.rev...@monash.edu Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Local Galaxy Instance MarkupSafe error Message-ID: 20111027064051.gg2...@bx.psu.edu Content-Type: text/plain; charset=us-ascii Jerico Nico De Leon Revote wrote: Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Hi Jerico, Are you using a cluster? If not, could you run: % python -ES ./scripts/fetch_eggs.py --nate Thanks, Jerico ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Mickael Escudero, Bioinformatics Officer
[galaxy-dev] Local Galaxy Instance MarkupSafe error
Hi there, I'm getting exactly the same problem with any job running on a TORQUE/PBS cluster. The jobs actually run fine as I can see the output and download it, but it's marked as failed in the galaxy history, with the following message: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The command `python -ES ./scripts/fetch_eggs.py` gives nothing as output. If I run the same tools locally there is no problem. Cheers Micka Message: 5 Date: Thu, 27 Oct 2011 15:08:17 +1100 From: Jerico Nico De Leon Revote jerico.rev...@monash.edu To: galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Local Galaxy Instance MarkupSafe error Message-ID: cap9ulhyipxyb2cwtqosm3mquuu55arxy5por1dbahej1uco...@mail.gmail.com Content-Type: text/plain; charset=iso-8859-1 Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Thanks, Jerico -- next part -- An HTML attachment was scrubbed... URL: http://lists.bx.psu.edu/pipermail/galaxy-dev/attachments/20111027/cd777ff3/attachment-0001.html -- Message: 6 Date: Thu, 27 Oct 2011 16:58:13 +1100 From: Jerico Nico De Leon Revote jerico.rev...@monash.edu To: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Local Galaxy Instance MarkupSafe error Message-ID: cap9ulhbenenkqc50uzyxoca4w-zucdy8obs4i6b3e2qyfmy...@mail.gmail.com Content-Type: text/plain; charset=iso-8859-1 Just to follow-up on this. The MarkupSafe egg is definitely present on the eggs directory and the servers are ran through virtualenv. On 27 October 2011 15:08, Jerico Nico De Leon Revote jerico.rev...@monash.edu wrote: Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Thanks, Jerico -- next part -- An HTML attachment was scrubbed... URL: http://lists.bx.psu.edu/pipermail/galaxy-dev/attachments/20111027/f8baadf6/attachment-0001.html -- Message: 7 Date: Thu, 27 Oct 2011 02:40:51 -0400 From: Nate Coraor n...@bx.psu.edu To: Jerico Nico De Leon Revote jerico.rev...@monash.edu Cc: galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Local Galaxy Instance MarkupSafe error Message-ID: 20111027064051.gg2...@bx.psu.edu Content-Type: text/plain; charset=us-ascii Jerico Nico De Leon Revote wrote: Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Hi Jerico, Are you using a cluster? If not, could you run: % python -ES ./scripts/fetch_eggs.py --nate Thanks, Jerico ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Mickael Escudero, Bioinformatics Officer ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local Galaxy Instance MarkupSafe error
Jerico Nico De Leon Revote wrote: Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Hi Jerico, Are you using a cluster? If not, could you run: % python -ES ./scripts/fetch_eggs.py --nate Thanks, Jerico ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Local Galaxy Instance MarkupSafe error
Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Thanks, Jerico ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local Galaxy Instance MarkupSafe error
Just to follow-up on this. The MarkupSafe egg is definitely present on the eggs directory and the servers are ran through virtualenv. On 27 October 2011 15:08, Jerico Nico De Leon Revote jerico.rev...@monash.edu wrote: Hi, I'm just doing a simple get-data from UCSC on our local Galaxy instance and got the following error: WARNING:galaxy.eggs:Warning: MarkupSafe (a dependent egg of Mako) cannot be fetched The job box then is displayed as red on the history panel. The job runner states that the job finished normally on the cluster. Galaxy is checkout from galaxy-central (changeset: 6176:34fffbf01183). Thanks, Jerico ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local Galaxy implimentation
Lee: I am CC'ing you e-mail to galaxy-dev list. You can get a lot of useful answers by posting to this list. Maintaining Galaxy is not different from maintaining other software resources. To run it locally you will need to have a system administration supervision as with any other high performance system. It is wonderful that you are sending your staff members to Galaxy Community Conference as they will be able to interact with key Galaxy Team members such as Nate Coraor, who runs Penn State instance here. Thanks! anton On Feb 25, 2011, at 6:14 AM, Lee Hazelwood wrote: Dear Prof Nekrutenko, The university where I work is looking to locally implement your elegant software to enable us to view and analyse the rapidly growing amount of sequence data. We will be sending two staff members to the conference in Lunteren in May to get right into the detail on how we would achieve this. However, with all things I need to request some time/ support for running such a system in advance of May. If you could help by briefly answering the questions below I would much appreciate it. * What is a realistic amount of someones time to implement Galaxy on a Linux server so that it can be used by researchers using the Web? * What is the usual time overhead to keep it running and how skilled would they have to be? I should say, this will be for local biological researchers only, approx 20-30 dealing in Exome and Whole genome sequences. Best regards Lee -- Dr Lee Hazelwood BHRC Bioinformatics Technology Group Leader Biomedical and Health Research Centre Faculty of Biological Sciences Garstang Building, University of Leeds, Leeds, LS2 9JT, UK. E-mail: l.d.hazelw...@leeds.ac.ukmailto:l.d.hazelw...@leeds.ac.uk Anton Nekrutenko http://nekrut.bx.psu.edu http://usegalaxy.org ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local Galaxy implimentation
Hi Lee * What is a realistic amount of someones time to implement Galaxy on a Linux server so that it can be used by researchers using the Web? Galaxy is simple to set up and works out-of-the-box. You will have a functional Galaxy server within minutes (yes, 'within minutes'). Providing Galaxy in a production environment will need a few extra steps (see: https://bitbucket.org/galaxy/galaxy-central/wiki/Config/ProductionServer ). Depending on your needs, not all the steps are required. I highly recommend, that you use PostgreSQL or MySQL (instead of the built in SQLite) from day 1. Nevertheless, you should have a reliable server within a few days. * What is the usual time overhead to keep it running and how skilled would they have to be? Galaxy itself needs hardly any maintenance. However, it is advised to update the code on a regularly basis. Currently we do it every three months. We have a test and a production server, so the update process is split over two days, with a couple of days testing in between. The actual code updates have been very smooth so far and require about an hour of downtime. As long as you have support from your IT-Department running the hardware, you don't need extraordinary skills. Galaxy is written in Python, but you don't need to be a Python programmer (even if you want to make a few changes), as long as you have some basic programming skills in another language. There is a lot of information on the wiki (sometimes, the info is difficult to find). And if you get stuck, you can always send an e-mail to galaxy-dev Of course, it all changes when you add your own tools. Although, the actual addition is simple. You might need some extra time for testing and improving your tools. Also, don't forget the time required to teach the use of Galaxy in general and the use of the self-written tools to your colleagues at the bench. Regards, Hans Hans-Rudolf Hotz, PhD Bioinformatics Support Friedrich Miescher Institute for Biomedical Research Maulbeerstrasse 66 4058 Basel/Switzerland I should say, this will be for local biological researchers only, approx 20-30 dealing in Exome and Whole genome sequences. Best regards Lee -- Dr Lee Hazelwood BHRC Bioinformatics Technology Group Leader Biomedical and Health Research Centre Faculty of Biological Sciences Garstang Building, University of Leeds, Leeds, LS2 9JT, UK. E-mail: l.d.hazelw...@leeds.ac.ukmailto:l.d.hazelw...@leeds.ac.uk Anton Nekrutenko http://nekrut.bx.psu.edu http://usegalaxy.org ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local Galaxy implimentation
Hi Lee, If setting up Galaxy on an existing supported cluster, I would estimate the setup time for an experienced sysadmin to be a day or two, although that number can be larger if a lot of tools that depend on external binaries (e.g. NGS mappers) and locally cached reference genomes are desired, or you'd like to use the Sample Tracking/sequencer integration features. Once running, it usually only needs intervention if more disk space is needed, or to perform updates to new versions. New stable versions are released about once a month and should usually only take about 10 minutes to apply, although we reccommend reading the accompanying Development News Brief. Reading the brief ensures there aren't any surprises, and to be made aware of any new configuration options or features. This task is fairly simple but a basic to intermediate knowledge of system administration will be helpful here. You may want to have your staff review the documentation at http://usegalaxy.org/production to get an idea of what's involved in setting up a production Galaxy server. Anton Nekrutenko wrote: Lee: I am CC'ing you e-mail to galaxy-dev list. You can get a lot of useful answers by posting to this list. Maintaining Galaxy is not different from maintaining other software resources. To run it locally you will need to have a system administration supervision as with any other high performance system. It is wonderful that you are sending your staff members to Galaxy Community Conference as they will be able to interact with key Galaxy Team members such as Nate Coraor, who runs Penn State instance here. Thanks! anton On Feb 25, 2011, at 6:14 AM, Lee Hazelwood wrote: Dear Prof Nekrutenko, The university where I work is looking to locally implement your elegant software to enable us to view and analyse the rapidly growing amount of sequence data. We will be sending two staff members to the conference in Lunteren in May to get right into the detail on how we would achieve this. However, with all things I need to request some time/ support for running such a system in advance of May. If you could help by briefly answering the questions below I would much appreciate it. * What is a realistic amount of someones time to implement Galaxy on a Linux server so that it can be used by researchers using the Web? * What is the usual time overhead to keep it running and how skilled would they have to be? I should say, this will be for local biological researchers only, approx 20-30 dealing in Exome and Whole genome sequences. Best regards Lee -- Dr Lee Hazelwood BHRC Bioinformatics Technology Group Leader Biomedical and Health Research Centre Faculty of Biological Sciences Garstang Building, University of Leeds, Leeds, LS2 9JT, UK. E-mail: l.d.hazelw...@leeds.ac.ukmailto:l.d.hazelw...@leeds.ac.uk Anton Nekrutenko http://nekrut.bx.psu.edu http://usegalaxy.org ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] local galaxy display problem
Ryan Golhar wrote: I've implemented the Serving Galaxy at a sub directory (such as /galaxy), made the change to Apache, and the universe_wsgi.ini. When I view the page through Apache using http://server.name/galaxy, the page is garbled like something didn't take. Hi Ryan, Did you restart Galaxy after setting up the [filter:proxy-prefix] section and the 'filter-with = proxy-prefix' setting? I did. I restarted both Apache and Galaxy. I verified it using 'ps -ef | grep python' to see that it in fact was shutdown, prior to restarting. Okay, if you're using the static content proxying as in the instructions, can you verify that those paths are correct? For example, the URL: http://server.name/galaxy/static/images/galaxyIcon_noText.png Should display a small icon. --nate ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] local galaxy display problem
Ryan Golhar wrote: SOLVED. I copied/paste the Rewrite rules for apache from the wiki without updating the paths to my installation...once I did that, all worked well. Whoops, I should read ahead before I reply. Glad it's working! --nate On 2/17/11 5:07 PM, Ryan Golhar wrote: I've set up a local instance of Galaxy following the instructions in the wiki for a production instance. I'm running on CentOS with Apache, and a mysql database server. Everything looks great when viewed directly through the Galaxy web server on port 8080. I've implemented the Serving Galaxy at a sub directory (such as /galaxy), made the change to Apache, and the universe_wsgi.ini. When I view the page through Apache using http://server.name/galaxy, the page is garbled like something didn't take. Hi Ryan, Did you restart Galaxy after setting up the [filter:proxy-prefix] section and the 'filter-with = proxy-prefix' setting? I did. I restarted both Apache and Galaxy. I verified it using 'ps -ef | grep python' to see that it in fact was shutdown, prior to restarting. ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] local galaxy display problem
I've set up a local instance of Galaxy following the instructions in the wiki for a production instance. I'm running on CentOS with Apache, and a mysql database server. Everything looks great when viewed directly through the Galaxy web server on port 8080. I've implemented the Serving Galaxy at a sub directory (such as /galaxy), made the change to Apache, and the universe_wsgi.ini. When I view the page through Apache using http://server.name/galaxy, the page is garbled like something didn't take. I've restarted Apache and Galaxy but its still not work... -- CONFIDENTIALITY NOTICE: This email communication may contain private, confidential, or legally privileged information intended for the sole use of the designated and/or duly authorized recipient(s). If you are not the intended recipient or have received this email in error, please notify the sender immediately by email and permanently delete all copies of this email including all attachments without reading them. If you are the intended recipient, secure the contents in a manner that conforms to all applicable state and/or federal requirements related to privacy and confidentiality of such information. attachment: golharam.vcf___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] local galaxy display problem
Ryan Golhar wrote: I've set up a local instance of Galaxy following the instructions in the wiki for a production instance. I'm running on CentOS with Apache, and a mysql database server. Everything looks great when viewed directly through the Galaxy web server on port 8080. I've implemented the Serving Galaxy at a sub directory (such as /galaxy), made the change to Apache, and the universe_wsgi.ini. When I view the page through Apache using http://server.name/galaxy, the page is garbled like something didn't take. Hi Ryan, Did you restart Galaxy after setting up the [filter:proxy-prefix] section and the 'filter-with = proxy-prefix' setting? --nate I've restarted Apache and Galaxy but its still not work... -- CONFIDENTIALITY NOTICE: This email communication may contain private, confidential, or legally privileged information intended for the sole use of the designated and/or duly authorized recipient(s). If you are not the intended recipient or have received this email in error, please notify the sender immediately by email and permanently delete all copies of this email including all attachments without reading them. If you are the intended recipient, secure the contents in a manner that conforms to all applicable state and/or federal requirements related to privacy and confidentiality of such information. begin:vcard fn:Ryan Golhar, Ph.D. n:Golhar;Ryan org:The Cancer Institute of NJ;Cancer Informatics Core/Bioinformatics adr:5th floor;;120 Albany St;New Brunswick;NJ;08901;USA email;internet:golha...@umdnj.edu title:NGS Bioinformatics Specialist tel;work:(732) 235-6613 tel;fax:(732) 235-6267 tel;cell:(732) 236-1176 x-mozilla-html:FALSE url:http://www.cinj.org version:2.1 end:vcard ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local galaxy installation problem. Please help.
Hi Nate, Thank you for your reply. I am getting there, but I need a bit more help. Firstly, yes, I had an alias hg (which was history | grep $1). I got rid of it and hg is working fine. Secondly, I installed the latest version of zlib (1.2.5) and I can go further with sh run.sh, but it hangs after it tells me: File /usr/local/lib/python2.5/SocketServer.py, line 341, in server_bind self.socket.bind(self.server_address) File string, line 1, in bind socket.error: (98, 'Address already in use') When I point my browser to http://vixen:8080 (vixen is the very machine on which I am implementing galaxy), a tomcat page comes up (telling me that tomcat is running fine). Here's my question: Is there a way to run galaxy on ports other than 8080? Thank you! Regards, Tena Sakai PS: Thank you for updating the link on wiki page. On 2/17/11 9:57 AM, Nate Coraor n...@bx.psu.edu wrote: Nate Coraor wrote: $ uname -v #1 SMP Tue Jul 22 18:01:05 EDT 2008 $ uname -r 2.6.9-78.0.1.ELsmp RHEL 4, I believe? $ cd galaxy-dist $ sh run.sh ...lots of lines here... Fetch successful. ...more lines here... ysam/csamtools.py, line 6, in __bootstrap__ ImportError: /home/tsakai/.python-eggs/pysam-0.1.1_kanwei_90e03180969d-py2.5-linux-x86_64 -ucs2.egg-tmp/pysam/csamtools.so: undefined symbol: deflateSetHeader $ echo $? 1 So, I am lost now. Can somebody please give me a tip as to how to proceed? I'll dig in to this last bit and get back to you ASAP. It would appear that your zlib does not contain deflateSetHeader, which was added to zlib in version 1.2.2.1, released in 2004. RHEL 4.X apparently ships with zlib 1.2.1.2. If you have a newer zlib installed somewhere, you can work around this problem by setting $LD_LIBRARY_PATH to point to the directory with the newer libz.so.1. LD_LIBRARY_PATH is a hack so this is not ideal, but you may not want to override libz globally by changing something like /etc/ld.so.conf. --nate ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] local galaxy display problem
SOLVED. I copied/paste the Rewrite rules for apache from the wiki without updating the paths to my installation...once I did that, all worked well. On 2/17/11 5:07 PM, Ryan Golhar wrote: I've set up a local instance of Galaxy following the instructions in the wiki for a production instance. I'm running on CentOS with Apache, and a mysql database server. Everything looks great when viewed directly through the Galaxy web server on port 8080. I've implemented the Serving Galaxy at a sub directory (such as /galaxy), made the change to Apache, and the universe_wsgi.ini. When I view the page through Apache using http://server.name/galaxy, the page is garbled like something didn't take. Hi Ryan, Did you restart Galaxy after setting up the [filter:proxy-prefix] section and the 'filter-with = proxy-prefix' setting? I did. I restarted both Apache and Galaxy. I verified it using 'ps -ef | grep python' to see that it in fact was shutdown, prior to restarting. ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Local galaxy installation problem. Please help.
Hi Nate, Please ignore my previous post. I got it working. I changed an entry in universe_wsgi.ini file. From #host = 127.0.0.1 to host = vixen.egcrc.org. It appears to be working. Many thanks for your help. Regards, Tena Sakai On 2/17/11 1:48 PM, Tena Sakai tsa...@gallo.ucsf.edu wrote: Hi Nate, I am still not there, though I am getting closer... I changed the port to in universe_wsgi.ini and Here's what appears on my screen: galaxy.web.buildapp DEBUG 2011-02-17 13:42:08,285 Enabling 'eval exceptions' middleware galaxy.web.buildapp DEBUG 2011-02-17 13:42:08,286 Enabling 'trans logger' middleware galaxy.web.buildapp DEBUG 2011-02-17 13:42:08,286 Enabling 'config' middleware galaxy.web.buildapp DEBUG 2011-02-17 13:42:08,287 Enabling 'x-forwarded-host' middleware Starting server in PID 28721. serving on http://127.0.0.1: When I try to go there on my browser, the complaint from browser (I used both safari and firefox) is: Firefox can't establish a connection to the server at vixen:. Or Safari can¹t open the page ³http://vixen:/² because Safari can¹t connect to the server ³vixen². But when I change to 8080, I do get a page from tomcat. I see that the process is running: $ ps axl | grep 28721 0 1026 28721 28588 15 0 390520 83456 finish Tl pts/9 0:14 python ./scripts/paster.py serve universe_wsgi.ini 0 1026 28781 27316 16 0 51116 668 pipe_w S+ pts/9 0:00 grep 28721 Any clue, why I can't get there by browser? Thank you. Regards, Tena Sakai On 2/17/11 12:57 PM, Nate Coraor n...@bx.psu.edu wrote: Tena Sakai wrote: Hi Nate, Thank you for your reply. I am getting there, but I need a bit more help. Firstly, yes, I had an alias hg (which was history | grep $1). I got rid of it and hg is working fine. Secondly, I installed the latest version of zlib (1.2.5) and I can go further with sh run.sh, but it hangs after it tells me: File /usr/local/lib/python2.5/SocketServer.py, line 341, in server_bind self.socket.bind(self.server_address) File string, line 1, in bind socket.error: (98, 'Address already in use') When I point my browser to http://vixen:8080 (vixen is the very machine on which I am implementing galaxy), a tomcat page comes up (telling me that tomcat is running fine). Here's my question: Is there a way to run galaxy on ports other than 8080? Yes, in the file universe_wsgi.ini in the root of the Galaxy directory, one of the options near the top of the file is 'port ='. This can be changed to whatever you prefer. --nate Thank you! Regards, Tena Sakai PS: Thank you for updating the link on wiki page. On 2/17/11 9:57 AM, Nate Coraor n...@bx.psu.edu wrote: Nate Coraor wrote: $ uname -v #1 SMP Tue Jul 22 18:01:05 EDT 2008 $ uname -r 2.6.9-78.0.1.ELsmp RHEL 4, I believe? $ cd galaxy-dist $ sh run.sh ...lots of lines here... Fetch successful. ...more lines here... ysam/csamtools.py, line 6, in __bootstrap__ ImportError: /home/tsakai/.python-eggs/pysam-0.1.1_kanwei_90e03180969d-py2.5-linux-x86 _ 64 -ucs2.egg-tmp/pysam/csamtools.so: undefined symbol: deflateSetHeader $ echo $? 1 So, I am lost now. Can somebody please give me a tip as to how to proceed? I'll dig in to this last bit and get back to you ASAP. It would appear that your zlib does not contain deflateSetHeader, which was added to zlib in version 1.2.2.1, released in 2004. RHEL 4.X apparently ships with zlib 1.2.1.2. If you have a newer zlib installed somewhere, you can work around this problem by setting $LD_LIBRARY_PATH to point to the directory with the newer libz.so.1. LD_LIBRARY_PATH is a hack so this is not ideal, but you may not want to override libz globally by changing something like /etc/ld.so.conf. --nate ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
