Re: [galaxy-dev] running two Galaxy installations on one server - was: Re: Can not clean my galaxy datasets
No, I'm sure it's possible. I can't say how easy it is, I've never tested it myself, but I would assume one could use virtual hosts or something similar. I personally just like keeping such things as separate as possible, and having them as separate VMs makes it a little easier to distinctly maintain each. chris On 12/22/2011 09:13 AM, liram_va...@agilent.com wrote: Hi Chris, Thank you. Just to make sure that I fully understand: Basically you say that there are no easy way to run two Galaxy process on the same Linux operating system? Thanks, Liram -Original Message- From: Fields, Christopher J [mailto:cjfie...@illinois.edu] Sent: Thursday, December 22, 2011 4:50 PM To: Hans-Rudolf Hotz Cc: VARDI,LIRAM (A-Labs,ex1); galaxy-...@bx.psu.edu; BEN-DOR,AMIR (A-Labs,ex1) Subject: Re: [galaxy-dev] running two Galaxy installations on one server - was: Re: Can not clean my galaxy datasets We are anticipating a similar issue and decided it would be easier to just have separate VMs where possible. I know this isn't always possible, but it is a very easy way to both manage multiple galaxy instances and keep the various galaxy instances completely separate. Chris On Dec 22, 2011, at 6:10 AM, Hans-Rudolf Hotzh...@fmi.ch wrote: On 12/22/2011 11:46 AM, liram_va...@agilent.com wrote: Hi Hans, Thank you for your reply. Anyway, I tried to install Galaxy again by using the site check-out (hg clone https://...) and it still cause problems. Specifically, those were my steps: 1) I Installed galaxy in different location. 2) After run run.sh in the first time, I changed universe_wsgi.ini to listen on different port. It is hard to believe that your new test installation running with SQLite interferes with your existing installation running with PostgreSQL(?). Are you sure, there is not a third 'party' involved? Also, I would stop at this step (ie: changed universe_wsgi.ini to listen on different port) and run your installation first like this. And wait with copying anything from the exiting code or data or directories. Till you manage to get the two web site to run in parallel without interfering with each other. Regards, Hans 3) Manually, I copied tools and tool-data directories from the old instance of Galaxy to the new location (In order to get my new tools). 4) Also, I copied the old config tool file: tool_conf.xml 5) Then, Run the new Galaxy instance. In parallel, the old Galaxy is already on line. The problem is that any operation on the new Galaxy instance cause to the other Galaxy to Throw the following error message: URL: http://ip address.../user/login?use_panels=False File '/home/liram/sdb1/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py', line 364 in respond app_iter = self.application(environ, detect_start_response) File '/home/liram/sdb1/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/debug/prints.py', line 98 in __call__ environ, self.app) File '/home/liram/sdb1/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/wsgilib.py', line 539 in intercept_output app_iter = application(environ, replacement_start_response) . . . . File '/home/liram/sdb1/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg/sqlalchemy/engine/base.py', line 931 in _handle_dbapi_exception raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect) OperationalError: (OperationalError) database is locked u'UPDATE galaxy_session SET update_time=?, is_valid=? WHERE galaxy_session.id = ?' ['2011-12-22 10:27:05.978992', 0, 158] I suppose that the problem is again caused due to collision between the databases. But how can I separate between them? Thanks you!! Liram -Original Message- From: Hans-Rudolf Hotz [mailto:h...@fmi.ch] Sent: Wednesday, December 21, 2011 4:59 PM To: VARDI,LIRAM (A-Labs,ex1) Cc: galaxy-...@bx.psu.edu; BEN-DOR,AMIR (A-Labs,ex1) Subject: running two Galaxy installations on one server - was: Re: [galaxy-dev] Can not clean my galaxy datasets Hi Liram I was suggesting: instead of copying the existing Galaxy directory into another location. Download and install a new Galaxy installation in a separate location on your file system. for the installation procedure, see: http://wiki.g2.bx.psu.edu/Admin/Get%20Galaxy all you need to do in the new installation is changing the port number. It will create its own file system and by default it will use its own SQLite database. Hence, there should be no interference with your existing, production Galaxy installation. Regards, Hans On 12/21/2011 09:16 AM, liram_va...@agilent.com wrote: Hi Hans, Thank you for your quick reply. Sorry, But I'm a rookie Galaxy user, So may you pls explain me in details the meaning by running a second Galaxy server in parallel using a fresh check-out ? Thank a lot for your assistance! Liram -Original Message- From: Hans-Rudolf Hotz [mailto:h...@fmi.ch] Sent: Monday, December 19, 2011 7:01 PM To: VARDI,LIRAM
Re: [galaxy-dev] running two Galaxy installations on one server - was: Re: Can not clean my galaxy datasets
On 12/22/2011 09:57 AM, amir_ben-...@agilent.com wrote: Hi Hans, Thank you again on your prompt answer. From your reply below, it is clear that it should be possible to run multiple Galaxy servers under a single linux - this is exactly what we are trying to do: running two galaxy servers (one production one for test). I would think this is relatively common scenario (i.e., setting up two servers), so it might be a good idea to figure out and post instructions on the wiki how to achieve this. http://wiki.g2.bx.psu.edu/Admin/Config/Apache%20Proxy Note under 'Basic Configuration'. We are really novice with regards to the linux OS (installed it within a 'virtualbox' just to be able to run the galaxy server) - so there is no local linux sysadmin to consult (or someone that knows how to work with the database). We were hoping it would be relatively easy process to setup a galaxy server, perhaps we were too naïve... Maybe a virtualbox-specific configuration issue, e.g. specify resources available to the VM. Is there a limit on mem, CPU, open ports, etc? We do not understand the collision between the two instances, as they were installed in two different locations altogether (the production server is under '/home/liram/sdb1/galaxy-dist', and the test server is under'/home/liram/test/galaxy-dist') . Somehow, it seems that the two server do try to access some shared resource - we have no clue what is the shared resource and how to fix this. Are you sure you have separate directories specifies with Apache? For instance, are both configured as the web server root (see link above)? Let us know if there is some log file or additional information we can send you as we are completely stuck now :-( Thanks again for your help, Amir Ben-Dor As Hans implied, might be nice to see how you configured Apache and Galaxy. chris -Original Message- From: Hans-Rudolf Hotz [mailto:h...@fmi.ch] Sent: Thursday, December 22, 2011 5:26 PM To: VARDI,LIRAM (A-Labs,ex1) Cc: cjfie...@illinois.edu; galaxy-...@bx.psu.edu; BEN-DOR,AMIR (A-Labs,ex1) Subject: Re: [galaxy-dev] running two Galaxy installations on one server - was: Re: Can not clean my galaxy datasets On 12/22/2011 04:13 PM, liram_va...@agilent.com wrote: Hi Chris, Thank you. Just to make sure that I fully understand: Basically you say that there are no easy way to run two Galaxy process on the same Linux operating system? Hi Liram I am sorry, but I can't leave this statement out there without strongly objecting! I have been running up to 5 Galaxy servers (one production and four test servers) on the same box without any problems. It is difficult to help you remotely, with only seeing snippets from your errors. But I am sure your local sysadmin can help you setting up the relational databases and making sure the file systems for your two servers are really separate from each other. Hans Thanks, Liram -Original Message- From: Fields, Christopher J [mailto:cjfie...@illinois.edu] Sent: Thursday, December 22, 2011 4:50 PM To: Hans-Rudolf Hotz Cc: VARDI,LIRAM (A-Labs,ex1); galaxy-...@bx.psu.edu; BEN-DOR,AMIR (A-Labs,ex1) Subject: Re: [galaxy-dev] running two Galaxy installations on one server - was: Re: Can not clean my galaxy datasets We are anticipating a similar issue and decided it would be easier to just have separate VMs where possible. I know this isn't always possible, but it is a very easy way to both manage multiple galaxy instances and keep the various galaxy instances completely separate. Chris On Dec 22, 2011, at 6:10 AM, Hans-Rudolf Hotzh...@fmi.ch wrote: On 12/22/2011 11:46 AM, liram_va...@agilent.com wrote: Hi Hans, Thank you for your reply. Anyway, I tried to install Galaxy again by using the site check-out (hg clone https://...) and it still cause problems. Specifically, those were my steps: 1) I Installed galaxy in different location. 2) After run run.sh in the first time, I changed universe_wsgi.ini to listen on different port. It is hard to believe that your new test installation running with SQLite interferes with your existing installation running with PostgreSQL(?). Are you sure, there is not a third 'party' involved? Also, I would stop at this step (ie: changed universe_wsgi.ini to listen on different port) and run your installation first like this. And wait with copying anything from the exiting code or data or directories. Till you manage to get the two web site to run in parallel without interfering with each other. Regards, Hans 3) Manually, I copied tools and tool-data directories from the old instance of Galaxy to the new location (In order to get my new tools). 4) Also, I copied the old config tool file: tool_conf.xml 5) Then, Run the new Galaxy instance. In parallel, the old Galaxy is already on line. The problem is that any operation on the new Galaxy instance cause to the other Galaxy to Throw the following error message: URL: http://ip
Re: [galaxy-dev] Tools requiring java can't find java
The following has a bit of advice on that: http://www.admon.org/difference-between-login-shell-and-non-login-shell/ chris On Aug 30, 2011, at 1:03 PM, Chorny, Ilya wrote: How do I set my path in a non login shell? From: George Marselis [mailto:george.marse...@kaust.edu.sa] Sent: Tuesday, August 30, 2011 9:40 AM To: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Tools requiring java can't find java Yes, your java path is not set, because you are using an non-login shell. Indeed, you should be talking to your sysadmins. George Marselis, systems administrator Building #2, Level 4, room 4327 Computational Bioscience Research Center, KAUST Land: +966-2-808-2944, Mobile: +966-56-321-7713, Skype: project2501a From: Chorny, Ilya icho...@illumina.com Date: Tue, 30 Aug 2011 19:23:42 +0300 To: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Tools requiring java can't find java This is probably a question for our sys admin but when I run tools such as GATK I get an error java command not found. I can execute java from the command line and it works. Further I can grab the drmaa submission script (ie galaxy_job_id.sh) and qsub it from the command line and it finds JAVA. Any thoughts? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.com Website: www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Tools requiring java can't find java
Gah, wrong link, apologies (the other one is okay, if a bit wonky): http://www.linuxfromscratch.org/blfs/view/cvs/postlfs/profile.html chrus On Aug 30, 2011, at 1:12 PM, Chris Fields wrote: The following has a bit of advice on that: http://www.admon.org/difference-between-login-shell-and-non-login-shell/ chris On Aug 30, 2011, at 1:03 PM, Chorny, Ilya wrote: How do I set my path in a non login shell? From: George Marselis [mailto:george.marse...@kaust.edu.sa] Sent: Tuesday, August 30, 2011 9:40 AM To: Chorny, Ilya; galaxy-dev@lists.bx.psu.edu Subject: Re: [galaxy-dev] Tools requiring java can't find java Yes, your java path is not set, because you are using an non-login shell. Indeed, you should be talking to your sysadmins. George Marselis, systems administrator Building #2, Level 4, room 4327 Computational Bioscience Research Center, KAUST Land: +966-2-808-2944, Mobile: +966-56-321-7713, Skype: project2501a From: Chorny, Ilya icho...@illumina.com Date: Tue, 30 Aug 2011 19:23:42 +0300 To: galaxy-dev@lists.bx.psu.edu galaxy-dev@lists.bx.psu.edu Subject: [galaxy-dev] Tools requiring java can't find java This is probably a question for our sys admin but when I run tools such as GATK I get an error java command not found. I can execute java from the command line and it works. Further I can grab the drmaa submission script (ie galaxy_job_id.sh) and qsub it from the command line and it finds JAVA. Any thoughts? Thanks, Ilya Ilya Chorny Ph.D. Bioinformatics Scientist I Illumina, Inc. 9885 Towne Centre Drive San Diego, CA 92121 Work: 858.202.4582 Email: icho...@illumina.com Website: www.illumina.com ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Quota
I recall this being mentioned at the Galaxy Conference, and I think there are quotas in place for the PSU servers. Maybe it's in place but not documented yet? chris On Aug 12, 2011, at 7:49 AM, Steve Taylor wrote: Hi, Are there any plans to have a disk space quota for Galaxy users that can be set by admins? Thanks, Steve ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] job status when SGE kills/aborts job
On Jul 30, 2011, at 4:58 AM, Peter Cock wrote: On Saturday, July 30, 2011, Shantanu Pavgi pa...@uab.edu wrote: The -V option is not for verbose mode but for exporting your shell environment. Refer to qsub manual for details: Specifies that all environment variables active within the qsub utility be exported to the context of the job. We are already using it in our configuration as needed. That is such a common need it would be great to have it in the Galaxy documentation as an example of using native SGE options with drmaa:// in universe_wsgi.ini Plus the http://linux.die.net/man/5/sge_complex link. Thanks! I think we are having problem with the galaxy (or drmaa Python lib) parsing correct drmaa/SGE messages and not with the drmaa URL configuration. Thoughts? I'd try adding a few debug log/print statements to the code to try and diagnose it. Peter The option is the same for Torque/PBS as well. We'll have the chance (or misfortune depending on how you look at it) of testing both SGE and Torque locally. chris ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] BWA sampe runs 'forever'
On Jul 29, 2011, at 12:44 PM, Assaf Gordon wrote: Chris Fields wrote, On 07/29/2011 12:35 PM: On Jul 29, 2011, at 11:00 AM, Assaf Gordon wrote: Question for galaxy maintainers: have you encountered situations where BWA jobs run 'forever' (for days) ? ... I'm wondering if this is common enough to justify adding -A to the wrapper, or is it rare enough and can be ignored. Can it be added as an optional argument? So that if it is selected, the setting is passed to the tool, otherwise not. Obviously, it can be added to the XML wrapper. I'm asking if other people encountered this situation with Paired-end mapping, and whether it's rare or not. If there are cases where this does occur (as you have pointed out), then the XML wrapper should have the option and maybe even a bit of documentation pointing this out (even if it is a rare event). I wouldn't want this running on a cluster where users are charged, for instance. From a user's perspective, this kind of parameter makes no sense: it basically tells the user: run BWA without this parameter, and if after 24 hours the job is still running, kill it and re-run with this parameter. That's not the way to build user-friendly tools :( -gordon It's definitely not intuitive and violates least surprise. If I explicitly set any parameter, I don't expect the application to possibly (and silently) assume I am wrong and change it anyway. chris ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Updating tools on new hg based Galaxy Tool Shed
On Jun 16, 2011, at 10:23 AM, Peter Cock wrote: On Thu, Jun 16, 2011 at 3:26 PM, Greg Von Kuster g...@bx.psu.edu wrote: Hi Peter, On Jun 16, 2011, at 10:10 AM, Peter Cock wrote: Are you expecting tool authors to work primarily at the hg level? You didn't answer that one ;) Not necessarily, but hg is the basis for uploading and downloading tools. I'm not sure if it will be possible to completely eliminate the requirement of a tool developer using hg, but we're making every attempt to do so. Good. Why are you so averse to using hg? Because git suits me better? ;) Seriously, I have no strong aversion to hg - what puts me off a little is the need to have one repo per tool or tool-suite, compared to my current setup where all by tools are in a branch from the main Galaxy repo. There would be a significant time and effort cost in switching, made worse by having multiple tools on the Tool Shed. I'm actually thinking of this (requiring hg knowledge) as a more general issue, namely a potential impediment to new Tool Shed contributors. Peter I'm of the opposite mind; I'm not sure there is an absolute need to have one's tools be a branch or fork of the main tool shed, though I agree there is also utility in allowing that (having a customized 'main' repo, or optimizing tools already present). Having completely separate repos for tools seems cleaner, focusing development on those tools alone (not any of the others present in a branch) and pushes maintenance of the tool back to the tool developer themselves (e.g. there is no need for the galaxy devs to 'pull' in changes at any point into a main repo). This might also allow the galaxy devs to designate a set of 'blessed' or supported tools in various repositories. Re: git: as Peter knows I'm also primarily a git/github user. It is feasible at some future point to allow git/github repos. For instance, one could possibly integrate github usage via this: http://hg-git.github.com/ Of course, I think it's much more important that any additional vcs integration wait until the new tool shed interface stabilizes somewhat, but (at least from the github perspective) seems like it shouldn't be terribly hard to do. chris ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] The new hg based Galaxy Tool Shed
(apologies in advance, limiting my response to the two questions below) On Jun 1, 2011, at 11:54 AM, Peter Cock wrote: On Wed, Jun 1, 2011 at 5:25 PM, Nate Coraor n...@bx.psu.edu wrote: Peter Cock wrote: Well, yes and no - as long as there are competing versions of a Galaxy tool (e.g. from an original author and a fork by a second author), and they use the same ID in their XML, you have a clash. This will have to be considered in the (automated) install interface. i.e. In general, when installing or updating any tool, there may be existing versions of some components already present. In fact two completely unrelated tools could even have the same XML ID by accident. I agree there could be a problem with tool ID uniqueness. We've talked about suggesting that people namespace their tool IDs to prevent this, but nothing formal has materialized at this point. That sounds sensible, and the sooner the better. Agreed. I think simple namespace prefixes (maybe hg account?) is the easiest option. I'm not immediately sold on this plan. To me one of the big plus points of having a single Official Tool Shed looked after by the Galaxy team is the convenience factor (a one stop shop), which requires critical mass, plus whatever QA happens as part of the current approval process. I would regard it as a step backwards if in order to hunt for a wrapper for a given tool, I had to resort to Google in order to find all the individual Galaxy Tool Sheds. It'll be possible for people to run their own Tool Sheds if they'd like, for whatever purpose - and this may be necessary for sharing extremely large data which we can't possibly host at the main Shed, but there should be an aggregator somewhere which lists all of the available public Sheds and makes it easy to add them as new sources to your Galaxy install. Like a slightly more organized Debian APT system. If there is an official meta tool shed aggregator, that would address my main concern about fragmenting things. Not sure how feasible this is, but could you use hg subrepositories for this purpose? For instance, have a 'blessed' set of galaxy tool sheds (as subrepos) listed in a main tool shed repository. One of the nice advantages of this is it could allow one to use git or svn, though I think sticking with hg-only repos is the simplest option for now. chris PS - wonderful conference, sorry that Peter couldn't make it! ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
