Re: [galaxy-dev] Shell script to start Galaxy in multi-server environment
On 8/8/12 4:06 PM, "Nate Coraor" wrote: >On Aug 8, 2012, at 2:30 PM, Karger, Amir wrote: > >>> >> >> Meanwhile, we're able to restart, and get happy log messages from the >>jobrunner and two web "servers" (two servers running on different ports >>of a Tomcat host). And I can do an upload, which runs locally. But when >>I try to do a blast, which is supposed to submit to the cluster (and ran >>just fine on our old install), it hangs and never starts. I would think >>the database is working OK, since it shows me new history items when I >>upload and stuff. The web Galaxy log shows that I went to the tool page, >>and then has a ton of loads to root/history_item_updates, but nothing >>else. The job handler Galaxy log has nothing since the PID messages when >>the server started up most recently. >> >> A quick search of the archives didn't find anything obvious. (I don't >>have any obvious words to search for.) Any thoughts about where I should >>start looking to track this down? > >Hi Amir, > >If you aren't setting job_manager and job_handlers in your config, each >server will consider itself the manager and handler. If not configured >to run jobs, this may result in jobs failing to run. I'd suggest >explicitly defining a manager and handlers. > >--nate Sigh. We have both job_manager and job_handlers set to the same server. It seems like our runner app may be getting into some kind of sleeping state. I was unable to upload a file, which had worked before. However, when I restarted the runner, it picked up the upload job and successfully uploaded it AND picked up the previously queued tab2fasta job, and I believe completed it successfully too. (There's an error due to a missing filetype, which I guess makes stderr non-empty and makes Galaxy think it was unsuccessful. But I can confirm that the job was in fact run on our cluster.) Running paster.py ... --status claims that the process is still running. So what would make the runner go to "sleep" like that and how do I stop it from happening? Thanks, -Amir ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Shell script to start Galaxy in multi-server environment
On Aug 8, 2012, at 2:30 PM, Karger, Amir wrote: >> From: Nate Coraor [mailto:n...@bx.psu.edu] >> On Aug 2, 2012, at 2:56 PM, Karger, Amir wrote: >> >>> We're upgrading to a late June Galaxy from a last-year Galaxy. We noticed >>> that the docs say you no longer need 2 different .ini files. Great! >>> Unfortunately, the multiprocess.sh in contrib/ still assumes you have >>> multiple .ini files. >> multiprocess.sh is out of date, so I've removed it from galaxy-central. >> run.sh >> can start and stop all of your processes now, as described at: >> >> http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application >> %20Scaling > > Thanks. Of course, reading some other people's posts and the wiki, it looks > like it's not *required* to merge, just recommended. Which means our existing > system of running the different scripts on different hosts should continue to > work. We figure we can put off the merge thing for a bit. > > Meanwhile, we're able to restart, and get happy log messages from the > jobrunner and two web "servers" (two servers running on different ports of a > Tomcat host). And I can do an upload, which runs locally. But when I try to > do a blast, which is supposed to submit to the cluster (and ran just fine on > our old install), it hangs and never starts. I would think the database is > working OK, since it shows me new history items when I upload and stuff. The > web Galaxy log shows that I went to the tool page, and then has a ton of > loads to root/history_item_updates, but nothing else. The job handler Galaxy > log has nothing since the PID messages when the server started up most > recently. > > A quick search of the archives didn't find anything obvious. (I don't have > any obvious words to search for.) Any thoughts about where I should start > looking to track this down? Hi Amir, If you aren't setting job_manager and job_handlers in your config, each server will consider itself the manager and handler. If not configured to run jobs, this may result in jobs failing to run. I'd suggest explicitly defining a manager and handlers. --nate > > Thanks, > > -Amir > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Shell script to start Galaxy in multi-server environment
> From: Nate Coraor [mailto:n...@bx.psu.edu] > On Aug 2, 2012, at 2:56 PM, Karger, Amir wrote: > > > We're upgrading to a late June Galaxy from a last-year Galaxy. We noticed > > that the docs say you no longer need 2 different .ini files. Great! > > Unfortunately, the multiprocess.sh in contrib/ still assumes you have > > multiple .ini files. > multiprocess.sh is out of date, so I've removed it from galaxy-central. > run.sh > can start and stop all of your processes now, as described at: > > http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application > %20Scaling Thanks. Of course, reading some other people's posts and the wiki, it looks like it's not *required* to merge, just recommended. Which means our existing system of running the different scripts on different hosts should continue to work. We figure we can put off the merge thing for a bit. Meanwhile, we're able to restart, and get happy log messages from the jobrunner and two web "servers" (two servers running on different ports of a Tomcat host). And I can do an upload, which runs locally. But when I try to do a blast, which is supposed to submit to the cluster (and ran just fine on our old install), it hangs and never starts. I would think the database is working OK, since it shows me new history items when I upload and stuff. The web Galaxy log shows that I went to the tool page, and then has a ton of loads to root/history_item_updates, but nothing else. The job handler Galaxy log has nothing since the PID messages when the server started up most recently. A quick search of the archives didn't find anything obvious. (I don't have any obvious words to search for.) Any thoughts about where I should start looking to track this down? Thanks, -Amir ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Shell script to start Galaxy in multi-server environment
On Aug 2, 2012, at 2:56 PM, Karger, Amir wrote: > We're upgrading to a late June Galaxy from a last-year Galaxy. We noticed > that the docs say you no longer need 2 different .ini files. Great! > Unfortunately, the multiprocess.sh in contrib/ still assumes you have > multiple .ini files. > > So the question is, assuming we correctly set up the different web > servers, job managers, and job handlers servers in universe_wsgi.ini, > what's the command line we should be giving to run Galaxy on each type of > server? The wiki Admin/Config pages for Performance and Scaling and > Cluster and that sort of thing had some info on editing the .ini, but I > didn't see what my .sh should look like there. Pointers to websites, > emails, or existing .sh files appreciated. Hi Amir, multiprocess.sh is out of date, so I've removed it from galaxy-central. run.sh can start and stop all of your processes now, as described at: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application%20Scaling --nate > > Thanks, > > -Amir Karger > Senior Research Computing Consultant > Harvard Medical School Research Computing > amir_kar...@hms.harvard.edu > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Shell script to start Galaxy in multi-server environment
The docs about that are here: http://wiki.g2.bx.psu.edu/Admin/Config/Performance/Web%20Application%20Scaling I am working on combining our separate ini files myself, but ran into some odd problems with cluster job submission that weren't present with the original split files (e.g. the combined one didn't work, but the split one did). I haven't managed to debug that just yet, and won't be able to until I get back from vacation in a week. chris On Aug 2, 2012, at 1:56 PM, "Karger, Amir" wrote: > We're upgrading to a late June Galaxy from a last-year Galaxy. We noticed > that the docs say you no longer need 2 different .ini files. Great! > Unfortunately, the multiprocess.sh in contrib/ still assumes you have > multiple .ini files. > > So the question is, assuming we correctly set up the different web > servers, job managers, and job handlers servers in universe_wsgi.ini, > what's the command line we should be giving to run Galaxy on each type of > server? The wiki Admin/Config pages for Performance and Scaling and > Cluster and that sort of thing had some info on editing the .ini, but I > didn't see what my .sh should look like there. Pointers to websites, > emails, or existing .sh files appreciated. > > Thanks, > > -Amir Karger > Senior Research Computing Consultant > Harvard Medical School Research Computing > amir_kar...@hms.harvard.edu > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Shell script to start Galaxy in multi-server environment
We're upgrading to a late June Galaxy from a last-year Galaxy. We noticed that the docs say you no longer need 2 different .ini files. Great! Unfortunately, the multiprocess.sh in contrib/ still assumes you have multiple .ini files. So the question is, assuming we correctly set up the different web servers, job managers, and job handlers servers in universe_wsgi.ini, what's the command line we should be giving to run Galaxy on each type of server? The wiki Admin/Config pages for Performance and Scaling and Cluster and that sort of thing had some info on editing the .ini, but I didn't see what my .sh should look like there. Pointers to websites, emails, or existing .sh files appreciated. Thanks, -Amir Karger Senior Research Computing Consultant Harvard Medical School Research Computing amir_kar...@hms.harvard.edu ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
