[galaxy-dev] Galaxy tools' priority level (nice)
Hi all, Our Galaxy is running on a shared server which will sometimes be running other computationally demanding jobs (outside of Galaxy). In some cases I'd like these to have priority, perhaps by having Galaxy run the tool child processes at a nice level of 10 (say). Is there any built in way to control the Unix priority level (e.g. nice or ionice) used to run tasks? I don't see anything on here, but perhaps I'm looking in the wrong place: https://bitbucket.org/galaxy/galaxy-central/wiki/Config/WebApplicationScaling Thanks, Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy tools' priority level (nice)
Hi Peter, Peter Cock wrote, On 04/21/2011 11:00 AM: Is there any built in way to control the Unix priority level (e.g. nice or ionice) used to run tasks? I don't see anything on here, but perhaps I'm looking in the wrong place: https://bitbucket.org/galaxy/galaxy-central/wiki/Config/WebApplicationScaling I've been experimenting with 'nice' as well (for other servers), and it seems the consensus is that 'nice' is broken. Linus puts it in a colorful way here: https://lwn.net/Articles/418739/ [...] Seriously. Nobody _ever_ does nice make, unless they are seriously repressed beta-males (eg MIS people who get shouted at when they do system maintenance unless they hide in dark corners and don't get discovered). It just doesn't happen. One recommended solution is to use cgroups, which can control CPU, Disk I/O, Memory usage and network load, as explained here: http://broadcast.oreilly.com/2009/06/manage-your-performance-with-cgroups-and-projects.html With cgroups there will be no need to change anything in galaxy (just apply a cgroup to the galaxy user, or something similar). Unfortunately, cgroups requires a recent kernel, so it might not be applicable in your case. -gordon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy tools' priority level (nice)
On Thu, Apr 21, 2011 at 5:43 PM, Assaf Gordon gor...@cshl.edu wrote: Hi Peter, Peter Cock wrote, On 04/21/2011 11:00 AM: Is there any built in way to control the Unix priority level (e.g. nice or ionice) used to run tasks? I don't see anything on here, but perhaps I'm looking in the wrong place: https://bitbucket.org/galaxy/galaxy-central/wiki/Config/WebApplicationScaling I've been experimenting with 'nice' as well (for other servers), and it seems the consensus is that 'nice' is broken. Linus puts it in a colorful way here: https://lwn.net/Articles/418739/ [...] Seriously. Nobody _ever_ does nice make, unless they are seriously repressed beta-males (eg MIS people who get shouted at when they do system maintenance unless they hide in dark corners and don't get discovered). It just doesn't happen. I know nice isn't perfect, but in the case of the sys admin setting up Galaxy, we don't have the human laziness problem to overcome: We could ensure all job tasks get run with nice 10 automatically, without the Galaxy users having to do anything special. One recommended solution is to use cgroups, which can control CPU, Disk I/O, Memory usage and network load, as explained here: http://broadcast.oreilly.com/2009/06/manage-your-performance-with-cgroups-and-projects.html With cgroups there will be no need to change anything in galaxy (just apply a cgroup to the galaxy user, or something similar). Yes, but I want to treat the Galaxy webserver differently from the compute jobs it launches. Unfortunately, cgroups requires a recent kernel, so it might not be applicable in your case. -gordon Are you using cgroups on your Galaxy? Peter ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Galaxy tools' priority level (nice)
Peter Cock wrote, On 04/21/2011 12:59 PM: On Thu, Apr 21, 2011 at 5:43 PM, Assaf Gordon gor...@cshl.edu wrote: Hi Peter, Peter Cock wrote, On 04/21/2011 11:00 AM: Is there any built in way to control the Unix priority level (e.g. nice or ionice) used to run tasks? I don't see anything on here, but perhaps I'm looking in the wrong place: https://bitbucket.org/galaxy/galaxy-central/wiki/Config/WebApplicationScaling I've been experimenting with 'nice' as well (for other servers), and it seems the consensus is that 'nice' is broken. Linus puts it in a colorful way here: https://lwn.net/Articles/418739/ [...] Seriously. Nobody _ever_ does nice make, unless they are seriously repressed beta-males (eg MIS people who get shouted at when they do system maintenance unless they hide in dark corners and don't get discovered). It just doesn't happen. I know nice isn't perfect, but in the case of the sys admin setting up Galaxy, we don't have the human laziness problem to overcome: We could ensure all job tasks get run with nice 10 automatically, without the Galaxy users having to do anything special. The article goes to explain further why nice is not the best option, the above quote was just a jest. But you are right that it's better than nothing. One recommended solution is to use cgroups, which can control CPU, Disk I/O, Memory usage and network load, as explained here: http://broadcast.oreilly.com/2009/06/manage-your-performance-with-cgroups-and-projects.html With cgroups there will be no need to change anything in galaxy (just apply a cgroup to the galaxy user, or something similar). Yes, but I want to treat the Galaxy webserver differently from the compute jobs it launches. I'm certain there's a way to set limits per processes, not just per users - but I haven't experimented with it. Unfortunately, cgroups requires a recent kernel, so it might not be applicable in your case. Are you using cgroups on your Galaxy? Only on my development system - unfortunately my production serve uses CentOS 5 with an old kernel and no cgroup support. -gordon ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
