On 11/12/2014 02:32 AM, Reuti wrote:
Am 12.11.2014 um 00:09 schrieb Christopher Heiny:
On 11/10/2014 11:21 PM, Reuti wrote:
Am 11.11.2014 um 06:37 schrieb Christopher Heiny:
Hi all,
We're running OGS 2011.11p1, with a 1:1 slot:core mapping. We mostly run submit single
threaded array jobs, one slot per task. We now want to multi-thread some of the programs
in order to speed up processing, but need a way to allocate <n> slots to an <n>
threaded task.
The first idea was to use a parallel environment to manage this, but since the
jobs are all with script (rather than a binary), we got bitten by this bug:
http://gridengine.org/pipermail/dev/2011-December/000081.html
Nevertheless you can submit the jobscript as binary, it might be necessary to
specify the complete path to the script though.
If you do that, GE will ignore all the #$ escapes in the script.
That's true.
An elaborate task could be to write a JSV to read the file and add all the
extra SGE options found therein to the submission command, hence no change for
the user.
That seems a little more (OK, a lot more) elaborate than I wanted to get
into, especially with an upgrade in the near future. The attached
Python wrapper works fine for us, since all the qsub options are escaped
within the script anyway.
Cheers,
Chris
#!/usr/bin/python
#
# Wrapper for submitting qsub array tasks.
import subprocess;
import sys;
import os;
def usage(exit_code):
sys.stdout.write("Usage: \n");
sys.stdout.write(" qwrap <script>\n");
sys.exit(exit_code);
if len(sys.argv) != 2:
sys.stderr.write("ERROR: Incorrect number of arguments\n");
usage(1);
script = sys.argv[1];
f = open(script, "r");
lines = f.readlines();
f.close();
escapes=[]
for l in lines:
if l.startswith("#$"):
escapes.append(l.replace("#$", "").strip());
command_line = ["qsub"];
for e in escapes:
command_line = command_line + e.split();
command_line = command_line + ["-b", "y", script];
try:
subprocess.check_call(command_line);
except subprocess.CalledProcessError, e:
sys.stderr.write("ERROR: Command submission failed with error code: %d\n", e.returncode);
sys.stderr.write("Command was: %s\n", (" ".join(command_line)));
sys.exit(e.returncode);
_______________________________________________
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users
