On Jul 21, 2011, at 6:24 PM, Shantanu Pavgi wrote: > > We have configured galaxy to work with our SGE cluster using drmaa job runner > interface. We are using 'unified method' for this install and > both TEMP environment variable and new_file_path in universe_wsgi.ini file > have been configured correctly. However, we are seeing some errors where > local /tmp space on compute nodes is being referenced by the galaxy tools. > Specifically we saw it mentioned in error messages from following tools: > * bwa_wrapper and upload tools: 'No space left on device: /tmp...' > * sam_to_bam tool: 'Error sorting alignments from /tmp/filename..' > > Shouldn't it be referencing TEMP environment variable or new_file_path > configuration value? Is it getting overridden by TMP or TMPDIR variables in > python wrapper code? Has anyone else experienced similar issue? > >
Further debugging showed that there are two more temporary directory related environment variables - TMP and TMPDIR which were pointing local /tmp location on compute nodes. We tried to set these variables in our shell environment (drmaa URL uses -V to export current shell env) however SGE overwrote TMP and TMPDIR before actual job execution. The TEMP variable remained unmodified by SGE scheduler. The galaxy tools seemed to be using temporary directory space pointed by TMP and TMPDIR and hence we saw local /tmp related errors mentioned in my earlier post. We have temporarily fixed this problem by hard coding TEMP, TMP and TMPDIR values in job template files - 1. lib/galaxy/jobs/runners/drmaa.py 2. lib/galaxy/jobs/runners/sge.py This will affect all jobs submitted through the galaxy. It will be helpful to know how it can be handled in the tool wrapper scripts. It seems like it can be handled in the tool wrapper scripts using directory prefix while creating the tempfile ( http://docs.python.org/library/tempfile.html ). Any thoughts/comments? Are there any other SGE users having similar issue? Also, there is a similar thread started today regarding usage of temporary directory location in tool wrapper scripts. It would be helpful to know how galaxy or tool wrappers are using the temporary directory and ways to configure it. -- Thanks, Shantanu. ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
