On Jul 21, 2011, at 6:24 PM, Shantanu Pavgi wrote:

> We have configured galaxy to work with our SGE cluster using drmaa job runner 
> interface. We are using 'unified method' for this install and 
> both TEMP environment variable and new_file_path in universe_wsgi.ini file 
> have been configured correctly. However, we are seeing some errors where 
> local /tmp space on compute nodes is being referenced by the galaxy tools. 
> Specifically we saw it mentioned in error messages from following tools: 
> * bwa_wrapper and upload tools: 'No space left on device: /tmp...' 
> * sam_to_bam tool: 'Error sorting alignments from /tmp/filename..'
> Shouldn't it be referencing TEMP environment variable or new_file_path 
> configuration value? Is it getting overridden by TMP or TMPDIR variables in 
> python wrapper code? Has anyone else experienced similar issue? 

Further debugging showed that there are two more temporary directory related 
environment variables - TMP and TMPDIR which were pointing local /tmp location 
on compute nodes. We tried to set these variables in our shell environment 
(drmaa URL uses -V to export current shell env) however SGE overwrote TMP and 
TMPDIR before actual job execution. The TEMP variable remained unmodified by 
SGE scheduler.

The galaxy tools seemed to be using temporary directory space pointed by TMP 
and TMPDIR and hence we saw local /tmp related errors mentioned in my earlier 

We have temporarily fixed this problem by hard coding TEMP, TMP and TMPDIR 
values in job template files - 
1. lib/galaxy/jobs/runners/drmaa.py  
2. lib/galaxy/jobs/runners/sge.py  

This will affect all jobs submitted through the galaxy. It will be helpful to 
know how it can be handled in the tool wrapper scripts. It seems like it can be 
handled in the tool wrapper scripts using directory prefix while creating the 
tempfile ( http://docs.python.org/library/tempfile.html ). Any 
thoughts/comments?  Are there any other SGE users having similar issue? 

Also, there is a similar thread started today regarding usage of temporary 
directory location in tool wrapper scripts. It would be helpful to know how 
galaxy or tool wrappers are using the temporary directory and ways to configure 

Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:


Reply via email to