Hi Nate, I wrote the code such that if a machine had a large local scratch space then you could comment it out and it would go to the default setting of writing to /tmp or whatever is defined by SGE. You are correct that the environmental variables defined in the Galaxy startup script do not get passed when running as an actual user. Further I separated TMPDIR from new_files_path because new_files_path is used by Galaxy for other reasons such as uploading files and metadata.
Best, Ilya -----Original Message----- From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] Sent: Tuesday, September 06, 2011 7:21 AM To: Chorny, Ilya Cc: Assaf Gordon; Shantanu Pavgi; galaxydev psu Subject: Re: [galaxy-dev] TEMP variable in cluster install Chorny, Ilya wrote: > Nate, > > We ran into this issue with /tmp not having enough space. We came up with a > generic solution and were wondering if you might want to add it to > galaxy-central. We did not want to modify SGE as it could affect other jobs > that use the cluster. > > We created a TMPDIR option in universe_wsgi.ini and then modified drama.py > to include an "export TMPDIR=TMPDIR" to the drm_tmplate if TMPDIR is defined > in universe_wsgi.ini. Do you want to pull this code into galaxy central. > It's like 5 lines of code. This is also something you can put in the Galaxy user's ~/.sge_request, although with your actual user code this would be more difficult. There's actually already a temp directory option, new_file_path, this would just need to be added to the template. But I don't know that it's a good idea, since each node may have a large local scratch space and the path to that space may differ (this is the case in our environment, where we actually run jobs on multiple clusters). > > Thanks and have a good weekend, > > Ilya > > BTW, any luck reviewing the run as actual user code? I should be doing it today. --nate > > > -----Original Message----- > From: galaxy-dev-boun...@lists.bx.psu.edu > [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Assaf Gordon > Sent: Thursday, August 04, 2011 2:31 PM > To: Shantanu Pavgi; galaxydev psu > Subject: Re: [galaxy-dev] TEMP variable in cluster install > > As a follow-up for SGE + TMP variables, > > I've also encountered problems with settings TEMP,TMP and TMPDIR variables. > Tried setting "~/.sge_request" and changing the Galaxy environment variables > before starting python - nothing worked - TMPDIR and TMP were always set to > "/tmp/all.q.XXXXX" by SGE . > > What finally worked is changing the SGE configuration queue, and simply > setting the "tmpdir" variable to my desired temp directory. > > Do that by running: > $ sudo qconf -mq all.q > > and changing the line: > tmpdir /tmp > to > tmpdir /my/temporary/directory/path > > > Problem solved :) > No more messing around with TMP,TMPDIR variables in any Galaxy related source > files. > > Hope this helps, > -gordon > > > > > Shantanu Pavgi wrote, On 07/26/2011 05:40 PM: > > > > On Jul 21, 2011, at 6:24 PM, Shantanu Pavgi wrote: > > > >> > >> We have configured galaxy to work with our SGE cluster using drmaa > >> job runner interface. We are using 'unified method' for this install and > >> both TEMP environment variable and new_file_path in universe_wsgi.ini file > >> have been configured correctly. However, we are seeing some errors where > >> local /tmp space on compute nodes is being referenced by the galaxy tools. > >> Specifically we saw it mentioned in error messages from following tools: > >> * bwa_wrapper and upload tools: 'No space left on device: /tmp...' > >> * sam_to_bam tool: 'Error sorting alignments from /tmp/filename..' > >> > >> Shouldn't it be referencing TEMP environment variable or new_file_path > >> configuration value? Is it getting overridden by TMP or TMPDIR variables > >> in python wrapper code? Has anyone else experienced similar issue? > >> > >> > > > > Further debugging showed that there are two more temporary directory > > related environment variables - TMP and TMPDIR which were pointing local > > /tmp location on compute nodes. We tried to set these variables in our > > shell environment (drmaa URL uses -V to export current shell env) however > > SGE overwrote TMP and TMPDIR before actual job execution. The TEMP variable > > remained unmodified by SGE scheduler. > > > > The galaxy tools seemed to be using temporary directory space pointed by > > TMP and TMPDIR and hence we saw local /tmp related errors mentioned in my > > earlier post. > > > > We have temporarily fixed this problem by hard coding TEMP, TMP and > > TMPDIR values in job template files - 1. > > lib/galaxy/jobs/runners/drmaa.py 2. lib/galaxy/jobs/runners/sge.py > > > > This will affect all jobs submitted through the galaxy. It will be helpful > > to know how it can be handled in the tool wrapper scripts. It seems like it > > can be handled in the tool wrapper scripts using directory prefix while > > creating the tempfile ( http://docs.python.org/library/tempfile.html ). Any > > thoughts/comments? Are there any other SGE users having similar issue? > > > > Also, there is a similar thread started today regarding usage of temporary > > directory location in tool wrapper scripts. It would be helpful to know how > > galaxy or tool wrappers are using the temporary directory and ways to > > configure it. > > > > -- > > Thanks, > > Shantanu. > > ___________________________________________________________ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this and other > > Galaxy lists, please use the interface at: > > > > http://lists.bx.psu.edu/ > > ___________________________________________________________ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this and other Galaxy > lists, please use the interface at: > > http://lists.bx.psu.edu/ > ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
