Re: [galaxy-dev] Uploading a Directory of Files - IOError: [Errno 28] No space left on device
On Feb 7, 2013, at 2:29 PM, greg wrote: > I think I found the problem. The TMPDIR environment variable was set > to /tmp/5393732.1.f03.q for jobs galaxy was running. (I guess the > admins do this?) SGE does this to create a job-specific temporary directory that it can clean up after each job terminates. > I updated /usr/local/galaxy/job_environment_setup_file and also > /home/galaxy/.bashrc to set TMPDIR to /scratch/galaxy and it seems to > work now. Great, thanks for letting us know. --nate > > Thanks for the help. > > -Greg > > On Thu, Feb 7, 2013 at 1:19 PM, greg wrote: >> Could I modify /misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py >> to print out debug information like host, os.environ, >> tempfile.gettempdir(), etc? >> >> Would I be able to see its stdout from galaxy or the log, or is there >> something special I need to do to retrieve the information? >> >> On Thu, Feb 7, 2013 at 1:12 PM, greg wrote: >>> Update: >>> >>> When I run as the Galaxy user, Python does have the right temp directory: >>> >> tempfile.gettempdir() >>> '/scratch/galaxy' >>> >>> So does that mean this upload job isn't running as galaxy, or is >>> skipping the job_environment_setup_file? Or could something else be >>> going on? >>> >>> Any ideas, now I'm really stuck. >>> >>> Thanks, >>> >>> Greg >>> >>> >>> >>> On Wed, Feb 6, 2013 at 3:35 PM, greg wrote: Ok, when I ran Python in my last two emails I was running as myself, not the galaxy user, and only the galaxy user has write permission to /scratch/galaxy So that's why Python was ignoring /scratch/galaxy for me. If it doesn't have write access it tries the next temp directory in its list. I'm going to try debugging as the galaxy user next. -Greg On Wed, Feb 6, 2013 at 3:21 PM, greg wrote: > Hi Nate, > > I don't see $TMPDIR being set on the cluster, in addition to my > previous email I ran: > print os.environ.keys() > ['KDE_IS_PRELINKED', 'FACTERLIB', 'LESSOPEN', 'SGE_CELL', 'LOGNAME', > 'USER', 'INPUTRC', 'QTDIR', 'PATH', 'PS1', 'LANG', 'KDEDIR', 'TERM', > 'SHELL', 'TEMP', 'QTINC', 'G_BROKEN_FILENAMES', 'SGE_EXECD_PORT', > 'HISTSIZE', 'KDE_NO_IPV6', 'MANPATH', 'HOME', 'SGE_ROOT', 'QTLIB', > 'VIRTUAL_ENV', 'SGE_CLUSTER_NAME', '_', 'SSH_CONNECTION', 'SSH_TTY', > 'HOSTNAME', 'SSH_CLIENT', 'SHLVL', 'PWD', 'MAIL', 'LS_COLORS', > 'SGE_QMASTER_PORT'] > > But I think we've narrowed it down to something interfering with > Python deciding the temp file location. I just can't figure out what. > > > > On Wed, Feb 6, 2013 at 3:18 PM, Nate Coraor wrote: >> On Feb 6, 2013, at 3:00 PM, greg wrote: >> >>> Thanks Nate, >>> >>> It turns out I already had this as the first line of my job setup file: >>> >>> export TEMP=/scratch/galaxy >>> >>> But when I look in that directory, there's plenty of free space, and I >>> also don't see any recent files there. So I'm wondering if the upload >>> jobs aren't seeing that for some reason. >>> >>> Any ideas on how I could diagnose this more? >> >> Hi Greg, >> >> The first place to look would be in lib/galaxy/datatypes/sniff.py, line >> 96: >> >>fd, temp_name = tempfile.mkstemp() >> >> If you print temp_name, that will tell you what file the upload tool is >> writing to. You may also want to take a look at: >> >>http://docs.python.org/2/library/tempfile.html#tempfile.tempdir >> >> Some cluster environments set $TMPDIR, and if that is set, $TEMP will >> not be used. >> >> --nate >> >>> >>> -Greg >>> >>> >>> Relevant info? >>> >>> grep env galaxy-dist/universe_wsgi.ini >>> environment_setup_file = /usr/local/galaxy/job_environment_setup_file >>> cat /usr/local/galaxy/job_environment_setup_file >>> export TEMP=/scratch/galaxy >>> #active Python virtual env just for galaxy >>> source /usr/local/galaxy/galaxy_python/bin/activate >>> ... path setup lines ... >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor wrote: On Feb 6, 2013, at 2:32 PM, greg wrote: > Hi guys, > > When I try to upload a directory of files from a server directory I'm > seeing the error below. > > It appears to be trying to write to a temp directory somewhere that > I'm guessing doesn't have enough space? Is there a way I can direct > where it writes to for temporary files like this? Hi Greg, There are a few ways. For some parts of Galaxy, you will want to set new_file_path in universe_wsgi.ini to a suitable temp space. However, this
Re: [galaxy-dev] Uploading a Directory of Files - IOError: [Errno 28] No space left on device
I think I found the problem. The TMPDIR environment variable was set to /tmp/5393732.1.f03.q for jobs galaxy was running. (I guess the admins do this?) I updated /usr/local/galaxy/job_environment_setup_file and also /home/galaxy/.bashrc to set TMPDIR to /scratch/galaxy and it seems to work now. Thanks for the help. -Greg On Thu, Feb 7, 2013 at 1:19 PM, greg wrote: > Could I modify /misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py > to print out debug information like host, os.environ, > tempfile.gettempdir(), etc? > > Would I be able to see its stdout from galaxy or the log, or is there > something special I need to do to retrieve the information? > > On Thu, Feb 7, 2013 at 1:12 PM, greg wrote: >> Update: >> >> When I run as the Galaxy user, Python does have the right temp directory: >> > tempfile.gettempdir() >> '/scratch/galaxy' >> >> So does that mean this upload job isn't running as galaxy, or is >> skipping the job_environment_setup_file? Or could something else be >> going on? >> >> Any ideas, now I'm really stuck. >> >> Thanks, >> >> Greg >> >> >> >> On Wed, Feb 6, 2013 at 3:35 PM, greg wrote: >>> Ok, when I ran Python in my last two emails I was running as myself, >>> not the galaxy user, and only the galaxy user has write permission to >>> /scratch/galaxy >>> >>> So that's why Python was ignoring /scratch/galaxy for me. If it >>> doesn't have write access it tries the next temp directory in its >>> list. >>> >>> I'm going to try debugging as the galaxy user next. >>> >>> -Greg >>> >>> On Wed, Feb 6, 2013 at 3:21 PM, greg wrote: Hi Nate, I don't see $TMPDIR being set on the cluster, in addition to my previous email I ran: >>> print os.environ.keys() ['KDE_IS_PRELINKED', 'FACTERLIB', 'LESSOPEN', 'SGE_CELL', 'LOGNAME', 'USER', 'INPUTRC', 'QTDIR', 'PATH', 'PS1', 'LANG', 'KDEDIR', 'TERM', 'SHELL', 'TEMP', 'QTINC', 'G_BROKEN_FILENAMES', 'SGE_EXECD_PORT', 'HISTSIZE', 'KDE_NO_IPV6', 'MANPATH', 'HOME', 'SGE_ROOT', 'QTLIB', 'VIRTUAL_ENV', 'SGE_CLUSTER_NAME', '_', 'SSH_CONNECTION', 'SSH_TTY', 'HOSTNAME', 'SSH_CLIENT', 'SHLVL', 'PWD', 'MAIL', 'LS_COLORS', 'SGE_QMASTER_PORT'] But I think we've narrowed it down to something interfering with Python deciding the temp file location. I just can't figure out what. On Wed, Feb 6, 2013 at 3:18 PM, Nate Coraor wrote: > On Feb 6, 2013, at 3:00 PM, greg wrote: > >> Thanks Nate, >> >> It turns out I already had this as the first line of my job setup file: >> >> export TEMP=/scratch/galaxy >> >> But when I look in that directory, there's plenty of free space, and I >> also don't see any recent files there. So I'm wondering if the upload >> jobs aren't seeing that for some reason. >> >> Any ideas on how I could diagnose this more? > > Hi Greg, > > The first place to look would be in lib/galaxy/datatypes/sniff.py, line > 96: > > fd, temp_name = tempfile.mkstemp() > > If you print temp_name, that will tell you what file the upload tool is > writing to. You may also want to take a look at: > > http://docs.python.org/2/library/tempfile.html#tempfile.tempdir > > Some cluster environments set $TMPDIR, and if that is set, $TEMP will not > be used. > > --nate > >> >> -Greg >> >> >> Relevant info? >> >> >>> grep env galaxy-dist/universe_wsgi.ini >> environment_setup_file = /usr/local/galaxy/job_environment_setup_file >> >>> cat /usr/local/galaxy/job_environment_setup_file >> export TEMP=/scratch/galaxy >> #active Python virtual env just for galaxy >> source /usr/local/galaxy/galaxy_python/bin/activate >> ... path setup lines ... >> >> >> >> >> >> >> >> >> >> >> On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor wrote: >>> On Feb 6, 2013, at 2:32 PM, greg wrote: >>> Hi guys, When I try to upload a directory of files from a server directory I'm seeing the error below. It appears to be trying to write to a temp directory somewhere that I'm guessing doesn't have enough space? Is there a way I can direct where it writes to for temporary files like this? >>> >>> Hi Greg, >>> >>> There are a few ways. For some parts of Galaxy, you will want to set >>> new_file_path in universe_wsgi.ini to a suitable temp space. However, >>> this is not the case for the upload tool. >>> Am I understanding right, that these upload jobs are running on our cluster? I think it would be a problem if its trying to use the default temp directory on each cluster node since they aren't provisioned with much space. >>> >>> This is correct. On the cluster, add something to your user's shell >>> startup
Re: [galaxy-dev] Uploading a Directory of Files - IOError: [Errno 28] No space left on device
Could I modify /misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py to print out debug information like host, os.environ, tempfile.gettempdir(), etc? Would I be able to see its stdout from galaxy or the log, or is there something special I need to do to retrieve the information? On Thu, Feb 7, 2013 at 1:12 PM, greg wrote: > Update: > > When I run as the Galaxy user, Python does have the right temp directory: > tempfile.gettempdir() > '/scratch/galaxy' > > So does that mean this upload job isn't running as galaxy, or is > skipping the job_environment_setup_file? Or could something else be > going on? > > Any ideas, now I'm really stuck. > > Thanks, > > Greg > > > > On Wed, Feb 6, 2013 at 3:35 PM, greg wrote: >> Ok, when I ran Python in my last two emails I was running as myself, >> not the galaxy user, and only the galaxy user has write permission to >> /scratch/galaxy >> >> So that's why Python was ignoring /scratch/galaxy for me. If it >> doesn't have write access it tries the next temp directory in its >> list. >> >> I'm going to try debugging as the galaxy user next. >> >> -Greg >> >> On Wed, Feb 6, 2013 at 3:21 PM, greg wrote: >>> Hi Nate, >>> >>> I don't see $TMPDIR being set on the cluster, in addition to my >>> previous email I ran: >>> >> print os.environ.keys() >>> ['KDE_IS_PRELINKED', 'FACTERLIB', 'LESSOPEN', 'SGE_CELL', 'LOGNAME', >>> 'USER', 'INPUTRC', 'QTDIR', 'PATH', 'PS1', 'LANG', 'KDEDIR', 'TERM', >>> 'SHELL', 'TEMP', 'QTINC', 'G_BROKEN_FILENAMES', 'SGE_EXECD_PORT', >>> 'HISTSIZE', 'KDE_NO_IPV6', 'MANPATH', 'HOME', 'SGE_ROOT', 'QTLIB', >>> 'VIRTUAL_ENV', 'SGE_CLUSTER_NAME', '_', 'SSH_CONNECTION', 'SSH_TTY', >>> 'HOSTNAME', 'SSH_CLIENT', 'SHLVL', 'PWD', 'MAIL', 'LS_COLORS', >>> 'SGE_QMASTER_PORT'] >>> >>> But I think we've narrowed it down to something interfering with >>> Python deciding the temp file location. I just can't figure out what. >>> >>> >>> >>> On Wed, Feb 6, 2013 at 3:18 PM, Nate Coraor wrote: On Feb 6, 2013, at 3:00 PM, greg wrote: > Thanks Nate, > > It turns out I already had this as the first line of my job setup file: > > export TEMP=/scratch/galaxy > > But when I look in that directory, there's plenty of free space, and I > also don't see any recent files there. So I'm wondering if the upload > jobs aren't seeing that for some reason. > > Any ideas on how I could diagnose this more? Hi Greg, The first place to look would be in lib/galaxy/datatypes/sniff.py, line 96: fd, temp_name = tempfile.mkstemp() If you print temp_name, that will tell you what file the upload tool is writing to. You may also want to take a look at: http://docs.python.org/2/library/tempfile.html#tempfile.tempdir Some cluster environments set $TMPDIR, and if that is set, $TEMP will not be used. --nate > > -Greg > > > Relevant info? > > >> grep env galaxy-dist/universe_wsgi.ini > environment_setup_file = /usr/local/galaxy/job_environment_setup_file > >> cat /usr/local/galaxy/job_environment_setup_file > export TEMP=/scratch/galaxy > #active Python virtual env just for galaxy > source /usr/local/galaxy/galaxy_python/bin/activate > ... path setup lines ... > > > > > > > > > > > On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor wrote: >> On Feb 6, 2013, at 2:32 PM, greg wrote: >> >>> Hi guys, >>> >>> When I try to upload a directory of files from a server directory I'm >>> seeing the error below. >>> >>> It appears to be trying to write to a temp directory somewhere that >>> I'm guessing doesn't have enough space? Is there a way I can direct >>> where it writes to for temporary files like this? >> >> Hi Greg, >> >> There are a few ways. For some parts of Galaxy, you will want to set >> new_file_path in universe_wsgi.ini to a suitable temp space. However, >> this is not the case for the upload tool. >> >>> Am I understanding right, that these upload jobs are running on our >>> cluster? I think it would be a problem if its trying to use the >>> default temp directory on each cluster node since they aren't >>> provisioned with much space. >> >> This is correct. On the cluster, add something to your user's shell >> startup files (or see the environment_setup_file option in >> universe_wsgi.ini) that will set the $TEMP or $TMPDIR environment >> variable to a suitable temp space. >> >> --nate >> >>> >>> Please advise. >>> >>> Thanks, >>> >>> Greg >>> >>> >>> >>> Miscellaneous information:Traceback (most recent call last): File >>> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >>> 384, in __main__() File >>> "/misc/local/galaxy/galaxy-dist/tools/dat
Re: [galaxy-dev] Uploading a Directory of Files - IOError: [Errno 28] No space left on device
Update: When I run as the Galaxy user, Python does have the right temp directory: >>> tempfile.gettempdir() '/scratch/galaxy' So does that mean this upload job isn't running as galaxy, or is skipping the job_environment_setup_file? Or could something else be going on? Any ideas, now I'm really stuck. Thanks, Greg On Wed, Feb 6, 2013 at 3:35 PM, greg wrote: > Ok, when I ran Python in my last two emails I was running as myself, > not the galaxy user, and only the galaxy user has write permission to > /scratch/galaxy > > So that's why Python was ignoring /scratch/galaxy for me. If it > doesn't have write access it tries the next temp directory in its > list. > > I'm going to try debugging as the galaxy user next. > > -Greg > > On Wed, Feb 6, 2013 at 3:21 PM, greg wrote: >> Hi Nate, >> >> I don't see $TMPDIR being set on the cluster, in addition to my >> previous email I ran: >> > print os.environ.keys() >> ['KDE_IS_PRELINKED', 'FACTERLIB', 'LESSOPEN', 'SGE_CELL', 'LOGNAME', >> 'USER', 'INPUTRC', 'QTDIR', 'PATH', 'PS1', 'LANG', 'KDEDIR', 'TERM', >> 'SHELL', 'TEMP', 'QTINC', 'G_BROKEN_FILENAMES', 'SGE_EXECD_PORT', >> 'HISTSIZE', 'KDE_NO_IPV6', 'MANPATH', 'HOME', 'SGE_ROOT', 'QTLIB', >> 'VIRTUAL_ENV', 'SGE_CLUSTER_NAME', '_', 'SSH_CONNECTION', 'SSH_TTY', >> 'HOSTNAME', 'SSH_CLIENT', 'SHLVL', 'PWD', 'MAIL', 'LS_COLORS', >> 'SGE_QMASTER_PORT'] >> >> But I think we've narrowed it down to something interfering with >> Python deciding the temp file location. I just can't figure out what. >> >> >> >> On Wed, Feb 6, 2013 at 3:18 PM, Nate Coraor wrote: >>> On Feb 6, 2013, at 3:00 PM, greg wrote: >>> Thanks Nate, It turns out I already had this as the first line of my job setup file: export TEMP=/scratch/galaxy But when I look in that directory, there's plenty of free space, and I also don't see any recent files there. So I'm wondering if the upload jobs aren't seeing that for some reason. Any ideas on how I could diagnose this more? >>> >>> Hi Greg, >>> >>> The first place to look would be in lib/galaxy/datatypes/sniff.py, line 96: >>> >>> fd, temp_name = tempfile.mkstemp() >>> >>> If you print temp_name, that will tell you what file the upload tool is >>> writing to. You may also want to take a look at: >>> >>> http://docs.python.org/2/library/tempfile.html#tempfile.tempdir >>> >>> Some cluster environments set $TMPDIR, and if that is set, $TEMP will not >>> be used. >>> >>> --nate >>> -Greg Relevant info? > grep env galaxy-dist/universe_wsgi.ini environment_setup_file = /usr/local/galaxy/job_environment_setup_file > cat /usr/local/galaxy/job_environment_setup_file export TEMP=/scratch/galaxy #active Python virtual env just for galaxy source /usr/local/galaxy/galaxy_python/bin/activate ... path setup lines ... On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor wrote: > On Feb 6, 2013, at 2:32 PM, greg wrote: > >> Hi guys, >> >> When I try to upload a directory of files from a server directory I'm >> seeing the error below. >> >> It appears to be trying to write to a temp directory somewhere that >> I'm guessing doesn't have enough space? Is there a way I can direct >> where it writes to for temporary files like this? > > Hi Greg, > > There are a few ways. For some parts of Galaxy, you will want to set > new_file_path in universe_wsgi.ini to a suitable temp space. However, > this is not the case for the upload tool. > >> Am I understanding right, that these upload jobs are running on our >> cluster? I think it would be a problem if its trying to use the >> default temp directory on each cluster node since they aren't >> provisioned with much space. > > This is correct. On the cluster, add something to your user's shell > startup files (or see the environment_setup_file option in > universe_wsgi.ini) that will set the $TEMP or $TMPDIR environment > variable to a suitable temp space. > > --nate > >> >> Please advise. >> >> Thanks, >> >> Greg >> >> >> >> Miscellaneous information:Traceback (most recent call last): File >> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >> 384, in __main__() File >> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >> 373, in __main__ add_file( dataset, >> Job Standard Error >> >> >> Traceback (most recent call last): >> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >> line 384, in >> __main__() >> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >> line 373, in __main__ >> add_file( dataset, registry, json_file, output_path ) >> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upl
Re: [galaxy-dev] Uploading a Directory of Files - IOError: [Errno 28] No space left on device
Ok, when I ran Python in my last two emails I was running as myself, not the galaxy user, and only the galaxy user has write permission to /scratch/galaxy So that's why Python was ignoring /scratch/galaxy for me. If it doesn't have write access it tries the next temp directory in its list. I'm going to try debugging as the galaxy user next. -Greg On Wed, Feb 6, 2013 at 3:21 PM, greg wrote: > Hi Nate, > > I don't see $TMPDIR being set on the cluster, in addition to my > previous email I ran: > print os.environ.keys() > ['KDE_IS_PRELINKED', 'FACTERLIB', 'LESSOPEN', 'SGE_CELL', 'LOGNAME', > 'USER', 'INPUTRC', 'QTDIR', 'PATH', 'PS1', 'LANG', 'KDEDIR', 'TERM', > 'SHELL', 'TEMP', 'QTINC', 'G_BROKEN_FILENAMES', 'SGE_EXECD_PORT', > 'HISTSIZE', 'KDE_NO_IPV6', 'MANPATH', 'HOME', 'SGE_ROOT', 'QTLIB', > 'VIRTUAL_ENV', 'SGE_CLUSTER_NAME', '_', 'SSH_CONNECTION', 'SSH_TTY', > 'HOSTNAME', 'SSH_CLIENT', 'SHLVL', 'PWD', 'MAIL', 'LS_COLORS', > 'SGE_QMASTER_PORT'] > > But I think we've narrowed it down to something interfering with > Python deciding the temp file location. I just can't figure out what. > > > > On Wed, Feb 6, 2013 at 3:18 PM, Nate Coraor wrote: >> On Feb 6, 2013, at 3:00 PM, greg wrote: >> >>> Thanks Nate, >>> >>> It turns out I already had this as the first line of my job setup file: >>> >>> export TEMP=/scratch/galaxy >>> >>> But when I look in that directory, there's plenty of free space, and I >>> also don't see any recent files there. So I'm wondering if the upload >>> jobs aren't seeing that for some reason. >>> >>> Any ideas on how I could diagnose this more? >> >> Hi Greg, >> >> The first place to look would be in lib/galaxy/datatypes/sniff.py, line 96: >> >> fd, temp_name = tempfile.mkstemp() >> >> If you print temp_name, that will tell you what file the upload tool is >> writing to. You may also want to take a look at: >> >> http://docs.python.org/2/library/tempfile.html#tempfile.tempdir >> >> Some cluster environments set $TMPDIR, and if that is set, $TEMP will not be >> used. >> >> --nate >> >>> >>> -Greg >>> >>> >>> Relevant info? >>> >>> grep env galaxy-dist/universe_wsgi.ini >>> environment_setup_file = /usr/local/galaxy/job_environment_setup_file >>> cat /usr/local/galaxy/job_environment_setup_file >>> export TEMP=/scratch/galaxy >>> #active Python virtual env just for galaxy >>> source /usr/local/galaxy/galaxy_python/bin/activate >>> ... path setup lines ... >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor wrote: On Feb 6, 2013, at 2:32 PM, greg wrote: > Hi guys, > > When I try to upload a directory of files from a server directory I'm > seeing the error below. > > It appears to be trying to write to a temp directory somewhere that > I'm guessing doesn't have enough space? Is there a way I can direct > where it writes to for temporary files like this? Hi Greg, There are a few ways. For some parts of Galaxy, you will want to set new_file_path in universe_wsgi.ini to a suitable temp space. However, this is not the case for the upload tool. > Am I understanding right, that these upload jobs are running on our > cluster? I think it would be a problem if its trying to use the > default temp directory on each cluster node since they aren't > provisioned with much space. This is correct. On the cluster, add something to your user's shell startup files (or see the environment_setup_file option in universe_wsgi.ini) that will set the $TEMP or $TMPDIR environment variable to a suitable temp space. --nate > > Please advise. > > Thanks, > > Greg > > > > Miscellaneous information:Traceback (most recent call last): File > "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line > 384, in __main__() File > "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line > 373, in __main__ add_file( dataset, > Job Standard Error > > > Traceback (most recent call last): > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 384, in > __main__() > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 373, in __main__ > add_file( dataset, registry, json_file, output_path ) > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 270, in add_file > line_count, converted_path = sniff.convert_newlines( dataset.path, > in_place=in_place ) > File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py", > line 99, in convert_newlines > fp.write( "%s\n" % line.rstrip( "\r\n" ) ) > IOError: [Errno 28] No space left on device > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your s
Re: [galaxy-dev] Uploading a Directory of Files - IOError: [Errno 28] No space left on device
Hi Nate, I don't see $TMPDIR being set on the cluster, in addition to my previous email I ran: >>> print os.environ.keys() ['KDE_IS_PRELINKED', 'FACTERLIB', 'LESSOPEN', 'SGE_CELL', 'LOGNAME', 'USER', 'INPUTRC', 'QTDIR', 'PATH', 'PS1', 'LANG', 'KDEDIR', 'TERM', 'SHELL', 'TEMP', 'QTINC', 'G_BROKEN_FILENAMES', 'SGE_EXECD_PORT', 'HISTSIZE', 'KDE_NO_IPV6', 'MANPATH', 'HOME', 'SGE_ROOT', 'QTLIB', 'VIRTUAL_ENV', 'SGE_CLUSTER_NAME', '_', 'SSH_CONNECTION', 'SSH_TTY', 'HOSTNAME', 'SSH_CLIENT', 'SHLVL', 'PWD', 'MAIL', 'LS_COLORS', 'SGE_QMASTER_PORT'] But I think we've narrowed it down to something interfering with Python deciding the temp file location. I just can't figure out what. On Wed, Feb 6, 2013 at 3:18 PM, Nate Coraor wrote: > On Feb 6, 2013, at 3:00 PM, greg wrote: > >> Thanks Nate, >> >> It turns out I already had this as the first line of my job setup file: >> >> export TEMP=/scratch/galaxy >> >> But when I look in that directory, there's plenty of free space, and I >> also don't see any recent files there. So I'm wondering if the upload >> jobs aren't seeing that for some reason. >> >> Any ideas on how I could diagnose this more? > > Hi Greg, > > The first place to look would be in lib/galaxy/datatypes/sniff.py, line 96: > > fd, temp_name = tempfile.mkstemp() > > If you print temp_name, that will tell you what file the upload tool is > writing to. You may also want to take a look at: > > http://docs.python.org/2/library/tempfile.html#tempfile.tempdir > > Some cluster environments set $TMPDIR, and if that is set, $TEMP will not be > used. > > --nate > >> >> -Greg >> >> >> Relevant info? >> >> >>> grep env galaxy-dist/universe_wsgi.ini >> environment_setup_file = /usr/local/galaxy/job_environment_setup_file >> >>> cat /usr/local/galaxy/job_environment_setup_file >> export TEMP=/scratch/galaxy >> #active Python virtual env just for galaxy >> source /usr/local/galaxy/galaxy_python/bin/activate >> ... path setup lines ... >> >> >> >> >> >> >> >> >> >> >> On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor wrote: >>> On Feb 6, 2013, at 2:32 PM, greg wrote: >>> Hi guys, When I try to upload a directory of files from a server directory I'm seeing the error below. It appears to be trying to write to a temp directory somewhere that I'm guessing doesn't have enough space? Is there a way I can direct where it writes to for temporary files like this? >>> >>> Hi Greg, >>> >>> There are a few ways. For some parts of Galaxy, you will want to set >>> new_file_path in universe_wsgi.ini to a suitable temp space. However, this >>> is not the case for the upload tool. >>> Am I understanding right, that these upload jobs are running on our cluster? I think it would be a problem if its trying to use the default temp directory on each cluster node since they aren't provisioned with much space. >>> >>> This is correct. On the cluster, add something to your user's shell >>> startup files (or see the environment_setup_file option in >>> universe_wsgi.ini) that will set the $TEMP or $TMPDIR environment variable >>> to a suitable temp space. >>> >>> --nate >>> Please advise. Thanks, Greg Miscellaneous information:Traceback (most recent call last): File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 384, in __main__() File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 373, in __main__ add_file( dataset, Job Standard Error Traceback (most recent call last): File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 384, in __main__() File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 373, in __main__ add_file( dataset, registry, json_file, output_path ) File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 270, in add_file line_count, converted_path = sniff.convert_newlines( dataset.path, in_place=in_place ) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py", line 99, in convert_newlines fp.write( "%s\n" % line.rstrip( "\r\n" ) ) IOError: [Errno 28] No space left on device ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ >>> > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Uploading a Directory of Files - IOError: [Errno 28] No space left on device
Hmm, I narrowed down the problem some more: For some reason Python isn't respecting the "TEMP" environment variable so it's trying to write to /tmp on whichever node it's running on. I really don't understand why Python isn't respecting it. The docs seem to suggest it should: http://docs.python.org/2/library/tempfile.html#tempfile.tempdir I ran: >qlogin > source /usr/local/galaxy/job_environment_setup_file > python Python 2.7.1 (r271:86832, Apr 4 2011, 13:23:54) [GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import tempfile >>> tempfile.gettempdir() '/tmp' >>> import os >>> os.environ['TEMP'] '/scratch/galaxy' On Wed, Feb 6, 2013 at 3:00 PM, greg wrote: > Thanks Nate, > > It turns out I already had this as the first line of my job setup file: > > export TEMP=/scratch/galaxy > > But when I look in that directory, there's plenty of free space, and I > also don't see any recent files there. So I'm wondering if the upload > jobs aren't seeing that for some reason. > > Any ideas on how I could diagnose this more? > > -Greg > > > Relevant info? > > >>grep env galaxy-dist/universe_wsgi.ini > environment_setup_file = /usr/local/galaxy/job_environment_setup_file > >>cat /usr/local/galaxy/job_environment_setup_file > export TEMP=/scratch/galaxy > #active Python virtual env just for galaxy > source /usr/local/galaxy/galaxy_python/bin/activate > ... path setup lines ... > > > > > > > > > > > On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor wrote: >> On Feb 6, 2013, at 2:32 PM, greg wrote: >> >>> Hi guys, >>> >>> When I try to upload a directory of files from a server directory I'm >>> seeing the error below. >>> >>> It appears to be trying to write to a temp directory somewhere that >>> I'm guessing doesn't have enough space? Is there a way I can direct >>> where it writes to for temporary files like this? >> >> Hi Greg, >> >> There are a few ways. For some parts of Galaxy, you will want to set >> new_file_path in universe_wsgi.ini to a suitable temp space. However, this >> is not the case for the upload tool. >> >>> Am I understanding right, that these upload jobs are running on our >>> cluster? I think it would be a problem if its trying to use the >>> default temp directory on each cluster node since they aren't >>> provisioned with much space. >> >> This is correct. On the cluster, add something to your user's shell startup >> files (or see the environment_setup_file option in universe_wsgi.ini) that >> will set the $TEMP or $TMPDIR environment variable to a suitable temp space. >> >> --nate >> >>> >>> Please advise. >>> >>> Thanks, >>> >>> Greg >>> >>> >>> >>> Miscellaneous information:Traceback (most recent call last): File >>> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >>> 384, in __main__() File >>> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >>> 373, in __main__ add_file( dataset, >>> Job Standard Error >>> >>> >>> Traceback (most recent call last): >>> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >>> line 384, in >>>__main__() >>> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >>> line 373, in __main__ >>>add_file( dataset, registry, json_file, output_path ) >>> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >>> line 270, in add_file >>>line_count, converted_path = sniff.convert_newlines( dataset.path, >>> in_place=in_place ) >>> File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py", >>> line 99, in convert_newlines >>>fp.write( "%s\n" % line.rstrip( "\r\n" ) ) >>> IOError: [Errno 28] No space left on device >>> ___ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >> ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Uploading a Directory of Files - IOError: [Errno 28] No space left on device
On Feb 6, 2013, at 3:00 PM, greg wrote: > Thanks Nate, > > It turns out I already had this as the first line of my job setup file: > > export TEMP=/scratch/galaxy > > But when I look in that directory, there's plenty of free space, and I > also don't see any recent files there. So I'm wondering if the upload > jobs aren't seeing that for some reason. > > Any ideas on how I could diagnose this more? Hi Greg, The first place to look would be in lib/galaxy/datatypes/sniff.py, line 96: fd, temp_name = tempfile.mkstemp() If you print temp_name, that will tell you what file the upload tool is writing to. You may also want to take a look at: http://docs.python.org/2/library/tempfile.html#tempfile.tempdir Some cluster environments set $TMPDIR, and if that is set, $TEMP will not be used. --nate > > -Greg > > > Relevant info? > > >> grep env galaxy-dist/universe_wsgi.ini > environment_setup_file = /usr/local/galaxy/job_environment_setup_file > >> cat /usr/local/galaxy/job_environment_setup_file > export TEMP=/scratch/galaxy > #active Python virtual env just for galaxy > source /usr/local/galaxy/galaxy_python/bin/activate > ... path setup lines ... > > > > > > > > > > > On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor wrote: >> On Feb 6, 2013, at 2:32 PM, greg wrote: >> >>> Hi guys, >>> >>> When I try to upload a directory of files from a server directory I'm >>> seeing the error below. >>> >>> It appears to be trying to write to a temp directory somewhere that >>> I'm guessing doesn't have enough space? Is there a way I can direct >>> where it writes to for temporary files like this? >> >> Hi Greg, >> >> There are a few ways. For some parts of Galaxy, you will want to set >> new_file_path in universe_wsgi.ini to a suitable temp space. However, this >> is not the case for the upload tool. >> >>> Am I understanding right, that these upload jobs are running on our >>> cluster? I think it would be a problem if its trying to use the >>> default temp directory on each cluster node since they aren't >>> provisioned with much space. >> >> This is correct. On the cluster, add something to your user's shell startup >> files (or see the environment_setup_file option in universe_wsgi.ini) that >> will set the $TEMP or $TMPDIR environment variable to a suitable temp space. >> >> --nate >> >>> >>> Please advise. >>> >>> Thanks, >>> >>> Greg >>> >>> >>> >>> Miscellaneous information:Traceback (most recent call last): File >>> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >>> 384, in __main__() File >>> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >>> 373, in __main__ add_file( dataset, >>> Job Standard Error >>> >>> >>> Traceback (most recent call last): >>> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >>> line 384, in >>> __main__() >>> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >>> line 373, in __main__ >>> add_file( dataset, registry, json_file, output_path ) >>> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >>> line 270, in add_file >>> line_count, converted_path = sniff.convert_newlines( dataset.path, >>> in_place=in_place ) >>> File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py", >>> line 99, in convert_newlines >>> fp.write( "%s\n" % line.rstrip( "\r\n" ) ) >>> IOError: [Errno 28] No space left on device >>> ___ >>> Please keep all replies on the list by using "reply all" >>> in your mail client. To manage your subscriptions to this >>> and other Galaxy lists, please use the interface at: >>> >>> http://lists.bx.psu.edu/ >> ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Uploading a Directory of Files - IOError: [Errno 28] No space left on device
Thanks Nate, It turns out I already had this as the first line of my job setup file: export TEMP=/scratch/galaxy But when I look in that directory, there's plenty of free space, and I also don't see any recent files there. So I'm wondering if the upload jobs aren't seeing that for some reason. Any ideas on how I could diagnose this more? -Greg Relevant info? >grep env galaxy-dist/universe_wsgi.ini environment_setup_file = /usr/local/galaxy/job_environment_setup_file >cat /usr/local/galaxy/job_environment_setup_file export TEMP=/scratch/galaxy #active Python virtual env just for galaxy source /usr/local/galaxy/galaxy_python/bin/activate ... path setup lines ... On Wed, Feb 6, 2013 at 2:37 PM, Nate Coraor wrote: > On Feb 6, 2013, at 2:32 PM, greg wrote: > >> Hi guys, >> >> When I try to upload a directory of files from a server directory I'm >> seeing the error below. >> >> It appears to be trying to write to a temp directory somewhere that >> I'm guessing doesn't have enough space? Is there a way I can direct >> where it writes to for temporary files like this? > > Hi Greg, > > There are a few ways. For some parts of Galaxy, you will want to set > new_file_path in universe_wsgi.ini to a suitable temp space. However, this > is not the case for the upload tool. > >> Am I understanding right, that these upload jobs are running on our >> cluster? I think it would be a problem if its trying to use the >> default temp directory on each cluster node since they aren't >> provisioned with much space. > > This is correct. On the cluster, add something to your user's shell startup > files (or see the environment_setup_file option in universe_wsgi.ini) that > will set the $TEMP or $TMPDIR environment variable to a suitable temp space. > > --nate > >> >> Please advise. >> >> Thanks, >> >> Greg >> >> >> >> Miscellaneous information:Traceback (most recent call last): File >> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >> 384, in __main__() File >> "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line >> 373, in __main__ add_file( dataset, >> Job Standard Error >> >> >> Traceback (most recent call last): >> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >> line 384, in >>__main__() >> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >> line 373, in __main__ >>add_file( dataset, registry, json_file, output_path ) >> File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", >> line 270, in add_file >>line_count, converted_path = sniff.convert_newlines( dataset.path, >> in_place=in_place ) >> File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py", >> line 99, in convert_newlines >>fp.write( "%s\n" % line.rstrip( "\r\n" ) ) >> IOError: [Errno 28] No space left on device >> ___ >> Please keep all replies on the list by using "reply all" >> in your mail client. To manage your subscriptions to this >> and other Galaxy lists, please use the interface at: >> >> http://lists.bx.psu.edu/ > ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Uploading a Directory of Files - IOError: [Errno 28] No space left on device
On Feb 6, 2013, at 2:32 PM, greg wrote: > Hi guys, > > When I try to upload a directory of files from a server directory I'm > seeing the error below. > > It appears to be trying to write to a temp directory somewhere that > I'm guessing doesn't have enough space? Is there a way I can direct > where it writes to for temporary files like this? Hi Greg, There are a few ways. For some parts of Galaxy, you will want to set new_file_path in universe_wsgi.ini to a suitable temp space. However, this is not the case for the upload tool. > Am I understanding right, that these upload jobs are running on our > cluster? I think it would be a problem if its trying to use the > default temp directory on each cluster node since they aren't > provisioned with much space. This is correct. On the cluster, add something to your user's shell startup files (or see the environment_setup_file option in universe_wsgi.ini) that will set the $TEMP or $TMPDIR environment variable to a suitable temp space. --nate > > Please advise. > > Thanks, > > Greg > > > > Miscellaneous information:Traceback (most recent call last): File > "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line > 384, in __main__() File > "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line > 373, in __main__ add_file( dataset, > Job Standard Error > > > Traceback (most recent call last): > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 384, in >__main__() > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 373, in __main__ >add_file( dataset, registry, json_file, output_path ) > File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", > line 270, in add_file >line_count, converted_path = sniff.convert_newlines( dataset.path, > in_place=in_place ) > File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py", > line 99, in convert_newlines >fp.write( "%s\n" % line.rstrip( "\r\n" ) ) > IOError: [Errno 28] No space left on device > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Uploading a Directory of Files - IOError: [Errno 28] No space left on device
Hi guys, When I try to upload a directory of files from a server directory I'm seeing the error below. It appears to be trying to write to a temp directory somewhere that I'm guessing doesn't have enough space? Is there a way I can direct where it writes to for temporary files like this? Am I understanding right, that these upload jobs are running on our cluster? I think it would be a problem if its trying to use the default temp directory on each cluster node since they aren't provisioned with much space. Please advise. Thanks, Greg Miscellaneous information:Traceback (most recent call last): File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 384, in __main__() File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 373, in __main__ add_file( dataset, Job Standard Error Traceback (most recent call last): File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 384, in __main__() File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 373, in __main__ add_file( dataset, registry, json_file, output_path ) File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 270, in add_file line_count, converted_path = sniff.convert_newlines( dataset.path, in_place=in_place ) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/datatypes/sniff.py", line 99, in convert_newlines fp.write( "%s\n" % line.rstrip( "\r\n" ) ) IOError: [Errno 28] No space left on device ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
