[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Roundup Robot added the comment: New changeset b368fc93dca8 by Serhiy Storchaka in branch '2.7': Issue #16800: tempfile.gettempdir() no longer left temporary files when http://hg.python.org/cpython/rev/b368fc93dca8 New changeset 377123f10820 by Serhiy Storchaka in branch '3.2': Issue #16800: tempfile.gettempdir() no longer left temporary files when http://hg.python.org/cpython/rev/377123f10820 New changeset 6f432bb11b28 by Serhiy Storchaka in branch '3.3': Issue #16800: tempfile.gettempdir() no longer left temporary files when http://hg.python.org/cpython/rev/6f432bb11b28 New changeset b66a5b41d82f by Serhiy Storchaka in branch 'default': Issue #16800: tempfile.gettempdir() no longer left temporary files when http://hg.python.org/cpython/rev/b66a5b41d82f -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Roundup Robot added the comment: New changeset a43f67e95ef0 by Serhiy Storchaka in branch '2.7': Fix for issue #16800: Use buffered write to handle EINTR. http://hg.python.org/cpython/rev/a43f67e95ef0 New changeset 4622206db91b by Serhiy Storchaka in branch '3.2': Fix for issue #16800: Use buffered write to handle EINTR. http://hg.python.org/cpython/rev/4622206db91b New changeset 2fb03fe354e3 by Serhiy Storchaka in branch '3.3': Fix for issue #16800: Use buffered write to handle EINTR. http://hg.python.org/cpython/rev/2fb03fe354e3 New changeset fec33725f319 by Serhiy Storchaka in branch 'default': Fix for issue #16800: Use buffered write to handle EINTR. http://hg.python.org/cpython/rev/fec33725f319 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Serhiy Storchaka added the comment: Thank you for explanation, Antoine. Thank you for your contribution, Amir. -- assignee: - serhiy.storchaka resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Antoine Pitrou added the comment: There are reasons to use buffered I/O rather than os.write: os.write can fail with EINTR, for example. -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Serhiy Storchaka added the comment: In general it looks good to me. But we can get rid of Python file and use only os.open/os.write/os.close. Here is an updated patch which carefully close a file descriptor and remove a file even if write or close failed. Amir Szekely, can you please submit a contributor form? http://python.org/psf/contrib/contrib-form/ http://python.org/psf/contrib/ -- nosy: +georg.brandl, ncoghlan, serhiy.storchaka Added file: http://bugs.python.org/file28966/fix_tempfile_leaving_files_behind_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Serhiy Storchaka added the comment: In general it looks good to me. But we can get rid of Python file and use only os.open/os.write/os.close. Here is an updated patch which carefully close a file descriptor and remove a file even if write or close failed. Amir Szekely, can you please submit a contributor form? http://python.org/psf/contrib/contrib-form/ http://python.org/psf/contrib/ -- Added file: http://bugs.python.org/file28967/fix_tempfile_leaving_files_behind_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Changes by Serhiy Storchaka storch...@gmail.com: -- Removed message: http://bugs.python.org/msg181470 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Changes by Serhiy Storchaka storch...@gmail.com: Removed file: http://bugs.python.org/file28967/fix_tempfile_leaving_files_behind_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Serhiy Storchaka added the comment:
And here are patches for 2.7 and 3.2 (there are some peculiarities, i.e. in
exception handling).
--
Added file:
http://bugs.python.org/file28968/fix_tempfile_leaving_files_behind_2-2.7.patch
Added file:
http://bugs.python.org/file28969/fix_tempfile_leaving_files_behind_2-3.2.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16800
___diff -r 20f0c5398e97 Lib/tempfile.py
--- a/Lib/tempfile.py Mon Feb 04 10:29:38 2013 -0500
+++ b/Lib/tempfile.py Tue Feb 05 20:57:18 2013 +0200
@@ -193,14 +193,16 @@
name = namer.next()
filename = _os.path.join(dir, name)
try:
-fd = _os.open(filename, flags, 0600)
-fp = _os.fdopen(fd, 'w')
-fp.write('blat')
-fp.close()
-_os.unlink(filename)
-del fp, fd
+fd = _os.open(filename, flags, 0o600)
+try:
+try:
+_os.write(fd, b'blat')
+finally:
+_os.close(fd)
+finally:
+_os.unlink(filename)
return dir
-except (OSError, IOError), e:
+except (OSError, IOError) as e:
if e[0] != _errno.EEXIST:
break # no point trying more names in this directory
pass
diff -r 20f0c5398e97 Lib/test/test_support.py
--- a/Lib/test/test_support.py Mon Feb 04 10:29:38 2013 -0500
+++ b/Lib/test/test_support.py Tue Feb 05 20:57:18 2013 +0200
@@ -1298,6 +1298,33 @@
except:
break
+@contextlib.contextmanager
+def swap_attr(obj, attr, new_val):
+Temporary swap out an attribute with a new object.
+
+Usage:
+with swap_attr(obj, attr, 5):
+...
+
+This will set obj.attr to 5 for the duration of the with: block,
+restoring the old value at the end of the block. If `attr` doesn't
+exist on `obj`, it will be created and then deleted at the end of the
+block.
+
+if hasattr(obj, attr):
+real_val = getattr(obj, attr)
+setattr(obj, attr, new_val)
+try:
+yield
+finally:
+setattr(obj, attr, real_val)
+else:
+setattr(obj, attr, new_val)
+try:
+yield
+finally:
+delattr(obj, attr)
+
def py3k_bytes(b):
Emulate the py3k bytes() constructor.
diff -r 20f0c5398e97 Lib/test/test_tempfile.py
--- a/Lib/test/test_tempfile.py Mon Feb 04 10:29:38 2013 -0500
+++ b/Lib/test/test_tempfile.py Tue Feb 05 20:57:18 2013 +0200
@@ -1,7 +1,9 @@
# tempfile.py unit tests.
import tempfile
+import errno
import os
import signal
+import shutil
import sys
import re
import warnings
@@ -202,8 +204,45 @@
test_classes.append(test__candidate_tempdir_list)
+# We test _get_default_tempdir some more by testing gettempdir.
-# We test _get_default_tempdir by testing gettempdir.
+class TestGetDefaultTempdir(TC):
+Test _get_default_tempdir().
+
+def test_no_files_left_behind(self):
+# use a private empty directory
+our_temp_directory = tempfile.mkdtemp()
+try:
+# force _get_default_tempdir() to consider our empty directory
+def our_candidate_list():
+return [our_temp_directory]
+
+with test_support.swap_attr(tempfile, _candidate_tempdir_list,
+our_candidate_list):
+# verify our directory is empty after _get_default_tempdir()
+tempfile._get_default_tempdir()
+self.assertEqual(os.listdir(our_temp_directory), [])
+
+def raise_OSError(*args, **kwargs):
+raise OSError(-1)
+
+with test_support.swap_attr(os, open, raise_OSError):
+# test again with failing os.open()
+with self.assertRaises(IOError) as cm:
+tempfile._get_default_tempdir()
+self.assertEqual(cm.exception.errno, errno.ENOENT)
+self.assertEqual(os.listdir(our_temp_directory), [])
+
+with test_support.swap_attr(os, write, raise_OSError):
+# test again with failing os.write()
+with self.assertRaises(IOError) as cm:
+tempfile._get_default_tempdir()
+self.assertEqual(cm.exception.errno, errno.ENOENT)
+self.assertEqual(os.listdir(our_temp_directory), [])
+finally:
+shutil.rmtree(our_temp_directory)
+
+test_classes.append(TestGetDefaultTempdir)
class test__get_candidate_names(TC):
diff -r 801cb3918212 Lib/tempfile.py
--- a/Lib/tempfile.py Tue Feb 05 10:49:49 2013 -0500
+++ b/Lib/tempfile.py Tue Feb
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Amir Szekely added the comment: I've submitted a contribution form by e-mail. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
New submission from Amir Szekely: tempfile._get_default_tempdir() tries to find a good temporary directory by attempting to create a file and write a string into it for all candidate directories. It deletes those files right after closing them. But if writing rather than creating them fails, the files get left behind. This can happen, for example, when the disk is full on Linux. It will let you create the file with open() and even write() to it. But as soon as close() is called, OSError is raised with errno 28 (No space left on device) and unlink() is never called. On our system (Linux 2.6.32, ext3), after filling the disk by mistake, we suddenly found dozens of random empty files scattered across the file system. The candidate list, where they were all found, even includes the root directory so the issue is quite visible. We noticed this problem on Python 2.4.4, but I have been able to reproduce it with today's tip. A possible solution would be wrapping everything between open() and unlink() in a try-finally block. This way, no matter what happens, the file will get deleted. I have attached a patch implementing this solution and a new test case that reproduces the issue. Verified with `make test make patchcheck`. -- components: Extension Modules files: fix_tempfile_leaving_files_behind.patch keywords: patch messages: 178362 nosy: kichik priority: normal severity: normal status: open title: tempfile._get_default_tempdir() leaves files behind when HD is full type: resource usage versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5 Added file: http://bugs.python.org/file28464/fix_tempfile_leaving_files_behind.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Amir Szekely added the comment: Reproduction details: [root@meh /]# uname -a Linux meh 2.6.32-131.17.1.el6-0.0.3.smp.pae.gcc4.1.x86.i686 #1 SMP Tue Dec 20 02:40:22 PST 2011 i686 GNU/Linux [root@meh /]# pwd / [root@meh /]# dd if=/dev/zero of=/tmp/BIG bs=4M dd: writing `/tmp/BIG': No space left on device 31+0 records in 30+0 records out 127254528 bytes (127 MB) copied, 3.7028 s, 34.4 MB/s [root@meh /]# ls bin boot dev etc home initrd lib lost+found media mnt opt proc root sbin srv sys tmp usr var [root@meh /]# python -c import tempfile; tempfile.mktemp() Traceback (most recent call last): File string, line 1, in ? File /usr/lib/python2.4/tempfile.py, line 362, in mktemp dir = gettempdir() File /usr/lib/python2.4/tempfile.py, line 268, in gettempdir tempdir = _get_default_tempdir() File /usr/lib/python2.4/tempfile.py, line 214, in _get_default_tempdir raise IOError, (_errno.ENOENT, IOError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/'] [root@meh /]# ls bin boot dev etc home initrd lib lost+found media mnt opt proc -qE-nI root sbin srv sys tmp usr var [root@meh /]# -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16800] tempfile._get_default_tempdir() leaves files behind when HD is full
Changes by Serhiy Storchaka storch...@gmail.com: -- components: +Library (Lib) -Extension Modules stage: - patch review versions: -Python 2.6, Python 3.1, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16800 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
