[issue21579] Python 3.4: tempfile.close attribute does not work

[Eryk Sun]

Eryk Sun added the comment:

For anyone interested, this issue is solvable on Windows by working around how 
O_TEMPORARY is implemented. To do this the _winapi module would need a wrapper 
for SetFileInformationByHandle (available in Vista+), which would need to 
support at least FileDispositionInfo. 

That said, the use case of replacing a file is better supported by calling the 
Windows API function ReplaceFile anyway. It isn't atomic, however. I don't 
think that's possible in general on Windows. The problem is that, unlike on 
Unix, an open file on Windows can't be anonymous. So if the replaced file is 
already open, it has to first be renamed before the replacement file can take 
its place. That creates a small window for things to go wrong.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Марк Коренберг]

Changes by Марк Коренберг :


--
resolution:  -> wont fix
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Марк Коренберг]

Марк Коренберг added the comment:

David Murray, in your code, if temporary file cannot be created, it attempts to 
unlink unknown name :).

But, I already have almost the same solution in production. I mention it 
sources in issue8604.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Марк Коренберг]

Марк Коренберг added the comment:

So, I'm required to re-implement NamedTemporaryFile by hand like that:


fd, name = tempfile.mkstemp(...)
try:
with io.open(fd, ...) as fff:
fff.write('hello')
fff.flush()
os.fdatasync(fff)
os.rename(name, ...)
except:
os.unlink(name)
raise


This is sad to see...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[R. David Murray]

R. David Murray added the comment:

You'd have to do that anyway if we implemented a delete=False constructor 
argument, since you want it deleted if there are any errors, and that's not 
what a delete=False API would do.  

If it were me, I'd write it (untested)

  @contextlib.contextmanager
  def open_for_atomic_replace(fn):
try:
fd, name = tempfile.mkstemp()
with io.open(fd) as fff:
yield fff
fff.flush()
os.fdatasync(fff)
os.rename(name, fn)
except BaseException:
os.unlink(name)
raise

which would make your code simpler than it is now.

Naming it 'open_for_atomic_replace' reminded me of issue 8604, which is what 
you really want, not delete=False.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[R. David Murray]

R. David Murray added the comment:

Oh, you are right of course.  I thought I was looking at _mkstemp_inner but in 
fact my edit window was over NamedTemporaryFile...I just wasn't paying 
attention.

I have no opinion myself as to whether it is worth the effort/code complexity 
to implement this behavior cross platform.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Eryk Sun]

Eryk Sun added the comment:

> To extend support for this to Windows, we can add a 
> feature to mkstmp to not use O_TEMPORARY

O_TEMPORARY is only used for NamedTemporaryFile, not mkstemp. 

Regarding NamedTemporaryFile, that can be worked around to keep Windows from 
deleting the file. Just open a second handle with DELETE (0x0001) access 
before closing the first one. Then close the first handle. Finally, use the 
second handle to call SetFileInformationByHandle [1] to remove the delete 
disposition. 

Since using O_TEMPORARY opens the file with DELETE access, you have to open the 
new handle with delete sharing. Unfortunately the CRT only provides O_TEMPORARY 
for opening a file with delete access and delete sharing, so just call 
CreateFile [2] directly instead. For example:

>>> import os, tempfile, msvcrt, ctypes
>>> kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
>>> f1 = tempfile.NamedTemporaryFile()
>>> f1.write(b'spam')
4
>>> h = kernel32.CreateFileW(f1.name, 0xC001, 7,
...  None, 3, 0x80, None)
>>> f1.close()
>>> os.path.exists(f1.name) # really, it does exist
False
>>> info = (ctypes.c_int * 1)(0)
>>> kernel32.SetFileInformationByHandle(h, 4, info, 4)
1
>>> os.path.exists(f1.name)
True
>>> f2 = os.fdopen(msvcrt.open_osfhandle(h, os.O_RDWR), 'r+')
>>> f2.read()
'spam'

Notice that right after f1 is closed, os.path.exists(f1.name) lies, saying the 
file no longer exists. The problem is you can't stat the file at this point 
because you can't open a *new* handle to a file that's marked for deletion. But 
you can use the existing handle to remove the delete disposition, after which 
the file is resurrected.

Also note that the FILE_DISPOSITION_INFO [3] docs say "t]his member has no 
effect if the handle was opened with FILE_FLAG_DELETE_ON_CLOSE". That's 
referring literally to the handle passed to SetFileInformationByHandle. 

Microsoft publishes the source for the FAT filesystem, so I could provide links 
for how this is implemented. Basically every kernel file object has a context 
control block (CCB) that points at an underlying kernel filesystem structure 
such as a file control block (FCB) or NTFS link/stream control block (LCB / 
SCB). When the file object is closed, the delete-on-close flag in the CCB gets 
transferred over to the underlying control structure and the delete disposition 
is set. You can't remove the CCB flag from the kernel file object (as the 
documentation correctly notes), but as long as you have an open handle to a 
file object whose CCB does not have this flag, you can use this other file 
handle to remove the delete disposition and make the file permanent. Just make 
sure to first close all file objects that do have the CCB delete-on-close flag 
because otherwise they'll just set the delete disposition again when closed.

[1]: https://msdn.microsoft.com/en-us/library/aa365539
[2]: https://msdn.microsoft.com/en-us/library/aa363858
[3]: https://msdn.microsoft.com/en-us/library/aa364221

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[R. David Murray]

R. David Murray added the comment:

I think it is a reasonable backward-compatibility fix to make it work as badly 
as it previously did ;).  I would not document it, and I do not consider it a 
high priority (as Serhiy said, this usage reached into object internals and 
thus was dangerous from the start).

Unless I'm missing something, there is no need to add any new features to 
support your use case, Марк, or for us to apply a backward compatibility fix 
for you to get your code working agian.  Your use case is already supported:

fd, name = tempfile.mkstemp(...)
with io.open(fd, ...) as f:
...

This has the added advantage that it will work on all python versions.  You can 
easy wrap it into a function for use in your application if you don't like 
needing two lines.

To extend support for this to Windows, we can add a feature to mkstmp to not 
use O_TEMPORARY.  Also, it would probably be worth adding the above as an 
example to the docs.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Марк Коренберг]

Марк Коренберг added the comment:

ping. THis is essential in our software, so we use private field in order to 
work-around this bug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Марк Коренберг]

Марк Коренберг added the comment:

No activity last week ? Why not to merge ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Марк Коренберг]

Марк Коренберг added the comment:

Please merge patch, proposed by Eduardo Seabra (Eduardo.Seabra). And document 
that this is UNIX-only solution.

--
versions: +Python 3.5, Python 3.6
Added file: http://bugs.python.org/file41055/test_tempfile.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

If it is possible to cancel the effect of the O_TEMPORARY flag, we can use it 
to implement this feature on all platforms. But if it is not possible, we have 
several options:

1. Just close this issue and do nothing more. This was undocumented and 
non-portable feature.

2. Implement delete as readonly property. This is similar to 1 but makes the 
failure loud.

3. Implement this feature on non-Windows, document that it is non-Windows only 
feature, and raise an exception in delete setter on Windows.

4. Same as 3, but emit deprecation warning in delete setter on non-Windows. In 
future this should be replaced with solution 2.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

See also issue14243.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Eduardo Seabra]

Eduardo Seabra added the comment:

I've attached a patch with @mmarkk proposal.

--
keywords: +patch
nosy: +Eduardo.Seabra
Added file: http://bugs.python.org/file35662/issue21579.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

OK, O_TMPFILE is not related. But Windows is related.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Марк Коренберг]

Марк Коренберг added the comment:

So, maybe API change? like delete=True|False|Maybe ? don't think that this is 
good decisions.

My approach is based on ext4 behaviour about delayed allocation and atomic file 
replacements.
In my case, either old file (with contents) or new file appear.
In any case, corrupted data cannot appear.
This is required behaviour for my application.


https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
http://lwn.net/Articles/322823/

===
auto_da_alloc(*)Many broken applications don't use fsync() when 
noauto_da_alloc replacing existing files via patterns such as
fd = open(foo.new)/write(fd,..)/close(fd)/
rename(foo.new, foo), or worse yet,
fd = open(foo, O_TRUNC)/write(fd,..)/close(fd).
If auto_da_alloc is enabled, ext4 will detect
the replace-via-rename and replace-via-truncate
patterns and force that any delayed allocation
blocks are allocated such that at the next
journal commit, in the default data=ordered
mode, the data blocks of the new file are forced
to disk before the rename() operation is
committed.  This provides roughly the same level
of guarantees as ext3, and avoids the
zero-length problem that can happen when a
system crashes before the delayed allocation
blocks are forced to disk.
===

So, this is essential functionality.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Марк Коренберг]

Марк Коренберг added the comment:

why not to make tmpfileobj.delete as property that really sets 
self._closer.delete ? this will fix my problem.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Yes, it would be easy to restore this behavior on Posix.

--
keywords: +easy
stage:  - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Berker Peksag]

Berker Peksag added the comment:

See issue 18879 for more information about the change.

--
nosy: +berker.peksag, pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +georg.brandl, ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

In any case this trick didn't work on Windows. And it can't work on Linux too 
when use new O_TMPFILE flag (issue21515).

--
nosy: +serhiy.storchaka

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Марк Коренберг]

Марк Коренберг added the comment:

Yes, but O_TMPFILE should be set ONLY  when used with TemporaryFile, not with 
NamedTemporaryFile. My problem refer only NamedTemporaryFile.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21579] Python 3.4: tempfile.close attribute does not work

[Марк Коренберг]

New submission from Марк Коренберг:

Suppose code:
=
import os
import tempfile

want_to_replace = 'zxc.dat'
tmpdir=os.path.dirname(os.path.realpath(want_to_replace))
with tempfile.NamedTemporaryFile(dir=tmpdir) as fff:
# do somewhat with fff here... and then:
fff.flush()
os.fdatasync(fff)
os.rename(fff.name, want_to_replace)
fff.delete = False
=
In python 3.3 and lower that works FINE. In Python 3.4 the fff._closer 
attribute was introduced, so fff.close=False stopped to work. I think this is 
major loss of functionality. The close attribute was not marked as private, 
so may be used in past.

--
components: Library (Lib)
messages: 219132
nosy: mmarkk
priority: normal
severity: normal
status: open
title: Python 3.4: tempfile.close attribute does not work
type: behavior
versions: Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com