[issue7443] test.support.unlink issue on Windows platform
Tim Golden added the comment: This has been covered off by work done with the test.support package including context managers for temporary files / directories, and a waitfor mechanism which waits some time if a file can't be accessed. -- resolution: - works for me status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
R. David Murray added the comment: The support package is fixed (I presume :), but there was also a desire expressed to expose this functionality in shutil, since it can arise in normal Python programs as well as in the test suite. Given that the fix has been successful in the test suite, shouldn't we open an issue to expose it in shutil? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Changes by Chris Jerdonek chris.jerdo...@gmail.com: -- nosy: +cjerdonek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Nick Coghlan added the comment: See #15496 for an alternative approach to solving this problem, at least in the test suite - as noted in that issue, the rename dance isn't sufficient when the problem gets triggered by a different sequence like: unlink(file_in_parent_dir) unlink(parent_dir) # This fails if file is still locked -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Changes by Nadeem Vawda nadeem.va...@gmail.com: -- nosy: +nadeem.vawda ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Andrew Svetlov andrew.svet...@gmail.com added the comment: FYI: implementation of unlink already has multiple calls for unicode version to process symlinks. Ansi version is the simple DeleteFileA call. Now we have different behavior for unicode and ansi variants as I can see. Now I'm working on unlink implementations partially borrowed from cygwin as Martin suggested. I still not have a final patch covering all known cases. Perhaps I can introduce it next week. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Jeff Dean chima...@gmail.com added the comment: * Patch Py_DeleteFileW in posixmodule.c so that it renames before deleting: should solve the problem overall but obviously has a possible wider impact, in general and on performance in particular. This rename might be a simple rename-to-guid or something more sophisticated such as the rename-to-recycler which cygwin uses. * Patch support.unlink in the test package to do the rename dance on the basis that it'll fix at least some of the problems with less impact overall. Opinions? I'm willing to do either. I vote for fixing the test package. File system extensions may track and record this activity. To use DropBox as an example, doing the rename and delete will cause the renamed and deleted file to be recorded. Just my opinion, but the code path to delete a file should be as short as possible. Making lots of other OS calls just doesn't seem right. I understand the wish to have a reliable unlink call but I'd be uncomfortable with a workaround that may be visible around the edges. -- nosy: +jdigital ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Nick Coghlan ncogh...@gmail.com added the comment: I'll throw in my 2 cents as to a possible way forward: - fix test.support.unlink in all current maintenance branches (2.7, 3.1, 3.2, default) - expose the feature to users as shutil.rmfile for 3.3 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Antoine Pitrou pit...@free.fr added the comment: * Patch Py_DeleteFileW in posixmodule.c so that it renames before deleting: should solve the problem overall but obviously has a possible wider impact, in general and on performance in particular. This rename might be a simple rename-to-guid or something more sophisticated such as the rename-to-recycler which cygwin uses. * Patch support.unlink in the test package to do the rename dance on the basis that it'll fix at least some of the problems with less impact overall. Opinions? I'm willing to do either. Well, since I'm not a Windows expert, I can only give an intuitive opinion. I think that we should start with patching support.unlink(); tweaking Py_DeleteFile() so as to do something more sophisticated than a simple removal sounds like it could hide some behaviour change that could hit some legitimate uses. (as an aside, for higher-level variants of OS functions, the shutil may be an appropriate recipient) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Antoine Pitrou pit...@free.fr added the comment: FWIW, Mercurial uses the following dance: http://selenic.com/repo/hg/file/463aca32a937/mercurial/windows.py#l296 (Mercurial is under the GPL, so we can't copy that code verbatim; but it can serve as an inspiration) -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Tim Golden m...@timgolden.me.uk added the comment: For clarity, while making unlink more robust is no bad thing, the error occurs when the unlink *succeeds* but a subsequent create of the same name fails. This happens when an indexer, Virus scanner or TortoiseSvn etc. has opened the file with SHARE_DELETE. This allows a DeleteFile to succeed but continues to hold an open handle on the file. A following test which uses an identically named file (such as the global TESTFN) will fail if not enough time has elapsed for the background process to release its handle. A good candidate to see this in action is the test for tarfile. I did start to undertake a conversion of TESTFN to a named temporary, but it started to sprawl all over the place and came up against a number of corner cases (eg where tests deliberately wanted two filenames to be the same) so I gave up. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Antoine Pitrou pit...@free.fr added the comment: I did start to undertake a conversion of TESTFN to a named temporary, but it started to sprawl all over the place and came up against a number of corner cases (eg where tests deliberately wanted two filenames to be the same) so I gave up. How about renaming to a unique random name just before the unlink(), as Mercurial does? Would it alleviate the problem or am I missing something? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Tim Golden m...@timgolden.me.uk added the comment: Well http://bugs.python.org/issue7443#msg102833 outlines the problems I encountered while trying to do essentially that. Nothing insurmountable, but definitely bigger than simply adding one line of code. Looks to me like there are two avenues of approach (and, given the chatter on this issue, several people willing to address them): * Patch Py_DeleteFileW in posixmodule.c so that it renames before deleting: should solve the problem overall but obviously has a possible wider impact, in general and on performance in particular. This rename might be a simple rename-to-guid or something more sophisticated such as the rename-to-recycler which cygwin uses. * Patch support.unlink in the test package to do the rename dance on the basis that it'll fix at least some of the problems with less impact overall. Opinions? I'm willing to do either. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Andrew Svetlov andrew.svet...@gmail.com added the comment: As I can see cygwin's unlink_nt uses Nt* functions family (NtSetInformationFile etc) which are part of Windows DDK. Do you like to use them or prefer SDK ones (say SetFileInformationByHandle)? In second case python unlink can borrow deletion schema from cygwin for modern Windows versions (Vista+) and return to legacy trivial DeleteFile call if OS is WinXP. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Changes by Andrew Svetlov andrew.svet...@gmail.com: -- versions: +Python 3.3 -Python 2.7, Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Nick Coghlan ncogh...@gmail.com added the comment: I closed issue 9627 as a duplicate of this - the buildbot failure referenced there was most likely due to something else holding open a temporary file that the test suite thought was closed. -- nosy: +ncoghlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Changes by Tim Golden m...@timgolden.me.uk: -- assignee: - tim.golden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Tim Golden m...@timgolden.me.uk added the comment: I put together a trivial patch against the 2.7 trunk (basically: I added a os.rename before the os.remove in test_support.unlink) and reran my test harness with test_zipfile... and it still failed because, of course, test_zipfile calls shutil.rmtree which bypasses the test_support.unlink altogether etc. etc. At this point several things occur to me: 1) There's little point in targetting the 2.x tree since 2.7 is due out any time now and is effectively end-of-line for 2.x and this isn't a release-blocker. Therefore whatever effort is brought to bear should be against the 3.x latest 2) This is a repeatable but relatively minority case: it only applies to Windows boxes and only to those where some behind-the-scenes process is holding this kind of lock on files for long enough to affect the tests. I'd certainly like to nail it but... 3) The amount of work -- and intrusion in the tests -- is quite substantial. Just looking [*] for straight os.unlink instances, without even considering shutil use gives 71 instances; os.remove gives another 90. That's even without the issues of the tests for shutil and os in any case. I'm willing to do the legwork of moving all the tests use, eg, support.unlink, support.rmtree and so on, but quis custodiet? who'll test the tests? grep os\.unlink *.py | wc -l grep os\.remove *.py | wc -l 4) All that said, the result should be a cleaner, more controllable test environment, at least for temp files. Another solution would be to rework the use of TESTFN on Windows to use a new temporary file every time. But that would be as much work and more than the unlink / rmtree work above. I'd like to hear opinions before I move forward with a non-trivial patch for this. For the sake of completeness, I attach a tiny test case which shows that the rename/remove approach should in fact work for the symptom we're seeing. -- Added file: http://bugs.python.org/file16869/test-case.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___import os, sys import traceback import win32file FILENAME = test def rename_and_remove (filename): os.rename (filename, filename + .deleted) os.remove (filename + .deleted) def remove_only (filename): os.remove (filename) def test (remove): open (FILENAME, w).close () hFile = win32file.CreateFile ( FILENAME, win32file.GENERIC_READ, win32file.FILE_SHARE_DELETE, None, win32file.OPEN_EXISTING, 0, 0 ) try: remove (FILENAME) try: open (FILENAME, w).close () except IOError: print FAIL else: print PASS finally: hFile.Close () try: open (FILENAME, w).close () except IOError: print FAIL else: print PASS if __name__ =='__main__': print print Should see FAIL-PASS test (remove_only) print print Should see PASS-PASS test (rename_and_remove) ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Jason R. Coombs jar...@jaraco.com added the comment: When I was working on a routine to checkout/patch/build/test/cleanup Python (see https://svn.jaraco.com/jaraco/python/jaraco.develop, and particularly scripts/test-windows-symlink-patch.py), I ran into a similar problem during the cleanup step. I tried using shutil.rmtree to clean up the folder that was checked out, but I repeatedly got 'access denied' exceptions. I ended up working around the problem by using subprocess and cmd.exe's rmdir /s /q. I think this demonstrates three facets to this problem: 1) It doesn't just affect the test suite. It happens to other Python programs that are using shutil.rmtree (and possibly remove/unlink) to remove files that are in use. 2) It doesn't have to be that way. At the very least, there could (should?) be a function in Python that replicates 'rmdir /s /q', which is not subject to the 'access denied' error. 3) We could use subprocess and cmd.exe to perform the unlink and rmtree operations in the test suite to bypass the transient failures until Python supports the behavior natively. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Martin v. Löwis mar...@v.loewis.de added the comment: 1. I agree that we should fix the unlinking problem on Windows. I also think that such a fix should be independent of the test suite - many people run into failed unlink problems. 2. Tim already said it, but I repeat: the common theory is that the culprit for this kind of problem is software like virus checkers, desktop search spiders, Tortoise, ... 3. I'm not convinced that rmdir/s/q *really* solves the problem reliably. Because it's a timing issue, it may be that the additional startup cost to invoke rmdir was enough to let the virus scanner win the race, so that rmdir actually had no problems with removing the file. 4. I believe the official party line for removing files on Windows is this: If DeleteFile fails, move the file to the trash bin (of the disk), and use NtSetInformationFile to set the delete disposition for the file. See cygwin's unlink_nt for an elaborate implementation of unlinking: http://tinyurl.com/y7w6rrj -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Tim Golden m...@timgolden.me.uk added the comment: I'm afraid that the problem doesn't lie in the unlink: DeleteFile succeeds. The problem is that the file is only marked for delete until such time as the last SHARE_DELETE handle on it is closed. Until that time, an attempt to (re)create the file for anything other than SHARE_DELETE will fail. As you say, it's a timing issue. Making os.unlink on Windows more robust may be a good idea, but it's not going to help this issue. See my test-case.py on an earlier message for reproduction: http://bugs.python.org/file16869 TJG -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Martin v. Löwis mar...@v.loewis.de added the comment: I'm afraid that the problem doesn't lie in the unlink: DeleteFile succeeds. The problem is that the file is only marked for delete until such time as the last SHARE_DELETE handle on it is closed. Then we shouldn't use DeleteFile in the first place to delete the file, but instead CreateFile, with DELETE access (and FILE_SHARE_DELETE sharing). If that fails, we need to move the file to the bin (see unlink_nt for details). Making os.unlink on Windows more robust may be a good idea, but it's not going to help this issue. See my test-case.py on an earlier message for reproduction: It certainly will help this case also. It would detect that the file is still open, and move it into the trash bin. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Tim Golden m...@timgolden.me.uk added the comment: Then we shouldn't use DeleteFile in the first place to delete the file, but instead CreateFile, with DELETE access (and FILE_SHARE_DELETE sharing). If that fails, we need to move the file to the bin (see unlink_nt for details). I see what you're getting at. I'm getting to the end of my day here, but I'll try to put a patch together for posixmodule.c when I can, if no-one else gets there first. Would you agree that py3k is the only target branch worth aiming for? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Martin v. Löwis mar...@v.loewis.de added the comment: Would you agree that py3k is the only target branch worth aiming for? Most certainly, yes. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Tim Golden m...@timgolden.me.uk added the comment: This is basically a rerun of this discussion a couple of years ago: http://mail.python.org/pipermail/python-dev/2008-April/078333.html The problem certainly still happens against trunk -- I have a semi-aggressive test-harness which can cause it to reproduce pretty much on-demand. I proposed an approach here: http://mail.python.org/pipermail/python-dev/2008-April/078339.html but when I started digging into test_support it all got a bit hairy because -- naturally -- test.support.unlink is used in a *lot* of places. In short, there's still a problem to be fixed. I believe that a rename-unlink dance would solve it, but only at the cost of affecting a lot of tests. -- nosy: +tim.golden ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
anatoly techtonik techto...@gmail.com added the comment: It is unlikely that it will go further then discussion unless this bug can be reliably reproduced to be debugged. If not testcase when at least Process Monitor log would be helpful. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Tim Golden m...@timgolden.me.uk added the comment: In one window run the attached script (assumes you have pywin32 installed) with a parameter of the directory the TESTFN file will end up in. Then run, eg, test_zipfile in another window. For me: c:\temp watch_dir.py C:\work_in_progress\make-snapshots\trunk\python\Lib C:\work_in_progress\make-snapshots\trunk\python\Lib ..\pcbuild\python.exe -m test.test_zipfile Obviously, you'd have to change the path to be wherever you're running the test suite from. The watch_dir script sits there looking for file activity, then takes and releases a delete-share handle on the file. It's enough to disrupt certain tests (such as test_zipfile) pretty much every time. Other tests are affected less, or only the first few times. Not sure why, but it's certainly enough to reproduce the general effect of TortoiseSVN or indexer or virus checker. -- Added file: http://bugs.python.org/file16839/watch_dir.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___import os, sys import winerror import win32file import win32con if __name__ == '__main__': path_to_watch = sys.argv[1] hDir = win32file.CreateFile ( path_to_watch, 1, # FILE_LIST_DIRECTORY win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE, None, win32con.OPEN_EXISTING, win32con.FILE_FLAG_BACKUP_SEMANTICS, None ) print = Watching, path_to_watch watching = set () handles = [] try: while 1: results = win32file.ReadDirectoryChangesW ( hDir, 1024, True, win32con.FILE_NOTIFY_CHANGE_FILE_NAME, None, None ) for action, filename in results: filename = os.path.join (path_to_watch, filename) if action == 1 and filename not in watching: try: handle = win32file.CreateFile ( filename, 0, win32file.FILE_SHARE_DELETE, None, win32file.OPEN_EXISTING, 0, 0 ) handles.append (handle) except win32file.error, (errno, module, message): if errno == winerror.ERROR_SHARING_VIOLATION: print .. Can't hold, repr (filename) else: print .. Problem with %r: %s % (filename, message) else: watching.add (filename) print .. Holding, repr (filename) handle.Close () handles.remove (handle) watching.discard (filename) print .. Released, repr (filename) finally: for handle in handles: handle.Close () hDir.Close () ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
R. David Murray rdmur...@bitdance.com added the comment: If the problem with the fix is that lots of tests use test_support.unlink, then I don't see why the rename dance can't be implemented in test_support.unlink. (Possibly conditioned on whether or not the tests are running on a windows platform.) Dealing with unlink problems is why that method exists in the first place. There are probably places in the test suite that *don't* use test_support.unlink, though, so fixing test_support.unlink will not necessarily fix all of the problems. We'll have to fix those other tests (probably by using the new test_support.unlink) as we find them. An actual patch will need a test that doesn't rely on win32file (ctypes would be OK). It may be necessary to rename to a unique filename, too. (To be clear, I think a unit test that reproduces the problem by doing an open with FILE_SHARE_DELETE is fine, we don't need a test that reproduces the race condition itself. The windows experts will correct me if I'm wrong :) I'm changing the stage to patch needed because it seems to me that using a technique like rename that doesn't introduce additional delays into the test suite is to be preferred. -- nosy: +r.david.murray stage: patch review - needs patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Changes by Jason R. Coombs jar...@jaraco.com: -- nosy: +jaraco ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Andrew Svetlov andrew.svet...@gmail.com added the comment: Florent, sorry. I have no Windows workstation now, so I cannot follow this issue. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
anatoly techtonik techto...@gmail.com added the comment: It would be nice to see standalone test case that illustrates the problem. -- nosy: +techtonik ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Sean Reifschneider j...@tummy.com added the comment: Andrew: There have been changes committed within the last week to #7712, which Florent suggested might be related. Can you please re-test this to see if it still exists, and if it does, bug me and I'll try to get some more movement on this. -- nosy: +jafo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Florent Xicluna la...@yahoo.fr added the comment: Note: the fix in test_linecache.py is useless. The with open(source_name, 'w') as source: context manager is in charge of closing the file on __exit__. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Florent Xicluna la...@yahoo.fr added the comment: Another report of something looking very similar on #7712. -- nosy: +brian.curtin, flox priority: low - high ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
New submission from Andrew Svetlov andrew.svet...@gmail.com: On Windows there are tiny delay between call to os.unlink and real file removing. Periodically it leads to unittest crashes in cases like this: test.support.unlink(filename) f = open(filename, 'wb') Proposed solution: wait in support.unlink for end of deletion asking os.stat for removed file (only if os.name == 'nt', of course). Also test.test_linecache:LineCacheTests.test_checkcache should be fixed - this one miss to close last opened file and Windows cannot remove it. Both patches for trunk and py3k is attached. -- components: Tests, Windows files: python3.patch keywords: patch messages: 95991 nosy: asvetlov severity: normal status: open title: test.support.unlink issue on Windows platform type: behavior versions: Python 2.7, Python 3.2 Added file: http://bugs.python.org/file15454/python3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Andrew Svetlov andrew.svet...@gmail.com added the comment: patch for python 2 is attached -- Added file: http://bugs.python.org/file15455/python2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Andrew Svetlov andrew.svet...@gmail.com added the comment: Problem can be reproduced with several run of test_bufio in both python versions. trunk contains more tests (+1 test case) so it's easer to catch 'access denied' exception on it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Changes by Brett Cannon br...@python.org: -- keywords: +needs review priority: - low stage: - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Martin v. Löwis mar...@v.loewis.de added the comment: I'm fairly skeptical that your analysis is right. Can you prove that a file still exists after unlink returns? -- nosy: +loewis ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7443] test.support.unlink issue on Windows platform
Andrew Svetlov andrew.svet...@gmail.com added the comment: You right, problem is not in os.unlink (Windows call DeleteFile) itself. I have TortoiseSVN (very popular Explorer extension to work with Subversion) installed. This tool run TSVNCache.exe process to update own data in background. TSVNCache.exe receive NotifyChangeDirectory events for svn checkout folders. support.TESTFN is tempfile name in current working directory. If I try to run unittests from some folder from python checkout TSVNCache.exe get change notify and analyze it. So, sometimes I can see race condition: - python: os.unlink(file) - TSVNCache.exe: get change event - TSVNCache.exe: query changes - python: open(file) - oops, TSVNCache.exe still processing notification and lock deleted file by holding opened handles to it. According to MSDN for DeleteFile function: The DeleteFile function marks a file for deletion on close. Therefore, the file deletion does not occur until the last handle to the file is closed. Subsequent calls to CreateFile to open the file fail with ERROR_ACCESS_DENIED. We get exactly same. - TSVNCache.exe finish of change update and release handle. File is actually deleted. I see this situation in sysinternals Process Monitor tool. Probability of race condition is tiny but non-zero. Intensive create/drop/create again sequences can catch this one. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7443 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
