[Disclaimer: thought dump e-mail, signal to noise ratio may be subpar.] Sounds like you're at least making steps forward in the right direction, despite the activity probably being quite disheartening. Based on what you've said below and the rest of the conversation, here are my thoughts for an approach:
1. For a given python[_d].exe, always use the same test directory, but hash it against the entire python process path such that it's unique only for a given python instance. 2. Make sure every time a test wants a temp file, it gets a unique one in this directory. Sounds like your TESTFN modification would take care of this for those tests using TESTFN; if TESTFN is the preferred approach then any other tests using tempfile or hardcoding file names would then be changed to use this instead. 3. In order to address tests that either call the test_support methods for removing files/directories, or those that call os.(unlink|remove), do what ever needs to be done to make these no-ops on Windows if an error occurs. 4. At the end of the regrtest.py run, create a suspended arbitrary process (i.e. explorer.exe), hijack the main thread context of the process and inject a routine (i.e. overwrite the thread context's instruction pointers) that takes care of removing the temporary directory that was used for testing -- patiently re-trying if any failures occur until all rogue processes also accessing the file(s) stop doing so. Resume the thread before exiting python. Heh. Sounds crazy? It did to me as well, until I came across http://www.catch22.net/tuts/selfdel.asp, which documents the approach. It's not particularly necessary in our case, we could simply spawn another python process at the end of regrtest.py that patiently attempts to remove the test directory we just used when the python process that was executing regrtest.py exits. We could then modify regrtest.py such that it will use the same approach if the hashed test directory already exists at the start of a run and it can't remove it via os.unlink. If we *still* run into issues here on the buildbots, say if regrtest.py blocks on our helper process, which for the life of it can't remove some/all of the test files -- it'd be interesting to write something that grok's all open handles for all processes and attempts to figure out what it is that keeps these files open -- i.e. same thing that procexp.exe does when you search for a handle. Or, keeping it simple, rather than a separate process and hashed test directory based on python process path, just have a common directory, i.e. %TEMP%\python-regrtest, and use an incrementing sequence number for each test run's test directory, i.e. if there are directories 001, 002 and 003 in the temp dir, when regrtest.py starts, it'll try delete all of these -- if it can't (which is what we'd want if another test is already running), it adds 1 to the highest numbered directory it couldn't delete. Guess it all depends on how much effort we want to put into cleaning up our test directory really -- just ensuring tests get a clean area and unique file names each run is the easy part. Trent. ________________________________________ From: [EMAIL PROTECTED] [EMAIL PROTECTED] On Behalf Of Tim Golden [EMAIL PROTECTED] Sent: 03 April 2008 09:39 To: Python-Dev Subject: Re: [Python-Dev] fixing tests on windows [re tests which fail because something's holding a file open with SHARE_DELETE] Well I've tried my best, but I can't come up with a solution which guarantees to overcome this obstacle. I set up a fairly aggressive directory watcher which, when it sees a test file being created, takes a SHARE_DELETE handle on it and releases it immediately. (Guessing that this is what the other apps are doing). Running the test suite from HEAD, this generates all manner of access-denied type errors as per the original output. I used tempfile to generate a random TESTFN in the current directory rather than the static @test. And implemented the rename-shim discussed previously, renaming to a different random name, courtesy of mktemp. With those in place, most tests run without error. But I'm still getting errors in the same sort of areas which Steven B originally reported. The one error I can't see a way round is the access denied (which manifests as Permission Error) which is the documented result of attempting to open a file with a pending delete -- ie the delete succeeded but hasn't completed yet. An additional complication is that there are hundreds of instances throughout the tests where the test simply calls os.unlink/os.remove to undo the test file. To have some more robust central deletion I had to go through and update 68 tests. I'll keep trying, but in the current state I'm not convinced the situation's improved enough for me to bother uploading a patch. TJG _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
