Re: [Python-Dev] refleaks on trunk
On Tue, Apr 1, 2008 at 6:34 AM, Neal Norwitz <[EMAIL PROTECTED]> wrote:
> test_io is the only leaky test on trunk that I know of. I narrowed
> down the leaks to the code below.
>
> It's possible there are other leaks in test_io.
>
> n
> --
> import sys, gc
> import _fileio, io
>
> class FileIO(_fileio._FileIO, io.RawIOBase):
> def close(self):
> io.RawIOBase.close(self)
>
> def main():
> class MyFileIO(FileIO): pass
> f = MyFileIO('tt-test', "w")
> f.close()
>
> for i in range(10):
> main()
> print(gc.collect())
> print(sys.gettotalrefcount())
The problem is that the MyFileIO class is registered in
io.RawIOBase._abc_cache, and never freed.
This is a problem with ABCs: _abc_cache should be changed to a
WeakSet, like python3.0 did.
I filed http://bugs.python.org/issue2521 about this problem.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] FW: [issue2513] 64bit cross compilation on windows
I wrote: > FYI, I've uploaded a patch that provides for cross-compilation on > Windows between 32 and 64 bit platforms - all comments invited! While I have some people's attention I'd like to re-raise another issue I foresee for x64 builds. I've mentioned this over the last couple of months, but haven't got much of a response, so this seems like a reasonable opportunity to try one more time :) Currently, the "official" (by way of being de-facto) directory structure for a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 platforms. I believe this might cause problems for people trying to port their applications to 64bit platforms. My proposal is that we change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy of one of the platform subdirectories. The rationale is fairly simple: I'm quite certain that this new directory layout will break existing "native only" build processes (ie, those that aren't aware of cross-compilation). I'm quite certain that Mozilla will break, for example, and no cross-compilation process exists that I am aware of. Existing build tools already know about the PCBuild directory, and are focused almost exclusively towards native compilation - all such tools on non-x86 platforms are currently broken. My proposal would help avoid breakage for existing build processes or tools that try to natively target x64, and give a reasonable story for build processes that explicitly support cross-compilation. I believe the costs are fairly small - disk space is cheap and the extra complexity in the VS project files should be reasonable. What do people think? I think it's time to take the approach of "silent ascent", so unless I hear objections I'll upload a new patch which implements this and also takes it into account for Distutils builds. Cheers, Mark ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
On Tue, Apr 1, 2008 at 12:42 AM, Neal Norwitz <[EMAIL PROTECTED]> wrote: > The Windows buildbots are still failing because some tests keep files > opened. This causes subsequent tests which use the same file to fail. > > Here is a recent run which had a failure early on: > > http://www.python.org/dev/buildbot/stable/x86%20XP-3%20trunk/builds/1209/step-test/0 > > I'm assuming the first failure (test_bufio) was due to an open file. > (Can't tell, no error msg.) That means the problem was in that test > or an earlier test. The only earlier tests are: > > test_import > test_sys > test_descr > test_xdrlib > test_urllibnet > test_binhex > test_strptime > test_importhooks > test_copy > test_hmac > test_genericpath > test_complex > test_timeout > test_quopri > test_marshal > test_zipfile > test_mutants > test_csv > test_ucn > test_hash > test_wsgiref > test_mmap > test_ftplib > test_pickletools > test_codecmaps_cn > > Of those tests, only some of them reference TESTFN which is the probable > file: > > test_import 22 > test_descr 3 > test_urllibnet 2 > test_binhex 2 > test_genericpath 44 > test_complex 3 > test_marshal 33 > test_zipfile 136 > test_mutants 7 > test_mmap 32 > > I inspected test_mmap which had been modified somewhat recently and > tried to fix a few things. I'm not sure I helped and there are > probably more problems lurking. > > This issue may not be noticed in the default test run. It definitely > occurs when the tests are run in a random order as they are on the > buildbots. Try running: ./python.exe ./Lib/test/regrtest.py -r > > That should help provoke more errors. At the sprints, I ran into a bunch of similar errors running the test suite on my Windows Vista box, even on tests that were properly cleaning up after themselves in tearDown(). I even tried putting in sleeps as long as 1 second, to no avail. The only way to get the test suite to run without these errors was to stop the Windows Search Service, fully disable Icon Overlays for TortoiseSVN, and then close down all open folders. Any chance the boxes the tests are being run on are running the Windows Search Service or have Icon Overlays enabled for TortoiseSVN? (If anyone has any ideas of how to get around these problems, I'd love to hear them. Seems like I shouldn't have to disable these services.) Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] test_signal on osx g4
test_signal is failing on osx g4: test test_signal failed -- Traceback (most recent call last): File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 151, in test_main self.fail(tb) AssertionError: Traceback (most recent call last): File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 134, in test_main self.run_test() File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 80, in run_test child = subprocess.Popen(['kill', '-HUP', str(pid)]) File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", line 593, in __init__ errread, errwrite) File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", line 1078, in _execute_child data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB OSError: [Errno 4] Interrupted system call This is the code which reads stderr from the child process: data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB os.close(errpipe_read) if data != "": os.waitpid(self.pid, 0) child_exception = pickle.loads(data) raise child_exception I'm not an expert in this stuff my any stretch of the imagination, but shouldn't subprocess try harder to read this output? Something like: while True: try: data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB except OSError, err: if err.errno == errno.EINTR: continue else: raise else: os.close(errpipe_read) if data != "": os.waitpid(self.pid, 0) child_exception = pickle.loads(data) raise child_exception break Maybe not while True, but try a few times at least. Or is the system supposed to automatically restart interrupted system calls? Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
Steven Bethard wrote: > At the sprints, I ran into a bunch of similar errors running the test > suite on my Windows Vista box, even on tests that were properly > cleaning up after themselves in tearDown(). I even tried putting in > sleeps as long as 1 second, to no avail. The only way to get the test > suite to run without these errors was to stop the Windows Search > Service, fully disable Icon Overlays for TortoiseSVN, and then close > down all open folders. > > Any chance the boxes the tests are being run on are running the > Windows Search Service or have Icon Overlays enabled for TortoiseSVN? > > (If anyone has any ideas of how to get around these problems, I'd love > to hear them. Seems like I shouldn't have to disable these services.) I'm not sure I'm going to help here, but the reason that this happens is that certain services -- and the Desktop Search tools are notorious for this -- get file handles with FILE_SHARE_DELETE, which means that the file isn't actually removed when it's deleted; only when the last of those handles closes. I did see a suggestion somewhere that, to circumvent this issue, instead of removing it a file you need to move it to %TEMP% (or wherever) and delete it there. I'll try to knock up a test case to see if this works. TJG ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
Tim Golden wrote: > Steven Bethard wrote: >> At the sprints, I ran into a bunch of similar errors running the test >> suite on my Windows Vista box, even on tests that were properly >> cleaning up after themselves in tearDown(). I even tried putting in >> sleeps as long as 1 second, to no avail. The only way to get the test >> suite to run without these errors was to stop the Windows Search >> Service, fully disable Icon Overlays for TortoiseSVN, and then close >> down all open folders. >> >> Any chance the boxes the tests are being run on are running the >> Windows Search Service or have Icon Overlays enabled for TortoiseSVN? >> >> (If anyone has any ideas of how to get around these problems, I'd love >> to hear them. Seems like I shouldn't have to disable these services.) > > I'm not sure I'm going to help here, but the reason that > this happens is that certain services -- and the Desktop > Search tools are notorious for this -- get file handles with > FILE_SHARE_DELETE, which means that the file isn't actually > removed when it's deleted; only when the last of those handles > closes. > > I did see a suggestion somewhere that, to circumvent this issue, > instead of removing it a file you need to move it to %TEMP% > (or wherever) and delete it there. I'll try to knock up a test > case to see if this works. OK: to confirm, the following test seems to indicate that doing a (local - same volume) rename followed by a remove will do the right thing, even when a FILE_SHARE_DELETE handle is held. If this is the thing to do, presumably test_support should grow a "remove_file" which does something of this sort? TJG import os, sys import win32file FILENAME = "test" def rename_and_remove (filename): os.rename (filename, filename + ".deleted") os.remove (filename + ".deleted") def remove_only (filename): os.remove (filename) def test (remove): open (FILENAME, "w").close () hFile = win32file.CreateFile ( FILENAME, win32file.GENERIC_READ, win32file.FILE_SHARE_DELETE, None, win32file.OPEN_EXISTING, 0, 0 ) try: remove (FILENAME) try: open (FILENAME, "w").close () except IOError: print "Couldn't recreate" else: print "Could recreate" finally: hFile.Close () try: open (FILENAME, "w").close () except IOError: print "Couldn't recreate" else: print "Could recreate" if __name__ =='__main__': print print "Should not work" test (remove_only) print print "Should work" test (rename_and_remove) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
2008/4/1, Tim Golden <[EMAIL PROTECTED]>: > If this is the thing to do, presumably test_support should > grow a "remove_file" which does something of this sort? +1 (I was thinking exactly that). Regards, -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista <[EMAIL PROTECTED]> wrote: > 2008/4/1, Tim Golden <[EMAIL PROTECTED]>: > > If this is the thing to do, presumably test_support should > > grow a "remove_file" which does something of this sort? > > +1 (I was thinking exactly that). +1 here too. That looks like a great solution. Of course, once it's in test_support, we need to fix *all* file removals in the test suite. ;-) Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Tracing at a more granular level (branch coverage)
Howdy Folks- I hope this is not too off topic. Sadly the current sys.settrace only allows tracing at the line level. This isn't sufficient to do branch and path coverage. The main problem is that many boolean operations can appear on a single line, and so you can't be sure which conditional branch was taken using the current tracing method. After speaking about code complexity/testing at Pycon [0], with a few people there, and seeing a few people have made some strides or shown interested in metrics/code graphs/flow and branch coverage I think that there is interest in this. One proposed method of getting branch coverage was to use Dalke's Python4Ply [1] to translate code so that all branches occur on their own line. Then using line coverage on that and converting it back to branch coverage of the original code. The maintainer of coverage.py suggested that we look into patching python instead to trace at a more granular level. His feeling was that there would be too many corner cases and the translation would get hairy quite quickly. Sadly in my 8 years of python experience I've yet to touch any c based guts of python. I'm looking for advice on how to get finer grain tracing with sys.settrace. Any advice or suggestions? There is a quorum of people (at least 5 others) who would be very interested in this functionality because it could lead to some cool tools built on top of it. (With the end goal that python code be cleaner, simpler and better tested). thanks much, -matt harrison 0 - http://panela.blog-city.com/pycon_2008_managing_complexity_slides.htm 1 - http://www.dalkescientific.com/Python/python4ply.html ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
On 1 Apr, 18:27, "Steven Bethard" <[EMAIL PROTECTED]> wrote: > On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista > > <[EMAIL PROTECTED]> wrote: > > 2008/4/1, Tim Golden <[EMAIL PROTECTED]>: > > > If this is the thing to do, presumably test_support should > > > grow a "remove_file" which does something of this sort? > > > +1 (I was thinking exactly that). > > +1 here too. That looks like a great solution. Of course, once it's > in test_support, we need to fix *all* file removals in the test suite. > ;-) > > Steve Why not just modifying test_support.unlink() like this? Surely more convenient than modifying the whole suite. def unlink(filename): try: if os.name == 'nt': os.rename(filename, filename + ".deleted") filename = filename + ".deleted" os.unlink(filename) except OSError: pass ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
Giampaolo Rodola' wrote: > > On 1 Apr, 18:27, "Steven Bethard" <[EMAIL PROTECTED]> wrote: >> On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista >> >> <[EMAIL PROTECTED]> wrote: >>> 2008/4/1, Tim Golden <[EMAIL PROTECTED]>: >>> > If this is the thing to do, presumably test_support should >>> > grow a "remove_file" which does something of this sort? >>> +1 (I was thinking exactly that). >> +1 here too. That looks like a great solution. Of course, once it's >> in test_support, we need to fix *all* file removals in the test suite. >> ;-) >> >> Steve > > Why not just modifying test_support.unlink() like this? > Surely more convenient than modifying the whole suite. > > > def unlink(filename): > try: > if os.name == 'nt': > os.rename(filename, filename + ".deleted") > filename = filename + ".deleted" > os.unlink(filename) > except OSError: > pass Funnily enough, that's exactly what the patch I've put together does. Have to look out, though for anywhere which just calls os.unlink rather than going through test_support. TJG ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
On 1 Apr, 21:03, "Giampaolo Rodola'" <[EMAIL PROTECTED]> wrote: > On 1 Apr, 18:27, "Steven Bethard" <[EMAIL PROTECTED]> wrote: > > > On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista > > > <[EMAIL PROTECTED]> wrote: > > > 2008/4/1, Tim Golden <[EMAIL PROTECTED]>: > > > > If this is the thing to do, presumably test_support should > > > > grow a "remove_file" which does something of this sort? > > > > +1 (I was thinking exactly that). > > > +1 here too. That looks like a great solution. Of course, once it's > > in test_support, we need to fix *all* file removals in the test suite. > > ;-) > > > Steve > > Why not just modifying test_support.unlink() like this? > Surely more convenient than modifying the whole suite. > > def unlink(filename): > try: > if os.name == 'nt': > os.rename(filename, filename + ".deleted") > filename = filename + ".deleted" > os.unlink(filename) > except OSError: > pass Another solution, probably better: def unlink(filename): try: os.unlink(filename) except OSError: if os.name == 'nt': try: os.rename(filename, filename + ".deleted") os.unlink(filename + ".deleted") except OSError: pass ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
On 1 Apr, 21:06, Tim Golden <[EMAIL PROTECTED]> wrote: > Giampaolo Rodola' wrote: > > > On 1 Apr, 18:27, "Steven Bethard" <[EMAIL PROTECTED]> wrote: > >> On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista > > >> <[EMAIL PROTECTED]> wrote: > >>> 2008/4/1, Tim Golden <[EMAIL PROTECTED]>: > >>> > If this is the thing to do, presumably test_support should > >>> > grow a "remove_file" which does something of this sort? > >>> +1 (I was thinking exactly that). > >> +1 here too. That looks like a great solution. Of course, once it's > >> in test_support, we need to fix *all* file removals in the test suite. > >> ;-) > > >> Steve > > > Why not just modifying test_support.unlink() like this? > > Surely more convenient than modifying the whole suite. > > > def unlink(filename): > > try: > > if os.name == 'nt': > > os.rename(filename, filename + ".deleted") > > filename = filename + ".deleted" > > os.unlink(filename) > > except OSError: > > pass > > Funnily enough, that's exactly what the patch I've > put together does. Sorry but maybe I misunderstood what you said above. It seems to me you proposed to add a new "remove_file" function to test_support while the solution I suggested was modifying the current test_support.unlink() function in a similar manner you proposed and have all tests use it wherever it is possible. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
Giampaolo Rodola' wrote: >>> Why not just modifying test_support.unlink() like this? >>> Surely more convenient than modifying the whole suite. >>> def unlink(filename): >>> try: >>> if os.name == 'nt': >>> os.rename(filename, filename + ".deleted") >>> filename = filename + ".deleted" >>> os.unlink(filename) >>> except OSError: >>> pass Tim Golden >> Funnily enough, that's exactly what the patch I've >> put together does. > > Sorry but maybe I misunderstood what you said above. > It seems to me you proposed to add a new "remove_file" function to > test_support while the solution I suggested was modifying the current > test_support.unlink() function in a similar manner you proposed and > have all tests use it wherever it is possible. I wasn't very clear. What I posted earlier was by way of a proof-of-concept and did indeed use a "remove" function. After other people's +1 I have put together a possible patch which does what you're suggesting. I haven't posted it yet and I need to work through the test files checking what the impact would be. I like your second suggestion BTW, altho' I'd probably skip the "if os.name == 'nt'" line and just give it a try in any case. TJG ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
Tim Golden wrote: > Giampaolo Rodola' wrote: >> On 1 Apr, 18:27, "Steven Bethard" <[EMAIL PROTECTED]> wrote: >>> On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista >>> >>> <[EMAIL PROTECTED]> wrote: 2008/4/1, Tim Golden <[EMAIL PROTECTED]>: > If this is the thing to do, presumably test_support should > grow a "remove_file" which does something of this sort? +1 (I was thinking exactly that). >>> +1 here too. That looks like a great solution. Of course, once it's >>> in test_support, we need to fix *all* file removals in the test suite. >>> ;-) >>> >>> Steve >> Why not just modifying test_support.unlink() like this? >> Surely more convenient than modifying the whole suite. >> >> >> def unlink(filename): >> try: >> if os.name == 'nt': >> os.rename(filename, filename + ".deleted") >> filename = filename + ".deleted" >> os.unlink(filename) >> except OSError: >> pass > > Funnily enough, that's exactly what the patch I've > put together does. Have to look out, though for > anywhere which just calls os.unlink rather than > going through test_support. Which turns out to be at least 117 instances: C:\work-in-progress\python26\Lib\test>grep "os.unlink" *.py | wc -l 117 (including the patch I just posted for os.access. Oops). TJG ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_signal on osx g4
this is http://bugs.python.org/issue1068268 On Tue, Apr 1, 2008 at 5:05 PM, <[EMAIL PROTECTED]> wrote: > test_signal is failing on osx g4: > >test test_signal failed -- Traceback (most recent call last): > File > "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 151, > in test_main >self.fail(tb) >AssertionError: Traceback (most recent call last): > File > "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 134, > in test_main >self.run_test() > File > "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 80, > in run_test >child = subprocess.Popen(['kill', '-HUP', str(pid)]) > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", > line 593, in __init__ >errread, errwrite) > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", > line 1078, in _execute_child >data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB >OSError: [Errno 4] Interrupted system call > > This is the code which reads stderr from the child process: > >data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB >os.close(errpipe_read) >if data != "": >os.waitpid(self.pid, 0) >child_exception = pickle.loads(data) >raise child_exception > > I'm not an expert in this stuff my any stretch of the imagination, but > shouldn't subprocess try harder to read this output? Something like: > >while True: >try: >data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 > MB >except OSError, err: >if err.errno == errno.EINTR: >continue >else: >raise >else: >os.close(errpipe_read) >if data != "": >os.waitpid(self.pid, 0) >child_exception = pickle.loads(data) >raise child_exception >break > > Maybe not while True, but try a few times at least. > > Or is the system supposed to automatically restart interrupted system > calls? > > Skip > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/schmir%40gmail.com > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] xmlrpclib and dates before 1900
Hi all, anyone care to take a look at: http://bugs.python.org/issue2014 It's about xmlrpclib not being able to send datetime objects with dates before 1900. I would like to see this go in and would also like to work on http://bugs.python.org/issue1745722 (xmlrpc wsgi support). But this only makes sense if someone also commits it... Regards, - Ralf ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tracing at a more granular level (branch coverage)
Hello, On Tue, Apr 1, 2008 at 9:02 PM, m h wrote: > Howdy Folks- > > I hope this is not too off topic. Sadly the current sys.settrace only > allows tracing at the line level. This isn't sufficient to do branch > and path coverage. The main problem is that many boolean operations > can appear on a single line, and so you can't be sure which > conditional branch was taken using the current tracing method. > > After speaking about code complexity/testing at Pycon [0], with a few > people there, and seeing a few people have made some strides or shown > interested in metrics/code graphs/flow and branch coverage I think > that there is interest in this. > > One proposed method of getting branch coverage was to use Dalke's > Python4Ply [1] to translate code so that all branches occur on their > own line. Then using line coverage on that and converting it back to > branch coverage of the original code. > > The maintainer of coverage.py suggested that we look into patching > python instead to trace at a more granular level. His feeling was > that there would be too many corner cases and the translation would > get hairy quite quickly. > > Sadly in my 8 years of python experience I've yet to touch any c based > guts of python. I'm looking for advice on how to get finer grain > tracing with sys.settrace. > > Any advice or suggestions? There is a quorum of people (at least 5 > others) who would be very interested in this functionality because it > could lead to some cool tools built on top of it. (With the end goal > that python code be cleaner, simpler and better tested). You can start by looking at the file Python/ceval.c, which contains the interpreter loop. More precisely, the function maybe_call_line_trace() is responsible to call sys.settrace. I think you will have to change the logic there. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] xmlrpclib and dates before 1900
> anyone care to take a look at: > http://bugs.python.org/issue2014 > It's about xmlrpclib not being able to send datetime objects with dates > before 1900. > I would like to see this go in and would also like to work on > http://bugs.python.org/issue1745722 > (xmlrpc wsgi support). > But this only makes sense if someone also commits it... Can you please explain why this is an important problem? Dates before 1900 have all passed long ago, so they shouldn't occur that often in real applications. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] FW: [issue2513] 64bit cross compilation on windows
> Currently, the "official" (by way of being de-facto) directory structure for > a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 > platforms. I believe this might cause problems for people trying to port > their applications to 64bit platforms. My proposal is that we change this > subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while > 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy > of one of the platform subdirectories. Can you explain what you mean by "native platform"? I usually perform cross-builds for amd64, so would then this directory remain empty? > The rationale is fairly simple: I'm quite certain that this new directory > layout will break existing "native only" build processes (ie, those that > aren't aware of cross-compilation). I'm quite certain that Mozilla will > break, for example, and no cross-compilation process exists that I am aware > of. Existing build tools already know about the PCBuild directory, and are > focused almost exclusively towards native compilation - all such tools on > non-x86 platforms are currently broken. The reverse may also be true: some tools may expect PCbuild to contain always x86 binaries, even on AMD64 - as they don't support non-x86 at all. Those tools might break if PCbuild suddenly starts containing AMD64 binaries. If this is all about *just* Mozilla, then I'd be much more in favour than if it was for some theoretical package. So if Mozilla builds with that kind of setup, and works correctly - go for it. If you have to fix Mozilla anyway, consider fixing it so it looks in the amd64 directory. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Thread-safe file objects, the return
This is not something that keeps me awake at night, but I am aware of it. Your solution (a counter) seems fine except I think perhaps the close() call should not raise IOError -- instead, it should set a flag so that the thread that makes the counter go to zero can close the thread (after all the file got closed while it was being used). There are of course other concurrency issues besides close -- what if two threads both try to do I/O on the file? What will the C stdio library do in that case? Are stdio files thread-safe at the C level? So (classically contradicting myself while I think the problem over more) perhaps any I/O operation should be disallowed while the file is in use by another thread? --Guido On Mon, Mar 31, 2008 at 1:09 PM, Antoine Pitrou <[EMAIL PROTECTED]> wrote: > > Hello, > > It seems this subject has had quite a bit of history. Tim Peters demonstrated > the problem in 2003 in this message: > http://mail.python.org/pipermail/python-dev/2003-June/036537.html > > In short, Python file objects release the GIL before calling any C stdlib > function on their embedded FILE pointer. Unfortunately, if another thread > calls fclose on the FILE pointer concurrently, the contents pointed to can > become garbage and the interpreter process crashes. Just by using the same > file object in two threads running pure Python code, you can crash the > interpreter. > > (another, easier-to-solve problem is that the FILE pointer stored in the > file object could become NULL at the point it is used by another thread. > If that was the only problem you could just store the FILE pointer in a > local variable before releasing the GIL et voilà) > > There was some discussion at the time about the possible resolution. I've > tried to fix the problem, and I've come to what I think is a satisfying > solution, which I can sum up as the following bullet points: > * Each file object gets a dedicated counter, which is incremented before > the bject releases the GIL and decremented after the GIL is taken again; thus > this counter keeps track of how many running "unlocked" sections of code are > using that particular file object. (please note the counter doesn't need its > own lock, since it is only modified in GIL-protected sections) > * In the close() method, if the aforementioned counter is greater than 0, > we refuse to call fclose and instead raise an IOError. > > This may seem like a worrying semantic change, but I don't think it is, for > the > following reasons: > 1) if we closed the FILE pointer anyway, the interpreter would likely crash > because another thread would be using garbage data (that's what we are trying > to fix after all!) > 2) if close() raises an IOError, it can be called again later, or at worse > fclose will be called when the file object is garbage collected > 3) close() can already raise an IOError if fclose fails for whatever reason > (although for sure it's probably very rare) > 4) it doesn't seem wrong to notify the programmer that his code is very > unsafe > > The patch is attached at http://bugs.python.org/issue815646 . It addresses > (or at least I hope it does) all potential problems with pure Python code, > threads, and the file object. It doesn't try to fix C extensions using the > PyFile_AsFile API and doing their own dirty things with the FILE pointer. It > could be a second step if the approach is accepted, but as noted in the 2003 > discussions it would probably involve a new API. Whether we want to introduce > such an API in Python 2.x while Python 3.0 has a different IO model anyway > is left open to discussion :) > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] FW: [issue2513] 64bit cross compilation on windows
Martin quoting me: > > Currently, the "official" (by way of being de-facto) directory > > structure for a build tree is 'PCBuild/.' for x86 builds and > > 'PCBuild/amd64' for x64 > > platforms. I believe this might cause problems for people trying to > > port their applications to 64bit platforms. My proposal is that we > > change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' > > exist, while 'PCBuild/.' is always the 'native' platform - ie, > > that 'PCBuild/.' be a copy of one of the platform subdirectories. > > Can you explain what you mean by "native platform"? I usually perform > cross-builds for amd64, so would then this directory remain empty? By "native platform", I mean it contains binaries that are native for the current platform. IOW, on x86 platforms, PCBuild would contain x86 binaries, and binaries compiled for AMD64 would live in the AMD64 directory. If in a given directory tree, you *only* did a cross-compile, then yes, PCBuild would remain empty - but I don't think that is a problem; tools that know how to cross-compile (such as the distutils after my patch is applied) will not look in the PCBuild directory, and tools that do not cross-compile will be unable to use the x64 binaries from the x86 platform anyway. In other words, I can't think of anything that would break by having an empty PCBuild directory in a tree that only contains binaries that can't be executed on the current platform. > > The rationale is fairly simple: I'm quite certain that this new > > directory layout will break existing "native only" build processes > > (ie, those that aren't aware of cross-compilation). I'm quite > > certain that Mozilla will break, for example, and no > > cross-compilation process exists that I am aware > > of. Existing build tools already know about the PCBuild directory, > > and are focused almost exclusively towards native compilation - all > > such tools on non-x86 platforms are currently broken. > > The reverse may also be true: some tools may expect PCbuild to contain > always x86 binaries, even on AMD64 - as they don't support non-x86 at > all. Those tools might break if PCbuild suddenly starts containing > AMD64 binaries. I agree. However, it is my assertion that there are very few build tools which expect the layout you describe, simply as the scheme has only been around for a few months, and only in Python 2.6 builds. Further, I assert that there are a greater number of build tools which do not support cross-compilation, but will build natively on x64 and expect 'PCBuild' to have libraries they can link with to create an x64 binary. I accept my assertions may be incorrect - in which case keeping the existing layout is indeed the best options. But I assert that my assertions are correct :) > If this is all about *just* Mozilla, then I'd be much more in favour > than if it was for some theoretical package. So if Mozilla builds with > that kind of setup, and works correctly - go for it. If you have to > fix Mozilla anyway, consider fixing it so it looks in the amd64 > directory. Nope - it's not about *just* Mozilla at all, and I don't expect that I personally will be involved in any 64bit issues for that platform. That just happens to be one build process that I know about, but I except it does some "typical" things other tools might do - like executing the python executable to sniff sys.prefix and looking for (eg) "Libs" and "PCBuild" to locate libraries to link with seems fairly likely to me. Even looking for the executable itself directly in PCBuild seems likely. On an x64 platform, such tools may well find the 32bit version, and it may even seem to work - but it is unlikely to be doing what they expect. Or they may find no binary at all. It seems that you previously agreed with this. From http://mail.python.org/pipermail/distutils-sig/2007-May/007668.html; you are replying to (I think) Kristján: | > I am baffled about why the build environment's layout matters, | > but once an .msi install can place the binaries in any | > old place it wants. The build structure doesn't have to | > reflect the final installed structure at all. | No. But still, people like to be able to "run" Python out of | a source check-out. This has been supported for a long time, | and more and more stuff was added to support it. For examples | within Python itself, see the support in distutils, getpathp.c, | PCbuild/rt.bat, and Tools/buildbot/*.bat. Reportedly (by | Mark), building Mozilla (the web browser) also "knows" | about PCbuild. Your comments exactly reflect my concern, and I believe them relevant for people working with native binaries on x64. But as implied above, I actually have zero personal interest in this at the moment - unlike the distutils cross-compile patch, which does scratch an itch of mine. It just seems it would make life easier for people down the track - but I'm happy to let it go if your position has changed or I have mis-understood it
Re: [Python-Dev] [Distutils] FW: [issue2513] 64bit cross compilation on windows
>> The reverse may also be true: some tools may expect PCbuild to contain >> always x86 binaries, even on AMD64 - as they don't support non-x86 at >> all. Those tools might break if PCbuild suddenly starts containing >> AMD64 binaries. > > I agree. However, it is my assertion that there are very few build tools > which expect the layout you describe, simply as the scheme has only been > around for a few months, and only in Python 2.6 builds. That PCbuild exists and contains x86 binaries? This scheme has been in place ever since the PCbuild directory got created, several years ago! > Nope - it's not about *just* Mozilla at all, and I don't expect that I > personally will be involved in any 64bit issues for that platform. That > just happens to be one build process that I know about, but I except it does > some "typical" things other tools might do - like executing the python > executable to sniff sys.prefix and looking for (eg) "Libs" and "PCBuild" to > locate libraries to link with seems fairly likely to me. Even looking for > the executable itself directly in PCBuild seems likely. On an x64 platform, > such tools may well find the 32bit version, and it may even seem to work - > but it is unlikely to be doing what they expect. Unless the *want* the x86 binaries, of course, which always had been in this place. > Your comments exactly reflect my concern, and I believe them relevant for > people working with native binaries on x64. No, I never cared about what binaries are "native". Instead, I wanted to see those binaries in PCbuild which had been built last. If you just built the x86 binaries, PCbuild should contain x86 binaries, even if you were running Win64. This is different from what you propose, but only slightly so (but perhaps importantly, depending on the scenario). > But as implied above, I > actually have zero personal interest in this at the moment - unlike the > distutils cross-compile patch, which does scratch an itch of mine. It just > seems it would make life easier for people down the track - but I'm happy to > let it go if your position has changed or I have mis-understood it. OK, if we don't have an actual use case, I suggest to leave things as is. People running into this problem should propose solutions themselves. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_signal on osx g4
I've tried to fix test_signal at least. For those particular calls, EINTR means that the kill I launched finished before I was expecting, so we can ignore it without retrying. Figuring out subprocess in general is a worthy goal but shouldn't block making the test less flaky. :) r62102. On Tue, Apr 1, 2008 at 8:05 AM, <[EMAIL PROTECTED]> wrote: > test_signal is failing on osx g4: > > test test_signal failed -- Traceback (most recent call last): > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", > line 151, in test_main > self.fail(tb) > AssertionError: Traceback (most recent call last): > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", > line 134, in test_main > self.run_test() > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", > line 80, in run_test > child = subprocess.Popen(['kill', '-HUP', str(pid)]) > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", line > 593, in __init__ > errread, errwrite) > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", line > 1078, in _execute_child > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > OSError: [Errno 4] Interrupted system call > > This is the code which reads stderr from the child process: > > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > os.close(errpipe_read) > if data != "": > os.waitpid(self.pid, 0) > child_exception = pickle.loads(data) > raise child_exception > > I'm not an expert in this stuff my any stretch of the imagination, but > shouldn't subprocess try harder to read this output? Something like: > > while True: > try: > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > except OSError, err: > if err.errno == errno.EINTR: > continue > else: > raise > else: > os.close(errpipe_read) > if data != "": > os.waitpid(self.pid, 0) > child_exception = pickle.loads(data) > raise child_exception > break > > Maybe not while True, but try a few times at least. > > Or is the system supposed to automatically restart interrupted system calls? > > Skip > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com > -- Namasté, Jeffrey Yasskin http://jeffrey.yasskin.info/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Thread-safe file objects, the return
On Tue, Apr 1, 2008 at 5:09 PM, Guido van Rossum <[EMAIL PROTECTED]> wrote: > This is not something that keeps me awake at night, but I am aware of > it. Your solution (a counter) seems fine except I think perhaps the > close() call should not raise IOError -- instead, it should set a flag > so that the thread that makes the counter go to zero can close the > thread (after all the file got closed while it was being used). > No, raising IOError is the right thing to do here. The problem is that calling close on a file implies that the close actually completed in the OS when it returns. The call should not return until the file object has actually been closed in the underlying layers. You can't leave it for later to be done by one of the other currently operating threads as you violate the close semantics and lose direct indication of an error that occurred during close. > There are of course other concurrency issues besides close -- what if > two threads both try to do I/O on the file? What will the C stdio > library do in that case? Are stdio files thread-safe at the C level? > So (classically contradicting myself while I think the problem over > more) perhaps any I/O operation should be disallowed while the file is > in use by another thread? > that does sound like the safest thing to do... -gps ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing tests on windows
On Tue, Apr 1, 2008 at 9:13 AM, Tim Golden <[EMAIL PROTECTED]> wrote: > Tim Golden wrote: > > Steven Bethard wrote: > >> At the sprints, I ran into a bunch of similar errors running the test > >> suite on my Windows Vista box, even on tests that were properly > >> cleaning up after themselves in tearDown(). I even tried putting in > >> sleeps as long as 1 second, to no avail. The only way to get the test > >> suite to run without these errors was to stop the Windows Search > >> Service, fully disable Icon Overlays for TortoiseSVN, and then close > >> down all open folders. > >> > >> Any chance the boxes the tests are being run on are running the > >> Windows Search Service or have Icon Overlays enabled for TortoiseSVN? > >> > >> (If anyone has any ideas of how to get around these problems, I'd love > >> to hear them. Seems like I shouldn't have to disable these services.) > > > > I'm not sure I'm going to help here, but the reason that > > this happens is that certain services -- and the Desktop > > Search tools are notorious for this -- get file handles with > > FILE_SHARE_DELETE, which means that the file isn't actually > > removed when it's deleted; only when the last of those handles > > closes. > > > > I did see a suggestion somewhere that, to circumvent this issue, > > instead of removing it a file you need to move it to %TEMP% > > (or wherever) and delete it there. I'll try to knock up a test > > case to see if this works. > > OK: to confirm, the following test seems to indicate that > doing a (local - same volume) rename followed by a remove > will do the right thing, even when a FILE_SHARE_DELETE handle > is held. > > If this is the thing to do, presumably test_support should > grow a "remove_file" which does something of this sort? > > TJG > > > import os, sys > import win32file > > FILENAME = "test" > > def rename_and_remove (filename): > os.rename (filename, filename + ".deleted") > os.remove (filename + ".deleted") Isn't this still going to run into problems when the rename fails because the earlier tests remove still left the .deleted file around due to some other running desktop search service that now has the .deleted file open? +1 on not supporting OSes with such bad filesystem semantics? ;) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
