Re: [Python-Dev] [Python-checkins] r45334 - python/trunk/Lib/test/leakers/test_gen1.py python/trunk/Lib/test/leakers/test_generator_cycle.py python/trunk/Lib/test/leakers/test_tee.py
On 4/13/06, Tim Peters [EMAIL PROTECTED] wrote: Rather than delete a leak test, perhaps we could simply move it into a new old-leaking-tests subdirectory? Likewise for crash tests. When the bug reappears, it's helfpul to have the focussed (whittled-down) old test that provoked it handy. Good point. I updated both READMEs. Hell, one of the tests I removed still leaked. Who knows what I was smoking. The removed test that really doesn't leak was moved into test generators. I think it was different than all the other existing tests. Feel free to tweak the READMEs or policy. I agree with the intent, however it may be implemented. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Calling Thomas Heller and Richard Jones
If Thomas Heller and Richard Jones haven't recently received email from me (and are reading this list, naturally) I'd appreciate it if they'd get in touch with me in the next day or so. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd www.holdenweb.com Love me, love my blog holdenweb.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 302 support for pydoc, runpy, etc.
Phillip J. Eby wrote: Anybody have any thoughts, comments, or questions? This all sounds good to me. Will it also eliminate the current quirk in the runpy emulation where it searches the metapath in a slightly different order than the actual import machinery does? (From what you wrote, I believe it will, but I'm not sure). Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int vs ssize_t in unicode
On 4/13/06, Martin v. Löwis [EMAIL PROTECTED] wrote:
Neal Norwitz wrote:
I just grepped for INT_MAX and there's a ton of them still (well 83 in
*/*.c). Some aren't an issue like posixmodule.c, those are
_SC_INT_MAX. marshal is probably ok, but all uses should be verified.
Really all uses of {INT,LONG}_{MIN,MAX} should be verified and
converted to PY_SSIZE_T_{MIN,MAX} as appropriate.
BTW, it would be great if someone could try to put together some tests
for bigmem machines. I'll add it to the todo wiki. The tests should
be broken up by those that require 2+ GB of memory, those that take
4+, etc. Many people won't have boxes with that much memory.
The test cases should test all methods (don't forget slicing
operations) at boundary points, particularly just before and after
2GB. Strings are probably the easiest. There's unicode too. lists,
dicts are good but will take more than 16 GB of RAM, so those can be
pushed out some.
I have some machines available for testing.
I replaced all the trivial ones; the remaining ones are (IMO) more
involved, or correct. In particular:
- collectionsmodule: deque is still restricted to 2GiB elements
- cPickle: pickling does not support huge strings (and probably
shouldn't); likewise marshal
- _sre is still limited to INT_MAX completely
I've got outstanding changes somewhere to clean up _sre.
- not sure why the mbcs codec is restricted to INT_MAX; somebody
should check the Win64 API whether the restriction can be
removed (most likely, it can)
- PyObject_CallFunction must be duplicated for PY_SSIZE_T_CLEAN,
then exceptions.c can be extended.
My new favorite static analysis tool is grep:
grep '(int)' */*.c | egrep -v 'sizeof(int)' | wc -l
418
I know a bunch of those aren't problematic, but a bunch are. Same
with long casts.
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r45321 - in python/trunk: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS
On 4/14/06, Anthony Baxter [EMAIL PROTECTED] wrote: On Friday 14 April 2006 02:31, Martin v. Löwis wrote: Tim Peters wrote: I'm not the one to decide, but at some time the traceback module should be rewritten to match the interpreter behavior. No argument from me about that. I also think the traceback module should be corrected, and the test cases updated, despite the objections that it may break other people's doctest test cases. Let me chime in with agreement (2.5 only of course). I don't mind one way or the other, but with the number of people working actively on the code at the moment, I think reverting broken patches that don't have trivial test fixes is the way to go. The buildbot system is useless, otherwise. That was the right thing to do (short of fixing all the missing tests, which requires actual thinking :-). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unicode vs buffer (array) design issue can crash interpreter
Neal Norwitz wrote: I'll leave this decision to Martin or someone else, since I'm not familiar with the ramifications. Since it was documented as unsigned, I think it's reasonable to consider changing. Though it could create signed-ness warnings in other modules. I'm not sure but it's possible it could create problems for C++ compilers since they are pickier. My concern is not so much that it becomes unsigned in 2.4.4, but that it stops being a typedef for wchar_t on Linux. C++ code that uses that assumption might stop compiling. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Procedure for sandbox branches in SVN?
Tim Peters wrote: Just do it. Branches under SVN are cheap, go fast, and are pretty easy to work with. Even better, because a branch is just another named directory in SVN's virtual file system, you can svn remove it when you're done with it (just like any other directory). That's all true. Of course, removing the branch won't free up any disk space - it will just remove the branch from the view (IOW, it is also easy to bring it back if it was removed mistakenly). Regards, Martn ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unicode vs buffer (array) design issue can crash interpreter
Martin v. Löwis wrote: Neal Norwitz wrote: I'll leave this decision to Martin or someone else, since I'm not familiar with the ramifications. Since it was documented as unsigned, I think it's reasonable to consider changing. Though it could create signed-ness warnings in other modules. I'm not sure but it's possible it could create problems for C++ compilers since they are pickier. My concern is not so much that it becomes unsigned in 2.4.4, but that it stops being a typedef for wchar_t on Linux. C++ code that uses that assumption might stop compiling. I'd argue that such code is broken anyway: 3rd party code simply cannot make any assumptions on the typedef behind Py_UNICODE. Note that you'd only see this change when compiling Python in the non-standard UCS4 setting on Linux. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 14 2006) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unicode vs buffer (array) design issue can crash interpreter
M.-A. Lemburg wrote: I'd argue that such code is broken anyway: 3rd party code simply cannot make any assumptions on the typedef behind Py_UNICODE. The code would work just fine now, but will stop working with the change. Users of the code might not be open to arguments. In any case, it's up to the release manager to decide. Note that you'd only see this change when compiling Python in the non-standard UCS4 setting on Linux. Sure. That happens to be the default on many Linux distributions, though. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Procedure for sandbox branches in SVN?
Tim Peters wrote: Just do it. Branches under SVN are cheap, go fast, and are pretty easy to work with. Even better, because a branch is just another named directory in SVN's virtual file system, you can svn remove it when you're done with it (just like any other directory). footnote: if you want to make huge amounts of branches (e.g. a branch for each bug or optimization idea you're working on), I recommend using the SVK frontend instead of SVN: http://svk.elixus.org/ /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New-style icons, .desktop file
Morning! I've done some tweaks to the previously-posted-about icon set, taking note of some of the comments here and on -list. In particular, amongst more minor changes: - added egg icon (based on zip) - flipped pycon to work better with shortcut arrow - emphasised borders of 32x32 version of pycon, and changed text colour, in order to distinguish more from pyc. In balance I think this is enough of a change, though Nick Coghlan's idea of just using the plus in this case also has merit - indeed this is what happens at 16x16 - built .ico files without the Windows Vista enormo-icons. If the icons *were* to go in the win32 distribution, it probably wouldn't make sense to spend the considerably larger filesize on Vista icons until such time as people are actually using Vista. - included PNG and SVG version of icons. The SVG unfortunately doesn't preserve all the effects of the original Xara files, partly because a few effects (feathering, bevels) can't be done in SVG 1.1, and partly because the current conversion process is bobbins (but we're working on that!). So the nice gradients and things are gone, but it should still be of use as a base for anyone wanting to hack on the icons. - excised file() in favour of open() ;-) Files and preview here: http://doxdesk.com/img/software/py/icons2.zip http://doxdesk.com/img/software/py/icons2.png Oh, and is the intention to deprecate the purple/green horizontal snake logo previously used for 'Python for Windows' (as well as PSF)? If so, Erik van B's installer graphic could probably do with a quick refresh. cheers, -- And Clover mailto:[EMAIL PROTECTED] http://www.doxdesk.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unicode vs buffer (array) design issue can crash interpreter
Martin v. Löwis wrote: M.-A. Lemburg wrote: I'd argue that such code is broken anyway: 3rd party code simply cannot make any assumptions on the typedef behind Py_UNICODE. The code would work just fine now, but will stop working with the change. Users of the code might not be open to arguments. Fair enough. Let's leave things as they are for 2.4, then. In any case, it's up to the release manager to decide. Note that you'd only see this change when compiling Python in the non-standard UCS4 setting on Linux. Sure. That happens to be the default on many Linux distributions, though. Unfortunately, that's true. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 14 2006) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New-style icons, .desktop file
Andrew Clover wrote: Morning! I've done some tweaks to the previously-posted-about icon set, taking note of some of the comments here and on -list. And I still think they're pretty :) - emphasised borders of 32x32 version of pycon, and changed text colour, in order to distinguish more from pyc. In balance I think this is enough of a change, though Nick Coghlan's idea of just using the plus in this case also has merit - indeed this is what happens at 16x16 A small problem with including the interactive prompt as part of that icon is that it leaves us without an appropriate icon for pythonw (the Windows .exe to start Python GUI applications without displaying a console window). The plain Plus-Python icon would work in both cases (the normal python executable and the pythonw variant). Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] IMPORTANT 2.5 API changes for C Extension Modules and Embedders
If you don't write or otherwise maintain Python Extension Modules written in C (or C++) or embed Python in your application, you can stop reading. Python 2.5 alpha 1 was released April 5, 2006. The second alpha should be released in a few weeks. There are several changes which can cause C extension modules or embedded applications to crash the interpreter if not fixed. Periodically, I will send out these reminders with updated information until 2.5 is released. * support for 64-bit sequences (eg, 2GB strings) * memory allocation modifications 64-bit changes -- There are important changes that are in 2.5 to support 64-bit systems. The 64-bit changes can cause Python to crash if your module is not upgraded to support the changes. Python was changed internally to use 64-bit values on 64-bit machines for indices. If you've got a machine with more than 16 GB of RAM, it would be great if you can test Python with large ( 2GB) strings and other sequences. For more details about the Python 2.5 schedule: http://www.python.org/dev/peps/pep-0356/ For more details about the 64-bit change: http://www.python.org/dev/peps/pep-0353/ How to fix your module: http://www.python.org/dev/peps/pep-0353/#conversion-guidelines The effbot wrote a program to check your code and find potential problems with the 64-bit APIs. http://svn.effbot.python-hosting.com/stuff/sandbox/python/ssizecheck.py Memory Allocation Modifications --- In previous versions of Python, it was possible to use different families of APIs (PyMem_* vs. PyObject_*) to allocate and free the same block of memory. APIs in these families include: PyMem_*:PyMem_Malloc, PyMem_Realloc, PyMem_Free, PyObject_*: PyObject_Malloc, PyObject_Realloc, PyObject_Free There are a few other APIs with similar names and also the macro variants. In 2.5, if allocate a block of memory with one family, you must reallocate or free with the same family. That means: If you allocate with PyMem_Malloc (or MALLOC), you must reallocate with PyMem_Realloc (or REALLOC) and free with PyMem_Free (or FREE). If you allocate with PyObject_Malloc (or MALLOC), you must reallocate with PyObject_Realloc (or REALLOC) and free with PyObject_Free (or FREE). Using inconsistent APIs can cause double frees or otherwise crash the interpreter. It is fine to mix and match functions or macros within the same family. Please test and upgrade your extension modules! Cheers, n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 359: The make Statement
Travis I generally like the idea. A different name would be better. Travis Here's a list of approximate synonyms that might work (ordered Travis by my preference...) ... lots of suggestions elided ... None of the alternatives seem better to me than make or create. In fact, many don't seem to me (as a native English speaker) to be fairly obscure synonyms. construct or build are maybe the best of the lot if neither make nor create are deemed worthy. I'm willing to put up with the pain of some breakage to have a better keyword. That's what __future__ imports are for. We need to consider that Python exists outside the English-speaking world. Making a strong connection between the keyword and its action is important. For that reason I'd much prefer simple, common keywords. Yes, I realize that some keywords in Python are more symbolic than that: def, lambda, del, elif, exec. But most keywords in Python are common English, understood by almost anyone having any facility with the language. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New-style icons, .desktop file
On Friday 14 April 2006 06:35, Andrew Clover wrote: Files and preview here: http://doxdesk.com/img/software/py/icons2.zip http://doxdesk.com/img/software/py/icons2.png Very nice! -Fred -- Fred L. Drake, Jr. fdrake at acm.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Procedure for sandbox branches in SVN?
[Martin v. Löwis] That's all true. Of course, removing the branch won't free up any disk space - it will just remove the branch from the view (IOW, it is also easy to bring it back if it was removed mistakenly). Right! I'm implicitly addressing a different issue, namely that two reasons for people disliking branches in CVS can go away by magic under SVN: 1. There's no way to attach a comment to a branch, explaining why it exists. In SVN, that's the checkin comment on the `copy` command that creates the branch. 2. Branches are in your face forever after, because the CVS docs strongly warn against trying to use the relevant commands even for tags (let alone branches). The CVS moral equivalent of svn list repo/branches keeps growing as a result, and shows up in each `cvs log`. The ever-growing clutter of long-obsolete info gets depressing 0.5 wink. The easy possibility in SVN for out of sight, out of mind is a huge improvement. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Debugging opportunity :-)
When possible finalizers were added to generators, the implementation at first said that all generators need finalizing. As a result, some old tests in test_generators.py started leaking cyclic trash. As a result of that, someone(s) added a number of explicit close gimmicks to those tests, to break the cycles and stop the leaks. Recently smarter gc code was added, so that most old generators should no longer say they need finalization. But the new close-it cruft added to test_generators wasn't removed, so I thought I'd try that. Alas, they still leaked without the close-it cruft. That eventually pointed to an off-by-one error in the new PyGen_NeedsFinalizing(), where reading up trash made it very likely that old generators containing an active loop would falsely claim they need finalization. Alas, fix that, and Python segfaults when running test_generators. I'm out of time for this now, so wrote up what I know, and attached a patch sufficient to reproduce it: http://www.python.org/sf/1470508 It's nominally assigned to Phillip, but for anyone who enjoys a significant debugging challenge, it's likely to be much more fun than skinning the Easter Bunny :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r45321 - in python/trunk: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS
Sorry, please ignore the post of mine I'm replying to here. I missed part of the thread, and Tim has already answered my question... On Fri, 14 Apr 2006, John J Lee wrote: [...] Assuming this is fixed in 2.5 final, is there some way to write doctests that work on both 2.4 and 2.5? If not, should something like doctest.IGNORE_EXCEPTION_DETAIL be added -- say IGNORE_EXCEPTION_MODULE? [...] ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Debugging opportunity :-)
At 03:49 PM 4/14/2006 -0400, Tim Peters wrote: That eventually pointed to an off-by-one error in the new PyGen_NeedsFinalizing(), where reading up trash made it very likely that old generators containing an active loop would falsely claim they need finalization. Alas, fix that, and Python segfaults when running test_generators. Even with the close() gimmicks still in place? If so, it sounds like the safest thing for a2 might be to claim that generators always need finalizing. :( ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 359: The make Statement
BJörn Lindqvist wrote: [nice way to declare properties with make] Of course, properties are only one of the many possible uses of the make statement. The make statement is useful in essentially any situation where a name is associated with a namespace. So, for So far, in this thread that is the only useful use of the make statement that has been presented. I'd like to see more examples. In SQLObject I would prefer: class Foo(SQLObject): make IntCol bar: notNull = True In FormEncode I would prefer: make Schema registration: make String name: max_length = 100 not_empty = True make PostalCode postal_code: not_empty = True make Int age: min = 18 In another thread on the python-3000 list I suggested (using : class Point(object): make setonce x: x coordinate make setonce y: y coordinate For a read-only x and y property (setonce because they have to be set to *something*, but just never re-set). Interfaces are nice: make interface IValidator: make attribute if_empty: If this attribute is not NoDefault, then this value will be used in lieue of an empty value default = NoDefault def to_python(value, state): ... Another descriptor, stricttype (http://svn.colorstudy.com/home/ianb/recipes/stricttype.py): class Pixel(object): make stricttype x: type = int make stricttype y: type = int (Both this descriptor and setonce need to know their name if they are going to store their value in the object in a stable location) It would be really cool if you could go through the standard library, and replace code there with code using the make statement. I think a patch showing how much nicer good Python code would be with the make statement would be a very convincing argument. I don't know if the standard library will have a whole lot; make is really only useful when frameworks are written to use it, and there's just not a lot of framework in the standard library. Maybe: make OptionParser myparser: make Option verbose: short = '-v' help = ... -- Ian Bicking / [EMAIL PROTECTED] / http://blog.ianbicking.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Debugging opportunity :-)
On 4/14/06, Phillip J. Eby [EMAIL PROTECTED] wrote: At 03:49 PM 4/14/2006 -0400, Tim Peters wrote: That eventuallypointed to an off-by-one error in the new PyGen_NeedsFinalizing(),where reading up trash made it very likely that old generatorscontaining an active loop would falsely claim they need finalization. Alas, fix that, and Python segfaults when running test_generators.Even with the close() gimmicks still in place?If so, it sounds like thesafest thing for a2 might be to claim that generators always need finalizing.:(No, it sounds like there's a bug in generator cleanup :-) From Tim's description, it isn't obvious to me that the bug is in the hack to clean up simple generator frames. I had finding generator-cycle leak in test_generators on my TODO list (as I'd noticed it was still leaking, too), I've now replaced that with finding the generator-cleanup crash Tim encountered :-) I don't know if I'll get to it this weekend, though, so if anyone else wants to hunt for it, don't let me stop you. -- Thomas Wouters [EMAIL PROTECTED]Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Debugging opportunity :-)
At 11:51 PM 4/14/2006 +0200, Thomas Wouters wrote: On 4/14/06, Phillip J. Eby mailto:[EMAIL PROTECTED][EMAIL PROTECTED] wrote: At 03:49 PM 4/14/2006 -0400, Tim Peters wrote: That eventually pointed to an off-by-one error in the new PyGen_NeedsFinalizing(), where reading up trash made it very likely that old generators containing an active loop would falsely claim they need finalization. Alas, fix that, and Python segfaults when running test_generators. Even with the close() gimmicks still in place? If so, it sounds like the safest thing for a2 might be to claim that generators always need finalizing. :( No, it sounds like there's a bug in generator cleanup :-) From Tim's description, it isn't obvious to me that the bug is in the hack to clean up simple generator frames. I had finding generator-cycle leak in test_generators on my TODO list (as I'd noticed it was still leaking, too), I've now replaced that with finding the generator-cleanup crash Tim encountered :-) I don't know if I'll get to it this weekend, though, so if anyone else wants to hunt for it, don't let me stop you. FYI, the smallest code I've found so far that reproduces the crash is: fun_tests = class LazyList: ... def __init__(self, g): ... self.v = None ... self.g = g ... ... def __iter__(self): ... yield 1 ... if self.v is None: ... self.v = self.g.next() ... yield self.v def loop(): ... for i in ll: ... yield i ll = LazyList(loop()) g=iter(ll) g.next() 1 g.next() 1 If you replace the fun_tests in test_generators with this, and remove all the other tests (e.g. pep_tests, tut_tests, etc.) you can still get it to crash. The code above is much shorter, but has the disadvantage that I don't really know what it's supposed to do, other than cause a crash. :) Interestingly, this code does *not* crash the interpreter on its own, only when run as a doctest (with or without regrtest). I still haven't figured out what the actual problem is, but at least this cuts out all the merge() and times() crud in the case this was derived from, reducing it to a pure zen essence of non-meaning. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Debugging opportunity :-)
On 4/15/06, Phillip J. Eby [EMAIL PROTECTED] wrote: Interestingly, this code does *not* crash the interpreter on its own, onlywhen run as a doctest (with or without regrtest).Actually, it does, you just have to force GC to clean up stuff (rather than wait for the interpreter-exit to just toss the whole lot out without giving it a second glance.) You do that by making it unreachable and calling gc.collect().centurion:~/python/python/trunk cat test_gencrash.pyimport gcclass LazyList: def __init__(self, g): self.v = None self.g = g def __iter__(self): yield 1 if self.v is None: self.v = self.g.next() yield self.vdef loop(): for i in ll: yield ill = LazyList(loop())g=iter(ll)g.next ()g.next()del g, llprint gc.collect()centurion:~/python/python/trunk ./python test_gencrash.pySegmentation fault (core dumped) I still haven't figuredout what the actual problem is, but at least this cuts out all the merge()and times() crud in the case this was derived from, reducing it to a purezen essence of non-meaning.:) I'm sure I know where the crash is coming from: one of the generator objects is being dealloced twice. The GC module tp_clears one of the frame objects, which deallocs a generator the frame referenced. That first dealloc calls _Py_ForgetReference and then calls gen_dealloc, which calls its tp_del (as it's a running generator, albeit a simple one), which raises an exception in the frame, which causes it to be cleaned up, which causes another generator object to be cleaned up and its tp_del method to be called, which - for some reason - tries to dealloc the first generator again. The second dealloc calls _Py_ForgetReference again, and _Py_ForgetReference doesn't like that (the ob_prev/ob_next debug ptrs are already wiped when it's called the second time, so it crashes.) The thing I don't understand is how the cleaning up of the second generator triggers a dealloc of the first generator. The refcount is 0 when the dealloc is called the first time (or _Py_ForgetReference, as well as a few other places, would have bombed out), and it is also 0 when it gets dealloced the second time. Since deallocation is triggered by DECREF'ing, that implies something is INCREF'ing the first generator somewhere -- but without having a valid reference, since the generator has been cleaned up by actually DECREF'ing the reference that a frame object held. Or, somehow, deallocation is triggered by something other than a DECREF. Hmm. No, the second dealloc is triggered by the Py_XDECREF in ceval.c, when clearing the stack after an exception, which is the right thing to do. Time to set some breakpoints and step through the code, I guess.(I first thought the problem was caused by gen_dealloc doing 'Py_DECREF(gen-gen_frame)' instead of 'frame = gen-gen_frame; gen-gen_frame = NULL; Py_DECREF(frame)', but that isn't the case. It should do it that way, I believe, but it's not the cause of this crash.) -- Thomas Wouters [EMAIL PROTECTED]Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Debugging opportunity :-)
At 06:26 PM 4/14/2006 -0400, Phillip J. Eby wrote: If you replace the fun_tests in test_generators with this, and remove all the other tests (e.g. pep_tests, tut_tests, etc.) you can still get it to crash. The code above is much shorter, but has the disadvantage that I don't really know what it's supposed to do, other than cause a crash. :) Interestingly, this code does *not* crash the interpreter on its own, only when run as a doctest (with or without regrtest). I still haven't figured out what the actual problem is, but at least this cuts out all the merge() and times() crud in the case this was derived from, reducing it to a pure zen essence of non-meaning. :) I believe this is now the irreducible minimum code necessary to produce the crash: fun_tests = def cycle(g): ... yield 1 ... yield 1 def root(): ... for i in c: ... yield i r = root() c = cycle(r) c.next() 1 r.next() 1 I have not been able to think of any way to further simplify this, that doesn't make the crash go away. For example, if I replace the for loop inside of root() with iter() and next() operations (either with or without a while loop), the crash does NOT occur. I don't know if that's a red herring, or if it means the crash is inherently about there being a for loop on the stack. I also know that the object 'r' has to be created before 'c' -- in previous iterations I found that just creating a cycle between two generators isn't enough; the one that has the paused for loop has to be created before the thing it's looping over. I'm guessing that this means the order they're listed in the GC tracking lists makes a difference. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Debugging opportunity :-)
On 4/15/06, Thomas Wouters [EMAIL PROTECTED] wrote: (I first thought the problem was caused by gen_dealloc doing 'Py_DECREF(gen-gen_frame)' instead of 'frame = gen-gen_frame; gen-gen_frame = NULL; Py_DECREF(frame)', but that isn't the case. It should do it that way, I believe, but it's not the cause of this crash.) fixes the crash.Ah, found the problem. After I hit 'send', I realized I hadn't checked frameobject's tp_clear, and sure enough, it calls Py_XDECREF in-place. That explains why the first generator object gets dealloced twice, although it doesn't explain why it doesn't blow up when it reaches a negative refcount. Fixing frameobject and genobject to both use Py_CLEAR() makes both the 'minimal' testcase and test_generators work. Your testcase also stops leaking, but alas, test_generators still leaks 255 references. WTF-time for me, meaning I hit the sack and not think about it until next week :) I'll upload a patch to Tim's bugreport after I remove my debugging cruft. All tp_clear/tp_traverse methods should really always use Py_CLEAR/Py_VISIT. I guess revisiting all tp_clear and tp_traverse methods is well worth putting on the TODO list, Bad-pun'ly y'rs,-- Thomas Wouters [EMAIL PROTECTED]Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Debugging opportunity :-)
At 01:11 AM 4/15/2006 +0200, Thomas Wouters wrote: On 4/15/06, Thomas Wouters mailto:[EMAIL PROTECTED][EMAIL PROTECTED] wrote: (I first thought the problem was caused by gen_dealloc doing 'Py_DECREF(gen-gen_frame)' instead of 'frame = gen-gen_frame; gen-gen_frame = NULL; Py_DECREF(frame)', but that isn't the case. It should do it that way, I believe, but it's not the cause of this crash.) fixes the crash.Ah, found the problem. After I hit 'send', I realized I hadn't checked frameobject's tp_clear, and sure enough, it calls Py_XDECREF in-place. That explains why the first generator object gets dealloced twice, although it doesn't explain why it doesn't blow up when it reaches a negative refcount. You're missing another piece of the puzzle: the problem is that since generators don't have a tp_clear, they don't know they're pointing to an invalid frame; it appears to still be running. So when the frame object releases items off the stack, it releases its reference to the generator, which is released normally, dropping out the local variable reference back to the generator whose frame is being cleared. That generator doesn't know it's holding a garbage frame, and thus proceeds to finalize it by resuming it... which then pops the blockstack, and f_stacktop is still valid, so popping the blockstack decrefs the frame a second time. Fixing frameobject and genobject to both use Py_CLEAR() makes both the 'minimal' testcase and test_generators work. It seems to me that frame_clear() should also set f_stacktop to NULL before doing any clearing; otherwise it's possible for a generator to think that the frame is still executable, and the double-decref could thus be replaced by a null pointer dereference. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Debugging opportunity :-)
[Phillip J. Eby] Even with the close() gimmicks still in place? If so, it sounds like the safest thing for a2 might be to claim that generators always need finalizing. :( LOL! Been there many times before: Python's gc really does work, but it's as unforgivingly delicate as any dance you're likely to learn while still mortal :-) This one was actually typical, ending with an unlikely comedy of errors involving microscopic details of how multiple objects' cleanup routines could barely manage to fool each other into tearing down an object more than once. Until you get to that point of clarity, it's also typical to fear that it can never be fixed ;-) You and Thomas did a wonderful of job of tracking this one down quickly. Good show! Take Easter off -- but only if you have to :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r45321 - in python/trunk: Lib/test/test_traceback.py Lib/traceback.py Misc/NEWS
[John J Lee] Assuming this is fixed in 2.5 final, is there some way to write doctests that work on both 2.4 and 2.5? If not, should something like doctest.IGNORE_EXCEPTION_DETAIL be added -- say IGNORE_EXCEPTION_MODULE? [also John] Sorry, please ignore the post of mine I'm replying to here. I missed part of the thread, and Tim has already answered my question... That's news to Tim ;-) It's not possible to add a new doctest option in 2.5 that would allow a doctest to work under both 2.4 and 2.5: the test would blow up under 2.4, because 2.4's doctest wouldn't recognize the (new in 2.5) option, and would raise a ValueError of its own griping about the unknown option name. It would be possible to add a new doctest option in 2.5 that would (just) allow a doctest running _under_ 2.5 to accept a bare name in an exception despite that the traceback module changed in 2.5 to produce some.dotted.name instead. That doesn't seem very useful, though (as above, it would force the test to fail when run under 2.4). Of course a doctest can contain any Python code, so there are many ways to write a doctest that passes under all versions of Python, without active help from doctest. For example, using the running example in this thread, this way works under all versions of Python with a decimal module: import decimal try: 1 / decimal.Decimal(0) except decimal.DivisionByZero: print good good Oddly enough, ELLIPSIS doesn't actually work for this purpose: import decimal 1 / decimal.Decimal(0) #doctest: +ELLIPSIS Traceback (most recent call last): [etc] ...DivisionByZero: x / 0 That test fails under 2.4, and would also fail under the presumably changed 2.5. The problem is that, as the docs say, when trying to guess where an exception message starts, doctest skips down to the first line after the Traceback line indented the same as the Traceback line and starting with an _alphanumeric_ character. There is no such line in this example (the exception line starts with a period), so doctest believes the example is showing expected stdout, not that it's showing an exception. The delightfully baffling result is that doctest complains that the example raises an exception when an exception wasn't expected. Stinking magic ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
