[Python-Dev] Re: builtin_id() returns negative numbers
James Y Knight wrote: However, last time this topic came up, this Tim Peters guy argued against it. ;) Quoting http://mail.python.org/pipermail/python-dev/2004-November/050049.html: Python doesn't promise to return a postive integer for id(), although it may have been nicer if it did. It's dangerous to change that now, because some code does depend on the 32 bit-ness as a signed integer accident of CPython's id() implementation on 32-bit machines. For example, code using struct.pack(), or code using one of ZODB's specialized int-key BTree types with id's as keys. can anyone explain the struct.pack and ZODB use cases? the first one doesn't make sense to me, and the other relies on Python *not* behaving as documented (which is worse than relying on undocumented behaviour, imo). /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] pymalloc on 2.1.3
does anyone remember if there were any big changes in pymalloc between the 2.1 series (where it was introduced) and 2.3 (where it was enabled by default). or in other words, is the 2.1.3 pymalloc stable enough for production use? (we're having serious memory fragmentation problems on a 2.1.3 system, and while I can patch/rebuild the interpreter if necessary, we cannot update the system right now...) /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pymalloc on 2.1.3
Fredrik Lundh [EMAIL PROTECTED] writes: does anyone remember if there were any big changes in pymalloc between the 2.1 series (where it was introduced) and 2.3 (where it was enabled by default). Yes. (Was it really 2.1? Time flies!) or in other words, is the 2.1.3 pymalloc stable enough for production use? Well, Tim posted ways of making it crash, but I don't know how likely they are to occur in non-malicious code. cvs log Objects/obmalloc.c might enlighten, or at least give an idea which months of the python-dev archive to search. Cheers, mwh -- Aardappel this I hate c++ is so old dash it's as old as C++, yes -- from Twisted.Quotes ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: builtin_id() returns negative numbers
[Fredrik Lundh] can anyone explain the struct.pack and ZODB use cases? the first one doesn't make sense to me, Not deep and surely not common, just possible. If you're on a 32-bit box and doing struct.pack(...i..., ... id(obj) ...), it in fact cannot fail now (no, that isn't guaranteed by the docs, it's just an implementation reality), but would fail if id() ever returned a positive long with the same bit pattern as a negative 32-bit int (OverflowError: long int too large to convert to int).. and the other relies on Python *not* behaving as documented (which is worse than relying on undocumented behaviour, imo). I don't know what you think the problem with ZODB's integer-flavored keys might be, then. The problem I'm thinking of is that by integer-flavored they really mean *C* int, not Python integer (which is C long). They're delicate enough that way that they already don't work right on most current 64-bit boxes whenever the value of a Python int doesn't in fact fit in the platform's C int: http://collector.zope.org/Zope/1592 If id() returned a long in some cases on 32-bit boxes, then code using id() as key (in an II or IO tree) or value (in an II or OI) tree would stop working. Again, the Python docs didn't guarantee this would work, and the int-flavored BTrees have 64-bit box bugs in their handling of integers, but the id()-as-key-or-value case has nevertheless worked wholly reliably until now on 32-bit boxes. Any change in visible behavior has the potential to break code -- that shouldn't be controversial, because it's so obvious, and so relentlessly proved in real life. It's a tradeoff. I've said I'm in favor of taking away the sign issue for id() in this case, although I'm not going to claim that no code will break as a result, and I'd be a lot more positive about it if we could use the time machine to change this behavior for 2.4. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pymalloc on 2.1.3
[Fredrik Lundh] does anyone remember if there were any big changes in pymalloc between the 2.1 series (where it was introduced) and 2.3 (where it was enabled by default). Yes, huge -- few original lines survived exactly, although many survived in intent. or in other words, is the 2.1.3 pymalloc stable enough for production use? Different question entirely wink. It _was_ used in production by some people, and happily so. Major differences: + 2.1 used a probabilistic scheme for guessing whether addresses passed to it were obtained from pymalloc or from the system malloc. It was easy for a malicous pure-Python program to corrupt pymalloc and/or malloc internals as a result, leading to things like segfaults, and even sneaky ways to mutate the Python bytecode stream. It's extremely unlikely that a non- malicious program could bump into these. + Horrid hackery went into 2.3's version to cater to broken extension modules that called PyMem functions without holding the GIL. 2.1's may not be as thread-safe in these cases. + 2.1's only fields requests up to 64 bytes, 2.3's up to 256 bytes. Changes in the dict implementation, and new-style classes, for 2.3 made it a pragmatic necessity to boost the limit for 2.3. (we're having serious memory fragmentation problems on a 2.1.3 system, and while I can patch/rebuild the interpreter if necessary, we cannot update the system right now...) I'd give it a shot -- pymalloc has always been very effective at handling large numbers of small objects gracefully. The meaning of small got 4x bigger since 2.1, which appeared to be a pure win, but 64 bytes was enough under 2.1 that most small instance dicts fit. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Exceptions *must*? be old-style classes?
Michael Hudson [EMAIL PROTECTED] writes: Michael Hudson [EMAIL PROTECTED] writes: I hope to have a new patch (which makes PyExc_Exception new-style, but allows arbitrary old-style classes as exceptions) soon. It may even pass bits of make test :) Done: http://www.python.org/sf/1104669 Now I think it's really done, apart from documentation. My design decision was to make Exception new-style. Things can be raised if they are instances of old-style classes or instances of Exception. If this meets with general agreement, I'd like to check the above patch in. It will break some highly introspective code, so it's IMO best to get it in early in the 2.5 cycle. The other option is to keep Exception old-style but allow new-style subclasses, but I think all this will do is break the above mentioned introspective code in a quieter way... The patch also updates the PendingDeprecationWarning on raising a string exception to a full DeprecationWarning (something that should be done anyway). Cheers, mwh -- python py.py ~/Source/python/dist/src/Lib/test/pystone.py Pystone(1.1) time for 5000 passes = 19129.1 This machine benchmarks at 0.261381 pystones/second ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Exceptions *must*? be old-style classes?
My design decision was to make Exception new-style. Things can be raised if they are instances of old-style classes or instances of Exception. If this meets with general agreement, I'd like to check the above patch in. I like it, but didn't you forget to mention that strings can still be raised? I think we can't break that (but we can insert a deprecation warning for this in 2.5 so we can hopefully deprecate it in 2.6, or 2.7 at the latest). The patch also updates the PendingDeprecationWarning on raising a string exception to a full DeprecationWarning (something that should be done anyway). What I said. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Exceptions *must*? be old-style classes?
Guido van Rossum [EMAIL PROTECTED] writes: My design decision was to make Exception new-style. Things can be raised if they are instances of old-style classes or instances of Exception. If this meets with general agreement, I'd like to check the above patch in. I like it, but didn't you forget to mention that strings can still be raised? I think we can't break that (but we can insert a deprecation warning for this in 2.5 so we can hopefully deprecate it in 2.6, or 2.7 at the latest). I try to forget that as much as possible :) The patch also updates the PendingDeprecationWarning on raising a string exception to a full DeprecationWarning (something that should be done anyway). What I said. :-) :) I'll try to bash the documentation into shape next. Cheers, mwh -- please realize that the Common Lisp community is more than 40 years old. collectively, the community has already been where every clueless newbie will be going for the next three years. so relax, please. -- Erik Naggum, comp.lang.lisp ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory Allocator Part 2: Did I get it right?
[Evan Jones] After I finally understood what thread-safety guarantees the Python memory allocator needs to provide, I went and did some hard thinking about the code this afternoon. I believe that my modifications provide the same guarantees that the original version did. I do need to declare the arenas array to be volatile, and leak the array when resizing it. Please correct me if I am wrong, but the situation that needs to be supported is this: As I said before, I don't think we need to support this any more. More, I think we should not -- the support code is excruciatingly subtle, it wasted plenty of your time trying to keep it working, and if we keep it in it's going to continue to waste time over the coming years (for example, in the short term, it will waste my time reviewing it). While one thread holds the GIL, any other thread can call PyObject_Free with a pointer that was returned by the system malloc. What _was_ supported was more generally that any number of threads could call PyObject_Free with pointers that were returned by the system malloc/realloc at the same time as a single thread, holding the GIL, was doing anything whatsoever (including executing any code inside obmalloc.c) Although that's a misleading way of expressing the actual intent; more on that below. The following situation is *not* supported: While one thread holds the GIL, another thread calls PyObject_Free with a pointer that was returned by PyObject_Malloc. Right, that was never supported (and I doubt it could be without introducing a new mutex in obmalloc.c). I'm hoping that I got things a little better this time around. I've submitted my updated patch to the patch tracker. For reference, I've included links to SourceForge and the previous thread. Thank you, Thank you! I probably can't make time to review anything before this weekend. I will try to then. I expect it would be easier if you ripped out the horrid support for PyObject_Free abuse; in a sane world, the release-build PyMem_FREE, PyMem_Del, and PyMem_DEL would expand to free instead of to PyObject_FREE (via changes to pymem.h). IOW, it was never the _intent_ that people be able to call PyObject_Free without holding the GIL. The need for that came from a different problem, that old code sometimes mixed calls to PyObject_New with calls to PyMem_DEL (or PyMem_FREE or PyMem_Del). It's for that latter reason that PyMem_DEL (and its synonyms) were changed to expand to PyObject_Free. This shouldn't be supported anymore. Because it _was_ supported, there was no way to tell whether PyObject_Free was being called because (a) we were catering to long-obsolete but once-loved code that called PyMem_DEL while holding the GIL and with a pointer obtained by PyObject_New; or, (b) somebody was calling PyMem_Del (etc) with a non-object pointer they had obtained from PyMem_New, or from the system malloc directly. It was never legit to do #a without holding the GIL. It was clear as mud whether it was legit to do #b without holding the GIL. If PyMem_Del (etc) change to expand to free in a release build, then #b can remain clear as mud without harming anyone. Nobody should be doing #a anymore. If someone still is, tough luck -- fix it, you've had years of warning is easy for me to live with at this stage. I suppose the other consideration is that already-compiled extension modules on non-Windows(*) systems will, if they're not recompiled, continue to call PyObject_Free everywhere they had a PyMem_Del/DEL/FREE call. If such code is calling it without holding the GIL, and obmalloc.c stops trying to support this insanity, then they're going to grow some thread races they woudn't have if they did recompile (to get such call sites remapped to the system free). I don't really care about that either: it's a general rule that virtually all Python API functions must be called with the GIL held, and there was never an exception in the docs for the PyMem_ family. (*) Windows is immune simply because the Windows Python is set up in such a way that you always have to recompile extension modules when Python's minor version number (the j in i.j.k) gets bumped. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory Allocator Part 2: Did I get it right?
[Tim Peters] As I said before, I don't think we need to support this any more. More, I think we should not -- the support code is excruciatingly subtle, it wasted plenty of your time trying to keep it working, and if we keep it in it's going to continue to waste time over the coming years (for example, in the short term, it will waste my time reviewing it). [Evan Jones] I do not have nearly enough experience in the Python world to evaluate this decision. I've only been programming in Python for about two years now, and as I am sure you are aware, this is my first patch that I have submitted to Python. I don't really know my way around the Python internals, beyond writing basic extensions in C. Martin's opinion is clearly the opposite of yours. ? This is all I recall Martin saying about this: http://mail.python.org/pipermail/python-dev/2005-January/051265.html I'm not certain it is acceptable to make this assumption. Why is it not possible to use the same approach that was previously used (i.e. leak the arenas array)? Do you have something else in mind? I'll talk with Martin about it if he still wants to. Martin, this miserable code must die! Basically, the debate seems to boil down to maintaining backwards compatibility at the cost of making the code in obmalloc.c harder to understand. The let it leak to avoid thread problems cruft is arguably the single most obscure bit of coding in Python's code base. I created it, so I get to say that wink. Even 100 lines of comments aren't enough to make it clear, as you've discovered. I've lost track of how many hours of my life have been pissed away explaining it, and its consequences (like how come this or that memory-checking program complains about the memory leak it causes), and the historical madness that gave rise to it in the beginning. I've had enough of it -- the only purpose this part ever had was to protect against C code that wasn't playing by the rules anyway. BFD. There are many ways to provoke segfaults with C code that breaks the rules, and there's just not anything that special about this way _except_ that I added objectionable (even at the time) hacks to preserve this kind of broken C code until authors had time to fix it. Time's up. The particular case that is being supported could definitely be viewed as a bug in the code that using obmalloc. It also likely is quite rare. However, until now it has been supported, so it is hard to judge exactly how much code would be affected. People spent many hours searching for affected code when it first went in, and only found a few examples then, in obscure extension modules. It's unlikely usage has grown. The hack was put it in for the dubious benefit of the few examples that were found then. It would definitely be a minor barrier to moving to Python 2.5. That's in part what python-dev is for. Of course nobody here has code that will break -- but the majority of high-use extension modules are maintained by people who read this list, so that's not as empty as it sounds. It's also what alpha and beta releases are for. Fear of change isn't a good enough reason to maintain this code. Is there some sort of consensus that is possible on this issue? Absolutely, provided it matches my view 0.5 wink. Rip it out, and if alpha/beta testing suggests that's a disaster, _maybe_ put it back in. ... It turns out that basically the only thing that would change would be removing the volatile specifiers from two of the global variables, plus it would remove about 100 lines of comments. :) The work was basically just hurting my brain trying to reason about the concurrency issues, not changing code. And the brain of everyone else who ever bumps into this. There's a high probability that if this code actually doesn't work (can you produce a formal proof of correctness for it? I can't -- and I tried), nothing can be done to repair it; and code this outrageously delicate has a decent chance of being buggy no matter how many people stare at it (overlooking that you + me isn't that many). You also mentioned before that removing the volatiles may have given a speed boost, and that's believable. I mentioned above the unending costs in explanations, and nuisance gripes from memory-integrity tools about the deliberate leaks. There are many kinds of ongoing costs here, and no _intended_ benefit anymore (it certainly wasn't my intent to cater to buggy C code forever). It was never legit to do #a without holding the GIL. It was clear as mud whether it was legit to do #b without holding the GIL. If PyMem_Del (etc) change to expand to free in a release build, then #b can remain clear as mud without harming anyone. Nobody should be doing #a anymore. If someone still is, tough luck -- fix it, you've had years of warning is easy for me to live with at this stage. Hmm... The issue is that case #a may not be an easy problem to diagnose: Many errors in C
[Python-Dev] Weekly Python Patch/Bug Summary
Patch / Bug Summary ___ Patches : 298 open (+14) / 2754 closed ( +6) / 3052 total (+20) Bugs: 823 open (+19) / 4829 closed (+17) / 5652 total (+36) RFE : 168 open ( +1) / 144 closed ( +2) / 312 total ( +3) New / Reopened Patches __ date.strptime and time.strptime as well (2005-02-04) http://python.org/sf/1116362 opened by Josh NameError in cookielib domain check (2005-02-04) CLOSED http://python.org/sf/1116583 opened by Chad Miller Minor improvement on 1116583 (2005-02-06) http://python.org/sf/1117114 opened by John J Lee cookielib and cookies with special names (2005-02-06) http://python.org/sf/1117339 opened by John J Lee cookielib LWPCookieJar and MozillaCookieJar exceptions (2005-02-06) http://python.org/sf/1117398 opened by John J Lee cookielib.LWPCookieJar incorrectly loads value-less cookies (2005-02-06) http://python.org/sf/1117454 opened by John J Lee urllib2 .getheaders attribute error (2005-02-07) http://python.org/sf/1117588 opened by Wummel replace md5 impl. with one having a more free license (2005-02-07) CLOSED http://python.org/sf/1117961 opened by Matthias Klose unknown locale: lt_LT (patch) (2005-02-08) http://python.org/sf/1118341 opened by Nerijus Baliunas Fix crash in xmlprase_GetInputContext in pyexpat.c (2005-02-08) http://python.org/sf/1118602 opened by Mathieu Fenniak enable time + timedelta (2005-02-08) http://python.org/sf/1118748 opened by Josh fix for a bug in Header.__unicode__() (2005-02-09) CLOSED http://python.org/sf/1119016 opened by Björn Lindqvist python -c readlink()s and stat()s '-c' (2005-02-09) http://python.org/sf/1119423 opened by Brian Foley patches to compile for AIX 4.1.x (2005-02-09) http://python.org/sf/1119626 opened by Stuart D. Gathman better datetime support for xmlrpclib (2005-02-10) http://python.org/sf/1120353 opened by Fred L. Drake, Jr. ZipFile.open - read-only file-like obj for files in archive (2005-02-11) http://python.org/sf/1121142 opened by Alan McIntyre Reference count bug fix (2005-02-12) http://python.org/sf/1121234 opened by Michiel de Hoon sha and md5 modules should use OpenSSL when possible (2005-02-12) http://python.org/sf/1121611 opened by Gregory P. Smith Python memory allocator: Free memory (2005-02-15) http://python.org/sf/1123430 opened by Evan Jones Patches Closed __ Add SSL certificate validation (2005-02-03) http://python.org/sf/1115631 closed by noonian NameError in cookielib domain check (2005-02-04) http://python.org/sf/1116583 closed by rhettinger replace md5 impl. with one having a more free license (2005-02-07) http://python.org/sf/1117961 closed by loewis fix for a bug in Header.__unicode__() (2005-02-09) http://python.org/sf/1119016 closed by sonderblade time.tzset() not built on Solaris (2005-01-04) http://python.org/sf/1096244 closed by bcannon OSATerminology extension fix (2004-06-25) http://python.org/sf/979784 closed by jackjansen New / Reopened Bugs ___ xmlrpclib: wrong decoding in '_stringify' (2005-02-04) CLOSED http://python.org/sf/1115989 opened by Dieter Maurer Prefix search is filesystem-centric (2005-02-04) http://python.org/sf/1116520 opened by Steve Holden Wrong match with regex, non-greedy problem (2005-02-05) CLOSED http://python.org/sf/1116571 opened by rengel Solaris 10 fails to compile complexobject.c (2005-02-04) http://python.org/sf/1116722 opened by Case Van Horsen Dictionary Evaluation Issue (2005-02-05) http://python.org/sf/1117048 opened by WalterBrunswick Typo in list.sort() documentation (2005-02-06) CLOSED http://python.org/sf/1117063 opened by Viktor Ferenczi sgmllib.SGMLParser (2005-02-06) CLOSED http://python.org/sf/1117302 opened by Paul Birnie SimpleHTTPServer and mimetypes: almost together (2005-02-06) http://python.org/sf/1117556 opened by Matthew L Daniel os.path.exists returns false negatives in MAC environments. (2005-02-07) http://python.org/sf/1117601 opened by Stephen Bennett profiler: Bad return and Bad call errors with exceptions (2005-02-06) http://python.org/sf/1117670 opened by Matthew Mueller in operator bug ? (2005-02-07) CLOSED http://python.org/sf/1117757 opened by Andrea Bolzonella BSDDB openhash (2005-02-07) http://python.org/sf/1117761 opened by Andrea Bolzonella lists coupled (2005-02-07) CLOSED http://python.org/sf/1118101 opened by chopf Error in representation of complex numbers(again) (2005-02-09) http://python.org/sf/1118729 opened by George Yoshida builtin file() vanishes (2005-02-08) CLOSED http://python.org/sf/1118977 opened by Barry Alan Scott Docs for set() omit constructor (2005-02-09) CLOSED
Re: [Python-Dev] Memory Allocator Part 2: Did I get it right?
Tim Peters wrote: I'm not certain it is acceptable to make this assumption. Why is it not possible to use the same approach that was previously used (i.e. leak the arenas array)? Do you have something else in mind? I'll talk with Martin about it if he still wants to. Martin, this miserable code must die! That's fine with me. I meant what I said: I'm not certain. The patch original claimed that it cannot possibly preserve this feature, and I felt that this claim was incorrect - indeed, Evan then understood the feature, and made it possible. I can personally accept breaking the code that still relies on the invalid APIs. The only problem is that it is really hard to determine whether some code *does* violate the API usage. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
