[Python-Dev] ssize_t question: longs in header files
$ grep long */*.h minus comments, etc yields several questions about whether some values should use Py_ssize_t rather than C longs. In particular: * hash values Include/abstract.h: long PyObject_Hash(PyObject *o); // also in object.h Include/object.h:typedef long (*hashfunc)(PyObject *); * ints: Include/intobject.h:long ob_ival; * thread values (probably no need to change since platform dependent) Attached is the whole list. This is just for C longs. The int list is much bigger and coming in a different msg. n header-longs Description: Binary data ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] ssize_t: ints in header files
Here's the int questions. The entire list is too big to post (50k), so it's here: http://www.python.org/neal/header-ints Should the following values be ints (limited to 2G)? * dict counts (ma_fill, ma_used, ma_mask) * line #s and column #s * AST (asdl.h) sequences * recursion limit * read/write/send/recv results and buf size * code object values like # args, # locals, stacksize, # instructions * sre (i think i have a patch to fix this somewhere) Please try to review the APIs and let us know if there are any values you think should be 64-bits on 64-bit platforms. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] feature request: inspect.isgenerator
Is there still time for new feature requests in Python 2.5? I am missing a isgenerator function in the inspect module. Right now I am using def isgenerator(func): return func.func_code.co_flags == 99 but it is rather ugly (horrible indeed). Michele Simionato ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssize_t: ints in header files
Neal Norwitz wrote: Should the following values be ints (limited to 2G)? * line #s and column #s Anyone with a source file more than 2G lines long or 2G chars wide deserves whatever they get. :-) Not-investing-in-a-1.3e10-pixel-wide-screen-any-time-soon-ly, Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
On May 28, 2006, at 5:34 PM, Thomas Wouters wrote:On 5/29/06, Bob Ippolito [EMAIL PROTECTED] wrote: On May 28, 2006, at 4:31 AM, Thomas Wouters wrote: I'm seeing a dubious failure of test_gzip and test_tarfile on my AMD64 machine. It's triggered by the recent struct changes, but I'd say it's probably caused by a bug/misfeature in zlibmodule: zlib.crc32 is the result of a zlib 'crc32' functioncall, which returns an unsigned long. zlib.crc32 turns that unsigned long into a (signed) Python int, which means a number beyond 131 goes negative on 32-bit systems and other systems with 32-bit longs, but stays positive on systems with 64-bit longs: (32-bit) zlib.crc32("foobabazr") -271938108 (64-bit) zlib.crc32("foobabazr") 4023029188 The old structmodule coped with that: struct.pack("l", -271938108) '\xc4\x8d\xca\xef' struct.pack("l", 4023029188) '\xc4\x8d\xca\xef' The new one does not: struct.pack("l", -271938108) '\xc4\x8d\xca\xef' struct.pack("l", 4023029188) Traceback (most recent call last): File "stdin", line 1, in module File "Lib/struct.py", line 63, in pack return o.pack(*args) struct.error: 'l' format requires -2147483647 = number = 2147483647 The structmodule should be fixed (and a test added ;) but I'm also wondering if zlib shouldn't be fixed. Now, I'm AMD64-centric, so my suggested fix would be to change the PyInt_FromLong() call to PyLong_FromUnsignedLong(), making zlib always return positive numbers -- it might break some code on 32-bit platforms, but that code is already broken on 64-bit platforms. But I guess I'm okay with the long being changed into an actual 32-bit signed number on 64-bit platforms, too.The struct module isn't what's broken here. All of the struct typeshave always had well defined bit sizes and alignment if you explicitly specify an endian, I and L are 32-bits everywhere, and Q is supported on platforms that don't have long long. The onlything that's changed is that it actually checks for errorsconsistently now. Yes. And that breaks things. I'm certain the behaviour is used in real-world code (and I don't mean just the gzip module.) It has always, as far as I can remember, accepted 'unsigned' values for the signed versions of ints, longs and long-longs (but not chars or shorts.) I agree that that's wrong, but I don't think changing struct to do the right thing should be done in 2.5. I don't even think it should be done in 2.6 -- although 3.0 is fine.Well, the behavior change is in response to a bug http://python.org/sf/1229380. If nothing else, we should at least fix the standard library such that it doesn't depend on struct bugs. This is the only way to find them :)Basically the struct module previously only checked for errors if you don't specify an endian. That's really strange and leads to very confusing results. The only code that really should be broken by this additional check is code that existed before Python had a long type and only signed values were available.-bob___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
On 5/29/06, Bob Ippolito [EMAIL PROTECTED] wrote: Well, the behavior change is in response to a bug http://python.org/sf/1229380.If nothing else, we should at least fix the standard library such that it doesn't depend on struct bugs. This is the only way to find them :) Feel free to comment how the zlib.crc32/gzip co-operation should be fixed. I don't see an obviously correct fix. The trunk is currently failing tests it shouldn't fail. Also note that the error isn't with feeding signed values to unsigned formats (which is what the bug is about) but the other way 'round, although I do believe both should be accepted for the time being, while generating a warning. Basically the struct module previously only checked for errors if you don't specify an endian. That's really strange and leads to very confusing results. The only code that really should be broken by this additional check is code that existed before Python had a long type and only signed values were available. Alas, reality is different. The fundamental difference between types in Python and in C causes this, and code using struct is usually meant specifically to bridge those two worlds. Furthermore, struct is often used *fix* that issue, by flipping sign bits if necessary: struct.unpack(l, struct.pack(l, 3221225472))(-1073741824,) struct.unpack(l, struct.pack(L, 3221225472))(-1073741824,) struct.unpack(l, struct.pack(l, -1073741824))(-1073741824,) struct.unpack(l, struct.pack(L, -1073741824))(-1073741824,) Before this change, you didn't have to check whether the value is negative before the struct.unpack/pack dance, regardless of which format character you used. This misfeature is used (and many would consider it convenient, even Pythonic, for struct to DWIM), breaking it suddenly is bad. -- Thomas Wouters [EMAIL PROTECTED]Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssize_t question: longs in header files
On 5/29/06, Neal Norwitz [EMAIL PROTECTED] wrote: * ints:Include/intobject.h:long ob_ival;I considered asking about this before, as it would give '64-bit power' to Win64 integers. It's a rather big change, though (lots of code assumes PyInts fit in signed longs, which would be untrue then.) On the other hand, if a separate type was used for PyInts (Py_pyint_t or whatever, defaulting to long, except on LLP64 systems), EWT's request for a 64-bit integer represented by C 'long long's would be a simple configure switch. -- Thomas Wouters [EMAIL PROTECTED]Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
On May 29, 2006, at 3:14 AM, Thomas Wouters wrote:On 5/29/06, Bob Ippolito [EMAIL PROTECTED] wrote: Well, the behavior change is in response to a bug http://python.org/sf/1229380. If nothing else, we should at least fix the standard library such that it doesn't depend on struct bugs. This is the only way to find them :) Feel free to comment how the zlib.crc32/gzip co-operation should be fixed. I don't see an obviously correct fix. The trunk is currently failing tests it shouldn't fail. Also note that the error isn't with feeding signed values to unsigned formats (which is what the bug is about) but the other way 'round, although I do believe both should be accepted for the time being, while generating a warning. Well, first I'm going to just correct the modules that are broken (zlib, gzip, tarfile, binhex and probably one or two others).Basically the struct module previously only checked for errors if you don't specify an endian. That's really strange and leads to very confusing results. The only code that really should be broken by this additional check is code that existed before Python had a long type and only signed values were available. Alas, reality is different. The fundamental difference between types in Python and in C causes this, and code using struct is usually meant specifically to bridge those two worlds. Furthermore, struct is often used *fix* that issue, by flipping sign bits if necessary: Well, in C you get a compiler warning for stuff like this. struct.unpack("l", struct.pack("l", 3221225472))(-1073741824,) struct.unpack("l", struct.pack("L", 3221225472))(-1073741824,) struct.unpack("l", struct.pack("l", -1073741824))(-1073741824,) struct.unpack("l", struct.pack("L", -1073741824))(-1073741824,) Before this change, you didn't have to check whether the value is negative before the struct.unpack/pack dance, regardless of which format character you used. This misfeature is used (and many would consider it convenient, even Pythonic, for struct to DWIM), breaking it suddenly is bad. struct doesn't really DWIM anyway, since integers are up-converted to longs and will overflow past what the (old or new) struct module will accept. Before there was a long type or automatic up-converting, the sign agnosticism worked.. but it doesn't really work correctly these days.We have two choices, either fix it to behave consistently broken everywhere for numbers of every size (modulo every number that comes in so that it fits), or have it do proper range checking. A compromise is to do proper range checking as a warning, and do the modulo math anyway... but is that what we really want?-bob___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
On 5/29/06, Bob Ippolito [EMAIL PROTECTED] wrote: A compromise is to do proper range checking as a warning, and do the modulo math anyway... but is that what we really want?I don't know about the rest of 'us', but that's what I want, yes: backward compatibility, and a warning to tell people to fix their code 'or else'. The prevalence of the warnings (outside of the stdlib) should give us a clue whether to make it an exception in 2.6 or wait for 2.7/3.0.Perhaps more people could chime in? Am I being too anal about backward compatibility here?-- Thomas Wouters [EMAIL PROTECTED] Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Syntax errors on continuation lines
On Sun, May 28, 2006, Roger Miller wrote: I am a recent subscriber to this list. A couple of months ago I downloaded the Python source out of curiosity, then decided to see if I could scratch a couple of small itches. Please post your patches to SourceForge, then post links back to the list. -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ I saw `cout' being shifted Hello world times to the left and stopped right there. --Steve Gonedes ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Syntax errors on continuation lines
Aahz wrote: On Sun, May 28, 2006, Roger Miller wrote: I am a recent subscriber to this list. A couple of months ago I downloaded the Python source out of curiosity, then decided to see if I could scratch a couple of small itches. +1 on the change, the error messages that report the wrong line can be very confusing. Michael Foord Please post your patches to SourceForge, then post links back to the list. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] DRAFT: python-dev summary for 2006-04-01 to 2006-04-15
Ok, here comes the first half of April. Again, please let me know if you have any comments or corrections (and thanks to those who already got back to me on the last one). I'm gonna try to keep these coming over the next week or so until I'm all caught up. = Announcements = - Python 2.5a1 Released - Python 2.5 alpha 1 was released on April 5th. Please download it and try it out, particularly if you are an extension writer or you embed Python -- you may want to change things to support 64-bit sequences, and if you have been using mismatched PyMem_* and PyObject_* allocation calls, you will need to fix these as well. Contributing threads: - `TRUNK FREEZE. 2.5a1, 00:00 UTC, Wednesday 5th of April. http://mail.python.org/pipermail/python-dev/2006-April/063324.html`__ - `outstanding items for 2.5 http://mail.python.org/pipermail/python-dev/2006-April/063328.html`__ - `chilling svn for 2.5a1 http://mail.python.org/pipermail/python-dev/2006-April/063403.html`__ - `Reminder: TRUNK FREEZE. 2.5a1, 00:00 UTC, Wednesday 5th of April. http://mail.python.org/pipermail/python-dev/2006-April/063426.html`__ - `RELEASED Python 2.5 (alpha 1) http://mail.python.org/pipermail/python-dev/2006-April/063441.html`__ - `TRUNK is UNFROZEN http://mail.python.org/pipermail/python-dev/2006-April/063443.html`__ - `segfault (double free?) when '''-string crosses line http://mail.python.org/pipermail/python-dev/2006-April/063572.html`__ - `pdb segfaults in 2.5 trunk? http://mail.python.org/pipermail/python-dev/2006-April/063591.html`__ - `IMPORTANT 2.5 API changes for C Extension Modules and Embedders http://mail.python.org/pipermail/python-dev/2006-April/063637.html`__ -- Contributor agreements -- If you're listed in the `Python acknowledgments`_, and you haven't signed a `contributor agreement`_, please submit one as soon as possible. Note that this includes even the folks that have been around forever -- the PSF would like to be as careful as possible on this one. .. _Python acknowledgments: http://svn.python.org/projects/python/trunk/Misc/ACKS .. _contributor agreement: http://www.python.org/psf/contrib-form-python.html Contributing threads: - `PSF Contributor Agreement for pysqlite http://mail.python.org/pipermail/python-dev/2006-April/063578.html`__ --- Firefox Python bug searches --- Anthony Baxter has created a `Firefox searchbar`_ for finding Python bugs by their SourceForge IDs. There are also two Firefox sidebar options: `Mark Hammond's`_ and `Daniel Lundin's`_. .. _Firefox searchbar: http://www.python.org/~anthony/searchbar/ .. _Mark Hammond's: http://starship.python.net/~skippy/mozilla/ .. _Daniel Lundin's: http://projects.edgewall.com/python-sidebar/ Contributing thread: - `Firefox searchbar engine for Python bugs http://mail.python.org/pipermail/python-dev/2006-April/063285.html`__ - Building Python with the free MS Toolkit compiler - Paul Moore documented how to build Python on Windows with the free MS Toolkit C++ compiler `on the wiki`_. The most up-to-date version of these instructions is in `PCbuild/readme.txt`_. .. _on the wiki: http://wiki.python.org/moin/Building_Python_with_the_free_MS_C_Toolkit .. _PCbuild/readme.txt: http://svn.python.org/projects/python/trunk/PCbuild/readme.txt Contributing thread: - `Building Python with the free MS Toolkit compiler http://mail.python.org/pipermail/python-dev/2006-April/063719.html`__ -- Please login to the wiki when you make changes -- Skip Montanaro has requested that anyone posting to the wiki sign in first, as this makes things easier for those monitoring changes to the wiki. When you're logged in, your changes appear with your name, and so can be immediately recognized as not being wiki-spam. Contributing thread: - `Suggestion: Please login to wiki when you make changes http://mail.python.org/pipermail/python-dev/2006-April/063447.html`__ --- Checking an older blamelist --- While the buildbot blamelist may scroll off the `main page`_ in a matter of hours, you can still see the blamelist for a particular revision by checking a particular build number, e.g. to see build 800 of the trunk on the G4 OSX, you could check http://www.python.org/dev/buildbot/trunk/g4%20osx.4%20trunk/builds/800. .. _main page: http://www.python.org/dev/buildbot/all/ Contributing threads: - `Preserving the blamelist http://mail.python.org/pipermail/python-dev/2006-April/063633.html`__ - `TODO Wiki (was: Preserving the blamelist) http://mail.python.org/pipermail/python-dev/2006-April/063673.html`__ = Summaries = - Garbage collection issues
Re: [Python-Dev] epoll implementation
On May 27, 2006, at 04:59, Alex Martelli wrote: I believe it's: kqueue on FreeBSD (for recent-enough versions thereof), otherwise epoll where available and nonbuggy, otherwise poll ditto, otherwise select -- that's roughly what Twisted uses for kqueue is not always faster. It depends on your application. The number of FDs and the fd values affect the point that kqueue wins over select. For a few FDs with low values select was always faster in my apps. Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Let's stop eating exceptions in dict lookup
Hi all, I've finally come around to writing a patch that stops dict lookup from eating all exceptions that occur during lookup, like rare bugs in user __eq__() methods. After another 2-hours long debugging session that turned out to be caused by that, I had a lot of motivation. http://python.org/sf/1497053 The patch doesn't change the PyDict_GetItem() interface, which is the historical core of the problem. It works around this issue by just moving the exception-eating bit there instead of in lookdict(), so it gets away with changing only dictobject.c (plus ceval.c's direct usage of ma_lookup for LOAD_GLOBAL). The benefit of this patch is that all other ways to work with dicts now correctly propagate exceptions, and this includes all the direct manipulation from Python code (including 'x=d[key]'). The reason I bring this up here is that I'm going to check it in 2.5, unless someone seriously objects. About the objection we need a better fix, PyDict_GetItem() should really be fixed and all its usages changed: this would be good, and also require some careful compatibility considerations, and quite some work in total; it would also give a patch which is basically a superset of mine, so I don't think I'm going in the wrong direction there. A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssize_t question: longs in header files
[Neal Norwitz] * ints: Include/intobject.h:long ob_ival; [Thomas Wouters] I considered asking about this before, as it would give '64-bit power' to Win64 integers. It's a rather big change, though (lots of code assumes PyInts fit in signed longs, which would be untrue then.) I expect that rather big is understatement -- more that the sheer bulk of it would make the Py_ssize_t changes look like a warmup exercise. Too late in the release cycle, IMO (and note that we're still stumbling into Py_ssize_t glitches despite that it's been in place for months). On the other hand, if a separate type was used for PyInts (Py_pyint_t or whatever, defaulting to long, except on LLP64 systems), EWT's request for a 64-bit integer represented by C 'long long's would be a simple configure switch. But then _all_ short ints in Python would be 64 bits, and EWT did not ask for a slowdown on all short int operations 0.5 wink. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssize_t question: longs in header files
[Neal Norwitz] * hash values Include/abstract.h: long PyObject_Hash(PyObject *o); // also in object.h Include/object.h:typedef long (*hashfunc)(PyObject *); We should leave these alone for now. There's no real connection between the width of a hash value and the number of elements in a container, and Py_ssize_t is conceptually only related to the latter. As you noted later, there is an unusually strong connection in the dict implementation, and it's been on my todo list for months to look into that. Certainly the struct _dictobject members need to change from int to Py_ssize_t (including ma_mask), but that can't be done mechanically (code changes may also be required, and this is one is much more math- than compiler-warning- driven). I'll note in passing that the debug-build obmalloc has no idea what to do with byte counts that don't fit in 4 bytes (it will probably left-truncate them in its internal debugging records). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
[Thomas Wouters] ... Perhaps more people could chime in? Am I being too anal about backward compatibility here? Yes and no ;-) Backward compatibility _is_ important, but there seems no way to know in this case whether struct's range-checking sloppiness was accidental or deliberate. Having fixed bugs in the old code several times, and been reduced to writing crap like this in the old test_struct.py: # XXX Most std integer modes fail to test for out-of-range. # The i and l codes appear to range-check OK on 32-bit boxes, but # fail to check correctly on some 64-bit ones (Tru64 Unix + Compaq C # reported by Mark Favas). BUGGY_RANGE_CHECK = bBhHiIlL I can't help but note several things: - If it _was_ intended that range-checking be sloppy, nobody bothered to document it. - Or even to write a comment in the code explaining that obscure intent. - When I implemented the Q (8-byte int) format code, I added correct range-checking in all cases, and nobody ever complained about that. - As noted in the comment above, we have gotten complaints about failures of struct range-checking at other integer widths. OTOH, BUGGY_RANGE_CHECK existed because I was too timid to risk making broken user code visibly broken. So, in all, I'm 95% sure 2.4's behavior is buggy, but 50% unsure that we need to warn about it before repairing it. Since you (Thomas) want warnings, and in theory it only affects the lightly-used standard modes, I do lean in favor of leaving the standard modes that _are_ broken (as above, not all are) broken in 2.5 but warning that this will change in 2.6. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
Thomas Wouters [EMAIL PROTECTED] wrote: On 5/29/06, Bob Ippolito [EMAIL PROTECTED] wrote: A compromise is to do proper range checking as a warning, and do the modulo math anyway... but is that what we really want? I don't know about the rest of 'us', but that's what I want, yes: backward compatibility, and a warning to tell people to fix their code 'or else'. The prevalence of the warnings (outside of the stdlib) should give us a clue whether to make it an exception in 2.6 or wait for 2.7/3.0. Perhaps more people could chime in? Am I being too anal about backward compatibility here? As a fairly heavy user of struct, I personally don't use struct to do modulos and/or sign manipulation (I mask before I pass), but a change in behavior seems foolish if people use that behavior. So far, I'm not aware of anyone complaining about Python 2.4's use, so it would seem to suggest that the current behavior is not incorrect. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
On 5/29/06, Tim Peters [EMAIL PROTECTED] wrote: [Thomas Wouters] ... Perhaps more people could chime in? Am I being too anal about backward compatibility here? Yes and no ;-) Backward compatibility _is_ important, but there seems no way to know in this case whether struct's range-checking sloppiness was accidental or deliberate. I'm pretty sure it was deliberate. I'm more than likely the original author of this code (since the struct module was originally mine), and I know I put in things like that in a few places to cope with hex/oct constants pasted from C headers, and the general messiness that ensued because of Python's lack of an unsigned int type. This is probably a case that was missed by PEP 237. I think we should do as Thomas proposes: plan to make it an error in 2.6 (or 2.7 if there's a big outcry, which I don't expect) and accept it with a warning in 2.5. - If it _was_ intended that range-checking be sloppy, nobody bothered to document it. Mea culpa. In those days we didn't document a lot of things. - Or even to write a comment in the code explaining that obscure intent. It never occurred to me that it was obscure; we did this all over the place (in PyArg_Parse too). - When I implemented the Q (8-byte int) format code, I added correct range-checking in all cases, and nobody ever complained about that. It's really only a practical concern for 32-bit values on 32-bit machines, where reasonable people can disagree over whether 0x is -1 or 4294967295. - As noted in the comment above, we have gotten complaints about failures of struct range-checking at other integer widths. OTOH, BUGGY_RANGE_CHECK existed because I was too timid to risk making broken user code visibly broken. So, in all, I'm 95% sure 2.4's behavior is buggy, but 50% unsure that we need to warn about it before repairing it. Since you (Thomas) want warnings, and in theory it only affects the lightly-used standard modes, I do lean in favor of leaving the standard modes that _are_ broken (as above, not all are) broken in 2.5 but warning that this will change in 2.6. I'm not sure what we gain by leaving other std modules depending on struct's brokenness broken. But I may be misinterpreting which modules you're referring to. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
From: Armin Rigo [EMAIL PROTECTED] I've finally come around to writing a patch that stops dict lookup from eating all exceptions that occur during lookup, like rare bugs in user __eq__() methods. Is there a performance impact? Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
[Guido] ... It's really only a practical concern for 32-bit values on 32-bit machines, where reasonable people can disagree over whether 0x is -1 or 4294967295. Then maybe we should only let that one slide 0.5 wink. ... [Tim] So, in all, I'm 95% sure 2.4's behavior is buggy, but 50% unsure that we need to warn about it before repairing it. Since you (Thomas) want warnings, and in theory it only affects the lightly-used standard modes, I do lean in favor of leaving the standard modes that _are_ broken (as above, not all are) broken in 2.5 but warning that this will change in 2.6. I'm not sure what we gain by leaving other std modules depending on struct's brokenness broken. But I may be misinterpreting which modules you're referring to. I think you're just reading module where I wrote mode. Standard mode is struct-module terminology, as in b !b b are standard modes but b is not a standard mode (it's native mode). But I got it backwards -- or maybe not ;-) It's confusing because it's so inconsistent (this under 2.4.3 on 32-bit Windows): struct.pack(B, -32) # std mode doesn't complain '\xe0' struct.pack(B, -32) # native mode does Traceback (most recent call last): File stdin, line 1, in ? struct.error: ubyte format requires 0=number=255 struct.pack(b, 255) # std mode doesn't complain '\xff' struct.pack(b, 255) # native mode does Traceback (most recent call last): File stdin, line 1, in ? struct.error: byte format requires -128=number=127 On the other hand, as I noted last time, some standard modes _do_ range-check -- but not correctly on some 64-bit boxes -- and not consistently across positive and negative out-of-range values, or across input types. Like: struct.pack(i, 2**32-1) # std and native modes complain Traceback (most recent call last): File stdin, line 1, in ? OverflowError: long int too large to convert to int struct.pack(i, 2**32-1) Traceback (most recent call last): File stdin, line 1, in ? OverflowError: long int too large to convert to int struct.pack(I, -1) # neither std nor native modes complain '\xff\xff\xff\xff' struct.pack(I, -1) '\xff\xff\xff\xff' struct.pack(I, -1L) # but both complain if the input is long Traceback (most recent call last): File stdin, line 1, in ? OverflowError: can't convert negative value to unsigned long struct.pack(I, -1L) Traceback (most recent call last): File stdin, line 1, in ? OverflowError: can't convert negative value to unsigned long In short, there's no way to explain what struct checks for in 2.4.3 short of drawing up an exhaustive table of standard-vs-native mode, format code, which direction a value may be out of range, and whether the value is given as a Python int or a long. At the sprint, I encouraged Bob to do complete range-checking. That's explainable. If we have to back off from that, then since the new code is consistent, I'm sure any warts he puts back in will be clearly look like warts ;-) I think we should do as Thomas proposes: plan to make it an error in 2.6 (or 2.7 if there's a big outcry, which I don't expect) and accept it with a warning in 2.5. That's what I arrived at, although 2.4.3's checking behavior is actually so inconsistent that it needs some defining (what exactly are we trying to still accept? e.g., that -1 doesn't trigger I complaints but that -1L does above? that one's surely a bug). To be clear, Thomas proposed accepting it (whatever that means) until 3.0. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
On 5/29/06, Armin Rigo [EMAIL PROTECTED] wrote: Hi all, I've finally come around to writing a patch that stops dict lookup from eating all exceptions that occur during lookup, like rare bugs in user __eq__() methods. After another 2-hours long debugging session that turned out to be caused by that, I had a lot of motivation. http://python.org/sf/1497053 The patch doesn't change the PyDict_GetItem() interface, which is the historical core of the problem. It works around this issue by just moving the exception-eating bit there instead of in lookdict(), so it gets away with changing only dictobject.c (plus ceval.c's direct usage of ma_lookup for LOAD_GLOBAL). The benefit of this patch is that all other ways to work with dicts now correctly propagate exceptions, and this includes all the direct manipulation from Python code (including 'x=d[key]'). The reason I bring this up here is that I'm going to check it in 2.5, unless someone seriously objects. +1, as long as (as you seem to imply) PyDict_GetItem() still swallows all exceptions. About the objection we need a better fix, PyDict_GetItem() should really be fixed and all its usages changed: this would be good, and also require some careful compatibility considerations, and quite some work in total; it would also give a patch which is basically a superset of mine, so I don't think I'm going in the wrong direction there. Fixing PyDict_GetItem() is a py3k issue, I think. Until then, there are way too many uses. I wouldn't be surprised if after INCREF and DECREF it's the most commonly used C API method... -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] feature request: inspect.isgenerator
Michele Simionato [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] Is there still time for new feature requests in Python 2.5? I am missing a isgenerator function in the inspect module. Right now I am using def isgenerator(func): return func.func_code.co_flags == 99 but it is rather ugly (horrible indeed). To me, part of the beauty of Python is that is so easy to write such things so compactly, so that we do not need megaAPIs with hundreds of functions and methods. tjr ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
On 5/29/06, Tim Peters [EMAIL PROTECTED] wrote: I think we should do as Thomas proposes: plan to make it an error in 2.6 (or 2.7 if there's a big outcry, which I don't expect) and accept it with a warning in 2.5. That's what I arrived at, although 2.4.3's checking behavior is actually so inconsistent that it needs some defining (what exactly are we trying to still accept? e.g., that -1 doesn't trigger I complaints but that -1L does above? that one's surely a bug). No, it reflects that (up to 2.3 I believe) 0x was -1 but 0xL was 4294967295L. To be clear, Thomas proposed accepting it (whatever that means) until 3.0. Fine with me. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
Perhaps more people could chime in? Am I being too anal about backward compatibility here? As a sometimes bug report reviewer, I would like the reported discrepancy between the public docs and visible code behavior fixed one way or the other (by changing the docs or code) since that is my working definition for whether something is a bug or not. Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remove METH_OLDARGS?
On 5/28/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Georg Brandl wrote: There's still a ton used under Modules. Also, if no flag is specified, it will default to 0 (ie, METH_OLDARGS). I wonder how many third party modules use METH_OLDARGS directly or more likely indirectly. These modules can be converted. I think that should be done for 2.5, but nothing else. Or, perhaps a deprecation warning could be generated if it is still used. I'll let Martin decide this one. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Hi Guido, On Mon, May 29, 2006 at 12:34:30PM -0700, Guido van Rossum wrote: +1, as long as (as you seem to imply) PyDict_GetItem() still swallows all exceptions. Yes. Fixing PyDict_GetItem() is a py3k issue, I think. Until then, there are way too many uses. I wouldn't be surprised if after INCREF and DECREF it's the most commonly used C API method... Alternatively, we could add a new C API function, and gradually replace PyDict_GetItem() uses with the the new one. I can't think of an obvious name, though... Maybe code should just start using PyMapping_GetItem() instead. It's not incredibly slower than PyDict_GetItem(), at least in the non-KeyError case. A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Hi Raymond, On Mon, May 29, 2006 at 12:20:44PM -0700, Raymond Hettinger wrote: I've finally come around to writing a patch that stops dict lookup from eating all exceptions that occur during lookup, like rare bugs in user __eq__() methods. Is there a performance impact? I believe that this patch is good anyway, because I consider my (and anybody's) debugging hours worth more than a few seconds of a long-running process. You get *really* obscure bugs this way. I would also point out that this is the kind of feature that should not be traded off for performance, otherwise we'd loose much of the point of Python. IMHO. As it turns out, I measured only 0.5% performance loss in Pystone. A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r46300 - in python/trunk: Lib/socket.py Lib/test/test_socket.py Lib/test/test_struct.py Modules/_struct.c Modules/arraymodule.c Modules/socketmodule.c
[python-checkins] * Added socket.recv_buf() and socket.recvfrom_buf() methods, that use the buffer protocol (send and sendto already did). * Added struct.pack_to(), that is the corresponding buffer compatible method to unpack_from(). [Guido] Hm... The file object has a similar method readinto(). Perhaps the methods introduced here could follow that lead instead of using two different new naming conventions? On 5/27/06, Bob Ippolito [EMAIL PROTECTED] wrote: (speaking specifically about struct and not socket) pack_to and unpack_from are named as such because they work with objects that support the buffer API (not file-like-objects). I couldn't find any existing convention for objects that manipulate buffers in such a way. Actually, fileobject.readinto(bufferobject) is the convention I started long ago (at the time the buffer was introduced. If there is an existing convention then I'd be happy to rename these. readinto seems to imply that some kind of position is being incremented. No -- it's like read() but instead of returning a new object it reads into an object you pass. Grammatically it only works if it's implemented on all buffer objects, but in this case it's implemented on the Struct type. It looks like structobject.pack_to(bufferobject, other args) is similar to fileobject.readinto(bufferobject) in that the buffer object receives the result of packing / reading. (Files don't have a writefrom() method because write() can be polymorphic. read() can't be due to the asymmetry in the API. I guess struct objects are different.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] feature request: inspect.isgenerator
Terry Reedy wrote: def isgenerator(func): return func.func_code.co_flags == 99 but it is rather ugly (horrible indeed). To me, part of the beauty of Python is that is so easy to write such things so compactly, so that we do not need megaAPIs with hundreds of functions and methods. so what's so easy about the magic constant 99 ? is there some natural and obvious connection between generators and that number that I'm missing, or is that constant perhaps best hidden inside some introspection support module? (I'm still not sure about the use case, though. shouldn't a program treat a generator just like any other callable that returns an iterator?) /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 42 - New kind of Temporary file
I think it's a very minor feature at best -- but I do admit to having implemented this at least once so I'm not against adding it as a new feature to the tempfile module. It's getting awfully late for 2.5 though... (feature freeze is at 2.5b1 which is planned for June 14 IIRC). --Guido On 5/28/06, Andrew Chambers [EMAIL PROTECTED] wrote: The mailing-list bot said introduce myself so... I'm Andy Chambers. I've been lurking on the web-interface to the list for a while. I recently started trying to implement one of the items on PEP 42 so I thought I should join the list and make myself known. The item I'm working on is the new kind of temporary file that only turns into a file when required (before which time presumably its a StringIO). Do we still want this? I suggest it be exposed as a class TempFile in the tempfile module. If yes, I'll write the test cases and submit a patch. Regards, Andy ___ Inbox full of spam? Get leading spam protection and 1GB storage with All New Yahoo! Mail. http://uk.docs.yahoo.com/nowyoucan.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remove METH_OLDARGS?
Neal Norwitz wrote: Also, it would be easy to detect METH_OLDARGS in PyCFunction_New and raise an appropriate exception. I agree with Martin this should raise a deprecation warning in 2.5. why? this is a clear case of unnecessary meddling. removing it won't remove much code (a whopping 11 lines is dedicated to this), nor give any speed ups whatsoever; all you're doing is generating extra work and support issues for a bunch of third-party developers. trust me, we have better things to do with our time. -1 on meddling with this before 3.0. /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] feature request: inspect.isgenerator
Fredrik Lundh wrote: Terry Reedy wrote: def isgenerator(func): return func.func_code.co_flags == 99 but it is rather ugly (horrible indeed). To me, part of the beauty of Python is that is so easy to write such things so compactly, so that we do not need megaAPIs with hundreds of functions and methods. so what's so easy about the magic constant 99 ? is there some natural and obvious connection between generators and that number that I'm missing, or is that constant perhaps best hidden inside some introspection support module? It seems to be a combination of CO_NOFREE, CO_GENERATOR, CO_OPTIMIZED and CO_NEWLOCALS. The first four CO_ constants are already in inspect.py, the newer ones (like CO_GENERATOR) aren't. I wonder whether a check shouldn't just return (co_flags 0x20), which is CO_GENERATOR. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Armin Rigo wrote: As it turns out, I measured only 0.5% performance loss in Pystone. umm. does Pystone even call lookdict? /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
On May 29, 2006, at 12:44 PM, Guido van Rossum wrote: On 5/29/06, Tim Peters [EMAIL PROTECTED] wrote: I think we should do as Thomas proposes: plan to make it an error in 2.6 (or 2.7 if there's a big outcry, which I don't expect) and accept it with a warning in 2.5. That's what I arrived at, although 2.4.3's checking behavior is actually so inconsistent that it needs some defining (what exactly are we trying to still accept? e.g., that -1 doesn't trigger I complaints but that -1L does above? that one's surely a bug). No, it reflects that (up to 2.3 I believe) 0x was -1 but 0xL was 4294967295L. Python 2.3 did a FutureWarning on 0x but its value was -1. Anyway, my plan is to make it such that all non-native format codes will behave exactly like C casting, but will do a DeprecationWarning for input numbers that were initially out of bounds. This behavior will be consistent across (python) int and long, and will be easy enough to explain in the docs (but still more complicated than values not representable by this data type will raise struct.error). This means that I'm also changing it so that struct.pack will not raise OverflowError for some longs, it will always raise struct.error or do a warning (as long as the input is int or long). Pseudocode looks kinda like this: def wrap_unsigned(x, CTYPE): if not (0 = x = CTYPE_MAX): DeprecationWarning() x = CTYPE_MAX return x Actually, should this be a FutureWarning or a DeprecationWarning? -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
[Guido] I think we should do as Thomas proposes: plan to make it an error in 2.6 (or 2.7 if there's a big outcry, which I don't expect) and accept it with a warning in 2.5. [Tim] That's what I arrived at, although 2.4.3's checking behavior is actually so inconsistent that it needs some defining (what exactly are we trying to still accept? e.g., that -1 doesn't trigger I complaints but that -1L does above? that one's surely a bug). [Guido] No, it reflects that (up to 2.3 I believe) 0x was -1 but 0xL was 4294967295L. But is it a bug _now_? That's what I mean by a bug. To me, this is simply inexplicable in 2.4 (and 2.5+): struct.pack(I, -1) # neither std nor native modes complain '\xff\xff\xff\xff' struct.pack(I, -1) '\xff\xff\xff\xff' struct.pack(I, -1L) # but both complain if the input is long Traceback (most recent call last): File stdin, line 1, in ? OverflowError: can't convert negative value to unsigned long struct.pack(I, -1L) Traceback (most recent call last): File stdin, line 1, in ? OverflowError: can't convert negative value to unsigned long Particulary for the standard modes, the behavior also varies across platforms (i.e., it wasn't true in any version of Python that 0x == -1 on most 64-bit boxes, and to have standard mode behavior vary according to platform isn't particularly standard :-)). To be clear, Thomas proposed accepting it (whatever that means) until 3.0. Fine with me. So who has a definition for what it means? I don't. Does every glitch have to be reproduced across the cross product of platform X format-code X input-type X native-vs-standard X direction-of-out-of-range? Or would people be happier if struct simply never checked for out-of-bounds? At least the latter is doable with finite effort, and is also explainable (for an integral code of N bytes, pack() stores the least-significant N bytes of the input viewed as a 2's-complement integer). I'd be happier with that than with something that can't be explained short of exhaustive listing of cases across 5 dimensions. Ditto with saying that for an integral type using N bytes, pack() accepts any integer in -(2**(8*N-1)) through 2**(8*N)-1, complains outside that range, and makes no complaining distinctions based on platform, input type, standard-vs-native mode, signed-or-unsigned format code, or direction of out-of-range. That's also explainable. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Contributor agreements (was Re: DRAFT: python-dev summary for 2006-04-01 to 2006-04-15)
Terry Reedy wrote: .. _contributor agreement: http://www.python.org/psf/contrib-form-python.html By itself, this link is inadequate since there is a blank for 'Initial License' with no choices indicated. So the summary needs a link to an instruction page also. The form is also a bit confusing since it appears to talk about future contributions but the last line tacked on to the bottom refers to the past. If it means both, then why not say so in the main text in a way that non-lawyers can understand? That's the modified version of the form for people like me that have contributed stuff before sending the form in (I have one sitting on my desk that will be going in the mail real soon now, honest. . .). There's another page somewhere in the PSF section that talks about contributor agreements in general - maybe that would be a better link for the summary? Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
On 5/29/06, Tim Peters [EMAIL PROTECTED] wrote: [Tim] To be clear, Thomas proposed accepting it (whatever that means) until 3.0.[Guido] Fine with me.So who has a definition for what it means? I know which 'it' I meant: the same 'it' as struct already accepts in Python 2.4 and before. Yes, it's inconsistent between formatcodes and valuetypes -- fixing that the whole point of the change -- but that's how you define 'compatibility'; struct, by default, should do what it did for Python 2.4, for all operating modes. It doesn't have to be more liberal than 2.4 (and preferably shouldn't, as that could break backward compatibility of some code -- much less common, though.)Making a list of which formatcodes accept what values (for what valuetypes) for 2.4 is easy enough (and should be added to the test suite, too ;-) -- I can do that in a few days if no one gets to it.-- Thomas Wouters [EMAIL PROTECTED] Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remove METH_OLDARGS?
On 5/29/06, Fredrik Lundh [EMAIL PROTECTED] wrote: this is a clear case of unnecessary meddling.removing it won't removemuch code (a whopping 11 lines is dedicated to this), nor give any speedups whatsoever; all you're doing is generating extra work and support issues for a bunch of third-party developers.trust me, we have betterthings to do with our time.-1 on meddling with this before 3.0.-1 from me, too, for the same reason. It would be nice if the use of PyArg_Parse could generate a (C) compile-time warning, but since it can't, I think a runtime warning for this minor case is just overkill. (The C compile-time warning would be useful in other cases, too... hmm, perhaps we can do some post-processing of .o files, looking for deprecated symbols left undefined...) -- Thomas Wouters [EMAIL PROTECTED]Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
Thomas Wouters wrote: I know which 'it' I meant: the same 'it' as struct already accepts in Python 2.4 and before. Yes, it's inconsistent between formatcodes and valuetypes -- fixing that the whole point of the change -- but that's how you define 'compatibility'; struct, by default, should do what it did for Python 2.4, for all operating modes. that's how you define compatibility for bug fix releases in Python. 2.5 is not 2.4.4. /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remove METH_OLDARGS?
Thomas Wouters wrote: On 5/29/06, *Fredrik Lundh* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: this is a clear case of unnecessary meddling. removing it won't remove much code (a whopping 11 lines is dedicated to this), nor give any speed ups whatsoever; all you're doing is generating extra work and support issues for a bunch of third-party developers. trust me, we have better things to do with our time. -1 on meddling with this before 3.0. -1 from me, too, for the same reason. It would be nice if the use of PyArg_Parse could generate a (C) compile-time warning, but since it can't, I think a runtime warning for this minor case is just overkill. (The C compile-time warning would be useful in other cases, too... hmm, perhaps we can do some post-processing of .o files, looking for deprecated symbols left undefined...) Isn't there at least a GCC __attribute__((deprecated))? Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
On 5/29/06, Fredrik Lundh [EMAIL PROTECTED] wrote: Thomas Wouters wrote: I know which 'it' I meant: the same 'it' as struct already accepts in Python 2.4 and before. Yes, it's inconsistent between formatcodes and valuetypes -- fixing that the whole point of the change -- but that's how you define 'compatibility'; struct, by default, should do what it did for Python 2.4, for all operating modes.that's how you define compatibility for bug fix releases in Python.2.5is not 2.4.4.Correct, and it is also not 2.6. Breaking perfectly working (and more to the point, non-complaining) code, even if it is for a good reason, is bad. It means, for instance, that I can not upgrade Python on any of the servers I manage, where Python gets used by clients. This is not a hypothetical problem, I've had it happen too often (although fortunately not often with Python, because of the sane policy on backward compatibility.) If 2.5 warns and does the old thing, the upgrade path is easy and defendable. This is also why there are future statements -- I distinctly recall making the same argument back then :-) The cost of continuing the misfeatures in struct for one release does not weigh up to the cost of breaking compatibility unwarned. -- Thomas Wouters [EMAIL PROTECTED]Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
On Mon, May 29, 2006 at 12:20:44PM -0700, Raymond Hettinger wrote: I've finally come around to writing a patch that stops dict lookup from eating all exceptions that occur during lookup, like rare bugs in user __eq__() methods. Is there a performance impact? I believe that this patch is good anyway, because I consider my (and anybody's) debugging hours worth more than a few seconds of a long-running process. You get *really* obscure bugs this way. I would also point out that this is the kind of feature that should not be traded off for performance, otherwise we'd loose much of the point of Python. IMHO. As it turns out, I measured only 0.5% performance loss in Pystone. Please run some better benchmarks and do more extensive assessments on the performance impact. The kind of obscure bug you're trying to kill does not affect 99.9% of Python users; however, loss of performance will affect everyone. This is arguably the most actively exercised part of the Python and should not be changed without carefully weighing purity vs practicality. FWIW, I applied a similar patch to set objects in Py2.5 so that they wouldn't eat exceptions. So, I'm truly sympathetic to the cause. However, dicts are much more critical. There needs to be a careful judgement based on measurements and assessments of whether there are real benefits for everyday Python users. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remove METH_OLDARGS?
Georg Brandl wrote: Isn't there at least a GCC __attribute__((deprecated))? Yes, but it applies to functions only. I guess you could make a __deprecated__ function, and then expand METH_OLDARGS to a call of that function. That won't help in cases where there are no flags given at all in the method structures. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssize_t question: longs in header files
Neal Norwitz wrote: minus comments, etc yields several questions about whether some values should use Py_ssize_t rather than C longs. In particular: * hash values Include/abstract.h: long PyObject_Hash(PyObject *o); // also in object.h Include/object.h:typedef long (*hashfunc)(PyObject *); I don't think these should be Py_ssize_t. Of course, a pointer type cannot naturally convert to a long on some systems, so where we use a pointer as a hash, we should do so carefully if sizeof(void*) sizeof(long); an xor of low and high word is probably good enough. Currently, the code does something way more complex (actually creating a long object); that should be replaced either by inlining long_hash, or by using an even simpler hash function for 64-bit pointers. * ints: Include/intobject.h:long ob_ival; As Tim says, this is way out of scope for 2.5. Guido said it is ok to change this to 64-bit ints in 2.6, but I expect some embedded system developers will start screaming when they hear that: 64-bit arithmetic is expensive on a 32-bit machine. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remove METH_OLDARGS?
Martin v. Löwis wrote: Georg Brandl wrote: Isn't there at least a GCC __attribute__((deprecated))? Yes, but it applies to functions only. I guess you could make a __deprecated__ function, and then expand METH_OLDARGS to a call of that function. That won't help in cases where there are no flags given at all in the method structures. I thought more about PyArg_Parse for __deprecated__. If you look at the patch on SF, it's creating warnings whenever a new CFunction is created with a METH_OLDARGS flag. I'm not sure this is the right way since it may mean that users of extensions using METH_OLDARGS might get a warning each time they get a bound method using it. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Hi Raymond, On Mon, May 29, 2006 at 02:02:25PM -0700, Raymond Hettinger wrote: Please run some better benchmarks and do more extensive assessments on the performance impact. At the moment, I'm trying to, but 2.5 HEAD keeps failing mysteriously on the tests I try to time, and even going into an infinite loop consuming all my memory - since the NFS sprint. Am I allowed to be grumpy here, and repeat that speed should not be used to justify bugs? I'm proposing a bug fix, I honestly don't care about 0.5% of speed. With a benchmark that passed, and that heavily uses instance and class attribute look-ups (richards), I don't even see any relevant difference. and assessments of whether there are real benefits for everyday Python users. It would have saved me two hours-long debugging sessions, and I consider myself an everyday Python user, so yes, I think so. Grumpy-ly yours, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssize_t question: longs in header files
On 5/29/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Neal Norwitz wrote: minus comments, etcyields several questions about whether some values should use Py_ssize_t rather than C longs.In particular: * ints:Include/intobject.h:long ob_ival; As Tim says, this is way out of scope for 2.5. Guido said it is okto change this to 64-bit ints in 2.6, but I expect some embeddedsystem developers will start screaming when they hear that: 64-bitarithmetic is expensive on a 32-bit machine. Well, those systems shouldn't have a 64-bit Py_ssize_t anyway, should they?-- Thomas Wouters [EMAIL PROTECTED]Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssize_t: ints in header files
Neal Norwitz wrote: Should the following values be ints (limited to 2G)? * dict counts (ma_fill, ma_used, ma_mask) I think Tim said he'll look into them. * line #s and column #s I think we should really formalize a limit, and then enforce it throughout. column numbers shouldn't exceed 16-bits, and line #s shouldn't exceed 31 bits. * AST (asdl.h) sequences we should first limit all the others, and then it should come out naturally that AST sequences can be happily limited to 31 bits. * recursion limit This should be Py_ssize_t. While the stack is typically more limited, it should be possible to configure it to exceed 4GiB on a 64-bit machine. * read/write/send/recv results and buf size This is very tricky. Often, the underlying C library has int there (e.g. msvcrt). Eventually, we should get rid of msvcrt, and then hope that the underlying system API can deal with larger buffers. * code object values like # args, # locals, stacksize, # instructions IMO, they should all get formally limited to 15 bits (i.e. short). I think some are already thus limited through the byte code format. Somebody would have to check, and document what the limits are. Either we end up with different limits for each one, or, more likely, the same limit, in which case I would suggest to introduce another symbolic type (e.g. Py_codesize_t or some such). Then we should consistently reject source code that exceeds such a limit, and elsewhere rely on the guarantee that the values will be always limited much more than the underlying data structures. * sre (i think i have a patch to fix this somewhere) This is a huge set of changes, I think. SRE *should* support strings larger than 4GiB. I could happily accept limitations for the regexes themselves (e.g. size of the compiled expression, number of repeats, etc). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Contributor agreements (was Re: DRAFT: python-dev summary for 2006-04-01 to 2006-04-15)
On 5/29/06, Nick Coghlan [EMAIL PROTECTED] wrote: Terry Reedy wrote: .. _contributor agreement: http://www.python.org/psf/contrib-form-python.html By itself, this link is inadequate since there is a blank for 'Initial License' with no choices indicated. So the summary needs a link to an instruction page also. The form is also a bit confusing since it appears to talk about future contributions but the last line tacked on to the bottom refers to the past. If it means both, then why not say so in the main text in a way that non-lawyers can understand? That's the modified version of the form for people like me that have contributed stuff before sending the form in (I have one sitting on my desk that will be going in the mail real soon now, honest. . .). There's another page somewhere in the PSF section that talks about contributor agreements in general - maybe that would be a better link for the summary? Sounds like it. (That wasn't anywhere in the thread.) Does http://www.python.org/psf/contrib/ seem like the right page to link to? Thanks, STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssize_t question: longs in header files
Thomas Wouters wrote: Neal Norwitz wrote: minus comments, etc yields several questions about whether some values should use Py_ssize_t rather than C longs. In particular: * ints: Include/intobject.h:long ob_ival; As Tim says, this is way out of scope for 2.5. Guido said it is ok to change this to 64-bit ints in 2.6, but I expect some embedded system developers will start screaming when they hear that: 64-bit arithmetic is expensive on a 32-bit machine. Well, those systems shouldn't have a 64-bit Py_ssize_t anyway, should they? No - but Guido said he wanted Python int to have a 64-bit representation everywhere, not a Py_ssize_t representation. I agree using Py_ssize_t would be a smaller change, and one that likely has less performance impact. It would still be a large change, and should be only done if we know for sure we don't want it to be a 64-bit type always the next day. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remove METH_OLDARGS?
Georg Brandl wrote: I thought more about PyArg_Parse for __deprecated__. Ah, PyArg_Parse can, of course. Having it actually do that should not hurt, either - but you need a reliable check whether the compiler supports __deprecated__. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remove METH_OLDARGS?
Guido van Rossum wrote: I think that should be done for 2.5, but nothing else. Or, perhaps a deprecation warning could be generated if it is still used. I'll let Martin decide this one. I just sent a message to some that producing a DeprecationWarning is fine, but now I read some opposition; given that this really just needs very little code to support it, and given that run-time warnings annoy the users, not the module authors, I agree that this should stay silently until 3.0. We should still continue to remove all uses we have ourselves. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ssize_t question: longs in header files
On 5/29/06, Martin v. Löwis [EMAIL PROTECTED] wrote: I agree using Py_ssize_t would be a smaller change, and one thatlikely has less performance impact. It would still be a large change,and should be only done if we know for sure we don't want it to be a 64-bit type always the next day.Well, implementing PyInt in terms of a new symbolic type, even if it defaults to 64-bit, should make choosing it to be a 32-bit type a lot easier, shouldn't it? Not that I think defaulting PyInt to use 'long long' is a good thing, considering we still only compile 'long long' support optionally. Better to stick with a common native type until all machines have 64-bit registers. Better for performance, too.But switching PyInts to use (a symbolic type of the same size as) Py_ssize_t means that, when the time comes that 32-bit architectures are rare, Win64 isn't left as the only platform (barring other LLP64 systems) that has slow 33-to-64-bit Python numbers (because they'd be PyLongs there, even though the platform has 64-bit registers.) Given the timeframe and the impact, though, perhaps we should just do it -- now -- in the p3yk branch and forget about 2.x; gives people all the more reason to switch, two years from now.-- Thomas Wouters [EMAIL PROTECTED]Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Armin Rigo wrote: At the moment, I'm trying to, but 2.5 HEAD keeps failing mysteriously on the tests I try to time, and even going into an infinite loop consuming all my memory - since the NFS sprint. Am I allowed to be grumpy here, and repeat that speed should not be used to justify bugs? not unless you can produce some code. unfounded accusations don't belong on this list (it's not like the sprinters didn't test the code on a whole bunch of platforms), and neither does lousy benchmarks (why are you repeating the 0.5% figure when pystone doesn't even test non-string dictionary behaviour? PyString_Eq cannot fail...) /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Re-hi,
On Mon, May 29, 2006 at 11:34:28PM +0200, Armin Rigo wrote:
At the moment, I'm trying to, but 2.5 HEAD keeps failing mysteriously on
the tests I try to time, and even going into an infinite loop consuming
all my memory
Ah, it's a corner case of str.find() whose behavior just changed.
Previously, 'abc'.find('', 100) would return -1, and now it returns 100.
Just to confuse matters, the same test with unicode returns 100, and has
always done so in the past. (Oh well, one of these again...)
So, we need to decide which behavior is right. One could argue that
the current 2.5 HEAD now has a consistent behavior, so it could be kept;
but there is an opposite argument as well, which is that some existing
programs like the one I was testing are now thrown into annoying
infinite loops because str.find() never returns -1 any more, even with
larger and larger start arguments. I believe that it's a pattern that
could be common in string-processing scripts, trying to match substrings
at various points in a string trusting that str.find() will eventually
return -1. It's harder to think of a case where a program previously
relied on unicode.find('',n) never returning -1. Also, introducing a
new way for programs to be caught in an infinite loop is probably not a
good idea.
Hum, my apologies for my grumpy comments about the NFS sprint. At
least, the unification of the string and unicode algorithm that was
started there is a good move, also because it exposes pre-existing
inconsistencies.
A bientot,
Armin.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integer representation (Was: ssize_t question: longs in header files)
Thomas Wouters wrote:
But switching PyInts to use (a symbolic type of the same size as)
Py_ssize_t means that, when the time comes that 32-bit architectures are
rare, Win64 isn't left as the only platform (barring other LLP64
systems) that has slow 33-to-64-bit Python numbers (because they'd be
PyLongs there, even though the platform has 64-bit registers.) Given the
timeframe and the impact, though, perhaps we should just do it -- now --
in the p3yk branch and forget about 2.x; gives people all the more
reason to switch, two years from now.
I thought Py3k will have a single integer type whose representation
varies depending on the value being represented.
I haven't seen an actual proposal for such a type, so let me make
one:
struct PyInt{
struct PyObject ob;
Py_ssize_t value_or_size;
char is_long;
digit ob_digit[1];
};
If is_long is false, then value_or_size is the value (represented
as Py_ssize_t), else the value is in ob_digit, and value_or_size
is the size.
PyLong_* will be synonyms for PyInt_*. PyInt_FromLong/AsLong will
continue to exist; PyInt_AsLong will indicate an overflow with -1.
Likewise, PyArg_ParseTuple i will continue to produce int, and
raise an exception (OverflowError?) when the value is out of range.
C code can then decide whether to parse a Python integer as
C int, long, long long, or ssize_t.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remove METH_OLDARGS?
Martin v. Löwis wrote: Guido van Rossum wrote: I think that should be done for 2.5, but nothing else. Or, perhaps a deprecation warning could be generated if it is still used. I'll let Martin decide this one. I just sent a message to some that producing a DeprecationWarning is fine, but now I read some opposition; given that this really just needs very little code to support it, and given that run-time warnings annoy the users, not the module authors, I agree that this should stay silently until 3.0. We should still continue to remove all uses we have ourselves. The only modules METH_OLDARGS is now used in are fl, sv, gl, cl and Tkinter (for the latter there is my patch on SF for review). sv and cl are listed in PEP 356 as possibly being removed, I don't know whether the other ones are used anymore. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Armin Rigo wrote:
Ah, it's a corner case of str.find() whose behavior just changed.
Previously, 'abc'.find('', 100) would return -1, and now it returns 100.
Just to confuse matters, the same test with unicode returns 100, and has
always done so in the past. (Oh well, one of these again...)
So, we need to decide which behavior is right. One could argue that
the current 2.5 HEAD now has a consistent behavior, so it could be kept;
but there is an opposite argument as well, which is that some existing
programs like the one I was testing are now thrown into annoying
infinite loops because str.find() never returns -1 any more, even with
larger and larger start arguments. I believe that it's a pattern that
could be common in string-processing scripts, trying to match substrings
at various points in a string trusting that str.find() will eventually
return -1.
well, the empty string is a valid substring of all possible strings
(there are no null strings in Python). you get the same behaviour
from slicing, the in operator, replace (this was discussed on the
list last week), count, etc.
if you're actually searching for a *non-empty* string, find() will
always return -1 sooner or later.
Hum, my apologies for my grumpy comments about the NFS sprint. At
least, the unification of the string and unicode algorithm that was
started there is a good move, also because it exposes pre-existing
inconsistencies.
here's my current favourite:
abc.count(, 100)
-96
/F
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Hi Fredrik, On Tue, May 30, 2006 at 12:01:46AM +0200, Fredrik Lundh wrote: not unless you can produce some code. unfounded accusations don't belong on this list (it's not like the sprinters didn't test the code on a whole bunch of platforms), and neither does lousy benchmarks (why are you repeating the 0.5% figure when pystone doesn't even test non-string dictionary behaviour? PyString_Eq cannot fail...) Sorry, I do apologize for my wording. I must admit that I was a bit apalled by the amount of reference leaks that Michael had to fix after the sprint, so jumped to conclusions a bit too fast when I saw by 1GB laptop swap after less than a minute. See my e-mail, which crossed yours, for the explanation. The reason I did not quote examples involving non-string dicts is that my patch makes the non-string case simpler, so -- as I expected, and as I have now measured -- marginally faster. All in all it's hard to say if there is a global consistent performance change. At this point I'd rather like to spend my time more interestingly; this might be by defending my point of view that very minor performance hits should not get in the way of fixes that avoid very obscure bugs, even only occasionally-occuring but still very obscure bugs. A bientot, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Armin Rigo wrote: Hi Raymond, On Mon, May 29, 2006 at 02:02:25PM -0700, Raymond Hettinger wrote: Please run some better benchmarks and do more extensive assessments on the performance impact. At the moment, I'm trying to, but 2.5 HEAD keeps failing mysteriously on the tests I try to time, and even going into an infinite loop consuming all my memory - since the NFS sprint. Am I allowed to be grumpy here, and repeat that speed should not be used to justify bugs? I'm proposing a bug fix, I honestly don't care about 0.5% of speed. If it is really 0.5%, then we're fine. Just remember that PyStone is an amazingly uninformative and crappy benchmark. The "justify bugs" terminology is pejorative and inaccurate. It is clear that the current dict behavior was a concious design decision and documented as such. Perhaps the decision sucked and should be changed, but it is not a bug. and I consider myself an everyday Python user, Something may have been lost in translation. Using it every day is not the same as being an everyday user ;-) There is no doubt that you routinely stress the language in ways ways that are not at all commonplace. All I'm asking is that there be a well thought-out assessment of whether the original design decision was struck the appropriate balance between practicality Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Hi Fredrik, On Tue, May 30, 2006 at 12:23:04AM +0200, Fredrik Lundh wrote: well, the empty string is a valid substring of all possible strings (there are no null strings in Python). you get the same behaviour from slicing, the in operator, replace (this was discussed on the list last week), count, etc. if you're actually searching for a *non-empty* string, find() will always return -1 sooner or later. I know this. These corner cases are debatable and different answers could be seen as correct, as I think is the case for find(). My point was different: I was worrying that the recent change in str.find() would needlessly send existing and working programs into infinite loops, which can be a particularly bad kind of failure for some applications. At least, it gave a 100% performance loss on the benchmark I was trying to run :-) A bientot, Armin. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
On May 29, 2006, at 1:18 PM, Bob Ippolito wrote:
On May 29, 2006, at 12:44 PM, Guido van Rossum wrote:
On 5/29/06, Tim Peters [EMAIL PROTECTED] wrote:
I think we should do as Thomas proposes: plan to make it an
error in
2.6 (or 2.7 if there's a big outcry, which I don't expect) and
accept
it with a warning in 2.5.
That's what I arrived at, although 2.4.3's checking behavior is
actually so inconsistent that it needs some defining (what exactly
are we trying to still accept? e.g., that -1 doesn't trigger I
complaints but that -1L does above? that one's surely a bug).
No, it reflects that (up to 2.3 I believe) 0x was -1 but
0xL was 4294967295L.
Python 2.3 did a FutureWarning on 0x but its value was -1.
Anyway, my plan is to make it such that all non-native format codes
will behave exactly like C casting, but will do a DeprecationWarning
for input numbers that were initially out of bounds. This behavior
will be consistent across (python) int and long, and will be easy
enough to explain in the docs (but still more complicated than
values not representable by this data type will raise struct.error).
This means that I'm also changing it so that struct.pack will not
raise OverflowError for some longs, it will always raise struct.error
or do a warning (as long as the input is int or long).
Pseudocode looks kinda like this:
def wrap_unsigned(x, CTYPE):
if not (0 = x = CTYPE_MAX):
DeprecationWarning()
x = CTYPE_MAX
return x
Actually, should this be a FutureWarning or a DeprecationWarning?
OK, this behavior is implemented in revision 46537:
(this is from ./python.exe -Wall)
import struct
struct.pack('B', 256)
Traceback (most recent call last):
File stdin, line 1, in module
File /Users/bob/src/python/Lib/struct.py, line 63, in pack
return o.pack(*args)
struct.error: ubyte format requires 0 = number = 255
struct.pack('B', -1)
Traceback (most recent call last):
File stdin, line 1, in module
File /Users/bob/src/python/Lib/struct.py, line 63, in pack
return o.pack(*args)
struct.error: ubyte format requires 0 = number = 255
struct.pack('B', 256)
/Users/bob/src/python/Lib/struct.py:63: DeprecationWarning: 'B'
format requires 0 = number = 255
return o.pack(*args)
'\x00'
struct.pack('B', -1)
/Users/bob/src/python/Lib/struct.py:63: DeprecationWarning: struct
integer wrapping is deprecated
return o.pack(*args)
/Users/bob/src/python/Lib/struct.py:63: DeprecationWarning: 'B'
format requires 0 = number = 255
return o.pack(*args)
'\xff'
struct.pack('B', 256L)
/Users/bob/src/python/Lib/struct.py:63: DeprecationWarning: 'B'
format requires 0 = number = 255
return o.pack(*args)
'\x00'
struct.pack('B', -1L)
/Users/bob/src/python/Lib/struct.py:63: DeprecationWarning: struct
integer wrapping is deprecated
return o.pack(*args)
/Users/bob/src/python/Lib/struct.py:63: DeprecationWarning: 'B'
format requires 0 = number = 255
return o.pack(*args)
'\xff'
In _struct.c, getting rid of the #define PY_STRUCT_WRAPPING 1 will
turn off this warning+wrapping nonsense and just raise errors for out
of range values. It'll also enable some additional performance hacks
(swapping out the host-endian table's pack and unpack functions with
the faster native versions).
-bob
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
Thomas Wouters wrote: If 2.5 warns and does the old thing, the upgrade path is easy and defendable. This is also why there are future statements -- I distinctly recall making the same argument back then :-) The cost of continuing the misfeatures in struct for one release does not weigh up to the cost of breaking compatibility unwarned. This really sounds to me like a __future__ import would be useful to get the fixed behaviour. Tim Delaney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Raymond Hettinger wrote:
If it is really 0.5%, then we're fine. Just remember that PyStone is an
amazingly uninformative and crappy benchmark.
Since Armin seems to not like having to justify his patch with any
performance testing, I wrote a handful of dict insertion exercises and
could find no real performance impact. In the case of an exception, it
was much faster, but that is because it is not inserting anything into
the dictionary.
IOW if it is a bad change in behavior. Previously, the exception was
swallowed and it was assumed to be a new key despite the exception. This
is an obscure use case that could creep up it's ugly head.
class K(int):
def __cmp__(self, o): raise Exception()
d = {}
for i in xrange(10):
d[K()] = i
for k in d.keys():
print d[k]
Despite the incomparability, this throws no error in previous versions
and the dict is still usable for the expected purpose.
--
Scott Dial
[EMAIL PROTECTED]
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Fredrik Lundh wrote: well, the empty string is a valid substring of all possible strings (there are no null strings in Python). you get the same behaviour from slicing, the in operator, replace (this was discussed on the list last week), count, etc. Although Tim pointed out that replace() only regards n+1 empty strings as existing in a string of lenth n. So for consistency, find() should only find them in those places, too. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] test_gzip/test_tarfile failure om AMD64
[Bob Ippolito]
...
Actually, should this be a FutureWarning or a DeprecationWarning?
Since it was never documented, UndocumentedBugGoingAwayError ;-)
Short of that, yes, DeprecationWarning. FutureWarning is for changes
in non-exceptional behavior (.e.g, if we swapped the meanings of
and in struct format codes, that would rate a FutureWarning
subclass, line InsaneFutureWarning).
OK, this behavior is implemented in revision 46537:
(this is from ./python.exe -Wall)
import struct
...
struct.pack('B', -1)
/Users/bob/src/python/Lib/struct.py:63: DeprecationWarning: struct
integer wrapping is deprecated
return o.pack(*args)
/Users/bob/src/python/Lib/struct.py:63: DeprecationWarning: 'B'
format requires 0 = number = 255
return o.pack(*args)
'\xff'
We certainly don't want to see two deprecation warnings for a single
deprecated behavior. I suggest eliminating the struct integer
wrapping warning, mostly because I had no idea what it _meant_
before reading the comments in _struct.c (wrapping is used most
often in a proxy or delegation context in Python these days). 'B'
format requires 0 = number = 255 is perfectly clear all by itself.
...
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
On 5/29/06, Greg Ewing [EMAIL PROTECTED] wrote: Fredrik Lundh wrote: well, the empty string is a valid substring of all possible strings (there are no null strings in Python). you get the same behaviour from slicing, the in operator, replace (this was discussed on the list last week), count, etc. Although Tim pointed out that replace() only regards n+1 empty strings as existing in a string of lenth n. So for consistency, find() should only find them in those places, too. And abc.count() should return 4. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integer representation (Was: ssize_t question: longs in header files)
[Adding the py3k list; please remove python-dev in followups.]
On 5/29/06, Martin v. Löwis [EMAIL PROTECTED] wrote:
I thought Py3k will have a single integer type whose representation
varies depending on the value being represented.
That's one proposal. Another is to have an abstract 'int' type with
two concrete subtypes, e.g. 'short' and 'long', corresponding to
today's int and long. At the C level the API should be unified so C
programmers are isolated from the difference (they aren't today).
I haven't seen an actual proposal for such a type,
I'm not sure that my proposal above has ever been said out loud. I'm
also not partial; I think we may have to do an experiment to decide.
so let me make one:
struct PyInt{
struct PyObject ob;
Py_ssize_t value_or_size;
char is_long;
digit ob_digit[1];
};
If is_long is false, then value_or_size is the value (represented
as Py_ssize_t), else the value is in ob_digit, and value_or_size
is the size.
Nice. I guess if we store the long value in big-endian order we could
drop is_long, since the first digit of the long would always be
nonzero. This would save a byte (on average) for the longs, but it
would do nothing for the wasted space for short ints.
PyLong_* will be synonyms for PyInt_*.
Why do we need to keep the PyLong_* APIs at all? Even at the Python
level we're not planning any backward compatibility features; at the C
level I like even more freedom to break things.
PyInt_FromLong/AsLong will
continue to exist; PyInt_AsLong will indicate an overflow with -1.
Likewise, PyArg_ParseTuple i will continue to produce int, and
raise an exception (OverflowError?) when the value is out of range.
C code can then decide whether to parse a Python integer as
C int, long, long long, or ssize_t.
Nice. I like the unified API and I like using Py_ssize_t instead of
long for the value; this ensures that an int can hold a pointer (if we
allow for signed pointers) and matches the native word size better on
Windows (I guess it makes no difference for any other platform, where
ssize_t and long already have the same size).
I worry about all the wasted space for alignment caused by the extra
flag byte though. That would be 4 byte per integer on 32-bit machines
(where they are currently 12 bytes) and 8 bytes on 64-bit machines
(where they are currently 24 bytes).
That's why I'd like my alternative proposal (int as ABC and two
subclasses that may remain anonymous to the Python user); it'll save
the alignment waste for short ints and will let us use a smaller int
type for the size for long ints (if we care about the latter).
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remove METH_OLDARGS?
On 5/29/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Georg Brandl wrote: I thought more about PyArg_Parse for __deprecated__. Ah, PyArg_Parse can, of course. Having it actually do that should not hurt, either - but you need a reliable check whether the compiler supports __deprecated__. Already done for gcc, see Py_DEPRECATED in pyport.h. Would be nice if someone could add support on Windows. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
[Greg Ewing] Although Tim pointed out that replace() only regards n+1 empty strings as existing in a string of lenth n. So for consistency, find() should only find them in those places, too. [Guido] And abc.count() should return 4. And it does, but too much context was missing in Greg's reply to make sense of his comment. In context Greg was seeming to say that abc.count(, 100) should return 0, and abc.find(, 100) should return -1, since the only empty substrings of abc are at slices 0:0, 1:1, 2:2 and 3:3. In fact we have abc.count(, 100) 1 uabc.count(, 100) 1 abc.find(, 100) 100 uabc.find(, 100) 100 today, although the idea that find() can return an index that doesn't exist in the string is particularly jarring. Since we also have: abc.rfind(, 100) 3 uabc.rfind(, 100) 3 it's twice as jarring that searching from the right finds the target at a _smaller_ index than searching from the left. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remove METH_OLDARGS?
On 5/29/06, Thomas Wouters [EMAIL PROTECTED] wrote:
On 5/29/06, Fredrik Lundh [EMAIL PROTECTED] wrote:
this is a clear case of unnecessary meddling. removing it won't remove
much code (a whopping 11 lines is dedicated to this), nor give any speed
ups whatsoever; all you're doing is generating extra work and support
issues for a bunch of third-party developers. trust me, we have better
things to do with our time.
-1 on meddling with this before 3.0.
-1 from me, too, for the same reason. It would be nice if the use of
PyArg_Parse could generate a (C) compile-time warning, but since it can't, I
think a runtime warning for this minor case is just overkill. (The C
compile-time warning would be useful in other cases, too... hmm, perhaps we
can do some post-processing of .o files, looking for deprecated symbols left
undefined...)
How can users find the implicit use of METH_OLDARGS in code like this:
static struct PyMethodDef gestalt_methods[] = {
{gestalt, gestalt_gestalt},
{NULL, NULL} /* Sentinel */
};
static PyMethodDef SwiMethods[]= {
{ swi, swi_swi,0},
{ NULL, NULL}
};
The first one was taken straight from one of Georg's checkins, the
second one was modified.
How many people knew we were still using METH_OLDARGS in these places?
I know this can be done with some simple scripts, but that won't
catch all cases. And many people won't run such a script, much less
write one.
OTOH, perhaps a deprecation warning on PyArgs_Parse() is sufficient?
What about that? It doesn't address other cases where OLDARGS are
used without PyArgs_Parse though.
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
[Armin Rigo] ... ... Am I allowed to be grumpy here, and repeat that speed should not be used to justify bugs? As a matter of fact, you are. OTOH, nobody at the sprint made that argument, so nobody actually feels shame on that count :-) I apologize for the insufficiently reviewed exception-rework patch anyway. The massive try/raise/except slowdown in 2.5a2 (compared to 2.4.3) was depressing and surprising, and figuring it out repairing it consumed various peoples' time across the whole week. By the time Saturday came around, I was fried enough that well, people have hacked on this all week, the slowdown finally got replaced by a speedup, and all the tests pass -- enough already, let's go home! ruled. So, as everyone suspected all along, it's Neal's fault for not running his refleak test more often ;-) ... It would have saved me two hours-long debugging sessions, and I consider myself an everyday Python user, so yes, I think so. I'll try to review your patch carefully Tuesday, but I'm willing to accept a slowdown for this too (although from a quick look I doubt there's a measurable one, and your approach is wholly sane). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] feature request: inspect.isgenerator
On 5/29/06, Georg Brandl [EMAIL PROTECTED] wrote: Fredrik Lundh wrote: is there some natural and obvious connection between generators and that number that I'm missing, or is that constant perhaps best hidden inside some introspection support module? It seems to be a combination of CO_NOFREE, CO_GENERATOR, CO_OPTIMIZED and CO_NEWLOCALS. The first four CO_ constants are already in inspect.py, the newer ones (like CO_GENERATOR) aren't. All these constants are declared in compiler.consts. Some are also defined in __future__. It would be better to have them all in a single place. I wonder whether a check shouldn't just return (co_flags 0x20), which is CO_GENERATOR. Makes more sense. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integer representation (Was: ssize_t question: longs in header files)
Guido van Rossum wrote:
struct PyInt{
struct PyObject ob;
Py_ssize_t value_or_size;
char is_long;
digit ob_digit[1];
};
Nice. I guess if we store the long value in big-endian order we could
drop is_long, since the first digit of the long would always be
nonzero. This would save a byte (on average) for the longs, but it
would do nothing for the wasted space for short ints.
Right; alternatively, the top-most bit of ob_digit[0] could also be
used, as longs have currently 15-bit digits.
Why do we need to keep the PyLong_* APIs at all? Even at the Python
level we're not planning any backward compatibility features; at the C
level I like even more freedom to break things.
Indeed, they should get dropped.
I worry about all the wasted space for alignment caused by the extra
flag byte though. That would be 4 byte per integer on 32-bit machines
(where they are currently 12 bytes) and 8 bytes on 64-bit machines
(where they are currently 24 bytes).
I think ints should get managed by PyMalloc in Py3k. With my proposal,
an int has 16 bytes on a 32-bit machine, so there wouldn't be any
wastage for PyMalloc (which allocates 16 bytes for 12-byte objects,
anyway). On a 64-bit machine, it would indeed waste 8 bytes per
int.
That's why I'd like my alternative proposal (int as ABC and two
subclasses that may remain anonymous to the Python user); it'll save
the alignment waste for short ints and will let us use a smaller int
type for the size for long ints (if we care about the latter).
I doubt they can remain anonymous. People often dispatch by type
(e.g. pickle, xmlrpclib, ...), and need to put the type into a
dictionary. If the type is anonymous, they will do
dispatch[type(0)] = marshal_int
dispatch[type(sys.maxint+1)] = marshal_int
Plus, their current code as
dispatch[int] = marshal_int
which will silently break (although it won't be silent if they also
have dispatch[long] = marshal_long).
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Guido van Rossum wrote: well, the empty string is a valid substring of all possible strings (there are no null strings in Python). you get the same behaviour from slicing, the in operator, replace (this was discussed on the list last week), count, etc. Although Tim pointed out that replace() only regards n+1 empty strings as existing in a string of lenth n. So for consistency, find() should only find them in those places, too. depends on how you interpret the reference to slices in the docs. abc[100:] is an empty string, and so is abc[100:100]. And abc.count() should return 4. it does, and has always done (afaik). abc.count(, 100) did use to return -96, though, which is hard to explain in terms of anything else. /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's stop eating exceptions in dict lookup
Armin Rigo wrote: I know this. These corner cases are debatable and different answers could be seen as correct, as I think is the case for find(). My point was different: I was worrying that the recent change in str.find() would needlessly send existing and working programs into infinite loops, which can be a particularly bad kind of failure for some applications. since abc.find(, 0) == 0, I would have thought that a program that searched for an empty string in a loop wouldn't get anywhere at all. /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
