Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes issue10761: tarfile.extractall failure when symlinked files are
On Thu, Apr 28, 2011 at 04:20:06PM +0200, Éric Araujo wrote: if hasattr(os, symlink) and hasattr(os, link): # For systems that support symbolic and hard links. if tarinfo.issym(): + if os.path.exists(targetpath): + os.unlink(targetpath) Is there a race condition here? The lock to avoid race conditions (if you were thinking along those lines) would usually be implemented at the higher level code which is using extractall in threads. A lock would only protect only against multi-threaded use of the tarfile module, which is probably quite rare and therefore not a real concern. The kind of race condition which can happen here is if an attacker creates targetpath between os.path.exists and os.unlink. Whether it is an exploitable flaw would need a detailed analysis, of course. Just out of curiosity, could you please elaborate on the potential threat of this? If the exists condition is true, targetpath already exists, so what use there is in overwriting it? If the condition is false, unlink isn't executed, so no harm either. What am I missing? Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Not-a-Number
Steven D'Aprano st...@pearwood.info writes: Robert Kern wrote: On 4/28/11 8:44 PM, Steven D'Aprano wrote: The real question should be, why does Python treat all NANs as signalling NANs instead of quiet NANs? I don't believe this helps anyone. Actually, Python treats all NaNs as quiet NaNs and never signalling NaNs. Sorry, did I get that backwards? I thought it was signalling NANs that cause a signal (in Python terms, an exception)? If I do x = 0.0/0 I get an exception instead of a NAN. Hence a signalling NAN. Robert has interpreted your “treats all NaNs as signalling NaNs” to mean “treats all objects that Python calls a NaN as signalling NaNs”, and is pointing out that no, the objects that Python calls “NaN” are all quiet NaNs. You might be clearer if you distinguish between what Python calls a NaN and what you call a NaN. It seems you're saying that some Python exception objects (e.g. ZeroDivisionError objects) are what you call NaNs, despite the fact that they're not what Python calls a NaN. -- \ “We can't depend for the long run on distinguishing one | `\ bitstream from another in order to figure out which rules | _o__) apply.” —Eben Moglen, _Anarchism Triumphant_, 1999 | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Not-a-Number
On Fri, Apr 29, 2011 at 3:28 PM, Steven D'Aprano st...@pearwood.info wrote: Robert Kern wrote: Actually, Python treats all NaNs as quiet NaNs and never signalling NaNs. Sorry, did I get that backwards? I thought it was signalling NANs that cause a signal (in Python terms, an exception)? If I do x = 0.0/0 I get an exception instead of a NAN. Hence a signalling NAN. Aside from the divide-by-zero case, we treat NaNs as quiet NaNs. This is largely due to the fact float operations are delegated to the underlying CPU, and SIGFPE is ignored by default. You can fiddle with it either by building and using the fpectl module, or else by switching to decimal.Decimal() instead (which offers much finer control over signalling through its thread local context information). The latter is by far the preferable course, unless you're targeting specific hardware with well-defined FPE behaviour. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal for a common benchmark suite
On Thu, Apr 28, 2011 at 11:10 PM, Stefan Behnel stefan...@behnel.de wrote: M.-A. Lemburg, 28.04.2011 22:23: Stefan Behnel wrote: DasIch, 28.04.2011 20:55: the CPython benchmarks have an extensive set of microbenchmarks in the pybench package Try not to care too much about pybench. There is some value in it, but some of its microbenchmarks are also tied to CPython's interpreter behaviour. For example, the benchmarks for literals can easily be considered dead code by other Python implementations so that they may end up optimising the benchmarked code away completely, or at least partially. That makes a comparison of the results somewhat pointless. The point of the micro benchmarks in pybench is to be able to compare them one-by-one, not by looking at the sum of the tests. If one implementation optimizes away some parts, then the comparison will show this fact very clearly - and that's the whole point. Taking the sum of the micro benchmarks only has some meaning as very rough indicator of improvement. That's why I wrote pybench: to get a better, more details picture of what's happening, rather than trying to find some way of measuring average use. This average is very different depending on where you look: for some applications method calls may be very important, for others, arithmetic operations, and yet others may have more need for fast attribute lookup. I wasn't talking about averages or sums, and I also wasn't trying to put down pybench in general. As it stands, it makes sense as a benchmark for CPython. However, I'm arguing that a substantial part of it does not make sense as a benchmark for PyPy and others. With Cython, I couldn't get some of the literal arithmetic benchmarks to run at all. The runner script simply bails out with an error when the benchmarks accidentally run faster than the initial empty loop. I imagine that PyPy would eventually even drop the loop itself, thus leaving nothing to compare. Does that tell us that PyPy is faster than Cython for arithmetic? I don't think it does. When I see that a benchmark shows that one implementation runs in 100% less time than another, I simply go *shrug* and look for a better benchmark to compare the two. I second here what Stefan says. This sort of benchmarks might be useful for CPython, but they're not particularly useful for PyPy or for comparisons (or any other implementation which tries harder to optimize stuff away). For example a method call in PyPy would be inlined and completely removed if method is empty, which does not measure method call overhead at all. That's why we settled on medium-to-large examples where it's more of an average of possible scenarios than just one. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Not-a-Number
Ben Finney wrote: Steven D'Aprano st...@pearwood.info writes: Robert Kern wrote: On 4/28/11 8:44 PM, Steven D'Aprano wrote: The real question should be, why does Python treat all NANs as signalling NANs instead of quiet NANs? I don't believe this helps anyone. Actually, Python treats all NaNs as quiet NaNs and never signalling NaNs. Sorry, did I get that backwards? I thought it was signalling NANs that cause a signal (in Python terms, an exception)? If I do x = 0.0/0 I get an exception instead of a NAN. Hence a signalling NAN. Robert has interpreted your “treats all NaNs as signalling NaNs” to mean “treats all objects that Python calls a NaN as signalling NaNs”, and is pointing out that no, the objects that Python calls “NaN” are all quiet NaNs. I'm sorry for my lack of clarity. I'm referring to functions which potentially produce NANs, not the exceptions themselves. A calculation which might have produced a (quiet) NAN as the result instead raises an exception (which I'm treating as equivalent to a signal). -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes issue10761: tarfile.extractall failure when symlinked files are
On Fri, Apr 29, 2011 at 4:26 PM, Eli Bendersky eli...@gmail.com wrote: On Thu, Apr 28, 2011 at 04:20:06PM +0200, Éric Araujo wrote: The kind of race condition which can happen here is if an attacker creates targetpath between os.path.exists and os.unlink. Whether it is an exploitable flaw would need a detailed analysis, of course. Just out of curiosity, could you please elaborate on the potential threat of this? If the exists condition is true, targetpath already exists, so what use there is in overwriting it? If the condition is false, unlink isn't executed, so no harm either. What am I missing? That's the detailed analysis part. What happens if other code deletes the path, and the unlink() call subsequently fails despite the successful exists() check? Hence why exception checking (as Nadeem posted) is typically the only right way to do things that access an external environment that supports multiple concurrent processes. For this kind of case, denial-of-service (i.e. an externally induced program crash) is likely to be the limit of the damage, so the threat isn't severe. Still worth avoiding the risk, though. Really tricky cases can lead to all sorts of fun and games, like manipulating programs that were granted elevated privileges into executing malicious code that was put in place using only user privileges (combining sudo and its ilk with python without passing -E and -s is an unfortunately-less-than-tricky way sysadmins can shoot themselves in the foot on that front). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Not-a-Number (was PyObject_RichCompareBool identity shortcut)
Terry Reedy writes: Python treats it as if it were a number: As I said, so did the committee, and that was its mistake that we are more or less stuck with. The committee didn't really have a choice. You could ask that they call NaNs something else, but some bit pattern is going to appear in the result register after each computation, and further operations may (try to) use that bit pattern. Seems reasonable to me to apply duck- typing and call those patterns numbers for the purpose of IEEE 754, and to define them in such a way that operating on them produces a non-NaN only when *all* numbers (including infinity) produce the same non-NaN. The alternative is to raise an exception whenever a NaN would be generated (but something is still going to appear in the register; I don't know any number that should be put there, do you?) That is excessively punishing to Python users and programmers, though, since Python handles exceptions by terminating the computation. (Kahan points out that signaling NaNs are essentially never used for this reason.) Other aspects of NaN behavior may be a mistake. But it's not clear to me, even after all the discussion in this thread. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] the role of assert in the standard library ?
On Fri, Apr 29, 2011 at 12:31 AM, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Apr 28, 2011, at 3:07 PM, Guido van Rossum wrote: On Thu, Apr 28, 2011 at 2:53 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Apr 28, 2011, at 1:27 PM, Holger Krekel wrote: On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossum gu...@python.org wrote: On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.ta...@gmail.com wrote: In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :) I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always. FWIW this is only true for the unittest module/pkg policy for writing and organising tests. There are other popular test frameworks like nose and pytest which promote using plain asserts within writing unit tests and also allow to write tests in functions. And judging from my tutorials and others places many people appreciate the ease of using asserts as compared to learning tons of new methods. YMMV. I've also observed that people appreciate using asserts with nose.py and py.test. They must not appreciate -O. :-) It might be nice if there were a pragma or module variable to selectively enable asserts for a given test module so that -O would turn-off asserts in the production code but leave them on in a test_suite. A way to tell Python if you are going to compile this module/path, don't turn off asserts, no matter what would be great. Then nose/py.test and whichever tools/apps could fine-tune the handling of asserts. (This would probably be better than marking things inline for those use cases). Then testing with -O would work nicely as well which would be appreciated :) best, holger Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes issue10761: tarfile.extractall failure when symlinked files are
On Fri, Apr 29, 2011 at 09:52, Nick Coghlan ncogh...@gmail.com wrote: On Fri, Apr 29, 2011 at 4:26 PM, Eli Bendersky eli...@gmail.com wrote: On Thu, Apr 28, 2011 at 04:20:06PM +0200, Éric Araujo wrote: The kind of race condition which can happen here is if an attacker creates targetpath between os.path.exists and os.unlink. Whether it is an exploitable flaw would need a detailed analysis, of course. Just out of curiosity, could you please elaborate on the potential threat of this? If the exists condition is true, targetpath already exists, so what use there is in overwriting it? If the condition is false, unlink isn't executed, so no harm either. What am I missing? That's the detailed analysis part. What happens if other code deletes the path, and the unlink() call subsequently fails despite the successful exists() check? Hence why exception checking (as Nadeem posted) is typically the only right way to do things that access an external environment that supports multiple concurrent processes. I completely understand this other code/thread deletes the path between exists() and unlink() case - it indeed is a race condition waiting to happen. What I didn't understand was Antoine's example of attacker creates targetpath between os.path.exists and os.unlink, and was asking for a more detailed example, since I'm not really experienced with security-oriented thinking. Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] the role of assert in the standard library ?
On Fri, Apr 29, 2011 at 9:55 AM, Holger Krekel holger.kre...@gmail.com wrote: On Fri, Apr 29, 2011 at 12:31 AM, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Apr 28, 2011, at 3:07 PM, Guido van Rossum wrote: On Thu, Apr 28, 2011 at 2:53 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Apr 28, 2011, at 1:27 PM, Holger Krekel wrote: On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossum gu...@python.org wrote: On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.ta...@gmail.com wrote: In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :) I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always. FWIW this is only true for the unittest module/pkg policy for writing and organising tests. There are other popular test frameworks like nose and pytest which promote using plain asserts within writing unit tests and also allow to write tests in functions. And judging from my tutorials and others places many people appreciate the ease of using asserts as compared to learning tons of new methods. YMMV. I've also observed that people appreciate using asserts with nose.py and py.test. They must not appreciate -O. :-) It might be nice if there were a pragma or module variable to selectively enable asserts for a given test module so that -O would turn-off asserts in the production code but leave them on in a test_suite. A way to tell Python if you are going to compile this module/path, don't turn off asserts, no matter what would be great. Then nose/py.test and whichever tools/apps could fine-tune the handling of asserts. (This would probably be better than marking things inline for those use cases). Then testing with -O would work nicely as well which would be appreciated :) best, holger Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com Any reason why -O behavior regarding removing asserts should not be changed? Or should I go to python-ideas? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal for a common benchmark suite
Maciej Fijalkowski wrote: On Thu, Apr 28, 2011 at 11:10 PM, Stefan Behnel stefan...@behnel.de wrote: M.-A. Lemburg, 28.04.2011 22:23: Stefan Behnel wrote: DasIch, 28.04.2011 20:55: the CPython benchmarks have an extensive set of microbenchmarks in the pybench package Try not to care too much about pybench. There is some value in it, but some of its microbenchmarks are also tied to CPython's interpreter behaviour. For example, the benchmarks for literals can easily be considered dead code by other Python implementations so that they may end up optimising the benchmarked code away completely, or at least partially. That makes a comparison of the results somewhat pointless. The point of the micro benchmarks in pybench is to be able to compare them one-by-one, not by looking at the sum of the tests. If one implementation optimizes away some parts, then the comparison will show this fact very clearly - and that's the whole point. Taking the sum of the micro benchmarks only has some meaning as very rough indicator of improvement. That's why I wrote pybench: to get a better, more details picture of what's happening, rather than trying to find some way of measuring average use. This average is very different depending on where you look: for some applications method calls may be very important, for others, arithmetic operations, and yet others may have more need for fast attribute lookup. I wasn't talking about averages or sums, and I also wasn't trying to put down pybench in general. As it stands, it makes sense as a benchmark for CPython. However, I'm arguing that a substantial part of it does not make sense as a benchmark for PyPy and others. With Cython, I couldn't get some of the literal arithmetic benchmarks to run at all. The runner script simply bails out with an error when the benchmarks accidentally run faster than the initial empty loop. I imagine that PyPy would eventually even drop the loop itself, thus leaving nothing to compare. Does that tell us that PyPy is faster than Cython for arithmetic? I don't think it does. When I see that a benchmark shows that one implementation runs in 100% less time than another, I simply go *shrug* and look for a better benchmark to compare the two. I second here what Stefan says. This sort of benchmarks might be useful for CPython, but they're not particularly useful for PyPy or for comparisons (or any other implementation which tries harder to optimize stuff away). For example a method call in PyPy would be inlined and completely removed if method is empty, which does not measure method call overhead at all. That's why we settled on medium-to-large examples where it's more of an average of possible scenarios than just one. If CPython were to start incorporating any specialising optimisations, pybench wouldn't be much use for CPython. The Unladen Swallow folks didn't like pybench as a benchmark. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes issue10761: tarfile.extractall failure when symlinked files are
On Fri, Apr 29, 2011 at 10:02 AM, Eli Bendersky eli...@gmail.com wrote: I completely understand this other code/thread deletes the path between exists() and unlink() case - it indeed is a race condition waiting to happen. What I didn't understand was Antoine's example of attacker creates targetpath between os.path.exists and os.unlink, and was asking for a more detailed example, since I'm not really experienced with security-oriented thinking. If targetpath is created after the os.path.exists() check, then os.unlink() will not be called, so os.symlink() will raise an exception when it sees that targetpath already exists. On Thu, Apr 28, 2011 at 5:44 PM, Antoine Pitrou solip...@pitrou.net wrote: On Thu, 28 Apr 2011 17:40:05 +0200 Nadeem Vawda nadeem.va...@gmail.com wrote: The deletion case could be handled like this: if tarinfo.issym(): +try: +os.unlink(targetpath) +except OSError as e: +if e.errno != errno.ENOENT: +raise os.symlink(tarinfo.linkname, targetpath) Someone can still create targetpath between the calls to unlink() and symlink() though. Like I said, that code only handles the case of targetpath being deleted. I can't think of a similarly easy fix for the creation case. You could solve that particular form of the problem with something like: if tarinfo.issym(): while True: try: os.symlink(tarinfo.linkname, targetpath) except OSError as e: if e.errno != errno.EEXIST: raise else: break try: os.unlink(targetpath) except OSError as e: if e.errno != errno.ENOENT: raise ... but that would open up another DOS vulnerability - if an attacker manages to keep re-creating targetpath in the window between unlink() and symlink(), the loop would never terminate. Also, the code is rather ugly ;-) Cheers, Nadeem ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal for a common benchmark suite
Mark Shannon wrote: Maciej Fijalkowski wrote: On Thu, Apr 28, 2011 at 11:10 PM, Stefan Behnel stefan...@behnel.de wrote: M.-A. Lemburg, 28.04.2011 22:23: Stefan Behnel wrote: DasIch, 28.04.2011 20:55: the CPython benchmarks have an extensive set of microbenchmarks in the pybench package Try not to care too much about pybench. There is some value in it, but some of its microbenchmarks are also tied to CPython's interpreter behaviour. For example, the benchmarks for literals can easily be considered dead code by other Python implementations so that they may end up optimising the benchmarked code away completely, or at least partially. That makes a comparison of the results somewhat pointless. The point of the micro benchmarks in pybench is to be able to compare them one-by-one, not by looking at the sum of the tests. If one implementation optimizes away some parts, then the comparison will show this fact very clearly - and that's the whole point. Taking the sum of the micro benchmarks only has some meaning as very rough indicator of improvement. That's why I wrote pybench: to get a better, more details picture of what's happening, rather than trying to find some way of measuring average use. This average is very different depending on where you look: for some applications method calls may be very important, for others, arithmetic operations, and yet others may have more need for fast attribute lookup. I wasn't talking about averages or sums, and I also wasn't trying to put down pybench in general. As it stands, it makes sense as a benchmark for CPython. However, I'm arguing that a substantial part of it does not make sense as a benchmark for PyPy and others. With Cython, I couldn't get some of the literal arithmetic benchmarks to run at all. The runner script simply bails out with an error when the benchmarks accidentally run faster than the initial empty loop. I imagine that PyPy would eventually even drop the loop itself, thus leaving nothing to compare. Does that tell us that PyPy is faster than Cython for arithmetic? I don't think it does. When I see that a benchmark shows that one implementation runs in 100% less time than another, I simply go *shrug* and look for a better benchmark to compare the two. I second here what Stefan says. This sort of benchmarks might be useful for CPython, but they're not particularly useful for PyPy or for comparisons (or any other implementation which tries harder to optimize stuff away). For example a method call in PyPy would be inlined and completely removed if method is empty, which does not measure method call overhead at all. That's why we settled on medium-to-large examples where it's more of an average of possible scenarios than just one. If CPython were to start incorporating any specialising optimisations, pybench wouldn't be much use for CPython. The Unladen Swallow folks didn't like pybench as a benchmark. This is all true, but I think there's a general misunderstanding of what pybench is. I wrote pybench in 1997 when I was working on optimizing the Python 1.5 implementation for use in an web application server. At the time, we had pystone and that was a really poor benchmark for determining of whether certain optimizations in the Python VM and compiler made sense or not. pybench was then improved and extended over the course of several years and then added to Python 2.5 in 2006. The benchmark is written as framework for micro benchmarks based on the assumption of a non-optimizing (byte code) compiler. As such it may or may not work with an optimizing compiler. The calibration part would likely have to be disabled for an optimizing compiler (run with -C 0) and a new set of benchmark tests would have to be added; one which tests the Python implementation at a higher level than the existing tests. That last part is something people tend to forget: pybench is not a monolithic application with a predefined and fixed set of tests. It's a framework that can be extended as needed. All you have to do is add a new module with test classes and import it in Setup.py. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 29 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 52 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___
Re: [Python-Dev] Proposal for a common benchmark suite
On 29/04/2011 11:04, M.-A. Lemburg wrote: Mark Shannon wrote: Maciej Fijalkowski wrote: On Thu, Apr 28, 2011 at 11:10 PM, Stefan Behnelstefan...@behnel.de wrote: M.-A. Lemburg, 28.04.2011 22:23: Stefan Behnel wrote: DasIch, 28.04.2011 20:55: the CPython benchmarks have an extensive set of microbenchmarks in the pybench package Try not to care too much about pybench. There is some value in it, but some of its microbenchmarks are also tied to CPython's interpreter behaviour. For example, the benchmarks for literals can easily be considered dead code by other Python implementations so that they may end up optimising the benchmarked code away completely, or at least partially. That makes a comparison of the results somewhat pointless. The point of the micro benchmarks in pybench is to be able to compare them one-by-one, not by looking at the sum of the tests. If one implementation optimizes away some parts, then the comparison will show this fact very clearly - and that's the whole point. Taking the sum of the micro benchmarks only has some meaning as very rough indicator of improvement. That's why I wrote pybench: to get a better, more details picture of what's happening, rather than trying to find some way of measuring average use. This average is very different depending on where you look: for some applications method calls may be very important, for others, arithmetic operations, and yet others may have more need for fast attribute lookup. I wasn't talking about averages or sums, and I also wasn't trying to put down pybench in general. As it stands, it makes sense as a benchmark for CPython. However, I'm arguing that a substantial part of it does not make sense as a benchmark for PyPy and others. With Cython, I couldn't get some of the literal arithmetic benchmarks to run at all. The runner script simply bails out with an error when the benchmarks accidentally run faster than the initial empty loop. I imagine that PyPy would eventually even drop the loop itself, thus leaving nothing to compare. Does that tell us that PyPy is faster than Cython for arithmetic? I don't think it does. When I see that a benchmark shows that one implementation runs in 100% less time than another, I simply go *shrug* and look for a better benchmark to compare the two. I second here what Stefan says. This sort of benchmarks might be useful for CPython, but they're not particularly useful for PyPy or for comparisons (or any other implementation which tries harder to optimize stuff away). For example a method call in PyPy would be inlined and completely removed if method is empty, which does not measure method call overhead at all. That's why we settled on medium-to-large examples where it's more of an average of possible scenarios than just one. If CPython were to start incorporating any specialising optimisations, pybench wouldn't be much use for CPython. The Unladen Swallow folks didn't like pybench as a benchmark. This is all true, but I think there's a general misunderstanding of what pybench is. pybench proved useful for IronPython. It certainly highlighted some performance problems with some of the basic operations it measures. All the best, Michael Foord I wrote pybench in 1997 when I was working on optimizing the Python 1.5 implementation for use in an web application server. At the time, we had pystone and that was a really poor benchmark for determining of whether certain optimizations in the Python VM and compiler made sense or not. pybench was then improved and extended over the course of several years and then added to Python 2.5 in 2006. The benchmark is written as framework for micro benchmarks based on the assumption of a non-optimizing (byte code) compiler. As such it may or may not work with an optimizing compiler. The calibration part would likely have to be disabled for an optimizing compiler (run with -C 0) and a new set of benchmark tests would have to be added; one which tests the Python implementation at a higher level than the existing tests. That last part is something people tend to forget: pybench is not a monolithic application with a predefined and fixed set of tests. It's a framework that can be extended as needed. All you have to do is add a new module with test classes and import it in Setup.py. -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] the role of assert in the standard library ?
On 28 April 2011 23:07, Guido van Rossum gu...@python.org wrote: On Thu, Apr 28, 2011 at 2:53 PM, Raymond Hettinger raymond.hettin...@gmail.com wrote: On Apr 28, 2011, at 1:27 PM, Holger Krekel wrote: On Thu, Apr 28, 2011 at 6:59 PM, Guido van Rossum gu...@python.org wrote: On Thu, Apr 28, 2011 at 12:54 AM, Tarek Ziadé ziade.ta...@gmail.com wrote: In my opinion assert should be avoided completely anywhere else than in the tests. If this is a wrong statement, please let me know why :) I would turn that around. The assert statement should not be used in unit tests; unit tests should use self.assertXyzzy() always. FWIW this is only true for the unittest module/pkg policy for writing and organising tests. There are other popular test frameworks like nose and pytest which promote using plain asserts within writing unit tests and also allow to write tests in functions. And judging from my tutorials and others places many people appreciate the ease of using asserts as compared to learning tons of new methods. YMMV. I've also observed that people appreciate using asserts with nose.py and py.test. They must not appreciate -O. :-) Personaly I'd love to get rid of all of -O's meanings apart from setting __debug__ to False. Then you can write a strip tool which could strip all docstrings, just unused docstrings (an improvement over -O), and any dead code resulting from setting __debug__ to either True or False. The last thing to do is place assert statements inside a if __debug__ block. That way you could use the strip tool on the modules under test but not on the test modules. Regards Floris PS: I actually wrote some prototype code for such a strip tool last year but never finished it off, so I'm pretty sure most of this is possible. -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Socket servers in the test suite
On 27.04.2011 23:23, Vinay Sajip wrote: I've been recently trying to improve the test coverage for the logging package, and have got to a not unreasonable point: logging/__init__.py 99% (96%) logging/config.py 89% (85%) logging/handlers.py 60% (54%) where the figures in parentheses include branch coverage measurements. BTW, didn't we agree not to put pragma comments into the stdlib code? Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Socket servers in the test suite
On Fri, Apr 29, 2011 at 9:44 PM, Georg Brandl g.bra...@gmx.net wrote: On 27.04.2011 23:23, Vinay Sajip wrote: I've been recently trying to improve the test coverage for the logging package, and have got to a not unreasonable point: logging/__init__.py 99% (96%) logging/config.py 89% (85%) logging/handlers.py 60% (54%) where the figures in parentheses include branch coverage measurements. BTW, didn't we agree not to put pragma comments into the stdlib code? I think some folks objected, but since they're essential to keeping track of progress in code coverage improvement efforts, there wasn't a consensus to leave them out. The pragmas themselves are easy enough to grep for, so it isn't like they don't leave a record of which lines may not be getting tested. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal for a common benchmark suite
Given those facts I think including pybench is a mistake. It does not allow for a fair or meaningful comparison between implementations which is one of the things the suite is supposed to be used for in the future. This easily leads to misinterpretation of the results from this particular benchmark and it negatively affects the performance data as a whole. The same applies to several Unladen Swallow microbenchmarks such as bm_call_method_*, bm_call_simple and bm_unpack_sequence. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal for a common benchmark suite
DasIch wrote: Given those facts I think including pybench is a mistake. It does not allow for a fair or meaningful comparison between implementations which is one of the things the suite is supposed to be used for in the future. This easily leads to misinterpretation of the results from this particular benchmark and it negatively affects the performance data as a whole. The same applies to several Unladen Swallow microbenchmarks such as bm_call_method_*, bm_call_simple and bm_unpack_sequence. I don't think we should exclude any implementation specific benchmarks from a common suite. They will not necessarily allow for comparisons between implementations, but will provide important information about the progress made in optimizing a particular implementation. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 29 2011) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 52 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal for a common benchmark suite
On Fri, 29 Apr 2011 14:29:46 +0200 DasIch dasdas...@googlemail.com wrote: Given those facts I think including pybench is a mistake. It does not allow for a fair or meaningful comparison between implementations which is one of the things the suite is supposed to be used for in the future. Including is quite vague. pybench is included in the suite of benchmarks at hg.python.org, but that doesn't mean it is given any particular importance: you can select whichever benchmarks you want to run when perf.py is executed (there are even several predefined benchmark groups, none of which pybench is a member IIRC). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Not-a-Number
Steven D'Aprano st...@pearwood.info writes: I'm sorry for my lack of clarity. I'm referring to functions which potentially produce NANs, not the exceptions themselves. A calculation which might have produced a (quiet) NAN as the result instead raises an exception (which I'm treating as equivalent to a signal). Yes, it produces a Python exception, which is not a Python NaN. If you want to talk about “signalling NaNs”, you'll have to distinguish that (every time!) so you're not misunderstood as referring to a Python NaN object. -- \ “It's my belief we developed language because of our deep inner | `\ need to complain.” —Jane Wagner, via Lily Tomlin | _o__) | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] What if replacing items in a dictionary returns the new dictionary?
It would be convenient if replacing items in a dictionary returns the
new dictionary, in a manner analogous to str.replace(). What do you
think?
::
# Current behavior
x = {'key1': 1}
x.update(key1=3) == None
x == {'key1': 3} # Original variable has changed
# Possible behavior
x = {'key1': 1}
x.replace(key1=3) == {'key1': 3}
x == {'key1': 1} # Original variable is unchanged
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What if replacing items in a dictionary returns the new dictionary?
Roy Hyunjin Han wrote:
It would be convenient if replacing items in a dictionary returns the
new dictionary, in a manner analogous to str.replace(). What do you
think?
::
# Current behavior
x = {'key1': 1}
x.update(key1=3) == None
x == {'key1': 3} # Original variable has changed
# Possible behavior
x = {'key1': 1}
x.replace(key1=3) == {'key1': 3}
x == {'key1': 1} # Original variable is unchanged
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/marks%40dcs.gla.ac.uk
Could you please post this to python-ideas, rather than python-dev
Python-dev is about aspects of the implementation,
not significant language changes.
Mark.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What if replacing items in a dictionary returns the new dictionary?
On Fri, 29 Apr 2011 10:27:46 -0400, Roy Hyunjin Han starsareblueandfara...@gmail.com wrote: It would be convenient if replacing items in a dictionary returns the new dictionary, in a manner analogous to str.replace(). What do you think? This belongs on python-ideas, but the short answer is no. The general language design principle (as I understand it) is that mutable object do not return themselves upon mutation, while immutable objects do return the new object. -- R. David Murray http://www.bitdance.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What if replacing items in a dictionary returns the new dictionary?
Hi! Seems like a question for python-ideas mailing list, not for python-dev.
On Fri, Apr 29, 2011 at 10:27:46AM -0400, Roy Hyunjin Han wrote:
It would be convenient if replacing items in a dictionary returns the
new dictionary, in a manner analogous to str.replace(). What do you
think?
::
# Current behavior
x = {'key1': 1}
x.update(key1=3) == None
x == {'key1': 3} # Original variable has changed
# Possible behavior
x = {'key1': 1}
x.replace(key1=3) == {'key1': 3}
x == {'key1': 1} # Original variable is unchanged
You can implement this in your own subclass of dict, no?
Oleg.
--
Oleg Broytmanhttp://phdru.name/p...@phdru.name
Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What if replacing items in a dictionary returns the new dictionary?
2011/4/29 R. David Murray rdmur...@bitdance.com: 2011/4/29 Roy Hyunjin Han starsareblueandfara...@gmail.com: It would be convenient if replacing items in a dictionary returns the new dictionary, in a manner analogous to str.replace() This belongs on python-ideas, but the short answer is no. The general language design principle (as I understand it) is that mutable object do not return themselves upon mutation, while immutable objects do return the new object. Thanks for the responses. Sorry for the mispost, I'll post things like this on python-ideas from now on. RHH ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What if replacing items in a dictionary returns the new dictionary?
You can implement this in your own subclass of dict, no? Yes, I just thought it would be convenient to have in the language itself, but the responses to my post seem to indicate that [not returning the updated object] is an intended language feature for mutable types like dict or list. class ReplaceableDict(dict): def replace(self, **kw): 'Works for replacing string-based keys' return dict(self.items() + kw.items()) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Not-a-Number
On 4/29/11 1:35 AM, Nick Coghlan wrote: On Fri, Apr 29, 2011 at 3:28 PM, Steven D'Apranost...@pearwood.info wrote: Robert Kern wrote: Actually, Python treats all NaNs as quiet NaNs and never signalling NaNs. Sorry, did I get that backwards? I thought it was signalling NANs that cause a signal (in Python terms, an exception)? If I do x = 0.0/0 I get an exception instead of a NAN. Hence a signalling NAN. Aside from the divide-by-zero case, we treat NaNs as quiet NaNs. And in fact, 0.0/0.0 is covered by the more general rule that x/0.0 raises ZeroDivisionError, not a rule that converts IEEE-754 INVALID exceptions into Python exceptions. Other operations that produce a NaN and issue an IEEE-754 INVALID signal do not raise a Python exception. But that's not the difference between signalling NaNs and quiet NaNs. A signalling NaN is one that when it is used as an *input* to an operation, it issues an INVALID signal, not whether a signal is issued when it is the *output* of an operation. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2011-04-22 - 2011-04-29)
Python tracker at http://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open2760 ( +8)
closed 20976 (+39)
total 23736 (+47)
Open issues with patches: 1201
Issues opened (33)
==
#3561: Windows installer should add Python and Scripts directories to
http://bugs.python.org/issue3561 reopened by ncoghlan
#10912: PyObject_RichCompare differs in behaviour from PyObject_RichCo
http://bugs.python.org/issue10912 reopened by ncoghlan
#10914: Python sub-interpreter test
http://bugs.python.org/issue10914 reopened by pitrou
#11895: pybench prep_times calculation error
http://bugs.python.org/issue11895 reopened by jcea
#11908: Weird `slice.stop or sys.maxint`
http://bugs.python.org/issue11908 opened by cool-RR
#11909: Doctest sees directives in strings when it should only see the
http://bugs.python.org/issue11909 opened by Devin Jeanpierre
#11910: test_heapq C tests are not skipped when _heapq is missing
http://bugs.python.org/issue11910 opened by ezio.melotti
#11912: Python shouldn't use the mprotect() system call
http://bugs.python.org/issue11912 opened by breun
#11913: sdist should allow for README.rst
http://bugs.python.org/issue11913 opened by ingy
#11914: pydoc modules/help('modules') crash in dirs with unreadable su
http://bugs.python.org/issue11914 opened by okopnik
#11916: A few errnos from OSX
http://bugs.python.org/issue11916 opened by pcarrier
#11920: ctypes: Strange bitfield structure sizing issue
http://bugs.python.org/issue11920 opened by Steve.Thompson
#11921: distutils2 should be able to compile an Extension based on the
http://bugs.python.org/issue11921 opened by dholth
#11924: Pickle and copyreg modules don't document the interface
http://bugs.python.org/issue11924 opened by jcea
#11925: test_ttk_guionly.test_traversal() failed on x86 Windows7 3.x
http://bugs.python.org/issue11925 opened by haypo
#11927: SMTP_SSL doesn't use port 465 by default
http://bugs.python.org/issue11927 opened by pitrou
#11928: fail on filename with space at the end
http://bugs.python.org/issue11928 opened by techtonik
#11930: Remove time.accept2dyear
http://bugs.python.org/issue11930 opened by belopolsky
#11931: Regular expression documentation patch
http://bugs.python.org/issue11931 opened by Retro
#11933: newer() function in dep_util.py mixes up new vs. old files due
http://bugs.python.org/issue11933 opened by jsjgruber
#11934: build with --prefix=/dev/null and zlib enabled in Modules/Setu
http://bugs.python.org/issue11934 opened by ysj.ray
#11935: MMDF/MBOX mailbox need utime
http://bugs.python.org/issue11935 opened by sdaoden
#11937: Interix support
http://bugs.python.org/issue11937 opened by mduft
#11939: Implement stat.st_dev and os.path.samefile on windows
http://bugs.python.org/issue11939 opened by amaury.forgeotdarc
#11941: Support st_atim, st_mtim and st_ctim attributes in os.stat_res
http://bugs.python.org/issue11941 opened by Arfrever
#11943: Add TLS-SRP (RFC 5054) support to ssl, _ssl, http, and urllib
http://bugs.python.org/issue11943 opened by sqs
#11944: Function call with * and generator hide exception raised by ge
http://bugs.python.org/issue11944 opened by falsetru
#11945: Adopt and document consistent semantics for handling NaN value
http://bugs.python.org/issue11945 opened by ncoghlan
#11948: Tutorial/Modules - small fix to better clarify the modules sea
http://bugs.python.org/issue11948 opened by sandro.tosi
#11949: Make float('nan') unorderable
http://bugs.python.org/issue11949 opened by belopolsky
#11950: logger use dict for loggers instead of WeakValueDictionary
http://bugs.python.org/issue11950 opened by mmarkk
#11953: Missing WSA* error codes
http://bugs.python.org/issue11953 opened by pitrou
#11954: 3.3 - 'make test' fails
http://bugs.python.org/issue11954 opened by Jason.Vas.Dias
Most recent 15 issues with no replies (15)
==
#11950: logger use dict for loggers instead of WeakValueDictionary
http://bugs.python.org/issue11950
#11935: MMDF/MBOX mailbox need utime
http://bugs.python.org/issue11935
#11934: build with --prefix=/dev/null and zlib enabled in Modules/Setu
http://bugs.python.org/issue11934
#11924: Pickle and copyreg modules don't document the interface
http://bugs.python.org/issue11924
#11916: A few errnos from OSX
http://bugs.python.org/issue11916
#11909: Doctest sees directives in strings when it should only see the
http://bugs.python.org/issue11909
#11898: Sending binary data with a POST request in httplib can cause U
http://bugs.python.org/issue11898
#11894: test_multiprocessing failure on AMD64 OpenIndiana 3.x: KeyEr
http://bugs.python.org/issue11894
#11893: Obsolete SSLFakeFile in smtplib?
http://bugs.python.org/issue11893
#11887: unittest fails on comparing str with bytes if python has the -
Re: [Python-Dev] Socket servers in the test suite
[Georg] BTW, didn't we agree not to put pragma comments into the stdlib code? I'd be grateful for a link to the prior discussion - it must have passed me by originally, and I searched python-dev on gmane but couldn't find any threads about this. [Nick] I think some folks objected, but since they're essential to keeping track of progress in code coverage improvement efforts, there wasn't a consensus to leave them out. The pragmas themselves are easy enough to grep for, so it isn't like they don't leave a record of which lines may not be getting tested. Yes - in theory the pragmas can give a false idea about coverage, but in practice they help increase the signal-to-noise ratio. As maintainer of a module, one'd only be kidding oneself by adding pragmas willy-nilly. The coverage reports are up-front about telling you how many lines were excluded, both in the summary HTML pages and the drill-downs HTML pages for individual modules. BTW, is there a public place somewhere showing stdlib coverage statistics? I looked on the buildbot pages as the likeliest home for them, but perhaps I missed them. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Not-a-Number
On Fri, Apr 29, 2011 at 11:31 AM, Robert Kern robert.k...@gmail.com wrote: .. And in fact, 0.0/0.0 is covered by the more general rule that x/0.0 raises ZeroDivisionError, not a rule that converts IEEE-754 INVALID exceptions into Python exceptions. It is unfortunate that official text of IEEE-754 is not freely available and as a result a lot of discussion in this thread is based on imperfect information. I find Kahan's Lecture Notes on the Status of IEEE Standard 754 for Binary Floating-Point Arithmetic [1] a reasonable reference in the absence of the official text. According to Kahan's notes, INVALID operation is defined as follows: Exception: INVALID operation. Signaled by the raising of the INVALID flag whenever an operation's operands lie outside its domain, this exception's default, delivered only because any other real or infinite value would most likely cause worse confusion, is NaN , which means “ Not a Number.” IEEE 754 specifies that seven invalid arithmetic operations shall deliver a NaN unless they are trapped: real √(Negative) , 0*∞ , 0.0/0.0 , ∞/∞, REMAINDER(Anything, 0.0) , REMAINDER( ∞, Anything) , ∞ - ∞ when signs agree (but ∞ + ∞ = ∞ when signs agree). Conversion from floating-point to other formats can be INVALID too, if their limits are violated, even if no NaN can be delivered. In contrast, Kahan describes DIVIDE by ZERO exception as a misnomer perpetrated for historical reasons. A better name for this exception is 'Infinite result computed Exactly from Finite operands.' Other operations that produce a NaN and issue an IEEE-754 INVALID signal do not raise a Python exception. Some do: math.sqrt(-1) Traceback (most recent call last): File stdin, line 1, in module ValueError: math domain error I think the only exception are the operations involving infinity. The likely rationale is that since infinity is not produced by python arithmetics, those who use inf are likely to expect inf*0 etc. to produce nan. The following seems to be an oversight: 1e300 * 1e300 inf compared to 1e300 ** 2 Traceback (most recent call last): File stdin, line 1, in module OverflowError: (34, 'Result too large') [1] http://www.cs.berkeley.edu/~wkahan/ieee754status/ieee754.ps ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Not-a-Number (was PyObject_RichCompareBool identity shortcut)
On Fri, Apr 29, 2011 at 12:10 AM, Stephen J. Turnbull step...@xemacs.org wrote: Other aspects of NaN behavior may be a mistake. But it's not clear to me, even after all the discussion in this thread. ISTM that the current behavior of NaN (never mind the identity issue) helps numeric experts write better code. For naive users, however, it causes puzzlement if they ever run into it. Decimal, for that reason, has a context that lets one specify different behaviors when a NaN is produced. Would it make sense to add a float context that also lets one specify what should happen? That could include returning Inf for 1.0/0.0 (for experts), or raising exceptions when NaNs are produced (for the numerically naive like myself). I could see a downside too, e.g. the correctness of code that passingly uses floats might be affected by the context settings. There's also the question of whether the float context should affect int operations; floats vs. ints is another can of worms since (in Python 3) we attempt to tie them together through 1/2 == 0.5, but ints have a much larger range than floats. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Not-a-Number (was PyObject_RichCompareBool identity shortcut)
On Fri, Apr 29, 2011 at 1:11 PM, Guido van Rossum gu...@python.org wrote: … Would it make sense to add a float context that also lets one specify what should happen? That could include returning Inf for 1.0/0.0 (for experts), or raising exceptions when NaNs are produced (for the numerically naive like myself). ISTM, this is approaching py4k territory. Adding contexts will not solve backward compatibility problem unless you introduce a quirks contexts that would preserve current warts and make it default. For what it's worth, I think the next major version of Python should use decimal as its main floating point type an leave binary floats to numerical experts. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Socket servers in the test suite
On 4/29/2011 3:11 PM, Terry Reedy wrote: On 4/29/2011 12:09 PM, Vinay Sajip wrote: BTW, is there a public place somewhere showing stdlib coverage statistics? I looked on the buildbot pages as the likeliest home for them, but perhaps I missed them. http://docs.python.org/devguide/coverage.html has a link to http://coverage.livinglogic.de/ which, however, currently has nothing for *.py. Perhaps a glitch/bug, as there used to be such. Anyone who knows the page owner might ask about this. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fwd: viewVC shows traceback on non utf-8 module markup
I know that the svn repo is now for legacy purposes only, but I doubt it is intended that the online source browser should raise exceptions. (See report below.) All the best, Michael Original Message Subject:viewVC shows traceback on non utf-8 module markup Date: Thu, 28 Apr 2011 17:47:12 +0900 From: Ikkei Shimomura ikkei.shimom...@gmail.com To: webmas...@python.org Hi, here is a report, I found some module markup shows traceback. like this: http://svn.python.org/view/python/trunk/Lib/heapq.py?view=markup UnicodeDecodeError: 'utf8' codec can't decode byte 0xe7 in position 1428: invalid continuation byte I do not know about latin-1 coding, this is just note what I found at that position It's \xe7 ... by Fran\xe7o ... and as I read the traceback, viewvc and pygment assumes utf-8 encoding, its hard-coded. Other modules which use non- utf8 encoding (searched by grep -r coding\: *.py) inspect, pydoc were ok. tarfile, shlex were not. Excute me for my writting broken English. Ikkei Shimomura ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Socket servers in the test suite
Terry Reedy tjreedy at udel.edu writes: http://coverage.livinglogic.de/ which, however, currently has nothing for *.py. Perhaps a glitch/bug, as there used to be such. Anyone who knows the page owner might ask about this. Thanks for the pointer, nevertheless, Terry. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python and super
Ricardo Kirkner wrote: I'll give you the example I came upon: I have a TestCase class, which inherits from both Django's TestCase and from some custom TestCases that act as mixin classes. So I have something like class MyTestCase(TestCase, Mixin1, Mixin2): ... now django's TestCase class inherits from unittest2.TestCase, which we found was not calling super. Even if this is a bug and should be fixed in unittest2, this is an example where I, as a consumer of django, shouldn't have to be worried about how django's TestCase class is implemented. I have to disagree -- anytime you are using somebody else's code you need to be aware of what it's supposed to do -- especially when playing with multiple inheritance. This response to the decorator I wrote for this situation may be helpful: Carl Banks wrote (on Python-List): The problem is that he was doing mixins wrong. Way wrong. Here is my advice on mixins: Mixins should almost always be listed first in the bases. (The only exception is to work around a technicality. Otherwise mixins go first.) If a mixin defines __init__, it should always accept self, *args and **kwargs (and no other arguments), and pass those on to super().__init__. Same deal with any other function that different sister classes might define in varied ways (such as __call__). A mixin should not accept arguments in __init__. Instead, it should burden the derived class to accept arguments on its behalf, and set attributes before calling super().__init__, which the mixin can access. If you insist on a mixin that accepts arguments in __init__, then it should should pop them off kwargs. Avoid using positional arguments, and never use named arguments. Always go through args and kwargs. If mixins follow these rules, they'll be reasonably safe to use on a variety of classes. (Maybe even safe enough to use in Django classes.) ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Not-a-Number
On Fri, Apr 29, 2011 at 11:35, Alexander Belopolsky alexander.belopol...@gmail.com wrote: On Fri, Apr 29, 2011 at 11:31 AM, Robert Kern robert.k...@gmail.com wrote: .. And in fact, 0.0/0.0 is covered by the more general rule that x/0.0 raises ZeroDivisionError, not a rule that converts IEEE-754 INVALID exceptions into Python exceptions. It is unfortunate that official text of IEEE-754 is not freely available and as a result a lot of discussion in this thread is based on imperfect information. I find Kahan's Lecture Notes on the Status of IEEE Standard 754 for Binary Floating-Point Arithmetic [1] a reasonable reference in the absence of the official text. According to Kahan's notes, INVALID operation is defined as follows: Exception: INVALID operation. Signaled by the raising of the INVALID flag whenever an operation's operands lie outside its domain, this exception's default, delivered only because any other real or infinite value would most likely cause worse confusion, is NaN , which means “ Not a Number.” IEEE 754 specifies that seven invalid arithmetic operations shall deliver a NaN unless they are trapped: real √(Negative) , 0*∞ , 0.0/0.0 , ∞/∞, REMAINDER(Anything, 0.0) , REMAINDER( ∞, Anything) , ∞ - ∞ when signs agree (but ∞ + ∞ = ∞ when signs agree). Conversion from floating-point to other formats can be INVALID too, if their limits are violated, even if no NaN can be delivered. In contrast, Kahan describes DIVIDE by ZERO exception as a misnomer perpetrated for historical reasons. A better name for this exception is 'Infinite result computed Exactly from Finite operands.' Nonetheless, the reason that *Python* raises a ZeroDivisionError is because it checks that the divisor is 0.0, not because 0.0/0.0 would issue an INVALID signal. I didn't mean that 0.0/0.0 is a Division by Zero error as defined in IEEE-754. This is another area where Python ignores the INVALID signal and does its own thing. Other operations that produce a NaN and issue an IEEE-754 INVALID signal do not raise a Python exception. Some do: math.sqrt(-1) Traceback (most recent call last): File stdin, line 1, in module ValueError: math domain error Right. Elsewhere I gave a more exhaustive list including this one. The other is int(nan), though that becomes a Python exception for a more fundamental reason (there is no integer value that can represent it) than that the IEEE-754 standard specifies that the operation should signal INVALID. Arithmetic operations on signalling NaNs don't raise an exception either. These are the minority *exceptions* to the majority of cases where operations on Python floats that would issue an INVALID signal do not raise Python exceptions. If you want to lump all of the inf-related cases together, that's fine, but arithmetic operations on signalling NaNs and comparisons with NaNs form two more groups of INVALID operations that do not raise Python exceptions. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Not-a-Number
Steven D'Aprano wrote: If I do x = 0.0/0 I get an exception instead of a NAN. But the exception you get is ZeroDivisionError, so I think Python is catching this before you get to the stage of producing a NaN. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
