[Python-Dev] Re: Prospective Peephole Transformation
Raymond Hettinger wrote: Based on some ideas from Skip, I had tried transforming the likes of x in (1,2,3) into x in frozenset([1,2,3]). When applicable, it substantially simplified the generated code and converted the O(n) lookup into an O(1) step. There were substantial savings even if the set contained only a single entry. savings in what? time or bytecode size? constructed micro-benchmarks, or examples from real-life code? do we have any statistics on real-life n values? /F ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: Prospective Peephole Transformation
Based on some ideas from Skip, I had tried transforming the likes of x in (1,2,3) into x in frozenset([1,2,3]) Fredrik savings in what? time or bytecode size? constructed Fredrik micro-benchmarks, or examples from real-life code? Fredrik do we have any statistics on real-life n values? My original suggestion wasn't based on performance issues. It was based on the notion of tuples-as-records and lists-as-arrays. Raymond had originally gone through the code and changed for x in [1,2,3]: to for x in (1,2,3): I suggested that since the standard library code is commonly used as an example of basic Python principles (that's probably not the right word), it should uphold that ideal tuple/list distinction. Raymond then translated for x in [1,2,3]: to for x in frozenset([1,2,3]): I'm unclear why the list in for x in [1,2,3] or if x not in [1,2,3] can't fairly easily be recognized as a constant and just be placed in the constants array. The bytecode would show n LOAD_CONST opcodes followed by BUILD_LIST then either a COMPARE_OP (in the test case) or GET_ITER+FOR_ITER (in the for loop case). I think the optimizer should be able to recognize both constructs fairly easily. I don't know if that would provide a performance increase or not. I was after separation of functionality between tuples and lists. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
RE: [Python-Dev] Re: Prospective Peephole Transformation
I'm unclear why the list in for x in [1,2,3] or if x not in [1,2,3] can't fairly easily be recognized as a constant and just be placed in the constants array. That part got done (at least for the if-statement). The question is whether the type transformation idea should be carried a step further so that a single step search operation replaces the linear search. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: Prospective Peephole Transformation
At 04:45 PM 2/18/05 +0100, Fredrik Lundh wrote: Phillip J. Eby wrote: Only contains expressions are translated: if x in [1,2,3] currently turns into: if x in (1,2,3) and I'm proposing that it go one step further: if x in Seachset([1,2,3]) ISTM that whenever I use a constant in-list like that, it's almost always with just a few (4) items, so it doesn't seem worth the extra effort (especially disrupting the marshal module) just to squeeze out those extra two comparisons and replace them with a hashing operation. it could be worth expanding them to if x == 1 or x == 2 or x == 3: though... C:\timeit -s a = 1 if a in (1, 2, 3): pass 1000 loops, best of 3: 0.11 usec per loop C:\timeit -s a = 1 if a == 1 or a == 2 or a == 3: pass 1000 loops, best of 3: 0.0691 usec per loop C:\timeit -s a = 2 if a == 1 or a == 2 or a == 3: pass 1000 loops, best of 3: 0.123 usec per loop C:\timeit -s a = 2 if a in (1, 2, 3): pass 1000 loops, best of 3: 0.143 usec per loop C:\timeit -s a = 3 if a == 1 or a == 2 or a == 3: pass 1000 loops, best of 3: 0.187 usec per loop C:\timeit -s a = 3 if a in (1, 2, 3): pass 100 loops, best of 3: 0.197 usec per loop C:\timeit -s a = 4 if a in (1, 2, 3): pass 100 loops, best of 3: 0.225 usec per loop C:\timeit -s a = 4 if a == 1 or a == 2 or a == 3: pass 1000 loops, best of 3: 0.161 usec per loop /F Were these timings done with the code that turns (1,2,3) into a constant? Also, I presume that these timings still include extra LOAD_FAST operations that could be replaced with DUP_TOP in the actual expansion, although I don't know how much difference that would make in practice, since saving the argument fetch might be offset by the need to swap and pop at the end. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Prospective Peephole Transformation
Raymond Hettinger wrote: Based on some ideas from Skip, I had tried transforming the likes of x in (1,2,3) into x in frozenset([1,2,3]). When applicable, it substantially simplified the generated code and converted the O(n) lookup into an O(1) step. There were substantial savings even if the set contained only a single entry. When disassembled, the bytecode is not only much shorter, it is also much more readable (corresponding almost directly to the original source). The problem with the transformation was that it didn't handle the case where x was non-hashable and it would raise a TypeError instead of returning False as it should. That situation arose once in the email module's test suite. To get it to work, I would have to introduce a frozenset subtype: class Searchset(frozenset): def __contains__(self, element): try: return frozenset.__contains__(self, element) except TypeError: return False Then, the transformation would be x in Searchset([1, 2, 3]). Since the new Searchset object goes in the constant table, marshal would have to be taught how to save and restore the object. This is a more complicated than the original frozenset version of the patch, so I would like to get feedback on whether you guys think it is worth it. Wouldn't it help a lot more if the compiler would detect that (1,2,3) is immutable and convert it into a constant at compile time ?! The next step would then be to have Python roll out these loops (in -O mode). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 18 2005) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fixing _PyEval_SliceIndex so that integer-like objects can be used
Hello again, There is a great discussion going on the numpy list regarding a proposed PEP for multidimensional arrays that is in the works. During this discussion as resurfaced regarding slicing with objects that are not IntegerType objects but that have a tp_as_number-nb_int method to convert to an int. Would it be possible to change _PyEval_SliceIndex in ceval.c so that rather than throwing an error if the indexing object is not an integer, the code first checks to see if the object has a tp_as_number-nb_int method and calls it instead. If this is acceptable, it is an easy patch. Thanks, -Travis Oliphant ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing _PyEval_SliceIndex so that integer-like objects can be used
Would it be possible to change _PyEval_SliceIndex in ceval.c so that rather than throwing an error if the indexing object is not an integer, the code first checks to see if the object has a tp_as_number-nb_int method and calls it instead. I don't think this is the right solution; since float has that method, it would allow floats to be used as slice indices, but that's not supposed to work (to protect yourself against irreproducible results due to rounding errors). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing _PyEval_SliceIndex so that integer-like objects can be used
Travis Oliphant wrote: Hello again, There is a great discussion going on the numpy list regarding a proposed PEP for multidimensional arrays that is in the works. During this discussion as resurfaced regarding slicing with objects that are not IntegerType objects but that have a tp_as_number-nb_int method to convert to an int. Would it be possible to change _PyEval_SliceIndex in ceval.c so that rather than throwing an error if the indexing object is not an integer, the code first checks to see if the object has a tp_as_number-nb_int method and calls it instead. You would also have to change apply_slice() since that also has a guard for checking the slice arguments are either NULL, int, or long objects. But I am +1 with it since the guard is already there for ints and longs to handle those properly and thus the common case does not slow down in any way. As long as it also accepts Python objects that define __int__ and not just C types that have the nb_int slot defined I am okay with this idea. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fix _PyEval_SliceIndex (Take two)
(More readable second paragraph) Hello again, There is a great discussion going on the numpy list regarding a proposed PEP for multidimensional arrays that is in the works. During this discussion a problem has resurfaced regarding slicing with objects that are not IntegerType objects but that have a tp_as_number-nb_int method. Would it be possible to change _PyEval_SliceIndex in ceval.c so that rather than raising an exception if the indexing object is not an integer, the code first checks to see if the object has a tp_as_number-nb_int method and trys it before raising an exception. If this is acceptable, it is an easy patch. Thanks, -Travis Oliphant ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing _PyEval_SliceIndex so that integer-like objects can be used
On Fri, 18 Feb 2005 13:28:34 -0800, Guido van Rossum [EMAIL PROTECTED] wrote: Would it be possible to change _PyEval_SliceIndex in ceval.c so that rather than throwing an error if the indexing object is not an integer, the code first checks to see if the object has a tp_as_number-nb_int method and calls it instead. I don't think this is the right solution; since float has that method, it would allow floats to be used as slice indices, but that's not supposed to work (to protect yourself against irreproducible results due to rounding errors). I wonder if floats are the special case here, not integer like objects. I've never been particularly happy about the confusion between the two roles of int() and it's C equivalents, i.e. casting and conversion. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing _PyEval_SliceIndex so that integer-like objects can be used
On Feb 18, 2005, at 4:36 PM, David Ascher wrote: On Fri, 18 Feb 2005 13:28:34 -0800, Guido van Rossum [EMAIL PROTECTED] wrote: Would it be possible to change _PyEval_SliceIndex in ceval.c so that rather than throwing an error if the indexing object is not an integer, the code first checks to see if the object has a tp_as_number-nb_int method and calls it instead. I don't think this is the right solution; since float has that method, it would allow floats to be used as slice indices, but that's not supposed to work (to protect yourself against irreproducible results due to rounding errors). I wonder if floats are the special case here, not integer like objects. I've never been particularly happy about the confusion between the two roles of int() and it's C equivalents, i.e. casting and conversion. All of the __special__ methods for this purpose seem to be usable only for conversion, not casting (__str__, __unicode__, etc.). The only way I've found to pass for a particular value type is to subclass one. We do this a lot in PyObjC. It ends up being a net win anyway, because you get free implementations of all the relevant methods, at the expense of having two copies of the value. The fact that these proxy objects are no longer visible-from-Python subclasses of Objective-C objects isn't really a big deal in our case, because the canonical Objective-C way to checking inheritance still work. The wrapper types use an attribute protocol for casting (__pyobjc_object__), and delegate to this object with __getattr__. from Foundation import * one = NSNumber.numberWithInt_(1) type(one).mro() [class 'objc._pythonify.OC_PythonInt', type 'int', type 'object'] isinstance(one, NSNumber) False isinstance(one.__pyobjc_object__, NSNumber) True one.isKindOfClass_(NSNumber) 1 type(one) class 'objc._pythonify.OC_PythonInt' type(one.__pyobjc_object__) objective-c class NSCFNumber at 0x300620 -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows Low Fragementation Heap yields speedup of ~15%
On Thu, 2005-02-17 at 22:38, Tim Peters wrote: Then you allocate a small object, marked 's': bbbsfff Isn't the whole point of obmalloc is that we don't want to allocate s on the heap, since it is small? I guess s could be an object that might potentially grow. One thing to take from that is that LFH can't be helping list-growing in a direct way either, if LFH (as seems likely) also needs to copy objects that grow in order to keep its internal memory segregated by size. The indirect benefit is still available, though: LFH may be helping simply by keeping smaller objects out of the general heap's hair. So then wouldn't this mean that there would have to be some sort of small object being allocated via the system malloc that is causing the poor behaviour? As you mention, I wouldn't think it would be list objects, since resizing lists using LFH should be *worse*. That would actually be something that is worth verifying, however. It could be that the Windows LFH is extra clever? I'm afraid the only you can know for sure is by obtaining detailed memory maps and analyzing them. Well, it would also be useful to find out what code is calling the system malloc. This would make it easy to examine the code and see if it should be calling obmalloc or the system malloc. Any good ideas for easily obtaining this information? I imagine that some profilers must be able to produce a complete call graph? Evan Jones ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory Allocator Part 2: Did I get it right?
Sorry for taking so long to get back to this thread, it has been one of those weeks for me. On Feb 16, 2005, at 2:50, Martin v. Löwis wrote: Evan then understood the feature, and made it possible. This is very true: it was a very useful exercise. I can personally accept breaking the code that still relies on the invalid APIs. The only problem is that it is really hard to determine whether some code *does* violate the API usage. Great. Please ignore the patch on SourceForge for a little while. I'll produce a revision 3 this weekend, without the compatibility hack. Evan Jones ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing _PyEval_SliceIndex so that integer-like objects can be used
Guido van Rossum wrote: Would it be possible to change _PyEval_SliceIndex in ceval.c so that rather than throwing an error if the indexing object is not an integer, the code first checks to see if the object has a tp_as_number-nb_int method and calls it instead. I don't think this is the right solution; since float has that method, it would allow floats to be used as slice indices, O.K., then how about if arrayobjects can make it in the core, then a check for a rank-0 integer-type arrayobject is allowed before raising an exception? -Travis ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows Low Fragementation Heap yields speedup of ~15%
[Tim Peters] ... Then you allocate a small object, marked 's': bbbsfff [Evan Jones] Isn't the whole point of obmalloc No, because it doesn't matter what follows that introduction: obmalloc has several points, including exploiting the GIL, heuristics aiming at reusing memory while it's still high in the memory heirarchy, almost never touching a piece of memory until it's actually needed, and so on. is that we don't want to allocate s on the heap, since it is small? That's one of obmalloc's goals, yes. But small is a relative adjective, not absolute. Because we're primarily talking about LFH here, the natural meaning for small in _this_ thread is 16KB, which is much larger than small means to obmalloc. The memory-map example applies just well to LFH as to obmalloc, by changing which meaning for small you have in mind. I guess s could be an object that might potentially grow. For example, list guts in Python are never handled by obmalloc, although the small fixed-size list _header_ object is always handled by obmalloc. One thing to take from that is that LFH can't be helping list-growing in a direct way either, if LFH (as seems likely) also needs to copy objects that grow in order to keep its internal memory segregated by size. The indirect benefit is still available, though: LFH may be helping simply by keeping smaller objects out of the general heap's hair. So then wouldn't this mean that there would have to be some sort of small object being allocated via the system malloc that is causing the poor behaviour? Yes. For example, a 300-character string could do it (that's not small to obmalloc, but is to LFH). Strings produced by pickling are very often that large, and especially in Zope (which uses pickles extensively under the covers -- reading and writing persistent objects in Zope all involve pickle strings). As you mention, I wouldn't think it would be list objects, since resizing lists using LFH should be *worse*. Until they get to LFH's boundary for small, and we have only the vaguest idea what Martin's app does here -- we know it grows lists containing 50K elements in the end, and ... well, that's all I really know about it wink. A well-known trick is applicable in that case, if Martin thinks it's worth the bother: grow the list to its final size once, at the start (overestimating if you don't know for sure). Then instead of appending, keep an index to the next free slot, same as you'd do in C. Then the list guts never move, so if that doesn't yield the same kind of speedup without using LFH, list copying wasn't actually the culprit to begin with. That would actually be something that is worth verifying, however. Not worth the time to me -- Windows is closed-source, and I'm too old to enjoy staring at binary disassemblies any more. Besides, list guts can't stay in LFH after the list exceeds 4K elements. If list-copying costs are significant here, they're far more likely to be due to copying lists over 4K elements than under -- copying a list takes O(len(list)) time. So the realloc() strategy used by LFH _probably_ isn't of _primary)_ interest here. It could be that the Windows LFH is extra clever? Sure -- that I doubt it moves Heaven Earth to cater to reallocs is just educated guessing. I wrote my first production heap manager at Cray Research, around 1979 wink. ... Well, it would also be useful to find out what code is calling the system malloc. This would make it easy to examine the code and see if it should be calling obmalloc or the system malloc. Any good ideas for easily obtaining this information? I imagine that some profilers must be able to produce a complete call graph? Windows supports extensive facilities for analyzing heap usage, even from an external process that attaches to the process you want to analyze. Ditto for profiling. But it's not easy, and I don't know of any free tools that are of real help. If someone were motivated enough, it would probably be easiest to run Martin's app on a Linux box, and use the free Linux tools to analyze it. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows Low Fragementation Heap yields speedup of ~15%
On Feb 18, 2005, at 17:51, Tim Peters wrote: grow the list to its final size once, at the start (overestimating if you don't know for sure). Then instead of appending, keep an index to the next free slot, same as you'd do in C. Then the list guts never move, so if that doesn't yield the same kind of speedup without using LFH, list copying wasn't actually the culprit to begin with. If this *does* improve the performance of his application by 15%, that would strongly argue for an addition to the list API similar to Java's ArrayList.ensureCapacity or the STL's vectorT::reserve. Since the list implementation already maintains separate ints for the list array size and the list occupied size, this would really just expose this implementation detail to Python. I don't like revealing the implementation in this fashion, but if it does make a significant performance difference, it could be worth it. http://java.sun.com/j2se/1.5.0/docs/api/java/util/ ArrayList.html#ensureCapacity(int) http://www.sgi.com/tech/stl/Vector.html#4 Evan Jones ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: Re: Re: Prospective Peephole Transformation
[Phillip J. Eby]
Still, it's rather interesting that tuple.__contains__ appears slower than a
series of LOAD_CONST and == operations, considering that the tuple
should be doing basically the same thing, only without bytecode fetch-and-
decode overhead. Maybe it's tuple.__contains__ that needs optimizing
here?
[Fredrik Lundh]
wouldn't be the first time...
How soon we forget wink. Fredrik introduced a pile of optimizations
special-casing the snot out of small integers into ceval.c a long time
ago, like this in COMPARE_OP:
case COMPARE_OP:
w = POP();
v = TOP();
if (PyInt_CheckExact(w) PyInt_CheckExact(v)) {
/* INLINE: cmp(int, int) */
register long a, b;
register int res;
a = PyInt_AS_LONG(v);
b = PyInt_AS_LONG(w);
switch (oparg) {
case PyCmp_LT: res = a b; break;
case PyCmp_LE: res = a = b; break;
case PyCmp_EQ: res = a == b; break;
case PyCmp_NE: res = a != b; break;
case PyCmp_GT: res = a b; break;
case PyCmp_GE: res = a = b; break;
case PyCmp_IS: res = v == w; break;
case PyCmp_IS_NOT: res = v != w; break;
default: goto slow_compare;
}
x = res ? Py_True : Py_False;
Py_INCREF(x);
}
else {
slow_compare:
x = cmp_outcome(oparg, v, w);
}
That's a hell of a lot faster than tuple comparison's deferral to
PyObject_RichCompareBool can be, even if we inlined the same blob
inside the latter (then we'd still have the additional overhead of
calling PyObject_RichCompareBool). As-is, PyObject_RichCompareBool()
has to do (relatively) significant work just to out find which
concrete comparision implementation to call.
As a result, i == j in Python source code, when i and j are little
ints, is much faster than comparing i and j via any other route in
Python. That's mostly really good, IMO -- /F's int optimizations are
of major value in real life. Context-dependent optimizations make
code performance less predictable too -- that's life.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
RE: [Python-Dev] Prospective Peephole Transformation
Wouldn't it help a lot more if the compiler would detect that
(1,2,3) is immutable and convert it into a constant at
compile time ?!
Yes. We've already gotten it to that point:
. . .
Cool. Does that work for all tuples in the program ?
It is limited to just tuples of constants (strings, ints, floats,
complex, None, and other tuples). Also, it is limited in its ability to
detect a nesting like: a=((1,2),(3,4)). One other limitation is that
floats like -0.23 are not recognized as constants because the initial
compilation still produces a UNARY_NEGATIVE operation:
dis.dis(compile('-0.23', '', 'eval'))
0 0 LOAD_CONST 0 (0.23001)
3 UNARY_NEGATIVE
4 RETURN_VALUE
I did a search on our code and Python's std lib. It turns
out that by far most such usages use either 2 or 3
values in the tuple. If you look at the types of the
values, the most common usages are strings and integers.
Right, those are the most common cases. The linear searches are
ubiquitous. Here's a small selection:
if comptype not in ('NONE', 'ULAW', 'ALAW', 'G722')
return tail.lower() in (.py, .pyw)
assert n in (2, 3, 4, 5)
if value[2] in ('F','n','N')
if sectName in (temp, cdata, ignore, include, rcdata)
if not decode or encoding in ('', '7bit', '8bit', 'binary'):
if (code in (301, 302, 303, 307) and m in (GET, HEAD)
Unfortunately, there are several common patterns that are skipped
because rarely changed globals/builtins cannot be treated as constants:
if isinstance(x, (int, float, complex)): # types are not constants
if op in (ROT_TWO, POP_TOP, LOAD_FAST): # global consts from
opcode.py
except (TypeError, KeyError, IndexError): # builtins are not
constant
I'd assume that you'll get somewhat different results
from your benchmark if you had integers in the tuple.
Nope, the results are substantially the same give or take 2usec.
Hmm, what if you'd teach tuples to do faster contains lookups for
string or integer only content, e.g. by introducing sub-types for
string-only and integer-only tuples ?!
For a linear search, tuples are already pretty darned good and leave
room for only microscopic O(n) improvements. The bigger win comes from
using a better algorithm and data structure -- hashing beats linear
search hands-down. The constant search time is faster for all n1,
resulting in much improved scalability. No tweaking of
tuple.__contains__() can match it.
Sets are the right data structure for fast membership testing. I would
love for sets to be used internally while letting users continue to
write the clean looking code shown above.
Raymond
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Prospective Peephole Transformation
[Raymond Hettinger] ... The problem with the transformation was that it didn't handle the case where x was non-hashable and it would raise a TypeError instead of returning False as it should. I'm very glad you introduced the optimization of building small constant tuples at compile-time. IMO, that was a pure win. I don't like this one, though. The meaning of x in (c1, c2, ..., c_n) is x == c1 or x == c2 or ... or x == c_n, and a transformation that doesn't behave exactly like the latter in all cases is simply wrong. Even if x isn't hashable, it could still be of a type that implements __eq__, and where x.__eq__(c_i) returned True for some i, and then False is plainly the wrong result. It could also be that x is of a type that is hashable, but where x.__hash__() raises TypeError at this point in the code. That could be for good or bad (bug) reasons, but suppressing the TypeError and converting into False would be a bad thing regardless. That situation arose once in the email module's test suite. I don't even care if no code in the standard library triggered a problem here: the transformation isn't semantically correct on the face of it. If we knew the type of x at compile-time, then sure, in most (almost all) cases we could know it was a safe transformation (and even without the hack to turn TypeError into False). But we don't know now, so the worst case has to be assumed: can't do this one now. Maybe someday, though. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows Low Fragementation Heap yields speedup of ~15%
[Tim Peters] grow the list to its final size once, at the start (overestimating if you don't know for sure). Then instead of appending, keep an index to the next free slot, same as you'd do in C. Then the list guts never move, so if that doesn't yield the same kind of speedup without using LFH, list copying wasn't actually the culprit to begin with. [Evan Jones] If this *does* improve the performance of his application by 15%, that would strongly argue for an addition to the list API similar to Java's ArrayList.ensureCapacity or the STL's vectorT::reserve. Since the list implementation already maintains separate ints for the list array size and the list occupied size, this would really just expose this implementation detail to Python. I don't like revealing the implementation in this fashion, but if it does make a significant performance difference, it could be worth it. That's a happy thought! It was first suggested for Python in 1991 wink, but before Python 2.4 the list implementation didn't have separate members for current size and capacity, so can't get there from here was the only response. It still wouldn't be trivial, because nothing in listobject.c now believes the allocated size ever needs to be preserved, and all len()-changing list operations ensure that not too much overallocation remains (see list_resize() in listobject.c for details). But let's see whether it would help first. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Proposal for a module to deal with hashing
Gregory P. Smith wrote: fyi - i've updated the python sha1/md5 openssl patch. it now replaces the entire sha and md5 modules with a generic hashes module that gives access to all of the hash algorithms supported by OpenSSL (including appropriate legacy interface wrappers and falling back to the old code when compiled without openssl). https://sourceforge.net/tracker/index.php?func=detailaid=1121611group_id=5470atid=305470 I don't quite like the module name 'hashes' that i chose for the generic interface (too close to the builtin hash() function). Other suggestions on a module name? 'digest' comes to mind. 'hashtools' and 'hashlib' would both have precedents in the standard library (itertools and urllib, for example). It occurs to me that such a module would provide a way to fix the bug with incorrectly hashable instances of new-style classes: Py class C: ... def __eq__(self, other): return True ... Py hash(C()) Traceback (most recent call last): File stdin, line 1, in ? TypeError: unhashable instance Py class C(object): ... def __eq__(self, other): return True ... Py hash(C()) 10357232 Guido wanted to fix this by eliminating object.__hash__, but that caused problems for Jython. If I remember that discussion correctly, the problem was that, in Jython, the default hash is _not_ simply hash(id(obj)) the way it is in CPython, so Python code needs a way to get access to the default implementation. A hashtools.default_hash that worked like the current object.__hash__ would seem to provide such a spelling, and allow object.__hash__ to be removed (fixing the above bug). Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.skystorm.net ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
RE: [Python-Dev] Prospective Peephole Transformation
I'm very glad you introduced the optimization of building small
constant tuples at compile-time. IMO, that was a pure win.
It's been out in the wild for a while now with no issues. I'm somewhat
happy with it.
the transformation isn't semantically correct on the
face of it.
Well that's the end of that.
What we really need is a clean syntax for specifying a constant
frozenset without compiler transformations of tuples. That would have
the further advantage of letting builtins and globals be used as element
values.
if isinstance(x, {int, float, complex}):
if opcode in {REPEAT, MIN_REPEAT, MAX_REPEAT}:
if (code in {301, 302, 303, 307} and m in {GET, HEAD}:
if op in (ROT_TWO, POP_TOP, LOAD_FAST)
Perhaps something other notation would be better but the idea is
basically the same.
Raymond
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Requesting that a class be a new-style class
This is something I've typed way too many times: Py class C(): File stdin, line 1 class C(): ^ SyntaxError: invalid syntax It's the asymmetry with functions that gets to me - defining a function with no arguments still requires parentheses in the definition statement, but defining a class with no bases requires the parentheses to be omitted. Which leads in to the real question: Does this *really* need to be a syntax error? Or could it be used as an easier way to spell class C(object):? Then, in Python 3K, simply drop support for omitting the parentheses from class definitions - require inheriting from ClassicClass instead. This would also have the benefit that the elimination of defaulting to classic classes would cause a syntax error rather than subtle changes in behaviour. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.skystorm.net ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] builtin_id() returns negative numbers
From: Armin Rigo [EMAIL PROTECTED]
Hi Tim,
On Thu, Feb 17, 2005 at 01:44:11PM -0500, Tim Peters wrote:
256 ** struct.calcsize('P')
Now if you'll just sign and fax a Zope contributor agreement, I'll
upgrade ZODB to use this slick trick wink.
I hereby donate this line of code to the public domain :-)
Damn... we can't use it then!
Seriously, on the Python lists there has been a discussion rejecting an
md5sum implementation because the author donated it to the public domain.
Apparently lawyers have decided that you can't give code away. Intellectual
charity is illegal :-)
Donovan Baardahttp://minkirri.apana.org.au/~abo/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] license issues with profiler.py and md5.h/md5c.c
From: Martin v. Löwis [EMAIL PROTECTED] Donovan Baarda wrote: This patch keeps the current md5c.c, md5module.c files and adds the following; _hashopenssl.c, hashes.py, md5.py, sha.py. [...] If all we wanted to do was fix the md5 module If we want to fix the licensing issues with the md5 module, this patch does not help at all, as it keeps the current md5 module (along with its licensing issues). So any patch to solve the problem will need to delete the code with the questionable license. It maybe half fixes it in that if Python is happy with the RSA one, they can continue to include it, and if Debian is unhappy with it, they can remove it and build against openssl. It doesn't fully fix the license problem. It is still worth considering because it doesn't make it worse, and it does allow Python to use much faster implementations and support other digest algorithms when openssl is available. Then, the approach in the patch breaks the promise that the md5 module is always there. It would require that OpenSSL is always there - a promise that we cannot make (IMO). It would be better if found an alternative md5c.c. I found one that was the libmd implementation that someone mildly tweaked and then slapped an LGPL on. I have a feeling that would make the lawyers tremble more than the public domain libmd one, unless they are happy that someone else is prepared to wear the grief for slapping a LGPL onto something public domain. Probably the best at the moment is the sourceforge one, which is listed as having a zlib/libpng licence. Donovan Baardahttp://minkirri.apana.org.au/~abo/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] license issues with profiler.py and md5.h/md5c.c
On Fri, Feb 18, 2005 at 10:06:24AM +0100, Martin v. L?wis wrote: Donovan Baarda wrote: This patch keeps the current md5c.c, md5module.c files and adds the following; _hashopenssl.c, hashes.py, md5.py, sha.py. [...] If all we wanted to do was fix the md5 module If we want to fix the licensing issues with the md5 module, this patch does not help at all, as it keeps the current md5 module (along with its licensing issues). So any patch to solve the problem will need to delete the code with the questionable license. Then, the approach in the patch breaks the promise that the md5 module is always there. It would require that OpenSSL is always there - a promise that we cannot make (IMO). I'm aware of that. My goals are primarily to get a good openssl based hashes/digest module going to be used instead of the built in implementations when openssl available because openssl is -so- much faster. Fixing the debian instigated md5 licensing issue is secondary and is something I'll get to later on after i work on the fun stuff. And as Donovan has said, the patch already does present debian with the option of dropping that md5 module and using the openssl derived one instead if they're desperate. based on laziness winning and the issue being so minor i hope they just wait for a patch from me that replaces the md5c.c with one of the acceptably licensed ones for their 2.3/2.4 packages. -g ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Requesting that a class be a new-style class
On 2005 Feb 19, at 06:03, Nick Coghlan wrote: This is something I've typed way too many times: Py class C(): File stdin, line 1 class C(): ^ SyntaxError: invalid syntax It's the asymmetry with functions that gets to me - defining a function with no arguments still requires parentheses in the definition statement, but defining a class with no bases requires the parentheses to be omitted. Seconded. It's always irked me enough that it's the only ``apology'' for Python syntax you'll see in the Nutshell -- top of p. 71, The syntax of the class statement has a small, tricky difference from that of the def statement etc. Which leads in to the real question: Does this *really* need to be a syntax error? Or could it be used as an easier way to spell class C(object):? -0 ... instinctively, I dread the task of explaining / teaching about the rationale for this somewhat kludgy transitional solution [[empty parentheses may be written OR omitted, with large difference in meaning, not very related to other cases of such parentheses]], even though I think you're right that it would make the future transition to 3.0 somewhat safer. Alex ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
