Re: [Python-Dev] Floor division
Guido van Rossum [EMAIL PROTECTED] wrote: (int)float_or_double truncates in C (even in KR C) /provided that/ the true result is representable as an int. Else behavior is undefined (may return -1, may cause a HW fault, ...). Actually, I have used Cs that didn't, but haven't seen any in over 10 years. C90 is unclear about its intent, but C99 is specific that truncation is towards zero. This is safe, at least for now. So Python uses C's modf() for float-int now, which is always defined for finite floats, and also truncates. Yes. And that is clearly documented and not currently likely to change, as far as I know. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Floor division
Tim Peters [EMAIL PROTECTED] wrote: It could, but who would have a (sane) use for a possibly 2000-bit quotient? Well, the 'exact rounding' camp in IEEE 754 seem to think that there is one :-) As you can gather, I can't think of one. Floating-point is an inherently inaccurate representation for anything other than small integers. This is a bit peculiar to me, because there are ways to compute remainder using a number of operations proportional to the log of the exponent difference. It could be that people who spend their life doing floating point forget how to work with integers ;-) Aargh! That is indeed the key! Given that I claim to know something about integer arithmetic, too, how can I have been so STUPID? Yes, you are right, and that is the only plausible way to calculate the remainder precisely. You don't get the quotient precisely, which is what my (insane) specification would have provided. I would nitpick with your example, because you don't want to reduce modulo 3.14 but modulo pi and therefore the modular arithmetic is rather more expensive (given Decimal). However, it STILL doesn't help to make remquo useful! The reason is that pi is input only to the floating-point precision, and so the result of remquo for very large arguments will depend more on the inaccuracy of pi as input than on the mathematical result. That makes remquo totally useless for the example you quote. Yes, I have implemented 'precise' range reduction, and there is no substitute for using an arbitrary precision pi value :-( But it isn't obviously WRONG. For floats, fmod(x, y) is exactly congruent to x modulo y -- I don't think it's possible to get more right than exactly right ;-) But, as a previous example of yours pointed out, it's NOT exactly right. It is also supposed to be in the range [0,y) and it isn't. -1%1e100 is mathematically wrong on two counts. 1a Cc: Tim Peters [EMAIL PROTECTED] Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Floor division
[Tim (misattributed to Guido)] (int)float_or_double truncates in C (even in KR C) /provided that/ the true result is representable as an int. Else behavior is undefined (may return -1, may cause a HW fault, ...). [Nick Maclaren] Actually, I have used Cs that didn't, but haven't seen any in over 10 years. I believe those. C90 is unclear about its intent, But am skeptical of that. I don't have a copy of C90 here, but before I wrote that I checked Kernighan Ritchie's seminal C book, Harbison Steele's generally excellent C: A Reference Manual (2nd ed), and a web version of Plauger Brodie's Standard C: http://www-ccs.ucsd.edu/c/ They all agree that the Cs they describe (all of which predate C99) convert floating to integral types via truncation, when possible. but C99 is specific that truncation is towards zero. As opposed to what? Truncation away from zero? I read truncation as implying toward 0, although the Plauger Brodie source is explicit about the integer part of X, truncated toward zero for the sake of logic choppers ;-) This is safe, at least for now. So Python uses C's modf() for float-int now, which is always defined for finite floats, and also truncates. Yes. And that is clearly documented and not currently likely to change, as far as I know. I don't expect to see another C standard in my lifetime, given that some major C compiler vendors still ignore C99 (and given that my expected remaining lifetime is much less than that of most people reading this ;-)). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Floor division
Tim Peters [EMAIL PROTECTED] wrote: [Tim (misattributed to Guido)] Apologies to both! C90 is unclear about its intent, But am skeptical of that. I don't have a copy of C90 here, but before I wrote that I checked Kernighan Ritchie's seminal C book, Harbison Steele's generally excellent C: A Reference Manual (2nd ed), and a web version of Plauger Brodie's Standard C: http://www-ccs.ucsd.edu/c/ They all agree that the Cs they describe (all of which predate C99) convert floating to integral types via truncation, when possible. I do. Kernighan Ritchie's seminal C book describes the Unix style of KR C - one of the reasons that ANSI/ISO had to make incompatible changes was that many important PC and embedded Cs differed. Harbison and Steele is generally reliable, but not always; I haven't looked at the last, but I would regard it suspiciously. What C90 says is: When a value of floating type is converted to integer type, the fractional part is discarded. There is other wording, but none relevant to this issue. Now, given the history of floating-point remainder, that is seriously ambiguous. but C99 is specific that truncation is towards zero. As opposed to what? Truncation away from zero? I read truncation as implying toward 0, although the Plauger Brodie source is explicit about the integer part of X, truncated toward zero for the sake of logic choppers ;-) Towards -infinity, of course. That was as common as truncation towards zero up until the 1980s. It was near-universal on twos complement floating-point systems, and not rare on signed magnitude ones. During the standardisation of C90, the BSI tried to explain to ANSI that this needed spelling out, but were ignored. C99 did not add the normative text (i.e., the value is truncated toward zero) because there was no ambiguity, after all! Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Floor division
[Tim Peters] ... [it would be possible for float.__divmod__ to return the exact quotient], but who would have a (sane) use for a possibly 2000-bit quotient? [Nick Maclaren] Well, the 'exact rounding' camp in IEEE 754 seem to think that there is one :-) As you can gather, I can't think of one. Floating-point is an inherently inaccurate representation for anything other than small integers. Well, bounded. Python's decimal fp can handle millions of digits, if you want that and are very patient ;-) OTOH, I am a fan of analyzing FP operations as if the inputs were in fact exactly what they claim to be, which 754 went a long way toward popularizing. That largely replaced mountains of idiosyncratic probabilistic arguments (and where it seemed no two debaters ever agreed on the proper approach) with a common approach that sometimes allows surprisingly sharp analysis. Since I spent a good part of my early career as a professional apologist for Seymour Cray's creative floating point, I'm probably much more grateful to leave sloppy arithmetic behind than most. ... This [that IBM's spec calls for an exception if remainder's inputs' exponents differ by too much] is a bit peculiar to me, because there are ways to compute remainder using a number of operations proportional to the log of the exponent difference. It could be that people who spend their life doing floating point forget how to work with integers ;-) Aargh! That is indeed the key! Given that I claim to know something about integer arithmetic, too, how can I have been so STUPID? You're not alone. I thought of this a decade ago, but have never seen it implemented, or even mentioned. All implementations of fmod I've seen emulate long binary division one bit at a time; the IBM spec seems to assume that's how it would be done for decimal too (why else mandate an exception instead if the exponent delta is large?); and so on. Yes, you are right, and that is the only plausible way to calculate the remainder precisely. You don't get the quotient precisely, which is what my (insane) specification would have provided. And, alas, without the last bit of the quotient IBM's remainder-near (== C99's remainder == 754's REM) can't resolve halfway cases in the mandated way. I would nitpick with your example, because you don't want to reduce modulo 3.14 but modulo pi I didn't have pi in mind at all. I just picked 3.14 as an arbitrary decimal modulus with few digits, to make the example easy to follow. Could just as well have been 2.72 (which has nothing to do with e either ;-)). and therefore the modular arithmetic is rather more expensive (given Decimal). However, it STILL doesn't help to make remquo useful! The reason is that pi is input only to the floating-point precision, and so the result of remquo for very large arguments will depend more on the inaccuracy of pi as input than on the mathematical result. That makes remquo totally useless for the example you quote. That's not our choice to make. Many math libraries still use a small approximation to pi for trig argument reduction, and that's their choice. Heck, a 66-bit value for pi is built in to the Pentium's FSIN and FCOS instructions. math.sin(math.pi) 1.2246063538223773e-016 That was on Windows with Python 2.5, using the MSVC compiler. The result indicates it uses the FSIN instruction. Same thing but under the Cygwin Python, whose libm uses infinite precision pi trig reduction: math.sin(math.pi) 1.2246467991473532e-16 That one is correct (close to the best double approximation to the sine of the best double approximation to pi). The FSIN result is off by about 164 billion(!) ULP, but few people care. Anyway, I simply gave cosine arg reduction as /an example/ of the kind of reduction strategy for which remquo can be useful. You said you were baffled by why C99 only required the last 3 bits. The example was a hint about why some functions can indeed find that to be useful, not an exhaustive rationale. It's really off-topic for Python-Dev, so I didn't/don't want to belabor it. Yes, I have implemented 'precise' range reduction, and there is no substitute for using an arbitrary precision pi value :-( I have too (for 754 doubles), coincidentally at the same time KC Ng was implementing it for fdlibm. I found some bugs in fdlibm's trig functions at the time by generating billions of random inputs and comparing his library-in-progress's results to mine. That was fun. My users (this was done for Kendall Square Research's libm) only noticed that the new trig functions were much faster than the old ones for small arguments -- nobody seemed to notice that the results were much better than the old ones, or arrived at much more slowly, for very large arguments. How satisfying ;-) But it isn't obviously WRONG. For floats, fmod(x, y) is exactly congruent to x modulo y -- I don't think it's possible to get more right than exactly right ;-) But, as a
Re: [Python-Dev] Floor division
Tim Peters [EMAIL PROTECTED] wrote: OTOH, I am a fan of analyzing FP operations as if the inputs were in fact exactly what they claim to be, which 754 went a long way toward popularizing. That largely replaced mountains of idiosyncratic probabilistic arguments (and where it seemed no two debaters ever agreed on the proper approach) with a common approach that sometimes allows surprisingly sharp analysis. Since I spent a good part of my early career as a professional apologist for Seymour Cray's creative floating point, I'm probably much more grateful to leave sloppy arithmetic behind than most. Well, I spent some of it working with code (and writing code) that was expected to work, unchanged, on an ICL 1900, CDC 6600/7600, IBM 370 and others. I have seen the harm caused by the 'exact arithmetic' mindset and so don't like it, but I agree about your objections to the probabilistic arguments (which were and are mostly twaddle). But that is seriously off-topic. [remquo] It's really off-topic for Python-Dev, so I didn't/don't want to belabor it. Agreed, except in one respect. I stand by my opinion that the C99 specification has no known PRACTICAL use (your example is correct, but I know of no such use in a real application), and so PLEASE don't copy it as a model for Python divmod/remainder. No, /Python's/ definition of mod is inexact for that example. fmod (which is not Python's definition) is always exact: fmod(-1, 1e100) = -1, and -1 is trivially exactly congruent to -1 modulo anything (including modulo 1e100). The result of fmod(x, y) has the same sign as x; Python's x.__mod__(y) has the same sign as y; and that makes all the difference in the world as to whether the exact result is always exactly representable as a float. Oops. You're right, of course. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python's C interface for types
I have a fair amount of my binary floating-point model written, though even of what I have done only some is debugged (and none has been rigorously tested). But I have hit some things that I can't work out, and one query reduced comp.lang.python to a stunned silence :-) Note that I am not intending to do all the following, at least for now, but I have had to restructure half a dozen times to match my implementation requirements to the C interface (as I have learnt more about Python!) and designing to avoid that is always good. Any pointers appreciated. I can't find any detailed description of the methods that I need to provide. Specifically: Does Python use classic division (nb_divide) and inversion (nb_invert) or are they entirely historical? Note that I can very easily provide the latter. Is there any documentation on the coercion function (nb_coerce)? It seems to have unusual properties. How critical is the 'numeric' property of the nb_hash function? I can certainly honour it, but is it worth it? I assume that Python will call nb_richcompare if defined and nb_compare if not. Is that right? Are the inplace methods used and, if so, what is their specification? I assume that I can ignore all of the allocation, deallocation and attribute handling functions, as the default for a VAR object is fine. That seems to work. Except for one thing! My base type is static, but I create some space for every derivation (and it can ONLY be used in derived form). The space creation is donein C but the derivation in Python. I assume that I need a class (not instance) destructor, but what should it do to free the space? Call C to Py_DECREF it? I assume that a class structure will never go away until after all instances have gone away (unless I use Py_DECREF), so a C pointer from an instance to something owned by the class is OK. Is there any documentation on how to support marshalling/pickling and the converse from C types? I would quite like to provide some attributes. They are 'simple' but need code executing to return them. I assume that means that they aren't simple enough, and have to be provided as methods (like conjugate). That's what I have done, anyway. Is there any obvious place for a reduction method to be hooked in? That is a method that takes a sequence, all members of which must be convertible to a single class, and returns a member of that class. Note that it specifically does NOT make sense on a single value of that class. Sorry about the length of this! Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Floor division
Nick, I didn't write that; Tim did. If you're going to enter into a pedantic discussion, at least get your attributions right, On 1/26/07, Nick Maclaren [EMAIL PROTECTED] wrote: Guido van Rossum [EMAIL PROTECTED] wrote: (int)float_or_double truncates in C (even in KR C) /provided that/ the true result is representable as an int. Else behavior is undefined (may return -1, may cause a HW fault, ...). Actually, I have used Cs that didn't, but haven't seen any in over 10 years. C90 is unclear about its intent, but C99 is specific that truncation is towards zero. This is safe, at least for now. So Python uses C's modf() for float-int now, which is always defined for finite floats, and also truncates. Yes. And that is clearly documented and not currently likely to change, as far as I know. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python's C interface for types
On 1/26/07, Nick Maclaren [EMAIL PROTECTED] wrote:
Does Python use classic division (nb_divide) and inversion (nb_invert)
or are they entirely historical? Note that I can very easily provide
the latter.
nb_divide is used for the division operation (/) when not using 'from
__future__ import division', and PyNumber_Divide(). It doesn't fall back to
foor_divide or true_divide.
nb_invert is used for bitwise inversion (~) and PyNumber_Invert(). It's not
historical, it's actual.
Is there any documentation on the coercion function (nb_coerce)? It
seems to have unusual properties.
I don't recall ever seeing useful documentation on coerce() and nb_coerce. I
suggest not to use it; it's gone in Python 3.0 anyway.
How critical is the 'numeric' property of the nb_hash function? I
can certainly honour it, but is it worth it?
Which numeric property? the fact that it returns a C long? Or that, for
natural numbers, it *seems* to return self? The former is quite important,
the latter not very. The important bit is that an object's hash() be equal
to the hash() of objects that it is considered equal to. That is to say, if
you have 'd = {5: five}' and you want 'f = yourtype(5); d[f]' to result in
five, hash(f) must be equal to hash(5). The builtin floats do that.
There's no strict requirement that equal objects must have equal hashes, but
people using sets and dicts really appreciate it. ('f == 5 and f not in
set([5])' to be True would be confusing.)
I assume that Python will call nb_richcompare if defined and
nb_compare if not. Is that right?
Yes.
Are the inplace methods used and, if so, what is their specification?
(Hah, my specialty.) Inplace methods are used for the augmented-assignment
statements: '+=' and the like. They are free to modify 'self' or return a
new object. Python falls back to the normal, non-inplace operations if the
inplace methods are not defined. I assume your floating-point type is
immutable, so you won't have to implement them. (If the type is to be
mutable, don't forget to make them unhashable, by not defining nb_hash.)
I assume that I can ignore all of the allocation, deallocation and
attribute handling functions, as the default for a VAR object is
fine. That seems to work.
Except for one thing! My base type is static, but I create some
space for every derivation (and it can ONLY be used in derived form).
The space creation is donein C but the derivation in Python. I
assume that I need a class (not instance) destructor, but what
should it do to free the space? Call C to Py_DECREF it?
Where do you allocate this space, and how do you allocate it? If it's space
you malloc() and store somewhere in the type struct, yecchh. You should not
just allocate stuff at the end of the type struct, as the type struct's
layout is not under your control (we actually extend the type struct as
needed, which is why newer features end up in less logical places at the end
of the struct ;) I would suggest using attributes of the type instead, with
the normal Python refcounting. That means the 'extra space' has to be an
actual Python object, though.
I assume that a class structure will never go away until after all
instances have gone away (unless I use Py_DECREF), so a C pointer
from an instance to something owned by the class is OK.
Correct.
Is there any documentation on how to support marshalling/pickling
and the converse from C types?
I don't you can make your own type marshallable. For pickle it's more or
less the same as for Python types. The pickle docs (and maybe
http://www.python.org/dev/peps/pep-0307/) probably cover what you want to
know. You can also look at one of the complexer builtin types that support
pickling, like the datetime types.
I would quite like to provide some attributes. They are 'simple'
but need code executing to return them. I assume that means that
they aren't simple enough, and have to be provided as methods
(like conjugate). That's what I have done, anyway.
You can use PyGetSetDef to get 'easy' attributes with getters and setters.
http://docs.python.org/api/type-structs.html#l2h-1020
Is there any obvious place for a reduction method to be hooked in?
That is a method that takes a sequence, all members of which must
be convertible to a single class, and returns a member of that
class. Note that it specifically does NOT make sense on a single
value of that class.
There's nothing I can think of that is a natural match for that in standard
Python methods. I would suggest just making it a classmethod.
(dict.fromkeysis a good example of a classmethod in C.)
As a final note: Python's source itself is a good source of answers and
examples. Almost all of the features of the Python API are used in a builtin
type or stdlib module, and for C source, Python's source is remarkably
readable ;-)
--
Thomas Wouters [EMAIL PROTECTED]
Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
Re: [Python-Dev] Python's C interface for types
Thanks very much! That answers most things. Yes, I had got many of my answers from searching the source, but there is clearly some history there, and it isn't always clear what is current. Here are a few responses to the areas of confusion: nb_invert is used for bitwise inversion (~) and PyNumber_Invert(). It's not historical, it's actual. Ah! So it's NOT 1/x! No relevant to floating-point, then. I don't recall ever seeing useful documentation on coerce() and nb_coerce. I suggest not to use it; it's gone in Python 3.0 anyway. Excellent! Task completed :-) Which numeric property? the fact that it returns a C long? Or that, for natural numbers, it *seems* to return self? The latter. hash(123) == hash(123.0) for example. It is a real pain for advanced formats. Making it the same for things that compare equal isn't a problem. [inplace ] I assume your floating-point type is immutable, so you won't have to implement them. I haven't done anything special to flag it as such, but it is. Where do you allocate this space, and how do you allocate it? If it's space you malloc() and store somewhere in the type struct, yecchh. You should not just allocate stuff at the end of the type struct, as the type struct's layout is not under your control (we actually extend the type struct as needed, which is why newer features end up in less logical places at the end of the struct ;) I would suggest using attributes of the type instead, with the normal Python refcounting. That means the 'extra space' has to be an actual Python object, though. PyMem_Malloc. I can certainly make it an attribute, as the overhead isn't large for a per-class object. It is just a block of mutable memory, opaque to the Python layer, and NOT containing any pointers! I don't you can make your own type marshallable. For pickle it's more or less the same as for Python types. The pickle docs (and maybe http://www.python.org/dev/peps/pep-0307/) probably cover what you want to know. You can also look at one of the complexer builtin types that support pickling, like the datetime types. The only documentation I have found is how to do it in Python. Is that what you mean? I will look at the datetime types. You can use PyGetSetDef to get 'easy' attributes with getters and setters. http://docs.python.org/api/type-structs.html#l2h-1020 I was put off by some of the warnings. I will revisit it. There's nothing I can think of that is a natural match for that in standard Python methods. I would suggest just making it a classmethod. (dict.fromkeysis a good example of a classmethod in C.) Thanks. That is a useful reference. Reductions are a problem in many languages. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python's C interface for types
Oops. Something else fairly major I forgot to ask. Python long. I can't find any clean way of converting to or from this, and would much rather not build a knowledge of long's internals into my code. Going via text is, of course, possible - but is not very efficient, even using hex/octal. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python's C interface for types
On 26/01/2007 17.03, Thomas Wouters wrote: How critical is the 'numeric' property of the nb_hash function? I can certainly honour it, but is it worth it? [...] There's no strict requirement that equal objects must have equal hashes, Uh? I thought that was the *only* strict requirement of hash. In fact the docs agree: __hash__( self) Called for the key object for dictionary operations, and by the built-in function hash(). Should return a 32-bit integer usable as a hash value for dictionary operations. The only required property is that objects which compare equal have the same hash value; [...] I personally consider *very* important that hash(5.0) == hash(5) (and that 5.0 == 5, of course). -- Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python's C interface for types
Nick Maclaren [EMAIL PROTECTED] wrote: Oops. Something else fairly major I forgot to ask. Python long. I can't find any clean way of converting to or from this, and would much rather not build a knowledge of long's internals into my code. Going via text is, of course, possible - but is not very efficient, even using hex/octal. See _PyLong_FromByteArray and _PyLong_AsByteArray . - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python's C interface for types
Giovanni Bajo [EMAIL PROTECTED] wrote: I personally consider *very* important that hash(5.0) == hash(5) (and that 5.0 == 5, of course). It gets a bit problematic with floating-point, when you can have different values exactly 5.0 and approximately 5.0. IEEE 754 has signed zeroes. And so it goes. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python's C interface for types
Josiah Carlson [EMAIL PROTECTED] wrote: See _PyLong_FromByteArray and _PyLong_AsByteArray . Oops! Thanks very much. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python's C interface for types
On Fri, Jan 26, 2007, Giovanni Bajo wrote: On 26/01/2007 17.03, Thomas Wouters wrote: There's no strict requirement that equal objects must have equal hashes, Uh? I thought that was the *only* strict requirement of hash. In fact the docs agree: __hash__( self) Called for the key object for dictionary operations, and by the built-in function hash(). Should return a 32-bit integer usable as a hash value for dictionary operations. The only required property is that objects which compare equal have the same hash value; [...] Possibly the docs should be updated, but this is only intended to apply to objects of the same type. I personally consider *very* important that hash(5.0) == hash(5) (and that 5.0 == 5, of course). Well, sure, but That's Different. -- Aahz ([EMAIL PROTECTED]) * http://www.pythoncraft.com/ I disrespectfully agree. --SJM ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python's C interface for types
Having looked into the answers a bit more deeply, I am afraid that I am still a bit puzzled. 1) As I understand it, PyMem_Malloc won't cause trouble, but won't be automatically freed, either, as it doesn't return a new reference. I don't think that immediately following it by PyCObject_FromVoidPtr (which is what I do) helps with that. What I need is some standard type that allows me to allocate an anonymous block of memory; yes, I can define such a type, but that seems excessive. Is there one? 2) _PyLong_FromByteArray and _PyLong_AsByteArray aren't in the API and have no comments. Does that mean that they are unstable, in the sense that they may change behaviour in new versions of Python? And will they be there in 3.0? Thanks for any help, again. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: [EMAIL PROTECTED] Tel.: +44 1223 334761Fax: +44 1223 334679 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python's C interface for types
Nick Maclaren schrieb: I personally consider *very* important that hash(5.0) == hash(5) (and that 5.0 == 5, of course). It gets a bit problematic with floating-point, when you can have different values exactly 5.0 and approximately 5.0. IEEE 754 has signed zeroes. And so it goes. [not sure what And so it goes means in English] It may be a bit problematic to implement, but I think a clean specification is possible. If a and b are numbers, and a==b, then hash(a)==hash(b). I'm not sure whether approximately 5.0 equals 5 or not: if it does, it should hash the same as 5, if it doesn't, it may or may not hash the same (whatever is easier to implement). For 0: hash(+0.0)==hash(-0.0)==hash(0)=hash(0L)=0 Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python's C interface for types
[Nick Maclaren] ... 2) _PyLong_FromByteArray and _PyLong_AsByteArray aren't in the API They're not in the public API, which is equivalent to that their names begin with a leading underscore. They're in the private API :-) and have no comments. The behavior of these functions, including return value and error conditions, is specified in longobject.h. Does that mean that they are unstable, in the sense that they may change behaviour in new versions of Python? They /may/ change, but they won't (== only common sense guarantees they won't change ;-)). And will they be there in 3.0? Almost certainly so. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
Martin v. Löwis wrote: Please try to come up with a patch (e.g. by putting a while(is_tripped) loop around the for loop). That isn't going to fix it. What's needed is to somehow atomically test and clear is_tripped at the beginning. You can put a while loop around it as well if you want, but it's not strictly necessary. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Problem with signals in a single threaded application
Nick Maclaren wrote: This one looks like an oversight in Python code, and so is a bug, but it is important to note that signals do NOT work reliably under any Unix or Microsoft system. That's a rather pessimistic way of putting it. In my experience, signals in Unix mostly do what they're meant to do quite reliably -- it's just a matter of understanding what they're meant to do. There may be bugs in certain systems that cause signals to get lost under obscure circumstances, but that's no reason for Python to make the situation worse by introducing bugs of its own. Two related signals received between two 'checkpoints' (i.e. when the signal is tested and cleared). You may only get one of them, and 'related' does not mean 'the same'. In the case where they're the same, I wouldn't say that the second signal has been lost. Rather, it's simply redundant -- a call to the handler is already pending, and will happen eventually. (If you're expecting one call per signal, then you're using the wrong mental model for signals.) I wasn't aware that this could happen between different signals. If it can, there must be some rationale as to why the second signal is considered redundant. Otherwise there's a bug in either the design or the implementation. A second signal received while the first is being 'handled' by the operating system or language run-time system. That one sounds odd to me. I would expect a signal received during the execution of a handler to be flagged and cause the handler to be called again after it returns. But then I'm used to the BSD signal model, which is relatively sane. A signal sent while the operating system is doing certain things to the application (including, sometimes, when it is swapped out or deep in I/O.) That sounds like an outright bug. I can't think of any earthly reason why the handler shouldn't be called eventually, if it remains installed and the process lives long enough. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Object creation hook
Kristján V. Jónsson wrote: (I was a bit dismayed that I couldn't assign to object.__init__ post-hoc from a python script, I'm fairly sure that is possible in Ruby :) It wouldn't do you much good anyway, because no existing subclass of object having its own __init__ method would call it. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Object creation hook
Short of using a memory debugger such as Valgrind or Purify, have you
considered looking for reference leaks? These may be the cause and
can be checked with pure python code. See how Lib/test/regrtest.py
handles the -R option.
n
--
On 1/24/07, Kristján V. Jónsson [EMAIL PROTECTED] wrote:
Thanks, but the question is really, how do I build a better debug hook than
sys.getobjects? so I argue this is a valid python-dev question.
We have been using gc.get_objects() but it has several problems:
1) It returns all objects in the system. This results in a list so long that
it often kills the system. Our system is of a scale that makes this very
risky.
2) There is no way to frame certain operations and get just those objects
that were created during their execution. In our case, we would like to get
the server cluster running, then frame a certain operation to get a callback
for all created objects, so that we could track that they were later
destroyed correctly. I have done this previously by storing the id()s of all
objects returned from gc.get_objects() and comparing them before and
after but this suffers from 1) above, and the ambiguity of id() in the face
of objects being created and destroyed.
Working with the metaclasses sounds reasonable if one has the luxury of
changing the entire codebase to use a different metaclass. It also doesn't
work with old style classes (plenty of those), c extensions, and builtins.
(I was a bit dismayed that I couldn't assign to object.__init__ post-hoc from
a python script, I'm fairly sure that is possible in Ruby :) but I
digress...)
My latest attempt was to modify _PyObject_GC_TRACK(o) in objimpl.h, adding
this final line:
if (PyCCP_CreationHook) PyCCP_CreationHookFunc(o);\
The function then looks like this:
void PyCCP_CreationHookFunc(PyObject * obj)
{
if (PyCCP_CreationHook) {
PyObject *result, *tmp = PyCCP_CreationHook;
PyCCP_CreationHook = 0; //guard against recursion
result = PyObject_CallFunctionObjArgs(PyCCP_CreationHook,
obj, 0);
Py_XDECREF(result);
if (!result)
PyErr_Clear();
PyCCP_CreationHook = tmp;
}
}
Doing this, and setting a no-op function as as the PyCCP_CreationHook, does
seem to work for a while in the interactive python shell, but it then crashes
and I haven't been able to work out the crash. In any case, doing stuff at
the point of GC list insertion is very hairy, especially in the face of
__del__ methods and such (of which we don't have many)
I am waiting to get Rational Purify set up on my machine and then I'll maybe
be able to understand the crash case better.
Cheers,
Kristján
-Original Message-
From: Martin v. Löwis [mailto:[EMAIL PROTECTED]
Sent: 23. janúar 2007 23:32
To: Kristján V. Jónsson
Cc: 'python-dev@python.org'
Subject: Re: [Python-Dev] Object creation hook
Kristján V. Jónsson schrieb:
I am trying to insert a hook into python enabling a callback for all
just-created objects. The intention is to debug and find memory leaks,
e.g. by having the hook function insert the object into a WeakKeyDictionary.
I'd like to point out that this isn't a python-dev question, but more
appropriate for comp.lang.python (as it is of the how do I x with
Python? kind).
I would use a debug build, and use sys.getobjects to determine all
objects and find leaks.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Weekly Python Patch/Bug Summary
Patch / Bug Summary ___ Patches : 421 open ( -2) / 3549 closed (+10) / 3970 total ( +8) Bugs: 943 open (-17) / 6471 closed (+25) / 7414 total ( +8) RFE : 260 open ( +2) / 250 closed ( +1) / 510 total ( +3) New / Reopened Patches __ rlcompleter tab completion in pdb (2007-01-22) http://python.org/sf/1641544 opened by Stephen Emslie logging leaks loggers (2007-01-22) CLOSED http://python.org/sf/1641790 opened by TH Fix error/crash in AST: syntaxerror in complex ifs (2007-01-23) http://python.org/sf/1642547 opened by Thomas Wouters comments to clarify complexobject.c (2007-01-23) http://python.org/sf/1642844 opened by Jim Jewett Fix Bug 1362475 Text.edit_modified() doesn't work (2007-01-24) http://python.org/sf/1643641 opened by Matthias Kievernagel ctypes leaks memory (2007-01-24) CLOSED http://python.org/sf/1643874 opened by Thomas Heller file - open in stdlib (2007-01-25) http://python.org/sf/1644218 opened by Daniel Nogradi Allow importing built-in submodules (2007-01-25) http://python.org/sf/1644818 opened by Miguel Lobo Patches Closed __ Fix for #1601399 (urllib2 does not close sockets properly) (2007-01-03) http://python.org/sf/1627441 closed by gbrandl C99 _Bool support for struct (2006-12-07) http://python.org/sf/1610575 closed by loewis logging leaks loggers (2007-01-22) http://python.org/sf/1641790 closed by vsajip Patch for #1586414 to avoid fragmentation on Windows (2006-10-31) http://python.org/sf/1587674 closed by gustaebel smtplib email renames (2007-01-16) http://python.org/sf/1637162 closed by gbrandl urllib2: email.Utils-email.utils (2007-01-16) http://python.org/sf/1637159 closed by gbrandl urllib: change email.Utils - email.utils (2007-01-16) http://python.org/sf/1637157 closed by gbrandl tarfile extraction does not honor umask (2006-06-16) http://python.org/sf/1507247 closed by gustaebel Fix crash when replacing sys.stdout in sitecustomize (2007-01-08) http://python.org/sf/1630975 closed by twouters ctypes leaks memory (2007-01-24) http://python.org/sf/1643874 closed by theller New / Reopened Bugs ___ 2.3.6.4 Error in append and extend descriptions (2007-01-21) CLOSED http://python.org/sf/1641109 opened by ilalopoulos Python 2.5 gets curses.h warning on HPUX (2007-01-22) http://python.org/sf/1642054 opened by Roy Smith Grammatical Error (2007-01-23) CLOSED http://python.org/sf/1643150 opened by Steve Miller function breakpoints in pdb (2007-01-24) http://python.org/sf/1643369 opened by decitre Emphasize buffering issues when sys.stdin is used (2007-01-24) http://python.org/sf/1643712 opened by Raghuram Devarakonda Problem with signals in a single-threaded application (2007-01-24) http://python.org/sf/1643738 opened by Ulisses Furquim strptime %U broken (2007-01-24) CLOSED http://python.org/sf/1643943 opened by Brian Nahas Error arrow offset wrong (2007-01-25) CLOSED http://python.org/sf/1644239 opened by Cees Timmerman ./configure --prefix=/ breaks, won't build C modules (2007-01-25) http://python.org/sf/1644987 opened by Jim Shankland MIME renderer: wrong header line break with long subject? (2007-01-26) http://python.org/sf/1645148 opened by kxroberto Bugs Closed ___ Newline skipped in for line in file (2007-01-16) http://python.org/sf/1636950 closed by bcannon Over-zealous keyword-arguments check for built-in set class (2006-05-11) http://python.org/sf/1486663 closed by gbrandl urllib2 does not close sockets properly (2006-11-22) http://python.org/sf/1601399 closed by gbrandl subprocess: error redirecting i/o from non-console process (2006-11-27) http://python.org/sf/1603907 closed by astrand Problem running a subprocess (2007-01-13) http://python.org/sf/1634739 closed by astrand subprocess.py: O(N**2) bottleneck (2006-11-17) http://python.org/sf/1598181 closed by astrand 2.3.6.4 Error in append and extend descriptions (2007-01-21) http://python.org/sf/1641109 closed by gbrandl asyncore.py and quot;handle_exptquot; (2002-12-16) http://python.org/sf/654766 closed by sf-robot Segfault provoked by generators and exceptions (2006-10-18) http://python.org/sf/1579370 closed by loewis gen_iternext: Assertion `f-f_back != ((void *)0)' failed (2006-05-06) http://python.org/sf/1483133 closed by nnorwitz Python polls unnecessarily every 0.1 second when interactive (2006-09-05) http://python.org/sf/1552726 closed by akuchling subprocess interpreted double quotation wrong on windows (2006-03-09) http://python.org/sf/1446119 closed by astrand tarfile.extract() may cause file fragmentation on Windows XP
