[issue10557] Malformed error message from float()

Mark Dickinson added the comment: Closing. -- status: pending -> closed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsub

[issue10557] Malformed error message from float()

Changes by Mark Dickinson : -- status: open -> pending ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://m

[issue10557] Malformed error message from float()

Mark Dickinson added the comment: Sure, this can be closed as far as I'm concerned. -- status: pending -> open ___ Python tracker ___

[issue10557] Malformed error message from float()

Changes by Serhiy Storchaka : -- status: open -> pending ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:/

[issue10557] Malformed error message from float()

Serhiy Storchaka added the comment: Can this issue be closed? -- nosy: +serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list m

[issue10557] Malformed error message from float()

Changes by STINNER Victor : -- versions: +Python 3.3 -Python 3.1 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue10557] Malformed error message from float()

Changes by Alexander Belopolsky : -- versions: +Python 3.1 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Sat, Dec 4, 2010 at 3:11 PM, Mark Dickinson wrote: > > Mark Dickinson added the comment: >.. One issue is that we'd still need the char* -> double operations, partly >because > PyOS_string_to_double is part of the public API, and partly to continue t

[issue10557] Malformed error message from float()

Mark Dickinson added the comment: > What do you think about adding number parsers that operate directly on > Py_UNICODE* strings? I think that might make some sense. It's not without difficulties, though. One issue is that we'd still need the char* -> double operations, partly because PyOS_

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Sat, Dec 4, 2010 at 6:03 AM, Mark Dickinson wrote: .. > I don't much like the extra boilerplate that's introduced (and repeated) > in longobject.c, floatobject.c and complexobject.c, though. > Yes, that's exactly what I meant when I called that code "

[issue10557] Malformed error message from float()

Mark Dickinson added the comment: Looks okay, I guess. I don't much like the extra boilerplate that's introduced (and repeated) in longobject.c, floatobject.c and complexobject.c, though. -- ___ Python tracker _

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: Committed in revision 87007. As a bug fix, this needs to be backported to 3.1, but PyUnicode_TransformDecimalToASCII() should probably be renamed to _PyUnicode_TransformDecimalToASCII() to avoid introduction of a new feature. -- resolution: ->

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: Hopefully this is the last iteration before commit. As discussed, I took whitespace processing out of PyUnicode_TransformDecimalToASCII() and made it public. Whitespace conversion in int()/float()/complex() is repetitious and can be optimized by, for

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Thu, Dec 2, 2010 at 9:53 PM, Alexander Belopolsky wrote: .. > .. The honest reason for the exclusion is that I gave up chasing a bug that > only shows > in full regrtest runs. I have realized where the problem was. PyUnicode_FromUnicode() "helpfully"

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Fri, Dec 3, 2010 at 4:45 AM, Marc-Andre Lemburg wrote: .. >> On Thu, Dec 2, 2010 at 4:34 PM, Marc-Andre Lemburg >> wrote: >> .. >>>  * Please change the API _PyUnicode_NormalizeDecimal() to >>>   PyUnicode_ConvertToASCIIDecimal() - that's closer to wh

[issue10557] Malformed error message from float()

Mark Dickinson added the comment: > According to comments in the code and verified by inspection, > PyOS_string_to_double does not accept any whitespace. Bah. You're right, of course. :-) Any whitespace (post PyUnicode_EncodeDecimal) is handled in PyFloat_FromString, using Py_ISSPACE.

[issue10557] Malformed error message from float()

Eric Smith added the comment: According to comments in the code and verified by inspection, PyOS_string_to_double does not accept any whitespace. -- ___ Python tracker ___

[issue10557] Malformed error message from float()

Mark Dickinson added the comment: > Are you sure ? I'm not sure how the underlying PyOS_string_to_double() > (IIRC) works. I believe it accepts ASCII whitespace (i.e., chars ' ', '\t', '\f', '\n', '\r', '\v'), and nothing else. -- ___ Python tracke

[issue10557] Malformed error message from float()

Changes by STINNER Victor : -- nosy: -haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue10557] Malformed error message from float()

Marc-Andre Lemburg added the comment: Alexander Belopolsky wrote: > > Alexander Belopolsky added the comment: > > On Thu, Dec 2, 2010 at 4:34 PM, Marc-Andre Lemburg > wrote: > .. >> * Please change the API _PyUnicode_NormalizeDecimal() to >> PyUnicode_ConvertToASCIIDecimal() - that's clos

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: I am submitting a new patch that excludes int() changes. The honest reason for the exclusion is that I gave up chasing a bug that only shows in full regrtest runs. (Marc, I don't think it is related to what you thought was a missing norm decref: that

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Thu, Dec 2, 2010 at 6:32 PM, Stefan Krah wrote: .. > I like the public name. Extension authors can use it and be sure that > their programs accept exactly the same numeric strings as the rest of > Python. > > Are you worried that the semantics might ch

[issue10557] Malformed error message from float()

Stefan Krah added the comment: Alexander Belopolsky wrote: > On Thu, Dec 2, 2010 at 4:34 PM, Marc-Andre Lemburg > wrote: > .. > >  * Please change the API _PyUnicode_NormalizeDecimal() to > >   PyUnicode_ConvertToASCIIDecimal() - that's closer to what > >   it does. > > > > Are you sure it is

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Thu, Dec 2, 2010 at 4:34 PM, Marc-Andre Lemburg wrote: .. >  * Please change the API _PyUnicode_NormalizeDecimal() to >   PyUnicode_ConvertToASCIIDecimal() - that's closer to what >   it does. > Are you sure it is a good idea to give it a public name?

[issue10557] Malformed error message from float()

Marc-Andre Lemburg added the comment: Alexander Belopolsky wrote: > > Alexander Belopolsky added the comment: > > I am submitting a patch (issue10557b.diff) for commit review. As Marc > suggested, decimal conversion is now performed on Py_UNICODE characters. For > this purpose, I introduce

[issue10557] Malformed error message from float()

Mark Dickinson added the comment: Ah yes, you're right: this shouldn't be a hard fix. I withdraw my suggestion for a separate patch. :-) Checking that: complex('\xa0(\xa02+3j\xa0)\xa0') == complex(2.0, 3.0) would probably be enough. -- ___ Pyt

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Thu, Dec 2, 2010 at 1:28 PM, Alexander Belopolsky wrote: .. > Can you suggest a test case? I mean for complex(). -- ___ Python tracker

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Thu, Dec 2, 2010 at 12:54 PM, Mark Dickinson wrote: .. > The classes of whitespace accepted in each position are the same.  IIUC, with > your patch, > that consistency would be lost---is that right? Good point. I thought The PyUnicode_EncodeDecimal()

[issue10557] Malformed error message from float()

Mark Dickinson added the comment: Just to clarify: I'm not opposed to allowing arbitrary Unicode whitespace in the float, int, complex constructors (indeed, it's probably a good thing). But I'd like to see the change made consistently; for the complex constructor this looks a bit involved,

[issue10557] Malformed error message from float()

Mark Dickinson added the comment: Is the stripping of whitespace necessary for this fix? Currently, the complex constructor accepts whitespace both inside and outside the (optional) parentheses: >>> complex(' ( 2+3j ) ') (2+3j) The classes of whitespace accepted in each position are the same

[issue10557] Malformed error message from float()

Changes by Alexander Belopolsky : Added file: http://bugs.python.org/file19907/issue10557b.diff ___ Python tracker ___ ___ Python-bugs-list ma

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: I am submitting a patch (issue10557b.diff) for commit review. As Marc suggested, decimal conversion is now performed on Py_UNICODE characters. For this purpose, I introduced _PyUnicode_NormalizeDecimal() function that takes Py_UNICODE and returns a PyU

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Mon, Nov 29, 2010 at 4:41 AM, Marc-Andre Lemburg wrote: .. > It would be better to copy and iterate over the Unicode string first, > replacing any decimal code points with ASCII ones and then call the > UTF-8 encoder. > Good idea. > The code as it st

[issue10557] Malformed error message from float()

Marc-Andre Lemburg added the comment: Alexander Belopolsky wrote: > > Alexander Belopolsky added the comment: > > After a bit of svn archeology, it does appear that Arabic-Indic digits' > support was deliberate at least in the sense that the feature was tested for > when the code was first

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: After a bit of svn archeology, it does appear that Arabic-Indic digits' support was deliberate at least in the sense that the feature was tested for when the code was first committed. See r15000. The test migrated from file to file over the last 10 year

[issue10557] Malformed error message from float()

Changes by Alexander Belopolsky : -- Removed message: http://bugs.python.org/msg122725 ___ Python tracker ___ ___ Python-bugs-list mai

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: Sending this by e-mail was not a good idea ... On Sun, Nov 28, 2010 at 3:30 PM, Stefan Krah wrote: .. >> UnicodeEncodeError: 'decimal' codec can't encode character '\u066b' > > Hmm, looks like a bug? I think U+066B is correct. > Really? What about >>>

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Sun, Nov 28, 2010 at 3:30 PM, Stefan Krah wrote: .. >> UnicodeEncodeError: 'decimal' codec can't encode character '\u066b' > > Hmm, looks like a bug? I think U+066B is correct. > Really? What about Traceback (most recent call last): File "", line 1

[issue10557] Malformed error message from float()

Stefan Krah added the comment: > UnicodeEncodeError: 'decimal' codec can't encode character '\u066b' Hmm, looks like a bug? I think U+066B is correct. -- ___ Python tracker ___

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Sun, Nov 28, 2010 at 3:09 PM, Stefan Krah wrote: .. > Decimal point: U+066B Well, not so fast: Traceback (most recent call last): File "", line 1, in UnicodeEncodeError: 'decimal' codec can't encode character '\u066b' in position 1: invalid decima

[issue10557] Malformed error message from float()

Stefan Krah added the comment: > (Anyone knows whether Arabic numbers are written right to left or left > to right? What is the proper decimal point character?) Thousands separator: U+066C Decimal point: U+066B ١٢٣٬١٢٣٫١٢ should be: 123,123.12 Wikipedia says that digits are arranged in the

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: Issue #10567 demonstrated the problem of relying on the Unicode database in Python builtins. Apparently, Unicode does not guarantee stability of the character categories. On the other hand, we are already tied to UCD for the language definition. Maybe

[issue10557] Malformed error message from float()

Marc-Andre Lemburg added the comment: float('½') > Traceback (most recent call last): > File "", line 1, in > ValueError: could not convert string to float: � > float('42½') > Traceback (most recent call last): > File "", line 1, in > ValueError Note that fractional Unicode cod

[issue10557] Malformed error message from float()

Marc-Andre Lemburg added the comment: Mark Dickinson wrote: > > Mark Dickinson added the comment: > > About Alexander's solution: might it make more sense to have > PyUnicode_EncodeDecimal raise for inputs like this? I see it as > PyUnicode_EncodeDecimal's job to turn the unicode input in

[issue10557] Malformed error message from float()

Stefan Krah added the comment: > PyUnicode_EncodeDecimal() is either unnecessary or should be exposed > as a codec. I'm depending on PyUnicode_EncodeDecimal in cdecimal. In fact, it saved me quite a bit of trouble. I wouldn't be surprised if other extension writers use it as well. -- n

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Sun, Nov 28, 2010 at 12:00 PM, Mark Dickinson wrote: > > Mark Dickinson added the comment: > > About Alexander's solution:  might it make more sense to have > PyUnicode_EncodeDecimal raise > for inputs like this? No, I think PyOS_string_to_double()

[issue10557] Malformed error message from float()

Mark Dickinson added the comment: > Did your chocolate packaging use European digits or Arabic-Indic ones? > Note that they have different bidi properties: Good question; I think it was Arabic-Indic digits, but to be honest I don't remember. (It wasn't *all* that recently.) --

[issue10557] Malformed error message from float()

Alexander Belopolsky added the comment: On Sun, Nov 28, 2010 at 11:37 AM, Mark Dickinson wrote: > > Mark Dickinson added the comment: > >> I am not sure, whether support for non-ascii digits in float() >> constructor is worth maintaining. > > I'd be very happy to drop such support.  If you all

[issue10557] Malformed error message from float()

Mark Dickinson added the comment: About Alexander's solution: might it make more sense to have PyUnicode_EncodeDecimal raise for inputs like this? I see it as PyUnicode_EncodeDecimal's job to turn the unicode input into usable ASCII (or raise an exception); it looks like that's not happeni

[issue10557] Malformed error message from float()

Mark Dickinson added the comment: > I am not sure, whether support for non-ascii digits in float() > constructor is worth maintaining. I'd be very happy to drop such support. If you allow alternative digit sets, trying to work out exactly what should be supported and what shouldn't gets very

[issue10557] Malformed error message from float()

Changes by Eric Smith : -- nosy: +eric.smith ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue10557] Malformed error message from float()

Ezio Melotti added the comment: FWIW the UnicodeError comes from PyUnicode_EncodeDecimal (unicodeobject.c:6212) and the "ValueError: could not convert string to float" with the buggy � comes from PyOS_string_to_double (pystrtod.c:316). Maybe PyOS_string_to_double should be fixed to display t

[issue10557] Malformed error message from float()

Ezio Melotti added the comment: I think float() should support non-ascii digits but I agree that it would be better to avoid UnicodeErrors and convert them to ValueErrors so that >>> float('١٢٣٤.٥٦') 1234.56 and >>> float('½') Traceback (most recent call last): File "", line 1, in ValueEr

[issue10557] Malformed error message from float()

New submission from Alexander Belopolsky : >>> float('½') Traceback (most recent call last): File "", line 1, in ValueError: could not convert string to float: � >>> float('42½') Traceback (most recent call last): File "", line 1, in ValueError With the attached patch, float-error.diff