Re: [IronPython] [python] Re: Import decimal failure with Turkish Locale (IP 1.1)
On 10/31/07, Michael Foord <[EMAIL PROTECTED]> wrote: > > > However - string literals in code should either be in ascii *or* in a > declared encoding. Some way out of this situation would be nice... > (Thinking subclasses of strings...) The only place I know of that follows this kind of approach is SQL Server, where all char fields are associated with a specific sort order. The original code in Decimal is pretty crazy -- even without the newly recognized dangers of the "upper" function. To the best of my knowledge, Turkish is the only locale in which an unaccented latin character has an upper-case value that's different from what its uppercase value would be in en-US. -- Curt Hagenlocher [EMAIL PROTECTED] ___ Users mailing list [email protected] http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] [python] Re: Import decimal failure with Turkish Locale (IP 1.1)
Cool, I'll send out an update after I find out more. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michael Foord Sent: Thursday, November 01, 2007 2:55 PM To: Discussion of IronPython Subject: Re: [IronPython] [python] Re: Import decimal failure with Turkish Locale (IP 1.1) Dino Viehland wrote: > One option might be a non-technical solution: Instead of you redistributing > the library (or modified library) we distribute it w/ IronPython - and then > you're just including the combined package. There's other reasons why it'd > be good for us to do this (help, encodings, warnings, etc...). Unfortunately > there's some internal process required before we cross that hurdle that we've > simply been putting off. > > I'm kicking off the discussions that are necessary to see if we can get this > going. It might take a couple of weeks (or who knows, maybe longer) so a > little patience might be required. Anyway, let me know what you think of > that solution. > Having a 'patched where necessary' distribution of the standard library come with IronPython would be *great*. +1000 to that idea. :-) Michael http://www.manning.com/foord > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michael Foord > Sent: Wednesday, October 31, 2007 5:26 PM > To: Discussion of IronPython > Subject: Re: [IronPython] [python] Re: Import decimal failure with Turkish > Locale (IP 1.1) > > Hello Dino (etc), > > Hmm... as usual it is a harder problem than it first seems. > > Curt's suggestion is helpful (and will probably get us out of a fix - so > thanks), but patching the standard library (which we do distribute > separately) is something we had hoped to avoid. To be honest that code > in 'decimal.py' seems odd and a weird way of achieving what it does. > > However - string literals in code should either be in ascii *or* in a > declared encoding. Some way out of this situation would be nice... > (Thinking subclasses of strings...) > > Michael > http://www.manning.com/foord > > Dino Viehland wrote: > >> This would have been my suggestion - this is a tough issue - do you have an >> expected behavior you'd like to see? I can imagine a few: >> 1. IronPython supports both byte-array strings and normal strings, >> and therefore "it just works" >> 2. IronPython never uses the current locale for string operations, >> effectively breaking Unicode >> 3. IronPython does a conversion into ASCII before doing attribute >> access, and somehow converts the upper case turkish I into an ASCII turkish >> I. >> >> Those 3 options are problematic at best. An easier solution would be what >> Curt suggested (I'm assuming you're redistributing the standard library >> yourself). I was also thinking that there should be a preexisting way to do >> this in Python but I don't see an upper function that lets you specify the >> locale or force it to do so against an invariance culture. Maybe just a >> call to string.translate instead? >> >> >> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Curt >> Hagenlocher >> Sent: Wednesday, October 31, 2007 11:15 AM >> To: Discussion of IronPython >> Subject: Re: [IronPython] Import decimal failure with Turkish Locale (IP 1.1) >> >> On 10/31/07, Michael Foord <[EMAIL PROTECTED]> wrote: >> >> The issue is because one of the method names is '_round_ceiling'. In the >> Turkish locale, the uppercase version of this is "ROUND_CEİLİNG" (in >> Turkish uppercase of "i" is "I" with a dot on top of it). Obviously the >> lookup of the corresponding global fails. >> >> In CPython the name is a byte-string, and so '.upper' just uses the >> ascii uppercase rather than the locale sensitive version - so we see >> failing to import 'decimal.py' as an IronPython bug. As this badly >> impacts Resolver we would *love* to see this fixed soon... >> >> Why not change decimal.py to use ToUpperInvariant()? Are you sharing the >> file with CPython? >> >> -- >> Curt Hagenlocher >> [EMAIL PROTECTED] >> ___ >> Users mailing list >> [email protected] >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> >> > > ___ > Users mailing list > [email protected] > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > ___ > Users mailing list > [email protected] > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > ___ Users mailing list [email protected] http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ Users mailing list [email protected] http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] [python] Re: Import decimal failure with Turkish Locale (IP 1.1)
Dino Viehland wrote: > One option might be a non-technical solution: Instead of you redistributing > the library (or modified library) we distribute it w/ IronPython - and then > you're just including the combined package. There's other reasons why it'd > be good for us to do this (help, encodings, warnings, etc...). Unfortunately > there's some internal process required before we cross that hurdle that we've > simply been putting off. > > I'm kicking off the discussions that are necessary to see if we can get this > going. It might take a couple of weeks (or who knows, maybe longer) so a > little patience might be required. Anyway, let me know what you think of > that solution. > Having a 'patched where necessary' distribution of the standard library come with IronPython would be *great*. +1000 to that idea. :-) Michael http://www.manning.com/foord > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michael Foord > Sent: Wednesday, October 31, 2007 5:26 PM > To: Discussion of IronPython > Subject: Re: [IronPython] [python] Re: Import decimal failure with Turkish > Locale (IP 1.1) > > Hello Dino (etc), > > Hmm... as usual it is a harder problem than it first seems. > > Curt's suggestion is helpful (and will probably get us out of a fix - so > thanks), but patching the standard library (which we do distribute > separately) is something we had hoped to avoid. To be honest that code > in 'decimal.py' seems odd and a weird way of achieving what it does. > > However - string literals in code should either be in ascii *or* in a > declared encoding. Some way out of this situation would be nice... > (Thinking subclasses of strings...) > > Michael > http://www.manning.com/foord > > Dino Viehland wrote: > >> This would have been my suggestion - this is a tough issue - do you have an >> expected behavior you'd like to see? I can imagine a few: >> 1. IronPython supports both byte-array strings and normal strings, >> and therefore "it just works" >> 2. IronPython never uses the current locale for string operations, >> effectively breaking Unicode >> 3. IronPython does a conversion into ASCII before doing attribute >> access, and somehow converts the upper case turkish I into an ASCII turkish >> I. >> >> Those 3 options are problematic at best. An easier solution would be what >> Curt suggested (I'm assuming you're redistributing the standard library >> yourself). I was also thinking that there should be a preexisting way to do >> this in Python but I don't see an upper function that lets you specify the >> locale or force it to do so against an invariance culture. Maybe just a >> call to string.translate instead? >> >> >> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Curt >> Hagenlocher >> Sent: Wednesday, October 31, 2007 11:15 AM >> To: Discussion of IronPython >> Subject: Re: [IronPython] Import decimal failure with Turkish Locale (IP 1.1) >> >> On 10/31/07, Michael Foord <[EMAIL PROTECTED]> wrote: >> >> The issue is because one of the method names is '_round_ceiling'. In the >> Turkish locale, the uppercase version of this is "ROUND_CEİLİNG" (in >> Turkish uppercase of "i" is "I" with a dot on top of it). Obviously the >> lookup of the corresponding global fails. >> >> In CPython the name is a byte-string, and so '.upper' just uses the >> ascii uppercase rather than the locale sensitive version - so we see >> failing to import 'decimal.py' as an IronPython bug. As this badly >> impacts Resolver we would *love* to see this fixed soon... >> >> Why not change decimal.py to use ToUpperInvariant()? Are you sharing the >> file with CPython? >> >> -- >> Curt Hagenlocher >> [EMAIL PROTECTED] >> ___ >> Users mailing list >> [email protected] >> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com >> >> >> > > ___ > Users mailing list > [email protected] > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > ___ > Users mailing list > [email protected] > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > ___ Users mailing list [email protected] http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] [python] Re: Import decimal failure with Turkish Locale (IP 1.1)
One option might be a non-technical solution: Instead of you redistributing the library (or modified library) we distribute it w/ IronPython - and then you're just including the combined package. There's other reasons why it'd be good for us to do this (help, encodings, warnings, etc...). Unfortunately there's some internal process required before we cross that hurdle that we've simply been putting off. I'm kicking off the discussions that are necessary to see if we can get this going. It might take a couple of weeks (or who knows, maybe longer) so a little patience might be required. Anyway, let me know what you think of that solution. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michael Foord Sent: Wednesday, October 31, 2007 5:26 PM To: Discussion of IronPython Subject: Re: [IronPython] [python] Re: Import decimal failure with Turkish Locale (IP 1.1) Hello Dino (etc), Hmm... as usual it is a harder problem than it first seems. Curt's suggestion is helpful (and will probably get us out of a fix - so thanks), but patching the standard library (which we do distribute separately) is something we had hoped to avoid. To be honest that code in 'decimal.py' seems odd and a weird way of achieving what it does. However - string literals in code should either be in ascii *or* in a declared encoding. Some way out of this situation would be nice... (Thinking subclasses of strings...) Michael http://www.manning.com/foord Dino Viehland wrote: > This would have been my suggestion - this is a tough issue - do you have an > expected behavior you'd like to see? I can imagine a few: > 1. IronPython supports both byte-array strings and normal strings, > and therefore "it just works" > 2. IronPython never uses the current locale for string operations, > effectively breaking Unicode > 3. IronPython does a conversion into ASCII before doing attribute > access, and somehow converts the upper case turkish I into an ASCII turkish I. > > Those 3 options are problematic at best. An easier solution would be what > Curt suggested (I'm assuming you're redistributing the standard library > yourself). I was also thinking that there should be a preexisting way to do > this in Python but I don't see an upper function that lets you specify the > locale or force it to do so against an invariance culture. Maybe just a call > to string.translate instead? > > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Curt > Hagenlocher > Sent: Wednesday, October 31, 2007 11:15 AM > To: Discussion of IronPython > Subject: Re: [IronPython] Import decimal failure with Turkish Locale (IP 1.1) > > On 10/31/07, Michael Foord <[EMAIL PROTECTED]> wrote: > > The issue is because one of the method names is '_round_ceiling'. In the > Turkish locale, the uppercase version of this is "ROUND_CEİLİNG" (in > Turkish uppercase of "i" is "I" with a dot on top of it). Obviously the > lookup of the corresponding global fails. > > In CPython the name is a byte-string, and so '.upper' just uses the > ascii uppercase rather than the locale sensitive version - so we see > failing to import 'decimal.py' as an IronPython bug. As this badly > impacts Resolver we would *love* to see this fixed soon... > > Why not change decimal.py to use ToUpperInvariant()? Are you sharing the > file with CPython? > > -- > Curt Hagenlocher > [EMAIL PROTECTED] > ___ > Users mailing list > [email protected] > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > ___ Users mailing list [email protected] http://lists.ironpython.com/listinfo.cgi/users-ironpython.com ___ Users mailing list [email protected] http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
Re: [IronPython] [python] Re: Import decimal failure with Turkish Locale (IP 1.1)
Hello Dino (etc), Hmm... as usual it is a harder problem than it first seems. Curt's suggestion is helpful (and will probably get us out of a fix - so thanks), but patching the standard library (which we do distribute separately) is something we had hoped to avoid. To be honest that code in 'decimal.py' seems odd and a weird way of achieving what it does. However - string literals in code should either be in ascii *or* in a declared encoding. Some way out of this situation would be nice... (Thinking subclasses of strings...) Michael http://www.manning.com/foord Dino Viehland wrote: > This would have been my suggestion - this is a tough issue - do you have an > expected behavior you'd like to see? I can imagine a few: > 1. IronPython supports both byte-array strings and normal strings, > and therefore "it just works" > 2. IronPython never uses the current locale for string operations, > effectively breaking Unicode > 3. IronPython does a conversion into ASCII before doing attribute > access, and somehow converts the upper case turkish I into an ASCII turkish I. > > Those 3 options are problematic at best. An easier solution would be what > Curt suggested (I'm assuming you're redistributing the standard library > yourself). I was also thinking that there should be a preexisting way to do > this in Python but I don't see an upper function that lets you specify the > locale or force it to do so against an invariance culture. Maybe just a call > to string.translate instead? > > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Curt > Hagenlocher > Sent: Wednesday, October 31, 2007 11:15 AM > To: Discussion of IronPython > Subject: Re: [IronPython] Import decimal failure with Turkish Locale (IP 1.1) > > On 10/31/07, Michael Foord <[EMAIL PROTECTED]> wrote: > > The issue is because one of the method names is '_round_ceiling'. In the > Turkish locale, the uppercase version of this is "ROUND_CEİLİNG" (in > Turkish uppercase of "i" is "I" with a dot on top of it). Obviously the > lookup of the corresponding global fails. > > In CPython the name is a byte-string, and so '.upper' just uses the > ascii uppercase rather than the locale sensitive version - so we see > failing to import 'decimal.py' as an IronPython bug. As this badly > impacts Resolver we would *love* to see this fixed soon... > > Why not change decimal.py to use ToUpperInvariant()? Are you sharing the > file with CPython? > > -- > Curt Hagenlocher > [EMAIL PROTECTED] > ___ > Users mailing list > [email protected] > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > ___ Users mailing list [email protected] http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
