I declare this thread irreparably broken. Do not make any decisions in this thread. Tell me (in another thread) when it's time to decide and I will.
On Sat, Aug 23, 2014 at 8:27 PM, Nick Coghlan <ncogh...@gmail.com> wrote: > On 24 August 2014 04:37, Oleg Broytman <p...@phdru.name> wrote: > > On Sat, Aug 23, 2014 at 06:40:37PM +0100, Paul Moore < > p.f.mo...@gmail.com> wrote: > >> Generally, it seems to be mostly a reaction to the repeated claims > >> that Python, or Windows, or whatever, is "broken". > > > > Ah, if that's the only problem I certainly can live with that. My > > problem is that it *seems* this anti-Unix attitude infiltrates Python > > core development. I very much hope I'm wrong and it really isn't. > > The POSIX locale based approach to handling encodings is genuinely > broken - it's almost as broken as code pages are on Windows. The > fundamental flaw is that locales encourage *bilingual* computing: > handling English plus one other language correctly. Given a global > internet, bilingual computing *is a fundamentally broken approach*. We > need multilingual computing (any human language, all the time), and > that means Unicode. > > As some examples of where bilingual computing breaks down: > > * My NFS client and server may have different locale settings > * My FTP client and server may have different locale settings > * My SSH client and server may have different locale settings > * I save a file locally and send it to someone with a different locale > setting > * I attempt to access a Windows share from a Linux client (or vice-versa) > * I clone my POSIX hosted git or Mercurial repository on a Windows client > * I have to connect my Linux client to a Windows Active Directory > domain (or vice-versa) > * I have to interoperate between native code and JVM code > > The entire computing industry is currently struggling with this > monolingual (ASCII/Extended ASCII/EBCDIC/etc) -> bilingual (locale > encoding/code pages) -> multilingual (Unicode) transition. It's been > going on for decades, and it's still going to be quite some time > before we're done. > > The POSIX world is slowly clawing its way towards a multilingual model > that actually works: UTF-8 > Windows (including the CLR) and the JVM adopted a different > multilingual model, but still one that actually works: UTF-16-LE > > POSIX is hampered by legacy ASCII defaults in various subsystems (most > notably the default locale) and the assumption that system metadata is > "just bytes" (an assumption that breaks down as soon as you have to > hand that metadata over to another machine that may have different > locale settings) > Windows is hampered by the fact they kept the old 8-bit APIs around > for backwards compatibility purposes, so applications using those APIs > are still only bilingual (at best) rather than multilingual. > JVM and CLR applications will at least handle the Basic Multilingual > Plane (UCS-2) correctly, but may not correctly handle code points > beyond the 16-bit boundary (this is the "Python narrow builds don't > handle Unicode correctly" problem that was resolved for Python 3.3+ by > PEP 393) > > Individual users (including some organisations) may have the luxury of > saying "well, all my clients and all my servers are POSIX, so I don't > care about interoperability with other platforms". As the providers of > a cross-platform runtime environment, we don't have that luxury - we > need to figure out how to get *all* the major platforms playing nice > with each other, regardless of whether they chose UTF-8 or UTF-16-LE > as the basis for their approach towards providing multilingual > computing environments. > > Historically, that question of cross platform interoperability for > open source software has been handled in a few different ways: > > * Don't really interoperate with anybody, reinvent all the wheels (the JVM > way) > * Emulate POSIX on Windows (the Cygwin/MinGW way) > * Let the application developer figure it out (the Python 2 way) > > The first approach is inordinately expensive - it took the resources > of Sun in its heyday to make it possible, and it effectively locks the > JVM out of certain kinds of computing (e.g. it's hard to do array > oriented programming in JVM languages, because the CPU and GPU > vectorisation features aren't readily accessible). > > The second approach prevents the creation of truly native Windows > applications, which makes it uncompelling as a way of attracting > Windows users - it sends a clear signal that the project doesn't > *really* care about supporting Windows as a platform, but instead only > grudgingly accepts that there are Windows users out there that might > like to use their software. > > The third approach is the one we tried for a long time with Python 2, > and essentially found to be an "experts only" solution. Yes, you can > *make* it work, but the runtime isn't set up so it works *by default*. > > The Unicode changes in Python 3 are a result of the Python core > development team saying "it really shouldn't be this hard for > application developers to get cross-platform interoperability between > correctly configured systems when dealing solely with correctly > encoded data and metadata". The idea of Python 3 is that applications > should require additional complexity solely to deal with *incorrectly* > configured systems and improperly encoded data and metadata (and, > ideally, the detection of the need for such handling should be "Python > 3 threw an exception" rather than "something further down the line > detected corrupted data"). > > This is software rather than magic, though - these improvements only > happen through people actually knuckling down and solving the related > problems. When folks complain about Python 3's operating system > interface handling causing problems in some situations? They're almost > always referring to areas where we're still relying on the locale > system on POSIX or the code page system on Windows. Both of those > approaches are irredeemably broken - the answer is to stop relying on > them, but appropriately updating the affected subsystems generally > isn't a trivial task. A lot of the affected code runs before the > interpreter is fully initialised, which makes it really hard to test, > and a lot of it is incredibly convoluted due to various configuration > options and platform specific details, which makes it incredibly hard > to modify without breaking anything. > > One of those areas is the fact that we still use the old 8-bit APIs to > interact with the Windows console. Those are just as broken in a > multilingual world as the other Windows 8-bit APIs, so Drekin came up > with a project to expose the Windows console as a UTF-16-LE stream > that uses the 16-bit APIs instead: > https://pypi.python.org/pypi/win_unicode_console > > I personally hope we'll be able to get the issues Drekin references > there resolved for Python 3.5 - if other folks hope for the same > thing, then one of the best ways to help that happen is to try out the > win_unicode_console module and provide feedback on what does and > doesn't work. > > Another was getting exceptions attempting to write OS data to > sys.stdout when the locale settings had been scrubbed from the > environment. For Python 3.5, we better tolerate that situation by > setting "errors=surrogateescape" on sys.stdout when the environment > claims "ascii" as a suitable encoding for talking to the operating > system (this is our way of saying "we don't actually believe you, but > also don't have the data we need to overrule you completely"). > > While I was going to wait for more feedback from Fedora folks before > pushing the idea again, this thread also makes me think it would be > worth our while to add more tools for dealing with surrogate escapes > and latin-1 binary data smuggling just to help make those techniques > more discoverable and accessible: > http://bugs.python.org/issue18814#msg225791 > > These various discussions are also giving me plenty of motivation to > get back to working on PEP 432 (the rewrite of the interpreter startup > sequence) for Python 3.5. A lot of these things are just plain hard to > change because of the complexity of the current startup code. > Redesigning that to use a cleaner, multiphase startup sequence that > gets the core interpreter running *before* configuring the operating > system integration should give us several more options when it comes > to dealing with some of these challenges. > > Regards, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido)
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
