On Thu, 2010-01-21 at 00:06 +0100, "Martin v. Löwis" wrote: > > Why only set an encoding on these streams when they're directly > > connected to a tty? > > If you are sending data to the terminal, you can be fairly certain > that the locale's encoding should be used. It's a convenience feature > for the interactive mode, so that Unicode strings print correctly. > > When sending data to a pipe or to a file, God knows what encoding > should have been used. If it's any XML file (for example), using the > locale's encoding would be incorrect, and the encoding declared > in the XML declaration should be used (or UTF-8 if no declaration > is included). If it's a HTTP socket, it really should be restricted > to ASCII in the headers, and then to the content-type. And so on. > > So in general, the applications should arrange to the the encoding > or encode themselves when they write to some output stream. If they > fail to do so, it's a bug in the application, not in Python. > > > I'll patch things to remove the isatty conditional if that's acceptable. > > It will make your Python release incompatible with everybody else's, > and will probably lead to moji-bake. Otherwise, it's fine.
Thanks (everyone) for your feedback It's clear that me unilaterally making this change is extremely unpopular, so I'm no longer planning to do so: maintaining consistency of behavior between different downstream distributions of CPython 2.* is the most important concern here. For reference I filed the tty patch as http://bugs.python.org/issue7745 (I don't seem to have rights to set it "closed->rejected" myself). One of my concerns here is the change of behavior between Python programs when run at a tty versus within a shell pipeline/cronjob/system daemon/etc, which I know many people find to be a gotcha; I know many developers who've been burned by this difference between development/deployment (myself included). I suspect I'm reinventing the wheel here, but one way of avoiding this "gotcha" is to set "PYTHONIOENCODING=ascii", to override the tty locale setting. Without this, these two cases have different behavior: [da...@brick ~]$ python -c 'print u"\u03b1\u03b2\u03b3"' αβγ [da...@brick ~]$ python -c 'print u"\u03b1\u03b2\u03b3"'|less Traceback (most recent call last): File "<string>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) With PYTHONIOENCODING=ascii, the two cases have the same behavior: [da...@brick ~]$ PYTHONIOENCODING=ascii python -c 'print u"\u03b1\u03b2 \u03b3"' Traceback (most recent call last): File "<string>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) [da...@brick ~]$ PYTHONIOENCODING=ascii python -c 'print u"\u03b1\u03b2 \u03b3"'|less Traceback (most recent call last): File "<string>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) (this is with my site.py reset back to the svn default) So I think there's a case for suggesting that developers set "PYTHONIOENCODING=ascii" in their environment, so ensure that attempts to write unicode to a std stream using defaults will fail immediately during the development cycle, rather than on deployment. (Though, alas, that will break the corresponding cases [1] for any python3 processes if they ever inherit that envvar). Hope this is helpful Dave [1] in that this works: [da...@brick ~]$ python3.1 -c 'print("\u03b1\u03b2\u03b3")' αβγ but this (naive and contrived) invocation of python3 from python2 fails: [da...@brick ~]$ PYTHONIOENCODING=ascii python2.6 -c "import os; os.system('python3.1 -c \'print(\"\u03b1\u03b2\u03b3\")\'')" Traceback (most recent call last): File "<string>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
