[issue1602] windows console doesn't print or input Unicode

2012-05-19 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Giampaolo: See #msg120700 for why that won't work, and the subsequent comments for what will work instead (basically, using WriteConsoleW and a workaround for a Windows API bug). Also see the prototype win_console.patch from

[issue1602] windows console doesn't print or input Unicode

2011-03-26 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Glenn wrote: So if flush checks that bit, maybe TextIOWriter could just call buffer.flush, and it would be fast if clean and slow if dirty? Yes. I'll benchmark how much overhead is added by the calls to flush; there's no point

[issue11395] print(s) fails on Windows with long strings

2011-03-26 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: If I understand the bug in the Windows console functions correctly, a limit of 32767 bytes might not always be small enough. The problem is that if two or more threads are concurrently using any console functions (which all use

[issue1602] windows console doesn't print or input Unicode

2011-03-25 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: First a minor correction: The new requirement would be that a correct app also needs to flush between a sequence of buffer.writes (that end in an incomplete line, or always if PYTHONUNBUFFERED or python -u is used

[issue1602] windows console doesn't print or input Unicode

2011-03-25 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: I wrote: A similar issue arises for stdin: to maintain strict compatibility, every read from a TextIOWrapper attached to an input console would have to drain the buffer of its buffer object, in case the app has read from

[issue1602] windows console doesn't print or input Unicode

2011-03-24 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: I wrote: The only caveat would be that if you write a partial line to the buffer object (or if you set the buffer object to be fully buffered and write to it), and then write to the text stream, the buffer wouldn't be flushed

[issue1602] windows console doesn't print or input Unicode

2011-03-24 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: I wrote: $ python3 -c 'import sys; sys.stdout.write(foo); sys.stdout.buffer.write(bbar); sys.stdout.write(baz\n)' barfoobaz Hmm, the behaviour actually would differ here: the proposed implementation would print foobaz bar

[issue1602] windows console doesn't print or input Unicode

2011-03-24 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Glenn Linderman wrote: Presently, a correct application only needs to flush between a sequence of writes and a sequence of buffer.writes. Right. The new requirement would be that a correct app also needs to flush between

[issue1602] windows console doesn't print or input Unicode

2011-03-22 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: (For anyone wondering about the hold-up on this bug, I ended up switching to Ubuntu. Not to worry, I now have Python 3 building in XP under VirtualBox -- which is further than I ever got with my broken Vista install :-/ It seems

[issue1602] windows console doesn't print or input Unicode

2011-02-02 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Feedback from Julie Solon of Microsoft: These console functions share a per-process heap that is 64K. There is some overhead, the heap can get fragmented, and calls from multiple threads all affect how much is available

[issue1602] windows console doesn't print utf8 (Py30a2)

2011-01-11 Thread David-Sarah Hopwood
Changes by David-Sarah Hopwood david-sa...@jacaranda.org: -- nosy: +BreamoreBoy versions: +Python 3.1, Python 3.2 -Python 3.3 Added file: http://bugs.python.org/file20360/doc-patch.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org

[issue1602] windows console doesn't print utf8 (Py30a2)

2011-01-11 Thread David-Sarah Hopwood
Changes by David-Sarah Hopwood david-sa...@jacaranda.org: -- nosy: +BreamoreBoy versions: +Python 3.1, Python 3.2 -Python 3.3 Added file: http://bugs.python.org/file20361/doc-patch.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org

[issue1602] windows console doesn't print utf8 (Py30a2)

2011-01-11 Thread David-Sarah Hopwood
Changes by David-Sarah Hopwood david-sa...@jacaranda.org: Added file: http://bugs.python.org/file20362/doc-patch.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1602

[issue1602] windows console doesn't print or input Unicode

2011-01-11 Thread David-Sarah Hopwood
Changes by David-Sarah Hopwood david-sa...@jacaranda.org: -- title: windows console doesn't print utf8 (Py30a2) - windows console doesn't print or input Unicode Added file: http://bugs.python.org/file20363/doc-patch.diff ___ Python tracker rep

[issue1602] windows console doesn't print utf8 (Py30a2)

2011-01-10 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: I'll have a look at the Py3k I/O internals and see what I can do. (Reopening a bug appears to need Coordinator permissions.) -- ___ Python tracker rep...@bugs.python.org http

[issue1602] windows console doesn't print utf8 (Py30a2)

2011-01-10 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: The script unicode2.py uses the console STD_OUTPUT_HANDLE iff sys.stdout.fileno()==1. You may have missed if not_a_console(hStdout): real_stdout = False. not_a_console uses GetFileType and GetConsoleMode to check whether

[issue1602] windows console doesn't print utf8 (Py30a2)

2011-01-10 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Note: Michael Kaplan's code checks whether GetConsoleMode failed due to ERROR_INVALID_HANDLE. My code intentionally doesn't do that, because it is correct and conservative to fall back to the non-console behaviour when

[issue5924] When setting complete PYTHONPATH on Python 3.x, paths in the PYTHONPATH are ignored

2011-01-10 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Looking at http://svn.python.org/view/python/branches/py3k/PC/getpathp.c?r1=73322r2=73321pathrev=73322 , wouldn't it be better to add a Py_WGETENV function? There are likely to be other cases where that would be the correct

[issue1602] windows console doesn't print utf8 (Py30a2)

2011-01-10 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: ... os.dup2() ... Good point, thanks. It would work to change os.dup2 so that if its second argument is 0, 1, or 2, it calls _get_osfhandle to get the Windows handle for that fd, and then reruns the console-detection logic

[issue1602] windows console doesn't print utf8 (Py30a2)

2011-01-09 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: haypo wrote: davidsarah wrote: It is certainly possible to write Unicode to the console successfully using WriteConsoleW Did you tried with characters not encodable to the code page and with character that cannot

[issue410547] os.statvfs support for Windows

2011-01-09 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Don't use win32file.GetDiskFreeSpace; the underlying Windows API only supports drives up to 2 GB (http://blogs.msdn.com/b/oldnewthing/archive/2007/11/01/5807020.aspx). Use GetFreeDiskSpaceEx, as the code I linked to does. I'm

[issue1602] windows console doesn't print utf8 (Py30a2)

2011-01-08 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: It is certainly possible to write Unicode to the console successfully using WriteConsoleW. This works regardless of the console code page, including 65001. The code a href=http://tahoe-lafs.org/trac/tahoe-lafs/browser/src

[issue1602] windows console doesn't print utf8 (Py30a2)

2011-01-08 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Glenn Linderman wrote: I skipped the unmangling of command-line arguments, because it produced an error I didn't understand, about needing a buffer protocol. If I understand correctly, that part isn't needed on Python 3 because

[issue2128] sys.argv is wrong for unicode strings

2011-01-08 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: The following code is being used to work around this issue for Python 2.x in Tahoe-LAFS: # This works around http://bugs.python.org/issue2128. GetCommandLineW = WINFUNCTYPE(LPWSTR)((GetCommandLineW, windll.kernel32

[issue2128] sys.argv is wrong for unicode strings

2011-01-08 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Sorry, missed out the imports: from ctypes import WINFUNCTYPE, windll, POINTER, byref, c_int from ctypes.wintypes import LPWSTR, LPCWSTR -- ___ Python tracker rep

[issue410547] os.statvfs support for Windows

2011-01-06 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Is there a portable way to get the available disk space by now? No, but http://tahoe-lafs.org/trac/tahoe-lafs/browser/src/allmydata/util/fileutil.py?rev=4894#L308 might be helpful (uses pywin32). -- nosy: +davidsarah

[issue6058] Add cp65001 to encodings/aliases.py

2010-10-23 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: This problem causes {{{os.getcwdu()}}} to fail when the console code page is set to 65001 (always, I think): {{{ t:\ver Microsoft Windows [Version 6.0.6002] t:\chcp Active code page: 65001 t:\python -c import os; print

[issue6058] Add cp65001 to encodings/aliases.py

2010-10-23 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: I said: There is only one correct way to encode/decode UTF-8. This is true modulo differences in the treatment of initial byte order marks. -- ___ Python tracker rep...@bugs.python.org

[issue6058] Add cp65001 to encodings/aliases.py

2010-10-23 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: I meant to say that the os.getcwdu() test in msg119440 was done with Windows native Python 2.6.2. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6058

[issue6058] Add cp65001 to encodings/aliases.py

2010-10-23 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Oops, false alarm. python -c import os; print repr(os.getcwdu()) works as expected, so the exception is part of issue 1602. (My command about there being no need to distinguish this codepage from UTF-8 stands

[issue7952] fileobject.c can switch between fread and fwrite without an intervening flush or seek, invoking undefined behaviour

2010-02-17 Thread David-Sarah Hopwood
New submission from David-Sarah Hopwood david-sa...@jacaranda.org: The C standard (any version, or POSIX), says in the description of fopen that: {{{ When a file is opened with update mode ( '+' as the second or third character in the mode argument), both input and output may be performed

[issue7952] fileobject.c can switch between fread and fwrite without an intervening flush or seek, invoking undefined behaviour

2010-02-17 Thread David-Sarah Hopwood
David-Sarah Hopwood david-sa...@jacaranda.org added the comment: Correction: when input is followed by output, the call needed to avoid undefined behaviour has to be to a file positioning function (fseek, fsetpos, or rewind, but not fflush). Since fileobject.c does not use wide I/O operations