[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-12 Thread Clemens Ladisch
Olivier Mascia wrote: > it looks evident that the command-line tool of sqlite3 is meant to be > compiled as a narrow I/O program. It uses the standard main() and not > wmain(), ... and does not attempt to use the wide chars in or out > channels. That's just because UTF-8 is the most portable

[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-12 Thread Dominique Devienne
On Fri, Feb 12, 2016 at 11:36 AM, Olivier Mascia wrote: > > So it would be a bad idea to change sqlite3's output depending on the > > current code page or font. > > I share your view on keeping things as unicode as possible, but the > frontier is thin before trying to impose our views onto

[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-12 Thread Olivier Mascia
Clemens, > Le 12 f?vr. 2016 ? 06:43, Clemens Ladisch a ?crit : > >> Convert from UTF8 to the codepage returned by GetConsoleOutputCP(). > > This would corrupt any characters not in that code page. > > When we need to handle Windows console I/O specially, we could just as > well use _O_U8TEXT,

[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-12 Thread Clemens Ladisch
Olivier Mascia wrote: >> Le 11 f?vr. 2016 ? 23:13, Clemens Ladisch a ?crit : >> - stdin from the console > > Convert from the codepage returned by GetConsoleCP() to UTF8. >> - stdout to the console > > Convert from UTF8 to the codepage returned by GetConsoleOutputCP(). This would corrupt any

[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-12 Thread Olivier Mascia
> Le 11 f?vr. 2016 ? 23:13, Clemens Ladisch a ?crit : > > As far as I can see, there are five problems: > - stdin from the console Convert from the codepage returned by GetConsoleCP() to UTF8. > - stdin redirected from a file Personal opinion: I'd like it to treat input as implicitly in UTF8

[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-11 Thread Clemens Ladisch
Richard Hipp wrote: > If you can provide solid suggestions for how to deal with the > unicode issue on windows consoles within the next day or so, we might > be able to incorporate changes into 3.11.0. But beyond that, I think > we'll need to push this off until 3.12.0. > > If y'all think that

[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-11 Thread Olivier Mascia
> Le 11 f?vr. 2016 ? 17:33, Olivier Mascia a ?crit : > > In between if any of you want to experiment with the path I followed, up to > now, here is a quick and dirty patch made on the amalgamation code from > https://www.sqlite.org/snapshot/sqlite-snapshot-201602101910.tar.gz Inline patch

[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-11 Thread Olivier Mascia
> Le 11 f?vr. 2016 ? 16:46, Olivier Mascia a ?crit : > > 3.12.0 is good target, imo. > The way I patched it works nicely for the interactivity, but more work needs > to be done for handling the command-line itself properly (it always is ANSI - > or Wide and not OEM-something or UTF-8). In

[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-11 Thread Dominique Devienne
On Thu, Feb 11, 2016 at 5:07 PM, Stephan Beal wrote: > You might want to look at: > > https://www.fossil-scm.org/index.html/finfo?name=src/utf8.c Interesting. Thanks for sharing. fossil_utf8_to_console() does seem to be WriteConsoleW-based, when the http://www.siao2.com

[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-11 Thread Stephan Beal
On Thu, Feb 11, 2016 at 4:46 PM, Olivier Mascia wrote: > The way I patched it works nicely for the interactivity, but more work > needs to be done for handling the command-line itself properly (it always > is ANSI - or Wide and not OEM-something or UTF-8). In addition my code will > need deep

[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-11 Thread Olivier Mascia
> Le 11 f?vr. 2016 ? 16:00, Richard Hipp a ?crit : > > Olivier, Dominique, Clemens: I *really* appreciate y'all looking into > this. Thank you so much. > > Please know that I am currently holding up the start of final release > testing for 3.11.0 on this issue. But I cannot hold it off much

[sqlite] Unicode issue on windows consoles. Was: Version 3.11.0 beta

2016-02-11 Thread Richard Hipp
On 2/11/16, Clemens Ladisch wrote: > Dominique Devienne wrote: >> On Thu, Feb 11, 2016 at 1:10 PM, Olivier Mascia wrote: >>> >>> If I'm changing shell.c (utf8_printf) to use some >>> sqlite3_win32_utf8_to_oem() (which is the same as >>> sqlite3_win32_utf8_to_mbcs, but always use CP_OEMCP) AND