Re: [PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!)
Added to Win32 TODO:
o Fix MSVC NLS support, like for to_char()
http://archives.postgresql.org/pgsql-hackers/2008-02/msg00485.php
http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
---
Gevik Babakhani wrote:
Hereby a patch that fixes NLS support on PG 8.3 compiled with MSVC.
There problem:
NLS support does not work on PG 8.3 compiled with MSVC.
I encountered this bug when I was trying to show localized months and days
using TO_CHAR.
The main reason for this problem is because Gettext on Windows does not
respond to LC_MESSAGES environment variable. Changing this variable should
trigger Gettext to load a different messages catalog which unfortunately
does not work on Windows. Gettext uses the locale of the current thread in
execution to determine which message catalog should be loaded.
This is all discussed in:
http://archives.postgresql.org/pgsql-hackers/2008-02/msg00485.php
How it is fixed:
Changing the LC_MESSAGES is done in pg_locale.c::pg_perm_setlocale(int
category, const char *locale).
In order to force Gettext to load a messages catalog we have to call
WIN32API::SetThreadLocale(unsigned long locale_id)
(You probably see it coming)
Our input parameter for locale in pg_perm_setlocale is a string and there is
no unified way in Windows to translate a locale string to a locale_id for
SetThreadLocale to use. Therefore we will use a static table of values to
anticipate the locale name:
pg_win32_locale_database::{0x041b,{sk,sk-SK,sk_SK,Slovak_Slovakia}}
Given any of sk,sk-SK,sk_SK the 0x041b is returned for
SetThreadLocale as input parameter.
(We are not quite done yet)
Gettext, internally uses a hack to force itself to reload. This hidden
feature is also explained in GetText docs. By incrementing a Gettext
internal variable (_nl_msg_cat_cntr) we force Gettext to reload on the next
LC_MESSAGES-MSGID query.
Tests:
- Tested on Win XP MSVC (VC++ 8.0)
- Just a routine make check test on my Linux box
- MINGW is not tested. (I do not have the installation)
Fix notes:
- Gettext behaves oddly when the env. variable LANGUAGE is set before
starting PG
- The locale names in the static table are case sensitive. 'nl_NL' !=
'NL_NL'
- At this moment I only have included locale names we actually support in PG
installation.
- Where are the JP locale .po and .mo files? These are not in sources!
- Even though there are 20+ locales in PG installation, does not mean those
are complete. Try TMMonth,TMDay on nl_NL. You will get English names :)
(My fault, hee hee, I haven't completed nl.po translations yet.)
TODO:
- Provide/complete the day and month names for all supported locales.
- Create docs for supported locale names on Windows. (Values of the static
table)
Regards,
Gevik Babakhani
PostgreSQL NL http://www.postgresql.nl
TrueSoftware BV http://www.truesoftware.nl
[ Attachment, skipping... ]
[ Attachment, skipping... ]
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
--
Bruce Momjian [EMAIL PROTECTED]http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
--
Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-patches
[PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!)
Hereby a patch that fixes NLS support on PG 8.3 compiled with MSVC.
There problem:
NLS support does not work on PG 8.3 compiled with MSVC.
I encountered this bug when I was trying to show localized months and days
using TO_CHAR.
The main reason for this problem is because Gettext on Windows does not
respond to LC_MESSAGES environment variable. Changing this variable should
trigger Gettext to load a different messages catalog which unfortunately
does not work on Windows. Gettext uses the locale of the current thread in
execution to determine which message catalog should be loaded.
This is all discussed in:
http://archives.postgresql.org/pgsql-hackers/2008-02/msg00485.php
How it is fixed:
Changing the LC_MESSAGES is done in pg_locale.c::pg_perm_setlocale(int
category, const char *locale).
In order to force Gettext to load a messages catalog we have to call
WIN32API::SetThreadLocale(unsigned long locale_id)
(You probably see it coming)
Our input parameter for locale in pg_perm_setlocale is a string and there is
no unified way in Windows to translate a locale string to a locale_id for
SetThreadLocale to use. Therefore we will use a static table of values to
anticipate the locale name:
pg_win32_locale_database::{0x041b,{sk,sk-SK,sk_SK,Slovak_Slovakia}}
Given any of sk,sk-SK,sk_SK the 0x041b is returned for
SetThreadLocale as input parameter.
(We are not quite done yet)
Gettext, internally uses a hack to force itself to reload. This hidden
feature is also explained in GetText docs. By incrementing a Gettext
internal variable (_nl_msg_cat_cntr) we force Gettext to reload on the next
LC_MESSAGES-MSGID query.
Tests:
- Tested on Win XP MSVC (VC++ 8.0)
- Just a routine make check test on my Linux box
- MINGW is not tested. (I do not have the installation)
Fix notes:
- Gettext behaves oddly when the env. variable LANGUAGE is set before
starting PG
- The locale names in the static table are case sensitive. 'nl_NL' !=
'NL_NL'
- At this moment I only have included locale names we actually support in PG
installation.
- Where are the JP locale .po and .mo files? These are not in sources!
- Even though there are 20+ locales in PG installation, does not mean those
are complete. Try TMMonth,TMDay on nl_NL. You will get English names :)
(My fault, hee hee, I haven't completed nl.po translations yet.)
TODO:
- Provide/complete the day and month names for all supported locales.
- Create docs for supported locale names on Windows. (Values of the static
table)
Regards,
Gevik Babakhani
PostgreSQL NL http://www.postgresql.nl
TrueSoftware BV http://www.truesoftware.nl
1.pg_locale.c.patch
Description: Binary data
Welcome to psql 8.3.0, the PostgreSQL interactive terminal.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit
Warning: Console code page (437) differs from Windows code page (1252)
8-bit characters might not work correctly. See psql reference
page Notes for Windows users for details.
Gevik=# set lc_messages to 'sv_SE';
SET
Gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from
generate_series(0,6) as s(a);
dates
---
Torsdag Februari 2008
Fredag Februari 2008
L÷rdag Februari 2008
S÷ndag Februari 2008
MÏndag Februari 2008
Tisdag Februari 2008
Onsdag Februari 2008
(7 rows)
Gevik=# set lc_messages to 'de_DE';
SET
Gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from
generate_series(0,6) as s(a);
dates
-
Donnerstag Februar 2008
Freitag Februar 2008
Samstag Februar 2008
Sonntag Februar 2008
Montag Februar 2008
Dienstag Februar 2008
Mittwoch Februar 2008
(7 rows)
Gevik=# set lc_messages to 'English_United_States';
SET
Gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from
generate_series(0,6) as s(a);
dates
-
Thursday February 2008
Friday February 2008
Saturday February 2008
Sunday February 2008
Monday February 2008
Tuesday February 2008
Wednesday February 2008
(7 rows)
Gevik=# set lc_messages to 'tr-TR';
SET
Gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from
generate_series(0,6) as s(a);
dates
--
Persembe Subat 2008
Cuma Subat 2008
Cumartesi Subat 2008
Pazar Subat 2008
Pazartesi Subat 2008
Sali Subat 2008
âarsamba Subat 2008
(7 rows)
Gevik=# set lc_messages to 'fr';
SET
Gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from
generate_series(0,6) as s(a);
dates
---
Jeudi FÎvrier 2008
Vendredi FÎvrier 2008
Samedi FÎvrier 2008
Dimanche FÎvrier 2008
Lundi FÎvrier 2008
Mardi FÎvrier 2008
Mercredi FÎvrier 2008
Re: [PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!)
Gevik Babakhani wrote: Hereby a patch that fixes NLS support on PG 8.3 compiled with MSVC. Haven't looked into the details of the patch yet, will do so. But the first thing I notice - you say this is only for MSVC, right? But the patch will also change the behaviour for the mingw build. Since you say you haven't tested on it, does the documentation imply that it would work on mingw, or is this likely to break that build? Perhaps it should be made conditional on MSVC only, and not on WIN32? //Magnus ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!)
Hi. Sorry, I don't understand the point of this patch. However, reality was confirmed. I use initdb -E UTF-8 --no-locale Gevik-san patch apply it. (So, ja is not contained.) http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/Gevik_afterpatch.png Were you changeful before and after the correction? P.S) I was thinking about the improvement of other relations. Regards, Hiroshi Saito - Original Message - From: Magnus Hagander [EMAIL PROTECTED] Gevik Babakhani wrote: Hereby a patch that fixes NLS support on PG 8.3 compiled with MSVC. Haven't looked into the details of the patch yet, will do so. But the first thing I notice - you say this is only for MSVC, right? But the patch will also change the behaviour for the mingw build. Since you say you haven't tested on it, does the documentation imply that it would work on mingw, or is this likely to break that build? Perhaps it should be made conditional on MSVC only, and not on WIN32? //Magnus ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!)
Gevik Babakhani wrote: Haven't looked into the details of the patch yet, will do so. But the first thing I notice - you say this is only for MSVC, right? But the patch will also change the behaviour for the mingw build. Since you say you haven't tested on it, does the documentation imply that it would work on mingw, or is this likely to break that build? Perhaps it should be made conditional on MSVC only, and not on WIN32? Thinking about the above... It is a good idea to have something like this be compiled only for MSVC. I guess we have to add an additional preprocessor directive from mkvcbuild to the solution (what about MSVC_COMPILER?) We have a directive called WIN32_ONLY_COMPILER that's used for this. It'll pick up MSVC and Borland C++ which normally behave at least almost the same. I am installing mingw to test the patch there. Chances are it will break because mingw does __declspec(dllimport) differently than msvc Thanks. //Magnus ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!)
Thank you. Is there any reason why JP locale files are not in normal installation? -Original Message- From: Hiroshi Saito [mailto:[EMAIL PROTECTED] Sent: Thursday, February 14, 2008 8:00 PM To: Magnus Hagander; Gevik Babakhani Cc: pgsql-patches@postgresql.org Subject: Re: [PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!) Hi. Sorry, I don't understand the point of this patch. However, reality was confirmed. I use initdb -E UTF-8 --no-locale Gevik-san patch apply it. (So, ja is not contained.) http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/Gevik_afterpatch.png Were you changeful before and after the correction? P.S) I was thinking about the improvement of other relations. Regards, Hiroshi Saito - Original Message - From: Magnus Hagander [EMAIL PROTECTED] Gevik Babakhani wrote: Hereby a patch that fixes NLS support on PG 8.3 compiled with MSVC. Haven't looked into the details of the patch yet, will do so. But the first thing I notice - you say this is only for MSVC, right? But the patch will also change the behaviour for the mingw build. Since you say you haven't tested on it, does the documentation imply that it would work on mingw, or is this likely to break that build? Perhaps it should be made conditional on MSVC only, and not on WIN32? //Magnus ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!)
We have a directive called WIN32_ONLY_COMPILER that's used for this. It'll pick up MSVC and Borland C++ which normally behave at least almost the same. I am installing mingw to test the patch there. Chances are it will break because mingw does __declspec(dllimport) differently than msvc Thanks. //Magnus Humm... I was expecting it to break... but it compiled just fine :) Here are the steps I took.. On a Win 2003 VM (VMWare): 1. Installed MinGW-5.1.3.exe 2. Installed MSYS-1.0.10.exe 3. Installed msysDTK-1.0.1.exe 4. Installed gettext-0.14.4.exe into C:\MinGW 4. Downloaded sources tarbal 5. ./configure --prefix=/home/gevik/build --without-zlib --enable-nls 6. make check, every thing was OK and 114 tests passed :) 7. make install,initdb...createdbetc..etc.. 8. set LC_MESSAGES and tested. See attachment :) The patch works both for MSVC and MINGW. Regards, Gevik. C:\msys\1.0\home\gevik\build\binpsql Welcome to psql 8.3.0, the PostgreSQL interactive terminal. Type: \copyright for distribution terms \h for help with SQL commands \? for help with psql commands \g or terminate with semicolon to execute query \q to quit Warning: Console code page (437) differs from Windows code page (1252) 8-bit characters might not work correctly. See psql reference page Notes for Windows users for details. gevik=# set lc_messages to 'sv_SE'; SET gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from generate_series(0,6) as s(a); dates --- Torsdag Februari 2008 Fredag Februari 2008 Lrdag Februari 2008 Sndag Februari 2008 Mndag Februari 2008 Tisdag Februari 2008 Onsdag Februari 2008 (7 rows) gevik=# set lc_messages to 'de_DE'; SET gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from generate_series(0,6) as s(a); dates - Donnerstag Februar 2008 Freitag Februar 2008 Samstag Februar 2008 Sonntag Februar 2008 Montag Februar 2008 Dienstag Februar 2008 Mittwoch Februar 2008 (7 rows) gevik=# set lc_messages to 'English_United_States'; SET gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from generate_series(0,6) as s(a); dates - Thursday February 2008 Friday February 2008 Saturday February 2008 Sunday February 2008 Monday February 2008 Tuesday February 2008 Wednesday February 2008 (7 rows) gevik=# set lc_messages to 'tr-TR'; SET gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from generate_series(0,6) as s(a); dates -- Persembe Subat 2008 Cuma Subat 2008 Cumartesi Subat 2008 Pazar Subat 2008 Pazartesi Subat 2008 Sali Subat 2008 arsamba Subat 2008 (7 rows) gevik=# set lc_messages to 'fr'; SET gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from generate_series(0,6) as s(a); dates -- Jeudi Fvrier 2008 Vendredi Fvrier 2008 Samedi Fvrier 2008 Dimanche Fvrier 2008 Lundi Fvrier 2008 Mardi Fvrier 2008 Mercredi Fvrier 2008 (7 rows) gevik=# set lc_messages to 'Spanish_Spain'; SET gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from generate_series(0,6) as s(a); dates --- Jueves Febrero 2008 Viernes Febrero 2008 Sbado Febrero 2008 Domingo Febrero 2008 Lunes Febrero 2008 Martes Febrero 2008 Mircoles Febrero 2008 (7 rows) gevik=# ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!)
Gevik Babakhani wrote: gevik=# set lc_messages to 'Spanish_Spain'; SET gevik=# select to_char((current_date + s.a),'TMDay TMMonth ') as dates from generate_series(0,6) as s(a); dates --- Jueves Febrero 2008 Viernes Febrero 2008 Sbado Febrero 2008 Domingo Febrero 2008 Lunes Febrero 2008 Martes Febrero 2008 Mircoles Febrero 2008 (7 rows) Hmm, interestingly you lost the diacritics here. The output is mangled for Saturday and Wednesday, which should read Sábado and Miércoles respectively. It is not good that the system allows you to output invalidly encoded data. What happens if you try setting lc_messages to Spanish_Spain.65001 instead? Ideally, it should be an error to set lc_messages to a value that's not compatible with the current encoding. Do we do that currently elsewhere? (Perhaps this is not a problem with your patch, but rather a problem that's worth fixing separately.) -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!)
Alvaro Herrera [EMAIL PROTECTED] writes: Ideally, it should be an error to set lc_messages to a value that's not compatible with the current encoding. Do we do that currently elsewhere? We don't currently enforce that, and I'm not sure it's possible to do so on non-Windows machines. AFAIR the POSIX API doesn't even associate a character set with anything but LC_CTYPE. Does it make any sense to wire in the assumption that locale names have the form something.encoding-id? If we did that, we could enforce that the encoding-id part matches LC_CTYPE, or maybe even just alter the presented values for other LC_foo variables to match. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!)
Hi. Um, my screen shot is looking at the problem. .. set LC_MESSAGES='de_DE.UTF-8'; .. it is not Japanese..however, result is Japanese message. From: Gevik Babakhani [EMAIL PROTECTED] Thank you. Is there any reason why JP locale files are not in normal installation? -Original Message- From: Hiroshi Saito [mailto:[EMAIL PROTECTED] Sent: Thursday, February 14, 2008 8:00 PM To: Magnus Hagander; Gevik Babakhani Cc: pgsql-patches@postgresql.org Subject: Re: [PATCHES] Fix for 8.3 MSVC locale (Was [HACKERS] NLS on MSVC strikes back!) Hi. Sorry, I don't understand the point of this patch. However, reality was confirmed. I use initdb -E UTF-8 --no-locale Gevik-san patch apply it. (So, ja is not contained.) http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/Gevik_afterpatch.png Were you changeful before and after the correction? P.S) I was thinking about the improvement of other relations. Regards, Hiroshi Saito ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
