Re: [HACKERS] Encoding and i18n
Am Sonntag, 7. Oktober 2007 schrieb Gregory Stark: Tom Lane [EMAIL PROTECTED] writes: Since nl_langinfo(CODESET) is supposedly determined only by LC_CTYPE, you could argue that strftime's results should be in that encoding regardless, It seems to me we aren't actually using strftime any more in any case. We seem to be using things like _(Monday) instead. I seem to recall that we don't use strftime *yet*, exactly because of this sort of issue. This was discussed before the 8.2 release. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Encoding and i18n
Am Samstag, 6. Oktober 2007 schrieb Tom Lane: It's not real clear to me whether, on a Unix machine, there is even supposed to be any difference between setting LC_TIME=es_ES.iso88591 and setting it to es_ES.utf8. Since nl_langinfo(CODESET) is supposedly determined only by LC_CTYPE, you could argue that strftime's results should be in that encoding regardless, and that the codeset component of other LC_ variables should be ignored. Some experimentation suggests that at least in glibc it doesn't work that way, and that there is in fact no principled way for you to find out what encoding strftime is giving you :-(. It might be useful to research whether that behavior is following the spec (POSIX or whatever). -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Encoding and i18n
Gregory Stark wrote: Reading the commit message about the TZ encoding issue I'm curious why this isn't a more widespread problem. How does gettext now what encoding we want messages in? How do we prevent things like to_char(now(),'month') from producing strings in an encoding different from the database's encoding? The PO files include encoding information, so it's easy for the server to recode them from that to the server (or client) encoding, as appropriate. Of course, then it is up to the translator to get it right ... but I think when he doesn't, people notice fairly quickly. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Encoding and i18n
Alvaro Herrera [EMAIL PROTECTED] writes: Gregory Stark wrote: Reading the commit message about the TZ encoding issue I'm curious why this isn't a more widespread problem. How does gettext now what encoding we want messages in? How do we prevent things like to_char(now(),'month') from producing strings in an encoding different from the database's encoding? The PO files include encoding information, so it's easy for the server to recode them from that to the server (or client) encoding, as appropriate. So does the _() macro automatically recode it to the current server encoding? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Encoding and i18n
Gregory Stark [EMAIL PROTECTED] writes: Reading the commit message about the TZ encoding issue I'm curious why this isn't a more widespread problem. How does gettext now what encoding we want messages in? How do we prevent things like to_char(now(),'month') from producing strings in an encoding different from the database's encoding? The short answer is it's all a house of cards, and if you troll the archives you will find plenty of bug reports traceable to misconfiguration in this area. The recent attempt to enforce that nl_langinfo(CODESET) matches the database encoding is a first step towards making this more bulletproof, but we're finding out that even that is harder than it looks. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Encoding and i18n
Gregory Stark wrote: Alvaro Herrera [EMAIL PROTECTED] writes: Gregory Stark wrote: Reading the commit message about the TZ encoding issue I'm curious why this isn't a more widespread problem. How does gettext now what encoding we want messages in? How do we prevent things like to_char(now(),'month') from producing strings in an encoding different from the database's encoding? The PO files include encoding information, so it's easy for the server to recode them from that to the server (or client) encoding, as appropriate. So does the _() macro automatically recode it to the current server encoding? Well, I'm not sure if it's _(), elog() or what, but it does get recoded. If I have a different client_encoding and get a NOTICE, then both the server and client get a message in the corresponding encoding. In fact this is the reason for the most common PANIC: stack overflow in elog.c error stack. When a message needs to be recoded but the recoding procedure errors out, it wants to report that and this one also fails, you get infinite recursion and nothing can get reported. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Encoding and i18n
Gregory Stark [EMAIL PROTECTED] writes: So does the _() macro automatically recode it to the current server encoding? From the gettext manual: --- gettext not only looks up a translation in a message catalog. It also converts the translation on the fly to the desired output character set. This is useful if the user is working in a different character set than the translator who created the message catalog, because it avoids distributing variants of message catalogs which differ only in the character set. The output character set is, by default, the value of nl_langinfo (CODESET), which depends on the LC_CTYPE part of the current locale. But programs which store strings in a locale independent way (e.g. UTF-8) can request that gettext and related functions return the translations in that encoding, by use of the bind_textdomain_codeset function. --- We don't currently call bind_textdomain_codeset, in part because of the lack of portability of names for codesets. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Encoding and i18n
Alvaro Herrera [EMAIL PROTECTED] writes: Gregory Stark wrote: So does the _() macro automatically recode it to the current server encoding? Well, I'm not sure if it's _(), elog() or what, but it does get recoded. If I have a different client_encoding and get a NOTICE, then both the server and client get a message in the corresponding encoding. Actually I was thinking about things like formatting.c which take localized strings and return them as data which can end up in the database. If they're in the wrong encoding then they'll be invalidly encoded strings in the database. In fact this is the reason for the most common PANIC: stack overflow in elog.c error stack. When a message needs to be recoded but the recoding procedure errors out, it wants to report that and this one also fails, you get infinite recursion and nothing can get reported. Ouch -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Encoding and i18n
Gregory Stark wrote: Alvaro Herrera [EMAIL PROTECTED] writes: Gregory Stark wrote: So does the _() macro automatically recode it to the current server encoding? Well, I'm not sure if it's _(), elog() or what, but it does get recoded. If I have a different client_encoding and get a NOTICE, then both the server and client get a message in the corresponding encoding. Actually I was thinking about things like formatting.c which take localized strings and return them as data which can end up in the database. If they're in the wrong encoding then they'll be invalidly encoded strings in the database. Oh, I didn't think of that. Let me see if I can get an invalid string into the database that way. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Encoding and i18n
Alvaro Herrera wrote: Actually I was thinking about things like formatting.c which take localized strings and return them as data which can end up in the database. If they're in the wrong encoding then they'll be invalidly encoded strings in the database. Oh, I didn't think of that. Let me see if I can get an invalid string into the database that way. I was quite certain when we closed most of these holes recently that we hadn't caught them all, so this wouldn't surprise me in the least. cheers andrew ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Encoding and i18n
Andrew Dunstan wrote: Alvaro Herrera wrote: Actually I was thinking about things like formatting.c which take localized strings and return them as data which can end up in the database. If they're in the wrong encoding then they'll be invalidly encoded strings in the database. Oh, I didn't think of that. Let me see if I can get an invalid string into the database that way. I was quite certain when we closed most of these holes recently that we hadn't caught them all, so this wouldn't surprise me in the least. It seems to work correctly: alvherre=# drop table week; DROP TABLE alvherre=# create table week (a text); CREATE TABLE alvherre=# \encoding utf8 alvherre=# insert into week select to_char(now()-'3 days'::interval, 'tmday'); INSERT 0 1 alvherre=# \encoding latin1 alvherre=# insert into week select to_char(now()-'3 days'::interval, 'tmday'); INSERT 0 1 alvherre=# select * from week; a --- miércoles miércoles (2 lignes) I tried on both a UTF8 and Latin1 terminal and it works OK in all cases. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Encoding and i18n
Alvaro Herrera [EMAIL PROTECTED] writes: I tried on both a UTF8 and Latin1 terminal and it works OK in all cases. The cases that would be interesting involve to_char's locale-specific format codes (eg Dy) along with LC_TIME settings that are deliberately incompatible with the database encoding. client_encoding is not relevant. It's not real clear to me whether, on a Unix machine, there is even supposed to be any difference between setting LC_TIME=es_ES.iso88591 and setting it to es_ES.utf8. Since nl_langinfo(CODESET) is supposedly determined only by LC_CTYPE, you could argue that strftime's results should be in that encoding regardless, and that the codeset component of other LC_ variables should be ignored. Some experimentation suggests that at least in glibc it doesn't work that way, and that there is in fact no principled way for you to find out what encoding strftime is giving you :-(. $ LANG=es_ES.utf8 date sáb oct 6 14:11:30 EDT 2007 $ LANG=es_ES.iso88591 date sáb oct 6 14:11:42 EDT 2007 $ LANG=en_US.iso88591 LC_TIME=es_ES.utf8 date sáb oct 6 14:12:10 EDT 2007 $ LC_CTYPE=en_US.iso88591 LC_TIME=es_ES.utf8 date sáb oct 6 14:12:34 EDT 2007 Perhaps a workable fix for this would be to try to mangle the LC_ settings we pass to setlocale() so that they all have the same codeset component (if any). It looks like the convention of .foo being a codeset name is fairly well standardized, even if the spelling of the codeset name is not ... regards, tom lane ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] Encoding and i18n
Tom Lane [EMAIL PROTECTED] writes: Since nl_langinfo(CODESET) is supposedly determined only by LC_CTYPE, you could argue that strftime's results should be in that encoding regardless, It seems to me we aren't actually using strftime any more in any case. We seem to be using things like _(Monday) instead. Except in my tests I don't get any French dates even when the server is started in French mode. I think we just don't have localizations for those strings yet. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Encoding and i18n
Gregory Stark [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] writes: Since nl_langinfo(CODESET) is supposedly determined only by LC_CTYPE, you could argue that strftime's results should be in that encoding regardless, It seems to me we aren't actually using strftime any more in any case. Sorry, I was using strftime as a generic standin for everything that LC_TIME affects. Trace the usage of backend/utils/adt/pg_locale.c to see what's really at stake there. The practical issues would likely be things like type money using a currency symbol that's given in the wrong encoding. And of course you did get the point that we already know a bogus LC_MESSAGES setting leads directly to error-stack-overflow PANIC. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Encoding and i18n
Gregory Stark wrote: It seems to me we aren't actually using strftime any more in any case. We seem to be using things like _(Monday) instead. Except in my tests I don't get any French dates even when the server is started in French mode. I think we just don't have localizations for those strings yet. This was already discussed [1]. I proposed a patch (that was rejected) because it calls setlocale() in every template pattern in to_char() IIRC. I coded a patch to implement the setlocale() caching mechanism but didn't send it. :( I'll take a look and this. [1] http://archives.postgresql.org/pgsql-hackers/2006-11/msg00523.php -- Euler Taveira de Oliveira http://www.timbira.com/ ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
[HACKERS] Encoding and i18n
Reading the commit message about the TZ encoding issue I'm curious why this isn't a more widespread problem. How does gettext now what encoding we want messages in? How do we prevent things like to_char(now(),'month') from producing strings in an encoding different from the database's encoding? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
