Re: [HACKERS] UTF8 on Debian
Gregory Stark <[EMAIL PROTECTED]> writes: > Hm, now things work correctly within the server if I play with text data > generated with convert_from(). But I can't seem to send utf-8 text directly= > in > psql any more: Did you rebuild libpq.so? It's got the bogus table embedded in it too. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] UTF8 on Debian
"Gregory Stark" <[EMAIL PROTECTED]> writes: > Hm, now things work correctly within the server if I play with text data > generated with convert_from(). But I can't seem to send utf-8 text directly in > psql any more: Ah, figured it out, nothing to do with the server. I didn't have the locale set to a UTF8 locale before starting up psql. I'm a bit puzzled how the locale psql is started up in works into this though. It seems to confuse psql's lexer at the least. But it goes to all this trouble to track client_encoding, shouldn't it be using that? Or at least warning you if the encoding it's run under doesn't match it? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] UTF8 on Debian
"Tom Lane" <[EMAIL PROTECTED]> writes: > Gregory Stark <[EMAIL PROTECTED]> writes: >> Something very strange is going on on my machine with UTF8: > > Argh, I missed rearranging a couple of tables in the recent patch to fix > up the encoding IDs. Will fix. Hm, now things work correctly within the server if I play with text data generated with convert_from(). But I can't seem to send utf-8 text directly in psql any more: postgres=# select 'ライセン'; ?column? -- ' (1 row) postgres=# show client_encoding; client_encoding - UTF8 (1 row) postgres=# show server_encoding; server_encoding - UTF8 (1 row) -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] UTF8 on Debian
Gregory Stark <[EMAIL PROTECTED]> writes: > ... This puts the order back in sync. I don't haven't looked at the > surrounding code enough to know if there aren't more problems like this > lurking or if there's a way to fix it so it isn't such a pain to maintain. The real problem is I believed the comment in pg_wchar.h that told me the only table that needed fixed was pg_enc2name :-( A check of Tatsuo's last couple of encoding-related commits confirms that there are only two order-sensitive tables, though, so I think we're good now. Also this time I troubled to run the src/test/mb/ tests, which I really should have tried earlier. Ah well. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] UTF8 on Debian
Gregory Stark <[EMAIL PROTECTED]> writes: > Something very strange is going on on my machine with UTF8: Argh, I missed rearranging a couple of tables in the recent patch to fix up the encoding IDs. Will fix. regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] UTF8 on Debian
The problem was simply that wchar.c has an array indexed by server encoding
number. This puts the order back in sync. I don't haven't looked at the
surrounding code enough to know if there aren't more problems like this
lurking or if there's a way to fix it so it isn't such a pain to maintain.
Index: src/backend/utils/mb/wchar.c
===
RCS file: /home/stark/src/REPOSITORY/pgsql/src/backend/utils/mb/wchar.c,v
retrieving revision 1.64
diff -u -r1.64 wchar.c
--- src/backend/utils/mb/wchar.c18 Sep 2007 17:41:17 - 1.64
+++ src/backend/utils/mb/wchar.c15 Oct 2007 21:53:06 -
@@ -1315,6 +1315,7 @@
{pg_euccn2wchar_with_len, pg_euccn_mblen, pg_euccn_dsplen,
pg_euccn_verifier, 2}, /* 2; PG_EUC_CN */
{pg_euckr2wchar_with_len, pg_euckr_mblen, pg_euckr_dsplen,
pg_euckr_verifier, 3}, /* 3; PG_EUC_KR */
{pg_euctw2wchar_with_len, pg_euctw_mblen, pg_euctw_dsplen,
pg_euctw_verifier, 4}, /* 4; PG_EUC_TW */
+ {pg_eucjp2wchar_with_len, pg_eucjp_mblen, pg_eucjp_dsplen,
pg_eucjp_verifier, 3}, /* 33; PG_EUC_JIS_2004 */
{pg_utf2wchar_with_len, pg_utf_mblen, pg_utf_dsplen, pg_utf8_verifier,
4}, /* 5; PG_UTF8 */
{pg_mule2wchar_with_len, pg_mule_mblen, pg_mule_dsplen,
pg_mule_verifier, 4}, /* 6; PG_MULE_INTERNAL */
{pg_latin12wchar_with_len, pg_latin1_mblen, pg_latin1_dsplen,
pg_latin1_verifier, 1}, /* 7; PG_LATIN1 */
@@ -1343,13 +1344,12 @@
{pg_latin12wchar_with_len, pg_latin1_mblen, pg_latin1_dsplen,
pg_latin1_verifier, 1}, /* 30; PG_WIN1254 */
{pg_latin12wchar_with_len, pg_latin1_mblen, pg_latin1_dsplen,
pg_latin1_verifier, 1}, /* 31; PG_WIN1255 */
{pg_latin12wchar_with_len, pg_latin1_mblen, pg_latin1_dsplen,
pg_latin1_verifier, 1}, /* 32; PG_WIN1257 */
- {pg_eucjp2wchar_with_len, pg_eucjp_mblen, pg_eucjp_dsplen,
pg_eucjp_verifier, 3}, /* 33; PG_EUC_JIS_2004 */
{0, pg_sjis_mblen, pg_sjis_dsplen, pg_sjis_verifier, 2},/* 34;
PG_SJIS */
{0, pg_big5_mblen, pg_big5_dsplen, pg_big5_verifier, 2},/* 35;
PG_BIG5 */
{0, pg_gbk_mblen, pg_gbk_dsplen, pg_gbk_verifier, 2}, /* 36;
PG_GBK */
{0, pg_uhc_mblen, pg_uhc_dsplen, pg_uhc_verifier, 2}, /* 37;
PG_UHC */
- {pg_johab2wchar_with_len, pg_johab_mblen, pg_johab_dsplen,
pg_johab_verifier, 3}, /* 38; PG_JOHAB */
{0, pg_gb18030_mblen, pg_gb18030_dsplen, pg_gb18030_verifier, 4},
/* 39; PG_GB18030 */
+ {pg_johab2wchar_with_len, pg_johab_mblen, pg_johab_dsplen,
pg_johab_verifier, 3}, /* 38; PG_JOHAB */
{0, pg_sjis_mblen, pg_sjis_dsplen, pg_sjis_verifier, 2} /* 40;
PG_SHIFT_JIS_2004 */
};
--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
Re: [HACKERS] UTF8 on Debian
"Gregory Stark" <[EMAIL PROTECTED]> writes:
> Something very strange is going on on my machine with UTF8:
>
> postgres=# show server_encoding;
> server_encoding
> -
> UTF8
> (1 row)
Hm. Well this doesn't look right:
(gdb) p pg_wchar_table[PG_UTF8]
$9 = {mb2wchar_with_len = 0x8364cff ,
mblen = 0x8364eda , dsplen = 0x8364f60 ,
mbverify = 0x83655cd , maxmblen = 4}
This looks like version skew around the file that Tom recently whacked around
to restore binary compatibility with 8.2. But I can't see how -- I don't have
any old header files in /usr/include or anything like that so I can't see how
I could have picked up an out-of-date header file.
--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
