Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-17 Thread Kyotaro HORIGUCHI
Hello, I suppose that testing for the two cases and additional one case which runs pg_do_encoding_conversion(), say latin1, would be enough to confirm that encoding/decoding is properly done, since the concrete conversion scheme is not significant this case. So I recommend that we

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-17 Thread Alvaro Herrera
Excerpts from Kyotaro HORIGUCHI's message of mar jul 17 05:01:10 -0400 2012: I think that's probably too much engineering for something that doesn't really warrant it. A real solution to this problem could be to create yet another new test file containing just this function definition and

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-13 Thread Alvaro Herrera
Excerpts from Kyotaro HORIGUCHI's message of jue jul 12 00:09:19 -0400 2012: Hmm... Sorry for immature patch.. No need to apologize. ... and this story hasn't ended yet, because one of the new tests is failing. See here:

Re: [SPAM] [MessageLimit][lowlimit] Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-11 Thread Alvaro Herrera
Excerpts from Alvaro Herrera's message of mar jul 10 16:23:57 -0400 2012: Excerpts from Kyotaro HORIGUCHI's message of mar jul 03 04:59:38 -0400 2012: Hello, Here is regression test runs on pg's also built with cygwin-gcc and VC++. The patches attached following, -

Re: [SPAM] [MessageLimit][lowlimit] Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-11 Thread Alex Hunsaker
On Wed, Jul 11, 2012 at 1:42 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: I have pushed these changes to HEAD, 9.2 and 9.1. Instead of the games with plperl_lc_*.out being copied around, I just used the ASCII version as plperl_lc_1.out and the UTF8 one as plperl_lc.out. ... and this

Re: [SPAM] [MessageLimit][lowlimit] Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-11 Thread Kyotaro HORIGUCHI
Hmm... Sorry for immature patch.. ... and this story hasn't ended yet, because one of the new tests is failing. See here: http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=magpiedt=2012-07-11%2010%3A00%3A04 The interesting part of the diff is: ... SELECT

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-11 Thread Kyotaro HORIGUCHI
Very sorry for rotten subject. I resent the message with correct subject. # Our mail server insisted that the message is spam. sigh.. Hmm... Sorry for immature patch.. ... and this story hasn't ended yet, because one of the new tests is failing. See here:

Re: [SPAM] [MessageLimit][lowlimit] Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-10 Thread Alvaro Herrera
Excerpts from Kyotaro HORIGUCHI's message of mar jul 03 04:59:38 -0400 2012: Hello, Here is regression test runs on pg's also built with cygwin-gcc and VC++. The patches attached following, - plperl_sql_ascii-4.patch : fix for pl/perl utf8 vs sql_ascii -

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-06 Thread Kyotaro HORIGUCHI
Hello, I've been stuck in mud trying to plperl work on windows environment. I saw many messages complaining that plperl wouldn't be built to work. For the convenience of those and myself, I describe the process of building postgresql with plperl on Windows with cygwin and VC++ I've done

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-05 Thread Alex Hunsaker
On Tue, Jul 3, 2012 at 2:59 AM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: Hello, Here is regression test runs on pg's also built with cygwin-gcc and VC++. Thank you! The patches attached following, - plperl_sql_ascii-4.patch : fix for pl/perl utf8 vs sql_ascii -

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-03 Thread Kyotaro HORIGUCHI
Hello, Here is regression test runs on pg's also built with cygwin-gcc and VC++. The patches attached following, - plperl_sql_ascii-4.patch : fix for pl/perl utf8 vs sql_ascii - plperl_sql_ascii_regress-1.patch : regression test for this patch. I

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-27 Thread Kyotaro HORIGUCHI
Hello, thank you for your sugestion. I agree. That is the fundamental question. I've coded just for my fun but I don't see not so much signicance to do that. We might omit the test for this which is non-ciritical and corner cases. We need these tests to work on Windows too, so fancy

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-22 Thread Kyotaro HORIGUCHI
Hello. The attached ugly patch does it. We seem should put NO_LOCALE=1 on the 'make check' command line for the encodings not compatible with the environmental locale, although it looks work. +REGRESS_LC0 = $(subst .sql,,$(shell cd sql; ls plperl_lc_$(shell echo snip. Hrm, that's quite

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-22 Thread Kyotaro HORIGUCHI
Hello. Renaming ret to quoted and str to ret as the patch attached might make it easily readable. I think I'm going to refrain from this because it will be more painful to backpatch. I've felt hesitation to do so, too. The new patch is indeed avoid leaks although which does not lasts

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-22 Thread Tom Lane
Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp writes: +REGRESS_LC0 = $(subst .sql,,$(shell cd sql; ls plperl_lc_$(shell echo Hrm, that's quite cute. I dunno if there is a more cannon way of doing the above-- but it seems to work. I'm not sure this regression test is worth it. I'm thinking

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Kyotaro HORIGUCHI
Hello, Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr and SvPVUTF8() when turning a perl string into a cstring. Right. I spent a bit longer time catching on pl/perl and now understand what is the problem... So I played a bit with this patch, and touched it a bit

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Kyotaro HORIGUCHI
Ouch! # UtfToLocal() seems to have a bug that always report illegal # encoding was UTF8 regardless of the real encoding. But # plper_lc_*.(sql|out) increases if the bug is fixed. This is not a bug. The error message invalid byte sequence for encoding UTF8 meant invalid INPUT byte sequence as

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Alex Hunsaker
On Wed, Jun 20, 2012 at 1:15 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr and SvPVUTF8() when turning a perl string into a cstring. Right.

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Alex Hunsaker
On Thu, Jun 21, 2012 at 5:22 AM, Kyotaro HORIGUCHI horiguchi.kyot...@lab.ntt.co.jp wrote: So I played a bit with this patch, and touched it a bit mainly [...] functions in Util.xs might leak some memory, so I made an attempt to Ok, Is it ok to look into the newer patch including fix of leaks

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Alvaro Herrera
Excerpts from Kyotaro HORIGUCHI's message of jue jun 21 08:02:58 -0400 2012: Ouch! # UtfToLocal() seems to have a bug that always report illegal # encoding was UTF8 regardless of the real encoding. But # plper_lc_*.(sql|out) increases if the bug is fixed. This is not a bug. The

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Alvaro Herrera
Excerpts from Kyotaro HORIGUCHI's message of jue jun 21 07:22:43 -0400 2012: Hello, Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr and SvPVUTF8() when turning a perl string into a cstring. Right. I spent a bit longer time catching on pl/perl and now

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Alvaro Herrera
Excerpts from Alex Hunsaker's message of jue jun 21 10:27:41 -0400 2012: On Wed, Jun 20, 2012 at 1:15 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: Seems like we missed the fact that we still did SvUTF8_on()

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-20 Thread Alvaro Herrera
Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr and SvPVUTF8() when turning a perl string into a cstring. Right. So I played a bit with this patch, and touched it a bit mainly just to add some more

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Robert Haas
On Mon, Jun 18, 2012 at 3:30 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr and SvPVUTF8() when turning a perl string into a cstring. Hmm,

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Alex Hunsaker
On Mon, Jun 18, 2012 at 1:30 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr and SvPVUTF8() when turning a perl string into a cstring. Hmm,

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar jun 19 11:36:41 -0400 2012: On Mon, Jun 18, 2012 at 3:30 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: Seems like we missed the fact that we still did SvUTF8_on()

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 3:03 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: That's what I thought.  I will commit to both branches soon, then. I think there might be three branches involved. Mind you, this should have been an open item, not a commitfest item. (Actually not even an open

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-18 Thread Alvaro Herrera
Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr and SvPVUTF8() when turning a perl string into a cstring. Hmm, this patch belongs into back branches too, right? Not just the current development

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-02-13 Thread Christoph Berg
Re: Alex Hunsaker 2012-02-10 cafapbrr9y1fu6gpvu+8ta8vty6qvcm3dfarkt8jg_ehgetx...@mail.gmail.com Does the attached fix the issue for you? Yes. :) Christoph -- c...@df7cb.de | http://www.df7cb.de/ signature.asc Description: Digital signature

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-02-10 Thread Alex Hunsaker
On Thu, Feb 9, 2012 at 03:21, Christoph Berg c...@df7cb.de wrote: Hi, we have a database that is storing strings in various encodings (and non-encodings, namely the arbitrary byte soup [ ... ] For this reason, the database uses sql_ascii encoding ...snip... In sql_ascii databases,