Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-17 Thread Alvaro Herrera
Excerpts from Kyotaro HORIGUCHI's message of mar jul 17 05:01:10 -0400 2012: > > I think that's probably too much engineering for something that doesn't > > really warrant it. A real solution to this problem could be to create > > yet another new test file containing just this function definitio

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-17 Thread Kyotaro HORIGUCHI
Hello, > > I suppose that testing for the two cases and additional > > one case which runs pg_do_encoding_conversion(), say latin1, > > would be enough to confirm that encoding/decoding is properly > > done, since the concrete conversion scheme is not significant > > this case. > > > > So I recom

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-13 Thread Alvaro Herrera
Excerpts from Kyotaro HORIGUCHI's message of jue jul 12 00:09:19 -0400 2012: > > Hmm... Sorry for immature patch.. No need to apologize. > > ... and this story hasn't ended yet, because one of the new tests is > > failing. See here: > > > > http://buildfarm.postgresql.org/cgi-bin/show_log.pl?

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-11 Thread Kyotaro HORIGUCHI
Very sorry for rotten subject. I resent the message with correct subject. # Our mail server insisted that the message is spam. sigh.. Hmm... Sorry for immature patch.. > ... and this story hasn't ended yet, because one of the new tests is > failing. See here: > > http://buildfarm.postgresql

Re: [SPAM] [MessageLimit][lowlimit] Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-11 Thread Kyotaro HORIGUCHI
Hmm... Sorry for immature patch.. > ... and this story hasn't ended yet, because one of the new tests is > failing. See here: > > http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=magpie&dt=2012-07-11%2010%3A00%3A04 > > The interesting part of the diff is: ... > SELECT encode(perl_utf_in

Re: [SPAM] [MessageLimit][lowlimit] Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-11 Thread Alex Hunsaker
On Wed, Jul 11, 2012 at 1:42 PM, Alvaro Herrera wrote: > >> I have pushed these changes to HEAD, 9.2 and 9.1. Instead of the games >> with plperl_lc_*.out being copied around, I just used the ASCII version >> as plperl_lc_1.out and the UTF8 one as plperl_lc.out. > > ... and this story hasn't ende

Re: [SPAM] [MessageLimit][lowlimit] Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-11 Thread Alvaro Herrera
Excerpts from Alvaro Herrera's message of mar jul 10 16:23:57 -0400 2012: > Excerpts from Kyotaro HORIGUCHI's message of mar jul 03 04:59:38 -0400 2012: > > Hello, Here is regression test runs on pg's also built with > > cygwin-gcc and VC++. > > > > The patches attached following, > > > > - plpe

Re: [SPAM] [MessageLimit][lowlimit] Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-10 Thread Alvaro Herrera
Excerpts from Kyotaro HORIGUCHI's message of mar jul 03 04:59:38 -0400 2012: > Hello, Here is regression test runs on pg's also built with > cygwin-gcc and VC++. > > The patches attached following, > > - plperl_sql_ascii-4.patch : fix for pl/perl utf8 vs sql_ascii > - plperl_sql_ascii_re

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-05 Thread Kyotaro HORIGUCHI
Hello, > > I've been stuck in mud trying to plperl work on windows > > environment. I saw many messages complaining that plperl wouldn't > > be built to work. For the convenience of those and myself, I > > describe the process of building postgresql with plperl on > > Windows with cygwin and VC++

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-05 Thread Alex Hunsaker
On Tue, Jul 3, 2012 at 2:59 AM, Kyotaro HORIGUCHI wrote: > Hello, Here is regression test runs on pg's also built with > cygwin-gcc and VC++. Thank you! > The patches attached following, > > - plperl_sql_ascii-4.patch : fix for pl/perl utf8 vs sql_ascii > - plperl_sql_ascii_regress-1.pat

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-07-03 Thread Kyotaro HORIGUCHI
Hello, Here is regression test runs on pg's also built with cygwin-gcc and VC++. The patches attached following, - plperl_sql_ascii-4.patch : fix for pl/perl utf8 vs sql_ascii - plperl_sql_ascii_regress-1.patch : regression test for this patch. I added

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-27 Thread Kyotaro HORIGUCHI
Hello, thank you for your sugestion. > > I agree. That is the fundamental question. I've coded just for my > > fun but I don't see not so much signicance to do that. We might > > omit the test for this which is non-ciritical and corner cases. > > We need these tests to work on Windows too, so fa

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-22 Thread Tom Lane
Kyotaro HORIGUCHI writes: >>> +REGRESS_LC0 = $(subst .sql,,$(shell cd sql; ls plperl_lc_$(shell echo >> Hrm, that's quite cute. I dunno if there is a more cannon way of doing >> the above-- but it seems to work. I'm not sure this regression test is >> worth it. I'm thinking maybe we should just r

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-22 Thread Kyotaro HORIGUCHI
Hello. > > Renaming ret to quoted and str to ret as the patch attached might > > make it easily readable. > > I think I'm going to refrain from this because it will be more painful > to backpatch. I've felt hesitation to do so, too. The new patch is indeed avoid leaks although which does not la

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Kyotaro HORIGUCHI
Hello. > > The attached ugly patch does it. We seem should put NO_LOCALE=1 > > on the 'make check' command line for the encodings not compatible > > with the environmental locale, although it looks work. > > +REGRESS_LC0 = $(subst .sql,,$(shell cd sql; ls plperl_lc_$(shell echo snip. > Hrm, that'

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Alvaro Herrera
Excerpts from Alex Hunsaker's message of jue jun 21 10:27:41 -0400 2012: > > On Wed, Jun 20, 2012 at 1:15 PM, Alvaro Herrera > wrote: > > Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: > > > >> Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr > >>

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Alvaro Herrera
Excerpts from Kyotaro HORIGUCHI's message of jue jun 21 07:22:43 -0400 2012: > Hello, > > > > Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr > > > and SvPVUTF8() when turning a perl string into a cstring. > > > > Right. > > I spent a bit longer time catching on pl/perl a

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Alvaro Herrera
Excerpts from Kyotaro HORIGUCHI's message of jue jun 21 08:02:58 -0400 2012: > > Ouch! > > > # UtfToLocal() seems to have a bug that always report illegal > > # encoding was "UTF8" regardless of the real encoding. But > > # plper_lc_*.(sql|out) increases if the bug is fixed. > > This is not a b

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Alex Hunsaker
On Thu, Jun 21, 2012 at 5:22 AM, Kyotaro HORIGUCHI wrote: >> So I played a bit with this patch, and touched it a bit mainly >> [...] functions in Util.xs might leak some memory, so I made an attempt to > Ok, Is it ok to look into the newer patch including fix of leaks > at first? Yeah :-). > V

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Alex Hunsaker
On Wed, Jun 20, 2012 at 1:15 PM, Alvaro Herrera wrote: > Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: > >> Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr >> and SvPVUTF8() when turning a perl string into a cstring. > > Right. > > So I played a bi

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Kyotaro HORIGUCHI
Ouch! > # UtfToLocal() seems to have a bug that always report illegal > # encoding was "UTF8" regardless of the real encoding. But > # plper_lc_*.(sql|out) increases if the bug is fixed. This is not a bug. The error message "invalid byte sequence for encoding "UTF8"" meant "invalid INPUT byte seq

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-21 Thread Kyotaro HORIGUCHI
Hello, > > Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr > > and SvPVUTF8() when turning a perl string into a cstring. > > Right. I spent a bit longer time catching on pl/perl and now understand what is the problem... > So I played a bit with this patch, and touched it

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-20 Thread Alvaro Herrera
Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: > Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr > and SvPVUTF8() when turning a perl string into a cstring. Right. So I played a bit with this patch, and touched it a bit mainly just to add some mor

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 3:03 PM, Alvaro Herrera wrote: > That's what I thought.  I will commit to both branches soon, then. I think there might be three branches involved. > Mind you, this should have been an "open item", not a commitfest item. > (Actually not even an open item.  We should have

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar jun 19 11:36:41 -0400 2012: > > On Mon, Jun 18, 2012 at 3:30 PM, Alvaro Herrera > wrote: > > Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: > > > >> Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr > >> an

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Alex Hunsaker
On Mon, Jun 18, 2012 at 1:30 PM, Alvaro Herrera wrote: > > Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: > >> Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr >> and SvPVUTF8() when turning a perl string into a cstring. > > Hmm, this patch belongs i

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Robert Haas
On Mon, Jun 18, 2012 at 3:30 PM, Alvaro Herrera wrote: > Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: > >> Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr >> and SvPVUTF8() when turning a perl string into a cstring. > > Hmm, this patch belongs int

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-18 Thread Alvaro Herrera
Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: > Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr > and SvPVUTF8() when turning a perl string into a cstring. Hmm, this patch belongs into back branches too, right? Not just the current development t

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-02-13 Thread Christoph Berg
Re: Alex Hunsaker 2012-02-10 > Does the attached fix the issue for you? Yes. :) Christoph -- c...@df7cb.de | http://www.df7cb.de/ signature.asc Description: Digital signature

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-02-10 Thread Alex Hunsaker
On Thu, Feb 9, 2012 at 03:21, Christoph Berg wrote: > Hi, > > we have a database that is storing strings in various encodings (and > non-encodings, namely the arbitrary byte soup [ ... ] > For this reason, the database uses > sql_ascii encoding > ...snip... > In sql_ascii databases, utf_e2u does

[HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-02-09 Thread Christoph Berg
Hi, we have a database that is storing strings in various encodings (and non-encodings, namely the arbitrary byte soup that you might see in email headers from the internet). For this reason, the database uses sql_ascii encoding. The columns are text, as most characters are ascii, so bytea didn't