Re: [HACKERS] Radix tree for character conversion

2017-03-27 Thread Kyotaro HORIGUCHI
Hmm, things are bit different. At Thu, 23 Mar 2017 12:13:07 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170323.121307.241436413.horiguchi.kyot...@lab.ntt.co.jp> > > Ok, I'll write a small script to generate a set of "conversion > > dump" and try

Re: [HACKERS] Radix tree for character conversion

2017-03-22 Thread Kyotaro HORIGUCHI
At Tue, 21 Mar 2017 13:10:48 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170321.131048.150321071.horiguchi.kyot...@lab.ntt.co.jp> > At Fri, 17 Mar 2017 13:03:35 +0200, Heikki Linnakangas > wrote in

Re: [HACKERS] Radix tree for character conversion

2017-03-20 Thread Kyotaro HORIGUCHI
Hello, At Fri, 17 Mar 2017 13:03:35 +0200, Heikki Linnakangas wrote in <01efd334-b839-0450-1b63-f2dea9326...@iki.fi> > On 03/17/2017 07:19 AM, Kyotaro HORIGUCHI wrote: > > I would like to use convert() function. It can be a large > > PL/PgSQL function or a series of "SELECT

Re: [HACKERS] Radix tree for character conversion

2017-03-17 Thread Heikki Linnakangas
On 03/17/2017 07:19 AM, Kyotaro HORIGUCHI wrote: At Mon, 13 Mar 2017 21:07:39 +0200, Heikki Linnakangas wrote in Hmm. A somewhat different approach might be more suitable for testing across versions, though. We could modify the

Re: [HACKERS] Radix tree for character conversion

2017-03-16 Thread Kyotaro HORIGUCHI
Thank you for committing this. At Mon, 13 Mar 2017 21:07:39 +0200, Heikki Linnakangas wrote in > On 03/13/2017 08:53 PM, Tom Lane wrote: > > Heikki Linnakangas writes: > >> It would be nice to run the map_checker

Re: [HACKERS] Radix tree for character conversion

2017-03-13 Thread Michael Paquier
On Tue, Mar 14, 2017 at 4:07 AM, Heikki Linnakangas wrote: > On 03/13/2017 08:53 PM, Tom Lane wrote: >> Heikki Linnakangas writes: >>> >>> It would be nice to run the map_checker tool one more time, though, to >>> verify that the mappings match those from

Re: [HACKERS] Radix tree for character conversion

2017-03-13 Thread Heikki Linnakangas
On 03/13/2017 08:53 PM, Tom Lane wrote: Heikki Linnakangas writes: It would be nice to run the map_checker tool one more time, though, to verify that the mappings match those from PostgreSQL 9.6. +1 Just to be sure, and after that the map checker can go to the dustbin.

Re: [HACKERS] Radix tree for character conversion

2017-03-13 Thread Tom Lane
Heikki Linnakangas writes: > I did some more kibitzing here and there, and committed. Thanks everyone! 111 files changed, 147742 insertions(+), 367346 deletions(-) Nice. > It would be nice to run the map_checker tool one more time, though, to > verify that the mappings match

Re: [HACKERS] Radix tree for character conversion

2017-03-13 Thread Heikki Linnakangas
On 03/06/2017 10:16 AM, Kyotaro HORIGUCHI wrote: At Fri, 3 Mar 2017 12:53:04 +0900, Michael Paquier wrote in On Thu, Mar 2, 2017 at 2:20 PM, Kyotaro HORIGUCHI

Re: [HACKERS] Radix tree for character conversion

2017-03-06 Thread Kyotaro HORIGUCHI
Hello, At Fri, 3 Mar 2017 12:53:04 +0900, Michael Paquier wrote in > On Thu, Mar 2, 2017 at 2:20 PM, Kyotaro HORIGUCHI > wrote: > > 5) Just remove plain map files

Re: [HACKERS] Radix tree for character conversion

2017-03-02 Thread Michael Paquier
On Thu, Mar 2, 2017 at 2:20 PM, Kyotaro HORIGUCHI wrote: > 5) Just remove plain map files and all related code. Addition to >that, Makefile stores hash digest of authority files in >Unicode/authoriy_hashes.txt or something that is managed by >git.

Re: [HACKERS] Radix tree for character conversion

2017-03-01 Thread Kyotaro HORIGUCHI
At Wed, 1 Mar 2017 14:34:23 +0900, Michael Paquier wrote in > On Tue, Feb 28, 2017 at 5:34 PM, Kyotaro HORIGUCHI > wrote: > > At Tue, 28 Feb 2017 15:20:06 +0900,

Re: [HACKERS] Radix tree for character conversion

2017-02-28 Thread Michael Paquier
On Tue, Feb 28, 2017 at 5:34 PM, Kyotaro HORIGUCHI wrote: > At Tue, 28 Feb 2017 15:20:06 +0900, Michael Paquier > wrote in > >> +conv.o: conv.c char_converter.c >>

Re: [HACKERS] Radix tree for character conversion

2017-02-28 Thread Kyotaro HORIGUCHI
At Tue, 28 Feb 2017 15:20:06 +0900, Michael Paquier wrote in > On Mon, Feb 27, 2017 at 5:37 PM, Kyotaro HORIGUCHI > wrote: > > At Wed, 22 Feb 2017 16:06:14 +0900,

Re: [HACKERS] Radix tree for character conversion

2017-02-27 Thread Michael Paquier
On Mon, Feb 27, 2017 at 5:37 PM, Kyotaro HORIGUCHI wrote: > At Wed, 22 Feb 2017 16:06:14 +0900, Michael Paquier > wrote in > >> In order to conduct sanity checks on

Re: [HACKERS] Radix tree for character conversion

2017-02-27 Thread Kyotaro HORIGUCHI
Hello, At Tue, 28 Feb 2017 08:00:22 +0530, Robert Haas wrote in > On Mon, Feb 27, 2017 at 2:07 PM, Kyotaro HORIGUCHI > wrote: > >> +# make_charmap - convert charset

Re: [HACKERS] Radix tree for character conversion

2017-02-27 Thread Robert Haas
On Mon, Feb 27, 2017 at 2:07 PM, Kyotaro HORIGUCHI wrote: >> +# make_charmap - convert charset table to charmap hash >> +# with checking duplicate source code >> Maybe this should be "with checking of duplicated source codes". > > Even though I'm not good

Re: [HACKERS] Radix tree for character conversion

2017-02-27 Thread Kyotaro HORIGUCHI
Thank you for the comment. At Wed, 22 Feb 2017 16:06:14 +0900, Michael Paquier wrote in > Thanks for the rebase. I have been spending sore time looking at this > patch. The new stuff in convutils.pm

Re: [HACKERS] Radix tree for character conversion

2017-02-21 Thread Michael Paquier
On Fri, Feb 3, 2017 at 1:18 PM, Kyotaro HORIGUCHI wrote: > Thanks to that Heikki have pushed the first two patches and a > part of the third, only one patch is remaining now. > > # Sorry for not separating KOI8 stuffs. > > At Tue, 31 Jan 2017 19:06:09 +0900 (Tokyo

Re: [HACKERS] Radix tree for character conversion

2017-02-02 Thread Kyotaro HORIGUCHI
Tnanks to that Heikki have pushed the first two patches and a part of the third, only one patch is remaining now. # Sorry for not separating KOI8 stuffs. At Tue, 31 Jan 2017 19:06:09 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in

Re: [HACKERS] Radix tree for character conversion

2017-01-31 Thread Kyotaro HORIGUCHI
At Tue, 31 Jan 2017 12:25:46 +0900, Michael Paquier wrote in > On Mon, Jan 30, 2017 at 3:37 PM, Kyotaro HORIGUCHI > wrote: > > Hello, this is the revised version of

Re: [HACKERS] Radix tree for character conversion

2017-01-30 Thread Michael Paquier
On Mon, Jan 30, 2017 at 3:37 PM, Kyotaro HORIGUCHI wrote: > Hello, this is the revised version of character conversion using radix tree. Thanks for the new version, I'll look at it once I am done with the cleanup of the current CF. For now I have moved it to the

Re: [HACKERS] Radix tree for character conversion

2017-01-29 Thread Kyotaro HORIGUCHI
Hello, this is the revised version of character conversion using radix tree. At Fri, 27 Jan 2017 17:33:57 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170127.173357.221584433.horiguchi.kyot...@lab.ntt.co.jp> > Hi, this is an intermediate report

Re: [HACKERS] Radix tree for character conversion

2017-01-27 Thread Kyotaro HORIGUCHI
Hi, this is an intermediate report without a patch. At Thu, 26 Jan 2017 21:42:12 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170126.214212.111556326.horiguchi.kyot...@lab.ntt.co.jp> > > >

Re: [HACKERS] Radix tree for character conversion

2017-01-26 Thread Kyotaro HORIGUCHI
Thank you for looking this. At Thu, 26 Jan 2017 16:28:16 +0900, Michael Paquier wrote in > On Tue, Jan 10, 2017 at 8:22 PM, Kyotaro HORIGUCHI > wrote: > >

Re: [HACKERS] Radix tree for character conversion

2017-01-26 Thread Kyotaro HORIGUCHI
At Thu, 26 Jan 2017 16:29:10 +0900, Michael Paquier wrote in > On Wed, Jan 25, 2017 at 7:18 PM, Ishii Ayumi wrote: > > I patched 4 patchset and run "make", but I got

Re: [HACKERS] Radix tree for character conversion

2017-01-25 Thread Michael Paquier
On Wed, Jan 25, 2017 at 7:18 PM, Ishii Ayumi wrote: > I patched 4 patchset and run "make", but I got failed. > Is this a bug or my mistake ? > I'm sorry if I'm wrong. > > [$(TOP)]$ patch -p1 < ../0001-Add-missing-semicolon.patch > [$(TOP)]$ patch -p1 <

Re: [HACKERS] Radix tree for character conversion

2017-01-25 Thread Michael Paquier
On Tue, Jan 10, 2017 at 8:22 PM, Kyotaro HORIGUCHI wrote: > [...patch...] Nobody has showed up yet to review this patch, so I am giving it a shot. The patch file sizes are scary at first sight, but after having a look: 36 files changed, 1411 insertions(+),

Re: [HACKERS] Radix tree for character conversion

2017-01-25 Thread Ishii Ayumi
HI, I patched 4 patchset and run "make", but I got failed. Is this a bug or my mistake ? I'm sorry if I'm wrong. [$(TOP)]$ patch -p1 < ../0001-Add-missing-semicolon.patch [$(TOP)]$ patch -p1 < ../0002-Correct-reference-resolution-syntax.patch [$(TOP)]$ patch -p1 <

Re: [HACKERS] Radix tree for character conversion

2016-12-12 Thread Kyotaro HORIGUCHI
Hello, I looked on this closer. The attached is the revised version of this patch. At Mon, 05 Dec 2016 19:29:54 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20161205.192954.12189.horiguchi.kyot...@lab.ntt.co.jp> > Apart from the aboves, I have

Re: [HACKERS] Radix tree for character conversion

2016-12-05 Thread Kyotaro HORIGUCHI
Hello, thank you for reviewing this. I compared mine and yours. The new patch works fine and gives smaller radix map files. It seems also to me more readable. At Fri, 2 Dec 2016 22:07:07 +0200, Heikki Linnakangas wrote in > On

Re: [HACKERS] Radix tree for character conversion

2016-12-02 Thread Heikki Linnakangas
On 12/02/2016 10:18 PM, Alvaro Herrera wrote: Heikki Linnakangas wrote: On 11/09/2016 10:38 AM, Kyotaro HORIGUCHI wrote: Thanks. The attached patch contains the patch by perlcritic. 0001,2,3 are Heikki's patch that are not modified since it is first proposed. It's a bit too big so I don't

Re: [HACKERS] Radix tree for character conversion

2016-12-02 Thread Alvaro Herrera
Heikki Linnakangas wrote: > On 11/09/2016 10:38 AM, Kyotaro HORIGUCHI wrote: > > Thanks. The attached patch contains the patch by perlcritic. > > > > 0001,2,3 are Heikki's patch that are not modified since it is > > first proposed. It's a bit too big so I don't attach them to this > > mail

Re: [HACKERS] Radix tree for character conversion

2016-12-02 Thread Heikki Linnakangas
On 11/09/2016 10:38 AM, Kyotaro HORIGUCHI wrote: Thanks. The attached patch contains the patch by perlcritic. 0001,2,3 are Heikki's patch that are not modified since it is first proposed. It's a bit too big so I don't attach them to this mail (again).

Re: [HACKERS] Radix tree for character conversion

2016-11-30 Thread Heikki Linnakangas
On 10/31/2016 06:11 PM, Daniel Gustafsson wrote: On 27 Oct 2016, at 09:23, Kyotaro HORIGUCHI wrote: At Tue, 25 Oct 2016 12:23:48 +0300, Heikki Linnakangas wrote in <08e7892a-d55c-eefe-76e6-7910bc8dd...@iki.fi> [..] The perl scripts are

Re: [HACKERS] Radix tree for character conversion

2016-11-28 Thread Kyotaro HORIGUCHI
Hello. I'll be off line until at least next Monday. So I move this to the next CF by myself. At Wed, 09 Nov 2016 17:38:53 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20161109.173853.77274443.horiguchi.kyot...@lab.ntt.co.jp> > Hello, thank you for

Re: [HACKERS] Radix tree for character conversion

2016-11-09 Thread Kyotaro HORIGUCHI
Hello, thank you for polishing this. At Wed, 9 Nov 2016 02:19:01 +0100, Daniel Gustafsson wrote in <80f34f25-bf6d-4bcd-9c38-42ed10d3f...@yesql.se> > > On 08 Nov 2016, at 17:37, Peter Eisentraut > > wrote: > > > > On 10/31/16 12:11 PM, Daniel

Re: [HACKERS] Radix tree for character conversion

2016-11-08 Thread Daniel Gustafsson
> On 08 Nov 2016, at 17:37, Peter Eisentraut > wrote: > > On 10/31/16 12:11 PM, Daniel Gustafsson wrote: >> I took a small stab at doing some cleaning of the Perl scripts, mainly around >> using the more modern (well, modern as in +15 years old) form for

Re: [HACKERS] Radix tree for character conversion

2016-11-08 Thread Peter Eisentraut
On 10/31/16 12:11 PM, Daniel Gustafsson wrote: > I took a small stab at doing some cleaning of the Perl scripts, mainly around > using the more modern (well, modern as in +15 years old) form for open(..), > avoiding global filehandles for passing scalar references and enforcing use > strict. Some

Re: [HACKERS] Radix tree for character conversion

2016-11-08 Thread Daniel Gustafsson
> On 08 Nov 2016, at 12:21, Kyotaro HORIGUCHI > wrote: > > Hello, this is the revising patch applies on top of the previous > patch. > > ... > > Finally the attached patch contains most of (virtually all of) > Daniel's suggestion and some modification by

Re: [HACKERS] Radix tree for character conversion

2016-11-08 Thread Kyotaro HORIGUCHI
Hello, this is the revising patch applies on top of the previous patch. Differences on map files are enormous but useless for discussion so they aren't included in this. (but can be generated) This still doesn't remove three .txt/.xml files since it heavily bloats the patch. I'm planning that

Re: [HACKERS] Radix tree for character conversion

2016-11-07 Thread Kyotaro HORIGUCHI
Hello, At Mon, 7 Nov 2016 17:19:29 +0100, Daniel Gustafsson wrote in <39e295b9-7391-40b6-911d-fe852e460...@yesql.se> > > On 07 Nov 2016, at 12:32, Daniel Gustafsson wrote: > > > >> On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI > >>

Re: [HACKERS] Radix tree for character conversion

2016-11-07 Thread Kyotaro HORIGUCHI
Hello, At Mon, 7 Nov 2016 12:32:55 +0100, Daniel Gustafsson wrote in > > On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI > > wrote: > > I'm not sure how the discussion about this goes, these patches

Re: [HACKERS] Radix tree for character conversion

2016-11-07 Thread Daniel Gustafsson
> On 07 Nov 2016, at 12:32, Daniel Gustafsson wrote: > >> On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI >> wrote: >> >> I'm not sure how the discussion about this goes, these patches >> makes me think about coding style of Perl. > > Some of

Re: [HACKERS] Radix tree for character conversion

2016-11-07 Thread Daniel Gustafsson
> On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI > wrote: > > Thank you for looling this. And thank you for taking the time to read my patches! > At Mon, 31 Oct 2016 17:11:17 +0100, Daniel Gustafsson wrote > in

Re: [HACKERS] Radix tree for character conversion

2016-11-04 Thread Kyotaro HORIGUCHI
Thank you for looling this. At Mon, 31 Oct 2016 17:11:17 +0100, Daniel Gustafsson wrote in <3fc648b5-2b7f-4585-9615-207a44b73...@yesql.se> > > On 27 Oct 2016, at 09:23, Kyotaro HORIGUCHI > > wrote: > > Perl scripts are to be messy, I believe.

Re: [HACKERS] Radix tree for character conversion

2016-10-31 Thread Daniel Gustafsson
> On 27 Oct 2016, at 09:23, Kyotaro HORIGUCHI > wrote: > > Hello, thank you very much for the work. My work became quite > easier with it. > > At Tue, 25 Oct 2016 12:23:48 +0300, Heikki Linnakangas > wrote in

Re: [HACKERS] Radix tree for character conversion

2016-10-30 Thread Kyotaro HORIGUCHI
Hello, At Fri, 28 Oct 2016 09:42:25 -0400, Tom Lane wrote in <13049.1477662...@sss.pgh.pa.us> > Robert Haas writes: > > On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI > > wrote: > >> Perhaps we can put the files

Re: [HACKERS] Radix tree for character conversion

2016-10-28 Thread David Fetter
On Fri, Oct 28, 2016 at 09:18:08AM -0400, Robert Haas wrote: > On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI > wrote: > > | COPYRIGHT AND PERMISSION NOTICE > > | > > | Copyright (c) 1991-2016 Unicode, Inc. All rights reserved. > > | Distributed under the

Re: [HACKERS] Radix tree for character conversion

2016-10-28 Thread Tom Lane
Robert Haas writes: > On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI > wrote: >> Perhaps we can put the files into our repositoy by providing some >> notifications. > Uggh, I don't much like advertising clauses. Even if the license

Re: [HACKERS] Radix tree for character conversion

2016-10-28 Thread Robert Haas
On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI wrote: > | COPYRIGHT AND PERMISSION NOTICE > | > | Copyright (c) 1991-2016 Unicode, Inc. All rights reserved. > | Distributed under the Terms of Use in http://www.unicode.org/copyright.html. > | > | Permission is

Re: [HACKERS] Radix tree for character conversion

2016-10-07 Thread Heikki Linnakangas
On 10/07/2016 06:55 PM, Robert Haas wrote: On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas wrote: Ouch. We should find and document an authoritative source for all the mappings we have... I think the next steps here are: 1. Find an authoritative source for all the

Re: [HACKERS] Radix tree for character conversion

2016-10-07 Thread Tom Lane
Robert Haas writes: > On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas wrote: >> Ouch. We should find and document an authoritative source for all the >> mappings we have... >> >> I think the next steps here are: >> >> 1. Find an authoritative source

Re: [HACKERS] Radix tree for character conversion

2016-10-07 Thread Robert Haas
On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas wrote: > Ouch. We should find and document an authoritative source for all the > mappings we have... > > I think the next steps here are: > > 1. Find an authoritative source for all the existing mappings. > 2. Generate the radix

Re: [HACKERS] Radix tree for character conversion

2016-10-07 Thread Heikki Linnakangas
On 10/07/2016 11:36 AM, Kyotaro HORIGUCHI wrote: The radix conversion function and map conversion script became more generic than the previous state. So I could easily added radix conversion of EUC_JP in addition to SjiftJIS. nm -S said that the size of radix tree data for sjis->utf8 conversion