Re: [HACKERS] Radix tree for character conversion

2017-03-27 Thread Kyotaro HORIGUCHI
Hmm, things are bit different. At Thu, 23 Mar 2017 12:13:07 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170323.121307.241436413.horiguchi.kyot...@lab.ntt.co.jp> > > Ok, I'll write a small script to generate a set of "conversion > > dump" and try to write README.sanity_check descr

Re: [HACKERS] Radix tree for character conversion

2017-03-22 Thread Kyotaro HORIGUCHI
At Tue, 21 Mar 2017 13:10:48 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170321.131048.150321071.horiguchi.kyot...@lab.ntt.co.jp> > At Fri, 17 Mar 2017 13:03:35 +0200, Heikki Linnakangas > wrote in <01efd334-b839-0450-1b63-f2dea9326...@iki.fi> > > On 03/17/2017 07:19 AM, Kyotaro

Re: [HACKERS] Radix tree for character conversion

2017-03-20 Thread Kyotaro HORIGUCHI
Hello, At Fri, 17 Mar 2017 13:03:35 +0200, Heikki Linnakangas wrote in <01efd334-b839-0450-1b63-f2dea9326...@iki.fi> > On 03/17/2017 07:19 AM, Kyotaro HORIGUCHI wrote: > > I would like to use convert() function. It can be a large > > PL/PgSQL function or a series of "SELECT convert(...)"s. The >

Re: [HACKERS] Radix tree for character conversion

2017-03-17 Thread Heikki Linnakangas
On 03/17/2017 07:19 AM, Kyotaro HORIGUCHI wrote: At Mon, 13 Mar 2017 21:07:39 +0200, Heikki Linnakangas wrote in Hmm. A somewhat different approach might be more suitable for testing across versions, though. We could modify the perl scripts slightly to print out SQL statements that exercise e

Re: [HACKERS] Radix tree for character conversion

2017-03-16 Thread Kyotaro HORIGUCHI
Thank you for committing this. At Mon, 13 Mar 2017 21:07:39 +0200, Heikki Linnakangas wrote in > On 03/13/2017 08:53 PM, Tom Lane wrote: > > Heikki Linnakangas writes: > >> It would be nice to run the map_checker tool one more time, though, to > >> verify that the mappings match those from Pos

Re: [HACKERS] Radix tree for character conversion

2017-03-13 Thread Michael Paquier
On Tue, Mar 14, 2017 at 4:07 AM, Heikki Linnakangas wrote: > On 03/13/2017 08:53 PM, Tom Lane wrote: >> Heikki Linnakangas writes: >>> >>> It would be nice to run the map_checker tool one more time, though, to >>> verify that the mappings match those from PostgreSQL 9.6. >> >> +1 Nice to login a

Re: [HACKERS] Radix tree for character conversion

2017-03-13 Thread Heikki Linnakangas
On 03/13/2017 08:53 PM, Tom Lane wrote: Heikki Linnakangas writes: It would be nice to run the map_checker tool one more time, though, to verify that the mappings match those from PostgreSQL 9.6. +1 Just to be sure, and after that the map checker can go to the dustbin. Hm, maybe we should

Re: [HACKERS] Radix tree for character conversion

2017-03-13 Thread Tom Lane
Heikki Linnakangas writes: > I did some more kibitzing here and there, and committed. Thanks everyone! 111 files changed, 147742 insertions(+), 367346 deletions(-) Nice. > It would be nice to run the map_checker tool one more time, though, to > verify that the mappings match those from Postgre

Re: [HACKERS] Radix tree for character conversion

2017-03-13 Thread Heikki Linnakangas
On 03/06/2017 10:16 AM, Kyotaro HORIGUCHI wrote: At Fri, 3 Mar 2017 12:53:04 +0900, Michael Paquier wrote in On Thu, Mar 2, 2017 at 2:20 PM, Kyotaro HORIGUCHI wrote: 5) Just remove plain map files and all related code. Addition to that, Makefile stores hash digest of authority files in

Re: [HACKERS] Radix tree for character conversion

2017-03-06 Thread Kyotaro HORIGUCHI
Hello, At Fri, 3 Mar 2017 12:53:04 +0900, Michael Paquier wrote in > On Thu, Mar 2, 2017 at 2:20 PM, Kyotaro HORIGUCHI > wrote: > > 5) Just remove plain map files and all related code. Addition to > >that, Makefile stores hash digest of authority files in > >Unicode/authoriy_hashes.txt

Re: [HACKERS] Radix tree for character conversion

2017-03-02 Thread Michael Paquier
On Thu, Mar 2, 2017 at 2:20 PM, Kyotaro HORIGUCHI wrote: > 5) Just remove plain map files and all related code. Addition to >that, Makefile stores hash digest of authority files in >Unicode/authoriy_hashes.txt or something that is managed by >git. That may be an idea to check for diff

Re: [HACKERS] Radix tree for character conversion

2017-03-01 Thread Kyotaro HORIGUCHI
At Wed, 1 Mar 2017 14:34:23 +0900, Michael Paquier wrote in > On Tue, Feb 28, 2017 at 5:34 PM, Kyotaro HORIGUCHI > wrote: > > At Tue, 28 Feb 2017 15:20:06 +0900, Michael Paquier > > wrote in > > > >> +conv.o: conv.c char_converter.c > >> This also can go away. > > > > Touching char_converte

Re: [HACKERS] Radix tree for character conversion

2017-02-28 Thread Michael Paquier
On Tue, Feb 28, 2017 at 5:34 PM, Kyotaro HORIGUCHI wrote: > At Tue, 28 Feb 2017 15:20:06 +0900, Michael Paquier > wrote in > >> +conv.o: conv.c char_converter.c >> This also can go away. > > Touching char_converter.c will be ignored if it is removed. Did > you mistake it for map_checker? That

Re: [HACKERS] Radix tree for character conversion

2017-02-28 Thread Kyotaro HORIGUCHI
At Tue, 28 Feb 2017 15:20:06 +0900, Michael Paquier wrote in > On Mon, Feb 27, 2017 at 5:37 PM, Kyotaro HORIGUCHI > wrote: > > At Wed, 22 Feb 2017 16:06:14 +0900, Michael Paquier > > wrote in > > > >> In order to conduct sanity checks on the shape of the radix tree maps > >> compared to the

Re: [HACKERS] Radix tree for character conversion

2017-02-27 Thread Michael Paquier
On Mon, Feb 27, 2017 at 5:37 PM, Kyotaro HORIGUCHI wrote: > At Wed, 22 Feb 2017 16:06:14 +0900, Michael Paquier > wrote in > >> In order to conduct sanity checks on the shape of the radix tree maps >> compared to the existing maps, having map_checker surely makes sense. >> Now in the final res

Re: [HACKERS] Radix tree for character conversion

2017-02-27 Thread Kyotaro HORIGUCHI
Hello, At Tue, 28 Feb 2017 08:00:22 +0530, Robert Haas wrote in > On Mon, Feb 27, 2017 at 2:07 PM, Kyotaro HORIGUCHI > wrote: > >> +# make_charmap - convert charset table to charmap hash > >> +# with checking duplicate source code > >> Maybe this should be "with checking of duplicated sour

Re: [HACKERS] Radix tree for character conversion

2017-02-27 Thread Robert Haas
On Mon, Feb 27, 2017 at 2:07 PM, Kyotaro HORIGUCHI wrote: >> +# make_charmap - convert charset table to charmap hash >> +# with checking duplicate source code >> Maybe this should be "with checking of duplicated source codes". > > Even though I'm not good English writer, 'duplicated codes' loo

Re: [HACKERS] Radix tree for character conversion

2017-02-27 Thread Kyotaro HORIGUCHI
Thank you for the comment. At Wed, 22 Feb 2017 16:06:14 +0900, Michael Paquier wrote in > Thanks for the rebase. I have been spending sore time looking at this > patch. The new stuff in convutils.pm is by far the interesting part of > the patch, where the building of the radix trees using a byt

Re: [HACKERS] Radix tree for character conversion

2017-02-21 Thread Michael Paquier
On Fri, Feb 3, 2017 at 1:18 PM, Kyotaro HORIGUCHI wrote: > Thanks to that Heikki have pushed the first two patches and a > part of the third, only one patch is remaining now. > > # Sorry for not separating KOI8 stuffs. > > At Tue, 31 Jan 2017 19:06:09 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI

Re: [HACKERS] Radix tree for character conversion

2017-02-02 Thread Kyotaro HORIGUCHI
Tnanks to that Heikki have pushed the first two patches and a part of the third, only one patch is remaining now. # Sorry for not separating KOI8 stuffs. At Tue, 31 Jan 2017 19:06:09 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170131.190609.254672218.horiguchi.kyot...@lab.ntt.co

Re: [HACKERS] Radix tree for character conversion

2017-01-31 Thread Kyotaro HORIGUCHI
At Tue, 31 Jan 2017 12:25:46 +0900, Michael Paquier wrote in > On Mon, Jan 30, 2017 at 3:37 PM, Kyotaro HORIGUCHI > wrote: > > Hello, this is the revised version of character conversion using radix tree. > > Thanks for the new version, I'll look at it once I am done with the > cleanup of the c

Re: [HACKERS] Radix tree for character conversion

2017-01-30 Thread Michael Paquier
On Mon, Jan 30, 2017 at 3:37 PM, Kyotaro HORIGUCHI wrote: > Hello, this is the revised version of character conversion using radix tree. Thanks for the new version, I'll look at it once I am done with the cleanup of the current CF. For now I have moved it to the CF 2017-03. -- Michael -- Sent

Re: [HACKERS] Radix tree for character conversion

2017-01-29 Thread Kyotaro HORIGUCHI
Hello, this is the revised version of character conversion using radix tree. At Fri, 27 Jan 2017 17:33:57 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170127.173357.221584433.horiguchi.kyot...@lab.ntt.co.jp> > Hi, this is an intermediate report without a patch. > > At Thu, 26 Jan

Re: [HACKERS] Radix tree for character conversion

2017-01-27 Thread Kyotaro HORIGUCHI
Hi, this is an intermediate report without a patch. At Thu, 26 Jan 2017 21:42:12 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170126.214212.111556326.horiguchi.kyot...@lab.ntt.co.jp> > > > 0003-Apply-pgperltidy-on-src-backend-utils-mb-Unicode.patch > > > > > > Before adding radi

Re: [HACKERS] Radix tree for character conversion

2017-01-26 Thread Kyotaro HORIGUCHI
Thank you for looking this. At Thu, 26 Jan 2017 16:28:16 +0900, Michael Paquier wrote in > On Tue, Jan 10, 2017 at 8:22 PM, Kyotaro HORIGUCHI > wrote: > > [...patch...] > > Nobody has showed up yet to review this patch, so I am giving it a shot. > > The patch file sizes are scary at first si

Re: [HACKERS] Radix tree for character conversion

2017-01-26 Thread Kyotaro HORIGUCHI
At Thu, 26 Jan 2017 16:29:10 +0900, Michael Paquier wrote in > On Wed, Jan 25, 2017 at 7:18 PM, Ishii Ayumi wrote: > > I patched 4 patchset and run "make", but I got failed. > > Is this a bug or my mistake ? > > I'm sorry if I'm wrong. > > > > [$(TOP)]$ patch -p1 < ../0001-Add-missing-semicolon

Re: [HACKERS] Radix tree for character conversion

2017-01-25 Thread Michael Paquier
On Wed, Jan 25, 2017 at 7:18 PM, Ishii Ayumi wrote: > I patched 4 patchset and run "make", but I got failed. > Is this a bug or my mistake ? > I'm sorry if I'm wrong. > > [$(TOP)]$ patch -p1 < ../0001-Add-missing-semicolon.patch > [$(TOP)]$ patch -p1 < ../0002-Correct-reference-resolution-syntax.p

Re: [HACKERS] Radix tree for character conversion

2017-01-25 Thread Michael Paquier
On Tue, Jan 10, 2017 at 8:22 PM, Kyotaro HORIGUCHI wrote: > [...patch...] Nobody has showed up yet to review this patch, so I am giving it a shot. The patch file sizes are scary at first sight, but after having a look: 36 files changed, 1411 insertions(+), 54398 deletions(-) Yes that's a surpri

Re: [HACKERS] Radix tree for character conversion

2017-01-25 Thread Ishii Ayumi
HI, I patched 4 patchset and run "make", but I got failed. Is this a bug or my mistake ? I'm sorry if I'm wrong. [$(TOP)]$ patch -p1 < ../0001-Add-missing-semicolon.patch [$(TOP)]$ patch -p1 < ../0002-Correct-reference-resolution-syntax.patch [$(TOP)]$ patch -p1 < ../0003-Apply-pgperltidy-on-src-

Re: [HACKERS] Radix tree for character conversion

2016-12-12 Thread Kyotaro HORIGUCHI
Hello, I looked on this closer. The attached is the revised version of this patch. At Mon, 05 Dec 2016 19:29:54 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20161205.192954.12189.horiguchi.kyot...@lab.ntt.co.jp> > Apart from the aboves, I have some trivial comments on the new >

Re: [HACKERS] Radix tree for character conversion

2016-12-05 Thread Kyotaro HORIGUCHI
Hello, thank you for reviewing this. I compared mine and yours. The new patch works fine and gives smaller radix map files. It seems also to me more readable. At Fri, 2 Dec 2016 22:07:07 +0200, Heikki Linnakangas wrote in > On 11/09/2016 10:38 AM, Kyotaro HORIGUCHI wrote: > > Thanks. The attac

Re: [HACKERS] Radix tree for character conversion

2016-12-02 Thread Heikki Linnakangas
On 12/02/2016 10:18 PM, Alvaro Herrera wrote: Heikki Linnakangas wrote: On 11/09/2016 10:38 AM, Kyotaro HORIGUCHI wrote: Thanks. The attached patch contains the patch by perlcritic. 0001,2,3 are Heikki's patch that are not modified since it is first proposed. It's a bit too big so I don't atta

Re: [HACKERS] Radix tree for character conversion

2016-12-02 Thread Alvaro Herrera
Heikki Linnakangas wrote: > On 11/09/2016 10:38 AM, Kyotaro HORIGUCHI wrote: > > Thanks. The attached patch contains the patch by perlcritic. > > > > 0001,2,3 are Heikki's patch that are not modified since it is > > first proposed. It's a bit too big so I don't attach them to this > > mail (again)

Re: [HACKERS] Radix tree for character conversion

2016-12-02 Thread Heikki Linnakangas
On 11/09/2016 10:38 AM, Kyotaro HORIGUCHI wrote: Thanks. The attached patch contains the patch by perlcritic. 0001,2,3 are Heikki's patch that are not modified since it is first proposed. It's a bit too big so I don't attach them to this mail (again). https://www.postgresql.org/message-id/08e78

Re: [HACKERS] Radix tree for character conversion

2016-11-30 Thread Heikki Linnakangas
On 10/31/2016 06:11 PM, Daniel Gustafsson wrote: On 27 Oct 2016, at 09:23, Kyotaro HORIGUCHI wrote: At Tue, 25 Oct 2016 12:23:48 +0300, Heikki Linnakangas wrote in <08e7892a-d55c-eefe-76e6-7910bc8dd...@iki.fi> [..] The perl scripts are still quite messy. For example, I lost the checks for

Re: [HACKERS] Radix tree for character conversion

2016-11-28 Thread Kyotaro HORIGUCHI
Hello. I'll be off line until at least next Monday. So I move this to the next CF by myself. At Wed, 09 Nov 2016 17:38:53 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20161109.173853.77274443.horiguchi.kyot...@lab.ntt.co.jp> > Hello, thank you for polishing this. > > At Wed, 9 Nov

Re: [HACKERS] Radix tree for character conversion

2016-11-09 Thread Kyotaro HORIGUCHI
Hello, thank you for polishing this. At Wed, 9 Nov 2016 02:19:01 +0100, Daniel Gustafsson wrote in <80f34f25-bf6d-4bcd-9c38-42ed10d3f...@yesql.se> > > On 08 Nov 2016, at 17:37, Peter Eisentraut > > wrote: > > > > On 10/31/16 12:11 PM, Daniel Gustafsson wrote: > >> I took a small stab at doing

Re: [HACKERS] Radix tree for character conversion

2016-11-08 Thread Daniel Gustafsson
> On 08 Nov 2016, at 17:37, Peter Eisentraut > wrote: > > On 10/31/16 12:11 PM, Daniel Gustafsson wrote: >> I took a small stab at doing some cleaning of the Perl scripts, mainly around >> using the more modern (well, modern as in +15 years old) form for open(..), >> avoiding global filehandles

Re: [HACKERS] Radix tree for character conversion

2016-11-08 Thread Peter Eisentraut
On 10/31/16 12:11 PM, Daniel Gustafsson wrote: > I took a small stab at doing some cleaning of the Perl scripts, mainly around > using the more modern (well, modern as in +15 years old) form for open(..), > avoiding global filehandles for passing scalar references and enforcing use > strict. Some

Re: [HACKERS] Radix tree for character conversion

2016-11-08 Thread Daniel Gustafsson
> On 08 Nov 2016, at 12:21, Kyotaro HORIGUCHI > wrote: > > Hello, this is the revising patch applies on top of the previous > patch. > > ... > > Finally the attached patch contains most of (virtually all of) > Daniel's suggestion and some modification by pgperltidy. Reading over this it looks

Re: [HACKERS] Radix tree for character conversion

2016-11-08 Thread Kyotaro HORIGUCHI
Hello, this is the revising patch applies on top of the previous patch. Differences on map files are enormous but useless for discussion so they aren't included in this. (but can be generated) This still doesn't remove three .txt/.xml files since it heavily bloats the patch. I'm planning that the

Re: [HACKERS] Radix tree for character conversion

2016-11-07 Thread Kyotaro HORIGUCHI
Hello, At Mon, 7 Nov 2016 17:19:29 +0100, Daniel Gustafsson wrote in <39e295b9-7391-40b6-911d-fe852e460...@yesql.se> > > On 07 Nov 2016, at 12:32, Daniel Gustafsson wrote: > > > >> On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI > >> wrote: > >> > >> I'm not sure how the discussion about this g

Re: [HACKERS] Radix tree for character conversion

2016-11-07 Thread Kyotaro HORIGUCHI
Hello, At Mon, 7 Nov 2016 12:32:55 +0100, Daniel Gustafsson wrote in > > On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI > > wrote: > > I'm not sure how the discussion about this goes, these patches > > makes me think about coding style of Perl. > > Some of this can absolutely be considered styl

Re: [HACKERS] Radix tree for character conversion

2016-11-07 Thread Daniel Gustafsson
> On 07 Nov 2016, at 12:32, Daniel Gustafsson wrote: > >> On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI >> wrote: >> >> I'm not sure how the discussion about this goes, these patches >> makes me think about coding style of Perl. > > Some of this can absolutely be considered style and more or le

Re: [HACKERS] Radix tree for character conversion

2016-11-07 Thread Daniel Gustafsson
> On 04 Nov 2016, at 08:34, Kyotaro HORIGUCHI > wrote: > > Thank you for looling this. And thank you for taking the time to read my patches! > At Mon, 31 Oct 2016 17:11:17 +0100, Daniel Gustafsson wrote > in <3fc648b5-2b7f-4585-9615-207a44b73...@yesql.se> >>> On 27 Oct 2016, at 09:23, Kyotar

Re: [HACKERS] Radix tree for character conversion

2016-11-04 Thread Kyotaro HORIGUCHI
Thank you for looling this. At Mon, 31 Oct 2016 17:11:17 +0100, Daniel Gustafsson wrote in <3fc648b5-2b7f-4585-9615-207a44b73...@yesql.se> > > On 27 Oct 2016, at 09:23, Kyotaro HORIGUCHI > > wrote: > > Perl scripts are to be messy, I believe. Anyway the duplicate > > check as been built into t

Re: [HACKERS] Radix tree for character conversion

2016-10-31 Thread Daniel Gustafsson
> On 27 Oct 2016, at 09:23, Kyotaro HORIGUCHI > wrote: > > Hello, thank you very much for the work. My work became quite > easier with it. > > At Tue, 25 Oct 2016 12:23:48 +0300, Heikki Linnakangas > wrote in <08e7892a-d55c-eefe-76e6-7910bc8dd...@iki.fi> >> >> [..] >> The perl scripts are st

Re: [HACKERS] Radix tree for character conversion

2016-10-30 Thread Kyotaro HORIGUCHI
Hello, At Fri, 28 Oct 2016 09:42:25 -0400, Tom Lane wrote in <13049.1477662...@sss.pgh.pa.us> > Robert Haas writes: > > On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI > > wrote: > >> Perhaps we can put the files into our repositoy by providing some > >> notifications. > > > Uggh, I don't

Re: [HACKERS] Radix tree for character conversion

2016-10-28 Thread David Fetter
On Fri, Oct 28, 2016 at 09:18:08AM -0400, Robert Haas wrote: > On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI > wrote: > > | COPYRIGHT AND PERMISSION NOTICE > > | > > | Copyright (c) 1991-2016 Unicode, Inc. All rights reserved. > > | Distributed under the Terms of Use in > > http://www.unicod

Re: [HACKERS] Radix tree for character conversion

2016-10-28 Thread Tom Lane
Robert Haas writes: > On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI > wrote: >> Perhaps we can put the files into our repositoy by providing some >> notifications. > Uggh, I don't much like advertising clauses. Even if the license were exactly compatible with ours, I'd be -1 on bloating ou

Re: [HACKERS] Radix tree for character conversion

2016-10-28 Thread Robert Haas
On Thu, Oct 27, 2016 at 3:23 AM, Kyotaro HORIGUCHI wrote: > | COPYRIGHT AND PERMISSION NOTICE > | > | Copyright (c) 1991-2016 Unicode, Inc. All rights reserved. > | Distributed under the Terms of Use in http://www.unicode.org/copyright.html. > | > | Permission is hereby granted, free of charge, to

Re: [HACKERS] Radix tree for character conversion

2016-10-07 Thread Heikki Linnakangas
On 10/07/2016 06:55 PM, Robert Haas wrote: On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas wrote: Ouch. We should find and document an authoritative source for all the mappings we have... I think the next steps here are: 1. Find an authoritative source for all the existing mappings. 2. Gen

Re: [HACKERS] Radix tree for character conversion

2016-10-07 Thread Tom Lane
Robert Haas writes: > On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas wrote: >> Ouch. We should find and document an authoritative source for all the >> mappings we have... >> >> I think the next steps here are: >> >> 1. Find an authoritative source for all the existing mappings. >> 2. Gener

Re: [HACKERS] Radix tree for character conversion

2016-10-07 Thread Robert Haas
On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas wrote: > Ouch. We should find and document an authoritative source for all the > mappings we have... > > I think the next steps here are: > > 1. Find an authoritative source for all the existing mappings. > 2. Generate the radix tree files directl

Re: [HACKERS] Radix tree for character conversion

2016-10-07 Thread Heikki Linnakangas
On 10/07/2016 11:36 AM, Kyotaro HORIGUCHI wrote: The radix conversion function and map conversion script became more generic than the previous state. So I could easily added radix conversion of EUC_JP in addition to SjiftJIS. nm -S said that the size of radix tree data for sjis->utf8 conversion