Re: GIN pending list pages not recycled promptly (was Re: [HACKERS] GIN improvements part 1: additional information)

2014-06-18 Thread Amit Langote
On Wed, Jan 22, 2014 at 9:12 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/22/2014 03:39 AM, Tomas Vondra wrote: What annoys me a bit is the huge size difference between the index updated incrementally (by a sequence of INSERT commands), and the index rebuilt from scratch using

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-24 Thread Alexander Korotkov
On Thu, Jan 23, 2014 at 9:27 AM, Alexander Korotkov aekorot...@gmail.comwrote: On Wed, Jan 22, 2014 at 9:28 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/22/2014 02:17 PM, Alexander Korotkov wrote: We already spent a lot of time with compression. Now we need to figure out

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-24 Thread Heikki Linnakangas
On 01/24/2014 10:03 AM, Alexander Korotkov wrote: ITSM I found this bug. ginVacuumPostingTreeLeaf re-encodes only some segments. Others are not even re-palloced. They are moved left in dataPlaceToPageLeafRecompress by memcpy. But it's incorrect to with memcpy, proper solution is memmove. Using

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-24 Thread Alexander Korotkov
On Fri, Jan 24, 2014 at 12:50 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/24/2014 10:03 AM, Alexander Korotkov wrote: ITSM I found this bug. ginVacuumPostingTreeLeaf re-encodes only some segments. Others are not even re-palloced. They are moved left in

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-24 Thread Heikki Linnakangas
On 01/24/2014 10:53 AM, Alexander Korotkov wrote: OK. What about previous fix in assert? Ah right, fixed that too now. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-24 Thread Alexander Korotkov
On Fri, Jan 24, 2014 at 1:19 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/24/2014 10:53 AM, Alexander Korotkov wrote: OK. What about previous fix in assert? Ah right, fixed that too now. Good, now my test-suite passed. Results are so. Time of operations event

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-22 Thread Heikki Linnakangas
On 01/22/2014 09:25 AM, Alexander Korotkov wrote: On Wed, Jan 22, 2014 at 1:21 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: Oh, I see what's going on. I had assumed that there cannot be duplicate insertions into the posting tree,

GIN pending list pages not recycled promptly (was Re: [HACKERS] GIN improvements part 1: additional information)

2014-01-22 Thread Heikki Linnakangas
On 01/22/2014 03:39 AM, Tomas Vondra wrote: What annoys me a bit is the huge size difference between the index updated incrementally (by a sequence of INSERT commands), and the index rebuilt from scratch using VACUUM FULL. It's a bit better with the patch (2288 vs. 2035 MB), but is there a

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-22 Thread Alexander Korotkov
On Wed, Jan 22, 2014 at 12:30 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/22/2014 09:25 AM, Alexander Korotkov wrote: On Wed, Jan 22, 2014 at 1:21 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: Oh, I see

Re: GIN pending list pages not recycled promptly (was Re: [HACKERS] GIN improvements part 1: additional information)

2014-01-22 Thread Alvaro Herrera
Heikki Linnakangas wrote: I wrote a little utility that scans all pages in a gin index, and prints out the flags indicating what kind of a page it is. The distribution looks like this: 19 DATA 7420 DATA LEAF 24701 DELETED 1 LEAF 1 META Hah. I think we need to

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-22 Thread Heikki Linnakangas
On 01/22/2014 02:17 PM, Alexander Korotkov wrote: We already spent a lot of time with compression. Now we need to figure out the result we want see. I spent quite long time debugging varbyte encoding without segments. Finally, it passed very many tests without any problems. Now, it is just piece

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-22 Thread Alexander Korotkov
On Wed, Jan 22, 2014 at 9:28 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/22/2014 02:17 PM, Alexander Korotkov wrote: We already spent a lot of time with compression. Now we need to figure out the result we want see. I spent quite long time debugging varbyte encoding without

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-21 Thread Heikki Linnakangas
On 01/21/2014 04:02 AM, Tomas Vondra wrote: On 20.1.2014 19:30, Heikki Linnakangas wrote: Attached is a yet another version, with more bugs fixed and more comments added and updated. I would appreciate some heavy-testing of this patch now. If you could re-run the tests you've been using, that

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-21 Thread Alexander Korotkov
On Mon, Jan 20, 2014 at 10:30 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/17/2014 08:49 PM, Alexander Korotkov wrote: On Fri, Jan 17, 2014 at 10:38 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/17/2014 01:05 PM, Alexander Korotkov wrote: Seems to be fixed

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-21 Thread Alexander Korotkov
On Tue, Jan 21, 2014 at 4:28 PM, Alexander Korotkov aekorot...@gmail.comwrote: I noticed that the gin vacuum redo routine is dead code, except for the data-leaf page handling, because we never remove entries or internal nodes (page deletion is a separate wal record type). And the data-leaf

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-21 Thread Heikki Linnakangas
On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: On 01/21/2014 04:02 AM, Tomas Vondra wrote: On 20.1.2014 19:30, Heikki Linnakangas wrote: Attached is a yet another version, with more bugs fixed and more comments added and updated. I would appreciate some heavy-testing of this patch now. If

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-21 Thread Tomas Vondra
On 21.1.2014 22:21, Heikki Linnakangas wrote: On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: On 01/21/2014 04:02 AM, Tomas Vondra wrote: On 20.1.2014 19:30, Heikki Linnakangas wrote: Attached is a yet another version, with more bugs fixed and more comments added and updated. I would

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-21 Thread Alexander Korotkov
On Wed, Jan 22, 2014 at 1:21 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: On 01/21/2014 04:02 AM, Tomas Vondra wrote: On 20.1.2014 19:30, Heikki Linnakangas wrote: Attached is a yet another version, with more bugs fixed and more

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-20 Thread Heikki Linnakangas
On 01/17/2014 08:49 PM, Alexander Korotkov wrote: On Fri, Jan 17, 2014 at 10:38 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/17/2014 01:05 PM, Alexander Korotkov wrote: Seems to be fixed in the attached version of patch. Another improvement in this version: only left-most

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-20 Thread Tomas Vondra
On 20.1.2014 19:30, Heikki Linnakangas wrote: Attached is a yet another version, with more bugs fixed and more comments added and updated. I would appreciate some heavy-testing of this patch now. If you could re-run the tests you've been using, that could be great. I've tested the WAL

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-17 Thread Alexander Korotkov
On Wed, Jan 15, 2014 at 10:47 AM, Alexander Korotkov aekorot...@gmail.comwrote: On Wed, Jan 15, 2014 at 5:17 AM, Tomas Vondra t...@fuzzy.cz wrote: On 14.1.2014 00:38, Tomas Vondra wrote: On 13.1.2014 18:07, Alexander Korotkov wrote: On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-17 Thread Heikki Linnakangas
On 01/17/2014 01:05 PM, Alexander Korotkov wrote: Seems to be fixed in the attached version of patch. Another improvement in this version: only left-most segments and further are re-encoded. Left part of page are left untouched. I'm looking into this now. A few quick notes: * Even when you

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-17 Thread Alexander Korotkov
On Fri, Jan 17, 2014 at 10:38 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/17/2014 01:05 PM, Alexander Korotkov wrote: Seems to be fixed in the attached version of patch. Another improvement in this version: only left-most segments and further are re-encoded. Left part of

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-14 Thread Heikki Linnakangas
On 01/13/2014 07:07 PM, Alexander Korotkov wrote: I've fixed this bug and many other bug. Now patch passes test suite that I've used earlier. The results are so: Operations time: event | period ---+- index_build |

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-14 Thread Alexander Korotkov
On Tue, Jan 14, 2014 at 12:34 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/13/2014 07:07 PM, Alexander Korotkov wrote: I've fixed this bug and many other bug. Now patch passes test suite that I've used earlier. The results are so: Operations time: event |

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-14 Thread Tomas Vondra
On 14.1.2014 00:38, Tomas Vondra wrote: On 13.1.2014 18:07, Alexander Korotkov wrote: On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra t...@fuzzy.cz mailto:t...@fuzzy.cz wrote: On 8.1.2014 22:58, Alexander Korotkov wrote: Thanks for reporting. Fixed version is attached. I've

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-14 Thread Alexander Korotkov
On Wed, Jan 15, 2014 at 5:17 AM, Tomas Vondra t...@fuzzy.cz wrote: On 14.1.2014 00:38, Tomas Vondra wrote: On 13.1.2014 18:07, Alexander Korotkov wrote: On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra t...@fuzzy.cz mailto:t...@fuzzy.cz wrote: On 8.1.2014 22:58, Alexander Korotkov

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-13 Thread Alexander Korotkov
On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra t...@fuzzy.cz wrote: On 8.1.2014 22:58, Alexander Korotkov wrote: Thanks for reporting. Fixed version is attached. I've tried to rerun the 'archie' benchmark with the current patch, and once again I got PANIC: could not split GIN page,

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-13 Thread Tomas Vondra
On 13.1.2014 18:07, Alexander Korotkov wrote: On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra t...@fuzzy.cz mailto:t...@fuzzy.cz wrote: On 8.1.2014 22:58, Alexander Korotkov wrote: Thanks for reporting. Fixed version is attached. I've tried to rerun the 'archie' benchmark with

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-13 Thread Etsuro Fujita
Peter Eisentraut wrote: On 12/10/13, 2:44 PM, Alexander Korotkov wrote: However, patch didn't apply to head. Corrected version is attached. Update the pgstattuple regression test results. The latest version of the patch still doesn't pass the test. I'll look at the patch in further detail.

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-10 Thread Tomas Vondra
On 8.1.2014 22:58, Alexander Korotkov wrote: Thanks for reporting. Fixed version is attached. I've tried to rerun the 'archie' benchmark with the current patch, and once again I got PANIC: could not split GIN page, didn't fit I reran it with '--enable-cassert' and with that I got TRAP:

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-08 Thread Alexander Korotkov
On Mon, Jan 6, 2014 at 12:35 PM, Amit Langote amitlangot...@gmail.comwrote: On Sat, Dec 21, 2013 at 4:36 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Yet another version. The encoding/decoding code is now quite isolated in ginpostinglist.c, so it's easy to experiment with

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-06 Thread Amit Langote
On Sat, Dec 21, 2013 at 4:36 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Yet another version. The encoding/decoding code is now quite isolated in ginpostinglist.c, so it's easy to experiment with different encodings. This patch uses varbyte encoding again. I got a bit carried away,

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-24 Thread Peter Eisentraut
On 12/10/13, 2:44 PM, Alexander Korotkov wrote: However, patch didn't apply to head. Corrected version is attached. Update the pgstattuple regression test results. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-20 Thread Heikki Linnakangas
On 12/19/2013 10:44 AM, Heikki Linnakangas wrote: On 12/19/2013 08:37 AM, Oleg Bartunov wrote: Guys, before digging deep into the art of comp/decomp world I'd like to know if you familiar with results of http://wwwconference.org/www2008/papers/pdf/p387-zhangA.pdf paper and some newer research

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-20 Thread Heikki Linnakangas
On 12/19/2013 03:33 PM, Heikki Linnakangas wrote: On 12/17/2013 12:49 AM, Heikki Linnakangas wrote: On 12/17/2013 12:22 AM, Alexander Korotkov wrote: On Mon, Dec 16, 2013 at 3:30 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/12/2013 06:44 PM, Alexander Korotkov wrote: When

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-20 Thread Alvaro Herrera
Heikki Linnakangas escribió: I believe that eliminates all encodings in the Simple family, as well as PForDelta, and surprisingly also Rice encoding. For example, if you have three items in consecutive offsets, the differences between them are encoded as 11 in rice encoding. If you remove the

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-20 Thread Alexander Korotkov
On Fri, Dec 20, 2013 at 11:43 PM, Alvaro Herrera alvhe...@2ndquadrant.comwrote: Heikki Linnakangas escribió: I believe that eliminates all encodings in the Simple family, as well as PForDelta, and surprisingly also Rice encoding. For example, if you have three items in consecutive

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-20 Thread Alvaro Herrera
Alexander Korotkov escribió: On Fri, Dec 20, 2013 at 11:43 PM, Alvaro Herrera alvhe...@2ndquadrant.comwrote: Heikki Linnakangas escribió: I believe that eliminates all encodings in the Simple family, as well as PForDelta, and surprisingly also Rice encoding. For example, if you

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-20 Thread Alexander Korotkov
On Fri, Dec 20, 2013 at 11:36 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/19/2013 03:33 PM, Heikki Linnakangas wrote: On 12/17/2013 12:49 AM, Heikki Linnakangas wrote: On 12/17/2013 12:22 AM, Alexander Korotkov wrote: On Mon, Dec 16, 2013 at 3:30 PM, Heikki Linnakangas

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-19 Thread Heikki Linnakangas
On 12/19/2013 08:37 AM, Oleg Bartunov wrote: Guys, before digging deep into the art of comp/decomp world I'd like to know if you familiar with results of http://wwwconference.org/www2008/papers/pdf/p387-zhangA.pdf paper and some newer research ? Yeah, I saw that paper. Do we agree in what

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-19 Thread Heikki Linnakangas
On 12/17/2013 12:49 AM, Heikki Linnakangas wrote: On 12/17/2013 12:22 AM, Alexander Korotkov wrote: On Mon, Dec 16, 2013 at 3:30 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/12/2013 06:44 PM, Alexander Korotkov wrote: When values are packed into small groups, we have to

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-18 Thread Alexander Korotkov
On Tue, Dec 17, 2013 at 2:49 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/17/2013 12:22 AM, Alexander Korotkov wrote: On Mon, Dec 16, 2013 at 3:30 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/12/2013 06:44 PM, Alexander Korotkov wrote: When values are

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-18 Thread Heikki Linnakangas
On 12/18/2013 01:45 PM, Alexander Korotkov wrote: On Tue, Dec 17, 2013 at 2:49 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/17/2013 12:22 AM, Alexander Korotkov wrote: 2) Storage would be easily extendable to hold additional information as well. Better compression shouldn't

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-18 Thread Oleg Bartunov
Guys, before digging deep into the art of comp/decomp world I'd like to know if you familiar with results of http://wwwconference.org/www2008/papers/pdf/p387-zhangA.pdf paper and some newer research ? Do we agree in what we really want ? Basically, there are three main features: size,

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-16 Thread Heikki Linnakangas
On 12/12/2013 06:44 PM, Alexander Korotkov wrote: I've thought about different algorithms little more. General problem I see is online update. We need it while it is typically not covered by researches at all. We already have to invent small index in the end of page. Different encoding methods

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-16 Thread Alexander Korotkov
On Mon, Dec 16, 2013 at 3:30 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/12/2013 06:44 PM, Alexander Korotkov wrote: I've thought about different algorithms little more. General problem I see is online update. We need it while it is typically not covered by researches at

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-16 Thread Heikki Linnakangas
On 12/17/2013 12:22 AM, Alexander Korotkov wrote: On Mon, Dec 16, 2013 at 3:30 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/12/2013 06:44 PM, Alexander Korotkov wrote: When values are packed into small groups, we have to either insert inefficiently encoded value or

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-12 Thread Alexander Korotkov
On Tue, Dec 10, 2013 at 12:26 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/09/2013 11:34 AM, Alexander Korotkov wrote: On Mon, Dec 9, 2013 at 1:18 PM, Heikki Linnakangas hlinnakan...@vmware.comwrote: Even if we use varbyte encoding, I wonder if it would be better to treat

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-10 Thread Alexander Korotkov
On Tue, Dec 10, 2013 at 12:26 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 12/09/2013 11:34 AM, Alexander Korotkov wrote: On Mon, Dec 9, 2013 at 1:18 PM, Heikki Linnakangas hlinnakan...@vmware.comwrote: Even if we use varbyte encoding, I wonder if it would be better to treat

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-09 Thread Heikki Linnakangas
On 12/08/2013 09:56 PM, Alexander Korotkov wrote: On Fri, Nov 29, 2013 at 11:17 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I'll continue reviewing next week.. Got dragged into other things and didn't make any progress on this last week. I'm trying again now. Good. Thanks for

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-09 Thread Alexander Korotkov
On Mon, Dec 9, 2013 at 1:18 PM, Heikki Linnakangas hlinnakan...@vmware.comwrote: On 12/08/2013 09:56 PM, Alexander Korotkov wrote: On Fri, Nov 29, 2013 at 11:17 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I'll continue reviewing next week.. Got dragged into other things and

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-09 Thread Heikki Linnakangas
On 12/09/2013 11:34 AM, Alexander Korotkov wrote: On Mon, Dec 9, 2013 at 1:18 PM, Heikki Linnakangas hlinnakan...@vmware.comwrote: Even if we use varbyte encoding, I wonder if it would be better to treat block + offset number as a single 48-bit integer, rather than encode them separately. That

Re: [HACKERS] GIN improvements part 1: additional information

2013-12-08 Thread Alexander Korotkov
On Fri, Nov 29, 2013 at 11:17 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 11/29/2013 11:41 AM, Heikki Linnakangas wrote: On 11/28/2013 09:19 AM, Alexander Korotkov wrote: On Wed, Nov 27, 2013 at 1:14 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 11/26/13

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-29 Thread Heikki Linnakangas
On 11/29/2013 11:41 AM, Heikki Linnakangas wrote: On 11/28/2013 09:19 AM, Alexander Korotkov wrote: On Wed, Nov 27, 2013 at 1:14 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 11/26/13 15:34, Alexander Korotkov wrote: What's your plans about GIN now? I tried to rebase packed

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-27 Thread Alexander Korotkov
On Wed, Nov 27, 2013 at 1:14 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 11/26/13 15:34, Alexander Korotkov wrote: What's your plans about GIN now? I tried to rebase packed posting lists with head. But I found that you've changed interface of placeToPage function. That's

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-26 Thread Alexander Korotkov
Hi! On Wed, Nov 20, 2013 at 9:02 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 06.11.2013 17:36, Alvaro Herrera wrote: Just for my own illumination, can someone explain this bit? + If a posting list is too large to store in-line in a key entry, a posting tree + is created. A

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-26 Thread Alexander Korotkov
On Tue, Nov 26, 2013 at 5:34 PM, Alexander Korotkov aekorot...@gmail.comwrote: On Wed, Nov 20, 2013 at 9:02 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 06.11.2013 17:36, Alvaro Herrera wrote: Just for my own illumination, can someone explain this bit? + If a posting list is

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-26 Thread Heikki Linnakangas
On 11/26/13 15:34, Alexander Korotkov wrote: What's your plans about GIN now? I tried to rebase packed posting lists with head. But I found that you've changed interface of placeToPage function. That's conflicts with packed posting lists, because dataPlaceToPageLeaf needs not only offset number

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-26 Thread Peter Eisentraut
This patch needs to be rebased. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-20 Thread Heikki Linnakangas
On 06.11.2013 17:36, Alvaro Herrera wrote: Just for my own illumination, can someone explain this bit? + If a posting list is too large to store in-line in a key entry, a posting tree + is created. A posting tree is a B-tree structure, where the ItemPointer is + used as the key. At the

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-15 Thread Peter Eisentraut
On 11/14/13, 6:00 AM, Alexander Korotkov wrote: Sorry, I posted buggy version of patch. Attached version is correct. This patch crashes the hstore the pg_trgm regression tests. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-15 Thread Alexander Korotkov
On Fri, Nov 15, 2013 at 8:58 PM, Peter Eisentraut pete...@gmx.net wrote: On 11/14/13, 6:00 AM, Alexander Korotkov wrote: Sorry, I posted buggy version of patch. Attached version is correct. This patch crashes the hstore the pg_trgm regression tests. What exact version did you try 14 or 16?

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-15 Thread Peter Eisentraut
On 11/15/13, 12:24 PM, Alexander Korotkov wrote: On Fri, Nov 15, 2013 at 8:58 PM, Peter Eisentraut pete...@gmx.net mailto:pete...@gmx.net wrote: On 11/14/13, 6:00 AM, Alexander Korotkov wrote: Sorry, I posted buggy version of patch. Attached version is correct. This patch

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-15 Thread Alexander Korotkov
On Fri, Nov 15, 2013 at 9:56 PM, Peter Eisentraut pete...@gmx.net wrote: On 11/15/13, 12:24 PM, Alexander Korotkov wrote: On Fri, Nov 15, 2013 at 8:58 PM, Peter Eisentraut pete...@gmx.net mailto:pete...@gmx.net wrote: On 11/14/13, 6:00 AM, Alexander Korotkov wrote: Sorry, I

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-14 Thread Alexander Korotkov
On Tue, Nov 5, 2013 at 9:49 PM, Heikki Linnakangas hlinnakan...@vmware.comwrote: On 04.11.2013 23:44, Alexander Korotkov wrote: On Mon, Oct 21, 2013 at 11:12 PM, Alexander Korotkov aekorot...@gmail.comwrote: Attached version of patch is debugged. I would like to note that number of bugs

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-14 Thread Alexander Korotkov
On Thu, Nov 14, 2013 at 2:17 PM, Alexander Korotkov aekorot...@gmail.comwrote: On Tue, Nov 5, 2013 at 9:49 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 04.11.2013 23:44, Alexander Korotkov wrote: On Mon, Oct 21, 2013 at 11:12 PM, Alexander Korotkov aekorot...@gmail.comwrote:

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-14 Thread Alexander Korotkov
On Thu, Nov 14, 2013 at 3:00 PM, Alexander Korotkov aekorot...@gmail.comwrote: On Thu, Nov 14, 2013 at 2:17 PM, Alexander Korotkov aekorot...@gmail.comwrote: On Tue, Nov 5, 2013 at 9:49 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 04.11.2013 23:44, Alexander Korotkov wrote:

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-06 Thread Heikki Linnakangas
On 04.11.2013 23:44, Alexander Korotkov wrote: Attached version of patch has support of old page format. Patch still needs more documentation and probably refactoring, but I believe idea is clear and it can be reviewed. In the patch I have to revert change of null category placement for

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-06 Thread Alvaro Herrera
Heikki Linnakangas escribió: On 04.11.2013 23:44, Alexander Korotkov wrote: Attached version of patch has support of old page format. Patch still needs more documentation and probably refactoring, but I believe idea is clear and it can be reviewed. In the patch I have to revert change of null

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-05 Thread Heikki Linnakangas
On 04.11.2013 23:44, Alexander Korotkov wrote: On Mon, Oct 21, 2013 at 11:12 PM, Alexander Korotkov aekorot...@gmail.comwrote: Attached version of patch is debugged. I would like to note that number of bugs was low and it wasn't very hard to debug. I've rerun tests on it. You can see that

Re: [HACKERS] GIN improvements part 1: additional information

2013-11-04 Thread Alexander Korotkov
On Mon, Oct 21, 2013 at 11:12 PM, Alexander Korotkov aekorot...@gmail.comwrote: Attached version of patch is debugged. I would like to note that number of bugs was low and it wasn't very hard to debug. I've rerun tests on it. You can see that numbers are improved as the result of your

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-21 Thread Alexander Korotkov
On Thu, Oct 10, 2013 at 3:57 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 09.10.2013 02:04, Tomas Vondra wrote: On 8.10.2013 21:59, Heikki Linnakangas wrote: On 08.10.2013 17:47, Alexander Korotkov wrote: Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondrat...@fuzzy.cz

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-20 Thread Bruce Momjian
On Sat, Oct 19, 2013 at 05:11:55PM -0400, Bruce Momjian wrote: I am in Moscow with Alexander and we were discussing GIN pg_upgrade issues. One option we have already discussed was to take the old GIN index code and put it in a separate directory, and call the new GIN index something

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-19 Thread Bruce Momjian
On Thu, Oct 3, 2013 at 05:24:49PM -0400, Bruce Momjian wrote: On Thu, Oct 3, 2013 at 02:48:20PM -0400, Robert Haas wrote: On Thu, Oct 3, 2013 at 2:43 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: It seems we've all but decided that we'll require reindexing GIN indexes in

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-12 Thread Alexander Korotkov
On Sat, Oct 12, 2013 at 1:55 AM, Tomas Vondra t...@fuzzy.cz wrote: On 10.10.2013 13:57, Heikki Linnakangas wrote: On 09.10.2013 02:04, Tomas Vondra wrote: On 8.10.2013 21:59, Heikki Linnakangas wrote: On 08.10.2013 17:47, Alexander Korotkov wrote: Hi, Tomas! On Sun, Oct 6, 2013 at

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-12 Thread Tomas Vondra
On 12.10.2013 12:11, Alexander Korotkov wrote: On Sat, Oct 12, 2013 at 1:55 AM, Tomas Vondra t...@fuzzy.cz mailto:t...@fuzzy.cz wrote: Yup, this version fixed the issues. I haven't been able to do any benchmarks yet, all I have is some basic stats | HEAD |

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-11 Thread Tomas Vondra
On 10.10.2013 13:57, Heikki Linnakangas wrote: On 09.10.2013 02:04, Tomas Vondra wrote: On 8.10.2013 21:59, Heikki Linnakangas wrote: On 08.10.2013 17:47, Alexander Korotkov wrote: Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondrat...@fuzzy.cz wrote: I've attempted to rerun the

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-10 Thread Heikki Linnakangas
On 09.10.2013 02:04, Tomas Vondra wrote: On 8.10.2013 21:59, Heikki Linnakangas wrote: On 08.10.2013 17:47, Alexander Korotkov wrote: Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondrat...@fuzzy.cz wrote: I've attempted to rerun the benchmarks tests I did a few weeks ago, but I

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-10 Thread Peter Eisentraut
You linked to this email from the commitfest entry, but there is no patch here. You probably meant a different email. Check please. On Tue, 2013-10-08 at 21:48 +0300, Heikki Linnakangas wrote: On 04.10.2013 14:13, Alexander Korotkov wrote: On Fri, Oct 4, 2013 at 12:28 PM, Heikki

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-08 Thread Alexander Korotkov
Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondra t...@fuzzy.cz wrote: I've attempted to rerun the benchmarks tests I did a few weeks ago, but I got repeated crashes when loading the data (into a table with tsvector+gin index). Right before a crash, theres this message in the log:

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-08 Thread Heikki Linnakangas
On 04.10.2013 14:13, Alexander Korotkov wrote: On Fri, Oct 4, 2013 at 12:28 PM, Heikki Linnakangashlinnakan...@vmware.com wrote: In the attached patch, I in fact already did that for data leaf pages, but didn't change the format of non-leaf pages yet. If we want to support pg_upgrade, we

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-08 Thread Heikki Linnakangas
On 08.10.2013 17:47, Alexander Korotkov wrote: Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondrat...@fuzzy.cz wrote: I've attempted to rerun the benchmarks tests I did a few weeks ago, but I got repeated crashes when loading the data (into a table with tsvector+gin index). Right

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-08 Thread Tomas Vondra
On 8.10.2013 21:59, Heikki Linnakangas wrote: On 08.10.2013 17:47, Alexander Korotkov wrote: Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondrat...@fuzzy.cz wrote: I've attempted to rerun the benchmarks tests I did a few weeks ago, but I got repeated crashes when loading the data

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-04 Thread Heikki Linnakangas
Aside from the pg_upgrade discussion, here's an updated version of the patch, rebased over master. It also contains some other misc refactoring I've done while reading through the patch. I haven't tested this much, I may well have also broken something, but I wanted to post an update before

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-04 Thread Alexander Korotkov
On Fri, Oct 4, 2013 at 12:28 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Aside from the pg_upgrade discussion, here's an updated version of the patch, rebased over master. It also contains some other misc refactoring I've done while reading through the patch. I haven't tested this

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-03 Thread Heikki Linnakangas
On 23.09.2013 18:35, Bruce Momjian wrote: On Sun, Sep 15, 2013 at 01:14:45PM +0400, Alexander Korotkov wrote: On Sat, Jun 29, 2013 at 12:56 PM, Heikki Linnakangashlinnakan...@vmware.com wrote: There's a few open questions: 1. How are we going to handle pg_upgrade? It would be nice

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-03 Thread Robert Haas
On Thu, Oct 3, 2013 at 2:43 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: It seems we've all but decided that we'll require reindexing GIN indexes in 9.4. I thought the consensus in Ottawa was strongly against that. I'm not aware that anyone has subsequently changed their position on

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-03 Thread Alexander Korotkov
On Thu, Oct 3, 2013 at 10:48 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Oct 3, 2013 at 2:43 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: It seems we've all but decided that we'll require reindexing GIN indexes in 9.4. I thought the consensus in Ottawa was strongly

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-03 Thread Heikki Linnakangas
On 03.10.2013 23:37, Alexander Korotkov wrote: 2) Insert kluges into GIN to support both old and new formats. So, kluges are kluges :) I don't see elegant way to do it for now, because formats are very different. Hmm. All you need is some code to read the old format, and a function to convert

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-03 Thread Alexander Korotkov
On Fri, Oct 4, 2013 at 12:41 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 03.10.2013 23:37, Alexander Korotkov wrote: 2) Insert kluges into GIN to support both old and new formats. So, kluges are kluges :) I don't see elegant way to do it for now, because formats are very

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-03 Thread Bruce Momjian
On Thu, Oct 3, 2013 at 02:48:20PM -0400, Robert Haas wrote: On Thu, Oct 3, 2013 at 2:43 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: It seems we've all but decided that we'll require reindexing GIN indexes in 9.4. I thought the consensus in Ottawa was strongly against that. I'm

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-03 Thread Alvaro Herrera
Bruce Momjian escribió: Agreed. I was stating only that this is easy for pg_upgrade. One cool thing is that the upgrades completes, and the indexes are there, but just marked as invalid until the REINDEX. One other point Alexander made is that the new GIN indexes will be smaller so most

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-03 Thread Heikki Linnakangas
On 03.10.2013 23:54, Alexander Korotkov wrote: ItemPointers compression reduce occupied space in all normal cases. It's not very realistic, but it could increase space in worst case. That would lead to page split after conversion. Are we going to support such case? Hmm, that's probably rare

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-03 Thread Alexander Korotkov
On Fri, Oct 4, 2013 at 12:37 AM, Alexander Korotkov aekorot...@gmail.comwrote: On Thu, Oct 3, 2013 at 10:48 PM, Robert Haas robertmh...@gmail.comwrote: On Thu, Oct 3, 2013 at 2:43 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: It seems we've all but decided that we'll require

Re: [HACKERS] GIN improvements part 1: additional information

2013-10-03 Thread Bruce Momjian
On Fri, Oct 4, 2013 at 02:23:33AM +0400, Alexander Korotkov wrote: I came to idea that I like option #4 more than option #2. If we try to make new GIN work with old page formats we have to maintain 3 use cases: 1) old GIN with old page format (because of old releases) 2) new GIN with old

Re: [HACKERS] GIN improvements part 1: additional information

2013-09-26 Thread Alexander Korotkov
On Wed, Sep 25, 2013 at 5:22 PM, Peter Eisentraut pete...@gmx.net wrote: On 9/23/13 5:36 PM, Alexander Korotkov wrote: In the attached version of patch double finding of ItemPointer during insert is avoided. Overhead becomes lower as expected. Fails cpluspluscheck:

Re: [HACKERS] GIN improvements part 1: additional information

2013-09-25 Thread Peter Eisentraut
On 9/23/13 5:36 PM, Alexander Korotkov wrote: In the attached version of patch double finding of ItemPointer during insert is avoided. Overhead becomes lower as expected. Fails cpluspluscheck: ./src/include/access/gin_private.h: In function ‘char* ginDataPageLeafReadItemPointer(char*,

Re: [HACKERS] GIN improvements part 1: additional information

2013-09-23 Thread Bruce Momjian
On Sun, Sep 15, 2013 at 01:14:45PM +0400, Alexander Korotkov wrote: On Sat, Jun 29, 2013 at 12:56 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: There's a few open questions: 1. How are we going to handle pg_upgrade? It would be nice to be able to read the old page

Re: [HACKERS] GIN improvements part 1: additional information

2013-09-23 Thread Alexander Korotkov
On Mon, Sep 23, 2013 at 12:47 AM, Alexander Korotkov aekorot...@gmail.comwrote: It's probably an option to select 64 entries instead of 32. There is still some regression in update speed. However, there is also room for improvement patch. It searches item index entries 2 times on insert: in

  1   2   >