Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Tue, Aug 11, 2015 at 2:58 AM, Jeff Janes wrote: > On Thu, Oct 30, 2014 at 5:30 AM, Fujii Masao wrote: >> >> On Thu, Oct 30, 2014 at 7:30 PM, Etsuro Fujita >> wrote: >> >> > + { >> > +

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Thu, Oct 30, 2014 at 5:30 AM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Oct 30, 2014 at 7:30 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: + { + {pending_list_cleanup_size, PGC_USERSET, CLIENT_CONN_STATEMENT, +

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Wed, Nov 12, 2014 at 12:40 AM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: On Tue, Nov 11, 2014 at 10:14 PM, Robert Haas robertmh...@gmail.com wrote: Not to kibitz too much after-the-fact, but wouldn't it be better to give this a name that has GIN in it

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Wed, Nov 12, 2014 at 9:30 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Nov 12, 2014 at 12:40 AM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: On Tue, Nov 11, 2014 at 10:14 PM, Robert Haas robertmh...@gmail.com wrote: Not to kibitz too much

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

(2014/11/11 2:31), Fujii Masao wrote: On Mon, Nov 10, 2014 at 4:15 PM, Etsuro Fujita The patch looks good to me except for the following point: *** a/src/backend/access/gin/ginfast.c --- b/src/backend/access/gin/ginfast.c *** *** 25,30 --- 25,32 #include

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Tue, Nov 11, 2014 at 7:01 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: (2014/11/11 2:31), Fujii Masao wrote: On Mon, Nov 10, 2014 at 4:15 PM, Etsuro Fujita The patch looks good to me except for the following point: *** a/src/backend/access/gin/ginfast.c ---

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Tue, Nov 11, 2014 at 7:11 AM, Fujii Masao masao.fu...@gmail.com wrote: OK, so if there are no objections of others, I'll mark this as Ready for Committer. I just pushed this. Thanks! Not to kibitz too much after-the-fact, but wouldn't it be better to give this a name that has GIN in it

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Tue, Nov 11, 2014 at 10:14 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Nov 11, 2014 at 7:11 AM, Fujii Masao masao.fu...@gmail.com wrote: OK, so if there are no objections of others, I'll mark this as Ready for Committer. I just pushed this. Thanks! Not to kibitz too much

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

Fujii Masao masao.fu...@gmail.com writes: On Tue, Nov 11, 2014 at 10:14 PM, Robert Haas robertmh...@gmail.com wrote: Not to kibitz too much after-the-fact, but wouldn't it be better to give this a name that has GIN in it somewhere? Maybe. gin_pending_list_cleanup_size? gin_pending_list_limit?

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Mon, Nov 10, 2014 at 4:15 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: (2014/11/06 23:38), Fujii Masao wrote: On Tue, Nov 4, 2014 at 12:04 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: IIUC, I think that min = 0 disables fast update, so ISTM that it'd be appropriate to set

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

(2014/11/06 23:38), Fujii Masao wrote: On Tue, Nov 4, 2014 at 12:04 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: IIUC, I think that min = 0 disables fast update, so ISTM that it'd be appropriate to set min to some positive value. And ISTM that the idea of using the min value of

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Tue, Nov 4, 2014 at 12:04 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: IIUC, I think that min = 0 disables fast update, so ISTM that it'd be appropriate to set min to some positive value. And ISTM that the idea of using the min value of work_mem is not so bad. OK. I changed the min

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

(2014/10/30 21:30), Fujii Masao wrote: On Thu, Oct 30, 2014 at 7:30 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: Here are my review comments. * The patch applies cleanly and make and make check run successfully. I think that the patch is mostly good. Thanks! Attached is the

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

(2014/10/09 11:49), Etsuro Fujita wrote: (2014/10/08 22:51), Fujii Masao wrote: On Wed, Sep 24, 2014 at 11:10 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: On Wed, Sep 10, 2014 at 10:37 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Wed, Sep 10, 2014 at 12:15

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Thu, Oct 30, 2014 at 7:30 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: (2014/10/09 11:49), Etsuro Fujita wrote: (2014/10/08 22:51), Fujii Masao wrote: On Wed, Sep 24, 2014 at 11:10 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: On Wed, Sep 10, 2014 at 10:37 PM, Alvaro

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Wed, Sep 24, 2014 at 11:10 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: (2014/09/13 2:42), Fujii Masao wrote: On Wed, Sep 10, 2014 at 10:37 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Wed, Sep 10, 2014 at 12:15 PM, Etsuro Fujita

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

(2014/10/08 22:51), Fujii Masao wrote: On Wed, Sep 24, 2014 at 11:10 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: On Wed, Sep 10, 2014 at 10:37 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Wed, Sep 10, 2014 at 12:15 PM, Etsuro Fujita

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

(2014/09/13 2:42), Fujii Masao wrote: On Wed, Sep 10, 2014 at 10:37 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Wed, Sep 10, 2014 at 12:15 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: PENDING_LIST_CLEANUP_SIZE and work_mem, for this setting. Wouldn't it

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Wed, Sep 10, 2014 at 10:37 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Wed, Sep 10, 2014 at 12:15 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: PENDING_LIST_CLEANUP_SIZE and work_mem, for this setting. Wouldn't it be easy-to-use to have only one

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

(2014/09/10 12:31), Fujii Masao wrote: On Wed, Sep 10, 2014 at 12:15 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: (2014/09/09 22:17), Fujii Masao wrote: Attached is the updated version of the patch. I took a quick review on the patch. It looks good to me, but one thing I'm

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Wed, Sep 10, 2014 at 5:35 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: (2014/09/10 12:31), Fujii Masao wrote: On Wed, Sep 10, 2014 at 12:15 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: (2014/09/09 22:17), Fujii Masao wrote: Attached is the updated version of the patch.

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

Fujii Masao wrote: On Wed, Sep 10, 2014 at 12:15 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: PENDING_LIST_CLEANUP_SIZE and work_mem, for this setting. Wouldn't it be easy-to-use to have only one parameter, PENDING_LIST_CLEANUP_SIZE? How about setting PENDING_LIST_CLEANUP_SIZE to

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Tue, Sep 9, 2014 at 1:28 AM, Jeff Janes jeff.ja...@gmail.com wrote: On Sun, Aug 17, 2014 at 7:46 PM, Fujii Masao masao.fu...@gmail.com wrote: Thanks for reviewing the patch! ISTM that I failed to make the patch from my git repository... Attached is the rebased version. I get some

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

(2014/09/09 22:17), Fujii Masao wrote: On Tue, Sep 9, 2014 at 1:28 AM, Jeff Janes jeff.ja...@gmail.com wrote: I get some compiler warnings on v2 of this patch: reloptions.c:219: warning: excess elements in struct initializer reloptions.c:219: warning: (near initialization for 'intRelOpts[15]')

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Wed, Sep 10, 2014 at 12:15 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: (2014/09/09 22:17), Fujii Masao wrote: On Tue, Sep 9, 2014 at 1:28 AM, Jeff Janes jeff.ja...@gmail.com wrote: I get some compiler warnings on v2 of this patch: reloptions.c:219: warning: excess elements in

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Sun, Aug 17, 2014 at 7:46 PM, Fujii Masao masao.fu...@gmail.com wrote: Thanks for reviewing the patch! ISTM that I failed to make the patch from my git repository... Attached is the rebased version. I get some compiler warnings on v2 of this patch: reloptions.c:219: warning: excess

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Sat, Aug 16, 2014 at 4:23 PM, Sawada Masahiko sawada.m...@gmail.com wrote: On Fri, Aug 8, 2014 at 11:45 PM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Aug 8, 2014 at 11:43 PM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Apr 14, 2014 at 11:40 PM, Tom Lane t...@sss.pgh.pa.us wrote:

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Fri, Aug 8, 2014 at 11:45 PM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Aug 8, 2014 at 11:43 PM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Apr 14, 2014 at 11:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: On Tue, Apr 1, 2014 at 1:41 AM,

PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Mon, Apr 14, 2014 at 11:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: On Tue, Apr 1, 2014 at 1:41 AM, Robert Haas robertmh...@gmail.com wrote: Should we try to install some hack around fastupdate for 9.4? I fear the divergence between reasonable values

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Fri, Aug 8, 2014 at 11:43 PM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Apr 14, 2014 at 11:40 PM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: On Tue, Apr 1, 2014 at 1:41 AM, Robert Haas robertmh...@gmail.com wrote: Should we try to install some hack

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Fri, Aug 8, 2014 at 11:43 PM, Fujii Masao masao.fu...@gmail.com wrote: The attached patch introduces... A patch perhaps? :) -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Tue, Apr 1, 2014 at 1:41 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Mar 20, 2014 at 1:12 PM, Jesper Krogh jes...@krogh.cc wrote: On 15/03/14 20:27, Heikki Linnakangas wrote: That said, I didn't expect the difference to be quite that big when you're appending to the end of the

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

Fujii Masao masao.fu...@gmail.com writes: On Tue, Apr 1, 2014 at 1:41 AM, Robert Haas robertmh...@gmail.com wrote: Should we try to install some hack around fastupdate for 9.4? I fear the divergence between reasonable values of work_mem and reasonable sizes for that list is only going to

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Thu, Mar 20, 2014 at 1:12 PM, Jesper Krogh jes...@krogh.cc wrote: On 15/03/14 20:27, Heikki Linnakangas wrote: That said, I didn't expect the difference to be quite that big when you're appending to the end of the table. When the new entries go to the end of the posting lists, you only need

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

I came up with the attached patch, to reduce the WAL volume of GIN insertions. It become fairly large, but I guess that's not too surprising as the old WAL-logging method was basically to dump the whole page to WAL record. This is now a lot more fine-grained and smarter. I separated

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On 15/03/14 20:27, Heikki Linnakangas wrote: That said, I didn't expect the difference to be quite that big when you're appending to the end of the table. When the new entries go to the end of the posting lists, you only need to recompress and WAL-log the last posting list, which is max 256

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Mon, Mar 17, 2014 at 10:44 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao escribió: On Sun, Mar 16, 2014 at 7:15 AM, Alexander Korotkov aekorot...@gmail.com wrote: That could be optimized, but I figured we can live with it, thanks to the fastupdate feature. Fastupdate

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Sun, Mar 16, 2014 at 7:15 AM, Alexander Korotkov aekorot...@gmail.com wrote: On Sat, Mar 15, 2014 at 11:27 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 03/15/2014 08:40 PM, Fujii Masao wrote: Hi, I executed the following statements in HEAD and 9.3, and compared the size of

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

Fujii Masao escribió: On Sun, Mar 16, 2014 at 7:15 AM, Alexander Korotkov aekorot...@gmail.com wrote: That could be optimized, but I figured we can live with it, thanks to the fastupdate feature. Fastupdate allows amortizing that cost over several insertions. But of course, you explicitly

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On 03/17/2014 03:20 PM, Fujii Masao wrote: On Sun, Mar 16, 2014 at 7:15 AM, Alexander Korotkov aekorot...@gmail.com wrote: On Sat, Mar 15, 2014 at 11:27 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I ran pg_xlogdump | grep Gin and checked the size of GIN-related WAL, and then found

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

Heikki Linnakangas hlinnakan...@vmware.com writes: 2. Instead of storing the new compressed posting list in the WAL record, store only the new item pointers added to the page. WAL replay would then have to duplicate the work done in the main insertion code path: find the right posting lists

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On 03/17/2014 04:33 PM, Tom Lane wrote: Heikki Linnakangas hlinnakan...@vmware.com writes: 2. Instead of storing the new compressed posting list in the WAL record, store only the new item pointers added to the page. WAL replay would then have to duplicate the work done in the main insertion

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Mon, Mar 17, 2014 at 10:54 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Heap and B-tree WAL records also rely on PageAddItem etc. to reconstruct the page, instead of making a physical copy of the modified parts. And _bt_restore_page even inserts the items physically in different

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

Robert Haas robertmh...@gmail.com writes: On Mon, Mar 17, 2014 at 10:54 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Heap and B-tree WAL records also rely on PageAddItem etc. to reconstruct the page, instead of making a physical copy of the modified parts. And _bt_restore_page even

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On 03/17/2014 05:35 PM, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Mar 17, 2014 at 10:54 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: The imminent danger I see is if we change the logic on how the items are divided into posting lists, and end up in a situation

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On 03/15/2014 08:40 PM, Fujii Masao wrote: Hi, I executed the following statements in HEAD and 9.3, and compared the size of WAL which were generated by data insertion in GIN index. - CREATE EXTENSION pg_trgm; CREATE TABLE hoge (col1 text); CREATE INDEX hogeidx ON hoge

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Sat, Mar 15, 2014 at 11:27 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 03/15/2014 08:40 PM, Fujii Masao wrote: Hi, I executed the following statements in HEAD and 9.3, and compared the size of WAL which were generated by data insertion in GIN index. -