Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

On Tue, Nov 30, 2021 at 5:09 PM Peter Geoghegan wrote: > Attached draft patch attempts to explain things in this area within > the nbtree README. There is a much shorter comment about it within > vacuumlazy.c. I am concerned about GiST index-only scans themselves, > of course, but I discovered

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

On Tue, Nov 30, 2021 at 5:09 PM Peter Geoghegan wrote: > I believe that there have been several historic reasons why we need a > cleanup lock during nbtree VACUUM, and that there is only one > remaining reason for it today. So the history is unusually complicated. Minor correction: we actually

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

On Fri, Nov 5, 2021 at 3:26 AM Andrey Borodin wrote: > > 4 нояб. 2021 г., в 20:58, Peter Geoghegan написал(а): > > That's a pretty unlikely scenario. And even > > if it happened it would only happen once (until the next time we get > > unlucky). What are the chances of somebody noticing a more

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

> 4 нояб. 2021 г., в 20:58, Peter Geoghegan написал(а): > That's a pretty unlikely scenario. And even > if it happened it would only happen once (until the next time we get > unlucky). What are the chances of somebody noticing a more or less > once-off, slightly wrong answer? I'd say next to

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

On Thu, Nov 4, 2021 at 8:52 AM Andrey Borodin wrote: > Let's enumerate steps how things can go wrong. > > Backend1: Index-Only scan returns tid and xs_hitup with index_tuple1 on > index_page1 pointing to heap_tuple1 on page1 > > Backend2: Remove index_tuple1 and heap_tuple1 > > Backend3: Mark

Re: Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

  04.11.2021, 04:33, "Peter Geoghegan" :But what about index-only scans, which GiST also supports? I thinkthat the rules are different there, even though index-only scans usean MVCC snapshot.Let's enumerate steps how things can go wrong.Backend1: Index-Only scan returns tid and xs_hitup with

Why doesn't GiST VACUUM require a super-exclusive lock, like nbtree VACUUM?

The code in gistvacuum.c is closely based on similar code in nbtree.c, except that it only acquires an exclusive lock -- not a super-exclusive lock. I suspect that that's because it seemed unnecessary; nbtree plain index scans have their own special reasons for this, that don't apply to GiST.

Re: Questions/Observations related to Gist vacuum

On Thu, Jan 9, 2020 at 4:41 PM Mahendra Singh Thalor wrote: > > On Mon, 9 Dec 2019 at 14:37, Amit Kapila wrote: > > > > On Mon, Dec 9, 2019 at 2:27 PM Amit Kapila wrote: > > > > > > I have modified the patch for the above points and additionally ran > > > pgindent. Let me know what you think

Re: Questions/Observations related to Gist vacuum

On Mon, 9 Dec 2019 at 14:37, Amit Kapila wrote: > > On Mon, Dec 9, 2019 at 2:27 PM Amit Kapila wrote: > > > > I have modified the patch for the above points and additionally ran > > pgindent. Let me know what you think about the attached patch? > > > > A new version with a slightly modified

Re: Questions/Observations related to Gist vacuum

On Mon, Dec 9, 2019 at 2:37 PM Amit Kapila wrote: > > On Mon, Dec 9, 2019 at 2:27 PM Amit Kapila wrote: > > > > I have modified the patch for the above points and additionally ran > > pgindent. Let me know what you think about the attached patch? > > > > A new version with a slightly modified

Re: Questions/Observations related to Gist vacuum

On Mon, Dec 9, 2019 at 2:27 PM Amit Kapila wrote: > > I have modified the patch for the above points and additionally ran > pgindent. Let me know what you think about the attached patch? > A new version with a slightly modified commit message. -- With Regards, Amit Kapila. EnterpriseDB:

Re: Questions/Observations related to Gist vacuum

On Fri, Oct 25, 2019 at 9:22 PM Masahiko Sawada wrote: > > On Wed, Oct 23, 2019 at 8:14 PM Amit Kapila wrote: > > > > On Tue, Oct 22, 2019 at 2:17 PM Dilip Kumar wrote: > > > > > > On Tue, Oct 22, 2019 at 10:53 AM Amit Kapila > > > wrote: > > > > > > I have modified as we discussed. Please

Re: Questions/Observations related to Gist vacuum

On Wed, Oct 23, 2019 at 8:14 PM Amit Kapila wrote: > > On Tue, Oct 22, 2019 at 2:17 PM Dilip Kumar wrote: > > > > On Tue, Oct 22, 2019 at 10:53 AM Amit Kapila > > wrote: > > > > > > > Basically, only IndexBulkDeleteResult is now shared across the stage > > > > so we can move all members to

Re: Questions/Observations related to Gist vacuum

On Tue, Oct 22, 2019 at 10:53 AM Amit Kapila wrote: > > On Tue, Oct 22, 2019 at 10:50 AM Dilip Kumar wrote: > > > > On Tue, Oct 22, 2019 at 9:10 AM Amit Kapila wrote: > > > > > > On Fri, Oct 18, 2019 at 4:51 PM Dilip Kumar wrote: > > > > > > > > I have prepared a first version of the patch.

Re: Questions/Observations related to Gist vacuum

On Tue, Oct 22, 2019 at 10:50 AM Dilip Kumar wrote: > > On Tue, Oct 22, 2019 at 9:10 AM Amit Kapila wrote: > > > > On Fri, Oct 18, 2019 at 4:51 PM Dilip Kumar wrote: > > > > > > I have prepared a first version of the patch. Currently, I am > > > performing an empty page deletion for all the

Re: Questions/Observations related to Gist vacuum

On Tue, Oct 22, 2019 at 9:10 AM Amit Kapila wrote: > > On Fri, Oct 18, 2019 at 4:51 PM Dilip Kumar wrote: > > > > I have prepared a first version of the patch. Currently, I am > > performing an empty page deletion for all the cases. > > > > Few comments: > -- > 1. > -/* > -

Re: Questions/Observations related to Gist vacuum

On Fri, Oct 18, 2019 at 4:51 PM Dilip Kumar wrote: > > I have prepared a first version of the patch. Currently, I am > performing an empty page deletion for all the cases. > Few comments: -- 1. -/* - * State kept across vacuum stages. - */ typedef struct { -

Re: Questions/Observations related to Gist vacuum

On Mon, Oct 21, 2019 at 2:58 PM Andrey Borodin wrote: > > > > > 21 окт. 2019 г., в 11:12, Dilip Kumar написал(а): > > > > On Mon, Oct 21, 2019 at 2:30 PM Andrey Borodin wrote: > >> > >> I've took a look into the patch, and cannot understand one simple thing... > >> We should not call

Re: Questions/Observations related to Gist vacuum

> 21 окт. 2019 г., в 11:12, Dilip Kumar написал(а): > > On Mon, Oct 21, 2019 at 2:30 PM Andrey Borodin wrote: >> >> I've took a look into the patch, and cannot understand one simple thing... >> We should not call gistvacuum_delete_empty_pages() for same gist_stats twice. >> Another way once

Re: Questions/Observations related to Gist vacuum

On Mon, Oct 21, 2019 at 2:30 PM Andrey Borodin wrote: > > Hi! > > > 18 окт. 2019 г., в 13:21, Dilip Kumar написал(а): > > > > On Fri, Oct 18, 2019 at 10:55 AM Amit Kapila > > wrote: > >> > >> > >> I think we can do it in general as adding some check for parallel > >> vacuum there will look bit

Re: Questions/Observations related to Gist vacuum

Hi! > 18 окт. 2019 г., в 13:21, Dilip Kumar написал(а): > > On Fri, Oct 18, 2019 at 10:55 AM Amit Kapila wrote: >> >> >> I think we can do it in general as adding some check for parallel >> vacuum there will look bit hackish. > I agree with that point. > It is not clear if we get enough >>

Re: Questions/Observations related to Gist vacuum

On Mon, Oct 21, 2019 at 11:23 AM Amit Kapila wrote: > > On Fri, Oct 18, 2019 at 10:48 AM Amit Kapila wrote: > > > > Thanks for the test. It shows that prior to patch the memory was > > getting leaked in TopTransactionContext during multi-pass vacuum and > > after patch, there is no leak. I

Re: Questions/Observations related to Gist vacuum

On Fri, Oct 18, 2019 at 10:48 AM Amit Kapila wrote: > > Thanks for the test. It shows that prior to patch the memory was > getting leaked in TopTransactionContext during multi-pass vacuum and > after patch, there is no leak. I will commit the patch early next > week unless Heikki or someone

Re: Questions/Observations related to Gist vacuum

On Fri, Oct 18, 2019 at 10:55 AM Amit Kapila wrote: > > On Fri, Oct 18, 2019 at 9:41 AM Dilip Kumar wrote: > > > > On Wed, Oct 16, 2019 at 7:22 PM Heikki Linnakangas wrote: > > > > > > On 16 October 2019 12:57:03 CEST, Amit Kapila > > > wrote: > > > >On Tue, Oct 15, 2019 at 7:13 PM Heikki

Re: Questions/Observations related to Gist vacuum

On Fri, Oct 18, 2019 at 9:41 AM Dilip Kumar wrote: > > On Wed, Oct 16, 2019 at 7:22 PM Heikki Linnakangas wrote: > > > > On 16 October 2019 12:57:03 CEST, Amit Kapila > > wrote: > > >On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas > > >wrote: > > >> All things > > >> considered, I'm not

Re: Questions/Observations related to Gist vacuum

On Fri, Oct 18, 2019 at 9:34 AM Dilip Kumar wrote: > > On Thu, Oct 17, 2019 at 6:32 PM Dilip Kumar wrote: > > > > On Thu, 17 Oct 2019, 14:59 Amit Kapila, wrote: > >> > >> On Thu, Oct 17, 2019 at 1:47 PM Dilip Kumar wrote: > >> > > >> > On Thu, Oct 17, 2019 at 12:27 PM Heikki Linnakangas > >>

Re: Questions/Observations related to Gist vacuum

On Wed, Oct 16, 2019 at 7:22 PM Heikki Linnakangas wrote: > > On 16 October 2019 12:57:03 CEST, Amit Kapila wrote: > >On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas > >wrote: > >> All things > >> considered, I'm not sure which is better. > > > >Yeah, this is a tough call to make, but if we

Re: Questions/Observations related to Gist vacuum

using the MemoryContextStats function from gdb. For testing this, in gistvacuumscan every time, after it resets the page_set_context, I have collected the sample. I have collected 3 samples for both the head and the patch. We can clearly see that on the head the memory is getting accumulated in the Top

Re: Questions/Observations related to Gist vacuum

On Thu, 17 Oct 2019, 14:59 Amit Kapila, wrote: > On Thu, Oct 17, 2019 at 1:47 PM Dilip Kumar wrote: > > > > On Thu, Oct 17, 2019 at 12:27 PM Heikki Linnakangas > wrote: > > > > > > On 17/10/2019 05:31, Amit Kapila wrote: > > > > > > > > The patch looks good to me. I have slightly modified the

Re: Questions/Observations related to Gist vacuum

On Thu, Oct 17, 2019 at 1:47 PM Dilip Kumar wrote: > > On Thu, Oct 17, 2019 at 12:27 PM Heikki Linnakangas wrote: > > > > On 17/10/2019 05:31, Amit Kapila wrote: > > > > > > The patch looks good to me. I have slightly modified the comments and > > > removed unnecessary initialization. > > > > >

Re: Questions/Observations related to Gist vacuum

On Thu, Oct 17, 2019 at 12:27 PM Heikki Linnakangas wrote: > > On 17/10/2019 05:31, Amit Kapila wrote: > > On Wed, Oct 16, 2019 at 11:20 AM Dilip Kumar wrote: > >> > >> On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas wrote: > >>> > >>> On 15/10/2019 09:37, Amit Kapila wrote: > While

Re: Questions/Observations related to Gist vacuum

On 17/10/2019 05:31, Amit Kapila wrote: On Wed, Oct 16, 2019 at 11:20 AM Dilip Kumar wrote: On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas wrote: On 15/10/2019 09:37, Amit Kapila wrote: While reviewing a parallel vacuum patch [1], we noticed a few things about $SUBJECT implemented in

Re: Questions/Observations related to Gist vacuum

On Thu, Oct 17, 2019 at 9:15 AM Amit Kapila wrote: > > On Wed, Oct 16, 2019 at 7:21 PM Heikki Linnakangas wrote: > > > > On 16 October 2019 12:57:03 CEST, Amit Kapila > > wrote: > > >On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas > > >wrote: > > >> All things > > >> considered, I'm not

Re: Questions/Observations related to Gist vacuum

On Wed, Oct 16, 2019 at 7:21 PM Heikki Linnakangas wrote: > > On 16 October 2019 12:57:03 CEST, Amit Kapila wrote: > >On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas > >wrote: > >> All things > >> considered, I'm not sure which is better. > > > >Yeah, this is a tough call to make, but if we

Re: Questions/Observations related to Gist vacuum

On Wed, Oct 16, 2019 at 11:20 AM Dilip Kumar wrote: > > On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas wrote: > > > > On 15/10/2019 09:37, Amit Kapila wrote: > > > While reviewing a parallel vacuum patch [1], we noticed a few things > > > about $SUBJECT implemented in commit - > > >

Re: Questions/Observations related to Gist vacuum

On 16 October 2019 12:57:03 CEST, Amit Kapila wrote: >On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas >wrote: >> All things >> considered, I'm not sure which is better. > >Yeah, this is a tough call to make, but if we can allow it to delete >the pages in bulkdelete conditionally for parallel

Re: Questions/Observations related to Gist vacuum

On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas wrote: > > On 15/10/2019 09:37, Amit Kapila wrote: > > 2. Right now, in gistbulkdelete we make a note of empty leaf pages and > > internals pages and then in the second pass during gistvacuumcleanup, > > we delete all the empty leaf pages. I was

Re: Questions/Observations related to Gist vacuum

On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas wrote: > > On 15/10/2019 09:37, Amit Kapila wrote: > > While reviewing a parallel vacuum patch [1], we noticed a few things > > about $SUBJECT implemented in commit - > > 7df159a620b760e289f1795b13542ed1b3e13b87. > > > > 1. A new memory context

Re: Questions/Observations related to Gist vacuum

On 15/10/2019 09:37, Amit Kapila wrote: While reviewing a parallel vacuum patch [1], we noticed a few things about $SUBJECT implemented in commit - 7df159a620b760e289f1795b13542ed1b3e13b87. 1. A new memory context GistBulkDeleteResult->page_set_context has been introduced, but it doesn't seem

Questions/Observations related to Gist vacuum

While reviewing a parallel vacuum patch [1], we noticed a few things about $SUBJECT implemented in commit - 7df159a620b760e289f1795b13542ed1b3e13b87. 1. A new memory context GistBulkDeleteResult->page_set_context has been introduced, but it doesn't seem to be used. 2. Right now, in gistbulkdelete

Re: GiST VACUUM

On Wed, Jul 24, 2019 at 11:33 AM Heikki Linnakangas wrote: > That's probably how it's going to go, but hey, doesn't hurt to ask :-). I think that it would be fine to be conservative with nbtree, and only target the master branch. The problem is annoying, certainly, but it's not likely to make a

Re: GiST VACUUM

On 24/07/2019 21:02, Peter Geoghegan wrote: On Wed, Jul 24, 2019 at 10:30 AM Heikki Linnakangas wrote: Pushed this now, to master and REL_12_STABLE. Now, B-tree indexes still have the same problem, in all versions. Any volunteers to write a similar fix for B-trees? I was hoping that you'd

Re: GiST VACUUM

On Wed, Jul 24, 2019 at 10:30 AM Heikki Linnakangas wrote: > Pushed this now, to master and REL_12_STABLE. > > Now, B-tree indexes still have the same problem, in all versions. Any > volunteers to write a similar fix for B-trees? I was hoping that you'd work on it. :-) Any reason to think

Re: GiST VACUUM

On 22/07/2019 16:09, Heikki Linnakangas wrote: Unless something comes up, I'll commit this tomorrow. Pushed this now, to master and REL_12_STABLE. Now, B-tree indexes still have the same problem, in all versions. Any volunteers to write a similar fix for B-trees? - Heikki

Re: GiST VACUUM

ble in a later release? I came up with the attached. Instead of having a general purpose "widen" function, it adds GetFullRecentGlobalXmin(), to return a 64-bit version of RecentGlobalXmin. That's enough for this Gist vacuum patch. In addition to that change, I modified the GistP

Re: GiST VACUUM

On Fri, Jun 28, 2019 at 3:32 AM Thomas Munro wrote: > > On Thu, Jun 27, 2019 at 11:38 PM Heikki Linnakangas wrote: > > * I moved the logic to extend a 32-bit XID to 64-bits to a new helper > > function in varsup.c. > > I'm a bit uneasy about making this into a generally available function > for

Re: GiST VACUUM

On Thu, Apr 4, 2019 at 8:15 AM Heikki Linnakangas wrote: > I think we should fix this in a similar manner in B-tree, too, but that > can be done separately. For B-tree, we need to worry about > backwards-compatibility, but that seems simple enough; we just need to > continue to understand old

Re: GiST VACUUM

On Thu, Jun 27, 2019 at 11:38 PM Heikki Linnakangas wrote: > * I moved the logic to extend a 32-bit XID to 64-bits to a new helper > function in varsup.c. I'm a bit uneasy about making this into a generally available function for reuse. I think we should instead come up with a very small number

Re: GiST VACUUM

On 27/06/2019 20:15, Andrey Borodin wrote: But I have stupid question again, about this code: https://github.com/x4m/postgres_g/commit/096d5586537d29ff316ca8ce074bbe1b325879ee#diff-754126824470cb8e68fd5e32af6d3bcaR417 nextFullXid = ReadNextFullTransactionId();

Re: GiST VACUUM

> 27 июня 2019 г., в 16:38, Heikki Linnakangas написал(а): > > I haven't done any testing on this yet. Andrey, would you happen to have an > environment ready to test this? Patches do not deadlock and delete pages on "rescan test" - setup that we used to detect "left jumps" in during

Re: GiST VACUUM

> 27 июня 2019 г., в 16:38, Heikki Linnakangas написал(а): > > I haven't done any testing on this yet. Andrey, would you happen to have an > environment ready to test this? Thanks! I will do some testing this evening (UTC+5). But I'm not sure I can reliably test wraparound of xids... I

Re: GiST VACUUM

On 26/06/2019 06:07, Thomas Munro wrote: On Wed, Jun 26, 2019 at 2:33 PM Michael Paquier wrote: On Tue, Jun 25, 2019 at 02:38:43PM +0500, Andrey Borodin wrote: I feel a little uncomfortable with number 0x7fff right in code. PG_INT32_MAX... MaxTransactionId / 2? Yeah, that makes

Re: GiST VACUUM

On Wed, Jun 26, 2019 at 2:33 PM Michael Paquier wrote: > On Tue, Jun 25, 2019 at 02:38:43PM +0500, Andrey Borodin wrote: > > I feel a little uncomfortable with number 0x7fff right in code. > > PG_INT32_MAX... MaxTransactionId / 2? -- Thomas Munro https://enterprisedb.com

Re: GiST VACUUM

On Tue, Jun 25, 2019 at 02:38:43PM +0500, Andrey Borodin wrote: > I feel a little uncomfortable with number 0x7fff right in code. PG_INT32_MAX... -- Michael signature.asc Description: PGP signature

Re: GiST VACUUM

Hi! Thanks for clarification, now I understand these patches better. > 25 июня 2019 г., в 13:10, Heikki Linnakangas написал(а): > >> Also, I did not understand this optimization: >> +/* >> + * We can skip this if the page was deleted so long ago, that no scan >> can possibly >> +

Re: GiST VACUUM

(Thanks for the reminder on this, Michael!) On 05/04/2019 08:39, Andrey Borodin wrote: 4 апр. 2019 г., в 20:15, Heikki Linnakangas написал(а): I suggest that we do the attached. It fixes this for GiST. The patch changes expands the "deletion XID" to 64-bits, and changes where it's stored.

Re: GiST VACUUM

Heikki, On Thu, Apr 04, 2019 at 06:15:21PM +0300, Heikki Linnakangas wrote: > I think we should fix this in a similar manner in B-tree, too, but that can > be done separately. For B-tree, we need to worry about > backwards-compatibility, but that seems simple enough; we just need to > continue to

Re: GiST VACUUM

Hi! > 4 апр. 2019 г., в 20:15, Heikki Linnakangas написал(а): > > On 25/03/2019 15:20, Heikki Linnakangas wrote: >> On 24/03/2019 18:50, Andrey Borodin wrote: >>> I was working on new version of gist check in amcheck and understand one >>> more thing: >>> >>> /* Can this page be recycled yet?

Re: GiST VACUUM

On 25/03/2019 15:20, Heikki Linnakangas wrote: On 24/03/2019 18:50, Andrey Borodin wrote: I was working on new version of gist check in amcheck and understand one more thing: /* Can this page be recycled yet? */ bool gistPageRecyclable(Page page) { return PageIsNew(page) ||

Re: GiST VACUUM

On 24/03/2019 18:50, Andrey Borodin wrote: I was working on new version of gist check in amcheck and understand one more thing: /* Can this page be recycled yet? */ bool gistPageRecyclable(Page page) { return PageIsNew(page) || (GistPageIsDeleted(page) &&

Re: GiST VACUUM

> 22 марта 2019 г., в 17:03, Heikki Linnakangas написал(а): > I was working on new version of gist check in amcheck and understand one more thing: /* Can this page be recycled yet? */ bool gistPageRecyclable(Page page) { return PageIsNew(page) || (GistPageIsDeleted(page) &&

Re: GiST VACUUM

> 22 марта 2019 г., в 19:37, Heikki Linnakangas написал(а): > > On 21/03/2019 19:04, Heikki Linnakangas wrote: >> Attached is the latest patch version, to be applied on top of the >> IntegerSet patch. > > And committed. Thanks Andrey! > > - Heikki Cool! Thank you very much! At the

Re: GiST VACUUM

On 22/03/2019 13:37, Heikki Linnakangas wrote: On 21/03/2019 19:04, Heikki Linnakangas wrote: Attached is the latest patch version, to be applied on top of the IntegerSet patch. And committed. Thanks Andrey! This caused the buildfarm to go pink... I was able to reproduce it, by running the

Re: GiST VACUUM

On 21/03/2019 19:04, Heikki Linnakangas wrote: Attached is the latest patch version, to be applied on top of the IntegerSet patch. And committed. Thanks Andrey! - Heikki

Re: GiST VACUUM

On 22/03/2019 10:00, Andrey Borodin wrote: 22 марта 2019 г., в 1:04, Heikki Linnakangas написал(а): PS. for Gist, we could almost use the LSN / NSN mechanism to detect the case that a deleted page is reused: Add a new field to the GiST page header, to store a new "deleteNSN" field. When a page

Re: GiST VACUUM

> 22 марта 2019 г., в 1:04, Heikki Linnakangas написал(а): > ... > When I started testing this, I quickly noticed that empty pages were not > being deleted nearly as much as I expected. I tracked it to this check in > gistdeletepage: > >> + if (GistFollowRight(leafPage) >> +

Re: GiST VACUUM

et. Something to keep in mind, if we have to change the page format anyway, for some reason. - Heikki >From d7a77ad483251b62514778d2235516f6f9237ad7 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Wed, 20 Mar 2019 20:24:44 +0200 Subject: [PATCH 2/2] Delete empty pages during GiST

Re: GiST VACUUM

> 21 марта 2019 г., в 2:30, Heikki Linnakangas написал(а): > one remaining issue that needs to be fixed: > > During Hot Standby, the B-tree code writes a WAL reord, when a deleted > page is recycled, to prevent the deletion from being replayed too early > in the hot standby. See

Re: GiST VACUUM

ompact representation at leaf items, using Simple-8b algorithm, so that clusters of nearby values take less space. This doesn't include any use of the code yet, but we have two patches in the works that would benefit from this: * the GiST vacuum patch, to track empty GiST pages and internal GiST

Re: GiST VACUUM

ompact representation at leaf items, using Simple-8b algorithm, so that clusters of nearby values take less space. This doesn't include any use of the code yet, but we have two patches in the works that would benefit from this: * the GiST vacuum patch, to track empty GiST pages and internal GiST

Re: GiST VACUUM

On Tue, Mar 5, 2019 at 8:21 AM Heikki Linnakangas wrote: > On 05/03/2019 02:26, Andrey Borodin wrote: > >> I also tried your amcheck tool with this. It did not report any > >> errors. > >> > >> Attached is also latest version of the patch itself. It is the > >> same as your latest patch v19,

Re: GiST VACUUM

but this variable could stop gistvacuum_recycle_pages() when everything is recycled already. Thanks! Best regards, Andrey Borodin. 0001-Add-block-set-data-structure-v22.patch Description: Binary data 0002-Delete-pages-during-GiST-VACUUM-v22.patch Description: Binary data 0003-Minor-

Re: GiST VACUUM

x %d", + blockno, test_limit, point_index); + } + } + + blockset_free(bs); + bms_free(bms); +} diff --git a/src/test/modules/test_blockset/test_blockset.control b/src/test/modules/test_blockset/test_blockset.control new file mode 100644 index 000..fdb7598c5a7 --- /dev/null +++ b/s

Re: GiST VACUUM

> 5 марта 2019 г., в 18:21, Heikki Linnakangas написал(а): > > On 05/03/2019 02:26, Andrey Borodin wrote: >> >> That's cool! I'll work on 2nd step of these patchset to make >> blockset data structure prettier and less hacky. > > Committed the first patch. Thanks for the patch! That's cool!

Re: GiST VACUUM

On 05/03/2019 02:26, Andrey Borodin wrote: I also tried your amcheck tool with this. It did not report any errors. Attached is also latest version of the patch itself. It is the same as your latest patch v19, except for some tiny comment kibitzing. I'll mark this as Ready for Committer in the

Re: GiST VACUUM

Hi! Thanks for fixing gist amcheck! The idea of using these two patches together seems so obvious now, but never actually visited my mind before. > 4 марта 2019 г., в 18:04, Heikki Linnakangas написал(а): > Thanks! As I noted at >

Re: GiST VACUUM

the commitfest app, and will try to commit it in the next couple of days. - Heikki gist-vacuum-test.sh Description: application/shellscript diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c index 3f52b8f4dc..cad4a2a46e 100644 --- a/src/backend/access/gist/gist.c +

Re: GiST VACUUM

On 04/01/2019 21:26, Andrey Borodin wrote: 3 янв. 2019 г., в 23:47, Andrey Borodin написал(а): * Bitmapset stores 32 bit signed integers, but BlockNumber is unsigned. So this will fail with an index larger than 2^31 blocks. That's perhaps academical, I don't think anyone will try to create a

Re: GiST VACUUM

3 янв. 2019 г., в 23:47, Andrey Borodin написал(а): > >> * Bitmapset stores 32 bit signed integers, but BlockNumber is unsigned. So >> this will fail with an index larger than 2^31 blocks. That's perhaps >> academical, I don't think anyone will try to create a 16 TB GiST index any >> time

Re: GiST VACUUM

e) && > TransactionIdPrecedes(GistPageGetDeleteXid(page), RecentGlobalXmin)) I've switched using RecentGlobalDataXmin to RecentGlobalXmin, because we have done so in similar mechanics for GIN (for uniformity with B-tree). Thanks for working on this! Best regards, Andrey Borodin. 0002-Delete-pages-during-GiST-VACUUM-v19.patch Description: Binary data 0001-Physical-GiST-scan-in-VACUUM-v19.patch Description: Binary data

Re: GiST VACUUM

be large ... but + * just in case, let's hand-optimize into a loop. + */ + if (recurse_to != InvalidBlockNumber) + { + blkno = recurse_to; + goto restart; + } } -- 2.19.2 >From 62fbe0ce5506e006b92dbfb07aee7414040d982f Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Wed, 2 Jan 2019 1

Re: GiST VACUUM

On 28/10/2018 19:32, Andrey Borodin wrote: Hi everyone! 2 окт. 2018 г., в 6:14, Michael Paquier написал(а): Andrey, your latest patch does not apply. I am moving this to the next CF, waiting for your input. I'm doing preps for CF. Here's rebased version. Thanks, I had another look at

Re: GiST VACUUM

> On Sun, Oct 28, 2018 at 6:32 PM Andrey Borodin wrote: > > Hi everyone! > > > 2 окт. 2018 г., в 6:14, Michael Paquier написал(а): > > Andrey, your latest patch does not apply. I am moving this to the next > > CF, waiting for your input. > > I'm doing preps for CF. > Here's rebased version.

Re: GiST VACUUM

Hi everyone! > 2 окт. 2018 г., в 6:14, Michael Paquier написал(а): > Andrey, your latest patch does not apply. I am moving this to the next > CF, waiting for your input. I'm doing preps for CF. Here's rebased version. Best regards, Andrey Borodin. 0002-Delete-pages-during-GiST-V

Re: GiST VACUUM

On Mon, Aug 06, 2018 at 11:12:00PM +0500, Andrey Borodin wrote: > Done. Added function bms_make_empty(int size) Andrey, your latest patch does not apply. I am moving this to the next CF, waiting for your input. -- Michael signature.asc Description: PGP signature

Re: GiST VACUUM

e the bitmap. > OK, I'll think of proper resize function (ensure capacity, to be precise). Done. Added function bms_make_empty(int size) Best regards, Andrey Borodin. 0002-Delete-pages-during-GiST-VACUUM-v16.patch Description: Binary data 0001-Physical-GiST-scan-in-VACUUM-v16.patch Description: Binary data

Re: GiST VACUUM

Hi! > 5 авг. 2018 г., в 16:18, Heikki Linnakangas написал(а): > > On 31/07/18 23:06, Andrey Borodin wrote: >>> On a typical GiST index, what's the ratio of leaf vs. internal >>> pages? Perhaps an array would indeed be better. > > >> Typical GiST has around 200 tuples per internal page. I've

Re: GiST VACUUM

On 31/07/18 23:06, Andrey Borodin wrote: On a typical GiST index, what's the ratio of leaf vs. internal pages? Perhaps an array would indeed be better. > Typical GiST has around 200 tuples per internal page. I've switched to List since it's more efficient than bitmap. Hmm. A ListCell is 16

Re: GiST VACUUM

have to note that default growth strategy of Bitmap is not good: we will be repallocing byte by byte. > > A straightforward little optimization would be to skip scanning the internal > pages, when the first scan didn't find any empty pages. And stop the scan of > the internal pages a

Re: GiST VACUUM

On 29/07/18 14:47, Andrey Borodin wrote: Fixed both problems. PFA v14. Thanks, took a really quick look at this. The text being added to README is outdated for these latest changes. In second step I still use paloc's memory, but only to store two bitmaps: bitmap of internal pages and bitmap

Re: GiST VACUUM

ariable mapNumPages. I do not know why it didn't trigger failure last time, but now it was reproduced on my machine reliably. Fixed both problems. PFA v14. Best regards, Andrey Borodin. 0002-Delete-pages-during-GiST-VACUUM-v14.patch Description: Binary data 0001-Physical-GiST-scan-in-VACUUM-v14.patch Description: Binary data

Re: GiST VACUUM

On Tue, Jul 24, 2018 at 6:04 AM, Andrey Borodin wrote: >> 21 июля 2018 г., в 17:11, Andrey Borodin написал(а): >> <0001-Physical-GiST-scan-in-VACUUM-v13.patch> > > Just in case, here's second part of patch series with actual page deletion. Hi Andrey, Cfbot says:

Re: GiST VACUUM

stead of bitmap, but it will create seriously more work for cpu to compute hashes. Best regards, Andrey Borodin. 0002-Delete-pages-during-GiST-VACUUM-v13.patch Description: Binary data 0001-Physical-GiST-scan-in-VACUUM-v13.patch Description: Binary data

Re: GiST VACUUM

Hi! > 19 июля 2018 г., в 23:26, Andrey Borodin написал(а): > > I'm working on triggering left split during vacuum. Will get back when done. > Thanks! Here's patch including some messy hacks to trigger NSN and FollowRight jumps during VACUUM. To trigger FollowRight GiST sometimes forget to

Re: GiST VACUUM

> 19 июля 2018 г., в 16:28, Heikki Linnakangas написал(а): > Hmm. So, while we are scanning the right sibling, which was moved to > lower-numbered block because of a concurrent split, the original page is > split again? That's OK, we've already scanned all the tuples on the original > page,

Re: GiST VACUUM

On 19/07/18 14:42, Andrey Borodin wrote: 19.07.2018, 15:20, "Heikki Linnakangas" : On 19/07/18 13:52, Andrey Borodin wrote: Hi! 19 июля 2018 г., в 1:12, Heikki Linnakangas mailto:hlinn...@iki.fi>> написал(а): Yeah, please, I think this is the way to go.

Re: GiST VACUUM

19.07.2018, 15:20, "Heikki Linnakangas" :On 19/07/18 13:52, Andrey Borodin wrote: Hi! 19 июля 2018 г., в 1:12, Heikki Linnakangas  написал(а): Yeah, please, I think this is the way to go. Here's v11 divided into proposed steps.Thanks, one quick question: /*

Re: GiST VACUUM

On 19/07/18 13:52, Andrey Borodin wrote: Hi! 19 июля 2018 г., в 1:12, Heikki Linnakangas написал(а): Yeah, please, I think this is the way to go. Here's v11 divided into proposed steps. Thanks, one quick question: /* We should not unlock buffer if we are going

Re: GiST VACUUM

physical scan only reads internal pages. I can omit that bitmap, if I'll scan everything. Also, I can replace emptyLeafs bitmap with array\list, but I do not really think it will be big. Anyway I propose focusing on first step. Best regards, Andrey Borodin. 0002-Delete-pages-during-GiST-V

Re: GiST VACUUM

On 18/07/18 21:27, Andrey Borodin wrote: Hi! 18 июля 2018 г., в 16:02, Heikki Linnakangas написал(а): , but I think it would be better to split this into two patches as follows: 1st patch: Scan the index in physical rather than logical order. No attempt at deleting empty pages yet. 2nd

  1   2   >