Re: Improve search for missing parent downlinks in amcheck

2020-03-11 Thread Peter Geoghegan
On Wed, Mar 11, 2020 at 2:02 AM Alexander Korotkov wrote: > Thank you! Pushed with this comment revised! Thanks! -- Peter Geoghegan

Re: Improve search for missing parent downlinks in amcheck

2020-03-11 Thread Alexander Korotkov
On Wed, Mar 11, 2020 at 7:19 AM Peter Geoghegan wrote: > This looks committable. I only noticed one thing: The comments above > bt_target_page_check() need to be updated to reflect the new check, > which no longer has anything to do with "heapallindexed = true". Thank you! Pushed with this

Re: Improve search for missing parent downlinks in amcheck

2020-03-10 Thread Peter Geoghegan
On Tue, Mar 10, 2020 at 8:30 AM Alexander Korotkov wrote: > Yes, current example looks confusing in this aspect. But your comment > spotted to me an algorithmic issue. We don't match highkey of > leftmost child against parent pivot key. But we can. The "if > (!BlockNumberIsValid(blkno))"

Re: Improve search for missing parent downlinks in amcheck

2020-03-10 Thread Alexander Korotkov
On Tue, Mar 10, 2020 at 3:07 AM Peter Geoghegan wrote: > On Sun, Mar 8, 2020 at 2:52 PM Alexander Korotkov > wrote: > > I've revised this comment. Hopefully it's better now. > > I think that the new comments about why we need a low key for the page > are much better now. Good, thank you. > >

Re: Improve search for missing parent downlinks in amcheck

2020-03-09 Thread Peter Geoghegan
On Sun, Mar 8, 2020 at 2:52 PM Alexander Korotkov wrote: > I've revised this comment. Hopefully it's better now. I think that the new comments about why we need a low key for the page are much better now. > I've updated this comment using terms "cousin page" and "subtree". I > didn't refer

Re: Improve search for missing parent downlinks in amcheck

2020-03-09 Thread Alexander Korotkov
On Mon, Mar 9, 2020 at 12:52 AM Alexander Korotkov wrote: > Attached patch also has revised commit message. I'll wait for your > response before commit. Oh, I found that I haven't attached the patch. -- Alexander Korotkov Postgres Professional: http://www.postgrespro.com The Russian

Re: Improve search for missing parent downlinks in amcheck

2020-03-08 Thread Alexander Korotkov
Hi, Peter! On Tue, Mar 3, 2020 at 3:04 AM Peter Geoghegan wrote: > Apologies for the delayed response. I was a little tired from the > deduplication project. No problem. Apologies for the delayed revision as well. > I taught pageinspect to display a "htid" field for pivot tuples > recently,

Re: Improve search for missing parent downlinks in amcheck

2020-03-02 Thread Peter Geoghegan
Hi Alexander, Apologies for the delayed response. I was a little tired from the deduplication project. On Mon, Feb 24, 2020 at 2:54 PM Alexander Korotkov wrote: > Thank you for your review. The revised version is attached. This has bitrot, because of the deduplication patch. Shouldn't be hard

Re: Improve search for missing parent downlinks in amcheck

2020-02-24 Thread Alexander Korotkov
Hi! Thank you for your review. The revised version is attached. On Wed, Feb 19, 2020 at 1:16 AM Peter Geoghegan wrote: > On Tue, Feb 18, 2020 at 2:16 AM Alexander Korotkov > wrote: > > > Don't need the "!P_ISLEAF()" here. > > > > Why don't I need. bt_downlink_connectivity_check() checks one

Re: Improve search for missing parent downlinks in amcheck

2020-02-18 Thread Peter Geoghegan
On Tue, Feb 18, 2020 at 2:16 AM Alexander Korotkov wrote: > Great, thank you very much! No problem! My remarks here are based on "amcheck-btree-improve-missing-parent-downlinks-check-6.patch". I have found a false positive corruption report bug in this latest version -- see note below about

Re: Improve search for missing parent downlinks in amcheck

2020-02-18 Thread Alexander Korotkov
On Tue, Feb 18, 2020 at 1:15 PM Alexander Korotkov wrote: > On Fri, Jan 24, 2020 at 4:31 AM Peter Geoghegan wrote: > > On Wed, Jan 22, 2020 at 6:41 PM Alexander Korotkov > > wrote: > > > Rebased patch is attached. Sorry for so huge delay. > > > > I really like this patch. Your interest in

Re: Improve search for missing parent downlinks in amcheck

2020-02-18 Thread Alexander Korotkov
Hi, Peter! On Fri, Jan 24, 2020 at 4:31 AM Peter Geoghegan wrote: > On Wed, Jan 22, 2020 at 6:41 PM Alexander Korotkov > wrote: > > Rebased patch is attached. Sorry for so huge delay. > > I really like this patch. Your interest in amcheck is something that > makes me feel good about having put

Re: Improve search for missing parent downlinks in amcheck

2020-01-23 Thread Peter Geoghegan
On Thu, Jan 23, 2020 at 5:40 PM Peter Geoghegan wrote: > I suppose the alternative is to get the high key of the parent's left > sibling, rather than going to the parent's parent (i.e. our > grandparent). That would probably be the best way to get a separator > key to compare against the high key

Re: Improve search for missing parent downlinks in amcheck

2020-01-23 Thread Peter Geoghegan
On Thu, Jan 23, 2020 at 5:31 PM Peter Geoghegan wrote: > In summary: I suppose that we can also solve "the cousin problem" > quite easily, but only for rightmost cousins within a subtree -- > leftmost cousins might be too messy to verify for it to be worth it. > We don't want to have to jump two

Re: Improve search for missing parent downlinks in amcheck

2020-01-23 Thread Peter Geoghegan
On Wed, Jan 22, 2020 at 6:41 PM Alexander Korotkov wrote: > Rebased patch is attached. Sorry for so huge delay. I really like this patch. Your interest in amcheck is something that makes me feel good about having put so much work into it myself. Here are some review comments: > + /* > +

Re: Improve search for missing parent downlinks in amcheck

2020-01-22 Thread Alexander Korotkov
On Fri, Jan 17, 2020 at 1:05 AM Tomas Vondra wrote: > On Fri, Nov 29, 2019 at 03:03:01PM +0900, Michael Paquier wrote: > >On Mon, Aug 19, 2019 at 01:15:19AM +0300, Alexander Korotkov wrote: > >> The revised patch seems to fix all of above. > > > >The latest patch is failing to apply. Please

Re: Improve search for missing parent downlinks in amcheck

2020-01-16 Thread Tomas Vondra
On Fri, Nov 29, 2019 at 03:03:01PM +0900, Michael Paquier wrote: On Mon, Aug 19, 2019 at 01:15:19AM +0300, Alexander Korotkov wrote: The revised patch seems to fix all of above. The latest patch is failing to apply. Please provide a rebase. This still does not apply (per cputube). Can you

Re: Improve search for missing parent downlinks in amcheck

2019-11-28 Thread Michael Paquier
On Mon, Aug 19, 2019 at 01:15:19AM +0300, Alexander Korotkov wrote: > The revised patch seems to fix all of above. The latest patch is failing to apply. Please provide a rebase. -- Michael signature.asc Description: PGP signature

Re: Improve search for missing parent downlinks in amcheck

2019-08-18 Thread Alexander Korotkov
On Tue, Aug 13, 2019 at 11:44 PM Peter Geoghegan wrote: > > In this revision check for missing downlinks is combined with > > bt_downlink_check(). So, pages visited by bt_downlink_check() patch > > doesn't cause extra accessed. It only causes following additional > > page accesses: > > 1)

Re: Improve search for missing parent downlinks in amcheck

2019-08-13 Thread Peter Geoghegan
On Mon, Aug 12, 2019 at 12:01 PM Alexander Korotkov wrote: > BTW, there is next revision of patch I'm proposing for v13. Cool. > In this revision check for missing downlinks is combined with > bt_downlink_check(). So, pages visited by bt_downlink_check() patch > doesn't cause extra accessed.

Re: Improve search for missing parent downlinks in amcheck

2019-08-12 Thread Alexander Korotkov
On Fri, Jul 19, 2019 at 3:21 AM Peter Geoghegan wrote: > On Tue, Apr 30, 2019 at 5:58 PM Peter Geoghegan wrote: > > I will think about a simple fix, but after the upcoming point release. > > There is no hurry. > > Attached draft patch uses RelationGetNumberOfBlocks() to size each of > the two

Re: Improve search for missing parent downlinks in amcheck

2019-07-18 Thread Peter Geoghegan
On Tue, Apr 30, 2019 at 5:58 PM Peter Geoghegan wrote: > I will think about a simple fix, but after the upcoming point release. > There is no hurry. Attached draft patch uses RelationGetNumberOfBlocks() to size each of the two Bloom filters that may be used by amcheck to perform verification.

Re: Improve search for missing parent downlinks in amcheck

2019-07-08 Thread Peter Geoghegan
On Sun, Jul 7, 2019 at 7:53 PM Thomas Munro wrote: > On Wed, May 1, 2019 at 12:58 PM Peter Geoghegan wrote: > > I will think about a simple fix, but after the upcoming point release. > > There is no hurry. > > A bureaucratic question: What should the status be for this CF entry? I have plans

Re: Improve search for missing parent downlinks in amcheck

2019-07-07 Thread Thomas Munro
On Wed, May 1, 2019 at 12:58 PM Peter Geoghegan wrote: > On Sun, Apr 28, 2019 at 10:15 AM Alexander Korotkov > wrote: > > I think this definitely not bug fix. Bloom filter was designed to be > > lossy, no way blaming it for that :) > > I will think about a simple fix, but after the upcoming

Re: Improve search for missing parent downlinks in amcheck

2019-04-30 Thread Peter Geoghegan
On Sun, Apr 28, 2019 at 10:15 AM Alexander Korotkov wrote: > I think this definitely not bug fix. Bloom filter was designed to be > lossy, no way blaming it for that :) I will think about a simple fix, but after the upcoming point release. There is no hurry. -- Peter Geoghegan

Re: Improve search for missing parent downlinks in amcheck

2019-04-28 Thread Alexander Korotkov
On Sun, Apr 28, 2019 at 4:36 AM Peter Geoghegan wrote: > On Sat, Apr 27, 2019 at 5:13 PM Alexander Korotkov > wrote: > > Yes, increasing of Bloom filter size also helps. But my intention was > > to make non-lossy check here. > > Why is that your intention? Do you want to do this as a feature

Re: Improve search for missing parent downlinks in amcheck

2019-04-27 Thread Peter Geoghegan
On Sat, Apr 27, 2019 at 5:13 PM Alexander Korotkov wrote: > Yes, increasing of Bloom filter size also helps. But my intention was > to make non-lossy check here. Why is that your intention? Do you want to do this as a feature for Postgres 13, or do you want to treat this as a bug that we need

Re: Improve search for missing parent downlinks in amcheck

2019-04-27 Thread Peter Geoghegan
On Sat, Apr 27, 2019 at 5:13 PM Alexander Korotkov wrote: > Yes, increasing of Bloom filter size also helps. But my intention was > to make non-lossy check here. I agree that that might be a good goal, but I am interested in knowing if there is something naive about how the downlinkfilter Bloom

Re: Improve search for missing parent downlinks in amcheck

2019-04-27 Thread Alexander Korotkov
On Tue, Apr 16, 2019 at 10:00 PM Peter Geoghegan wrote: > > On Mon, Apr 15, 2019 at 7:30 PM Alexander Korotkov > wrote: > > Currently we amcheck supports lossy checking for missing parent > > downlinks. It collects bitmap of downlink hashes and use it to check > > subsequent tree level. We've

Re: Improve search for missing parent downlinks in amcheck

2019-04-27 Thread Peter Geoghegan
On Sat, Apr 27, 2019 at 4:57 PM Alexander Korotkov wrote: > "rootdescend" is cool type of check. Thank you for noticing, I wasn't aware > of it. > But can it detect the missing downlink in following situation? > > A > / \ > B <-> C <-> D > > Here A has downlinks to B and D,

Re: Improve search for missing parent downlinks in amcheck

2019-04-27 Thread Alexander Korotkov
On Tue, Apr 16, 2019 at 10:04 PM Peter Geoghegan wrote: > > On Tue, Apr 16, 2019 at 12:00 PM Peter Geoghegan wrote: > > Can you be more specific? What was the cause of the corruption? I'm > > always very interested in hearing about cases that amcheck could have > > detected, but didn't. > >

Re: Improve search for missing parent downlinks in amcheck

2019-04-25 Thread Peter Geoghegan
On Tue, Apr 16, 2019 at 12:00 PM Peter Geoghegan wrote: > On Mon, Apr 15, 2019 at 7:30 PM Alexander Korotkov > wrote: > > Currently we amcheck supports lossy checking for missing parent > > downlinks. It collects bitmap of downlink hashes and use it to check > > subsequent tree level. We've

Re: Improve search for missing parent downlinks in amcheck

2019-04-16 Thread Peter Geoghegan
On Tue, Apr 16, 2019 at 12:00 PM Peter Geoghegan wrote: > Can you be more specific? What was the cause of the corruption? I'm > always very interested in hearing about cases that amcheck could have > detected, but didn't. FWIW, v4 indexes in Postgres 12 will support the new "rootdescend"

Re: Improve search for missing parent downlinks in amcheck

2019-04-16 Thread Peter Geoghegan
On Mon, Apr 15, 2019 at 7:30 PM Alexander Korotkov wrote: > Currently we amcheck supports lossy checking for missing parent > downlinks. It collects bitmap of downlink hashes and use it to check > subsequent tree level. We've experienced some large corrupted indexes > which pass this check due

Improve search for missing parent downlinks in amcheck

2019-04-15 Thread Alexander Korotkov
Hi! Currently we amcheck supports lossy checking for missing parent downlinks. It collects bitmap of downlink hashes and use it to check subsequent tree level. We've experienced some large corrupted indexes which pass this check due to its looseness. However, it seems to me we can implement