Re: Improve search for missing parent downlinks in amcheck

2020-03-11 Thread Peter Geoghegan
On Wed, Mar 11, 2020 at 2:02 AM Alexander Korotkov wrote: > Thank you! Pushed with this comment revised! Thanks! -- Peter Geoghegan

Re: Improve search for missing parent downlinks in amcheck

2020-03-11 Thread Alexander Korotkov
On Wed, Mar 11, 2020 at 7:19 AM Peter Geoghegan wrote: > This looks committable. I only noticed one thing: The comments above > bt_target_page_check() need to be updated to reflect the new check, > which no longer has anything to do with "heapallindexed = true". Thank you! Pushed with this comme

Re: Improve search for missing parent downlinks in amcheck

2020-03-10 Thread Peter Geoghegan
On Tue, Mar 10, 2020 at 8:30 AM Alexander Korotkov wrote: > Yes, current example looks confusing in this aspect. But your comment > spotted to me an algorithmic issue. We don't match highkey of > leftmost child against parent pivot key. But we can. The "if > (!BlockNumberIsValid(blkno))" branc

Re: Improve search for missing parent downlinks in amcheck

2020-03-10 Thread Alexander Korotkov
On Tue, Mar 10, 2020 at 3:07 AM Peter Geoghegan wrote: > On Sun, Mar 8, 2020 at 2:52 PM Alexander Korotkov > wrote: > > I've revised this comment. Hopefully it's better now. > > I think that the new comments about why we need a low key for the page > are much better now. Good, thank you. > > I

Re: Improve search for missing parent downlinks in amcheck

2020-03-09 Thread Peter Geoghegan
On Sun, Mar 8, 2020 at 2:52 PM Alexander Korotkov wrote: > I've revised this comment. Hopefully it's better now. I think that the new comments about why we need a low key for the page are much better now. > I've updated this comment using terms "cousin page" and "subtree". I > didn't refer the

Re: Improve search for missing parent downlinks in amcheck

2020-03-09 Thread Alexander Korotkov
On Mon, Mar 9, 2020 at 12:52 AM Alexander Korotkov wrote: > Attached patch also has revised commit message. I'll wait for your > response before commit. Oh, I found that I haven't attached the patch. -- Alexander Korotkov Postgres Professional: http://www.postgrespro.com The Russian Postgre

Re: Improve search for missing parent downlinks in amcheck

2020-03-08 Thread Alexander Korotkov
Hi, Peter! On Tue, Mar 3, 2020 at 3:04 AM Peter Geoghegan wrote: > Apologies for the delayed response. I was a little tired from the > deduplication project. No problem. Apologies for the delayed revision as well. > I taught pageinspect to display a "htid" field for pivot tuples > recently, ma

Re: Improve search for missing parent downlinks in amcheck

2020-03-02 Thread Peter Geoghegan
Hi Alexander, Apologies for the delayed response. I was a little tired from the deduplication project. On Mon, Feb 24, 2020 at 2:54 PM Alexander Korotkov wrote: > Thank you for your review. The revised version is attached. This has bitrot, because of the deduplication patch. Shouldn't be hard

Re: Improve search for missing parent downlinks in amcheck

2020-02-24 Thread Alexander Korotkov
Hi! Thank you for your review. The revised version is attached. On Wed, Feb 19, 2020 at 1:16 AM Peter Geoghegan wrote: > On Tue, Feb 18, 2020 at 2:16 AM Alexander Korotkov > wrote: > > > Don't need the "!P_ISLEAF()" here. > > > > Why don't I need. bt_downlink_connectivity_check() checks one l

Re: Improve search for missing parent downlinks in amcheck

2020-02-18 Thread Peter Geoghegan
On Tue, Feb 18, 2020 at 2:16 AM Alexander Korotkov wrote: > Great, thank you very much! No problem! My remarks here are based on "amcheck-btree-improve-missing-parent-downlinks-check-6.patch". I have found a false positive corruption report bug in this latest version -- see note below about inco

Re: Improve search for missing parent downlinks in amcheck

2020-02-18 Thread Alexander Korotkov
On Tue, Feb 18, 2020 at 1:15 PM Alexander Korotkov wrote: > On Fri, Jan 24, 2020 at 4:31 AM Peter Geoghegan wrote: > > On Wed, Jan 22, 2020 at 6:41 PM Alexander Korotkov > > wrote: > > > Rebased patch is attached. Sorry for so huge delay. > > > > I really like this patch. Your interest in amche

Re: Improve search for missing parent downlinks in amcheck

2020-02-18 Thread Alexander Korotkov
Hi, Peter! On Fri, Jan 24, 2020 at 4:31 AM Peter Geoghegan wrote: > On Wed, Jan 22, 2020 at 6:41 PM Alexander Korotkov > wrote: > > Rebased patch is attached. Sorry for so huge delay. > > I really like this patch. Your interest in amcheck is something that > makes me feel good about having put

Re: Improve search for missing parent downlinks in amcheck

2020-01-23 Thread Peter Geoghegan
On Thu, Jan 23, 2020 at 5:40 PM Peter Geoghegan wrote: > I suppose the alternative is to get the high key of the parent's left > sibling, rather than going to the parent's parent (i.e. our > grandparent). That would probably be the best way to get a separator > key to compare against the high key

Re: Improve search for missing parent downlinks in amcheck

2020-01-23 Thread Peter Geoghegan
On Thu, Jan 23, 2020 at 5:31 PM Peter Geoghegan wrote: > In summary: I suppose that we can also solve "the cousin problem" > quite easily, but only for rightmost cousins within a subtree -- > leftmost cousins might be too messy to verify for it to be worth it. > We don't want to have to jump two o

Re: Improve search for missing parent downlinks in amcheck

2020-01-23 Thread Peter Geoghegan
On Wed, Jan 22, 2020 at 6:41 PM Alexander Korotkov wrote: > Rebased patch is attached. Sorry for so huge delay. I really like this patch. Your interest in amcheck is something that makes me feel good about having put so much work into it myself. Here are some review comments: > + /* > +*

Re: Improve search for missing parent downlinks in amcheck

2020-01-22 Thread Alexander Korotkov
On Fri, Jan 17, 2020 at 1:05 AM Tomas Vondra wrote: > On Fri, Nov 29, 2019 at 03:03:01PM +0900, Michael Paquier wrote: > >On Mon, Aug 19, 2019 at 01:15:19AM +0300, Alexander Korotkov wrote: > >> The revised patch seems to fix all of above. > > > >The latest patch is failing to apply. Please provi

Re: Improve search for missing parent downlinks in amcheck

2020-01-16 Thread Tomas Vondra
On Fri, Nov 29, 2019 at 03:03:01PM +0900, Michael Paquier wrote: On Mon, Aug 19, 2019 at 01:15:19AM +0300, Alexander Korotkov wrote: The revised patch seems to fix all of above. The latest patch is failing to apply. Please provide a rebase. This still does not apply (per cputube). Can you p

Re: Improve search for missing parent downlinks in amcheck

2019-11-28 Thread Michael Paquier
On Mon, Aug 19, 2019 at 01:15:19AM +0300, Alexander Korotkov wrote: > The revised patch seems to fix all of above. The latest patch is failing to apply. Please provide a rebase. -- Michael signature.asc Description: PGP signature

Re: Improve search for missing parent downlinks in amcheck

2019-08-18 Thread Alexander Korotkov
On Tue, Aug 13, 2019 at 11:44 PM Peter Geoghegan wrote: > > In this revision check for missing downlinks is combined with > > bt_downlink_check(). So, pages visited by bt_downlink_check() patch > > doesn't cause extra accessed. It only causes following additional > > page accesses: > > 1) Downli

Re: Improve search for missing parent downlinks in amcheck

2019-08-13 Thread Peter Geoghegan
On Mon, Aug 12, 2019 at 12:01 PM Alexander Korotkov wrote: > BTW, there is next revision of patch I'm proposing for v13. Cool. > In this revision check for missing downlinks is combined with > bt_downlink_check(). So, pages visited by bt_downlink_check() patch > doesn't cause extra accessed. I

Re: Improve search for missing parent downlinks in amcheck

2019-08-12 Thread Alexander Korotkov
On Fri, Jul 19, 2019 at 3:21 AM Peter Geoghegan wrote: > On Tue, Apr 30, 2019 at 5:58 PM Peter Geoghegan wrote: > > I will think about a simple fix, but after the upcoming point release. > > There is no hurry. > > Attached draft patch uses RelationGetNumberOfBlocks() to size each of > the two Blo

Re: Improve search for missing parent downlinks in amcheck

2019-07-18 Thread Peter Geoghegan
On Tue, Apr 30, 2019 at 5:58 PM Peter Geoghegan wrote: > I will think about a simple fix, but after the upcoming point release. > There is no hurry. Attached draft patch uses RelationGetNumberOfBlocks() to size each of the two Bloom filters that may be used by amcheck to perform verification. Th

Re: Improve search for missing parent downlinks in amcheck

2019-07-08 Thread Peter Geoghegan
On Sun, Jul 7, 2019 at 7:53 PM Thomas Munro wrote: > On Wed, May 1, 2019 at 12:58 PM Peter Geoghegan wrote: > > I will think about a simple fix, but after the upcoming point release. > > There is no hurry. > > A bureaucratic question: What should the status be for this CF entry? I have plans to

Re: Improve search for missing parent downlinks in amcheck

2019-07-07 Thread Thomas Munro
On Wed, May 1, 2019 at 12:58 PM Peter Geoghegan wrote: > On Sun, Apr 28, 2019 at 10:15 AM Alexander Korotkov > wrote: > > I think this definitely not bug fix. Bloom filter was designed to be > > lossy, no way blaming it for that :) > > I will think about a simple fix, but after the upcoming poin

Re: Improve search for missing parent downlinks in amcheck

2019-04-30 Thread Peter Geoghegan
On Sun, Apr 28, 2019 at 10:15 AM Alexander Korotkov wrote: > I think this definitely not bug fix. Bloom filter was designed to be > lossy, no way blaming it for that :) I will think about a simple fix, but after the upcoming point release. There is no hurry. -- Peter Geoghegan

Re: Improve search for missing parent downlinks in amcheck

2019-04-28 Thread Alexander Korotkov
On Sun, Apr 28, 2019 at 4:36 AM Peter Geoghegan wrote: > On Sat, Apr 27, 2019 at 5:13 PM Alexander Korotkov > wrote: > > Yes, increasing of Bloom filter size also helps. But my intention was > > to make non-lossy check here. > > Why is that your intention? Do you want to do this as a feature for

Re: Improve search for missing parent downlinks in amcheck

2019-04-27 Thread Peter Geoghegan
On Sat, Apr 27, 2019 at 5:13 PM Alexander Korotkov wrote: > Yes, increasing of Bloom filter size also helps. But my intention was > to make non-lossy check here. Why is that your intention? Do you want to do this as a feature for Postgres 13, or do you want to treat this as a bug that we need to

Re: Improve search for missing parent downlinks in amcheck

2019-04-27 Thread Peter Geoghegan
On Sat, Apr 27, 2019 at 5:13 PM Alexander Korotkov wrote: > Yes, increasing of Bloom filter size also helps. But my intention was > to make non-lossy check here. I agree that that might be a good goal, but I am interested in knowing if there is something naive about how the downlinkfilter Bloom

Re: Improve search for missing parent downlinks in amcheck

2019-04-27 Thread Alexander Korotkov
On Tue, Apr 16, 2019 at 10:00 PM Peter Geoghegan wrote: > > On Mon, Apr 15, 2019 at 7:30 PM Alexander Korotkov > wrote: > > Currently we amcheck supports lossy checking for missing parent > > downlinks. It collects bitmap of downlink hashes and use it to check > > subsequent tree level. We've e

Re: Improve search for missing parent downlinks in amcheck

2019-04-27 Thread Peter Geoghegan
On Sat, Apr 27, 2019 at 4:57 PM Alexander Korotkov wrote: > "rootdescend" is cool type of check. Thank you for noticing, I wasn't aware > of it. > But can it detect the missing downlink in following situation? > > A > / \ > B <-> C <-> D > > Here A has downlinks to B and D, wh

Re: Improve search for missing parent downlinks in amcheck

2019-04-27 Thread Alexander Korotkov
On Tue, Apr 16, 2019 at 10:04 PM Peter Geoghegan wrote: > > On Tue, Apr 16, 2019 at 12:00 PM Peter Geoghegan wrote: > > Can you be more specific? What was the cause of the corruption? I'm > > always very interested in hearing about cases that amcheck could have > > detected, but didn't. > > FWIW,

Re: Improve search for missing parent downlinks in amcheck

2019-04-25 Thread Peter Geoghegan
On Tue, Apr 16, 2019 at 12:00 PM Peter Geoghegan wrote: > On Mon, Apr 15, 2019 at 7:30 PM Alexander Korotkov > wrote: > > Currently we amcheck supports lossy checking for missing parent > > downlinks. It collects bitmap of downlink hashes and use it to check > > subsequent tree level. We've exp

Re: Improve search for missing parent downlinks in amcheck

2019-04-16 Thread Peter Geoghegan
On Tue, Apr 16, 2019 at 12:00 PM Peter Geoghegan wrote: > Can you be more specific? What was the cause of the corruption? I'm > always very interested in hearing about cases that amcheck could have > detected, but didn't. FWIW, v4 indexes in Postgres 12 will support the new "rootdescend" verifica

Re: Improve search for missing parent downlinks in amcheck

2019-04-16 Thread Peter Geoghegan
On Mon, Apr 15, 2019 at 7:30 PM Alexander Korotkov wrote: > Currently we amcheck supports lossy checking for missing parent > downlinks. It collects bitmap of downlink hashes and use it to check > subsequent tree level. We've experienced some large corrupted indexes > which pass this check due t