Testing Index Skip scan

2022-06-28 Thread Alexandre Felipe
included in the attached file). server_version 12.8 On slack #pgsql-hackers channel, @sfrost, it was suggested that what I described is achieved by index skip scan. How can I get a development build to test this feature? create table ordered_skip_index_scan_test (id serial not null constraint

Re: MDAM techniques and Index Skip Scan patch

2022-03-28 Thread Peter Geoghegan
On Mon, Mar 28, 2022 at 7:07 PM Tom Lane wrote: > Right, that's the case I had in mind --- apologies if my terminology > was faulty. btree can actually handle such a case now, but what it > fails to do is re-descend from the tree root instead of plowing > forward in the index to find the next

Re: MDAM techniques and Index Skip Scan patch

2022-03-28 Thread Tom Lane
Peter Geoghegan writes: > The terminology in this area is a mess. MySQL calls > SELECT-DISTINCT-that-matches-an-index "loose index scans". I think > that you're talking about skip scan when you say "loose index scan". > Skip scan is where there is an omitted prefix of columns in the SQL > query

Re: MDAM techniques and Index Skip Scan patch

2022-03-28 Thread Peter Geoghegan
On Mon, Mar 28, 2022 at 5:21 PM Tom Lane wrote: > In any case, what I was on about is _bt_preprocess_keys() and > adjacent code. I'm surprised that those aren't more expensive > than one palloc in _bt_first. Maybe that logic falls through very > quickly in simple cases, though. I assume that

Re: MDAM techniques and Index Skip Scan patch

2022-03-28 Thread Peter Geoghegan
On Tue, Mar 22, 2022 at 1:55 PM Tom Lane wrote: > Peter asked me off-list to spend some time thinking about the overall > direction we ought to be pursuing here. Thanks for taking a look! "5.5 Exploiting Key Prefixes" and "5.6 Ordered Retrieval" from "Modern B-Tree Techniques" are also good,

Re: MDAM techniques and Index Skip Scan patch

2022-03-28 Thread Tom Lane
Peter Geoghegan writes: > We could get rid of dynamic allocations for BTStackData in > _bt_first(), perhaps. The problem is that there is no simple, > reasonable proof of the maximum height on a B-tree, even though a > B-Tree with more than 7 or 8 levels seems extraordinarily unlikely. Start

Re: MDAM techniques and Index Skip Scan patch

2022-03-28 Thread Peter Geoghegan
On Tue, Mar 22, 2022 at 4:06 PM Andres Freund wrote: > Are you thinking of just moving the setup stuff in nbtree (presumably parts of > _bt_first() / _bt_preprocess_keys()) or also stuff in > ExecIndexBuildScanKeys()? > > The latter does show up a bit more heavily in profiles than nbtree specific

Re: Index Skip Scan (new UniqueKeys)

2022-03-24 Thread Jesper Pedersen
On 6/9/20 06:22, Dmitry Dolgov wrote: Here is a new version of index skip scan patch, based on v8 patch for UniqueKeys implementation from [1]. I want to start a new thread to simplify navigation, hopefully I didn't forget anyone who actively participated in the discussion. This CommitFest

Re: MDAM techniques and Index Skip Scan patch

2022-03-24 Thread Jesper Pedersen
On 3/23/22 18:22, Dmitry Dolgov wrote: The CF item could be set to RwF, what would you say, Jesper? We want to thank the community for the feedback that we have received over the years for this feature. Hopefully a future implementation can use Tom's suggestions to get closer to a

Re: MDAM techniques and Index Skip Scan patch

2022-03-23 Thread Dmitry Dolgov
> On Wed, Mar 23, 2022 at 05:32:46PM -0400, Tom Lane wrote: > Dmitry Dolgov <9erthali...@gmail.com> writes: > > On Tue, Mar 22, 2022 at 04:55:49PM -0400, Tom Lane wrote: > >> In short: I would throw out just about all the planner infrastructure > >> that's been proposed so far. It looks bulky,

Re: MDAM techniques and Index Skip Scan patch

2022-03-23 Thread Tom Lane
Dmitry Dolgov <9erthali...@gmail.com> writes: > On Tue, Mar 22, 2022 at 04:55:49PM -0400, Tom Lane wrote: >> In short: I would throw out just about all the planner infrastructure >> that's been proposed so far. It looks bulky, expensive, and >> drastically undercommented, and I don't think it's

Re: MDAM techniques and Index Skip Scan patch

2022-03-23 Thread Dmitry Dolgov
> On Tue, Mar 22, 2022 at 04:55:49PM -0400, Tom Lane wrote: > Peter Geoghegan writes: > > Like many difficult patches, the skip scan patch is not so much > > troubled by problems with the implementation as it is troubled by > > *ambiguity* about the design. Particularly concerning how skip scan >

Re: MDAM techniques and Index Skip Scan patch

2022-03-22 Thread Tom Lane
Andres Freund writes: > On 2022-03-22 16:55:49 -0400, Tom Lane wrote: >> BTW, I've had a bee in my bonnet for a long time about whether some of >> nbtree's scan setup work could be done once during planning, rather than >> over again during each indexscan start. > It does show up in

Re: MDAM techniques and Index Skip Scan patch

2022-03-22 Thread Andres Freund
Hi, On 2022-03-22 16:55:49 -0400, Tom Lane wrote: > 4. I find each of the above ideas to be far more attractive than > optimizing SELECT-DISTINCT-that-matches-an-index, so I don't really > understand why the current patchsets seem to be driven largely > by that single use-case. It's something

Re: MDAM techniques and Index Skip Scan patch

2022-03-22 Thread Tom Lane
Peter Geoghegan writes: > Like many difficult patches, the skip scan patch is not so much > troubled by problems with the implementation as it is troubled by > *ambiguity* about the design. Particularly concerning how skip scan > meshes with existing designs, as well as future designs -- >

Re: MDAM techniques and Index Skip Scan patch

2022-03-22 Thread Thomas Munro
On Tue, Mar 22, 2022 at 2:34 PM Andres Freund wrote: > IMO it's pretty clear that having "duelling" patches below one CF entry is a > bad idea. I think they should be split, with inactive approaches marked as > returned with feeback or whatnot. I have the impression that this thread is getting

Re: MDAM techniques and Index Skip Scan patch

2022-03-22 Thread Dmitry Dolgov
PlannerInfo and Path structures with the list of relevant unique expressions. It specifies which keys must be unique on the query level, and allows to leverage this into on the path level. At the moment only distinctClause makes use of such mechanism, which enables potential use of index skip scan. O

Re: MDAM techniques and Index Skip Scan patch

2022-03-21 Thread Andres Freund
Hi, On 2022-01-22 22:37:19 +0100, Dmitry Dolgov wrote: > > On Fri, Jan 14, 2022 at 04:03:41PM +0800, Julien Rouhaud wrote: > > Hi, > > > > On Fri, Jan 14, 2022 at 08:55:26AM +0100, Dmitry Dolgov wrote: > > > > > > FYI, I've attached this thread to the CF item as an informational one, > > > but as

Re: Index Skip Scan (new UniqueKeys)

2022-01-24 Thread Zhihong Yu
On Mon, Jan 24, 2022 at 8:51 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Sun, Jan 23, 2022 at 04:25:04PM -0800, Zhihong Yu wrote: > > On Sat, Jan 22, 2022 at 1:32 PM Dmitry Dolgov <9erthali...@gmail.com> > wrote: > > > Besides that the new patch version contains some cleaning up and >

Re: Index Skip Scan (new UniqueKeys)

2022-01-24 Thread Dmitry Dolgov
> On Sun, Jan 23, 2022 at 04:25:04PM -0800, Zhihong Yu wrote: > On Sat, Jan 22, 2022 at 1:32 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > Besides that the new patch version contains some cleaning up and > > addresses commentaries around leaf page pinning from [1]. The idea > > behind the

Re: Index Skip Scan (new UniqueKeys)

2022-01-23 Thread Zhihong Yu
dexScan support was also relocated into a separate patch, the first > > three patches are now only about IndexOnlyScan. > > > > * Last bits of reviews were incorporated and rebased. > > While the patch is still waiting for a review, I was motivated by the > thread [1] to t

Re: MDAM techniques and Index Skip Scan patch

2022-01-22 Thread Dmitry Dolgov
> On Fri, Jan 14, 2022 at 04:03:41PM +0800, Julien Rouhaud wrote: > Hi, > > On Fri, Jan 14, 2022 at 08:55:26AM +0100, Dmitry Dolgov wrote: > > > > FYI, I've attached this thread to the CF item as an informational one, > > but as there are some patches posted here, folks may get confused. For > >

Re: Index Skip Scan (new UniqueKeys)

2022-01-22 Thread Dmitry Dolgov
rest of them as an experiment field. > > * IndexScan support was also relocated into a separate patch, the first > three patches are now only about IndexOnlyScan. > > * Last bits of reviews were incorporated and rebased. While the patch is still waiting for a review, I was moti

Re: MDAM techniques and Index Skip Scan patch

2022-01-14 Thread Julien Rouhaud
Hi, On Fri, Jan 14, 2022 at 08:55:26AM +0100, Dmitry Dolgov wrote: > > FYI, I've attached this thread to the CF item as an informational one, > but as there are some patches posted here, folks may get confused. For > those who have landed here with no context, I feel obliged to mention > that

Re: MDAM techniques and Index Skip Scan patch

2022-01-13 Thread Dmitry Dolgov
> On Thu, Jan 13, 2022 at 03:27:08PM +, Floris Van Nee wrote: > > > > Could you send a rebased version? In the meantime I will change the status > > on the cf app to Waiting on Author. > > Attached a rebased version. FYI, I've attached this thread to the CF item as an informational one, but

Re: MDAM techniques and Index Skip Scan patch

2022-01-13 Thread Julien Rouhaud
Hi, On Sat, Oct 23, 2021 at 07:30:47PM +, Floris Van Nee wrote: > > From the patch series above, v9-0001/v9-0002 is the UniqueKeys patch series, > which focuses on the planner. v2-0001 is Dmitry's patch that extends it to > make it possible to use UniqueKeys for the skip scan. Both of these

Re: MDAM techniques and Index Skip Scan patch

2021-10-25 Thread Jesper Pedersen
can effort should read it thoroughly (if they haven't already). It's easy to see why people get excited about skip scan [2]. But there is a bigger picture here. Thanks for starting this thread ! The Index Skip Scan patch could affect a lot of areas, so keeping MDAM in mind is definitely important.

Re: MDAM techniques and Index Skip Scan patch

2021-10-23 Thread Dmitry Dolgov
> On Thu, Oct 21, 2021 at 07:16:00PM -0700, Peter Geoghegan wrote: > > My general concern is that the skip scan patch may currently be > structured in a way that paints us into a corner, MDAM-wise. > > Note that the MDAM paper treats skipping a prefix of columns as a case > where the prefix is

MDAM techniques and Index Skip Scan patch

2021-10-21 Thread Peter Geoghegan
I returned to the 1995 paper "Efficient Search of Multidimensional B-Trees" [1] as part of the process of reviewing v39 of the skip scan patch, which was posted back in May. It's a great paper, and anybody involved in the skip scan effort should read it thoroughly (if they haven't already). It's

Re: Index Skip Scan (new UniqueKeys)

2021-05-21 Thread Dmitry Dolgov
ons. It specifies which keys must be unique on the query level, and allows to leverage this into on the path level. At the moment only distinctClause makes use of such mechanism, which enables potential use of index skip scan. Originally proposed by David Rowley, based on the UniqueKey patch imple

Re: Index Skip Scan (new UniqueKeys)

2021-03-17 Thread Tomas Vondra
> per parts of the original patch series. >> I suggest looking for XXX and FIXME comments in all the review patches. >> >> >> 0001 >> >> >> >> >> 0002 >> >> > > In fact both 0001 & 0002 belong to another thread,

Re: Index Skip Scan (new UniqueKeys)

2021-03-17 Thread Dmitry Dolgov
ts in all the review patches. > > > 0001 > > > > > 0002 > > In fact both 0001 & 0002 belong to another thread, which these days span [1], [2]. I've included them only because they happened to be a dependency for index skip scan following David suggestions

Re: Index Skip Scan (new UniqueKeys)

2021-01-28 Thread Dmitry Dolgov
> On Thu, Jan 28, 2021 at 09:49:26PM +0900, Masahiko Sawada wrote: > Hi Dmitry, > > Status update for a commitfest entry. > > This patch entry has been "Waiting on Author" on CF app and the > discussion seems inactive from the last CF. Could you share the > current status of this patch? Heikki

Re: Index Skip Scan (new UniqueKeys)

2021-01-28 Thread Masahiko Sawada
Hi Dmitry, On Sun, Oct 25, 2020 at 1:45 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Tue, Oct 06, 2020 at 05:20:39PM +0200, Dmitry Dolgov wrote: > > > On Mon, Sep 21, 2020 at 05:59:32PM -0700, Peter Geoghegan wrote: > > > > > > * I see the following compiler warning: > > > > > >

Re: Index Skip Scan (new UniqueKeys)

2020-12-05 Thread Thomas Munro
On Wed, Dec 2, 2020 at 9:59 AM Heikki Linnakangas wrote: > On 01/12/2020 22:21, Dmitry Dolgov wrote: > >> On Mon, Nov 30, 2020 at 04:42:20PM +0200, Heikki Linnakangas wrote: > >> - I'm surprised you need a new index AM function (amskip) for this. Can't > >> you just restart the scan with

Re: Index Skip Scan (new UniqueKeys)

2020-12-05 Thread Dmitry Dolgov
> On Tue, Dec 01, 2020 at 10:59:22PM +0200, Heikki Linnakangas wrote: > > > > - Does this optimization apply to bitmap index scans? > > > > No, from what I understand it doesn't. > > Would it be hard to add? Don't need to solve everything in the first > version of this, but I think in principle

Re: Index Skip Scan (new UniqueKeys)

2020-12-01 Thread Heikki Linnakangas
On 01/12/2020 22:21, Dmitry Dolgov wrote: On Mon, Nov 30, 2020 at 04:42:20PM +0200, Heikki Linnakangas wrote: I had a quick look at this patch. I haven't been following this thread, so sorry if I'm repeating old arguments, but here we go: Thanks! - I'm surprised you need a new index AM

Re: Index Skip Scan (new UniqueKeys)

2020-12-01 Thread Dmitry Dolgov
> On Mon, Nov 30, 2020 at 04:42:20PM +0200, Heikki Linnakangas wrote: > > I had a quick look at this patch. I haven't been following this thread, so > sorry if I'm repeating old arguments, but here we go: Thanks! > - I'm surprised you need a new index AM function (amskip) for this. Can't > you

Re: Index Skip Scan (new UniqueKeys)

2020-11-30 Thread Heikki Linnakangas
On 24/10/2020 19:45, Dmitry Dolgov wrote: Here is a new version which doesn't require "scanstart" argument and contains few other changes to address the issues mentioned earlier. It's also based on the latest UniqueKeys patches with the valgrind issue fixed (as before they're attached also just

Re: Index Skip Scan (new UniqueKeys)

2020-10-06 Thread Dmitry Dolgov
> On Mon, Sep 21, 2020 at 05:59:32PM -0700, Peter Geoghegan wrote: > > * I see the following compiler warning: > > /code/postgresql/patch/build/../source/src/backend/optimizer/path/uniquekeys.c: > In function ‘populate_baserel_uniquekeys’: >

Re: Index Skip Scan (new UniqueKeys)

2020-09-21 Thread Peter Geoghegan
On Mon, Sep 21, 2020 at 5:59 PM Peter Geoghegan wrote: > That's all I have for now. One more thing. I don't think that this should be a bitwise AND: if ((offnum > maxoff) & (so->currPos.nextPage == P_NONE)) {

Re: Index Skip Scan (new UniqueKeys)

2020-09-21 Thread Peter Geoghegan
On Sat, Aug 15, 2020 at 7:09 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > Here is a new version that hopefully address most of the concerns > mentioned in this thread so far. As before, first two patches are taken > from UniqueKeys thread and attached only for the reference. List of > changes

Re: Index Skip Scan (new UniqueKeys)

2020-08-02 Thread Andy Fan
Hi David: Thanks for looking into this. On Fri, Jul 31, 2020 at 11:07 AM David Rowley wrote: > On Mon, 13 Jul 2020 at 10:18, Floris Van Nee > wrote: > > One question about the unique keys - probably for Andy or David: I've > looked in the archives to find arguments for/against using Expr

Re: Index Skip Scan (new UniqueKeys)

2020-07-30 Thread David Rowley
On Mon, 13 Jul 2020 at 10:18, Floris Van Nee wrote: > One question about the unique keys - probably for Andy or David: I've looked > in the archives to find arguments for/against using Expr nodes or > EquivalenceClasses in the Unique Keys patch. However, I couldn't really find > a clear answer

Re: Index Skip Scan (new UniqueKeys)

2020-07-27 Thread Dmitry Dolgov
> On Tue, Jul 21, 2020 at 04:35:55PM -0700, Peter Geoghegan wrote: > > > Well, it's obviously wrong, thanks for noticing. What is necessary is to > > compare two index tuples, the start and the next one, to test if they're > > the same (in which case if I'm not mistaken probably we can compare

RE: Index Skip Scan (new UniqueKeys)

2020-07-23 Thread Floris Van Nee
> > One UniqueKey can have multiple corresponding expressions, which gives us > also possibility of having one unique key with (t1.a, t2.a) and it looks now > similar to EquivalenceClass. > I believe the current definition of a unique key with two expressions (t1.a, t2.a) means that it's

Re: Index Skip Scan (new UniqueKeys)

2020-07-23 Thread Dmitry Dolgov
> On Tue, Jul 14, 2020 at 06:18:50PM +, Floris Van Nee wrote: > > Due to the other changes I made in > create_distinct_paths/query_has_uniquekeys_for, it will choose a correct plan > now, even without the EC_MUST_BE_REDUNDANT check though, so it's difficult to > give an actual failing test

Re: Index Skip Scan (new UniqueKeys)

2020-07-21 Thread Peter Geoghegan
On Sat, Jul 11, 2020 at 9:10 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > + currItem = >currPos.items[so->currPos.lastItem]; > > > + itup = (IndexTuple) (so->currTuples + > > > currItem->tupleOffset); > > > + nextOffset =

Re: Generic Index Skip Scan

2020-07-15 Thread David Rowley
On Thu, 16 Jul 2020 at 07:52, Floris Van Nee wrote: > > Besides the great efforts that Dmitry et al. are putting into the skip scan > for DISTINCT queries [1], I’m also still keen on extending the use of it > further. I’d like to address the limited cases in which skipping can occur > here. A

RE: Index Skip Scan (new UniqueKeys)

2020-07-14 Thread Floris Van Nee
> > > - the uniquekeys that is built, still contains some redundant keys, that are > normally eliminated from the path keys lists. > > I guess you're talking about: > > + if (EC_MUST_BE_REDUNDANT(ec)) > + continue; > > Can you add some test cases to your changes to show the effect of it? It

Re: Index Skip Scan (new UniqueKeys)

2020-07-14 Thread Dmitry Dolgov
> On Sun, Jul 12, 2020 at 12:48:47PM +, Floris Van Nee wrote: > > > > Good point, thanks for looking at this. With the latest planner version > > there > > are indeed more possibilities to use skipping. It never occured to me that > > some of those paths will still rely on index scan

Re: Index Skip Scan (new UniqueKeys)

2020-07-11 Thread Dmitry Dolgov
> On Fri, Jul 10, 2020 at 05:03:37PM +, Floris Van Nee wrote: > > Also took another look at the patch now, and found a case of incorrect > data. It looks related to the new way of creating the paths, as I > can't recall seeing this in earlier versions. > > create table t1 as select a,b,b%5 as

Re: Index Skip Scan (new UniqueKeys)

2020-07-11 Thread Dmitry Dolgov
> On Wed, Jul 08, 2020 at 03:44:26PM -0700, Peter Geoghegan wrote: > > On Tue, Jun 9, 2020 at 3:20 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > * Btree-implementation contains btree specific code to implement amskip, > > introduced in the previous patch. > > The way that you're dealing

RE: Index Skip Scan (new UniqueKeys)

2020-07-10 Thread Floris Van Nee
Hi Dmitry, Also took another look at the patch now, and found a case of incorrect data. It looks related to the new way of creating the paths, as I can't recall seeing this in earlier versions. create table t1 as select a,b,b%5 as c, random() as d from generate_series(1, 10) a,

Re: Index Skip Scan (new UniqueKeys)

2020-07-08 Thread Peter Geoghegan
On Tue, Jun 9, 2020 at 3:20 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > * Btree-implementation contains btree specific code to implement amskip, > introduced in the previous patch. The way that you're dealing with B-Tree tuples here needs to account for posting list tuples: > +

Re: Index Skip Scan (new UniqueKeys)

2020-06-29 Thread Dmitry Dolgov
> On Thu, Jun 11, 2020 at 04:14:07PM +0800, Andy Fan wrote: > > I just get the rough idea of patch, looks we have to narrow down the > user cases where we can use this method. Consider the below example: Hi Not exactly narrow down, but rather get rid of wrong usage of skipping for index scan.

Re: Index Skip Scan (new UniqueKeys)

2020-06-11 Thread Andy Fan
On Tue, Jun 9, 2020 at 6:20 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > Hi, > > Here is a new version of index skip scan patch, based on v8 patch for > UniqueKeys implementation from [1]. I want to start a new thread to > simplify navigation, hopefully I didn't forge

Re: Index Skip Scan

2020-06-02 Thread Andy Fan
On Tue, Jun 2, 2020 at 9:38 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Tue, Jun 02, 2020 at 08:36:31PM +0800, Andy Fan wrote: > > > > > Other than that to summarize current open points for future readers > > > (this thread somehow became quite big): > > > > > > * Making UniqueKeys

Re: Index Skip Scan

2020-06-02 Thread Dmitry Dolgov
> On Tue, Jun 02, 2020 at 08:36:31PM +0800, Andy Fan wrote: > > > Other than that to summarize current open points for future readers > > (this thread somehow became quite big): > > > > * Making UniqueKeys usage more generic to allow using skip scan for more > > use cases (hopefully it was

Re: Index Skip Scan

2020-06-02 Thread Andy Fan
On Wed, Apr 8, 2020 at 3:41 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Mon, Apr 06, 2020 at 06:31:08PM +, Floris Van Nee wrote: > > > > > Hm, I wasn't aware about this one, thanks for bringing this up. Btw, > Floris, I > > > would appreciate if in the future you can make it more

Re: Index Skip Scan

2020-05-15 Thread Dilip Kumar
On Fri, 15 May 2020 at 6:06 PM, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Wed, May 13, 2020 at 02:37:21PM +0530, Dilip Kumar wrote: > > > > + if (_bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, > dir)) > > + { > > > > Here we expect whether the "next" unique key can be

Re: Index Skip Scan

2020-05-15 Thread Dmitry Dolgov
> On Wed, May 13, 2020 at 02:37:21PM +0530, Dilip Kumar wrote: > > + if (_bt_scankey_within_page(scan, so->skipScanKey, so->currPos.buf, dir)) > + { > > Here we expect whether the "next" unique key can be found on this page > or not, but we are using the function which suggested whether the >

Re: Index Skip Scan

2020-05-13 Thread Peter Geoghegan
On Mon, Jan 20, 2020 at 5:05 PM Peter Geoghegan wrote: > You can add another assertion that calls a new utility function in > bufmgr.c. That can use the same logic as this existing assertion in > FlushOneBuffer(): > > Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr))); > > We haven't

Re: Index Skip Scan

2020-05-13 Thread Dilip Kumar
On Mon, May 11, 2020 at 4:55 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Mon, May 11, 2020 at 04:04:00PM +0530, Dilip Kumar wrote: > > > > > > +static inline bool > > > > +_bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key, > > > > + Buffer buf, ScanDirection dir) > > > > +{

Re: Index Skip Scan

2020-05-11 Thread Dmitry Dolgov
> On Mon, May 11, 2020 at 04:04:00PM +0530, Dilip Kumar wrote: > > > > +static inline bool > > > +_bt_scankey_within_page(IndexScanDesc scan, BTScanInsert key, > > > + Buffer buf, ScanDirection dir) > > > +{ > > > + OffsetNumber low, high; > > > + Page page = BufferGetPage(buf); > > > +

Re: Index Skip Scan

2020-05-11 Thread Dilip Kumar
on the leaf > page, so there is a point in searching. Is that incorrect? But IIUC, here we want to decide whether we will get the next key in the current page or not? Is my understanding is correct? So if our key (the last tuple key) is equal to the high key means the max key on this page is t

Re: Index Skip Scan

2020-05-10 Thread Dmitry Dolgov
al is equal to the high key then also there is no point in > searching on the current page so we can directly skip. >From nbtree/README and comments to functions like _bt_split I've got an impression that the high key could be equal to the last item on the leaf page, so there is a poi

Re: Index Skip Scan

2020-05-10 Thread Dmitry Dolgov
Sorry for late reply. > On Tue, Apr 14, 2020 at 09:19:22PM +1200, David Rowley wrote: > > I've not yet looked at the latest patch, but I did put some thoughts > into an email on the other thread that's been discussing UniqueKeys > [1]. > > I'm keen to hear thoughts on the plan I mentioned over

Re: Index Skip Scan

2020-04-14 Thread David Rowley
On Wed, 8 Apr 2020 at 07:40, Dmitry Dolgov <9erthali...@gmail.com> wrote: > Other than that to summarize current open points for future readers > (this thread somehow became quite big): > > * Making UniqueKeys usage more generic to allow using skip scan for more > use cases (hopefully it was

RE: Index Skip Scan

2020-04-07 Thread Floris Van Nee
> > * Suspicious performance difference between different type of workload, > mentioned by Tomas (unfortunately I had no chance yet to investigate). > His benchmark results indeed most likely point to multiple comparisons being done. Since the most likely place where these occur is

Re: Index Skip Scan

2020-04-07 Thread Dmitry Dolgov
oot, List *sortclauses, List *tlist); +extern List *make_pathkeys_for_uniquekeys(PlannerInfo *root, + List *sortclauses, + List *tlist); extern void initialize_mergeclause_eclasses(PlannerInfo *root, RestrictInfo *restrictinfo); extern void upda

RE: Index Skip Scan

2020-04-06 Thread Floris Van Nee
> > Hm, I wasn't aware about this one, thanks for bringing this up. Btw, Floris, I > would appreciate if in the future you can make it more visible that changes > you > suggest contain some fixes. E.g. it wasn't clear for me from your previous > email > that that's the case, and it doesn't

Re: Index Skip Scan

2020-04-06 Thread Dmitry Dolgov
> On Mon, Apr 6, 2020 at 1:14 PM Floris Van Nee > wrote: > > > > There's a number of ways we can force a 'filter' rather than an > > 'index condition'. Hm, I wasn't aware about this one, thanks for bringing this up. Btw, Floris, I would appreciate if in the future you can make it more visible

Re: Index Skip Scan

2020-04-06 Thread Dilip Kumar
On Mon, Apr 6, 2020 at 1:14 PM Floris Van Nee wrote: > > > > > On Sun, Apr 05, 2020 at 04:30:51PM +0530, Dilip Kumar wrote: > > > > > > > > I was just wondering how the distinct will work with the "skip scan" > > > > if we have some filter? I mean every time we select the unique row > > > > based

RE: Index Skip Scan

2020-04-06 Thread Floris Van Nee
> > > On Sun, Apr 05, 2020 at 04:30:51PM +0530, Dilip Kumar wrote: > > > > > > I was just wondering how the distinct will work with the "skip scan" > > > if we have some filter? I mean every time we select the unique row > > > based on the prefix key and that might get rejected by an external > >

Re: Index Skip Scan

2020-04-05 Thread Dilip Kumar
On Sun, Apr 5, 2020 at 9:39 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Sun, Apr 05, 2020 at 04:30:51PM +0530, Dilip Kumar wrote: > > > > I was just wondering how the distinct will work with the "skip scan" > > if we have some filter? I mean every time we select the unique row > >

Re: Index Skip Scan

2020-04-05 Thread Dmitry Dolgov
> On Sun, Apr 05, 2020 at 04:30:51PM +0530, Dilip Kumar wrote: > > I was just wondering how the distinct will work with the "skip scan" > if we have some filter? I mean every time we select the unique row > based on the prefix key and that might get rejected by an external > filter right? Not

Re: Index Skip Scan

2020-04-05 Thread Dilip Kumar
On Wed, Mar 25, 2020 at 2:19 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Wed, Mar 25, 2020 at 11:31:56AM +0530, Dilip Kumar wrote: > > > > Seems like you forgot to add the uniquekey.c file in the > > v33-0001-Unique-key.patch. > > Oh, you're right, thanks. Here is the corrected patch.

Re: Index Skip Scan

2020-03-25 Thread Dmitry Dolgov
nfo *root, RestrictInfo *restrictinfo); extern void update_mergeclause_eclasses(PlannerInfo *root, @@ -240,4 +243,12 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *root, extern void add_paths_to_append_rel(PlannerInfo *root, RelOptInfo *rel, List *live_childrels);

Re: Index Skip Scan

2020-03-25 Thread Dilip Kumar
t look similar to what you had in mind? > > In this version if all conditions are met and there are corresponding > unique keys, a new index skip scan path will be added to > unique_pathlists. In case if requested distinct clauses match with > unique keys, create_distinct_paths can ch

Re: Index Skip Scan

2020-03-24 Thread Andy Fan
t messages around that message for > > the context. hope it helps. > > Thanks for pointing out to this thread! Somehow I've missed it, and now > looks like we need to make some efforts to align patches for index skip > scan and distincClause elimination. > Yes:). Looks Index skip sca

Re: Index Skip Scan

2020-03-24 Thread Dmitry Dolgov
how I've missed it, and now looks like we need to make some efforts to align patches for index skip scan and distincClause elimination.

Re: Index Skip Scan

2020-03-24 Thread Dmitry Dolgov
the Path's uniquekeys rather than what type of path it > is. Thanks for the suggestion. As a result of the discussion I've modified the patch, does it look similar to what you had in mind? In this version if all conditions are met and there are corresponding unique keys, a new index skip

Re: Index Skip Scan

2020-03-23 Thread Andy Fan
> > > On Mon, Mar 23, 2020 at 1:55 AM Floris Van Nee > wrote: > > I'm unsure which version number to give this patch (to continue with > numbers from previous skip scan patches, or to start numbering from scratch > again). It's a rather big change, so one could argue it's mostly a separate >

Re: Index Skip Scan

2020-03-22 Thread Thomas Munro
Hi Floris, On Sun, Mar 22, 2020 at 11:00 AM Floris Van Nee wrote: > create index on t1 (a,b,c); > select * from t1 where b in (100, 200); > Execution Time: 2.464 ms > Execution Time: 252.224 ms > Execution Time: 244.872 ms Wow. This is very cool work and I'm sure it will become a major

Re: Index Skip Scan

2020-03-11 Thread David Rowley
On Wed, 11 Mar 2020 at 16:44, Andy Fan wrote: >> >> >> I think the UniqueKeys may need to be changed from using >> EquivalenceClasses to use Exprs instead. > > > When I try to understand why UniqueKeys needs EquivalenceClasses, > see your comments here. I feel that FuncExpr can't be > used to

Re: Index Skip Scan

2020-03-11 Thread Andy Fan
On Tue, Mar 10, 2020 at 4:32 AM James Coleman wrote: > On Mon, Mar 9, 2020 at 3:56 PM Dmitry Dolgov <9erthali...@gmail.com> > wrote: > > > > Assuming we'll implement it in a way that we do not know about what kind > > of path type is that in create_distinct_path, then it can also work for > >

Re: Index Skip Scan

2020-03-10 Thread Andy Fan
> > > I think the UniqueKeys may need to be changed from using > EquivalenceClasses to use Exprs instead. > When I try to understand why UniqueKeys needs EquivalenceClasses, see your comments here. I feel that FuncExpr can't be used to as a UniquePath even we can create unique index on f(a) and

Re: Index Skip Scan

2020-03-10 Thread David Rowley
On Wed, 11 Mar 2020 at 01:38, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > >On Tue, Mar 10, 2020 at 09:29:32AM +1300, David Rowley wrote: > > There's also some weird looking assumptions that an EquivalenceMember > > can only be a Var in create_distinct_paths(). I think you're only > > saved

Re: Index Skip Scan

2020-03-10 Thread Dmitry Dolgov
> >On Tue, Mar 10, 2020 at 09:29:32AM +1300, David Rowley wrote: > On Tue, 10 Mar 2020 at 08:56, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > Assuming we'll implement it in a way that we do not know about what kind > > of path type is that in create_distinct_path, then it can also work for > >

Re: Index Skip Scan

2020-03-09 Thread James Coleman
On Mon, Mar 9, 2020 at 3:56 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > Assuming we'll implement it in a way that we do not know about what kind > of path type is that in create_distinct_path, then it can also work for > ProjectionPath or anything else (if UniqueKeys are present). But then

Re: Index Skip Scan

2020-03-09 Thread David Rowley
On Tue, 10 Mar 2020 at 08:56, Dmitry Dolgov <9erthali...@gmail.com> wrote: > Assuming we'll implement it in a way that we do not know about what kind > of path type is that in create_distinct_path, then it can also work for > ProjectionPath or anything else (if UniqueKeys are present). But then >

Re: Index Skip Scan

2020-03-09 Thread Dmitry Dolgov
> On Mon, Mar 09, 2020 at 10:27:26AM +1300, David Rowley wrote: > > > > I think the changes in create_distinct_paths() need more work. The > > > way I think this should work is that create_distinct_paths() gets to > > > know exactly nothing about what path types support the elimination of > > >

Re: Index Skip Scan

2020-03-08 Thread David Rowley
On Mon, 9 Mar 2020 at 03:21, Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > > > I've been looking over v32 of the patch and have a few comments > > regarding the planner changes. > > Thanks for the commentaries! > > > I think the changes in create_distinct_paths() need more work. The > > way

Re: Index Skip Scan

2020-03-08 Thread Dmitry Dolgov
> On Wed, Mar 04, 2020 at 11:32:00AM +1300, David Rowley wrote: > > I've been looking over v32 of the patch and have a few comments > regarding the planner changes. Thanks for the commentaries! > I think the changes in create_distinct_paths() need more work. The > way I think this should work

Re: Index Skip Scan

2020-03-07 Thread David Rowley
g skip-scan for join inputs, or on top of the join? The skip index scan Path would still be created at the base rel level, but the join path on the join relation would have one of the sub-paths of the join as an index skip scan. An example query that could make use of this is: SELECT * FROM some_table

Re: Index Skip Scan

2020-03-06 Thread Tomas Vondra
On Wed, Mar 04, 2020 at 11:32:00AM +1300, David Rowley wrote: On Tue, 18 Feb 2020 at 05:24, Dmitry Dolgov <9erthali...@gmail.com> wrote: Here is something similar to what I had in mind. (changing to this email address for future emails) Hi, I've been looking over v32 of the patch and have a

Re: Index Skip Scan

2020-03-03 Thread David Rowley
On Tue, 18 Feb 2020 at 05:24, Dmitry Dolgov <9erthali...@gmail.com> wrote: > Here is something similar to what I had in mind. (changing to this email address for future emails) Hi, I've been looking over v32 of the patch and have a few comments regarding the planner changes. I think the

Re: Index Skip Scan

2020-02-17 Thread Dmitry Dolgov
nfo *root, + List *sortclauses, + List *tlist); extern void initialize_mergeclause_eclasses(PlannerInfo *root, RestrictInfo *restrictinfo); extern void update_mergeclause_eclasses(PlannerInfo *root, @@ -240,4 +243,12 @@ extern PathKey *make_canonical_pathkey(PlannerInfo *r

Re: Index Skip Scan

2020-02-14 Thread Dmitry Dolgov
> On Fri, Feb 14, 2020 at 05:23:13PM +0900, Kyotaro Horiguchi wrote: > The first attached (renamed to .txt not to confuse the cfbots) is a > small patch that makes sure if _bt_readpage is called with the proper > condition as written in its comment, that is, caller must have pinned > and

  1   2   3   >