Re: Avoid full GIN index scan when possible

2020-01-17 Thread Alexander Korotkov
On Sat, Jan 18, 2020 at 12:48 AM Tom Lane wrote: > Alexander Korotkov writes: > > On Wed, Jan 15, 2020 at 2:03 AM Tom Lane wrote: > >> Hmm ... yeah, these test cases are not large enough to exercise any > >> lossy-page cases, are they? I doubt we should try to make a new regression > >> test

Re: Avoid full GIN index scan when possible

2020-01-17 Thread Tom Lane
Alexander Korotkov writes: > On Wed, Jan 15, 2020 at 2:03 AM Tom Lane wrote: >> Hmm ... yeah, these test cases are not large enough to exercise any >> lossy-page cases, are they? I doubt we should try to make a new regression >> test that is that big. (But if there is one already, maybe we

Re: Avoid full GIN index scan when possible

2020-01-17 Thread Alexander Korotkov
On Sat, Jan 18, 2020 at 12:33 AM Alexander Korotkov wrote: > So, I think we don't need so huge regression test to exercise this corner > case. Forgot to mention. I'm going to push v15 if no objections. -- Alexander Korotkov Postgres Professional: http://www.postgrespro.com The Russian

Re: Avoid full GIN index scan when possible

2020-01-17 Thread Alexander Korotkov
On Wed, Jan 15, 2020 at 2:03 AM Tom Lane wrote: > Alexander Korotkov writes: > > On Tue, Jan 14, 2020 at 9:43 PM Tom Lane wrote: > >> One thing I'm still not happy about is the comment in > >> collectMatchesForHeapRow. v12 failed to touch that at all, so I tried to > >> fill it in, but I'm not

Re: Avoid full GIN index scan when possible

2020-01-14 Thread Tom Lane
Alexander Korotkov writes: > On Tue, Jan 14, 2020 at 9:43 PM Tom Lane wrote: >> One thing I'm still not happy about is the comment in >> collectMatchesForHeapRow. v12 failed to touch that at all, so I tried to >> fill it in, but I'm not sure if my explanation is good. > I've tried to rephrase

Re: Avoid full GIN index scan when possible

2020-01-14 Thread Alexander Korotkov
On Wed, Jan 15, 2020 at 1:47 AM Alexander Korotkov wrote: > I also had concerns about how excludeOnly keys work with lossy pages. > I didn't find exact error. But I've added code, which skips > excludeOnly keys checks for lossy pages. They aren't going to exclude > any lossy page anyway. So,

Re: Avoid full GIN index scan when possible

2020-01-14 Thread Alexander Korotkov
Hi! On Tue, Jan 14, 2020 at 9:43 PM Tom Lane wrote: > Alexander Korotkov writes: > > Updated patch is attached. It contains more comments as well as commit > > message. > > I reviewed this a little bit. I agree this seems way more straightforward > than the patches we've been considering so

Re: Avoid full GIN index scan when possible

2020-01-14 Thread Tom Lane
Alexander Korotkov writes: > Updated patch is attached. It contains more comments as well as commit > message. I reviewed this a little bit. I agree this seems way more straightforward than the patches we've been considering so far. I wasn't too happy with the state of the comments, so I

Re: Avoid full GIN index scan when possible

2020-01-13 Thread Alexander Korotkov
Updated patch is attached. It contains more comments as well as commit message. On Sun, Jan 12, 2020 at 12:10 AM Alexander Korotkov wrote: > On Sat, Jan 11, 2020 at 3:19 PM Julien Rouhaud wrote: > > While at it, IIUC only excludeOnly key can have nrequired == 0 (if > > that's the case, this

Re: Avoid full GIN index scan when possible

2020-01-11 Thread Alexander Korotkov
Hi! Thank you for feedback! On Sat, Jan 11, 2020 at 3:19 PM Julien Rouhaud wrote: > On Sat, Jan 11, 2020 at 1:10 AM Alexander Korotkov > wrote: > > > > On Fri, Jan 10, 2020 at 7:36 PM Alexander Korotkov > > wrote: > > > > > > On Fri, Jan 10, 2020 at 6:31 PM Tom Lane wrote: > > >> > > >>

Re: Avoid full GIN index scan when possible

2020-01-11 Thread Julien Rouhaud
Hi, On Sat, Jan 11, 2020 at 1:10 AM Alexander Korotkov wrote: > > On Fri, Jan 10, 2020 at 7:36 PM Alexander Korotkov > wrote: > > > > On Fri, Jan 10, 2020 at 6:31 PM Tom Lane wrote: > >> > >> Alexander Korotkov writes: > >> > So, I think v10 is a version of patch, which can be committed after

Re: Avoid full GIN index scan when possible

2020-01-10 Thread Alexander Korotkov
On Fri, Jan 10, 2020 at 7:36 PM Alexander Korotkov wrote: > > On Fri, Jan 10, 2020 at 6:31 PM Tom Lane wrote: >> >> Alexander Korotkov writes: >> > So, I think v10 is a version of patch, which can be committed after >> > some cleanup. And we can try doing better nulls handling in a separate >>

Re: Avoid full GIN index scan when possible

2020-01-10 Thread Alexander Korotkov
On Fri, Jan 10, 2020 at 6:31 PM Tom Lane wrote: > Alexander Korotkov writes: > > So, I think v10 is a version of patch, which can be committed after > > some cleanup. And we can try doing better nulls handling in a separate > > patch. > > The cfbot reports that this doesn't pass regression

Re: Avoid full GIN index scan when possible

2020-01-10 Thread Tom Lane
Alexander Korotkov writes: > So, I think v10 is a version of patch, which can be committed after > some cleanup. And we can try doing better nulls handling in a separate > patch. The cfbot reports that this doesn't pass regression testing. I haven't looked into why not.

Re: Avoid full GIN index scan when possible

2020-01-08 Thread Alexander Korotkov
Hi, Tomas! Thank you for your feedback! On Mon, Jan 6, 2020 at 6:22 PM Tomas Vondra wrote: > Yeah, I can confirm those results, although on my system the timings are > a bit different (I haven't tested v8): > > |Query time, ms >

Re: Avoid full GIN index scan when possible

2020-01-06 Thread Tomas Vondra
On Fri, Dec 27, 2019 at 04:36:14AM +0300, Nikita Glukhov wrote: On 26.12.2019 4:59, Alexander Korotkov wrote:  I've tried to add patch #4 to comparison, but I've catch assertion failure. TRAP: FailedAssertion("key->includeNonMatching", File: "ginget.c", Line: 1340) There simply should be

Re: Avoid full GIN index scan when possible

2019-12-26 Thread Nikita Glukhov
On 26.12.2019 4:59, Alexander Korotkov wrote:  I've tried to add patch #4 to comparison, but I've catch assertion failure. TRAP: FailedAssertion("key->includeNonMatching", File: "ginget.c", Line: 1340) There simply should be inverted condition in the assertion:

Re: Avoid full GIN index scan when possible

2019-12-25 Thread Alexander Korotkov
On Wed, Dec 25, 2019 at 8:25 AM Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > Patch requires further polishing including comments, minor refactoring > etc. I'm going to continue work on this. > I also run the same performance comparison as Nikita [1] on my laptop. The results are

Re: Avoid full GIN index scan when possible

2019-12-24 Thread Alexander Korotkov
On Sat, Nov 23, 2019 at 2:39 AM Nikita Glukhov wrote: > Attached 8th version of the patches. I've read this thread. I decided to rewrite the patch in the way, which I find simple and more clear. Attached is the draft patch written from scratch except regression tests, which were copied "as

Re: Avoid full GIN index scan when possible

2019-11-25 Thread Tom Lane
Michael Paquier writes: > On Sat, Nov 23, 2019 at 02:35:50AM +0300, Nikita Glukhov wrote: >> Attached 8th version of the patches. > Please be careful here. The CF entry was still marked as waiting on > author, but you sent a new patch series which has not been reviewed. > I have moved this patc

Re: Avoid full GIN index scan when possible

2019-11-25 Thread Michael Paquier
On Sat, Nov 23, 2019 at 02:35:50AM +0300, Nikita Glukhov wrote: > Attached 8th version of the patches. Please be careful here. The CF entry was still marked as waiting on author, but you sent a new patch series which has not been reviewed. I have moved this patc to next CF instead. -- Michael

Re: Avoid full GIN index scan when possible

2019-09-02 Thread Alvaro Herrera
On 2019-Aug-07, Tom Lane wrote: > I think this would be committable as it stands, except that replacing > an ALL scan with an EVERYTHING scan could be a performance regression > if the index contains many null items. We need to do something about > that before committing. Nikita, any word on

Re: Avoid full GIN index scan when possible

2019-08-07 Thread Tom Lane
Nikita Glukhov writes: > Attached 6th version of the patches. I spent a bit of time looking at these. Attached is a proposed revision of the 0001 patch, with some minor changes: * I didn't adopt your move of the "Non-default modes require the index to have placeholders" test to after the

Re: Avoid full GIN index scan when possible

2019-08-02 Thread Nikita Glukhov
Attached 6th version of the patches. On 01.08.2019 22:28, Tom Lane wrote: Alexander Korotkov writes: On Thu, Aug 1, 2019 at 9:59 PM Tom Lane wrote: While I've not attempted to fix that here, I wonder whether we shouldn't fix it by just forcing forcedRecheck to true in any case where we

Re: Avoid full GIN index scan when possible

2019-08-01 Thread Tom Lane
Alexander Korotkov writes: > On Thu, Aug 1, 2019 at 9:59 PM Tom Lane wrote: >> While I've not attempted to fix that here, I wonder whether we shouldn't >> fix it by just forcing forcedRecheck to true in any case where we discard >> an ALL qualifier. > +1 for setting forcedRecheck in any case we

Re: Avoid full GIN index scan when possible

2019-08-01 Thread Alexander Korotkov
On Thu, Aug 1, 2019 at 9:59 PM Tom Lane wrote: > > Julien Rouhaud writes: > > On Thu, Aug 1, 2019 at 4:37 PM Tom Lane wrote: > >> Eyeing this a bit further ... doesn't scanPendingInsert also need > >> to honor so->forcedRecheck? Something along the lines of > > > I think so. > > Yeah, it does

Re: Avoid full GIN index scan when possible

2019-08-01 Thread Tom Lane
Julien Rouhaud writes: > On Thu, Aug 1, 2019 at 4:37 PM Tom Lane wrote: >> Eyeing this a bit further ... doesn't scanPendingInsert also need >> to honor so->forcedRecheck? Something along the lines of > I think so. Yeah, it does --- the updated pg_trgm test attached fails if it doesn't.

Re: Avoid full GIN index scan when possible

2019-08-01 Thread Julien Rouhaud
On Thu, Aug 1, 2019 at 4:37 PM Tom Lane wrote: > > Julien Rouhaud writes: > > Attached v4 that should address all comments. > > Eyeing this a bit further ... doesn't scanPendingInsert also need > to honor so->forcedRecheck? Something along the lines of > > -

Re: Avoid full GIN index scan when possible

2019-08-01 Thread Tom Lane
Julien Rouhaud writes: > Attached v4 that should address all comments. Eyeing this a bit further ... doesn't scanPendingInsert also need to honor so->forcedRecheck? Something along the lines of - tbm_add_tuples(tbm, , 1, recheck); +

Re: Avoid full GIN index scan when possible

2019-08-01 Thread Julien Rouhaud
On Thu, Aug 1, 2019 at 12:13 PM Julien Rouhaud wrote: > > On Thu, Aug 1, 2019 at 8:43 AM Thomas Munro wrote: > > > > On Wed, Jul 31, 2019 at 5:28 AM Tom Lane wrote: > > > Meanwhile, I looked at the v3 patch, and it seems like it might not be > > > too far from committable. I think we should

Re: Avoid full GIN index scan when possible

2019-08-01 Thread Julien Rouhaud
On Thu, Aug 1, 2019 at 8:43 AM Thomas Munro wrote: > > On Wed, Jul 31, 2019 at 5:28 AM Tom Lane wrote: > > Meanwhile, I looked at the v3 patch, and it seems like it might not be > > too far from committable. I think we should *not* let this get bogged > > down in questions of whether EXPLAIN

Re: Avoid full GIN index scan when possible

2019-08-01 Thread Thomas Munro
On Wed, Jul 31, 2019 at 5:28 AM Tom Lane wrote: > Meanwhile, I looked at the v3 patch, and it seems like it might not be > too far from committable. I think we should *not* let this get bogged > down in questions of whether EXPLAIN can report which index quals were > used or ignored. That's a

Re: Avoid full GIN index scan when possible

2019-07-30 Thread Tom Lane
Marc Cousin writes: > By the way, while preparing this, I noticed that it seems that during this > kind of index scan, the interrupt signal is masked > for a very long time. Control-C takes a very long while to cancel the query. > But it's an entirely different problem :) Yeah, that seems like

Re: Avoid full GIN index scan when possible

2019-07-01 Thread Marc Cousin
On 29/06/2019 00:23, Julien Rouhaud wrote: > On Fri, Jun 28, 2019 at 10:16 PM Tom Lane wrote: >> >> Tomas Vondra writes: >>> On Fri, Jun 28, 2019 at 03:03:19PM -0400, Tom Lane wrote: I not only don't want that function in indxpath.c, I don't even want it to be known/called from there.

Re: Avoid full GIN index scan when possible

2019-06-29 Thread Alexander Korotkov
On Sat, Jun 29, 2019 at 3:51 PM Julien Rouhaud wrote: > On Sat, Jun 29, 2019 at 12:25 PM Tomas Vondra > wrote: > > > > On Sat, Jun 29, 2019 at 11:10:03AM +0200, Julien Rouhaud wrote: > > >On Sat, Jun 29, 2019 at 12:51 AM Nikita Glukhov > > >> -- patched > > >> EXPLAIN ANALYZE SELECT * FROM test

Re: Avoid full GIN index scan when possible

2019-06-29 Thread Alexander Korotkov
On Sat, Jun 29, 2019 at 1:25 PM Tomas Vondra wrote: > A related issue is that during costing is too late to modify cardinality > estimates, so the 'Bitmap Index Scan' will be expected to return fewer > rows than it actually returns (after ignoring the full-scan quals). > Ignoring redundant quals

Re: Avoid full GIN index scan when possible

2019-06-29 Thread Alexander Korotkov
Hi! On Sat, Jun 29, 2019 at 1:52 AM Nikita Glukhov wrote: > We have a similar solution for this problem. The idea is to avoid full index > scan inside GIN itself when we have some GIN entries, and forcibly recheck > all tuples if triconsistent() returns GIN_MAYBE for the keys that emitted no >

Re: Avoid full GIN index scan when possible

2019-06-29 Thread Tomas Vondra
On Sat, Jun 29, 2019 at 03:28:11PM +0200, Julien Rouhaud wrote: On Sat, Jun 29, 2019 at 3:11 PM Tomas Vondra wrote: On Sat, Jun 29, 2019 at 02:50:51PM +0200, Julien Rouhaud wrote: >On Sat, Jun 29, 2019 at 12:25 PM Tomas Vondra >> A related issue is that during costing is too late to modify

Re: Avoid full GIN index scan when possible

2019-06-29 Thread Julien Rouhaud
On Sat, Jun 29, 2019 at 3:11 PM Tomas Vondra wrote: > > On Sat, Jun 29, 2019 at 02:50:51PM +0200, Julien Rouhaud wrote: > >On Sat, Jun 29, 2019 at 12:25 PM Tomas Vondra > >> A related issue is that during costing is too late to modify cardinality > >> estimates, so the 'Bitmap Index Scan' will be

Re: Avoid full GIN index scan when possible

2019-06-29 Thread Tomas Vondra
On Sat, Jun 29, 2019 at 02:50:51PM +0200, Julien Rouhaud wrote: On Sat, Jun 29, 2019 at 12:25 PM Tomas Vondra wrote: On Sat, Jun 29, 2019 at 11:10:03AM +0200, Julien Rouhaud wrote: >On Sat, Jun 29, 2019 at 12:51 AM Nikita Glukhov >> -- patched >> EXPLAIN ANALYZE SELECT * FROM test WHERE t

Re: Avoid full GIN index scan when possible

2019-06-29 Thread Julien Rouhaud
On Sat, Jun 29, 2019 at 12:25 PM Tomas Vondra wrote: > > On Sat, Jun 29, 2019 at 11:10:03AM +0200, Julien Rouhaud wrote: > >On Sat, Jun 29, 2019 at 12:51 AM Nikita Glukhov > >> -- patched > >> EXPLAIN ANALYZE SELECT * FROM test WHERE t LIKE '%1234%' AND t LIKE '%1%'; > >>

Re: Avoid full GIN index scan when possible

2019-06-29 Thread Tomas Vondra
On Sat, Jun 29, 2019 at 11:10:03AM +0200, Julien Rouhaud wrote: On Sat, Jun 29, 2019 at 12:51 AM Nikita Glukhov wrote:> On 29.06.2019 1:23, Julien Rouhaud wrote: But that kinda resembles stuff we already have - selectivity/cost. So why shouldn't this be considered as part of costing? Yeah,

Re: Avoid full GIN index scan when possible

2019-06-29 Thread Julien Rouhaud
On Sat, Jun 29, 2019 at 12:51 AM Nikita Glukhov wrote:> > On 29.06.2019 1:23, Julien Rouhaud wrote: > > But that kinda resembles stuff we already have - selectivity/cost. So > why shouldn't this be considered as part of costing? > > Yeah, I'm not entirely convinced that we need anything new here.

Re: Avoid full GIN index scan when possible

2019-06-28 Thread Tomas Vondra
On Fri, Jun 28, 2019 at 04:16:23PM -0400, Tom Lane wrote: Tomas Vondra writes: On Fri, Jun 28, 2019 at 03:03:19PM -0400, Tom Lane wrote: I not only don't want that function in indxpath.c, I don't even want it to be known/called from there. If we need the ability for the index AM to

Re: Avoid full GIN index scan when possible

2019-06-28 Thread Nikita Glukhov
Hi! On 29.06.2019 1:23, Julien Rouhaud wrote: But that kinda resembles stuff we already have - selectivity/cost. So why shouldn't this be considered as part of costing? Yeah, I'm not entirely convinced that we need anything new here. The cost estimate function can detect such situations, and

Re: Avoid full GIN index scan when possible

2019-06-28 Thread Julien Rouhaud
On Fri, Jun 28, 2019 at 10:16 PM Tom Lane wrote: > > Tomas Vondra writes: > > On Fri, Jun 28, 2019 at 03:03:19PM -0400, Tom Lane wrote: > >> I not only don't want that function in indxpath.c, I don't even want > >> it to be known/called from there. If we need the ability for the index > >> AM

Re: Avoid full GIN index scan when possible

2019-06-28 Thread Tom Lane
Tomas Vondra writes: > On Fri, Jun 28, 2019 at 03:03:19PM -0400, Tom Lane wrote: >> I not only don't want that function in indxpath.c, I don't even want >> it to be known/called from there. If we need the ability for the index >> AM to editorialize on the list of indexable quals (which I'm not

Re: Avoid full GIN index scan when possible

2019-06-28 Thread Tomas Vondra
On Fri, Jun 28, 2019 at 03:03:19PM -0400, Tom Lane wrote: Julien Rouhaud writes: On Fri, Jun 28, 2019 at 6:10 PM Tomas Vondra wrote: 2) I'm not sure it's a good idea to add dependency on a specific AM type into indxpath.c. At the moment there are only two places, both referring to

Re: Avoid full GIN index scan when possible

2019-06-28 Thread Tom Lane
Julien Rouhaud writes: > On Fri, Jun 28, 2019 at 6:10 PM Tomas Vondra > wrote: >> 2) I'm not sure it's a good idea to add dependency on a specific AM type >> into indxpath.c. At the moment there are only two places, both referring >> to BTREE_AM_OID, do we really hard-code another OID here? >>

Re: Avoid full GIN index scan when possible

2019-06-28 Thread Julien Rouhaud
On Fri, Jun 28, 2019 at 6:10 PM Tomas Vondra wrote: > > I've briefly looked at the patch today. I think the idea is worthwhile, Thanks! > 2) I'm not sure it's a good idea to add dependency on a specific AM type > into indxpath.c. At the moment there are only two places, both referring > to

Re: Avoid full GIN index scan when possible

2019-06-28 Thread Tomas Vondra
Hi, I've briefly looked at the patch today. I think the idea is worthwhile, but I found a couple of issues with the patch: 1) The index_selfuncs.h header is included in the wrong place, it should be included before lsyscache.h (because 'i' < 'l'). 2) I'm not sure it's a good idea to add

Re: Avoid full GIN index scan when possible

2019-06-28 Thread Julien Rouhaud
On Sun, Mar 24, 2019 at 11:52 AM Julien Rouhaud wrote: > > Marc (in Cc) reported me a problematic query using a GIN index hit in > production. The issue is that even if an GIN opclass says that the > index can be used for an operator, it's still possible that some > values aren't really

Avoid full GIN index scan when possible

2019-03-24 Thread Julien Rouhaud
Hi, Marc (in Cc) reported me a problematic query using a GIN index hit in production. The issue is that even if an GIN opclass says that the index can be used for an operator, it's still possible that some values aren't really compatible and requires a full index scan. One simple example is