Re: [HACKERS] Hash Indexes

2016-12-20 Thread Amit Kapila
On Tue, Dec 20, 2016 at 7:44 PM, Robert Haas wrote: > On Tue, Dec 20, 2016 at 9:01 AM, Amit Kapila wrote: >> On Tue, Dec 20, 2016 at 7:11 PM, Robert Haas wrote: >>> On Tue, Dec 20, 2016 at 4:51 AM, Amit Kapila >>> wrote: We have mainly four actions for squeeze operation, add tuples to the

Re: [HACKERS] Hash Indexes

2016-12-20 Thread Robert Haas
On Tue, Dec 20, 2016 at 9:01 AM, Amit Kapila wrote: > On Tue, Dec 20, 2016 at 7:11 PM, Robert Haas wrote: >> On Tue, Dec 20, 2016 at 4:51 AM, Amit Kapila wrote: >>> We have mainly four actions for squeeze operation, add tuples to the >>> write page, empty overflow page, unlinks overflow page, ma

Re: [HACKERS] Hash Indexes

2016-12-20 Thread Amit Kapila
On Tue, Dec 20, 2016 at 7:11 PM, Robert Haas wrote: > On Tue, Dec 20, 2016 at 4:51 AM, Amit Kapila wrote: >> We have mainly four actions for squeeze operation, add tuples to the >> write page, empty overflow page, unlinks overflow page, make it free >> by setting the corresponding bit in overflow

Re: [HACKERS] Hash Indexes

2016-12-20 Thread Robert Haas
On Tue, Dec 20, 2016 at 4:51 AM, Amit Kapila wrote: > We have mainly four actions for squeeze operation, add tuples to the > write page, empty overflow page, unlinks overflow page, make it free > by setting the corresponding bit in overflow page. Now, if we don't > log the changes to write page a

Re: [HACKERS] Hash Indexes

2016-12-20 Thread Amit Kapila
On Mon, Dec 19, 2016 at 11:05 PM, Robert Haas wrote: > On Sun, Dec 18, 2016 at 8:54 AM, Amit Kapila wrote: >>> I committed remove-hash-wrtbuf and fix_dirty_marking_v1 but I've got >>> some reservations about fix_lock_chaining_v1. ISTM that the natural >>> fix here would be to change the API cont

Re: [HACKERS] Hash Indexes

2016-12-19 Thread Robert Haas
On Sun, Dec 18, 2016 at 8:54 AM, Amit Kapila wrote: >> I committed remove-hash-wrtbuf and fix_dirty_marking_v1 but I've got >> some reservations about fix_lock_chaining_v1. ISTM that the natural >> fix here would be to change the API contract for _hash_freeovflpage so >> that it doesn't release t

Re: [HACKERS] Hash Indexes

2016-12-18 Thread Amit Kapila
On Fri, Dec 16, 2016 at 9:57 PM, Robert Haas wrote: > On Thu, Dec 15, 2016 at 11:33 AM, Amit Kapila wrote: >> Attached are the two patches on top of remove-hash-wrtbuf. Patch >> fix_dirty_marking_v1.patch allows to mark the buffer dirty in one of >> the corner cases in _hash_freeovflpage() and

Re: [HACKERS] Hash Indexes

2016-12-16 Thread Robert Haas
On Thu, Dec 15, 2016 at 11:33 AM, Amit Kapila wrote: > Attached are the two patches on top of remove-hash-wrtbuf. Patch > fix_dirty_marking_v1.patch allows to mark the buffer dirty in one of > the corner cases in _hash_freeovflpage() and avoids to mark dirty > without need in _hash_squeezebucket

Re: [HACKERS] Hash Indexes

2016-12-15 Thread Amit Kapila
On Wed, Dec 14, 2016 at 10:47 PM, Robert Haas wrote: > On Wed, Dec 14, 2016 at 4:27 AM, Amit Kapila wrote: >> >> Yeah, it will fix the problem in hashbucketcleanup, but there are two >> other problems that need to be fixed for which I can send a separate >> patch. By the way, as mentioned to you

Re: [HACKERS] Hash Indexes

2016-12-14 Thread Robert Haas
On Wed, Dec 14, 2016 at 4:27 AM, Amit Kapila wrote: >> It's not required for enabling WAL. You don't *have* to release and >> reacquire the buffer lock; in fact, that just adds overhead. > > If we don't release the lock, then it will break the general coding > pattern of writing WAL (Acquire pin

Re: [HACKERS] Hash Indexes

2016-12-14 Thread Amit Kapila
On Tue, Dec 13, 2016 at 11:30 PM, Robert Haas wrote: > On Mon, Dec 12, 2016 at 9:21 PM, Amit Kapila wrote: >> The reason is to make the operations consistent in master and standby. >> In WAL patch, for clearing the SPLIT_CLEANUP flag, we write a WAL and >> if we don't release the lock after writi

Re: [HACKERS] Hash Indexes

2016-12-13 Thread Robert Haas
On Mon, Dec 12, 2016 at 9:21 PM, Amit Kapila wrote: > The reason is to make the operations consistent in master and standby. > In WAL patch, for clearing the SPLIT_CLEANUP flag, we write a WAL and > if we don't release the lock after writing a WAL, the operation can > appear on standby even before

Re: [HACKERS] Hash Indexes

2016-12-13 Thread Jesper Pedersen
On 12/11/2016 11:37 PM, Amit Kapila wrote: On Sun, Dec 11, 2016 at 11:54 AM, Amit Kapila wrote: On Wed, Dec 7, 2016 at 2:02 AM, Jeff Janes wrote: With above fixes, the test ran successfully for more than a day. There was a small typo in the previous patch which is fixed in attached. I don'

Re: [HACKERS] Hash Indexes

2016-12-12 Thread Amit Kapila
On Tue, Dec 13, 2016 at 2:51 AM, Robert Haas wrote: > On Sun, Dec 11, 2016 at 1:24 AM, Amit Kapila wrote: >> >> With above fixes, the test ran successfully for more than a day. > > Instead of doing this: > > +_hash_chgbufaccess(rel, bucket_buf, HASH_WRITE, HASH_NOLOCK); > +_hash_chgbufacc

Re: [HACKERS] Hash Indexes

2016-12-12 Thread Robert Haas
On Sun, Dec 11, 2016 at 1:24 AM, Amit Kapila wrote: > The reason for this and the similar error in vacuum was that in one of > the corner cases after freeing the overflow page and updating the link > for the previous bucket, we were not marking the buffer as dirty. So, > due to concurrent activit

Re: [HACKERS] Hash Indexes

2016-12-11 Thread Amit Kapila
On Mon, Dec 12, 2016 at 10:25 AM, Jeff Janes wrote: > On Sun, Dec 11, 2016 at 8:37 PM, Amit Kapila > wrote: >> >> On Sun, Dec 11, 2016 at 11:54 AM, Amit Kapila >> wrote: >> > On Wed, Dec 7, 2016 at 2:02 AM, Jeff Janes wrote: >> > >> > With above fixes, the test ran successfully for more than a

Re: [HACKERS] Hash Indexes

2016-12-11 Thread Jeff Janes
On Sun, Dec 11, 2016 at 8:37 PM, Amit Kapila wrote: > On Sun, Dec 11, 2016 at 11:54 AM, Amit Kapila > wrote: > > On Wed, Dec 7, 2016 at 2:02 AM, Jeff Janes wrote: > > > > With above fixes, the test ran successfully for more than a day. > > > > There was a small typo in the previous patch which

Re: [HACKERS] Hash Indexes

2016-12-11 Thread Amit Kapila
On Tue, Dec 6, 2016 at 1:29 PM, Jeff Janes wrote: > On Thu, Dec 1, 2016 at 10:54 PM, Amit Kapila > wrote: > > With the latest HASH WAL patch applied, I get different but apparently > related errors > > 41993 UPDATE XX002 2016-12-05 22:28:45.333 PST:ERROR: index "foo_index_idx" > contains corrupt

Re: [HACKERS] Hash Indexes

2016-12-11 Thread Amit Kapila
On Sun, Dec 11, 2016 at 11:54 AM, Amit Kapila wrote: > On Wed, Dec 7, 2016 at 2:02 AM, Jeff Janes wrote: > > With above fixes, the test ran successfully for more than a day. > There was a small typo in the previous patch which is fixed in attached. I don't think it will impact the test results

Re: [HACKERS] Hash Indexes

2016-12-10 Thread Amit Kapila
On Wed, Dec 7, 2016 at 2:02 AM, Jeff Janes wrote: > On Tue, Dec 6, 2016 at 4:00 AM, Amit Kapila wrote: >> >> On Tue, Dec 6, 2016 at 1:29 PM, Jeff Janes wrote: >> > >> > >> > I just occasionally insert a bunch of equal tuples, which have to be in >> > overflow pages no matter how much splitting h

Re: [HACKERS] Hash Indexes

2016-12-06 Thread Jeff Janes
On Tue, Dec 6, 2016 at 4:00 AM, Amit Kapila wrote: > On Tue, Dec 6, 2016 at 1:29 PM, Jeff Janes wrote: > > > > > > I just occasionally insert a bunch of equal tuples, which have to be in > > overflow pages no matter how much splitting happens. > > > > I am getting vacuum errors against HEAD, aft

Re: [HACKERS] Hash Indexes

2016-12-06 Thread Amit Kapila
On Tue, Dec 6, 2016 at 1:29 PM, Jeff Janes wrote: > > > I just occasionally insert a bunch of equal tuples, which have to be in > overflow pages no matter how much splitting happens. > > I am getting vacuum errors against HEAD, after about 20 minutes or so (8 > cores). > > 49233 XX002 2016-12-05

Re: [HACKERS] Hash Indexes

2016-12-06 Thread Jeff Janes
On Thu, Dec 1, 2016 at 10:54 PM, Amit Kapila wrote: > On Thu, Dec 1, 2016 at 8:56 PM, Robert Haas wrote: > > On Thu, Dec 1, 2016 at 12:43 AM, Amit Kapila > wrote: > >> On Thu, Dec 1, 2016 at 2:18 AM, Robert Haas > wrote: > >>> On Wed, Nov 23, 2016 at 8:50 AM, Amit Kapila > wrote: > [ new

Re: [HACKERS] Hash Indexes

2016-12-05 Thread Robert Haas
On Fri, Dec 2, 2016 at 10:54 PM, Amit Kapila wrote: > On Sat, Dec 3, 2016 at 12:13 AM, Robert Haas wrote: >> On Fri, Dec 2, 2016 at 1:54 AM, Amit Kapila wrote: I want to split when the average bucket contains 10 pages worth of tuples. >>> >>> oh, I think what you mean to say is hack t

Re: [HACKERS] Hash Indexes

2016-12-02 Thread Amit Kapila
On Sat, Dec 3, 2016 at 12:13 AM, Robert Haas wrote: > On Fri, Dec 2, 2016 at 1:54 AM, Amit Kapila wrote: >>> I want to split when the average bucket >>> contains 10 pages worth of tuples. >> >> oh, I think what you mean to say is hack the code to bump fill factor >> and then test it. I was conf

Re: [HACKERS] Hash Indexes

2016-12-02 Thread Robert Haas
On Fri, Dec 2, 2016 at 1:54 AM, Amit Kapila wrote: >> I want to split when the average bucket >> contains 10 pages worth of tuples. > > oh, I think what you mean to say is hack the code to bump fill factor > and then test it. I was confused that how can user can do that from > SQL command. Yes,

Re: [HACKERS] Hash Indexes

2016-12-01 Thread Amit Kapila
On Thu, Dec 1, 2016 at 8:56 PM, Robert Haas wrote: > On Thu, Dec 1, 2016 at 12:43 AM, Amit Kapila wrote: >> On Thu, Dec 1, 2016 at 2:18 AM, Robert Haas wrote: >>> On Wed, Nov 23, 2016 at 8:50 AM, Amit Kapila >>> wrote: [ new patch ] >>> >>> Committed with some further cosmetic changes. >>

Re: [HACKERS] Hash Indexes

2016-12-01 Thread Robert Haas
On Thu, Dec 1, 2016 at 12:43 AM, Amit Kapila wrote: > On Thu, Dec 1, 2016 at 2:18 AM, Robert Haas wrote: >> On Wed, Nov 23, 2016 at 8:50 AM, Amit Kapila wrote: >>> [ new patch ] >> >> Committed with some further cosmetic changes. > > Thank you very much. > >> I think it would be worth testing th

Re: [HACKERS] Hash Indexes

2016-11-30 Thread Amit Kapila
On Thu, Dec 1, 2016 at 2:18 AM, Robert Haas wrote: > On Wed, Nov 23, 2016 at 8:50 AM, Amit Kapila wrote: >> [ new patch ] > > Committed with some further cosmetic changes. > Thank you very much. > I think it would be worth testing this code with very long overflow > chains by hacking the fill f

Re: [HACKERS] Hash Indexes

2016-11-30 Thread Robert Haas
On Wed, Nov 23, 2016 at 8:50 AM, Amit Kapila wrote: > [ new patch ] Committed with some further cosmetic changes. I guess I won't be very surprised if this turns out to have a few bugs yet, but I think it's in fairly good shape at this point. I think it would be worth testing this code with ver

Re: [HACKERS] Hash Indexes

2016-11-18 Thread Amit Kapila
On Fri, Nov 18, 2016 at 12:11 PM, Amit Kapila wrote: > On Thu, Nov 17, 2016 at 10:54 PM, Robert Haas wrote: >> On Thu, Nov 17, 2016 at 12:05 PM, Amit Kapila >> wrote: >> I think this comment is saying that we'll release the pin on the primary bucket page for now, and then reacquire it

Re: [HACKERS] Hash Indexes

2016-11-17 Thread Amit Kapila
On Thu, Nov 17, 2016 at 10:54 PM, Robert Haas wrote: > On Thu, Nov 17, 2016 at 12:05 PM, Amit Kapila wrote: > >>> I think this comment is saying that we'll release the pin on the >>> primary bucket page for now, and then reacquire it later if the user >>> reverses the scan direction. But that do

Re: [HACKERS] Hash Indexes

2016-11-17 Thread Robert Haas
On Thu, Nov 17, 2016 at 12:05 PM, Amit Kapila wrote: > Are you expecting a comment change here? > >> +old_blkno = _hash_get_oldblock_from_newbucket(rel, >> opaque->hasho_bucket); >> >> Couldn't you pass "bucket" here instead of "hasho->opaque_bucket"? I >> feel like I'm repeating this ad

Re: [HACKERS] Hash Indexes

2016-11-17 Thread Amit Kapila
On Thu, Nov 17, 2016 at 3:08 AM, Robert Haas wrote: > On Sat, Nov 12, 2016 at 12:49 AM, Amit Kapila wrote: >> You are right and I have changed the code as per your suggestion. > > So... > > +/* > + * We always maintain the pin on bucket page for whole scan > operation, > +

Re: [HACKERS] Hash Indexes

2016-11-16 Thread Robert Haas
On Sat, Nov 12, 2016 at 12:49 AM, Amit Kapila wrote: > You are right and I have changed the code as per your suggestion. So... +/* + * We always maintain the pin on bucket page for whole scan operation, + * so releasing the additional pin we have acquired here. +

Re: [HACKERS] Hash Indexes

2016-11-09 Thread Amit Kapila
On Thu, Nov 10, 2016 at 2:57 AM, Robert Haas wrote: > > Some more review: > > The API contract of _hash_finish_split seems a bit unfortunate. The > caller is supposed to have obtained a cleanup lock on both the old and > new buffers, but the first thing it does is walk the entire new bucket > ch

Re: [HACKERS] Hash Indexes

2016-11-09 Thread Robert Haas
On Wed, Nov 9, 2016 at 12:11 PM, Robert Haas wrote: > On Wed, Nov 9, 2016 at 11:41 AM, Amit Kapila wrote: >> On Wed, Nov 9, 2016 at 9:10 PM, Robert Haas wrote: >>> On Wed, Nov 9, 2016 at 9:04 AM, Amit Kapila wrote: >>> I think we can give a brief explanation right in the code comment. I >>> re

Re: [HACKERS] Hash Indexes

2016-11-09 Thread Robert Haas
On Wed, Nov 9, 2016 at 11:41 AM, Amit Kapila wrote: > On Wed, Nov 9, 2016 at 9:10 PM, Robert Haas wrote: >> On Wed, Nov 9, 2016 at 9:04 AM, Amit Kapila wrote: >> I think we can give a brief explanation right in the code comment. I >> referred to "decreasing the TIDs"; you are referring to "movi

Re: [HACKERS] Hash Indexes

2016-11-09 Thread Amit Kapila
On Wed, Nov 9, 2016 at 9:10 PM, Robert Haas wrote: > On Wed, Nov 9, 2016 at 9:04 AM, Amit Kapila wrote: > > I think we can give a brief explanation right in the code comment. I > referred to "decreasing the TIDs"; you are referring to "moving tuples > around". But I think that moving the tuples

Re: [HACKERS] Hash Indexes

2016-11-09 Thread Robert Haas
On Wed, Nov 9, 2016 at 9:04 AM, Amit Kapila wrote: > + * This function expects that the caller has acquired a cleanup lock on the > + * primary bucket page, and will with a write lock again held on the primary > + * bucket page. The lock won't necessarily be held continuously, though, > + * becau

Re: [HACKERS] Hash Indexes

2016-11-09 Thread Amit Kapila
On Wed, Nov 9, 2016 at 1:23 AM, Robert Haas wrote: > On Mon, Nov 7, 2016 at 9:51 PM, Amit Kapila wrote: >> [ new patches ] > > Attached is yet another incremental patch with some suggested changes. > > + * This expects that the caller has acquired a cleanup lock on the target > + * bucket (primar

Re: [HACKERS] Hash Indexes

2016-11-08 Thread Robert Haas
On Mon, Nov 7, 2016 at 9:51 PM, Amit Kapila wrote: > [ new patches ] Attached is yet another incremental patch with some suggested changes. + * This expects that the caller has acquired a cleanup lock on the target + * bucket (primary page of a bucket) and it is reponsibility of caller to + * re

Re: [HACKERS] Hash Indexes

2016-11-08 Thread Robert Haas
On Mon, Nov 7, 2016 at 9:51 PM, Amit Kapila wrote: >> Both the places _hash_squeezebucket() and _hash_splitbucket can use >> this optimization irrespective of rest of the patch. I will prepare a >> separate patch for these and send along with main patch after some >> testing. > > Done as a separ

Re: [HACKERS] Hash Indexes

2016-11-04 Thread Amit Kapila
On Fri, Nov 4, 2016 at 9:37 PM, Robert Haas wrote: > On Tue, Nov 1, 2016 at 9:09 PM, Robert Haas wrote: >> On Mon, Oct 24, 2016 at 10:30 AM, Amit Kapila >> wrote: >>> [ new patches ] >> >> I looked over parts of this today, mostly the hashinsert.c changes. > > Some more review... > > @@ -656,6

Re: [HACKERS] Hash Indexes

2016-11-04 Thread Amit Kapila
On Fri, Nov 4, 2016 at 6:37 PM, Robert Haas wrote: > On Thu, Nov 3, 2016 at 6:25 AM, Amit Kapila wrote: >>> +nblkno = _hash_get_newblk(rel, pageopaque); >>> >>> I think this is not a great name for this function. It's not clear >>> what "new blocks" refers to, exactly. I suggest >>> FIN

Re: [HACKERS] Hash Indexes

2016-11-04 Thread Robert Haas
On Tue, Nov 1, 2016 at 9:09 PM, Robert Haas wrote: > On Mon, Oct 24, 2016 at 10:30 AM, Amit Kapila wrote: >> [ new patches ] > > I looked over parts of this today, mostly the hashinsert.c changes. Some more review... @@ -656,6 +678,10 @@ _hash_squeezebucket(Relation rel, IndexTuple

Re: [HACKERS] Hash Indexes

2016-11-04 Thread Robert Haas
On Fri, Oct 28, 2016 at 12:33 AM, Amit Kapila wrote: > On Fri, Oct 28, 2016 at 2:52 AM, Robert Haas wrote: >> On Mon, Oct 24, 2016 at 10:30 AM, Amit Kapila >> wrote: Amit, can you please split the buffer manager changes in this patch into a separate patch? >>> >>> Sure, attached patch

Re: [HACKERS] Hash Indexes

2016-11-04 Thread Robert Haas
On Thu, Nov 3, 2016 at 6:25 AM, Amit Kapila wrote: >> +nblkno = _hash_get_newblk(rel, pageopaque); >> >> I think this is not a great name for this function. It's not clear >> what "new blocks" refers to, exactly. I suggest >> FIND_SPLIT_BUCKET(metap, bucket) or OLD_BUCKET_TO_NEW_BUCKET(m

Re: [HACKERS] Hash Indexes

2016-11-03 Thread Amit Kapila
On Wed, Nov 2, 2016 at 6:39 AM, Robert Haas wrote: > On Mon, Oct 24, 2016 at 10:30 AM, Amit Kapila wrote: >> [ new patches ] > > I looked over parts of this today, mostly the hashinsert.c changes. > > +/* > + * Copy bucket mapping info now; The comment in _hash_expandtable where > +

Re: [HACKERS] Hash Indexes

2016-11-01 Thread Robert Haas
On Mon, Oct 24, 2016 at 10:30 AM, Amit Kapila wrote: > [ new patches ] I looked over parts of this today, mostly the hashinsert.c changes. +/* + * Copy bucket mapping info now; The comment in _hash_expandtable where + * we copy this information and calls _hash_splitbucket explains w

Re: [HACKERS] Hash Indexes

2016-10-27 Thread Amit Kapila
On Fri, Oct 28, 2016 at 2:52 AM, Robert Haas wrote: > On Mon, Oct 24, 2016 at 10:30 AM, Amit Kapila wrote: >>> Amit, can you please split the buffer manager changes in this patch >>> into a separate patch? >> >> Sure, attached patch extend_bufmgr_api_for_hash_index_v1.patch does that. > > The add

Re: [HACKERS] Hash Indexes

2016-10-27 Thread Robert Haas
On Mon, Oct 24, 2016 at 10:30 AM, Amit Kapila wrote: >> Amit, can you please split the buffer manager changes in this patch >> into a separate patch? > > Sure, attached patch extend_bufmgr_api_for_hash_index_v1.patch does that. The additional argument to ConditionalLockBuffer() doesn't seem to be

Re: [HACKERS] Hash Indexes

2016-10-24 Thread Amit Kapila
On Mon, Oct 24, 2016 at 8:00 PM, Amit Kapila wrote: > On Thu, Sep 29, 2016 at 6:04 AM, Robert Haas wrote: >> On Wed, Sep 28, 2016 at 3:04 PM, Robert Haas wrote: > > > Thanks for the valuable feedback. > Forgot to mention that in addition to fixing the review comments, I had made an additional c

Re: [HACKERS] Hash Indexes

2016-10-20 Thread Robert Haas
On Tue, Oct 18, 2016 at 8:27 PM, Amit Kapila wrote: > By this problem, I mean to say deadlocks for suspended scans, that can > happen in btree for non-Mvcc or other type of scans where we don't > release pin during scan. In my mind, we have below options: > > a. problem of deadlocks for suspended

Re: [HACKERS] Hash Indexes

2016-10-18 Thread Amit Kapila
On Wed, Oct 19, 2016 at 5:57 AM, Amit Kapila wrote: > On Tue, Oct 18, 2016 at 10:52 PM, Robert Haas wrote: >> On Tue, Oct 18, 2016 at 5:37 AM, Amit Kapila wrote: >>> On Wed, Oct 5, 2016 at 10:22 PM, Robert Haas wrote: On Tue, Oct 4, 2016 at 12:36 AM, Amit Kapila wrote: > I think

Re: [HACKERS] Hash Indexes

2016-10-18 Thread Amit Kapila
On Tue, Oct 18, 2016 at 10:52 PM, Robert Haas wrote: > On Tue, Oct 18, 2016 at 5:37 AM, Amit Kapila wrote: >> On Wed, Oct 5, 2016 at 10:22 PM, Robert Haas wrote: >>> On Tue, Oct 4, 2016 at 12:36 AM, Amit Kapila >>> wrote: I think one way to avoid the risk of deadlock in above scenario is

Re: [HACKERS] Hash Indexes

2016-10-18 Thread Andres Freund
On 2016-10-18 13:38:14 -0400, Tom Lane wrote: > Robert Haas writes: > > On Tue, Oct 18, 2016 at 5:37 AM, Amit Kapila > > wrote: > >> I have implemented this idea and it works for MVCC scans. However, I > >> think this might not work for non-MVCC scans. Consider a case where > >> in Process-1,

Re: [HACKERS] Hash Indexes

2016-10-18 Thread Tom Lane
Robert Haas writes: > On Tue, Oct 18, 2016 at 5:37 AM, Amit Kapila wrote: >> I have implemented this idea and it works for MVCC scans. However, I >> think this might not work for non-MVCC scans. Consider a case where >> in Process-1, hash scan has returned one row and before it could check >> i

Re: [HACKERS] Hash Indexes

2016-10-18 Thread Robert Haas
On Tue, Oct 18, 2016 at 5:37 AM, Amit Kapila wrote: > On Wed, Oct 5, 2016 at 10:22 PM, Robert Haas wrote: >> On Tue, Oct 4, 2016 at 12:36 AM, Amit Kapila wrote: >>> I think one way to avoid the risk of deadlock in above scenario is to >>> take the cleanup lock conditionally, if we get the cleanu

Re: [HACKERS] Hash Indexes

2016-10-18 Thread Amit Kapila
On Wed, Oct 5, 2016 at 10:22 PM, Robert Haas wrote: > On Tue, Oct 4, 2016 at 12:36 AM, Amit Kapila wrote: >> I think one way to avoid the risk of deadlock in above scenario is to >> take the cleanup lock conditionally, if we get the cleanup lock then >> we will delete the items as we are doing in

Re: [HACKERS] Hash Indexes

2016-10-10 Thread Amit Kapila
On Mon, Oct 10, 2016 at 10:07 PM, Jeff Janes wrote: > On Mon, Oct 10, 2016 at 5:55 AM, Amit Kapila > wrote: >> >> On Thu, Sep 29, 2016 at 8:27 PM, Amit Kapila >> wrote: >> > On Thu, Sep 29, 2016 at 6:04 AM, Robert Haas >> > wrote: >> > >> >> Another thing I don't quite understand about this alg

Re: [HACKERS] Hash Indexes

2016-10-10 Thread Jeff Janes
On Mon, Oct 10, 2016 at 5:55 AM, Amit Kapila wrote: > On Thu, Sep 29, 2016 at 8:27 PM, Amit Kapila > wrote: > > On Thu, Sep 29, 2016 at 6:04 AM, Robert Haas > wrote: > > > >> Another thing I don't quite understand about this algorithm is that in > >> order to conditionally lock the target prima

Re: [HACKERS] Hash Indexes

2016-10-10 Thread Amit Kapila
On Thu, Sep 29, 2016 at 8:27 PM, Amit Kapila wrote: > On Thu, Sep 29, 2016 at 6:04 AM, Robert Haas wrote: > >> Another thing I don't quite understand about this algorithm is that in >> order to conditionally lock the target primary bucket page, we'd first >> need to read and pin it. And that doe

Re: [HACKERS] Hash Indexes

2016-10-06 Thread Amit Kapila
On Wed, Oct 5, 2016 at 10:22 PM, Robert Haas wrote: > On Tue, Oct 4, 2016 at 12:36 AM, Amit Kapila wrote: >> I think one way to avoid the risk of deadlock in above scenario is to >> take the cleanup lock conditionally, if we get the cleanup lock then >> we will delete the items as we are doing in

Re: [HACKERS] Hash Indexes

2016-10-05 Thread Robert Haas
On Tue, Oct 4, 2016 at 12:36 AM, Amit Kapila wrote: > I think one way to avoid the risk of deadlock in above scenario is to > take the cleanup lock conditionally, if we get the cleanup lock then > we will delete the items as we are doing in patch now, else it will > just mark the tuples as dead an

Re: [HACKERS] Hash Indexes

2016-10-03 Thread Amit Kapila
On Tue, Oct 4, 2016 at 10:06 AM, Amit Kapila wrote: > On Thu, Sep 29, 2016 at 8:27 PM, Amit Kapila wrote: >> On Thu, Sep 29, 2016 at 6:04 AM, Robert Haas wrote: >>> On Wed, Sep 28, 2016 at 3:04 PM, Robert Haas wrote: >>> >>> As I was looking at the old text regarding deadlock risk, I realized >

Re: [HACKERS] Hash Indexes

2016-10-03 Thread Amit Kapila
On Thu, Sep 29, 2016 at 8:27 PM, Amit Kapila wrote: > On Thu, Sep 29, 2016 at 6:04 AM, Robert Haas wrote: >> On Wed, Sep 28, 2016 at 3:04 PM, Robert Haas wrote: >> >> As I was looking at the old text regarding deadlock risk, I realized >> what may be a serious problem. Suppose process A is perf

Re: [HACKERS] Hash Indexes

2016-10-03 Thread Tom Lane
Jeff Janes writes: > I've done a simple comparison using pgbench's default transaction, in which > all the primary keys have been dropped and replaced with indexes of either > hash or btree type, alternating over many rounds. > I run 'pgbench -c16 -j16 -T 900 -M prepared' on an 8 core machine wit

Re: [HACKERS] Hash Indexes

2016-10-03 Thread Jeff Janes
On Thu, Sep 29, 2016 at 5:14 PM, Robert Haas wrote: > On Thu, Sep 29, 2016 at 8:07 PM, Peter Geoghegan wrote: > > On Wed, Sep 28, 2016 at 8:06 PM, Andres Freund > wrote: > >> On 2016-09-28 15:04:30 -0400, Robert Haas wrote: > >>> Andres already > >>> stated that he things working on btree-over-

Re: [HACKERS] Hash Indexes

2016-10-02 Thread Michael Paquier
On Mon, Oct 3, 2016 at 12:42 AM, Pavel Stehule wrote: > > > 2016-10-02 12:40 GMT+02:00 Michael Paquier : >> >> On Sun, Oct 2, 2016 at 3:31 AM, Greg Stark wrote: >> > On Fri, Sep 30, 2016 at 2:11 AM, Robert Haas >> > wrote: >> >> For one thing, we can stop shipping a totally broken feature in rel

Re: [HACKERS] Hash Indexes

2016-10-02 Thread Pavel Stehule
2016-10-02 12:40 GMT+02:00 Michael Paquier : > On Sun, Oct 2, 2016 at 3:31 AM, Greg Stark wrote: > > On Fri, Sep 30, 2016 at 2:11 AM, Robert Haas > wrote: > >> For one thing, we can stop shipping a totally broken feature in release > after release > > > > For what it's worth I'm for any patch th

Re: [HACKERS] Hash Indexes

2016-10-02 Thread Michael Paquier
On Sun, Oct 2, 2016 at 3:31 AM, Greg Stark wrote: > On Fri, Sep 30, 2016 at 2:11 AM, Robert Haas wrote: >> For one thing, we can stop shipping a totally broken feature in release >> after release > > For what it's worth I'm for any patch that can accomplish that. > > In retrospect I think we sho

Re: [HACKERS] Hash Indexes

2016-10-01 Thread Greg Stark
On Fri, Sep 30, 2016 at 2:11 AM, Robert Haas wrote: > For one thing, we can stop shipping a totally broken feature in release after > release For what it's worth I'm for any patch that can accomplish that. In retrospect I think we should have done the hash-over-btree thing ten years ago but we

Re: [HACKERS] Hash Indexes

2016-10-01 Thread ktm
Andres Freund : On 2016-09-30 17:39:04 +0100, Peter Geoghegan wrote: On Fri, Sep 30, 2016 at 4:46 PM, Robert Haas wrote: > I would just be very disappointed if, after the amount of work that > Amit and others have put into this project, the code gets rejected > because somebody thinks a differ

Re: [HACKERS] Hash Indexes

2016-09-30 Thread Amit Kapila
On 30-Sep-2016 10:26 PM, "Peter Geoghegan" wrote: > > On Fri, Sep 30, 2016 at 4:46 PM, Robert Haas wrote: > > I would just be very disappointed if, after the amount of work that > > Amit and others have put into this project, the code gets rejected > > because somebody thinks a different project

Re: [HACKERS] Hash Indexes

2016-09-30 Thread Andres Freund
On 2016-09-30 17:39:04 +0100, Peter Geoghegan wrote: > On Fri, Sep 30, 2016 at 4:46 PM, Robert Haas wrote: > > I would just be very disappointed if, after the amount of work that > > Amit and others have put into this project, the code gets rejected > > because somebody thinks a different project

Re: [HACKERS] Hash Indexes

2016-09-30 Thread Tom Lane
Peter Geoghegan writes: > On Fri, Sep 30, 2016 at 4:46 PM, Robert Haas wrote: >> The fact that we have hash indexes already and cannot remove them >> because too much other code depends on hash opclasses is also, in my >> opinion, a sufficiently good reason to pursue improving them. > I think th

Re: [HACKERS] Hash Indexes

2016-09-30 Thread Peter Geoghegan
On Fri, Sep 30, 2016 at 4:46 PM, Robert Haas wrote: > I would just be very disappointed if, after the amount of work that > Amit and others have put into this project, the code gets rejected > because somebody thinks a different project would have been more worth > doing. I wouldn't presume to te

Re: [HACKERS] Hash Indexes

2016-09-30 Thread Peter Geoghegan
On Fri, Sep 30, 2016 at 4:46 PM, Robert Haas wrote: > I would just be very disappointed if, after the amount of work that > Amit and others have put into this project, the code gets rejected > because somebody thinks a different project would have been more worth > doing. I wouldn't presume to te

Re: [HACKERS] Hash Indexes

2016-09-30 Thread Robert Haas
On Fri, Sep 30, 2016 at 7:47 AM, Peter Geoghegan wrote: > My only firm position is that it wouldn't be very hard to investigate > hash-over-btree to Andres' satisfaction, say, so, why not? I'm > surprised that this has caused consternation -- ISTM that Andres' > suggestion is *perfectly* reasonabl

Re: [HACKERS] Hash Indexes

2016-09-30 Thread Peter Geoghegan
On Fri, Sep 30, 2016 at 9:07 AM, Amit Kapila wrote: > Considering, I have missed the real intention of their suggestions, I think > such a serious objection on any work should be discussed on list. To answer > the actual objection, I have already mentioned upthread that we can deduce > from the

Re: [HACKERS] Hash Indexes

2016-09-30 Thread Amit Kapila
On 30-Sep-2016 6:24 AM, "Robert Haas" wrote: > > On Thu, Sep 29, 2016 at 8:29 PM, Andres Freund wrote: > > On September 29, 2016 5:28:00 PM PDT, Robert Haas wrote: > >>On Thu, Sep 29, 2016 at 8:16 PM, Andres Freund > >>wrote: > Well, I, for one, find it frustrating. It seems pretty unhelp

Re: [HACKERS] Hash Indexes

2016-09-29 Thread Robert Haas
On Thu, Sep 29, 2016 at 8:53 PM, Peter Geoghegan wrote: > On Fri, Sep 30, 2016 at 1:14 AM, Robert Haas wrote: >>> I, for one, agree with this position. >> >> Well, I, for one, find it frustrating. It seems pretty unhelpful to >> bring this up only after the code has already been written. The fi

Re: [HACKERS] Hash Indexes

2016-09-29 Thread Robert Haas
On Thu, Sep 29, 2016 at 8:29 PM, Andres Freund wrote: > On September 29, 2016 5:28:00 PM PDT, Robert Haas > wrote: >>On Thu, Sep 29, 2016 at 8:16 PM, Andres Freund >>wrote: Well, I, for one, find it frustrating. It seems pretty unhelpful to bring this up only after the code has alrea

Re: [HACKERS] Hash Indexes

2016-09-29 Thread Peter Geoghegan
On Fri, Sep 30, 2016 at 1:14 AM, Robert Haas wrote: >> I, for one, agree with this position. > > Well, I, for one, find it frustrating. It seems pretty unhelpful to > bring this up only after the code has already been written. The first > post on this thread was on May 10th. The first version o

Re: [HACKERS] Hash Indexes

2016-09-29 Thread Peter Geoghegan
On Fri, Sep 30, 2016 at 1:29 AM, Andres Freund wrote: >>To whom? In what context? > > Amit, over dinner. In case it matters, I also talked to Amit about this privately. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscrip

Re: [HACKERS] Hash Indexes

2016-09-29 Thread Andres Freund
On September 29, 2016 5:28:00 PM PDT, Robert Haas wrote: >On Thu, Sep 29, 2016 at 8:16 PM, Andres Freund >wrote: >>> Well, I, for one, find it frustrating. It seems pretty unhelpful to >>> bring this up only after the code has already been written. >> >> I brought this up in person at pgcon to

Re: [HACKERS] Hash Indexes

2016-09-29 Thread Robert Haas
On Thu, Sep 29, 2016 at 8:16 PM, Andres Freund wrote: >> Well, I, for one, find it frustrating. It seems pretty unhelpful to >> bring this up only after the code has already been written. > > I brought this up in person at pgcon too. To whom? In what context? -- Robert Haas EnterpriseDB: http

Re: [HACKERS] Hash Indexes

2016-09-29 Thread Andres Freund
On 2016-09-29 20:14:40 -0400, Robert Haas wrote: > On Thu, Sep 29, 2016 at 8:07 PM, Peter Geoghegan wrote: > > On Wed, Sep 28, 2016 at 8:06 PM, Andres Freund wrote: > >> On 2016-09-28 15:04:30 -0400, Robert Haas wrote: > >>> Andres already > >>> stated that he things working on btree-over-hash wo

Re: [HACKERS] Hash Indexes

2016-09-29 Thread Robert Haas
On Thu, Sep 29, 2016 at 8:07 PM, Peter Geoghegan wrote: > On Wed, Sep 28, 2016 at 8:06 PM, Andres Freund wrote: >> On 2016-09-28 15:04:30 -0400, Robert Haas wrote: >>> Andres already >>> stated that he things working on btree-over-hash would be more >>> beneficial than fixing hash, but at this po

Re: [HACKERS] Hash Indexes

2016-09-29 Thread Peter Geoghegan
On Wed, Sep 28, 2016 at 8:06 PM, Andres Freund wrote: > On 2016-09-28 15:04:30 -0400, Robert Haas wrote: >> Andres already >> stated that he things working on btree-over-hash would be more >> beneficial than fixing hash, but at this point it seems like he's the >> only one who takes that position.

Re: [HACKERS] Hash Indexes

2016-09-29 Thread Amit Kapila
On Thu, Sep 29, 2016 at 6:04 AM, Robert Haas wrote: > On Wed, Sep 28, 2016 at 3:04 PM, Robert Haas wrote: >> I'll write another email with my thoughts about the rest of the patch. > > I think that the README changes for this patch need a fairly large > amount of additional work. Here are a few t

Re: [HACKERS] Hash Indexes

2016-09-28 Thread Robert Haas
On Wed, Sep 28, 2016 at 3:04 PM, Robert Haas wrote: > I'll write another email with my thoughts about the rest of the patch. I think that the README changes for this patch need a fairly large amount of additional work. Here are a few things I notice: - The confusion between buckets and pages ha

Re: [HACKERS] Hash Indexes

2016-09-28 Thread Robert Haas
On Wed, Sep 28, 2016 at 3:06 PM, Andres Freund wrote: > On 2016-09-28 15:04:30 -0400, Robert Haas wrote: >> Andres already >> stated that he things working on btree-over-hash would be more >> beneficial than fixing hash, but at this point it seems like he's the >> only one who takes that position.

Re: [HACKERS] Hash Indexes

2016-09-28 Thread Andres Freund
On 2016-09-28 15:04:30 -0400, Robert Haas wrote: > Andres already > stated that he things working on btree-over-hash would be more > beneficial than fixing hash, but at this point it seems like he's the > only one who takes that position. Note that I did *NOT* take that position. I was saying that

Re: [HACKERS] Hash Indexes

2016-09-28 Thread Robert Haas
On Tue, Sep 27, 2016 at 3:06 PM, Jesper Pedersen wrote: > I have been running various tests, and applications with this patch together > with the WAL v5 patch [1]. > > As I havn't seen any failures and doesn't currently have additional feedback > I'm moving this patch to "Ready for Committer" for

Re: [HACKERS] Hash Indexes

2016-09-27 Thread Jesper Pedersen
On 09/20/2016 09:02 AM, Amit Kapila wrote: On Fri, Sep 16, 2016 at 11:22 AM, Amit Kapila wrote: I do want to work on it, but it is always possible that due to some other work this might get delayed. Also, I think there is always a chance that while doing that work, we face some problem due to

Re: [HACKERS] Hash Indexes

2016-09-24 Thread Mark Kirkwood
On 25/09/16 18:18, Amit Kapila wrote: On Sat, Sep 24, 2016 at 10:49 PM, Greg Stark wrote: On Thu, Sep 22, 2016 at 3:23 AM, Tom Lane wrote: But to kick the hash AM as such to the curb is to say "sorry, there will never be O(1) index lookups in Postgres". Well there's plenty of halfway solut

Re: [HACKERS] Hash Indexes

2016-09-24 Thread Amit Kapila
On Sat, Sep 24, 2016 at 10:49 PM, Greg Stark wrote: > On Thu, Sep 22, 2016 at 3:23 AM, Tom Lane wrote: >> But to kick the hash AM as such to the curb is to say >> "sorry, there will never be O(1) index lookups in Postgres". > > Well there's plenty of halfway solutions for that. We could move hash

Re: [HACKERS] Hash Indexes

2016-09-24 Thread Tom Lane
Greg Stark writes: > On Thu, Sep 22, 2016 at 3:23 AM, Tom Lane wrote: >> But to kick the hash AM as such to the curb is to say >> "sorry, there will never be O(1) index lookups in Postgres". > Well there's plenty of halfway solutions for that. We could move hash > indexes to contrib or even have

  1   2   >