Re: [HACKERS] GSoC on WAL-logging hash indexes

On 08/20/2014 02:43 AM, Michael Paquier wrote: > > > > On Thu, Jun 19, 2014 at 6:40 PM, Vik Fearing > wrote: > > On 04/30/2014 11:41 PM, Tom Lane wrote: > > We really oughta fix the WAL situation, not just band-aid around it. > > After re-reading thi

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Thu, Jun 19, 2014 at 6:40 PM, Vik Fearing wrote: > On 04/30/2014 11:41 PM, Tom Lane wrote: > > We really oughta fix the WAL situation, not just band-aid around it. > > After re-reading this thread, it is not clear that anyone is going to > work on it so I'll just ask: > > Is anyone working on

Re: [HACKERS] GSoC on WAL-logging hash indexes

On 04/30/2014 11:41 PM, Tom Lane wrote: > We really oughta fix the WAL situation, not just band-aid around it. After re-reading this thread, it is not clear that anyone is going to work on it so I'll just ask: Is anyone working on this? If not, I'd like to put it on my plate. -- Vik -- Sen

Re: [HACKERS] GSoC on WAL-logging hash indexes

Greg Stark writes: > But imnsho doing nothing is a bad idea. We should have long ago either > added WAL logging or removed the index type. We shouldn't have left a > foot-gun that large lying around for so long. We can't remove the hash index type, nor move it to an extension, because it is the o

Re: [HACKERS] GSoC on WAL-logging hash indexes

Stephen Frost wrote: > * Greg Stark (st...@mit.edu) wrote: > > I could do the legwork on either the GUC or moving hash indexes to an > > extension if there's a consensus. But I suspect either will be quite > > controversial... > If you'd like to build an extension which provides hash indexes- I s

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Wed, Apr 30, 2014 at 12:16 PM, Greg Stark wrote: > I think the key question was if someone wanted to develop a good hash > index would they start from our current hash index or would they be > better off starting from a fresh codebase? If it were me I'd start with the current code. It would

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Wed, Apr 30, 2014 at 11:19 AM, Peter Geoghegan wrote: > On Wed, Apr 30, 2014 at 11:03 AM, Jeff Janes wrote: > > If we don't put in the work to make them useful, then they won't ever > become > > useful. > > > > If we do put in the effort (and it would be considerable) then I think > they > >

Re: [HACKERS] GSoC on WAL-logging hash indexes

* Greg Stark (st...@mit.edu) wrote: > But imnsho doing nothing is a bad idea. We should have long ago either > added WAL logging or removed the index type. We shouldn't have left a > foot-gun that large lying around for so long. I certainly encourage you to work on it. :) > Incidentally something

Re: [HACKERS] GSoC on WAL-logging hash indexes

I think the key question was if someone wanted to develop a good hash index would they start from our current hash index or would they be better off starting from a fresh codebase? If the former then we should add WAL logging and throw GSOC and other people who ask for projects at it. If the latter

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Wed, Apr 30, 2014 at 11:03 AM, Jeff Janes wrote: > If we don't put in the work to make them useful, then they won't ever become > useful. > > If we do put in the effort (and it would be considerable) then I think they > will be. But you may be correct that the effort required would perhaps be

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Wed, Apr 30, 2014 at 2:02 PM, Peter Geoghegan wrote: >> Of course our current hash indexes have *more* not less contention >> than btree but I'm pretty comfortable chalking that up to quality of >> implementation rather than anything intrinsic. > > I am not convinced of that. In commit 76837c1

Re: [HACKERS] GSoC on WAL-logging hash indexes

On 2014-04-30 11:10:22 -0700, Jeff Janes wrote: > I've seen the simple pinning and unpinning of the root page (or the fast root, > whatever the first page we bother to pin on a regular basis is called) be a > point of contention.  When one index dominates the entire system workload, > that > one p

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Wed, Apr 30, 2014 at 11:02 AM, Peter Geoghegan wrote: > On Wed, Apr 30, 2014 at 10:11 AM, Robert Haas > wrote: > > I thought the theoretical advantage of hash indexes wasn't that they > > were smaller but that you avoided a central contention point (the > > btree root). > > The B-Tree root is

Re: [HACKERS] GSoC on WAL-logging hash indexes

All, (dropping pgsql-advocacy off the cc list) On 04/30/2014 10:11 AM, Robert Haas wrote: > On Wed, Apr 30, 2014 at 12:54 PM, Peter Geoghegan wrote: >> On Wed, Apr 30, 2014 at 5:55 AM, k...@rice.edu wrote: >>> I do not think that CPU costs matter as much as the O(1) probe to >>> get a result va

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Wed, Apr 30, 2014 at 12:26 AM, Peter Geoghegan wrote: > On Mon, Mar 3, 2014 at 8:12 AM, Robert Haas wrote: > >> As a GSoC student, I will implement WAL recovery of hash indexes using > the > >> other index types' WAL code as a guide. > > Frankly, I'm skeptical of the idea that hash indexes wi

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Wed, Apr 30, 2014 at 11:01 AM, Tom Lane wrote: > It might lead to good things later; and even if > it doesn't, the lack of WAL support is an embarrassment. I don't think it will, but I do agree that the current state of affairs is an embarrassment. -- Peter Geoghegan -- Sent via pgsql-ha

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Wed, Apr 30, 2014 at 10:11 AM, Robert Haas wrote: > I thought the theoretical advantage of hash indexes wasn't that they > were smaller but that you avoided a central contention point (the > btree root). The B-Tree root isn't really a central contention point at all. The locking/latching proto

Re: [HACKERS] GSoC on WAL-logging hash indexes

Robert Haas writes: > I thought the theoretical advantage of hash indexes wasn't that they > were smaller but that you avoided a central contention point (the > btree root). > Of course our current hash indexes have *more* not less contention > than btree but I'm pretty comfortable chalking that

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Wed, Apr 30, 2014 at 12:54 PM, Peter Geoghegan wrote: > On Wed, Apr 30, 2014 at 5:55 AM, k...@rice.edu wrote: >> I do not think that CPU costs matter as much as the O(1) probe to >> get a result value specifically for very large indexes/tables where >> even caching the upper levels of a B-tree

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Wed, Apr 30, 2014 at 5:55 AM, k...@rice.edu wrote: > I do not think that CPU costs matter as much as the O(1) probe to > get a result value specifically for very large indexes/tables where > even caching the upper levels of a B-tree index would kill your > working set in memory. I know, I know,

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Wed, Apr 30, 2014 at 12:26:20AM -0700, Peter Geoghegan wrote: > On Mon, Mar 3, 2014 at 8:12 AM, Robert Haas wrote: > >> As a GSoC student, I will implement WAL recovery of hash indexes using the > >> other index types' WAL code as a guide. > > Frankly, I'm skeptical of the idea that hash index

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Mon, Mar 3, 2014 at 8:12 AM, Robert Haas wrote: >> As a GSoC student, I will implement WAL recovery of hash indexes using the >> other index types' WAL code as a guide. Frankly, I'm skeptical of the idea that hash indexes will ever really be useful. I realize that that's a counter-intuitive co

Re: [HACKERS] GSoC on WAL-logging hash indexes

On 03/07/2014 03:48 PM, Robert Haas wrote: On Fri, Mar 7, 2014 at 4:34 AM, Heikki Linnakangas wrote: >Hmm. You suggested ensuring that a scan always has at least a pin, and split >takes a vacuum-lock. That ought to work. There's no need for the more >complicated maneuvers you described, ISTM t

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Thu, Mar 06, 2014 at 06:14:21PM -0500, Robert Haas wrote: > On Thu, Mar 6, 2014 at 3:44 PM, Jeff Janes wrote: > > > > I've been tempted to implement a new type of hash index that allows both WAL > > and high concurrency, simply by disallowing bucket splits. At the index > > creation time you u

Re: [HACKERS] GSoC on WAL-logging hash indexes

On 03/07/2014 03:48 PM, Robert Haas wrote: >So, there seems to be a few fairly simple and independent improvements to be >made: > >1. Replace the heavy-weight lock with pin & vacuum-lock. >2. Make it crash-safe, by adding WAL-logging >3. Only acquire the exclusive-lock (vacuum-lock after step 1)

Re: [HACKERS] GSoC on WAL-logging hash indexes

[ I just sent a reply to Greg Stark's email which touches on some of the same points you mentioned here; that was mostly written last night, and finished this morning before seeing this. ] On Fri, Mar 7, 2014 at 4:34 AM, Heikki Linnakangas wrote: > Hmm. You suggested ensuring that a scan always

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Thu, Mar 6, 2014 at 7:07 PM, Greg Stark wrote: > On Thu, Mar 6, 2014 at 11:14 PM, Robert Haas wrote: >>> I've been tempted to implement a new type of hash index that allows both WAL >>> and high concurrency, simply by disallowing bucket splits. At the index >>> creation time you use a storage

Re: [HACKERS] GSoC on WAL-logging hash indexes

On 03/06/2014 09:34 PM, Robert Haas wrote: On Thu, Mar 6, 2014 at 8:11 AM, Heikki Linnakangas wrote: I don't think it's necessary to improve concurrency just to get WAL-logging. Better concurrency is a worthy goal of its own, of course, but it's a separate concern. To some extent, I agree, bu

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Thu, Mar 6, 2014 at 11:14 PM, Robert Haas wrote: >> I've been tempted to implement a new type of hash index that allows both WAL >> and high concurrency, simply by disallowing bucket splits. At the index >> creation time you use a storage parameter to specify the number of buckets, >> and that

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Thu, Mar 6, 2014 at 3:44 PM, Jeff Janes wrote: > On Thu, Mar 6, 2014 at 11:34 AM, Robert Haas wrote: >> Putting the split-in-progress flag in the new bucket's primary page >> makes a lot of sense. I don't have any problem with dumping the rest >> of it for a first cut if we have a different l

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Thu, Mar 6, 2014 at 11:34 AM, Robert Haas wrote: > > Putting the split-in-progress flag in the new bucket's primary page > makes a lot of sense. I don't have any problem with dumping the rest > of it for a first cut if we have a different long-term plan for how to > improve concurrency, but I

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Thu, Mar 6, 2014 at 8:11 AM, Heikki Linnakangas wrote: > I don't think it's necessary to improve concurrency just to get WAL-logging. > Better concurrency is a worthy goal of its own, of course, but it's a > separate concern. To some extent, I agree, but only to some extent. To make hash inde

Re: [HACKERS] GSoC on WAL-logging hash indexes

(de-CC'ing pgsql-advocacy) On 03/06/2014 04:03 AM, Greg Stark wrote: On Mon, Mar 3, 2014 at 4:12 PM, Robert Haas wrote: Unfortunately, I don't believe that it's possible to do this easily today because of the way bucket splits are handled. I wrote about this previously here, with an idea for

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Mon, Mar 3, 2014 at 4:12 PM, Robert Haas wrote: > Unfortunately, I don't believe that it's possible to do this easily > today because of the way bucket splits are handled. I wrote about > this previously here, with an idea for solving the problem: We could just tackle this in the same incompl

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Mon, Mar 3, 2014 at 8:12 AM, Robert Haas wrote: > > Unfortunately, I don't believe that it's possible to do this easily > today because of the way bucket splits are handled. I wrote about > this previously here, with an idea for solving the problem: > > > http://www.postgresql.org/message-id/

Re: [HACKERS] GSoC on WAL-logging hash indexes

Thanks for alerting me to your previous idea. While I don't know enough about Postgresql internals to judge its merits yet, I'll write some pseudocode based on it in my proposal; and I'll relegate it to a "reach" proposal alongside a more straightforward one. Tan On Mon, Mar 3, 2014 at 8:12 AM, R

Re: [HACKERS] GSoC on WAL-logging hash indexes

On Sun, Mar 2, 2014 at 2:38 PM, Tan Tran wrote: > 2. Proposal > As a GSoC student, I will implement WAL recovery of hash indexes using the > other index types' WAL code as a guide. Roughly, I will: > - Devise a way to store and retrieve hashing data within the XLog data > structures. > - In the ex