Re: [HACKERS] foreign key locks, 2nd attempt

2012-04-05 Thread Peter Geoghegan
On 25 March 2012 09:17, Simon Riggs si...@2ndquadrant.com wrote: The main thing we're waiting on are the performance tests to confirm the lack of regression. I have extensively benchmarked the latest revision of the patch (tpc-b.sql), which I pulled from Alvaro's github account. The benchmark

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-25 Thread Simon Riggs
On Sat, Mar 17, 2012 at 10:45 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Here is v11.  This version is mainly updated to add pg_upgrade support, as discussed.  It also contains the README file that was posted earlier (plus wording fixes per Bruce), a couple of bug fixes, and some

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-17 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of lun mar 05 15:28:59 -0300 2012: On Mon, Feb 27, 2012 at 2:47 AM, Robert Haas robertmh...@gmail.com wrote: Regarding performance, the good thing about this patch is that if you have an operation that used to block, it might now not block.  So maybe

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-17 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of mar mar 06 18:33:13 -0300 2012: The lock modes are correct, appropriate and IMHO have meaningful names. No redesign required here. Not sure about the naming of some of the flag bits however. Feel free to suggest improvements ... I've probably seen

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-17 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of mar mar 06 17:28:12 -0300 2012: On Tue, Mar 6, 2012 at 7:39 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: We provide four levels of tuple locking strength: SELECT FOR KEY UPDATE is super-exclusive locking (used to delete tuples and more

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Alvaro Herrera
Excerpts from Bruce Momjian's message of vie mar 16 00:04:06 -0300 2012: On Tue, Mar 13, 2012 at 02:35:02PM -0300, Alvaro Herrera wrote: Excerpts from Bruce Momjian's message of mar mar 13 14:00:52 -0300 2012: On Tue, Mar 06, 2012 at 04:39:32PM -0300, Alvaro Herrera wrote:

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Alvaro Herrera
Excerpts from Alvaro Herrera's message of vie mar 16 10:36:11 -0300 2012: Now I am confused. Where do you see the word hint used by HEAP_XMAX_EXCL_LOCK and HEAP_XMAX_SHARED_LOCK. These are tuple infomask bits, not hints, meaning they are not optional or there just for performance.

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Robert Haas
On Thu, Mar 15, 2012 at 11:09 PM, Bruce Momjian br...@momjian.us wrote: On Tue, Mar 13, 2012 at 01:46:24PM -0400, Robert Haas wrote: On Mon, Mar 12, 2012 at 3:28 PM, Simon Riggs si...@2ndquadrant.com wrote: I agree with you that some worst case performance tests should be done. Could you

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Bruce Momjian
On Fri, Mar 16, 2012 at 10:40:01AM -0300, Alvaro Herrera wrote: Excerpts from Alvaro Herrera's message of vie mar 16 10:36:11 -0300 2012: Now I am confused. Where do you see the word hint used by HEAP_XMAX_EXCL_LOCK and HEAP_XMAX_SHARED_LOCK. These are tuple infomask bits, not

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Bruce Momjian
On Fri, Mar 16, 2012 at 10:08:07AM -0400, Robert Haas wrote: On Thu, Mar 15, 2012 at 11:09 PM, Bruce Momjian br...@momjian.us wrote: On Tue, Mar 13, 2012 at 01:46:24PM -0400, Robert Haas wrote: On Mon, Mar 12, 2012 at 3:28 PM, Simon Riggs si...@2ndquadrant.com wrote: I agree with you that

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-16 Thread Alvaro Herrera
Excerpts from Bruce Momjian's message of vie mar 16 15:22:05 -0300 2012: On Fri, Mar 16, 2012 at 10:08:07AM -0400, Robert Haas wrote: On Thu, Mar 15, 2012 at 11:09 PM, Bruce Momjian br...@momjian.us wrote: On Tue, Mar 13, 2012 at 01:46:24PM -0400, Robert Haas wrote: On Mon, Mar 12,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Wed, Mar 14, 2012 at 5:23 PM, Robert Haas robertmh...@gmail.com wrote: You still have HEAP_XMAX_{INVALID,COMMITTED} to reduce the pressure on mxid lookups, so I think something more sophisticated is needed to exercise that cost.  Not sure what. I don't think HEAP_XMAX_COMMITTED is much

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Thu, Mar 15, 2012 at 2:15 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Mar 14, 2012 at 6:10 PM, Noah Misch n...@leadboat.com wrote: Well, post-release, the cat is out of the bag: we'll be stuck with this whether the performance characteristics are acceptable or not. That's why we'd

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Thu, Mar 15, 2012 at 2:26 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Mar 14, 2012 at 9:17 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Agreed.  But speaking of that, why exactly do we fsync the multixact SLRU today? Good question.  So far, I can't think of a reason.  

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Thu, Mar 15, 2012 at 1:17 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: As things stand today Can I confirm where we are now? Is there another version of the patch coming out soon? --  Simon Riggs   http://www.2ndQuadrant.com/  PostgreSQL Development, 24x7 Support,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue mar 15 18:38:53 -0300 2012: On Thu, Mar 15, 2012 at 2:26 AM, Robert Haas robertmh...@gmail.com wrote: But that would only make sense if we thought that getting rid of the fsyncs would be more valuable than avoiding the blocking here, and I don't.

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue mar 15 18:46:44 -0300 2012: On Thu, Mar 15, 2012 at 1:17 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: As things stand today Can I confirm where we are now? Is there another version of the patch coming out soon? Yes, another version

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Thu, Mar 15, 2012 at 9:54 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Simon Riggs's message of jue mar 15 18:38:53 -0300 2012: On Thu, Mar 15, 2012 at 2:26 AM, Robert Haas robertmh...@gmail.com wrote: But that would only make sense if we thought that getting rid

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue mar 15 19:04:41 -0300 2012: On Thu, Mar 15, 2012 at 9:54 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Simon Riggs's message of jue mar 15 18:38:53 -0300 2012: On Thu, Mar 15, 2012 at 2:26 AM, Robert Haas

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Simon Riggs
On Thu, Mar 15, 2012 at 10:13 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Simon Riggs's message of jue mar 15 19:04:41 -0300 2012: On Thu, Mar 15, 2012 at 9:54 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Simon Riggs's message of jue mar 15

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Robert Haas
On Thu, Mar 15, 2012 at 5:07 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Mar 14, 2012 at 5:23 PM, Robert Haas robertmh...@gmail.com wrote: You still have HEAP_XMAX_{INVALID,COMMITTED} to reduce the pressure on mxid lookups, so I think something more sophisticated is needed to exercise

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Alvaro Herrera
Excerpts from Robert Haas's message of jue mar 15 21:37:36 -0300 2012: On Thu, Mar 15, 2012 at 5:07 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Mar 14, 2012 at 5:23 PM, Robert Haas robertmh...@gmail.com wrote: You still have HEAP_XMAX_{INVALID,COMMITTED} to reduce the pressure on

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Noah Misch
On Thu, Mar 15, 2012 at 08:37:36PM -0400, Robert Haas wrote: On Thu, Mar 15, 2012 at 5:07 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Mar 14, 2012 at 5:23 PM, Robert Haas robertmh...@gmail.com wrote: You still have HEAP_XMAX_{INVALID,COMMITTED} to reduce the pressure on mxid

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Bruce Momjian
On Tue, Mar 13, 2012 at 02:35:02PM -0300, Alvaro Herrera wrote: Excerpts from Bruce Momjian's message of mar mar 13 14:00:52 -0300 2012: On Tue, Mar 06, 2012 at 04:39:32PM -0300, Alvaro Herrera wrote: When there is a single locker in a tuple, we can just store the locking info

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Bruce Momjian
On Thu, Mar 15, 2012 at 11:04:06PM -0400, Bruce Momjian wrote: On Tue, Mar 13, 2012 at 02:35:02PM -0300, Alvaro Herrera wrote: Excerpts from Bruce Momjian's message of mar mar 13 14:00:52 -0300 2012: On Tue, Mar 06, 2012 at 04:39:32PM -0300, Alvaro Herrera wrote: When there is

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-15 Thread Bruce Momjian
On Tue, Mar 13, 2012 at 01:46:24PM -0400, Robert Haas wrote: On Mon, Mar 12, 2012 at 3:28 PM, Simon Riggs si...@2ndquadrant.com wrote: I agree with you that some worst case performance tests should be done. Could you please say what you think the worst cases would be, so those can be

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-14 Thread Robert Haas
On Tue, Mar 13, 2012 at 11:42 PM, Noah Misch n...@leadboat.com wrote: More often than that; each 2-member mxid takes 4 bytes in an offsets file and 10 bytes in a members file.  So, more like one fsync per ~580 mxids.  Note that we already fsync the multixact SLRUs today, so any increase will

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-14 Thread Noah Misch
On Wed, Mar 14, 2012 at 01:23:14PM -0400, Robert Haas wrote: On Tue, Mar 13, 2012 at 11:42 PM, Noah Misch n...@leadboat.com wrote: More often than that; each 2-member mxid takes 4 bytes in an offsets file and 10 bytes in a members file. ?So, more like one fsync per ~580 mxids. ?Note that

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-14 Thread Alvaro Herrera
Excerpts from Noah Misch's message of mié mar 14 19:10:00 -0300 2012: On Wed, Mar 14, 2012 at 01:23:14PM -0400, Robert Haas wrote: On Tue, Mar 13, 2012 at 11:42 PM, Noah Misch n...@leadboat.com wrote: More often than that; each 2-member mxid takes 4 bytes in an offsets file and 10

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-14 Thread Robert Haas
On Wed, Mar 14, 2012 at 6:10 PM, Noah Misch n...@leadboat.com wrote: Well, post-release, the cat is out of the bag: we'll be stuck with this whether the performance characteristics are acceptable or not. That's why we'd better be as sure as possible before committing to this implementation

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-14 Thread Robert Haas
On Wed, Mar 14, 2012 at 9:17 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Agreed.  But speaking of that, why exactly do we fsync the multixact SLRU today? Good question.  So far, I can't think of a reason.  nextMulti is critical, but we already fsync it with pg_control.  We could

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-13 Thread Bruce Momjian
On Tue, Mar 06, 2012 at 04:39:32PM -0300, Alvaro Herrera wrote: Here's a first attempt at a README illustrating this. I intend this to be placed in src/backend/access/heap/README.tuplock; the first three paragraphs are stolen from the comment in heap_lock_tuple, so I'd remove those from

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-13 Thread Robert Haas
On Mon, Mar 12, 2012 at 9:24 PM, Noah Misch n...@leadboat.com wrote: When we lock an update-in-progress row, we walk the t_ctid chain and lock all descendant tuples.  They may all have uncommitted xmins.  This is essential to ensure that the final outcome of the updating transaction does not

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-13 Thread Alvaro Herrera
Excerpts from Bruce Momjian's message of mar mar 13 14:00:52 -0300 2012: On Tue, Mar 06, 2012 at 04:39:32PM -0300, Alvaro Herrera wrote: When there is a single locker in a tuple, we can just store the locking info in the tuple itself. We do this by storing the locker's Xid in XMAX, and

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-13 Thread Robert Haas
On Mon, Mar 12, 2012 at 3:28 PM, Simon Riggs si...@2ndquadrant.com wrote: I agree with you that some worst case performance tests should be done. Could you please say what you think the worst cases would be, so those can be tested? That would avoid wasting time or getting anything backwards.

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-13 Thread Noah Misch
On Tue, Mar 13, 2012 at 01:46:24PM -0400, Robert Haas wrote: On Mon, Mar 12, 2012 at 3:28 PM, Simon Riggs si...@2ndquadrant.com wrote: I agree with you that some worst case performance tests should be done. Could you please say what you think the worst cases would be, so those can be

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-12 Thread Robert Haas
On Sun, Feb 26, 2012 at 9:47 PM, Robert Haas robertmh...@gmail.com wrote: Regarding performance, the good thing about this patch is that if you have an operation that used to block, it might now not block.  So maybe multixact-related operation is a bit slower than before, but if it allows you

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-12 Thread Simon Riggs
On Mon, Mar 12, 2012 at 5:28 PM, Robert Haas robertmh...@gmail.com wrote: In other words, we'd entirely avoid needing to make mxacts crash-safe, and we'd avoid most of the extra SLRU lookups that the current implementation requires; they'd only be needed when (and for as long as) the locked

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-12 Thread Robert Haas
On Mon, Mar 12, 2012 at 1:50 PM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, Mar 12, 2012 at 5:28 PM, Robert Haas robertmh...@gmail.com wrote: In other words, we'd entirely avoid needing to make mxacts crash-safe, and we'd avoid most of the extra SLRU lookups that the current

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-12 Thread Simon Riggs
On Mon, Mar 12, 2012 at 6:14 PM, Robert Haas robertmh...@gmail.com wrote: Considering that nobody's done any work to resolve the uncertainty about whether the worst-case performance characteristics of this patch are acceptable, and considering further that it was undergoing massive code churn

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-12 Thread Noah Misch
On Mon, Mar 12, 2012 at 01:28:11PM -0400, Robert Haas wrote: I spent some time thinking about this over the weekend, and I have an observation, and an idea. Here's the observation: I believe that locking a tuple whose xmin is uncommitted is always a noop, because if it's ever possible for a

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Gokulakannan Somasundaram
I feel sad, that i followed this topic very late. But i still want to put forward my views. Have we thought on the lines of how Robert has implemented relation level locks. In short it should go like this a) The locks for enforcing Referential integrity should be taken only when the rarest of the

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Simon Riggs
On Wed, Mar 7, 2012 at 9:24 AM, Gokulakannan Somasundaram gokul...@gmail.com wrote: I feel sad, that i followed this topic very late. But i still want to put forward my views. Have we thought on the lines of how Robert has implemented relation level locks. In short it should go like this a)

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Gokulakannan Somasundaram
Insert, Update and Delete don't take locks they simply mark the tuples they change with an xid. Anybody else wanting to wait on the lock just waits on the xid. We do insert a lock row for each xid, but not one per row changed. I mean the foreign key checks here. They take a Select for Share

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Simon Riggs
On Wed, Mar 7, 2012 at 10:18 AM, Gokulakannan Somasundaram gokul...@gmail.com wrote: Insert, Update and Delete don't take locks they simply mark the tuples they change with an xid. Anybody else wanting to wait on the lock just waits on the xid. We do insert a lock row for each xid, but not

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Gokulakannan Somasundaram
Please explain in detail your idea of how it will work. OK. I will try to explain the abstract idea, i have. a) Referential integrity gets violated, when there are referencing key values, not present in the referenced key values. We are maintaining the integrity by taking a Select for Share

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Simon Riggs
n Wed, Mar 7, 2012 at 11:37 AM, Gokulakannan Somasundaram gokul...@gmail.com wrote: Please explain in detail your idea of how it will work. So we will take some kind of lock, which will stop such a happening. ... May be someone can come up with better ideas than this. With respect, I don't

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of lun mar 05 16:34:10 -0300 2012: It does however, illustrate my next review comment which is that the comments and README items are sorely lacking here. It's quite hard to see how it works, let along comment on major design decisions. It would help

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Simon Riggs
On Mon, Mar 5, 2012 at 8:35 PM, Simon Riggs si...@2ndquadrant.com wrote: * Why do we need multixact to be persistent? Do we need every page of multixact to be persistent, or just particular pages in certain circumstances? Any page that contains at least one multi with an update as a member

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 7:39 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: We provide four levels of tuple locking strength: SELECT FOR KEY UPDATE is super-exclusive locking (used to delete tuples and more generally to update tuples modifying the values of the columns that make up the

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Robert Haas
Preliminary comment: This README is very helpful. On Tue, Mar 6, 2012 at 2:39 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: We provide four levels of tuple locking strength: SELECT FOR KEY UPDATE is super-exclusive locking (used to delete tuples and more generally to update tuples

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar mar 06 18:10:16 -0300 2012: Preliminary comment: This README is very helpful. Thanks. I feel silly that I didn't write it earlier. On Tue, Mar 6, 2012 at 2:39 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: We provide four levels of

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 9:10 PM, Robert Haas robertmh...@gmail.com wrote: Preliminary comment: This README is very helpful. On Tue, Mar 6, 2012 at 2:39 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: We provide four levels of tuple locking strength: SELECT FOR KEY UPDATE is

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Robert Haas
On Tue, Mar 6, 2012 at 4:27 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mar mar 06 18:10:16 -0300 2012: Preliminary comment: This README is very helpful. Thanks.  I feel silly that I didn't write it earlier. On Tue, Mar 6, 2012 at 2:39 PM,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Simon Riggs
On Mon, Feb 27, 2012 at 2:47 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Feb 23, 2012 at 11:01 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: This seems like a horrid mess that's going to be unsustainable both from a complexity and a performance standpoint.  The only reason

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of lun mar 05 15:28:59 -0300 2012: On Mon, Feb 27, 2012 at 2:47 AM, Robert Haas robertmh...@gmail.com wrote: From a performance standpoint, we really need to think not only about the cases where the patch wins, but also, and maybe more importantly,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Simon Riggs
On Mon, Mar 5, 2012 at 6:37 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Simon Riggs's message of lun mar 05 15:28:59 -0300 2012: On Mon, Feb 27, 2012 at 2:47 AM, Robert Haas robertmh...@gmail.com wrote: From a performance standpoint, we really need to think not only

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of lun mar 05 16:34:10 -0300 2012: On Mon, Mar 5, 2012 at 6:37 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: It does however, illustrate my next review comment which is that the comments and README items are sorely lacking here. It's quite hard to

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Simon Riggs
On Mon, Mar 5, 2012 at 7:53 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: My other comments so far are * some permutations commented out - no comments as to why Something of a fault with the isolation tester that it just shows output, there's no way to record expected output in the

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-27 Thread Heikki Linnakangas
On 23.02.2012 18:01, Alvaro Herrera wrote: Excerpts from Tom Lane's message of jue feb 23 12:28:20 -0300 2012: Alvaro Herreraalvhe...@commandprompt.com writes: Sure. The problem is that we are allowing updated rows to be locked (and locked rows to be updated). This means that we need to

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-27 Thread Noah Misch
On Mon, Feb 27, 2012 at 02:13:32PM +0200, Heikki Linnakangas wrote: On 23.02.2012 18:01, Alvaro Herrera wrote: As far as complexity, yeah, it's a lot more complex now -- no question about that. How about assigning a new, real, transaction id, to represent the group of transaction ids. The

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-27 Thread Simon Riggs
On Tue, Feb 28, 2012 at 12:28 AM, Noah Misch n...@leadboat.com wrote: On Mon, Feb 27, 2012 at 02:13:32PM +0200, Heikki Linnakangas wrote: On 23.02.2012 18:01, Alvaro Herrera wrote: As far as complexity, yeah, it's a lot more complex now -- no question about that. How about assigning a new,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-26 Thread Robert Haas
On Thu, Feb 23, 2012 at 11:01 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: This seems like a horrid mess that's going to be unsustainable both from a complexity and a performance standpoint.  The only reason multixacts were tolerable at all was that they had only one semantics.  

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-25 Thread Kevin Grittner
Vik Reykja vikrey...@gmail.com wrote: Kevin Grittner kevin.gritt...@wicourts.govwrote: One of the problems that Florian was trying to address is that people often have a need to enforce something with a lot of similarity to a foreign key, but with more subtle logic than declarative foreign

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-24 Thread Jeroen Vermeulen
On 2012-02-23 22:12, Noah Misch wrote: That alone would not simplify the patch much. INSERT/UPDATE/DELETE on the foreign side would still need to take some kind of tuple lock on the primary side to prevent primary-side DELETE. You then still face the complicated case of a tuple that's both

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-24 Thread Vik Reykja
On Thu, Feb 23, 2012 at 19:44, Kevin Grittner kevin.gritt...@wicourts.govwrote: One of the problems that Florian was trying to address is that people often have a need to enforce something with a lot of similarity to a foreign key, but with more subtle logic than declarative foreign keys

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Wed, Feb 22, 2012 at 5:00 PM, Noah Misch n...@leadboat.com wrote: All in all, I think this is in pretty much final shape.  Only pg_upgrade bits are still missing.  If sharp eyes could give this a critical look and knuckle-cracking testers could give it a spin, that would be helpful. Lack

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Jeroen Vermeulen
On 2012-02-23 10:18, Simon Riggs wrote: However, review of such a large patch should not be simply pass or fail. We should be looking back at the original problem and ask ourselves whether some subset of the patch could solve a useful subset of the problem. For me, that seems quite likely and

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Sun, Dec 4, 2011 at 12:20 PM, Noah Misch n...@leadboat.com wrote: Making pg_multixact persistent across clean shutdowns is no bridge to cross lightly, since it means committing to an on-disk format for an indefinite period.  We should do it; the benefits of this patch justify it, and I

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Thu, Feb 23, 2012 at 1:08 PM, Jeroen Vermeulen j...@xs4all.nl wrote: Simon, I think you had a reason why it couldn't work, but I didn't quite get your meaning and didn't want to distract things further at that stage.  You wrote that it doesn't do what KEY LOCKS are designed to do...  any

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue feb 23 11:15:45 -0300 2012: On Sun, Dec 4, 2011 at 12:20 PM, Noah Misch n...@leadboat.com wrote: Making pg_multixact persistent across clean shutdowns is no bridge to cross lightly, since it means committing to an on-disk format for an indefinite

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Thu, Feb 23, 2012 at 3:04 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Simon Riggs's message of jue feb 23 11:15:45 -0300 2012: On Sun, Dec 4, 2011 at 12:20 PM, Noah Misch n...@leadboat.com wrote: Making pg_multixact persistent across clean shutdowns is no bridge to

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: Sure. The problem is that we are allowing updated rows to be locked (and locked rows to be updated). This means that we need to store extended Xmax information in tuples that goes beyond mere locks, which is what we were doing previously --

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue feb 23 06:18:57 -0300 2012: On Wed, Feb 22, 2012 at 5:00 PM, Noah Misch n...@leadboat.com wrote: All in all, I think this is in pretty much final shape.  Only pg_upgrade bits are still missing.  If sharp eyes could give this a critical look

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue feb 23 12:12:13 -0300 2012: On Thu, Feb 23, 2012 at 3:04 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Sure.  The problem is that we are allowing updated rows to be locked (and locked rows to be updated).  This means that we need to store

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Tom Lane's message of jue feb 23 12:28:20 -0300 2012: Alvaro Herrera alvhe...@commandprompt.com writes: Sure. The problem is that we are allowing updated rows to be locked (and locked rows to be updated). This means that we need to store extended Xmax information in

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Kevin Grittner
Alvaro Herrera alvhe...@commandprompt.com wrote: As for sanity -- I regard multixacts as a way to store extended Xmax information. The original idea was obviously much more limited in that the extended info was just locking info. We've extended it but I don't think it's such a stretch.

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Kevin Grittner's message of jue feb 23 13:31:36 -0300 2012: Alvaro Herrera alvhe...@commandprompt.com wrote: As for sanity -- I regard multixacts as a way to store extended Xmax information. The original idea was obviously much more limited in that the extended info was

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Greg Smith
On 02/23/2012 10:43 AM, Alvaro Herrera wrote: I completely understand that you don't want to review this latest version of the patch; it's a lot of effort and I wouldn't inflict it on anybody who hasn't not volunteered. However, it doesn't seem to me that this is reason to boot the patch from

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Greg Smith's message of jue feb 23 14:48:13 -0300 2012: On 02/23/2012 10:43 AM, Alvaro Herrera wrote: I completely understand that you don't want to review this latest version of the patch; it's a lot of effort and I wouldn't inflict it on anybody who hasn't not volunteered.

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Greg Smith
On 02/23/2012 01:04 PM, Alvaro Herrera wrote: manual vacuum is teh sux0r I think you've just named my next conference talk submission. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Thu, Feb 23, 2012 at 4:01 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: As far as complexity, yeah, it's a lot more complex now -- no question about that. As far as complexity goes, would it be easier if we treated the UPDATE of a primary key column as a DELETE plus an INSERT?

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Kevin Grittner
Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Kevin Grittner's message: Since the limitation on what can be stored in xmax was the killer for Florian's attempt to support SELECT FOR UPDATE in a form which was arguably more useful (and certainly more convenient for those

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Noah Misch
On Thu, Feb 23, 2012 at 02:08:28PM +0100, Jeroen Vermeulen wrote: On 2012-02-23 10:18, Simon Riggs wrote: However, review of such a large patch should not be simply pass or fail. We should be looking back at the original problem and ask ourselves whether some subset of the patch could solve a

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Noah Misch's message of mié feb 22 14:00:07 -0300 2012: On Mon, Feb 13, 2012 at 07:16:58PM -0300, Alvaro Herrera wrote: On Mon, Jan 30, 2012 at 06:48:47PM -0500, Noah Misch wrote: On Tue, Jan 24, 2012 at 03:47:16PM -0300, Alvaro Herrera wrote: * Columns that are part of

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Noah Misch
On Thu, Feb 23, 2012 at 06:36:42PM -0300, Alvaro Herrera wrote: Excerpts from Noah Misch's message of mi?? feb 22 14:00:07 -0300 2012: On Mon, Feb 13, 2012 at 07:16:58PM -0300, Alvaro Herrera wrote: On Mon, Jan 30, 2012 at 06:48:47PM -0500, Noah Misch wrote: On Tue, Jan 24, 2012 at

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Noah Misch
On Thu, Feb 23, 2012 at 12:43:11PM -0300, Alvaro Herrera wrote: Excerpts from Simon Riggs's message of jue feb 23 06:18:57 -0300 2012: On Wed, Feb 22, 2012 at 5:00 PM, Noah Misch n...@leadboat.com wrote: All in all, I think this is in pretty much final shape. ??Only pg_upgrade bits are

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-22 Thread Noah Misch
On Mon, Feb 13, 2012 at 07:16:58PM -0300, Alvaro Herrera wrote: Okay, so this patch fixes the truncation and wraparound issues through a mechanism much like pg_clog's: it keeps track of the oldest possibly existing multis on each and every table, and then during tuple freezing those are

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-01 Thread Jim Nasby
On Jan 31, 2012, at 10:58 AM, Alvaro Herrera wrote: I think it's butt-ugly, but it's only slightly uglier than relfrozenxid which we're already stuck with. The slight amount of additional ugliness is that you're going to use an XID column to store a uint4 that is not an XID - but I don't have

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-01 Thread Alvaro Herrera
Excerpts from Jim Nasby's message of mié feb 01 21:33:47 -0300 2012: On Jan 31, 2012, at 10:58 AM, Alvaro Herrera wrote: I think it's butt-ugly, but it's only slightly uglier than relfrozenxid which we're already stuck with. The slight amount of additional ugliness is that you're going

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Robert Haas
On Mon, Jan 30, 2012 at 6:48 PM, Noah Misch n...@leadboat.com wrote: On Tue, Jan 24, 2012 at 03:47:16PM -0300, Alvaro Herrera wrote: The biggest item remaining is the point you raised about multixactid wraparound.  This is closely related to multixact truncation and the way checkpoints are to

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar ene 31 10:17:40 -0300 2012: I suspect you are right that it is unlikely, but OTOH that sounds like an extremely painful recovery procedure. We probably don't need to put a ton of thought into handling this case as efficiently as possible, but I

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Robert Haas
On Tue, Jan 31, 2012 at 9:19 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mar ene 31 10:17:40 -0300 2012: I suspect you are right that it is unlikely, but OTOH that sounds like an extremely painful recovery procedure.  We probably don't need to

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar ene 31 13:18:30 -0300 2012: On Tue, Jan 31, 2012 at 9:19 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mar ene 31 10:17:40 -0300 2012: I suspect you are right that it is unlikely, but OTOH that sounds

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Robert Haas
On Tue, Jan 31, 2012 at 11:58 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Well, we're already storing a multixact in Xmax, so it's not like we don't assume that we can store multis in space normally reserved for Xids.  What I've been wondering is not how ugly it is, but rather of the

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-30 Thread Noah Misch
On Tue, Jan 24, 2012 at 03:47:16PM -0300, Alvaro Herrera wrote: The biggest item remaining is the point you raised about multixactid wraparound. This is closely related to multixact truncation and the way checkpoints are to be handled. If we think that MultiXactId wraparound is possible, and

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-26 Thread Alvaro Herrera
Excerpts from Alvaro Herrera's message of mar ene 24 15:47:16 -0300 2012: Need more code changes for the following: * export FOR KEY UPDATE lock mode in SQL While doing this, I realized that there's an open item here regarding a transaction that locks a tuple, and then in an aborted

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-18 Thread Alvaro Herrera
Excerpts from Heikki Linnakangas's message of mar ene 17 03:21:28 -0300 2012: On 16.01.2012 21:52, Alvaro Herrera wrote: Excerpts from Heikki Linnakangas's message of lun ene 16 16:17:42 -0300 2012: On 15.01.2012 06:49, Alvaro Herrera wrote: - pg_upgrade bits are missing. I

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-18 Thread Noah Misch
On Wed, Jan 18, 2012 at 05:18:31PM -0300, Alvaro Herrera wrote: Excerpts from Heikki Linnakangas's message of mar ene 17 03:21:28 -0300 2012: On 16.01.2012 21:52, Alvaro Herrera wrote: I was initially thinking that pg_multixact should return the empty set if requested members of a multi

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-17 Thread Noah Misch
On Sun, Jan 15, 2012 at 01:49:54AM -0300, Alvaro Herrera wrote: - I'm not sure that the multixact truncation code is sane on checkpoints. It might be that I need to tweak more the pg_control info we keep about truncation. The whole truncation thing needs more testing, too. My largest

  1   2   >