Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Gokulakannan Somasundaram
Insert, Update and Delete don't take locks they simply mark the tuples they change with an xid. Anybody else wanting to wait on the lock just waits on the xid. We do insert a lock row for each xid, but not one per row changed. I mean the foreign key checks here. They take a Select for Share

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Simon Riggs
On Wed, Mar 7, 2012 at 10:18 AM, Gokulakannan Somasundaram gokul...@gmail.com wrote: Insert, Update and Delete don't take locks they simply mark the tuples they change with an xid. Anybody else wanting to wait on the lock just waits on the xid. We do insert a lock row for each xid, but not

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Gokulakannan Somasundaram
Please explain in detail your idea of how it will work. OK. I will try to explain the abstract idea, i have. a) Referential integrity gets violated, when there are referencing key values, not present in the referenced key values. We are maintaining the integrity by taking a Select for Share

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-07 Thread Simon Riggs
n Wed, Mar 7, 2012 at 11:37 AM, Gokulakannan Somasundaram gokul...@gmail.com wrote: Please explain in detail your idea of how it will work. So we will take some kind of lock, which will stop such a happening. ... May be someone can come up with better ideas than this. With respect, I don't

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of lun mar 05 16:34:10 -0300 2012: It does however, illustrate my next review comment which is that the comments and README items are sorely lacking here. It's quite hard to see how it works, let along comment on major design decisions. It would help

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Simon Riggs
On Mon, Mar 5, 2012 at 8:35 PM, Simon Riggs si...@2ndquadrant.com wrote: * Why do we need multixact to be persistent? Do we need every page of multixact to be persistent, or just particular pages in certain circumstances? Any page that contains at least one multi with an update as a member

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 7:39 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: We provide four levels of tuple locking strength: SELECT FOR KEY UPDATE is super-exclusive locking (used to delete tuples and more generally to update tuples modifying the values of the columns that make up the

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Robert Haas
Preliminary comment: This README is very helpful. On Tue, Mar 6, 2012 at 2:39 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: We provide four levels of tuple locking strength: SELECT FOR KEY UPDATE is super-exclusive locking (used to delete tuples and more generally to update tuples

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar mar 06 18:10:16 -0300 2012: Preliminary comment: This README is very helpful. Thanks. I feel silly that I didn't write it earlier. On Tue, Mar 6, 2012 at 2:39 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: We provide four levels of

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 9:10 PM, Robert Haas robertmh...@gmail.com wrote: Preliminary comment: This README is very helpful. On Tue, Mar 6, 2012 at 2:39 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: We provide four levels of tuple locking strength: SELECT FOR KEY UPDATE is

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-06 Thread Robert Haas
On Tue, Mar 6, 2012 at 4:27 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mar mar 06 18:10:16 -0300 2012: Preliminary comment: This README is very helpful. Thanks.  I feel silly that I didn't write it earlier. On Tue, Mar 6, 2012 at 2:39 PM,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Simon Riggs
On Mon, Feb 27, 2012 at 2:47 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Feb 23, 2012 at 11:01 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: This seems like a horrid mess that's going to be unsustainable both from a complexity and a performance standpoint.  The only reason

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of lun mar 05 15:28:59 -0300 2012: On Mon, Feb 27, 2012 at 2:47 AM, Robert Haas robertmh...@gmail.com wrote: From a performance standpoint, we really need to think not only about the cases where the patch wins, but also, and maybe more importantly,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Simon Riggs
On Mon, Mar 5, 2012 at 6:37 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Simon Riggs's message of lun mar 05 15:28:59 -0300 2012: On Mon, Feb 27, 2012 at 2:47 AM, Robert Haas robertmh...@gmail.com wrote: From a performance standpoint, we really need to think not only

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of lun mar 05 16:34:10 -0300 2012: On Mon, Mar 5, 2012 at 6:37 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: It does however, illustrate my next review comment which is that the comments and README items are sorely lacking here. It's quite hard to

Re: [HACKERS] foreign key locks, 2nd attempt

2012-03-05 Thread Simon Riggs
On Mon, Mar 5, 2012 at 7:53 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: My other comments so far are * some permutations commented out - no comments as to why Something of a fault with the isolation tester that it just shows output, there's no way to record expected output in the

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-27 Thread Heikki Linnakangas
On 23.02.2012 18:01, Alvaro Herrera wrote: Excerpts from Tom Lane's message of jue feb 23 12:28:20 -0300 2012: Alvaro Herreraalvhe...@commandprompt.com writes: Sure. The problem is that we are allowing updated rows to be locked (and locked rows to be updated). This means that we need to

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-27 Thread Noah Misch
On Mon, Feb 27, 2012 at 02:13:32PM +0200, Heikki Linnakangas wrote: On 23.02.2012 18:01, Alvaro Herrera wrote: As far as complexity, yeah, it's a lot more complex now -- no question about that. How about assigning a new, real, transaction id, to represent the group of transaction ids. The

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-27 Thread Simon Riggs
On Tue, Feb 28, 2012 at 12:28 AM, Noah Misch n...@leadboat.com wrote: On Mon, Feb 27, 2012 at 02:13:32PM +0200, Heikki Linnakangas wrote: On 23.02.2012 18:01, Alvaro Herrera wrote: As far as complexity, yeah, it's a lot more complex now -- no question about that. How about assigning a new,

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-26 Thread Robert Haas
On Thu, Feb 23, 2012 at 11:01 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: This seems like a horrid mess that's going to be unsustainable both from a complexity and a performance standpoint.  The only reason multixacts were tolerable at all was that they had only one semantics.  

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-25 Thread Kevin Grittner
Vik Reykja vikrey...@gmail.com wrote: Kevin Grittner kevin.gritt...@wicourts.govwrote: One of the problems that Florian was trying to address is that people often have a need to enforce something with a lot of similarity to a foreign key, but with more subtle logic than declarative foreign

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-24 Thread Jeroen Vermeulen
On 2012-02-23 22:12, Noah Misch wrote: That alone would not simplify the patch much. INSERT/UPDATE/DELETE on the foreign side would still need to take some kind of tuple lock on the primary side to prevent primary-side DELETE. You then still face the complicated case of a tuple that's both

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-24 Thread Vik Reykja
On Thu, Feb 23, 2012 at 19:44, Kevin Grittner kevin.gritt...@wicourts.govwrote: One of the problems that Florian was trying to address is that people often have a need to enforce something with a lot of similarity to a foreign key, but with more subtle logic than declarative foreign keys

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Wed, Feb 22, 2012 at 5:00 PM, Noah Misch n...@leadboat.com wrote: All in all, I think this is in pretty much final shape.  Only pg_upgrade bits are still missing.  If sharp eyes could give this a critical look and knuckle-cracking testers could give it a spin, that would be helpful. Lack

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Jeroen Vermeulen
On 2012-02-23 10:18, Simon Riggs wrote: However, review of such a large patch should not be simply pass or fail. We should be looking back at the original problem and ask ourselves whether some subset of the patch could solve a useful subset of the problem. For me, that seems quite likely and

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Sun, Dec 4, 2011 at 12:20 PM, Noah Misch n...@leadboat.com wrote: Making pg_multixact persistent across clean shutdowns is no bridge to cross lightly, since it means committing to an on-disk format for an indefinite period.  We should do it; the benefits of this patch justify it, and I

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Thu, Feb 23, 2012 at 1:08 PM, Jeroen Vermeulen j...@xs4all.nl wrote: Simon, I think you had a reason why it couldn't work, but I didn't quite get your meaning and didn't want to distract things further at that stage.  You wrote that it doesn't do what KEY LOCKS are designed to do...  any

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue feb 23 11:15:45 -0300 2012: On Sun, Dec 4, 2011 at 12:20 PM, Noah Misch n...@leadboat.com wrote: Making pg_multixact persistent across clean shutdowns is no bridge to cross lightly, since it means committing to an on-disk format for an indefinite

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Thu, Feb 23, 2012 at 3:04 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Simon Riggs's message of jue feb 23 11:15:45 -0300 2012: On Sun, Dec 4, 2011 at 12:20 PM, Noah Misch n...@leadboat.com wrote: Making pg_multixact persistent across clean shutdowns is no bridge to

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: Sure. The problem is that we are allowing updated rows to be locked (and locked rows to be updated). This means that we need to store extended Xmax information in tuples that goes beyond mere locks, which is what we were doing previously --

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue feb 23 06:18:57 -0300 2012: On Wed, Feb 22, 2012 at 5:00 PM, Noah Misch n...@leadboat.com wrote: All in all, I think this is in pretty much final shape.  Only pg_upgrade bits are still missing.  If sharp eyes could give this a critical look

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue feb 23 12:12:13 -0300 2012: On Thu, Feb 23, 2012 at 3:04 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Sure.  The problem is that we are allowing updated rows to be locked (and locked rows to be updated).  This means that we need to store

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Tom Lane's message of jue feb 23 12:28:20 -0300 2012: Alvaro Herrera alvhe...@commandprompt.com writes: Sure. The problem is that we are allowing updated rows to be locked (and locked rows to be updated). This means that we need to store extended Xmax information in

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Kevin Grittner
Alvaro Herrera alvhe...@commandprompt.com wrote: As for sanity -- I regard multixacts as a way to store extended Xmax information. The original idea was obviously much more limited in that the extended info was just locking info. We've extended it but I don't think it's such a stretch.

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Kevin Grittner's message of jue feb 23 13:31:36 -0300 2012: Alvaro Herrera alvhe...@commandprompt.com wrote: As for sanity -- I regard multixacts as a way to store extended Xmax information. The original idea was obviously much more limited in that the extended info was

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Greg Smith
On 02/23/2012 10:43 AM, Alvaro Herrera wrote: I completely understand that you don't want to review this latest version of the patch; it's a lot of effort and I wouldn't inflict it on anybody who hasn't not volunteered. However, it doesn't seem to me that this is reason to boot the patch from

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Greg Smith's message of jue feb 23 14:48:13 -0300 2012: On 02/23/2012 10:43 AM, Alvaro Herrera wrote: I completely understand that you don't want to review this latest version of the patch; it's a lot of effort and I wouldn't inflict it on anybody who hasn't not volunteered.

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Greg Smith
On 02/23/2012 01:04 PM, Alvaro Herrera wrote: manual vacuum is teh sux0r I think you've just named my next conference talk submission. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Simon Riggs
On Thu, Feb 23, 2012 at 4:01 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: As far as complexity, yeah, it's a lot more complex now -- no question about that. As far as complexity goes, would it be easier if we treated the UPDATE of a primary key column as a DELETE plus an INSERT?

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Kevin Grittner
Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Kevin Grittner's message: Since the limitation on what can be stored in xmax was the killer for Florian's attempt to support SELECT FOR UPDATE in a form which was arguably more useful (and certainly more convenient for those

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Noah Misch
On Thu, Feb 23, 2012 at 02:08:28PM +0100, Jeroen Vermeulen wrote: On 2012-02-23 10:18, Simon Riggs wrote: However, review of such a large patch should not be simply pass or fail. We should be looking back at the original problem and ask ourselves whether some subset of the patch could solve a

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Alvaro Herrera
Excerpts from Noah Misch's message of mié feb 22 14:00:07 -0300 2012: On Mon, Feb 13, 2012 at 07:16:58PM -0300, Alvaro Herrera wrote: On Mon, Jan 30, 2012 at 06:48:47PM -0500, Noah Misch wrote: On Tue, Jan 24, 2012 at 03:47:16PM -0300, Alvaro Herrera wrote: * Columns that are part of

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Noah Misch
On Thu, Feb 23, 2012 at 06:36:42PM -0300, Alvaro Herrera wrote: Excerpts from Noah Misch's message of mi?? feb 22 14:00:07 -0300 2012: On Mon, Feb 13, 2012 at 07:16:58PM -0300, Alvaro Herrera wrote: On Mon, Jan 30, 2012 at 06:48:47PM -0500, Noah Misch wrote: On Tue, Jan 24, 2012 at

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-23 Thread Noah Misch
On Thu, Feb 23, 2012 at 12:43:11PM -0300, Alvaro Herrera wrote: Excerpts from Simon Riggs's message of jue feb 23 06:18:57 -0300 2012: On Wed, Feb 22, 2012 at 5:00 PM, Noah Misch n...@leadboat.com wrote: All in all, I think this is in pretty much final shape. ??Only pg_upgrade bits are

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-22 Thread Noah Misch
On Mon, Feb 13, 2012 at 07:16:58PM -0300, Alvaro Herrera wrote: Okay, so this patch fixes the truncation and wraparound issues through a mechanism much like pg_clog's: it keeps track of the oldest possibly existing multis on each and every table, and then during tuple freezing those are

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-01 Thread Jim Nasby
On Jan 31, 2012, at 10:58 AM, Alvaro Herrera wrote: I think it's butt-ugly, but it's only slightly uglier than relfrozenxid which we're already stuck with. The slight amount of additional ugliness is that you're going to use an XID column to store a uint4 that is not an XID - but I don't have

Re: [HACKERS] foreign key locks, 2nd attempt

2012-02-01 Thread Alvaro Herrera
Excerpts from Jim Nasby's message of mié feb 01 21:33:47 -0300 2012: On Jan 31, 2012, at 10:58 AM, Alvaro Herrera wrote: I think it's butt-ugly, but it's only slightly uglier than relfrozenxid which we're already stuck with. The slight amount of additional ugliness is that you're going

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Robert Haas
On Mon, Jan 30, 2012 at 6:48 PM, Noah Misch n...@leadboat.com wrote: On Tue, Jan 24, 2012 at 03:47:16PM -0300, Alvaro Herrera wrote: The biggest item remaining is the point you raised about multixactid wraparound.  This is closely related to multixact truncation and the way checkpoints are to

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar ene 31 10:17:40 -0300 2012: I suspect you are right that it is unlikely, but OTOH that sounds like an extremely painful recovery procedure. We probably don't need to put a ton of thought into handling this case as efficiently as possible, but I

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Robert Haas
On Tue, Jan 31, 2012 at 9:19 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mar ene 31 10:17:40 -0300 2012: I suspect you are right that it is unlikely, but OTOH that sounds like an extremely painful recovery procedure.  We probably don't need to

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar ene 31 13:18:30 -0300 2012: On Tue, Jan 31, 2012 at 9:19 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Robert Haas's message of mar ene 31 10:17:40 -0300 2012: I suspect you are right that it is unlikely, but OTOH that sounds

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-31 Thread Robert Haas
On Tue, Jan 31, 2012 at 11:58 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Well, we're already storing a multixact in Xmax, so it's not like we don't assume that we can store multis in space normally reserved for Xids.  What I've been wondering is not how ugly it is, but rather of the

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-30 Thread Noah Misch
On Tue, Jan 24, 2012 at 03:47:16PM -0300, Alvaro Herrera wrote: The biggest item remaining is the point you raised about multixactid wraparound. This is closely related to multixact truncation and the way checkpoints are to be handled. If we think that MultiXactId wraparound is possible, and

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-26 Thread Alvaro Herrera
Excerpts from Alvaro Herrera's message of mar ene 24 15:47:16 -0300 2012: Need more code changes for the following: * export FOR KEY UPDATE lock mode in SQL While doing this, I realized that there's an open item here regarding a transaction that locks a tuple, and then in an aborted

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-18 Thread Alvaro Herrera
Excerpts from Heikki Linnakangas's message of mar ene 17 03:21:28 -0300 2012: On 16.01.2012 21:52, Alvaro Herrera wrote: Excerpts from Heikki Linnakangas's message of lun ene 16 16:17:42 -0300 2012: On 15.01.2012 06:49, Alvaro Herrera wrote: - pg_upgrade bits are missing. I

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-18 Thread Noah Misch
On Wed, Jan 18, 2012 at 05:18:31PM -0300, Alvaro Herrera wrote: Excerpts from Heikki Linnakangas's message of mar ene 17 03:21:28 -0300 2012: On 16.01.2012 21:52, Alvaro Herrera wrote: I was initially thinking that pg_multixact should return the empty set if requested members of a multi

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-17 Thread Noah Misch
On Sun, Jan 15, 2012 at 01:49:54AM -0300, Alvaro Herrera wrote: - I'm not sure that the multixact truncation code is sane on checkpoints. It might be that I need to tweak more the pg_control info we keep about truncation. The whole truncation thing needs more testing, too. My largest

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-17 Thread Noah Misch
On Mon, Jan 16, 2012 at 04:52:36PM -0300, Alvaro Herrera wrote: Excerpts from Heikki Linnakangas's message of lun ene 16 16:17:42 -0300 2012: On 15.01.2012 06:49, Alvaro Herrera wrote: --- 164,178 #define HEAP_HASVARWIDTH0x0002/* has variable-width attribute(s) */

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-16 Thread Heikki Linnakangas
On 15.01.2012 06:49, Alvaro Herrera wrote: --- 164,178 #define HEAP_HASVARWIDTH 0x0002 /* has variable-width attribute(s) */ #define HEAP_HASEXTERNAL 0x0004 /* has external stored attribute(s) */ #define HEAP_HASOID 0x0008 /*

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-16 Thread Alvaro Herrera
Excerpts from Heikki Linnakangas's message of lun ene 16 16:17:42 -0300 2012: On 15.01.2012 06:49, Alvaro Herrera wrote: --- 164,178 #define HEAP_HASVARWIDTH0x0002/* has variable-width attribute(s) */ #define HEAP_HASEXTERNAL0x0004/* has external

Re: [HACKERS] foreign key locks, 2nd attempt

2012-01-16 Thread Heikki Linnakangas
On 16.01.2012 21:52, Alvaro Herrera wrote: Excerpts from Heikki Linnakangas's message of lun ene 16 16:17:42 -0300 2012: On 15.01.2012 06:49, Alvaro Herrera wrote: - pg_upgrade bits are missing. I guess we'll need to rewrite pg_multixact contents in pg_upgrade. Is the page format

Re: [HACKERS] foreign key locks, 2nd attempt

2011-12-16 Thread Greg Smith
Sounds like there's still a few things left to research out on Alvaro's side, and I'm thinking there's a performance/reliability under load testing side of this that will take some work to validate too. Since I can't see all that happening fast enough to commit for a bit, I'm going to mark it

Re: [HACKERS] foreign key locks, 2nd attempt

2011-12-14 Thread Noah Misch
On Tue, Dec 13, 2011 at 06:36:21PM -0300, Alvaro Herrera wrote: Excerpts from Noah Misch's message of dom dic 04 09:20:27 -0300 2011: @@ -2725,11 +2884,20 @@ l2: oldtup.t_data-t_infomask = ~(HEAP_XMAX_COMMITTED | HEAP_XMAX_INVALID |

Re: [HACKERS] foreign key locks, 2nd attempt

2011-12-14 Thread Tom Lane
Noah Misch n...@leadboat.com writes: On Tue, Dec 13, 2011 at 06:36:21PM -0300, Alvaro Herrera wrote: Yeah, I've been wondering about this: do we have a problem already with exclusion constraints? I mean, if a concurrent inserter doesn't see the tuple that we've marked here as deleted while we

Re: [HACKERS] foreign key locks, 2nd attempt

2011-12-13 Thread Noah Misch
On Mon, Dec 12, 2011 at 05:20:39PM -0300, Alvaro Herrera wrote: Excerpts from Noah Misch's message of dom dic 04 09:20:27 -0300 2011: Second, I tried a SELECT FOR SHARE on a table of 1M tuples; this might incur some cost due to the now-guaranteed use of pg_multixact for FOR SHARE. See

Re: [HACKERS] foreign key locks, 2nd attempt

2011-12-13 Thread Alvaro Herrera
Excerpts from Noah Misch's message of mar dic 13 11:44:49 -0300 2011: On Mon, Dec 12, 2011 at 05:20:39PM -0300, Alvaro Herrera wrote: Excerpts from Noah Misch's message of dom dic 04 09:20:27 -0300 2011: Second, I tried a SELECT FOR SHARE on a table of 1M tuples; this might incur

Re: [HACKERS] foreign key locks, 2nd attempt

2011-12-13 Thread Noah Misch
On Tue, Dec 13, 2011 at 01:09:46PM -0300, Alvaro Herrera wrote: Excerpts from Noah Misch's message of mar dic 13 11:44:49 -0300 2011: On Mon, Dec 12, 2011 at 05:20:39PM -0300, Alvaro Herrera wrote: Excerpts from Noah Misch's message of dom dic 04 09:20:27 -0300 2011: Second, I

Re: [HACKERS] foreign key locks, 2nd attempt

2011-12-13 Thread Alvaro Herrera
Excerpts from Noah Misch's message of dom dic 04 09:20:27 -0300 2011: +/* + * If the tuple we're updating is locked, we need to preserve this in the + * new tuple's Xmax as well as in the old tuple. Prepare the new xmax + * value for these uses. + * + *

Re: [HACKERS] foreign key locks, 2nd attempt

2011-12-12 Thread Alvaro Herrera
Noah, Many thanks for this review. I'm going through items on it; definitely there are serious issues here, as well as minor things that also need fixing. Thanks for all the detail. I'll post an updated patch shortly (probably not today though); in the meantime, this bit: Excerpts from Noah

Re: [HACKERS] foreign key locks, 2nd attempt

2011-12-12 Thread Alvaro Herrera
Excerpts from Alvaro Herrera's message of lun dic 12 17:20:39 -0300 2011: I found that this is caused by mxid_to_string being leaked all over the place :-( I fixed it by making the returned string be a static that's malloced and then freed on the next call. There's still virtsize growth

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-21 Thread Robert Haas
On Sat, Nov 19, 2011 at 10:36 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On Thu, Nov 3, 2011 at 6:12 PM, Alvaro Herrera alvhe...@alvh.no-ip.org wrote: So Noah Misch proposed using the FOR KEY SHARE syntax, and that's what I have implemented here.  (There

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-21 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: On Sat, Nov 19, 2011 at 10:36 AM, Tom Lane t...@sss.pgh.pa.us wrote: It's already the case that RI triggers require access to special executor features that are not accessible at the SQL level.  I don't think the above argument is a compelling reason

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-19 Thread Simon Riggs
On Thu, Nov 10, 2011 at 8:17 PM, Christopher Browne cbbro...@gmail.com wrote: On Sun, Nov 6, 2011 at 2:28 AM, Jeroen Vermeulen j...@xs4all.nl wrote: On 2011-11-04 01:12, Alvaro Herrera wrote: I would like some opinions on the ideas on this patch, and on the patch itself.  If someone wants

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-19 Thread Simon Riggs
On Thu, Nov 3, 2011 at 6:12 PM, Alvaro Herrera alvhe...@alvh.no-ip.org wrote: So Noah Misch proposed using the FOR KEY SHARE syntax, and that's what I have implemented here.  (There was some discussion that instead of inventing new SQL syntax we could pass the necessary lock mode internally

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-19 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Thu, Nov 3, 2011 at 6:12 PM, Alvaro Herrera alvhe...@alvh.no-ip.org wrote: So Noah Misch proposed using the FOR KEY SHARE syntax, and that's what I have implemented here.  (There was some discussion that instead of inventing new SQL syntax we

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-11 Thread David Kerr
On Thu, Nov 10, 2011 at 03:17:59PM -0500, Christopher Browne wrote: - On Sun, Nov 6, 2011 at 2:28 AM, Jeroen Vermeulen j...@xs4all.nl wrote: - On 2011-11-04 01:12, Alvaro Herrera wrote: - - I would like some opinions on the ideas on this patch, and on the patch - itself.  If someone wants more

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-11 Thread Josh Berkus
An UPDATE that modifies the key columns will be blocked, just as now. Same with a DELETE. OK, so it prevents non-key data modifications from spilling to the referred rows --- nice. Yes. Eliminates the leading cause of deadlocks in Postgres applications. -- Josh Berkus PostgreSQL Experts

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-11 Thread Jeroen Vermeulen
On 2011-11-12 00:30, David Kerr wrote: Is this being suggested in lieu of Alvaro's patch? because it seems to be adding complexity to the system (multiple types of primary key definitions) instead of just fixing an obvious problem (over-aggressive locking done on FK checks). It wouldn't be a

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-10 Thread Bruce Momjian
Alvaro Herrera wrote: Hello, After some rather extensive rewriting, I submit the patch to improve foreign key locks. To recap, the point of this patch is to introduce a new lock tuple mode, that lets the RI code obtain a lighter lock on tuples, which doesn't conflict with updates that do

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-10 Thread Christopher Browne
On Sun, Nov 6, 2011 at 2:28 AM, Jeroen Vermeulen j...@xs4all.nl wrote: On 2011-11-04 01:12, Alvaro Herrera wrote: I would like some opinions on the ideas on this patch, and on the patch itself.  If someone wants more discussion on implementation details of each part of the patch, I'm happy to

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-10 Thread Pavel Stehule
2011/11/10 Christopher Browne cbbro...@gmail.com: On Sun, Nov 6, 2011 at 2:28 AM, Jeroen Vermeulen j...@xs4all.nl wrote: On 2011-11-04 01:12, Alvaro Herrera wrote: I would like some opinions on the ideas on this patch, and on the patch itself.  If someone wants more discussion on

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-10 Thread Kevin Grittner
Christopher Browne cbbro...@gmail.com wrote: There's value in having an immutability constraint on a column, where, in effect, you're not allowed to modify the value of the column, once assigned. +1 We would definitely use such a feature, should it become available. -Kevin -- Sent via

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-10 Thread Christopher Browne
On Thu, Nov 10, 2011 at 3:29 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Christopher Browne cbbro...@gmail.com wrote: There's value in having an immutability constraint on a column, where, in effect, you're not allowed to modify the value of the column, once assigned. +1  We would

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-10 Thread Alvaro Herrera
Excerpts from Bruce Momjian's message of jue nov 10 16:59:20 -0300 2011: Alvaro Herrera wrote: Hello, After some rather extensive rewriting, I submit the patch to improve foreign key locks. To recap, the point of this patch is to introduce a new lock tuple mode, that lets the RI

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-10 Thread Bruce Momjian
Alvaro Herrera wrote: Excerpts from Bruce Momjian's message of jue nov 10 16:59:20 -0300 2011: Alvaro Herrera wrote: Hello, After some rather extensive rewriting, I submit the patch to improve foreign key locks. To recap, the point of this patch is to introduce a new lock

Re: [HACKERS] foreign key locks, 2nd attempt

2011-11-06 Thread Jeroen Vermeulen
On 2011-11-04 01:12, Alvaro Herrera wrote: I would like some opinions on the ideas on this patch, and on the patch itself. If someone wants more discussion on implementation details of each part of the patch, I'm happy to provide a textual description -- please just ask. Jumping in a bit

<    1   2