Re: [HACKERS] Serializable Isolation without blocking

2010-01-13 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Nope, you're on target. Although - if I were you - I would post the ACCESS EXCLUSIVE lock version of the patch for feedback. I can't speak for anyone else, but I'll read it. Here you go! :-) This is the milestone of having full serializable

Re: [HACKERS] Serializable Isolation without blocking

2010-01-13 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: Here you go! :-) Hmmm... I just got a write skew example to show a snapshot isolation anomaly, so I've got something wrong still. :-( Will continue to work on it. Sorry. -Kevin -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Serializable Isolation without blocking

2010-01-13 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: This is the milestone of having full serializable behavior, albeit with horrible performance, using the simplest implementation possible. A tad too simple, as it turns out. It did it's main job of providing a simple milestone to identify

Re: [HACKERS] Serializable Isolation without blocking

2010-01-09 Thread Markus Wanner
Hi, Kevin Grittner wrote: Nicolas Barbier nicolas.barb...@gmail.com wrote: AFAICS, detecting a rw dependency where the read executes after the write is rather easy: the writer has created a row version that is not visible to the reader's snapshot. Therefore, any time a reader reads a

Re: [HACKERS] Serializable Isolation without blocking

2010-01-09 Thread Nicolas Barbier
2010/1/8 Kevin Grittner kevin.gritt...@wicourts.gov: Nicolas Barbier nicolas.barb...@gmail.com wrote: I assume here that PG's non-SI-compatible behavior of not always rollbacking any transaction that writes to a non-last version will be disabled in serializable mode. Can that currently

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Markus Wanner
Hi, Greg Stark wrote: I think we're still talking past the issue. Predicate locks are not row level, nor page level, nor table level. They're locks on predicates. Ie, you have to lock against values which aren't actually currently in the table at all. You need to be able to detect a conflict

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Markus Wanner
Hi, Kevin Grittner wrote: As I understand it, Greg's line of thinking is that we should use a technique which has never proven practical on a large scale: matching database changes against a list of predicate lock expressions. I find that approach to predicate locking pretty interesting.

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Greg Stark
On Thursday, January 7, 2010, Robert Haas robertmh...@gmail.com wrote: On Thu, Jan 7, 2010 at 2:40 PM, Greg Stark gsst...@mit.edu wrote: On Thu, Jan 7, 2010 at 11:08 AM, Markus Wanner mar...@bluegap.ch wrote: Row level locks are very fine grained, but those are spilled to disk in its current

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Robert Haas
On Fri, Jan 8, 2010 at 7:34 AM, Greg Stark gsst...@mit.edu wrote: On Thursday, January 7, 2010, Robert Haas robertmh...@gmail.com wrote: On Thu, Jan 7, 2010 at 2:40 PM, Greg Stark gsst...@mit.edu wrote: On Thu, Jan 7, 2010 at 11:08 AM, Markus Wanner mar...@bluegap.ch wrote: Row level locks are

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Markus Wanner mar...@bluegap.ch wrote: Greg Stark wrote: That's about predicate locks. I've been talking about SIREAD, which is a different thing (and which I don't consider to be a lock). The SIREAD thingie certainly doesn't help implement predicate locks. And predicate locking isn't

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Markus Wanner mar...@bluegap.ch wrote: Kevin Grittner wrote: As I understand it, Greg's line of thinking is that we should use a technique which has never proven practical on a large scale: matching database changes against a list of predicate lock expressions. I find that approach to

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Greg Stark gsst...@mit.edu wrote: well the one place you *cannot* attach them is on the tuples. The predicate locking schemes I've been reading about do attach locks to tuples, as *part* of a complete strategy. you need to new able to lock hypothetical new tuples which don't exist yet.

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
I had this flagged as needing a response, but it fell through the cracks yesterday. Apologies for the delayed response. Markus Wanner mar...@bluegap.ch wrote: I'm not clear if Kevin plans to go down to tuple level locking with granularity of the SIREAD thing. Eventually, where possible,

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Markus Wanner
Hi, Greg Stark wrote: well the one place you *cannot* attach them is on the tuples. because you need to new able to lock hypothetical new tuples which don't exist yet. Well, maybe attaching here is meant in a more general or theoretical sense. I think we all agree that adding them to the

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Markus Wanner
Hi, Kevin Grittner wrote: SIREAD locks need to be acquired according to the exact same rules as normal read locks in predicate locking schemes. Understood. I didn't take that into account at first. Thanks for pointing it out. (Whatever normal read locks are...) We're just using a lock level

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Markus Wanner
Hi, Kevin Grittner wrote: I had this flagged as needing a response, but it fell through the cracks yesterday. Apologies for the delayed response. No problem. Markus Wanner mar...@bluegap.ch wrote: When the Cahill paper talks about predicate locking, it *is* talking about what to lock with

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Markus Wanner mar...@bluegap.ch wrote: I don't remember reading about predicate locking in the paper I read. Either he didn't cover that in his first implementation (based on page level locking), or I've simply re-used that part of my in-brain-memory. If you read the first paper but not

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Nicolas Barbier
2010/1/8 Markus Wanner mar...@bluegap.ch: SIREAD atop predicate locking serves detecting vulnerable edges (I hope I'm using that term correctly here) between newly inserted tuples and reads, right? I was trying to figure if it would make sense to use predicate locking (instead of table or row

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Nicolas Barbier nicolas.barb...@gmail.com wrote: AFAICS, detecting a rw dependency where the read executes after the write is rather easy: the writer has created a row version that is not visible to the reader's snapshot. Therefore, any time a reader reads a non-last version of a row, there

Re: [HACKERS] Serializable Isolation without blocking

2010-01-08 Thread Kevin Grittner
Again, a somewhat tardy post from a question found in review. Markus Wanner mar...@bluegap.ch wrote: I suppose these more persistent write locks should be kept out of the DEFAULT lock method, too I fail to understand that part. What's the DEFAULT lock method? With some adjustment of

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Markus Wanner
Hi, Kevin Grittner wrote: We're very much on the same page. My goal was to get predicate locking that didn't miss anything, even though it was ridiculously coarse, then implement the simplest possible SSI on top of it, without worrying about optimizations, then incrementally move toward

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Nicolas Barbier
2010/1/7 Markus Wanner mar...@bluegap.ch: (It's interesting that with database page level granularity, he states that predicate locking would not be necessary. Instead any page can be locked at any time. For this to work, according to my reasoning, you'd have to know in advance on which page

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Markus Wanner
Hi, Kevin Grittner wrote: I'm probably not quite as clueless as you think on this; I realize that keeping SIREAD locks in memory will require many more slots for locks, escalation from tuple level to page or coarser when there are many on a table, or (most likely) both. ..oh, there's the

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Markus Wanner
Hi, Nicolas Barbier wrote: The specifics of relation databases can be entirely ignored in case serializability is provided on the page layer level. Aha, I now see very vaguely how that could work, yes. Thank you for elaborating on this. I agree that this isn't the best way forward for

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Kevin Grittner
Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: I think you should have users/kgrittner/postgres.git rather than serializable.git. serializable sounds more like the branch name. I'll wait a bit for other comments before taking any action.

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Magnus Hagander
On Thu, Jan 7, 2010 at 19:08, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: I think you should have users/kgrittner/postgres.git rather than serializable.git.  serializable sounds more like the

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Greg Stark
On Thu, Jan 7, 2010 at 11:08 AM, Markus Wanner mar...@bluegap.ch wrote: Row level locks are very fine grained, but those are spilled to disk in its current implementation. So those are an even worse fit for the needs of SIREAD. I think we're still talking past the issue. Predicate locks are

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Robert Haas
On Thu, Jan 7, 2010 at 2:40 PM, Greg Stark gsst...@mit.edu wrote: On Thu, Jan 7, 2010 at 11:08 AM, Markus Wanner mar...@bluegap.ch wrote: Row level locks are very fine grained, but those are spilled to disk in its current implementation. So those are an even worse fit for the needs of SIREAD.

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Jeff Davis
On Thu, 2010-01-07 at 15:02 -0500, Robert Haas wrote: I think we're still talking past the issue. Predicate locks are not row level, nor page level, nor table level. They're not? They're just floating out in space somewhere? There are several possible ways to implement predicate locks,

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Kevin Grittner
Jeff Davis pg...@j-davis.com wrote: Consider an exclusion constraint, which is a kind of predicate lock. You could say that the lock is in the index (now) -- but my first implementation used a shared memory structure instead, so it's clearly not required to exist in the index. You could also

Re: [HACKERS] Serializable Isolation without blocking

2010-01-07 Thread Jeff Davis
On Thu, 2010-01-07 at 15:01 -0600, Kevin Grittner wrote: As I understand it, Greg's line of thinking is that we should use a technique which has never proven practical on a large scale: matching database changes against a list of predicate lock expressions. It's not that I want to do it any

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: And use git so you can keep merging up to CVS HEAD easily. Regarding this, I was thinking that it would make sense to use a repository on git.postgresql.org to coordinate my work with any other people interested in the project. I would clone from

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Robert Haas
On Wed, Jan 6, 2010 at 4:29 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: And use git so you can keep merging up to CVS HEAD easily. Regarding this, I was thinking that it would make sense to use a repository on git.postgresql.org to

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Dimitri Fontaine
Kevin Grittner kevin.gritt...@wicourts.gov writes: I've read through some git tutorials, but there's a lot to digest and I'm not entirely sure this is a good way to proceed. I found that the following video is really helpful at grasping the concepts of git, that it exposes pretty directly even

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: I think you should have users/kgrittner/postgres.git rather than serializable.git. serializable sounds more like the branch name. Hmmm On a multi-year project it seemed more than remotely possible that responsibility for the project could shift,

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Robert Haas
On Wed, Jan 6, 2010 at 5:04 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: I think you should have users/kgrittner/postgres.git rather than serializable.git.  serializable sounds more like the branch name. Hmmm  On a multi-year project it

Re: [HACKERS] Serializable Isolation without blocking

2010-01-06 Thread Markus Wanner
Hi, Greg Stark wrote: aaah... I think I see where we've gone off track in previous discussions...you think postgres keeps row level locks in a shared memory data structure. It doesn't it stores all row level locks *in* the tuple itself. It only stores the lock in memory briefly while

Re: [HACKERS] Serializable Isolation without blocking

2010-01-03 Thread Kevin Grittner
Jeff Davis wrote: I started a wiki page here: http://wiki.postgresql.org/wiki/Serializable I've filled it in with all relevant information which came to mind. If you can spot something I missed, please feel free to correct that or let me know so that I can. -Kevin -- Sent via

Re: [HACKERS] Serializable Isolation without blocking

2010-01-01 Thread Albe Laurenz
Kevin Grittner wrote: It seems to me that the hard part of this problem is to describe the general mechanism by which conflicts will be detected, with specific references to the types of data structures that will be used to hold that information. Well, the general approach to tracking SIREAD

Re: [HACKERS] Serializable Isolation without blocking

2010-01-01 Thread Kevin Grittner
Albe Laurenz wrote: If I remember right, one necessity for the SIREAD lock technique was that SIREAD locks taken by a transaction have to be kept after the transaction has ended. Correct. An SIREAD lock must be held until completion of the last serializable transaction holding a snapshot

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Nicolas Barbier
[ Reviving this old thread because a recent one referred to it. ] 2009/5/7 Albe Laurenz laurenz.a...@wien.gv.at: Kevin Grittner wrote: maybe I misunderstood something. Consider a function makehighlander(personid integer) RETURNS void defined like this:    SELECT ishighlander INTO

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Greg Stark
On Thu, Dec 31, 2009 at 12:45 PM, Nicolas Barbier nicolas.barb...@gmail.com wrote: * To answer your specific question: T0 might not read that specific row, but the COUNT(..) definitely must read *something* that must be modified by T1 when it updates the ishighlander field: The problem occurs

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Greg Stark wrote: The unsolved problems that have been raised are: [legion] Yeah, that's why this is a two to four year project. And I would point out that if there *wasn't* a performance hit in serializable mode, none of the other isolation levels would exist. These less rigorous modes

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Greg Stark
On Thu, Dec 31, 2009 at 3:11 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Yeah, that's why this is a two to four year project.  And I would point out that if there *wasn't* a performance hit in serializable mode, none of the other isolation levels would exist.  These less rigorous

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Greg Stark wrote: Hm, this raises the issue that you'll have to figure out what should happen if two different transactions are using different isolation modes. Currently our two isolation modes only control behaviour within your transaction so they co-exist perfectly fine. ISTM you would

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Robert Haas
On Thu, Dec 31, 2009 at 12:10 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Greg Stark  wrote: Hm, this raises the issue that you'll have to figure out what should happen if two different transactions are using different isolation modes. Currently our two isolation modes only control

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Robert Haas wrote: It seems to me that the hard part of this problem is to describe the general mechanism by which conflicts will be detected, with specific references to the types of data structures that will be used to hold that information. Well, the general approach to tracking SIREAD

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Robert Haas
On Thu, Dec 31, 2009 at 1:43 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas  wrote: It seems to me that the hard part of this problem is to describe the general mechanism by which conflicts will be detected, with specific references to the types of data structures that will

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Jeff Davis
On Thu, 2009-12-31 at 09:11 -0600, Kevin Grittner wrote: Yeah, that's why this is a two to four year project. I started a wiki page here: http://wiki.postgresql.org/wiki/Serializable I didn't add much content yet, but can participants in this discussion please try to organize the various

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Robert Haas wrote: OK. I think it will become more clear whether the existing lock data structures are adequate as you move into detailed design. I've gotten far enough in reviewing it to be pretty convinced that they'll cover all the granularities I'm likely to want unless I get to the

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Jeff Davis wrote: I started a wiki page here: http://wiki.postgresql.org/wiki/Serializable I'll try to get that filled in with something useful over the weekend. I'm heading to a party soon, and may not be in shape to work on it tomorrow ;-) Happy New Year, all! -Kevin --

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Greg Stark
aaah... I think I see where we've gone off track in previous discussions...you think postgres keeps row level locks in a shared memory data structure. It doesn't it stores all row level locks *in* the tuple itself. It only stores the lock in memory briefly while actually acquiring the lock. Once

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Greg Stark wrote: aaah... I think I see where we've gone off track in previous discussions...you think postgres keeps row level locks in a shared memory data structure. It doesn't it stores all row level locks *in* the tuple itself. It only stores the lock in memory briefly while actually

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Robert Haas
On Thu, Dec 31, 2009 at 4:44 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: I wonder if you couldn't start with something ridiculously poor, like maybe an S2PL implementation with only table-level granularity - just make any operation that reads or writes a table grab an ACCESS

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Kevin Grittner
Robert Haas wrote: What predicate locking? If you take ACCESS EXCLUSIVE locks on every read, that should serialize all access to every table. Predicate locking wouldn't do anything, because the table would be completely inaccessible to all competing transactions. Yeah, that's the benefit

Re: [HACKERS] Serializable Isolation without blocking

2009-12-31 Thread Robert Haas
On Thu, Dec 31, 2009 at 7:45 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas  wrote: What predicate locking? If you take ACCESS EXCLUSIVE locks on every read, that should serialize all access to every table. Predicate locking wouldn't do anything, because the table would be

Re: [HACKERS] Serializable Isolation without blocking

2009-05-27 Thread Kevin Grittner
I wrote: Greg Stark st...@enterprisedb.com wrote: This thread has really been one of those cases where everyone thought they were having a different kind of discussion. Apparently so. In light of discussions at PGCon, I'm going to abandon this thread in favor of a discussion of what

Re: [HACKERS] Serializable Isolation without blocking

2009-05-27 Thread Simon Riggs
On Wed, 2009-05-27 at 13:34 -0500, Kevin Grittner wrote: For the record, it became clear that I did a bad job of communicating on this thread... You did good work, IMHO. Not everything will reach consensus and that's not your fault. first implement table level predicate locks, since that

Re: [HACKERS] Serializable Isolation without blocking

2009-05-27 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: Do we need table-level predicate locks at all? What would they give us? Why not just go straight for fine-grained page-level locks? I don't want to get too far into implementation discussions at this phase (see Tom's slides ;-)), but suffice it to

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Albe Laurenz
Kevin Grittner wrote: I still haven't actually read the paper so I should probably bow out from the conversation until I do. I was apparently already under one misapprehension as Laurenz just claimed the paper does not show how to prevent phantoms (phantom reads I assume?). Perhaps it's

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Kevin Grittner
Albe Laurenz laurenz.a...@wien.gv.at wrote: All the authors show with regard to predicate handling is handwaving, That is because predicate locking is a mature technology with many known implementations. The best technique for any database product will depend on that product, and their

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Greg Stark
On Mon, May 11, 2009 at 2:49 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Albe Laurenz laurenz.a...@wien.gv.at wrote: All the authors show with regard to predicate handling is handwaving, That is because predicate locking is a mature technology with many known implementations.  The

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Albe Laurenz
Kevin Grittner wrote: All the authors show with regard to predicate handling is handwaving, That is because predicate locking is a mature technology with many known implementations. The best technique for any database product will depend on that product, and their technique doesn't

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Kevin Grittner
Greg Stark st...@enterprisedb.com wrote: I thought the big problem with providing true serializability was the predicate locking. If it doesn't address that need then does this get us any closer? I thought the big problem was the perception that performance would suffer and that the level

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Greg Stark
On Mon, May 11, 2009 at 3:11 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Greg Stark st...@enterprisedb.com wrote: I thought the big problem with providing true serializability was the predicate locking. If it doesn't address that need then does this get us any closer? I thought

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Kevin Grittner
Albe Laurenz laurenz.a...@wien.gv.at wrote: In my first reply I wondered if the presence of concurrent read committed transactions would somehow affect the correctness of the algorithm, as the authors don't mention that. Yeah, I was concerned about that, too. In thinking it through I've

Re: [HACKERS] Serializable Isolation without blocking

2009-05-11 Thread Kevin Grittner
Greg Stark st...@enterprisedb.com wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: Greg Stark st...@enterprisedb.com wrote: I thought the big problem with providing true serializability was the predicate locking. If it doesn't address that need then does this get us any closer? I

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Albe Laurenz
Kevin Grittner wrote: What if there is an index on the ishighlander row? Then an index scan would find only one candidate to examine, and the other rows would not even be touched by the execution plan. I assume you're talking about this line of your function: SELECT count(*) INTO n

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Kevin Grittner
Greg Stark st...@enterprisedb.com wrote: On Thu, May 7, 2009 at 11:08 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: I would assume that SELECT shown above would either resolve to a table scan, in which case you would have to have an SIREAD lock at the table level That sounds like

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Greg Stark
On Fri, May 8, 2009 at 3:00 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: I've removed the broken email address for now -- please re-add the correct email address. I'll see what I can find.  When I last corresponded with him, he was still at the University of Sidney, working on his

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Greg Stark
On Fri, May 8, 2009 at 3:00 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Greg Stark st...@enterprisedb.com wrote: On Thu, May 7, 2009 at 11:08 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: I would assume that SELECT shown above would either resolve to a table scan, in which

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Kevin Grittner
Albe Laurenz laurenz.a...@wien.gv.at wrote: As far as I know, only the table rows that are found in the index scan are examined for visibility. Which would be only one in my example. S2PL locking schemes routinely use index range locks. But in your attempt to sketch a way how true

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Kevin Grittner
Greg Stark st...@enterprisedb.com wrote: Well I don't understand what storing locks in an index can accomplish if other queries might use other indexes or sequential scans to access the records and never see those locks. Or does this method only require that writers discover the locks and

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Greg Stark
On Fri, May 8, 2009 at 4:12 PM, Greg Stark st...@enterprisedb.com wrote: On Fri, May 8, 2009 at 3:49 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Greg Stark st...@enterprisedb.com wrote: Well I don't understand what storing locks in an index can accomplish if other queries might use

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Tom Lane
Greg Stark st...@enterprisedb.com writes: ... Argh, sorry, as soon as I hit send I realized this is wrong. Writers already need to insert into every index, so that's not a problem. What about HOT? regards, tom lane -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Greg Stark st...@enterprisedb.com writes: ... Argh, sorry, as soon as I hit send I realized this is wrong. Writers already need to insert into every index, so that's not a problem. What about HOT? I think that a read would need to lock both the row or

Re: [HACKERS] Serializable Isolation without blocking

2009-05-08 Thread Greg Stark
On Fri, May 8, 2009 at 3:49 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Greg Stark st...@enterprisedb.com wrote: Well I don't understand what storing locks in an index can accomplish if other queries might use other indexes or sequential scans to access the records and never see

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Albe Laurenz
Kevin Grittner wrote: While discussing potential changes to PostgreSQL documentation of transaction isolation levels, Emmanuel Cecchet pointed out an intriguing new paper[1] on a new algorithm to provide true serializable behavior in a MVCC based database, with no additional blocking;

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Gregory Stark
Albe Laurenz laurenz.a...@wien.gv.at writes: So I think one would have to add intention locks for rows considered in the WHERE clause to guarantee true serializability. Does the paper explain how to deal with rows considered in the WHERE clause which don't yet exist? Ie, SELECT count(*) WHERE

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Simon Riggs
On Tue, 2009-05-05 at 10:50 -0500, Kevin Grittner wrote: This paper describes a modification to the concurrency control algorithm of a database management system that automatically detects and prevents snapshot isolation anomalies at runtime for arbitrary applications, thus providing

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Heikki Linnakangas
Simon Riggs wrote: It wouldn't be 692 lines of code and even if it were the impact of that code would be such that it would need to be optional, since it would differ in definition from an existing SQL Standard isolation mode and it would have additional performance implications. I thought it

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 15:26 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: It wouldn't be 692 lines of code and even if it were the impact of that code would be such that it would need to be optional, since it would differ in definition from an existing SQL Standard isolation mode and

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Albe Laurenz
Greg Stark wrote: So I think one would have to add intention locks for rows considered in the WHERE clause to guarantee true serializability. Does the paper explain how to deal with rows considered in the WHERE clause which don't yet exist? Ie, SELECT count(*) WHERE foo needs to take out a

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Albe Laurenz laurenz.a...@wien.gv.at wrote: maybe I misunderstood something. Consider a function makehighlander(personid integer) RETURNS void defined like this: SELECT ishighlander INTO b FROM scots WHERE id=personid; IF b THEN RETURN; /* no need to do anything */

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Gregory Stark st...@enterprisedb.com wrote: Albe Laurenz laurenz.a...@wien.gv.at writes: So I think one would have to add intention locks for rows considered in the WHERE clause to guarantee true serializability. Does the paper explain how to deal with rows considered in the WHERE clause

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Albe Laurenz
Kevin Grittner wrote: maybe I misunderstood something. Consider a function makehighlander(personid integer) RETURNS void defined like this: SELECT ishighlander INTO b FROM scots WHERE id=personid; IF b THEN RETURN; /* no need to do anything */ END IF;

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Albe Laurenz laurenz.a...@wien.gv.at wrote: Kevin Grittner wrote: I do think you misunderstood. If there are two concurrent executions and each reads one row, there will be an SIREAD lock for each of those rows. As an example, let's say that one of them (T0) updates its row and does its

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Simon Riggs wrote: It wouldn't be 692 lines of code and even if it were the impact of that code would be such that it would need to be optional, since it would differ in definition from an existing SQL Standard isolation mode and

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Albe Laurenz
Kevin Grittner wrote: Where does T1 select rows that were modified by T0? It selects only one row, the one it modified itself, right? You have to select it to know whether to count it, right? We are getting closer. So an SIREAD lock is taken for every row that is examined during the

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Albe Laurenz laurenz.a...@wien.gv.at wrote: What if there is an index on the ishighlander row? Then an index scan would find only one candidate to examine, and the other rows would not even be touched by the execution plan. I assume you're talking about this line of your function:

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: It wouldn't be 692 lines of code Agreed. The original implementation was in an MVCC database which already supported full serializability using strict 2 phase locking and used page level locks. Both of these made the implementation simpler than it

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Please keep Michael Cahill copied on this thread, per his request. I just noticed the omission on a few messages and will forward them to him. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 11:15 -0500, Kevin Grittner wrote: Please keep Michael Cahill copied on this thread, per his request. I just noticed the omission on a few messages and will forward them to him. Apologies Michael, I see that my mail did remove you. That was a unconscious error; I was

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 10:56 -0500, Kevin Grittner wrote: It's clear that any new-theory solution will cost significantly more as the number of users increases, at least O(N^2), whereas simply waiting is only O(N), AFAICS. I'm not following your reasoning on the O(N^2). Could you

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: Each user must compare against work performed by all other users. O(N). There are N users, so O(N^2). Why does that apply here and not in the update conflict detection? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 12:39 -0500, Kevin Grittner wrote: Simon Riggs si...@2ndquadrant.com wrote: Each user must compare against work performed by all other users. O(N). There are N users, so O(N^2). Why does that apply here and not in the update conflict detection? I think the

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2009-05-07 at 12:39 -0500, Kevin Grittner wrote: Simon Riggs si...@2ndquadrant.com wrote: Each user must compare against work performed by all other users. O(N). There are N users, so O(N^2). Why does that apply here and not in the

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 15:10 -0500, Kevin Grittner wrote: The assertion that there is some need for each session to wade through something for every other session seems baseless to me. I'm wondering what I might be missing. That's Greg's point. Do we need full locking of everything we might

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Greg Stark
On Thu, May 7, 2009 at 6:31 PM, Simon Riggs si...@2ndquadrant.com wrote: Each user must compare against work performed by all other users. O(N). There are N users, so O(N^2). i think this logic only works if you must scan every item for every other user every time. If you have data structures

Re: [HACKERS] Serializable Isolation without blocking

2009-05-07 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: Do we need full locking of everything we might touch, or tracking of what we have touched? If you need the might touch then you either need to implement locking that will effect everybody (which ain't ever gonna fly round here), or you implement a

  1   2   >