Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-31 Thread Simon Riggs
On Wed, May 25, 2011 at 3:35 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On Wed, May 25, 2011 at 1:44 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, May 25, 2011 at 8:27 AM, Simon Riggs si...@2ndquadrant.com wrote: Design seemed relatively easy from

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-31 Thread Robert Haas
On Tue, May 31, 2011 at 9:22 AM, Simon Riggs si...@2ndquadrant.com wrote: The basis for this is that weak locks do not conflict with each other, whereas strong locks conflict with both strong and weak locks. (There's a couple of special cases which I ignore for now). (Using Robert's

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-27 Thread Robert Haas
On Tue, May 24, 2011 at 10:03 AM, Noah Misch n...@leadboat.com wrote: On Tue, May 24, 2011 at 08:53:11AM -0400, Robert Haas wrote: On Tue, May 24, 2011 at 5:07 AM, Noah Misch n...@leadboat.com wrote: This drops the part about only transferring fast-path entries once when a strong_lock_counts

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-27 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: When a strong lock is taken or released, we have to increment or decrement strong_lock_counts[fasthashpartition]. Here's the question: is that atomic? In other words, suppose that strong_lock_counts[42] starts out at 0, and two backends both do

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-27 Thread Noah Misch
On Fri, May 27, 2011 at 04:55:07PM -0400, Robert Haas wrote: When a strong lock is taken or released, we have to increment or decrement strong_lock_counts[fasthashpartition]. Here's the question: is that atomic? In other words, suppose that strong_lock_counts[42] starts out at 0, and two

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-25 Thread Simon Riggs
On Tue, May 24, 2011 at 6:37 PM, Robert Haas robertmh...@gmail.com wrote: That being said, it's a slight extra cost for all fast-path lockers to benefit the strong lockers, so I'm not prepared to guess whether it will pay off. Yeah.  Basically this entire idea is about trying to make life

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-25 Thread Robert Haas
On Wed, May 25, 2011 at 8:27 AM, Simon Riggs si...@2ndquadrant.com wrote: I got a bit lost with the description of a potential solution. It seemed like you were unaware that there is a local lock and a shared lock table, maybe just me? No, I'm not unaware of the local lock table. The point of

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-25 Thread Simon Riggs
On Wed, May 25, 2011 at 1:44 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, May 25, 2011 at 8:27 AM, Simon Riggs si...@2ndquadrant.com wrote: I got a bit lost with the description of a potential solution. It seemed like you were unaware that there is a local lock and a shared lock table,

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-25 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Wed, May 25, 2011 at 1:44 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, May 25, 2011 at 8:27 AM, Simon Riggs si...@2ndquadrant.com wrote: Design seemed relatively easy from there: put local lock table in shared memory for all procs. We then

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-25 Thread Robert Haas
On Wed, May 25, 2011 at 8:56 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, May 25, 2011 at 1:44 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, May 25, 2011 at 8:27 AM, Simon Riggs si...@2ndquadrant.com wrote: I got a bit lost with the description of a potential solution. It seemed

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-25 Thread Bruce Momjian
Simon Riggs wrote: On Tue, May 24, 2011 at 6:37 PM, Robert Haas robertmh...@gmail.com wrote: That being said, it's a slight extra cost for all fast-path lockers to benefit the strong lockers, so I'm not prepared to guess whether it will pay off. Yeah. ?Basically this entire idea is

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-24 Thread Noah Misch
On Mon, May 23, 2011 at 09:15:27PM -0400, Robert Haas wrote: On Fri, May 13, 2011 at 4:16 PM, Noah Misch n...@leadboat.com wrote: ? ? ? ?if (level = ShareUpdateExclusiveLock) ? ? ? ? ? ? ? ?++strong_lock_counts[my_strong_lock_count_partition] ? ? ? ? ? ? ? ?sfence ? ? ? ? ? ? ? ?if

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-24 Thread Robert Haas
On Tue, May 24, 2011 at 5:07 AM, Noah Misch n...@leadboat.com wrote: This drops the part about only transferring fast-path entries once when a strong_lock_counts cell transitions from zero to one. Right: that's because I don't think that's what we want to do. I don't think we want to transfer

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-24 Thread Noah Misch
On Tue, May 24, 2011 at 08:53:11AM -0400, Robert Haas wrote: On Tue, May 24, 2011 at 5:07 AM, Noah Misch n...@leadboat.com wrote: This drops the part about only transferring fast-path entries once when a strong_lock_counts cell transitions from zero to one. Right: that's because I don't

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-24 Thread Robert Haas
On Tue, May 24, 2011 at 10:03 AM, Noah Misch n...@leadboat.com wrote: Let's see if I understand the risk better now: the new system will handle lock load better, but when it does hit a limit, understanding why that happened will be more difficult.  Good point.  No silver-bullet ideas come to

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-24 Thread Noah Misch
On Tue, May 24, 2011 at 10:35:23AM -0400, Robert Haas wrote: On Tue, May 24, 2011 at 10:03 AM, Noah Misch n...@leadboat.com wrote: Let's see if I understand the risk better now: the new system will handle lock load better, but when it does hit a limit, understanding why that happened

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-24 Thread Robert Haas
On Tue, May 24, 2011 at 11:38 AM, Noah Misch n...@leadboat.com wrote: Another random idea for optimization: we could have a lock-free array with one entry per backend, indicating whether any fast-path locks are present.  Before acquiring its first fast-path lock, a backend writes a 1 into that

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-24 Thread Noah Misch
On Tue, May 24, 2011 at 11:52:54AM -0400, Robert Haas wrote: On Tue, May 24, 2011 at 11:38 AM, Noah Misch n...@leadboat.com wrote: Another random idea for optimization: we could have a lock-free array with one entry per backend, indicating whether any fast-path locks are present. ?Before

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-24 Thread Robert Haas
On Tue, May 24, 2011 at 12:34 PM, Noah Misch n...@leadboat.com wrote: There's a potentially-unbounded delay between when the subject backend reads strong_lock_counts[] and when it sets its fast-path-used flag.  (I didn't mean not yet seen in the sense that some memory load would not show the

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-23 Thread Robert Haas
On Fri, May 13, 2011 at 4:16 PM, Noah Misch n...@leadboat.com wrote:        if (level = ShareUpdateExclusiveLock)                ++strong_lock_counts[my_strong_lock_count_partition]                sfence                if (strong_lock_counts[my_strong_lock_count_partition] == 1)              

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-15 Thread Robert Haas
On Sat, May 14, 2011 at 1:33 PM, Jeff Janes jeff.ja...@gmail.com wrote: Would that risk be substantially worse than it currently is?  If a backend goes into the tank while holding access shared locks, it will still block access exclusive locks until it recovers.  And those queued access

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-14 Thread Robert Haas
On Fri, May 13, 2011 at 11:05 PM, Noah Misch n...@leadboat.com wrote: Incidentally, I used the term local lock because I assumed fast-path locks would still go through the lock manager far enough to populate the local lock table.  But there may be no reason to do so. Oh, good point. I think

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-14 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, May 13, 2011 at 11:05 PM, Noah Misch n...@leadboat.com wrote: Incidentally, I used the term local lock because I assumed fast-path locks would still go through the lock manager far enough to populate the local lock table. But there may be no

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-14 Thread Jeff Janes
On Fri, May 13, 2011 at 5:55 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, May 13, 2011 at 4:16 PM, Noah Misch n...@leadboat.com wrote: I wonder if, instead, we could signal all backends at marker 1 to dump the applicable parts of their local (memory) lock tables to files.  Or to

[HACKERS] Reducing overhead of frequent table locks

2011-05-13 Thread Noah Misch
On Fri, May 13, 2011 at 09:07:34AM -0400, Robert Haas wrote: Actually, it's occurred to me from time to time that it would be nice to eliminate ACCESS SHARE (and while I'm dreaming, maybe ROW SHARE and ROW EXCLUSIVE) locks for tables as well. Under normal operating conditions (i.e. no DDL

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-13 Thread Robert Haas
On Fri, May 13, 2011 at 4:16 PM, Noah Misch n...@leadboat.com wrote: The key is putting a rapid hard stop to all fast-path lock acquisitions and then reconstructing a valid global picture of the affected lock table regions. Your 1024-way table of strong lock counts sounds promising.  (Offhand,

Re: [HACKERS] Reducing overhead of frequent table locks

2011-05-13 Thread Noah Misch
On Fri, May 13, 2011 at 08:55:34PM -0400, Robert Haas wrote: On Fri, May 13, 2011 at 4:16 PM, Noah Misch n...@leadboat.com wrote: If I'm understanding correctly, your pseudocode would look roughly like this: ? ? ? ?if (level = ShareUpdateExclusiveLock) I think ShareUpdateExclusiveLock