Re: [HACKERS] group locking: incomplete patch, just for discussion

2015-07-28 Thread Robert Haas
On Wed, Nov 5, 2014 at 9:26 PM, Robert Haas robertmh...@gmail.com wrote: On Sun, Nov 2, 2014 at 7:31 AM, Simon Riggs si...@2ndquadrant.com wrote: The procgloballist stuff should be the subject of a separate patch which I agree with. Yes, I think that's probably a net improvement in robustness

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-12-25 Thread Noah Misch
On Thu, Nov 20, 2014 at 09:15:51PM +0100, Andres Freund wrote: On 2014-11-20 13:57:25 -0500, Robert Haas wrote: On Thu, Nov 20, 2014 at 12:19 PM, Andres Freund and...@2ndquadrant.com wrote: If the deadlock detector would kill the processes anyway, it doesn't matter, because

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-12-09 Thread Peter Geoghegan
On Sun, Nov 23, 2014 at 3:59 PM, Andres Freund and...@2ndquadrant.com wrote: The heavyweight locking issue is really killing me, though. I don't really understand why you're not content with just detecting deadlocks for now. Everything else seems like bells and whistles for later. I

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-24 Thread Robert Haas
On Thu, Nov 20, 2014 at 4:45 PM, Andres Freund and...@2ndquadrant.com wrote: How about a frontend process having created a relation that then starts a parallel query. Then the frontend process ERRORs out and, in the course of that, does smgrDoPendingDeletes(). Which will delete the new

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-23 Thread Andres Freund
On 2014-11-21 08:31:01 -0500, Robert Haas wrote: On Thu, Nov 20, 2014 at 8:31 PM, Andres Freund and...@2ndquadrant.com wrote: I can't follow that logic. Why do you think self-exclusive locks are the only ones that matter? All the others can be acquired by jumping in front of the lock's

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-21 Thread Robert Haas
On Thu, Nov 20, 2014 at 8:31 PM, Andres Freund and...@2ndquadrant.com wrote: I can't follow that logic. Why do you think self-exclusive locks are the only ones that matter? All the others can be acquired by jumping in front of the lock's waitqueue? That's is true, as a practical matter, in

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Jeff Davis
On Wed, 2014-11-19 at 11:03 -0500, Robert Haas wrote: But your proposal does not solve this problem: 1. Backend A-1 acquires AccessExclusiveLock on relation R. 2. Backend A-2 waits for AccessShareLock on relation R. The good news is that the deadlock detector should realize that since A-1

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Robert Haas
On Thu, Nov 20, 2014 at 4:21 AM, Jeff Davis pg...@j-davis.com wrote: The bad news, to borrow a phrase from Peter Geoghegan, is that it's an unprincipled deadlock; a user confronted with the news that her parallel scan has self-deadlocked will be justifiably dismayed. You seem to be raising

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Jeff Davis
On Thu, 2014-11-20 at 11:22 -0500, Robert Haas wrote: 2. Propagate pre-existing locks from the user backend to all the workers. I initially proposed #1, but now I think #2 solves more of the problems for less code. OK. The primary concern with that is unintended consequences. But it's

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Andres Freund
On 2014-11-20 11:22:37 -0500, Robert Haas wrote: On Thu, Nov 20, 2014 at 4:21 AM, Jeff Davis pg...@j-davis.com wrote: The bad news, to borrow a phrase from Peter Geoghegan, is that it's an unprincipled deadlock; a user confronted with the news that her parallel scan has self-deadlocked

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Robert Haas
On Thu, Nov 20, 2014 at 12:12 PM, Jeff Davis pg...@j-davis.com wrote: On Thu, 2014-11-20 at 11:22 -0500, Robert Haas wrote: 2. Propagate pre-existing locks from the user backend to all the workers. I initially proposed #1, but now I think #2 solves more of the problems for less code. OK.

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Robert Haas
On Thu, Nov 20, 2014 at 12:19 PM, Andres Freund and...@2ndquadrant.com wrote: Except that it opens us up for all kinds of concurrency bugs. I'm pretty strictly set against granting any self exclusive locks en-masse. If we do this by default for all granted locks when starting a worker backend

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Andres Freund
On 2014-11-20 13:57:25 -0500, Robert Haas wrote: On Thu, Nov 20, 2014 at 12:19 PM, Andres Freund and...@2ndquadrant.com wrote: Except that it opens us up for all kinds of concurrency bugs. I'm pretty strictly set against granting any self exclusive locks en-masse. If we do this by default

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Robert Haas
On Thu, Nov 20, 2014 at 3:15 PM, Andres Freund and...@2ndquadrant.com wrote: Haven't I responded to this a few times already? Not in a particularly convincing way at least. That's not a very helpful response. I see no way, even theoretically, that it can be sane for CheckTableNotInUse() to

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Andres Freund
On 2014-11-20 15:47:39 -0500, Robert Haas wrote: On Thu, Nov 20, 2014 at 3:15 PM, Andres Freund and...@2ndquadrant.com wrote: Haven't I responded to this a few times already? Not in a particularly convincing way at least. That's not a very helpful response. Well... I see no way, even

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Robert Haas
On Thu, Nov 20, 2014 at 4:45 PM, Andres Freund and...@2ndquadrant.com wrote: And you haven't answered how you'd like to fix that. The only answer you've given so far, that I can see, is that maybe we can foresee all the locks that the child might take and not initiate parallelism if any of

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-20 Thread Andres Freund
On 2014-11-20 20:21:07 -0500, Robert Haas wrote: On Thu, Nov 20, 2014 at 4:45 PM, Andres Freund and...@2ndquadrant.com wrote: And you haven't answered how you'd like to fix that. The only answer you've given so far, that I can see, is that maybe we can foresee all the locks that the child

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-19 Thread Robert Haas
On Tue, Nov 18, 2014 at 4:40 AM, Jeff Davis pg...@j-davis.com wrote: To reiterate the basic problem here, if we do nothing at all about the lock manager, a parallel backend can stall trying to grab an AccessExclusiveLock that the user backend alread holds, either because the user backend holds

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-19 Thread Amit Kapila
On Wed, Nov 19, 2014 at 9:33 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Nov 18, 2014 at 4:40 AM, Jeff Davis pg...@j-davis.com wrote: To reiterate the basic problem here, if we do nothing at all about the lock manager, a parallel backend can stall trying to grab an

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-18 Thread Amit Kapila
On Tue, Nov 18, 2014 at 3:10 PM, Jeff Davis pg...@j-davis.com wrote: On Mon, 2014-11-17 at 14:32 -0500, Robert Haas wrote: To reiterate the basic problem here, if we do nothing at all about the lock manager, a parallel backend can stall trying to grab an AccessExclusiveLock that the user

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-17 Thread Robert Haas
On Thu, Nov 13, 2014 at 4:57 PM, Jeff Davis pg...@j-davis.com wrote: On Thu, Nov 13, 2014 at 11:26 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Nov 13, 2014 at 3:38 AM, Jeff Davis pg...@j-davis.com wrote: If two backends both have an exclusive lock on the relation for a join

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-13 Thread Jeff Davis
Great discussion. Robert, I think you make a lot of very valid points, but philosophically I am in pretty strong agreement with Andres here. On Fri, 2014-10-31 at 14:29 -0400, Robert Haas wrote: So just refusing parallelism as soon as anything has taken an access exclusive lock doesn't sound

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-13 Thread Robert Haas
On Thu, Nov 13, 2014 at 2:26 AM, Jeff Davis pg...@j-davis.com wrote: The current deadlock detector would see a dependency graph like: A2 - B - A1 But with lock groups, it would see: (A1 A2) - B - (A1 A2) which is a cycle, and can be detected regardless of the synchronization method

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-13 Thread Robert Haas
On Thu, Nov 13, 2014 at 3:38 AM, Jeff Davis pg...@j-davis.com wrote: If two backends both have an exclusive lock on the relation for a join operation, that implies that they need to do their own synchronization, because obviously the lock manager is not doing it for them. This doesn't make

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-13 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Nov 13, 2014 at 3:38 AM, Jeff Davis pg...@j-davis.com wrote: If two backends both have an exclusive lock on the relation for a join operation, that implies that they need to do their own synchronization, because obviously the lock manager is

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-13 Thread Andres Freund
On November 13, 2014 8:50:18 PM CET, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Nov 13, 2014 at 3:38 AM, Jeff Davis pg...@j-davis.com wrote: If two backends both have an exclusive lock on the relation for a join operation, that implies that they need to

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-13 Thread Jeff Davis
On Thu, Nov 13, 2014 at 11:26 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Nov 13, 2014 at 3:38 AM, Jeff Davis pg...@j-davis.com wrote: If two backends both have an exclusive lock on the relation for a join operation, that implies that they need to do their own synchronization,

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-12 Thread Robert Haas
On Wed, Nov 12, 2014 at 2:57 AM, Jeff Davis pg...@j-davis.com wrote: Trying to catch up on this thread, please excuse me if these questions are already covered. Welcome to the party. The more, the merrier. :-) You mention the possibility of undetected deadlocks, which is surely

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-12 Thread Jeff Davis
On Wed, 2014-11-12 at 14:16 -0500, Robert Haas wrote: Detected deadlocks are fine. Improving the deadlock detector is the heart of what needs to be done here. OK, great. As you say, the lock requests we're talking about will rarely wait, so deadlocks won't be frequent. The issue is making

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-11 Thread Jeff Davis
On Wed, 2014-10-15 at 00:03 -0400, Robert Haas wrote: For parallelism, I think we need a concept of group locking. That is, suppose we have a user backend and N worker backends collaborating to execute some query. For the sake of argument, let's say it's a parallel CLUSTER, requiring a full

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-10 Thread Amit Kapila
On Fri, Nov 7, 2014 at 3:55 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Oct 31, 2014 at 9:07 AM, Andres Freund and...@2ndquadrant.com wrote: Maybe we can, as a first step, make those edges in the lock graph visible to the deadlock detector? It's pretty clear that undetected

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-10 Thread Robert Haas
On Mon, Nov 10, 2014 at 6:33 AM, Amit Kapila amit.kapil...@gmail.com wrote: I think this logic can sometimes block processes from acquiring a lock which no one is holding. Assume Group-1 (G-1) is waiting on two locks (Lock L1 on Table T1 and Lock L2 on Table T2) which are held by unrelated

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-06 Thread Robert Haas
On Wed, Nov 5, 2014 at 9:26 PM, Robert Haas robertmh...@gmail.com wrote: Yes, I think that's probably a net improvement in robustness quite apart from what we decide to do about any of the rest of this. I've attached it here as revise-procglobal-tracking.patch and will commit that bit if

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-06 Thread Jim Nasby
On 11/5/14, 8:26 PM, Robert Haas wrote: rhaas=# create table foo (a int); CREATE TABLE rhaas=# select test_group_locking('1.0:start,2.0:start,1.0:lock:AccessExclusiveLock:foo,2.0:lock:AccessExclusiveLock:foo'); NOTICE: starting worker 1.0 NOTICE: starting worker 2.0 NOTICE: instructing

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-06 Thread Robert Haas
On Fri, Oct 31, 2014 at 9:07 AM, Andres Freund and...@2ndquadrant.com wrote: Maybe we can, as a first step, make those edges in the lock graph visible to the deadlock detector? It's pretty clear that undetected deadlocks aren't ok, but detectable deadlocks in a couple corner cases might be

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-05 Thread Robert Haas
On Sun, Nov 2, 2014 at 7:31 AM, Simon Riggs si...@2ndquadrant.com wrote: The procgloballist stuff should be the subject of a separate patch which I agree with. Yes, I think that's probably a net improvement in robustness quite apart from what we decide to do about any of the rest of this. I've

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-03 Thread Greg Stark
On Sat, Nov 1, 2014 at 9:09 PM, Robert Haas robertmh...@gmail.com wrote: 1. Any non-trivial piece of PostgreSQL code is likely to contain syscache lookups. 2. Syscache lookups had better work in parallel workers, or they'll be all but useless. I've been using parallel sorts and index builds

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-03 Thread Tom Lane
Greg Stark st...@mit.edu writes: On Sat, Nov 1, 2014 at 9:09 PM, Robert Haas robertmh...@gmail.com wrote: 2. Syscache lookups had better work in parallel workers, or they'll be all but useless. I've been using parallel sorts and index builds in my mental model of how this will be used. I

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-03 Thread Robert Haas
On Mon, Nov 3, 2014 at 10:18 AM, Greg Stark st...@mit.edu wrote: On Sat, Nov 1, 2014 at 9:09 PM, Robert Haas robertmh...@gmail.com wrote: 1. Any non-trivial piece of PostgreSQL code is likely to contain syscache lookups. 2. Syscache lookups had better work in parallel workers, or they'll be

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-03 Thread Robert Haas
On Sun, Nov 2, 2014 at 7:31 AM, Simon Riggs si...@2ndquadrant.com wrote: The main question is still why any of this is necessary? If we are only acquiring fast path locks in workers, what need is there for any of this? The leader and workers never conflict with each other, so why would they

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-03 Thread Simon Riggs
On 3 November 2014 17:00, Robert Haas robertmh...@gmail.com wrote: On Sun, Nov 2, 2014 at 7:31 AM, Simon Riggs si...@2ndquadrant.com wrote: The main question is still why any of this is necessary? If we are only acquiring fast path locks in workers, what need is there for any of this? The

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-03 Thread Robert Haas
On Mon, Nov 3, 2014 at 1:02 PM, Simon Riggs si...@2ndquadrant.com wrote: OK, I think I'm happy with this as a general point. Cool! How will we automatically test this code? Good question. I can see two possible approaches: 1. We write a test_group_locking harness along the lines of

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-02 Thread Simon Riggs
On 15 October 2014 05:03, Robert Haas robertmh...@gmail.com wrote: At least to me, that simple scenario is clear-cut[1], but what do we do in more complicated situations? For example, suppose backends A and B are members of the same locking group. A locks a relation with AccessShareLock, an

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-02 Thread Simon Riggs
On 2 November 2014 10:39, Simon Riggs si...@2ndquadrant.com wrote: On 15 October 2014 05:03, Robert Haas robertmh...@gmail.com wrote: At least to me, that simple scenario is clear-cut[1], but what do we do in more complicated situations? For example, suppose backends A and B are members of

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-01 Thread Robert Haas
On Sat, Nov 1, 2014 at 1:55 PM, Andres Freund and...@2ndquadrant.com wrote: Where will those preexisting cache entries come from, exactly? The postmaster is forking the parallel worker, not the user backend. Several things: 1) The relcache init files load a fair bit 2) There's cache entries

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-01 Thread Robert Haas
On Sat, Nov 1, 2014 at 1:55 PM, Andres Freund and...@2ndquadrant.com wrote: I doubt it. There's a whole bunch of problems that just exist by virtue of having a group leader. What if the leader dies but the worker backend isn't in a section of code that does a CHECK_FOR_INTERRUPTS()? In between

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-01 Thread Andres Freund
On 2014-11-01 17:00:59 -0400, Robert Haas wrote: On Sat, Nov 1, 2014 at 1:55 PM, Andres Freund and...@2ndquadrant.com wrote: I doubt it. There's a whole bunch of problems that just exist by virtue of having a group leader. What if the leader dies but the worker backend isn't in a section of

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-01 Thread Robert Haas
On Sat, Nov 1, 2014 at 5:06 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-11-01 17:00:59 -0400, Robert Haas wrote: On Sat, Nov 1, 2014 at 1:55 PM, Andres Freund and...@2ndquadrant.com wrote: I doubt it. There's a whole bunch of problems that just exist by virtue of having a group

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-01 Thread Robert Haas
On Sat, Nov 1, 2014 at 1:09 AM, Simon Riggs si...@2ndquadrant.com wrote: What are the specific restrictions you are suggesting we place on parallel workers? We need that definition clear so we can decide how to mark the functions. If we don't know, which is easily understandable, then the best

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-01 Thread Andres Freund
On 2014-11-01 16:59:34 -0400, Robert Haas wrote: I completely fail to see why you'd generally think it's safe for two backends to hold the same self conflicting lock. Yes, within carefully restricted sections of code that might be doable. But I don't think you can fully enforce that.

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-11-01 Thread Robert Haas
On Sat, Nov 1, 2014 at 5:38 PM, Andres Freund and...@2ndquadrant.com wrote: That's just a mislabeled function. It's obviously not parallel safe at all. I see absolutely no problem with erroring out. I disagree. It's entirely parallel-safe, as long as you don't arbitrarily decide to have the

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Simon Riggs
On 31 October 2014 04:42, Amit Kapila amit.kapil...@gmail.com wrote: In fact it would be more sensible to lock the toast table earlier. It might make some sense to lock the toast table earlier for this particular case, but I don't think in general it will be feasible to lock all the tables

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Robert Haas
On Fri, Oct 31, 2014 at 6:41 AM, Simon Riggs si...@2ndquadrant.com wrote: Is it genuinely required for most parallel operations? I think it's clear that none of us knows the answer. Sure, the general case needs it, but is the general case the same thing as the reasonably common case? Well, I

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Andres Freund
On 2014-10-31 08:54:54 -0400, Robert Haas wrote: On Fri, Oct 31, 2014 at 6:41 AM, Simon Riggs si...@2ndquadrant.com wrote: Is it genuinely required for most parallel operations? I think it's clear that none of us knows the answer. Sure, the general case needs it, but is the general case the

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Robert Haas
On Fri, Oct 31, 2014 at 9:07 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-31 08:54:54 -0400, Robert Haas wrote: On Fri, Oct 31, 2014 at 6:41 AM, Simon Riggs si...@2ndquadrant.com wrote: Is it genuinely required for most parallel operations? I think it's clear that none of us

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Andres Freund
On 2014-10-31 10:08:59 -0400, Robert Haas wrote: On Fri, Oct 31, 2014 at 9:07 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-10-31 08:54:54 -0400, Robert Haas wrote: On Fri, Oct 31, 2014 at 6:41 AM, Simon Riggs si...@2ndquadrant.com wrote: Is it genuinely required for most

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Simon Riggs
On 31 October 2014 12:54, Robert Haas robertmh...@gmail.com wrote: 1. Turing's theorem being what it is, predicting what catalog tables the child might lock is not necessarily simple. The Pareto principle offers ways to cope with the world's lack of simplicity. You mentioned earlier that

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Robert Haas
On Fri, Oct 31, 2014 at 10:38 AM, Andres Freund and...@2ndquadrant.com wrote: I have serious doubts about the number of cases where it's correct to access relations in a second backend that are exclusively locked. Not so much when that happens for a user issued LOCK statement of course, but

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Simon Riggs
On 31 October 2014 18:29, Robert Haas robertmh...@gmail.com wrote: Suppose somebody fires off a parallel sort on a text column, or a parallel sequential scan involving a qual of the form textcol = 'zog'. We launch a bunch of workers to do comparisons; they do lookups against pg_collation.

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Robert Haas
On Fri, Oct 31, 2014 at 11:02 AM, Simon Riggs si...@2ndquadrant.com wrote: You mentioned earlier that functions would need to be marked proisparallel etc.. What conditions will that be protecting against? If we aren't going to support the general case where every single thing works, can we

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Andres Freund
On 2014-10-31 14:29:57 -0400, Robert Haas wrote: On Fri, Oct 31, 2014 at 10:38 AM, Andres Freund and...@2ndquadrant.com wrote: I have serious doubts about the number of cases where it's correct to access relations in a second backend that are exclusively locked. Not so much when that

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Robert Haas
On Fri, Oct 31, 2014 at 2:46 PM, Simon Riggs si...@2ndquadrant.com wrote: On 31 October 2014 18:29, Robert Haas robertmh...@gmail.com wrote: Suppose somebody fires off a parallel sort on a text column, or a parallel sequential scan involving a qual of the form textcol = 'zog'. We launch a

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Robert Haas
On Fri, Oct 31, 2014 at 3:32 PM, Andres Freund and...@2ndquadrant.com wrote: So just refusing parallelism as soon as anything has taken an access exclusive lock doesn't sound too bad to me. That restriction seems onerous to me; for example, it would prevent a parallel sort for CLUSTER or a

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Andres Freund
On 2014-10-31 15:56:42 -0400, Robert Haas wrote: On Fri, Oct 31, 2014 at 3:32 PM, Andres Freund and...@2ndquadrant.com wrote: So just refusing parallelism as soon as anything has taken an access exclusive lock doesn't sound too bad to me. That restriction seems onerous to me; for

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Robert Haas
On Fri, Oct 31, 2014 at 4:10 PM, Andres Freund and...@2ndquadrant.com wrote: I don't think that's correct. We only need to process local invalidation messages after CommandCounterIncrement(), which I anticipate prohibiting during parallel execution (short thought should convince you that

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Simon Riggs
On 31 October 2014 19:36, Robert Haas robertmh...@gmail.com wrote: It's an obscure case and its not the only solution either. I don't think that's an obscure situation at all. Do you really think a patch that could cause an attempt to VACUUM FULL a system catalog to suffer an undetected

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-31 Thread Simon Riggs
On 31 October 2014 18:47, Robert Haas robertmh...@gmail.com wrote: On Fri, Oct 31, 2014 at 11:02 AM, Simon Riggs si...@2ndquadrant.com wrote: You mentioned earlier that functions would need to be marked proisparallel etc.. What conditions will that be protecting against? If we aren't going

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-30 Thread Simon Riggs
On 30 October 2014 04:24, Amit Kapila amit.kapil...@gmail.com wrote: Locking the toast table of any main tables we access seems easily done. Though perhaps we should make weak locking of the toast table presumed. Do we have cases where the toast table can be accessed when the main table is

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-30 Thread Amit Kapila
On Thu, Oct 30, 2014 at 2:28 PM, Simon Riggs si...@2ndquadrant.com wrote: On 30 October 2014 04:24, Amit Kapila amit.kapil...@gmail.com wrote: Locking the toast table of any main tables we access seems easily done. Though perhaps we should make weak locking of the toast table presumed. Do

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-29 Thread Simon Riggs
On 28 October 2014 23:24, Robert Haas robertmh...@gmail.com wrote: You asked for my help, but I'd like to see some concrete steps towards an interim feature so I can see some benefit in a clear direction. Can we please have the first step we discussed? Parallel CREATE INDEX? (Note the

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-29 Thread Amit Kapila
On Wed, Oct 29, 2014 at 2:18 PM, Simon Riggs si...@2ndquadrant.com wrote: My proposal is we do a parallel index build scan... just as we discussed earlier, so you would be following the direction set by Dev Meeting, not just a proposal of mine. As I mentioned previously when you started

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-29 Thread Robert Haas
On Wed, Oct 29, 2014 at 4:48 AM, Simon Riggs si...@2ndquadrant.com wrote: If you do wish to pursue || Seq Scan, then a working prototype would help. It allows us to see that there is an open source solution we are working to solve the problems for. People can benchmark it, understand the

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-29 Thread Simon Riggs
On 29 October 2014 12:08, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Oct 29, 2014 at 2:18 PM, Simon Riggs si...@2ndquadrant.com wrote: My proposal is we do a parallel index build scan... just as we discussed earlier, so you would be following the direction set by Dev Meeting, not just

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-29 Thread Simon Riggs
On 29 October 2014 12:28, Robert Haas robertmh...@gmail.com wrote: I care much more about getting the general infrastructure in place to make parallel programming feasible in PostgreSQL than I do about getting one particular case working. And more than feasible: I want it to be relatively

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-29 Thread Robert Haas
On Wed, Oct 29, 2014 at 10:21 AM, Simon Riggs si...@2ndquadrant.com wrote: There is a real danger that your ta-dah moment sometime in the future contains flaws which need to be addressed, but we now have piles of questionable infrastructure lieing around. If you have similar doubts about

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-29 Thread Simon Riggs
On 29 October 2014 15:43, Robert Haas robertmh...@gmail.com wrote: On Wed, Oct 29, 2014 at 10:21 AM, Simon Riggs si...@2ndquadrant.com wrote: There is a real danger that your ta-dah moment sometime in the future contains flaws which need to be addressed, but we now have piles of questionable

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-29 Thread Amit Kapila
On Wed, Oct 29, 2014 at 7:38 PM, Simon Riggs si...@2ndquadrant.com wrote: On 29 October 2014 12:08, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Oct 29, 2014 at 2:18 PM, Simon Riggs si...@2ndquadrant.com wrote: CREATE INDEX has a lock on the table. Worker tasks can be simply banned

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-28 Thread Simon Riggs
On 16 October 2014 16:22, Robert Haas robertmh...@gmail.com wrote: Might I gently enquire what the something usable we are going to see in this release? I'm not up on current plans. I don't know how far I'm going to get for this release yet. I think pg_background is a pretty good milestone,

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-28 Thread Jim Nasby
On 10/28/14, 3:48 PM, Simon Riggs wrote: Given your description of pg_background it looks an awful lot like infrastructure to make Autonomous Transactions work, but it doesn't even do that. I guess it could do in a very small additional patch, so maybe it is useful for something. What do you

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-28 Thread Robert Haas
On Tue, Oct 28, 2014 at 4:48 PM, Simon Riggs si...@2ndquadrant.com wrote: On 16 October 2014 16:22, Robert Haas robertmh...@gmail.com wrote: Might I gently enquire what the something usable we are going to see in this release? I'm not up on current plans. I don't know how far I'm going to get

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-28 Thread Robert Haas
On Tue, Oct 28, 2014 at 7:22 PM, Jim Nasby jim.na...@bluetreble.com wrote: On 10/28/14, 3:48 PM, Simon Riggs wrote: Given your description of pg_background it looks an awful lot like infrastructure to make Autonomous Transactions work, but it doesn't even do that. I guess it could do in a very

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-28 Thread Robert Haas
On Tue, Oct 28, 2014 at 7:26 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Oct 28, 2014 at 7:22 PM, Jim Nasby jim.na...@bluetreble.com wrote: On 10/28/14, 3:48 PM, Simon Riggs wrote: Given your description of pg_background it looks an awful lot like infrastructure to make Autonomous

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-16 Thread Robert Haas
On Wed, Oct 15, 2014 at 2:55 PM, Simon Riggs si...@2ndquadrant.com wrote: On 15 October 2014 17:03, Robert Haas robertmh...@gmail.com wrote: Well, I'm fervently in agreement with you on one point: the first version of all this needs to be as simple as possible, or the time to get to the first

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-15 Thread Simon Riggs
On 15 October 2014 05:13, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: For parallelism, I think we need a concept of group locking. That is, suppose we have a user backend and N worker backends collaborating to execute some query. For the sake of argument,

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-15 Thread Robert Haas
On Wed, Oct 15, 2014 at 4:18 AM, Simon Riggs si...@2ndquadrant.com wrote: On 15 October 2014 05:13, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: For parallelism, I think we need a concept of group locking. That is, suppose we have a user backend and N worker

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-15 Thread Simon Riggs
On 15 October 2014 14:46, Robert Haas robertmh...@gmail.com wrote: When my family goes to a restaurant, any member of the party may ask for a table and the request is granted for the whole family. But the lock is released only when I pay the bill. Once we have the table, any stragglers know

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-15 Thread Robert Haas
On Wed, Oct 15, 2014 at 10:12 AM, Simon Riggs si...@2ndquadrant.com wrote: On 15 October 2014 14:46, Robert Haas robertmh...@gmail.com wrote: When my family goes to a restaurant, any member of the party may ask for a table and the request is granted for the whole family. But the lock is

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-15 Thread Simon Riggs
On 15 October 2014 17:03, Robert Haas robertmh...@gmail.com wrote: Well, I'm fervently in agreement with you on one point: the first version of all this needs to be as simple as possible, or the time to get to the first version will be longer than we can afford to wait. I think what we're

Re: [HACKERS] group locking: incomplete patch, just for discussion

2014-10-14 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: For parallelism, I think we need a concept of group locking. That is, suppose we have a user backend and N worker backends collaborating to execute some query. For the sake of argument, let's say it's a parallel CLUSTER, requiring a full table lock.