Re: [HACKERS] refresh materialized view concurrently

2013-07-12 Thread Hitoshi Harada
On Tue, Jul 9, 2013 at 12:50 PM, Kevin Grittner kgri...@ymail.com wrote: Thanks again! New patch attached. After a couple of more attempts trying to break it, I mark this as ready to go. One small question: why do we use multiple unique indexes if exist? One index isn't enough? -- Hitoshi

Re: [HACKERS] refresh materialized view concurrently

2013-07-12 Thread Kevin Grittner
Hitoshi Harada umi.tan...@gmail.com wrote: After a couple of more attempts trying to break it, I mark this as ready to go. Thanks. One small question:  why do we use multiple unique indexes if exist?   Two reasons. (1)  By only matching up rows which test as equal on all columns used in

Re: [HACKERS] refresh materialized view concurrently

2013-07-09 Thread Hitoshi Harada
On Sat, Jul 6, 2013 at 9:20 AM, Kevin Grittner kgri...@ymail.com wrote: Hitoshi Harada umi.tan...@gmail.com wrote: Oops! Indeed. Thanks for the careful testing. drop materialized view if exists mv; drop table if exists foo; create table foo(a, b) as values(1, 10); create materialized

Re: [HACKERS] refresh materialized view concurrently

2013-07-03 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Hitoshi Harada umi.tan...@gmail.com wrote: Other than these, I've found index is opened with NoLock, relying on ExclusiveLock of parent matview, and ALTER INDEX SET TABLESPACE or something similar can run concurrently, but it is presumably safe.  DROP

Re: [HACKERS] refresh materialized view concurrently

2013-07-03 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: Robert Haas robertmh...@gmail.com wrote: I doubt very much that this is safe.  And even if it is safe today, I think it's a bad idea, because we're likely to try to reduce lock levels in the future.  Taking no lock on a relation we're opening, even an

Re: [HACKERS] refresh materialized view concurrently

2013-07-03 Thread Robert Haas
On Wed, Jul 3, 2013 at 10:25 AM, Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kgri...@ymail.com writes: Robert Haas robertmh...@gmail.com wrote: I doubt very much that this is safe. And even if it is safe today, I think it's a bad idea, because we're likely to try to reduce lock levels

Re: [HACKERS] refresh materialized view concurrently

2013-07-03 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Jul 3, 2013 at 10:25 AM, Tom Lane t...@sss.pgh.pa.us wrote: I don't believe that that happens. If it does, it's a bug. Either the planner or the executor should be taking a lock on each index touched by a query. It seems Kevin's right. Not

Re: [HACKERS] refresh materialized view concurrently

2013-07-03 Thread Robert Haas
On Wed, Jul 3, 2013 at 10:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Wed, Jul 3, 2013 at 10:25 AM, Tom Lane t...@sss.pgh.pa.us wrote: I don't believe that that happens. If it does, it's a bug. Either the planner or the executor should be taking a

Re: [HACKERS] refresh materialized view concurrently

2013-07-03 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Jul 3, 2013 at 10:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: Are we somehow not going through ExecOpenIndices? I dunno. I just did a quick black-box test: CREATE TABLE foo (a int primary key); BEGIN; INSERT INTO foo VALUES (1); SELECT

Re: [HACKERS] refresh materialized view concurrently

2013-07-03 Thread Andres Freund
On 2013-07-03 11:08:32 -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Wed, Jul 3, 2013 at 10:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: Are we somehow not going through ExecOpenIndices? I dunno. I just did a quick black-box test: CREATE TABLE foo (a int primary

Re: [HACKERS] refresh materialized view concurrently

2013-07-03 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Tom Lane t...@sss.pgh.pa.us wrote: Are we somehow not going through ExecOpenIndices? I dunno.  I just did a quick black-box test: [ begin; insert; without commit ] No foo_pkey anywhere. That proves nothing, as

Re: [HACKERS] refresh materialized view concurrently

2013-07-03 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: OK.  I had seen that no locks were held after the insert and wasn't aware that we acquired and then released them for each insert within a transaction.  On the other hand, we acquire locks on all indexes even for a HOT UPDATE which uses a seqscan, and

Re: [HACKERS] refresh materialized view concurrently

2013-07-03 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kgri...@ymail.com writes: we acquire locks on all indexes even for a HOT UPDATE which uses a seqscan, and hold those until end of transaction.  Is there a reason for that? Sounds dubious to me; although in the HOT code it might be that

Re: [HACKERS] refresh materialized view concurrently

2013-07-03 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: Further testing shows that any UPDATE or DELETE statement acquires a RowExclusiveLock on every index on the table and holds it until end of transaction, whether or not any rows are affected and regardless of whether an index scan or a seqscan is used. 

Re: [HACKERS] refresh materialized view concurrently

2013-07-02 Thread Hitoshi Harada
On Thu, Jun 27, 2013 at 12:19 AM, Hitoshi Harada umi.tan...@gmail.comwrote: On Wed, Jun 26, 2013 at 1:38 AM, Kevin Grittner kgri...@ymail.com wrote: New version attached. Will take another look. Oops! drop materialized view if exists mv; drop table if exists foo; create table foo(a,

Re: [HACKERS] refresh materialized view concurrently

2013-07-02 Thread Robert Haas
On Tue, Jul 2, 2013 at 4:02 AM, Hitoshi Harada umi.tan...@gmail.com wrote: Other than these, I've found index is opened with NoLock, relying on ExclusiveLock of parent matview, and ALTER INDEX SET TABLESPACE or something similar can run concurrently, but it is presumably safe. DROP INDEX,

Re: [HACKERS] refresh materialized view concurrently

2013-06-27 Thread Hitoshi Harada
On Tue, Jun 25, 2013 at 9:07 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Jun 21, 2013 at 5:20 AM, Hitoshi Harada umi.tan...@gmail.com wrote: If I don't miss something, the requirement for the CONCURRENTLY option is to allow simple SELECT reader to read the matview concurrently

Re: [HACKERS] refresh materialized view concurrently

2013-06-27 Thread Hitoshi Harada
On Wed, Jun 26, 2013 at 1:38 AM, Kevin Grittner kgri...@ymail.com wrote: Hitoshi Harada umi.tan...@gmail.com wrote: I spent a few hours to review the patch. Thanks! As far as I can tell, the overall approach is as follows. - create a new temp heap as non-concurrent does, but with

Re: [HACKERS] refresh materialized view concurrently

2013-06-27 Thread Andres Freund
On 2013-06-27 00:12:07 -0700, Hitoshi Harada wrote: Two, until we get MVCC catalog scans, it's not safe to update any system catalog tuple without an AccessExclusiveLock on some locktag that will prevent concurrent catalog scans for that tuple. Under SnapshotNow semantics, concurrent

Re: [HACKERS] refresh materialized view concurrently

2013-06-27 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote: We can play cute tricks akin to what CREATE INDEX CONCURRENTLY currently does, i.e. wait for all other relations that could have possibly seen the old relfilenode (they must have at least a share lock on the relation) before dropping the actual

Re: [HACKERS] refresh materialized view concurrently

2013-06-27 Thread Kevin Grittner
Hitoshi Harada umi.tan...@gmail.com wrote: Kevin Grittner kgri...@ymail.com wrote: Hitoshi Harada umi.tan...@gmail.com wrote: As far as I can tell, the overall approach is as follows. - create a new temp heap as non-concurrent does, but with ExclusiveLock on the matview, so that reader

Re: [HACKERS] refresh materialized view concurrently

2013-06-25 Thread Robert Haas
On Fri, Jun 21, 2013 at 5:20 AM, Hitoshi Harada umi.tan...@gmail.com wrote: If I don't miss something, the requirement for the CONCURRENTLY option is to allow simple SELECT reader to read the matview concurrently while the view is populating the new data, and INSERT/UPDATE/DELETE and SELECT FOR

Re: [HACKERS] refresh materialized view concurrently

2013-06-21 Thread Hitoshi Harada
On Fri, Jun 14, 2013 at 9:05 AM, Kevin Grittner kgri...@ymail.com wrote: Attached is a patch for REFRESH MATERIALIZED VIEW CONCURRENTLY for 9.4 CF1. The goal of this patch is to allow a refresh without interfering with concurrent reads, using transactional semantics. I spent a few hours to

Re: [HACKERS] refresh materialized view concurrently

2013-06-21 Thread Hitoshi Harada
On Fri, Jun 21, 2013 at 2:20 AM, Hitoshi Harada umi.tan...@gmail.comwrote: On Fri, Jun 14, 2013 at 9:05 AM, Kevin Grittner kgri...@ymail.com wrote: Attached is a patch for REFRESH MATERIALIZED VIEW CONCURRENTLY for 9.4 CF1. The goal of this patch is to allow a refresh without interfering

Re: [HACKERS] refresh materialized view concurrently

2013-06-21 Thread Andres Freund
On 2013-06-21 02:43:23 -0700, Hitoshi Harada wrote: On Fri, Jun 21, 2013 at 2:20 AM, Hitoshi Harada umi.tan...@gmail.comwrote: On Fri, Jun 14, 2013 at 9:05 AM, Kevin Grittner kgri...@ymail.com wrote: Attached is a patch for REFRESH MATERIALIZED VIEW CONCURRENTLY for 9.4 CF1. The

Re: [HACKERS] refresh materialized view concurrently

2013-06-17 Thread Simon Riggs
On 17 June 2013 00:43, Kevin Grittner kgri...@ymail.com wrote: Especially when one is known to be better than the other already. What is the hypothetical technique you're arguing is inferior? For my own part, I haven't gotten beyond the phase of knowing that to meet all requests for the

Re: [HACKERS] refresh materialized view concurrently

2013-06-17 Thread Heikki Linnakangas
On 14.06.2013 19:05, Kevin Grittner wrote: Attached is a patch for REFRESH MATERIALIZED VIEW CONCURRENTLY for 9.4 CF1. The goal of this patch is to allow a refresh without interfering with concurrent reads, using transactional semantics. It is my hope to get this committed during this CF to

Re: [HACKERS] refresh materialized view concurrently

2013-06-17 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: There are multiple features all requiring efficient change set extraction. It seems extremely relevant to begin discussing what that mechanism might be in each case Changeset extraction has nothing to do with this patch, and cannot possibly be useful

Re: [HACKERS] refresh materialized view concurrently

2013-06-17 Thread Simon Riggs
On 17 June 2013 12:13, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 14.06.2013 19:05, Kevin Grittner wrote: Attached is a patch for REFRESH MATERIALIZED VIEW CONCURRENTLY for 9.4 CF1. The goal of this patch is to allow a refresh without interfering with concurrent reads, using

Re: [HACKERS] refresh materialized view concurrently

2013-06-17 Thread Simon Riggs
On 17 June 2013 13:15, Kevin Grittner kgri...@ymail.com wrote: Simon Riggs si...@2ndquadrant.com wrote: There are multiple features all requiring efficient change set extraction. It seems extremely relevant to begin discussing what that mechanism might be in each case Changeset extraction

Re: [HACKERS] refresh materialized view concurrently

2013-06-17 Thread Kevin Grittner
Heikki Linnakangas hlinnakan...@vmware.com wrote: On 14.06.2013 19:05, Kevin Grittner wrote: Attached is a patch for REFRESH MATERIALIZED VIEW CONCURRENTLY for 9.4 CF1.  The goal of this patch is to allow a refresh without interfering with concurrent reads, using transactional semantics. It

Re: [HACKERS] refresh materialized view concurrently

2013-06-17 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: Kevin Grittner kgri...@ymail.com wrote: Changeset extraction has nothing to do with this patch, and cannot possibly be useful for it.  Please keep discussion which is completely unrelated to this patch off this thread. You mentioned incremental

Re: [HACKERS] refresh materialized view concurrently

2013-06-17 Thread Nicolas Barbier
2013/6/17 Heikki Linnakangas hlinnakan...@vmware.com: +errmsg(concurrent refresh requires a unique index on just columns for all rows of the materialized view))); Maybe my english is failing me here, but I don’t understand the “just” part. Nicolas -- A.

Re: [HACKERS] refresh materialized view concurrently

2013-06-17 Thread Kevin Grittner
Nicolas Barbier nicolas.barb...@gmail.com wrote: 2013/6/17 Heikki Linnakangas hlinnakan...@vmware.com: +    errmsg(concurrent refresh requires a unique index on just columns for all rows of the materialized view))); Maybe my english is failing me here, but I

Re: [HACKERS] refresh materialized view concurrently

2013-06-17 Thread Robert Haas
On Mon, Jun 17, 2013 at 11:21 AM, Kevin Grittner kgri...@ymail.com wrote: Nicolas Barbier nicolas.barb...@gmail.com wrote: 2013/6/17 Heikki Linnakangas hlinnakan...@vmware.com: +errmsg(concurrent refresh requires a unique index on just columns for all rows of

Re: [HACKERS] refresh materialized view concurrently

2013-06-17 Thread Josh Berkus
On 06/17/2013 04:13 AM, Heikki Linnakangas wrote: On 14.06.2013 19:05, Kevin Grittner wrote: Attached is a patch for REFRESH MATERIALIZED VIEW CONCURRENTLY for 9.4 CF1. The goal of this patch is to allow a refresh without interfering with concurrent reads, using transactional semantics. It

Re: [HACKERS] refresh materialized view concurrently

2013-06-16 Thread Simon Riggs
On 14 June 2013 17:05, Kevin Grittner kgri...@ymail.com wrote: Attached is a patch for REFRESH MATERIALIZED VIEW CONCURRENTLY for 9.4 CF1. The goal of this patch is to allow a refresh without interfering with concurrent reads, using transactional semantics. Is there a reason to keep the

Re: [HACKERS] refresh materialized view concurrently

2013-06-16 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: On 14 June 2013 17:05, Kevin Grittner kgri...@ymail.com wrote: Attached is a patch for REFRESH MATERIALIZED VIEW CONCURRENTLY for 9.4 CF1.  The goal of this patch is to allow a refresh without interfering with concurrent reads, using transactional