Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Tom Lane
Pavan Deolasee [EMAIL PROTECTED] writes: How about storing the snapshot which we used during planning in CachedPlanSource, if at least one index was seen unusable because its CREATE INDEX transaction was seen as in-progress ? I'm getting tired of repeating this, but: the planner doesn't use a

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Pavan Deolasee
Tom Lane wrote: I'm getting tired of repeating this, but: the planner doesn't use a snapshot. System catalogs run on SnapshotNow. I am really sorry if I sound foolish here. I am NOT suggesting that we use snapshot to read system catalogs. I understand that system catalogs run on SnapshotNow

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Florian G. Pflug
Tom Lane wrote: Pavan Deolasee [EMAIL PROTECTED] writes: How about storing the snapshot which we used during planning in CachedPlanSource, if at least one index was seen unusable because its CREATE INDEX transaction was seen as in-progress ? I'm getting tired of repeating this, but: the

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Tom Lane
Pavan Deolasee [EMAIL PROTECTED] writes: What I am suggesting is to use ActiveSnapshot (actually Florian's idea) to decide whether the transaction that created index was still running when we started. Isn't it the case that some snapshot will be active when we plan ? I do not think you can

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Florian G. Pflug
Tom Lane wrote: Pavan Deolasee [EMAIL PROTECTED] writes: What I am suggesting is to use ActiveSnapshot (actually Florian's idea) to decide whether the transaction that created index was still running when we started. Isn't it the case that some snapshot will be active when we plan ? I do not

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Pavan Deolasee
On 3/30/07, Florian G. Pflug [EMAIL PROTECTED] wrote: What about doing PREPARE myplan select ... ; outside of a transaction? Will this be execute inside a transaction? I checked that. PREPARE runs with ActiveSnapshot set. Thanks, Pavan -- EnterpriseDB http://www.enterprisedb.com

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Tom Lane
Florian G. Pflug [EMAIL PROTECTED] writes: Tom Lane wrote: I do not think you can assume that the plan won't be used later with some older snapshot. So maybe we'd need to use the SerializableSnapshot created at the start of each transaction for this check, and not the ActiveSnapshot? Could

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Simon Riggs
On Fri, 2007-03-30 at 11:44 +0530, Pavan Deolasee wrote: ISTM that the run-another-transaction-afterwards idea would have same problem with plan invalidation. When the second transaction commits, the relcache invalidation event is generated. The event may get consumed by other backends, but

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Pavan Deolasee
On 3/30/07, Tom Lane [EMAIL PROTECTED] wrote: That might work, but it doesn't seem to address the core objection: there's no mechanism to cause the query to be replanned once the snapshot is new enough, because no relcache inval will happen. So most likely existing backends will keep using

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Florian G. Pflug
Tom Lane wrote: Florian G. Pflug [EMAIL PROTECTED] writes: Tom Lane wrote: I do not think you can assume that the plan won't be used later with some older snapshot. So maybe we'd need to use the SerializableSnapshot created at the start of each transaction for this check, and not the

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Pavan Deolasee
On 3/30/07, Florian G. Pflug [EMAIL PROTECTED] wrote: My idea was to store a list of xid's together with the cached plan that are assumed to be uncommitted accoring to the IndexSnapshot. The query is replanned if upon execution the IndexSnapshot assumes that one of these xid's is committed.

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Florian G. Pflug
Pavan Deolasee wrote: On 3/30/07, Florian G. Pflug [EMAIL PROTECTED] wrote: My idea was to store a list of xid's together with the cached plan that are assumed to be uncommitted accoring to the IndexSnapshot. The query is replanned if upon execution the IndexSnapshot assumes that one of these

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Tom Lane
Pavan Deolasee [EMAIL PROTECTED] writes: Actually, if we are using Serializable Snapshot then there is no chance to replan the query before the transaction completes and the next transaction to start in the session must see the index and hence we must replan. So it would be enough just to

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Simon Riggs
On Fri, 2007-03-30 at 13:54 -0400, Tom Lane wrote: Pavan Deolasee [EMAIL PROTECTED] writes: Actually, if we are using Serializable Snapshot then there is no chance to replan the query before the transaction completes and the next transaction to start in the session must see the index and

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Pavan Deolasee
On 3/31/07, Simon Riggs [EMAIL PROTECTED] wrote: On Fri, 2007-03-30 at 13:54 -0400, Tom Lane wrote: Hm. So anytime we reject a potentially useful index as being not valid yet, we mark the plan as only good for this top-level transaction? That seems possibly workable --- in particular it

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-30 Thread Pavan Deolasee
On 3/30/07, Tom Lane [EMAIL PROTECTED] wrote: I do not think you can assume that the plan won't be used later with some older snapshot. Consider recursive plpgsql functions for a counterexample: the inner occurrence might be the first to arrive at a given line of the function, hence the first

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Florian G. Pflug
Pavan Deolasee wrote: In this specific context, this particular case is easy to handle because we are only concerned about the serializable transactions started before CREATE INDEX commits. If PREPARE can see the new index, it implies that the CI transaction is committed. So the transaction

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Pavan Deolasee
On 3/29/07, Florian G. Pflug [EMAIL PROTECTED] wrote: Yes, but the non-index plan PREPARE generated will be used until the end of the session, nut only until the end of the transaction. Frankly I don't know this works, but are you sure that the plan will be used until the end of the session

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Florian G. Pflug
Pavan Deolasee wrote: On 3/29/07, Florian G. Pflug [EMAIL PROTECTED] wrote: Yes, but the non-index plan PREPARE generated will be used until the end of the session, nut only until the end of the transaction. Frankly I don't know this works, but are you sure that the plan will be used until

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Tom Lane
Florian G. Pflug [EMAIL PROTECTED] writes: Pavan Deolasee wrote: Frankly I don't know this works, but are you sure that the plan will be used until the end of the session ? Even if thats the case, it can happen even today if we create a new index, but the existing sessions will use the stale

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Pavan Deolasee
On 3/29/07, Tom Lane [EMAIL PROTECTED] wrote: It will replan at the first use of the plan after seeing the relcache inval sent by commit of the index-creating transaction. If you have two separate transactions to create an index and then mark it valid later, everything's fine because there

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Gregory Stark
Tom Lane [EMAIL PROTECTED] writes: However, if you design something where an index becomes usable due to the passage of time rather than an explicit mark-valid step, there's gonna be a problem. I'd suggest trying to stick to the way CREATE INDEX CONCURRENTLY does it... I'm a bit skeptical

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Pavan Deolasee
On 3/29/07, Gregory Stark [EMAIL PROTECTED] wrote: Besides, it seems if people are happy to have indexes take a long time to build they could just do a concurrent build. I think we discussed this earlier. One of the down-side of CIC is that it needs two complete heap scans. Apart from that

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Bruce Momjian
Pavan Deolasee wrote: Earlier we were talking about not inserting any HOT tuples until the index became valid. The goal of having an xid on the index was so we would know when we could start doing HOT updates again. That seems like a much lesser cost than not being able to use the index

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Gregory Stark
Pavan Deolasee [EMAIL PROTECTED] writes: What I am proposing is to keep index unusable for existing transactions. The index is available for all new transactions even if there are unfinished existing transactions. Ah thanks, that makes a lot more sense. Sorry for the false alarm. --

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Simon Riggs
On Thu, 2007-03-29 at 22:08 +0530, Pavan Deolasee wrote: On 3/29/07, Tom Lane [EMAIL PROTECTED] wrote: It will replan at the first use of the plan after seeing the relcache inval sent by commit of the index-creating transaction. If you have

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Simon Riggs
On Thu, 2007-03-29 at 13:55 -0400, Bruce Momjian wrote: Pavan Deolasee wrote: Earlier we were talking about not inserting any HOT tuples until the index became valid. The goal of having an xid on the index was so we would know when we could start doing HOT updates again. That seems

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: ISTM that the run-another-transaction-afterwards idea is the only one that does everything I think we need. I really do wish we could put in a wait, like CIC, but I just think it will break existing programs. Actually, there's a showstopper objection to

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Florian G. Pflug
Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: ISTM that the run-another-transaction-afterwards idea is the only one that does everything I think we need. I really do wish we could put in a wait, like CIC, but I just think it will break existing programs. Actually, there's a

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Simon Riggs
On Thu, 2007-03-29 at 17:27 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: ISTM that the run-another-transaction-afterwards idea is the only one that does everything I think we need. I really do wish we could put in a wait, like CIC, but I just think it will break existing

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Florian G. Pflug
Simon Riggs wrote: On Thu, 2007-03-29 at 17:27 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: ISTM that the run-another-transaction-afterwards idea is the only one that does everything I think we need. I really do wish we could put in a wait, like CIC, but I just think it will

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-29 Thread Pavan Deolasee
On 3/30/07, Simon Riggs [EMAIL PROTECTED] wrote: Pavan, ISTM you have misunderstood Tom slightly. Oh, yes. Now that I re-read Tom's comment, his plan invalidation design and code, I understand things better. Having the index invisible to all current transactions is acceptable. Ok.

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-28 Thread Pavan Deolasee
On 3/23/07, Pavan Deolasee [EMAIL PROTECTED] wrote: Its slightly different for the HOT-chains created by this transaction which is creating the index. We should index the latest version of the row which is not yet committed. But thats ok because when CREATE INDEX commits this latest version

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-28 Thread Simon Riggs
On Wed, 2007-03-28 at 22:24 +0530, Pavan Deolasee wrote: Just when I thought we have nailed down CREATE INDEX, I realized that there something more to worry. The problem is with the HOT-chains created by our own transaction which is creating the index. We thought it will be enough to index

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-28 Thread Florian G. Pflug
Simon Riggs wrote: On Wed, 2007-03-28 at 22:24 +0530, Pavan Deolasee wrote: Just when I thought we have nailed down CREATE INDEX, I realized that there something more to worry. The problem is with the HOT-chains created by our own transaction which is creating the index. We thought it will be

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-28 Thread Pavan Deolasee
On 3/28/07, Tom Lane [EMAIL PROTECTED] wrote: Florian G. Pflug [EMAIL PROTECTED] writes: Couldn't you store the creating transaction's xid in pg_index, and let other transaction check that against their snapshot like they would for any tuple's xmin or xmax? What snapshot? I keep having to

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-28 Thread Tom Lane
Florian G. Pflug [EMAIL PROTECTED] writes: Couldn't you store the creating transaction's xid in pg_index, and let other transaction check that against their snapshot like they would for any tuple's xmin or xmax? What snapshot? I keep having to remind people that system catalog operations are

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-28 Thread Pavan Deolasee
On 3/28/07, Simon Riggs [EMAIL PROTECTED] wrote: Set it at the end, not the beginning. At the end of what ? It does not help to set it at the end of CREATE INDEX because the transaction may not commit immediately. In the meantime, many new transactions may start with transaction id

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-28 Thread Florian G. Pflug
Tom Lane wrote: Florian G. Pflug [EMAIL PROTECTED] writes: Couldn't you store the creating transaction's xid in pg_index, and let other transaction check that against their snapshot like they would for any tuple's xmin or xmax? What snapshot? I keep having to remind people that system

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-28 Thread Simon Riggs
On Wed, 2007-03-28 at 23:42 +0530, Pavan Deolasee wrote: On 3/28/07, Simon Riggs [EMAIL PROTECTED] wrote: Set it at the end, not the beginning. At the end of what ? It does not help to set it at the end of CREATE INDEX because the transaction may not commit

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-28 Thread Florian G. Pflug
Pavan Deolasee wrote: On 3/28/07, Tom Lane [EMAIL PROTECTED] wrote: Florian G. Pflug [EMAIL PROTECTED] writes: Couldn't you store the creating transaction's xid in pg_index, and let other transaction check that against their snapshot like they would for any tuple's xmin or xmax? What

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-28 Thread Pavan Deolasee
On 3/29/07, Florian G. Pflug [EMAIL PROTECTED] wrote: Pavan Deolasee wrote: Tom, please correct me if I am wrong. But ISTM that this idea might work in this context. In get_relation_info(), we would check if xcreate xid stored in pg_index for the index under consideration is seen committed

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-23 Thread Hannu Krosing
Ühel kenal päeval, N, 2007-03-22 kell 23:30, kirjutas Pavan Deolasee: On 3/22/07, Tom Lane [EMAIL PROTECTED] wrote: Pavan Deolasee [EMAIL PROTECTED] writes: When CREATE INDEX starts, it acquires ShareLock on the table. At this point we may have one or

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-23 Thread Hannu Krosing
Ühel kenal päeval, N, 2007-03-22 kell 07:09, kirjutas Andrew Dunstan: Pavan Deolasee wrote: What I am hearing from many users is that its probably not such a nice thing to put such restriction. Thats fair. It really helps to think about a solution once you know what is acceptable and what

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-23 Thread Hannu Krosing
Ühel kenal päeval, K, 2007-03-21 kell 14:06, kirjutas Merlin Moncure: On 3/21/07, Florian G. Pflug [EMAIL PROTECTED] wrote: Pavan Deolasee wrote: On 3/21/07, Merlin Moncure [EMAIL PROTECTED] wrote: On 3/21/07, Pavan Deolasee [EMAIL PROTECTED] wrote: It seems much simpler to me do

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-23 Thread Pavan Deolasee
On 3/23/07, Hannu Krosing [EMAIL PROTECTED] wrote: My argument is that its enough to index only the LIVE tuple which is at the end of the chain if we don't use the new index for queries in transactions which were started before CREATE INDEX. You mean, which were started before CREATE

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-23 Thread Florian G. Pflug
Pavan Deolasee wrote: There is a slight hole in that SERIALIZABLE transactions won't be able to use any indexes they build during their transaction, since they may need to be able to see prior data, but I don't think anybody is going to complain about that restriction. Anyone? Oh, I did not

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-23 Thread Pavan Deolasee
On 3/23/07, Florian G. Pflug [EMAIL PROTECTED] wrote: Why exactly can't a SERIALIZABLE transaction use the index it created itself? If you add a pointer to the root of all HOT update chains where either the HEAD is alive, or some tuple is visible to the transaction creating the index,

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-23 Thread Florian G. Pflug
Pavan Deolasee wrote: On 3/23/07, Florian G. Pflug [EMAIL PROTECTED] wrote: Why exactly can't a SERIALIZABLE transaction use the index it created itself? If you add a pointer to the root of all HOT update chains where either the HEAD is alive, or some tuple is visible to the transaction

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-23 Thread Bruce Momjian
Hannu Krosing wrote: I don't think it is a good idea to store xid's anywhere but in xmin/xmax columns, as doing so would cause nasty xid wraparound problems. Instead you should wait, after completeing the index , for all concurrent transactions to end before you mark the index as usable for

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-23 Thread Bruce Momjian
Pavan Deolasee wrote: Xids are unstable and will come back to bite you after 2G transactions. Why not just use the isindvalid flag ? Who would set the flag to true ? Unless of course we are waiting in CREATE INDEX. But that seems to be less acceptable to me. Agreed, and we have the

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Csaba Nagy
speaking with pavan off list he seems to think that only 'create index' is outside transaction, not the other ddl flavors of it because they are generally acquiring a excl lock. so, in that sense it is possibly acceptable to me although still a pretty tough pill to swallow (thinking, guc

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Pavan Deolasee
On 3/22/07, Csaba Nagy [EMAIL PROTECTED] wrote: speaking with pavan off list he seems to think that only 'create index' is outside transaction, not the other ddl flavors of it because they are generally acquiring a excl lock. so, in that sense it is possibly acceptable to me although still

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Andrew Dunstan
Pavan Deolasee wrote: What I am hearing from many users is that its probably not such a nice thing to put such restriction. Thats fair. It really helps to think about a solution once you know what is acceptable and what is not. That's likely to be the reaction for almost any restriction you

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Merlin Moncure
On 3/22/07, Pavan Deolasee [EMAIL PROTECTED] wrote: On 3/22/07, Csaba Nagy [EMAIL PROTECTED] wrote: speaking with pavan off list he seems to think that only 'create index' is outside transaction, not the other ddl flavors of it because they are generally acquiring a excl lock. so, in that

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Pavan Deolasee
On 3/21/07, Bruce Momjian [EMAIL PROTECTED] wrote: A different idea is to flag the _index_ as using HOT for the table or not, using a boolean in pg_index. The idea is that when a new index is created, it has its HOT boolean set to false and indexes all tuples and ignores HOT chains. Then

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Tom Lane
Pavan Deolasee [EMAIL PROTECTED] writes: When CREATE INDEX starts, it acquires ShareLock on the table. At this point we may have one or more HOT-update chains in the table. Tuples in this chain may be visible to one or more running transactions. The fact that we have ShareLock on the table

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Pavan Deolasee
On 3/22/07, Tom Lane [EMAIL PROTECTED] wrote: Pavan Deolasee [EMAIL PROTECTED] writes: When CREATE INDEX starts, it acquires ShareLock on the table. At this point we may have one or more HOT-update chains in the table. Tuples in this chain may be visible to one or more running transactions.

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Bruce Momjian
Pavan Deolasee wrote: My argument is that its enough to index only the LIVE tuple which is at the end of the chain if we don't use the new index for queries in transactions which were started before CREATE INDEX. I am proposing to do that by storing an xid in the pg_index row. A special case

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Simon Riggs
On Thu, 2007-03-22 at 22:11 +0530, Pavan Deolasee wrote: With this background, I propose to index ONLY the head of the HOT-chain. The TID of the root tuple is used instead of the actual TID of the tuple being indexed. This index will not be available to the transactions which are started

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: There is a slight hole in that SERIALIZABLE transactions won't be able to use any indexes they build during their transaction, since they may need to be able to see prior data, but I don't think anybody is going to complain about that restriction. Anyone?

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Simon Riggs
On Thu, 2007-03-22 at 16:16 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: There is a slight hole in that SERIALIZABLE transactions won't be able to use any indexes they build during their transaction, since they may need to be able to see prior data, but I don't think anybody

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Russell Smith
Simon Riggs wrote: On Thu, 2007-03-22 at 16:16 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: There is a slight hole in that SERIALIZABLE transactions won't be able to use any indexes they build during their transaction, since they may need to be able to see prior data,

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-22 Thread Pavan Deolasee
On 3/23/07, Simon Riggs [EMAIL PROTECTED] wrote: The ShareLock taken by CREATE INDEX guarantees all transactions that wrote data to the table have completed and that no new data can be added until after the index build commits. So the end of the chain is visible to CREATE INDEX and won't

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Bruce Momjian wrote: I have read the HOT discussion and wanted to give my input. The major issue is that CREATE INDEX might require a HOT chain to be split apart if one of the new indexed columns changed in the HOT chain. To expand a little more, the problem is that when you split those HOT

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Heikki Linnakangas
Bruce Momjian wrote: A different idea is to flag the _index_ as using HOT for the table or not, using a boolean in pg_index. The idea is that when a new index is created, it has its HOT boolean set to false and indexes all tuples and ignores HOT chains. Then doing lookups using that index, the

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Simon Riggs
On Wed, 2007-03-21 at 10:47 -0400, Bruce Momjian wrote: A different idea is to flag the _index_ as using HOT for the table or not, using a boolean in pg_index. The idea is that when a new index is created, it has its HOT boolean set to false and indexes all tuples and ignores HOT chains.

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Heikki Linnakangas
Sorry, I was a bit too quick to respond. I didn't understand at first how this differs from Pavan's/Simon's proposals. Let me answer my own questions. Heikki Linnakangas wrote: Bruce Momjian wrote: A different idea is to flag the _index_ as using HOT for the table or not, using a boolean in

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Gregory Stark
Bruce Momjian [EMAIL PROTECTED] writes: We also add a boolean to pg_class to indicate no new HOT chains should be created and set that to false once the new index is created. Since we have all the index info in the relcache we could just skim through all the indexes when we build the relcache

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Heikki Linnakangas wrote: Bruce Momjian wrote: A different idea is to flag the _index_ as using HOT for the table or not, using a boolean in pg_index. The idea is that when a new index is created, it has its HOT boolean set to false and indexes all tuples and ignores HOT chains. Then

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Pavan Deolasee
On 3/21/07, Bruce Momjian [EMAIL PROTECTED] wrote: Bruce Momjian wrote: I have read the HOT discussion and wanted to give my input. The major issue is that CREATE INDEX might require a HOT chain to be split apart if one of the new indexed columns changed in the HOT chain. To expand a

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Simon Riggs wrote: On Wed, 2007-03-21 at 10:47 -0400, Bruce Momjian wrote: A different idea is to flag the _index_ as using HOT for the table or not, using a boolean in pg_index. The idea is that when a new index is created, it has its HOT boolean set to false and indexes all tuples and

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Ah, sounds like you have the idea clearly now. Great. --- Heikki Linnakangas wrote: Sorry, I was a bit too quick to respond. I didn't understand at first how this differs from Pavan's/Simon's proposals. Let me answer

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
As stated in later email, I think we should focus on the xid idea because it is more flexible. --- Gregory Stark wrote: Bruce Momjian [EMAIL PROTECTED] writes: We also add a boolean to pg_class to indicate no new HOT

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Pavan Deolasee wrote: On 3/21/07, Bruce Momjian [EMAIL PROTECTED] wrote: Bruce Momjian wrote: I have read the HOT discussion and wanted to give my input. The major issue is that CREATE INDEX might require a HOT chain to be split apart if one of the new indexed columns changed in

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Pavan Deolasee
On 3/21/07, Bruce Momjian [EMAIL PROTECTED] wrote: I am worried that will require CREATE INDEX to wait for a long time. Not unless there are long running transactions. We are not waiting for the lock, but only for the current transactions to finish. Is the pg_index xid idea too

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Gregory Stark
Bruce Momjian [EMAIL PROTECTED] writes: As stated in later email, I think we should focus on the xid idea because it is more flexible. Sorry if I was unclear. I agree, my comment and questions are all predicated on the assumption that we would go with xids.

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Gregory Stark wrote: Bruce Momjian [EMAIL PROTECTED] writes: As stated in later email, I think we should focus on the xid idea because it is more flexible. Sorry if I was unclear. I agree, my comment and questions are all predicated on the assumption that we would go with xids. OK,

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Pavan Deolasee wrote: On 3/21/07, Bruce Momjian [EMAIL PROTECTED] wrote: I am worried that will require CREATE INDEX to wait for a long time. Not unless there are long running transactions. We are not waiting for the lock, but only for the current transactions to finish. Waiting

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Effectively, my idea is not to chill/break the HOT chains during index creation, but rather to abandon them and wait for VACUUM to clean them up. My idea is much closer to the idea of a bit per index on every tuple, except the tuple xmax and pg_index xid replace them.

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Merlin Moncure
On 3/21/07, Pavan Deolasee [EMAIL PROTECTED] wrote: It seems much simpler to me do something like this. But important question is whether the restriction that CREATE INDEX can not be run in a transaction block is acceptable ? yikes -- this is huge, huge price to pay, IMHO. Think about DDL

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Pavan Deolasee
On 3/21/07, Merlin Moncure [EMAIL PROTECTED] wrote: On 3/21/07, Pavan Deolasee [EMAIL PROTECTED] wrote: It seems much simpler to me do something like this. But important question is whether the restriction that CREATE INDEX can not be run in a transaction block is acceptable ? yikes -- this

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Pavan Deolasee
On 3/21/07, Bruce Momjian [EMAIL PROTECTED] wrote: Effectively, my idea is not to chill/break the HOT chains during index creation, but rather to abandon them and wait for VACUUM to clean them up. My idea is much closer to the idea of a bit per index on every tuple, except the tuple xmax and

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Florian G. Pflug
Pavan Deolasee wrote: On 3/21/07, Merlin Moncure [EMAIL PROTECTED] wrote: On 3/21/07, Pavan Deolasee [EMAIL PROTECTED] wrote: It seems much simpler to me do something like this. But important question is whether the restriction that CREATE INDEX can not be run in a transaction block is

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Merlin Moncure
On 3/21/07, Florian G. Pflug [EMAIL PROTECTED] wrote: Pavan Deolasee wrote: On 3/21/07, Merlin Moncure [EMAIL PROTECTED] wrote: On 3/21/07, Pavan Deolasee [EMAIL PROTECTED] wrote: It seems much simpler to me do something like this. But important question is whether the restriction that

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Pavan Deolasee wrote: On 3/21/07, Bruce Momjian [EMAIL PROTECTED] wrote: Effectively, my idea is not to chill/break the HOT chains during index creation, but rather to abandon them and wait for VACUUM to clean them up. My idea is much closer to the idea of a bit per index on every

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Bruce Momjian wrote: Also, I am wondering whether the information that which index is used to fetch a tuple is always available. I haven't checked, but do we have that information in lossy bitmap heapscan ? Oh, that is an interesting problem because an index might have one index entry

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Bruce Momjian wrote: Bruce Momjian wrote: Also, I am wondering whether the information that which index is used to fetch a tuple is always available. I haven't checked, but do we have that information in lossy bitmap heapscan ? Oh, that is an interesting problem because an index

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Simon Riggs
On Wed, 2007-03-21 at 13:29 -0400, Bruce Momjian wrote: Pavan Deolasee wrote: On 3/21/07, Bruce Momjian [EMAIL PROTECTED] wrote: I am worried that will require CREATE INDEX to wait for a long time. Not unless there are long running transactions. We are not waiting for

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Heikki Linnakangas
Bruce Momjian wrote: Bruce Momjian wrote: Bruce Momjian wrote: Also, I am wondering whether the information that which index is used to fetch a tuple is always available. I haven't checked, but do we have that information in lossy bitmap heapscan ? Oh, that is an interesting problem because

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Heikki Linnakangas wrote: Bruce Momjian wrote: Bruce Momjian wrote: Bruce Momjian wrote: Also, I am wondering whether the information that which index is used to fetch a tuple is always available. I haven't checked, but do we have that information in lossy bitmap heapscan ? Oh, that

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Bruce Momjian
Bruce Momjian wrote: You don't need to scan the whole page like in the lossy bitmap mode, just all the tuples in the HOT-chain. You need to somehow pass the information that multiple indexes have been used in the bitmap scan to the bitmap heapscan node, so that it knows when the

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-21 Thread Andrew Dunstan
Bruce Momjian wrote: Anyway, perhaps we can leave the bitmap scan part to someone more familiar with that part of the code, like Tom. Yeah. We all know he doesn't have enough to do ... cheers andrew ---(end of broadcast)--- TIP 5:

Re: [HACKERS] CREATE INDEX and HOT - revised design

2007-03-20 Thread Bruce Momjian
I have read the HOT discussion and wanted to give my input. The major issue is that CREATE INDEX might require a HOT chain to be split apart if one of the new indexed columns changed in the HOT chain. As for the outline below, there is no way we are going to add new ALTER TABLE and CHILL