Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-05-10 Thread Tom Lane
I forgot to mention it in the commit message, but this patch http://archives.postgresql.org/pgsql-committers/2006-05/msg00069.php includes a fix to avoid tuple concurrently updated failures in CREATE INDEX's update of the parent relation's pg_class entry. I'm still not convinced that that's where

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Tom Lane
Wes [EMAIL PROTECTED] writes: Where do you want me to send it to? pgsql-bugs would be appropriate. Of course, since your hardware is different, it may not show up since it appears to be a timing thing.. I'm on a PB G4 1Ghz. My G4 is in the shop at the moment, but Apple promised it back by

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Wes
On 4/25/06 1:01 AM, Tom Lane [EMAIL PROTECTED] wrote: Where do you want me to send it to? pgsql-bugs would be appropriate. It's not that small that I would want to post it to a list. Also, I think I've sanitized the data, but I wouldn't want to post it on a public list. Or are you just

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Tom Lane
Wes [EMAIL PROTECTED] writes: On 4/25/06 1:01 AM, Tom Lane [EMAIL PROTECTED] wrote: Where do you want me to send it to? pgsql-bugs would be appropriate. It's not that small that I would want to post it to a list. Also, I think I've sanitized the data, but I wouldn't want to post it on a

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Zeugswetter Andreas DCP SD
[ shrug... ] Worksforme. There is a short interval at the end of the first CREATE INDEX on the table where the problem would happen if another CREATE INDEX tries to modify the pg_class row before the first one's committed. I did a pg_dumpall and removed the index creation commands.

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Wes
On 4/25/06 2:18 AM, Tom Lane [EMAIL PROTECTED] wrote: So invent some made-up data. I'd be seriously surprised if this behavior has anything to do with the precise data being indexed. Experiment around till you've got something you don't mind posting that exhibits the behavior you see. My

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Zeugswetter Andreas DCP SD
Wes, you could most likely solve your immediate problem if you did an analyze before creating the indexes. I can try that. Is that going to be a reasonable thing to do when there's 100 million rows per table? I obviously want to minimize the number of sequential passes through the

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Tom Lane
Zeugswetter Andreas DCP SD [EMAIL PROTECTED] writes: My answer to this would be to (have an option to) ommit this relpages and reltuples update. Wouldn't help, unless you want to get rid of relhasindex too. Wes, you could most likely solve your immediate problem if you did an analyze before

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Wes
On 4/25/06 9:58 AM, Tom Lane [EMAIL PROTECTED] wrote: I can't think of any effect in the PG code that would make CREATE INDEX commands tend to converge rather than diverge, so maybe there is some external effect here. Wes, is your machine a dual processor by any chance? Which OS X version

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Tom Lane
Wes [EMAIL PROTECTED] writes: A while back, I saw a posting (I think from Tom) to the effect of index creation converging due to disk caching. It was along the lines of the slower index would be reading from disk data cached by the first index creation's read. When the faster creation went

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Alvaro Herrera
I'm late to this thread, but maybe we can make the process of storing the new data in pg_class take a lock using LockObject() or something like that to serialize the access to the pg_class row. The idea would be that this lock doesn't conflict with a LockRelation(), but it would of course

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-25 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: I'm late to this thread, but maybe we can make the process of storing the new data in pg_class take a lock using LockObject() or something like that to serialize the access to the pg_class row. I'm inclined to think that the right solution is to fix

Catalog Access (was: [HACKERS] [GENERAL] Concurrency problem building indexes)

2006-04-25 Thread Jim C. Nasby
On Tue, Apr 25, 2006 at 12:48:04PM -0400, Alvaro Herrera wrote: I'm late to this thread, but maybe we can make the process of storing the new data in pg_class take a lock using LockObject() or something like that to serialize the access to the pg_class row. The idea would be that this lock

Re: Catalog Access (was: [HACKERS] [GENERAL] Concurrency problem building indexes)

2006-04-25 Thread Martijn van Oosterhout
On Tue, Apr 25, 2006 at 12:25:35PM -0500, Jim C. Nasby wrote: Is there anything in comments/docs/list archives about why catalog access uses a bunch of 'magic' instead of treating catalog tables the same as every other table? I realize that ultimately you have to bootstrap somehow (kinda hard

Re: Catalog Access (was: [HACKERS] [GENERAL] Concurrency problem building indexes)

2006-04-25 Thread Tom Lane
Martijn van Oosterhout kleptog@svana.org writes: I think the basic problem is that DDL can't really work within a transaction. If I do an ALTER TABLE, some of these changes need to show up to concurrent transactions (maybe creating a unique index?). The point is that DDL can't be MVCC. If for

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Jim C. Nasby
moving to -hackers On Sun, Apr 23, 2006 at 09:06:59AM -0500, Wes wrote: Postgres: 8.1.3 OS: Mac OS X 10.4.6 I've run into another concurrency issue - parallel building of indexes. When I try to build multiple indexes at a time, I randomly get: ERROR: tuple concurrently updated The

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Tom Lane
Jim C. Nasby [EMAIL PROTECTED] writes: Since this seems to only be an issue due to trying to update pg_class for the table, perhaps CREATE INDEX can just ignore errors there? Lessee, where would ignoring an error potentially cause the greatest damage? I can hardly think of a less critical

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Jim C. Nasby
On Mon, Apr 24, 2006 at 08:14:33PM -0400, Tom Lane wrote: Jim C. Nasby [EMAIL PROTECTED] writes: Since this seems to only be an issue due to trying to update pg_class for the table, perhaps CREATE INDEX can just ignore errors there? Lessee, where would ignoring an error potentially cause

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Tom Lane
Jim C. Nasby [EMAIL PROTECTED] writes: Was the code ever changed so that it won't update relpages if the number is the same? Long ago. I just tested it again, and AFAICS you can create indexes concurrently so long as the underlying table isn't changing (ie, neither reltuples nor relpages

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Jim C. Nasby
On Mon, Apr 24, 2006 at 08:42:41PM -0400, Tom Lane wrote: Jim C. Nasby [EMAIL PROTECTED] writes: Was the code ever changed so that it won't update relpages if the number is the same? Long ago. I just tested it again, and AFAICS you can create indexes concurrently so long as the

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Wes
On 4/24/06 7:54 PM, Jim C. Nasby [EMAIL PROTECTED] wrote: Long ago. I just tested it again, and AFAICS you can create indexes concurrently so long as the underlying table isn't changing (ie, neither reltuples nor relpages changes). Hrm, the OP seemed to find a case that was having

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Wes
Long ago. I just tested it again, and AFAICS you can create indexes concurrently so long as the underlying table isn't changing (ie, neither reltuples nor relpages changes). Hrm, the OP seemed to find a case that was having problems:

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Tom Lane
Wes [EMAIL PROTECTED] writes: I can duplicate the error at will. [ shrug... ] Worksforme. There is a short interval at the end of the first CREATE INDEX on the table where the problem would happen if another CREATE INDEX tries to modify the pg_class row before the first one's committed. That

Re: [HACKERS] [GENERAL] Concurrency problem building indexes

2006-04-24 Thread Wes
On 4/24/06 11:02 PM, Tom Lane [EMAIL PROTECTED] wrote: [ shrug... ] Worksforme. There is a short interval at the end of the first CREATE INDEX on the table where the problem would happen if another CREATE INDEX tries to modify the pg_class row before the first one's committed. That would