Re: [PATCHES] [HACKERS] Index creation takes for ever

2004-03-17 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: Where was it posted anyway? Found it: http://groups.google.com/groups?hl=enlr=ie=UTF-8selm=200312010450.hB14ovH16330%40candle.pha.pa.usrnum=8 Thanks. The original patch is much older than I thought --- I was looking in the

Re: [PATCHES] [HACKERS] Index creation takes for ever

2004-03-17 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: Where was it posted anyway? Found it: http://groups.google.com/groups?hl=enlr=ie=UTF-8selm=200312010450.hB14ovH16330%40candle.pha.pa.usrnum=8 Thanks. The original patch is much older than I thought ---

Re: [PATCHES] [HACKERS] Index creation takes for ever

2004-03-16 Thread Bruce Momjian
Here is more detail on the patch. --- Manfred Koizar wrote: On Mon, 1 Dec 2003 00:02:54 -0500 (EST), Bruce Momjian [EMAIL PROTECTED] wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: And if it doesn't

Re: [PATCHES] [HACKERS] Index creation takes for ever

2004-03-16 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Where are we on this? It seems like a win to me. I thought it was a bad idea, although I no longer remember the details. If I remember correctly, you didn't like the index routines reading the tuple information, or something like

Re: [PATCHES] [HACKERS] Index creation takes for ever

2004-03-16 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: If I remember correctly, you didn't like the index routines reading the tuple information, or something like that, but there was a performance benefit for duplicate keys, so I think we should re-investigate this. I don't see the actual patch either in

Re: [PATCHES] [HACKERS] Index creation takes for ever

2004-03-16 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: If I remember correctly, you didn't like the index routines reading the tuple information, or something like that, but there was a performance benefit for duplicate keys, so I think we should re-investigate this. I don't see the

Re: [PATCHES] [HACKERS] Index creation takes for ever

2003-12-02 Thread Manfred Koizar
On Mon, 01 Dec 2003 13:32:10 -0500, Tom Lane [EMAIL PROTECTED] wrote: Manfred Koizar [EMAIL PROTECTED] writes: comparetup_index() compares two IndexTuples. The structure IndexTupleData consists basically of not much more than an ItemPointer, and the patch is not much more than adding a

Re: [PATCHES] [HACKERS] Index creation takes for ever

2003-12-01 Thread Manfred Koizar
On Mon, 1 Dec 2003 00:02:54 -0500 (EST), Bruce Momjian [EMAIL PROTECTED] wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: And if it doesn't help index creation speed, at least the resulting index has better correlation. ... which has been shown by the example in the original

Re: [PATCHES] [HACKERS] Index creation takes for ever

2003-12-01 Thread Tom Lane
Manfred Koizar [EMAIL PROTECTED] writes: comparetup_index() compares two IndexTuples. The structure IndexTupleData consists basically of not much more than an ItemPointer, and the patch is not much more than adding a comparison of two ItemPointers. So how does the patch introduce a new low

Re: [PATCHES] [HACKERS] Index creation takes for ever

2003-11-30 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: If qsort is to blame, then maybe this patch could help. It sorts equal key values on item pointer. And if it doesn't help index creation speed, at least the resulting index has better correlation. I will try to apply it within the next 48 hours. I

Re: [PATCHES] [HACKERS] Index creation takes for ever

2003-11-30 Thread Bruce Momjian
Patch removed from queue. --- Manfred Koizar wrote: On Mon, 01 Sep 2003 08:46:09 -0400, Tom Lane [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] writes: it took 69 minutes to finish, 75% of this time was devoted to create 2

Re: [PATCHES] [HACKERS] Index creation takes for ever

2003-11-30 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: If qsort is to blame, then maybe this patch could help. It sorts equal key values on item pointer. And if it doesn't help index creation speed, at least the resulting index has better correlation. I will try to apply it within

Re: [HACKERS] Index creation takes for ever

2003-09-08 Thread Zeugswetter Andreas SB SD
I don't think so, because the patch does nothing to keep the sort order once the index is initially created. As Tom mentioned, we might not want to keep the tid's in order after the index is created because he wants the most recent tid's first, so the expired ones migrate to the end.

Re: [HACKERS] Index creation takes for ever

2003-09-08 Thread Manfred Koizar
On Mon, 8 Sep 2003 11:31:05 +0200, Zeugswetter Andreas SB SD [EMAIL PROTECTED] wrote: As Tom mentioned, we might not want to keep the tid's in order after the index is created because he wants the most recent tid's first, so the expired ones migrate to the end. But on average this argument

Re: [HACKERS] Index creation takes for ever

2003-09-07 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: I assume this completes this TODO: * Order duplicate index entries by tid for faster heap lookups I don't know why that TODO entry exists, but I think the idea is counterproductive. The existing btree code will tend to put newer versions of a row

Re: [PATCHES] [HACKERS] Index creation takes for ever

2003-09-07 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: I assume this completes this TODO: * Order duplicate index entries by tid for faster heap lookups I don't know why that TODO entry exists, but I think the idea is counterproductive. The existing btree code will tend to put newer

Re: [PATCHES] [HACKERS] Index creation takes for ever

2003-09-07 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: * Order duplicate index entries by tid for faster heap lookups I don't know why that TODO entry exists, but I think the idea is counterproductive. I assume you are talking about a unique index that probably only has a few non-expired

Re: [PATCHES] [HACKERS] Index creation takes for ever

2003-09-07 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Tom Lane wrote: * Order duplicate index entries by tid for faster heap lookups I don't know why that TODO entry exists, but I think the idea is counterproductive. I assume you are talking about a unique index that probably only

Re: [HACKERS] Index creation takes for ever

2003-09-07 Thread Manfred Koizar
On Sun, 7 Sep 2003 11:43:42 -0400 (EDT), Bruce Momjian [EMAIL PROTECTED] wrote: I assume this completes this TODO: * Order duplicate index entries by tid for faster heap lookups I don't think so, because the patch does nothing to keep the sort order once the index is initially created.

Re: [HACKERS] Index creation takes for ever

2003-09-01 Thread ohp
: Re: [HACKERS] Index creation takes for ever [EMAIL PROTECTED] writes: I've then pg_dump'ed the database and recreate an other both on 7.3.4 and 7.4b Both are still running after more than 30 minutes of CPU (100% cpu taken) creating the levt_lu_ligne_evt_key. That's hard to believe. I

Re: [HACKERS] Index creation takes for ever

2003-09-01 Thread Tom Lane
[EMAIL PROTECTED] writes: it took 69 minutes to finish, 75% of this time was devoted to create 2 indexes on varchar(2) with value being 'O', 'N' or null; I wonder if it's a configuration matter. I still say it's either strcoll or qsort's fault. Try swapping in our own version of qsort to see

Re: [HACKERS] Index creation takes for ever

2003-09-01 Thread ohp
] To: [EMAIL PROTECTED] Cc: pgsql-hackers list [EMAIL PROTECTED] Subject: Re: [HACKERS] Index creation takes for ever [EMAIL PROTECTED] writes: it took 69 minutes to finish, 75% of this time was devoted to create 2 indexes on varchar(2) with value being 'O', 'N' or null; I wonder if it's

Re: [HACKERS] Index creation takes for ever

2003-09-01 Thread Tom Lane
[EMAIL PROTECTED] writes: I've made some tests with your qsort and it DEFINITIVLY help ~3 mn instead of 69. However this is for 7.3.4 I've got no probs with 7.4b. Did something change in btree creation? Hmm, I wouldn't have thought so, but perhaps we did change something that would affect

Re: [HACKERS] Index creation takes for ever

2003-08-30 Thread Bruce Momjian
Tom Lane wrote: [EMAIL PROTECTED] writes: This is on unixware 7 (both 7.3.4 and 7.4b) I'm on the FR language (I'll re-initdb whith lang=C to see what happens) Okay. If you find it's still slow in C locale, the next thing to try would be forcing use of our own qsort, as we already do

Re: [HACKERS] Index creation takes for ever

2003-08-30 Thread ohp
PROTECTED] To: Tom Lane [EMAIL PROTECTED] Cc: [EMAIL PROTECTED], pgsql-hackers list [EMAIL PROTECTED] Subject: Re: [HACKERS] Index creation takes for ever Tom Lane wrote: [EMAIL PROTECTED] writes: This is on unixware 7 (both 7.3.4 and 7.4b) I'm on the FR language (I'll re-initdb whith

[HACKERS] Index creation takes for ever

2003-08-28 Thread ohp
Hi every one, I've tried to reindex one of my customer's table to gain some disk space. I had to stop after 90 m cpu... I've then pg_dump'ed the database and recreate an other both on 7.3.4 and 7.4b Both are still running after more than 30 minutes of CPU (100% cpu taken) creating the

Re: [HACKERS] Index creation takes for ever

2003-08-28 Thread Tom Lane
[EMAIL PROTECTED] writes: I've then pg_dump'ed the database and recreate an other both on 7.3.4 and 7.4b Both are still running after more than 30 minutes of CPU (100% cpu taken) creating the levt_lu_ligne_evt_key. That's hard to believe. I get regression=# SELECT levt_lu,count(*) from

Re: [HACKERS] Index creation takes for ever

2003-08-28 Thread Tom Lane
[EMAIL PROTECTED] writes: This is on unixware 7 (both 7.3.4 and 7.4b) I'm on the FR language (I'll re-initdb whith lang=C to see what happens) Okay. If you find it's still slow in C locale, the next thing to try would be forcing use of our own qsort, as we already do for Solaris. You'd need

Re: [HACKERS] Index creation takes for ever

2003-08-28 Thread ohp
] Index creation takes for ever [EMAIL PROTECTED] writes: This is on unixware 7 (both 7.3.4 and 7.4b) I'm on the FR language (I'll re-initdb whith lang=C to see what happens) Okay. If you find it's still slow in C locale, the next thing to try would be forcing use of our own qsort, as we

Re: [HACKERS] Index creation takes for ever

2003-08-28 Thread Tom Lane
[EMAIL PROTECTED] writes: I've reinitdb (on 7.4b) with LANG=C and it worked. So I reinitDB with LANG=FR and used LANG=C to psql -f xxx.sql template1 to recreate the db and it worked too... That's weird. I don't understand why an initdb in the same locale would make the problem go away. I

Re: [HACKERS] Index creation takes for ever

2003-08-28 Thread ohp
Lane [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: pgsql-hackers list [EMAIL PROTECTED] Subject: Re: [HACKERS] Index creation takes for ever [EMAIL PROTECTED] writes: I've reinitdb (on 7.4b) with LANG=C and it worked. So I reinitDB with LANG=FR and used LANG=C to psql -f xxx.sql template1