Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-25 Thread Curt Sampson
On Mon, 24 Jun 2002, Tom Lane wrote: There are a lot of other things we desperately need to spend time on that would not amount to re-engineering large quantities of OS-level code. Given that most Unixen have perfectly respectable disk management subsystems, we prefer to tune our code to

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-25 Thread Bruce Momjian
Curt Sampson wrote: On Mon, 24 Jun 2002, Tom Lane wrote: There are a lot of other things we desperately need to spend time on that would not amount to re-engineering large quantities of OS-level code. Given that most Unixen have perfectly respectable disk management subsystems, we

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Curt Sampson
On Sun, 23 Jun 2002, Bruce Momjian wrote: Yes, I don't see writing to two files vs. one to be any win, especially when we need to fsync both of them. What I would really like is to avoid the double I/O of writing to WAL and to the data file; improving that would be a huge win. You mean,

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread J. R. Nield
On Sun, 2002-06-23 at 21:29, J. R. Nield wrote: If is impossible to do what you want. You can not protect against... Wow. The number of typo's in that last one was just amazing. I even started with one. Have an nice weekend everybody :-) ;jrnield -- J. R. Nield [EMAIL PROTECTED]

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread J. R. Nield
On Sun, 2002-06-23 at 15:36, Bruce Momjian wrote: Yes, I don't see writing to two files vs. one to be any win, especially when we need to fsync both of them. What I would really like is to avoid the double I/O of writing to WAL and to the data file; improving that would be a huge win. If

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Bruce Momjian
J. R. Nield wrote: On Sun, 2002-06-23 at 15:36, Bruce Momjian wrote: Yes, I don't see writing to two files vs. one to be any win, especially when we need to fsync both of them. What I would really like is to avoid the double I/O of writing to WAL and to the data file; improving that

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Bruce Momjian
Curt Sampson wrote: On 23 Jun 2002, J. R. Nield wrote: So since we have all this buffering designed especially to meet our needs, and since the OS buffering is in the way, can someone explain to me why postgresql would ever open a file without the O_DSYNC flag if the platform supports

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Curt Sampson
On 23 Jun 2002, J. R. Nield wrote: If is impossible to do what you want. You can not protect against partial writes without writing pages twice and calling fdatasync between them while going through a generic filesystem. I agree with this. The best disk array will not protect you if the

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Bruce Momjian
J. R. Nield wrote: So since we have all this buffering designed especially to meet our needs, and since the OS buffering is in the way, can someone explain to me why postgresql would ever open a file without the O_DSYNC flag if the platform supports it? We sync only WAL, not the other

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: The only thing I've been able to think of that seems like it might improve matters is to make the WAL writing logic aware of the layout of buffer pages --- specifically, to know that our pages generally contain an uninteresting hole in the middle, and

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Tom Lane
On Sun, 23 Jun 2002, Bruce Momjian wrote: Yes, I don't see writing to two files vs. one to be any win, especially when we need to fsync both of them. What I would really like is to avoid the double I/O of writing to WAL and to the data file; improving that would be a huge win. I don't

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Bruce Momjian
Tom Lane wrote: On Sun, 23 Jun 2002, Bruce Momjian wrote: Yes, I don't see writing to two files vs. one to be any win, especially when we need to fsync both of them. What I would really like is to avoid the double I/O of writing to WAL and to the data file; improving that would be a

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Tom Lane
J. R. Nield [EMAIL PROTECTED] writes: Also, postgreSQL can't recover from any other type of block corruption, while the commercial systems can. Say again? Would it not be the case that things like read-ahead, grouping writes, and caching written data are probably best done by PostgreSQL,

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: Does anyone know what the major barriers to infinite log replay are in PostgreSQL? I'm trying to look for everything that might need to be changed outside xlog.c, but surely this has come up before. Searching the archives hasn't revealed much. This

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Bruce Momjian
J. R. Nield wrote: This I don't quite understand. Assuming you're using a SCSI drive (and this mostly applies to ATAPI/IDE, too), you can do naught but align block writes to the structure of the underlying device. When you initiate a SCSI WRITE command, you start by telling the device at

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread J. R. Nield
On Sun, 2002-06-23 at 23:40, Curt Sampson wrote: On 23 Jun 2002, J. R. Nield wrote: If is impossible to do what you want. You can not protect against partial writes without writing pages twice and calling fdatasync between them while going through a generic filesystem. I agree with

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Bruce Momjian
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Does anyone know what the major barriers to infinite log replay are in PostgreSQL? I'm trying to look for everything that might need to be changed outside xlog.c, but surely this has come up before. Searching the archives hasn't

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread J. R. Nield
On Mon, 2002-06-24 at 17:16, Tom Lane wrote: I think you have been missing the point... Yes, this appears to be the case. Thanks especially to Curt for clearing things up for me. -- J. R. Nield [EMAIL PROTECTED] ---(end of broadcast)---

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-24 Thread Curt Sampson
On 24 Jun 2002, J. R. Nield wrote: All I'm saying is that the entire postgresql block write must be converted into exactly one SCSI write command in all cases, and I don't know a portable way to ensure this. No, there's no portable way. All you can do is give the admin who is able to set

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-23 Thread J. R. Nield
On Sat, 2002-06-22 at 19:17, Bruce Momjian wrote: J. R. Nield wrote: One other point: Page pre-image logging is fundamentally the same as what Jim Grey's book[1] would call careful writes. I don't believe they should be in the XLOG, because we never need to keep the pre-images after

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-23 Thread Curt Sampson
On 23 Jun 2002, J. R. Nield wrote: On Sat, 2002-06-22 at 19:17, Bruce Momjian wrote: J. R. Nield wrote: One other point: Page pre-image logging is fundamentally the same as what Jim Grey's book[1] would call careful writes. I don't believe they should be in the XLOG, because we

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-23 Thread Tom Lane
Curt Sampson [EMAIL PROTECTED] writes: This should also allow us to disable completely the ping-pong writes if we have a disk subsystem that we trust. If we have a disk subsystem we trust, we just disable fsync on the WAL and the performance issue largely goes away. I concur with Bruce: the

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-23 Thread Curt Sampson
On Sun, 23 Jun 2002, Tom Lane wrote: Curt Sampson [EMAIL PROTECTED] writes: This should also allow us to disable completely the ping-pong writes if we have a disk subsystem that we trust. If we have a disk subsystem we trust, we just disable fsync on the WAL and the performance issue

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-23 Thread J. R. Nield
On Sun, 2002-06-23 at 11:19, Tom Lane wrote: Curt Sampson [EMAIL PROTECTED] writes: This should also allow us to disable completely the ping-pong writes if we have a disk subsystem that we trust. If we have a disk subsystem we trust, we just disable fsync on the WAL and the performance

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-23 Thread J. R. Nield
On Sun, 2002-06-23 at 12:10, Curt Sampson wrote: So what we would really need to implement, if we wanted to be more efficient with trusted disk subsystems, would be the option of writing to the log only the changed row or changed part of the row, or writing the entire changed page. I don't

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-23 Thread Curt Sampson
On 23 Jun 2002, J. R. Nield wrote: So since we have all this buffering designed especially to meet our needs, and since the OS buffering is in the way, can someone explain to me why postgresql would ever open a file without the O_DSYNC flag if the platform supports it? It's more code, if

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-22 Thread J. R. Nield
On Thu, 2002-06-20 at 21:58, Bruce Momjian wrote: I was wondering, how does knowing the block is corrupt help MS SQL? Right now, we write changed pages to WAL, then later write them to disk. I have always been looking for a way to prevent these WAL writes. The 512-byte bit seems

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-22 Thread Bruce Momjian
J. R. Nield wrote: One other point: Page pre-image logging is fundamentally the same as what Jim Grey's book[1] would call careful writes. I don't believe they should be in the XLOG, because we never need to keep the pre-images after we're sure the buffer has made it to the disk. Instead,

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-06-20 Thread Curt Sampson
On Thu, 20 Jun 2002, Bruce Momjian wrote: MS SQL Server has an interesting way of dealing with this. They have a torn bit in each 512-byte chunk of a page, and this bit is set the same for each chunk. When they are about to write out a page, they first flip all of the torn bits and then

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-25 Thread Bruce Momjian
Michael Loftis wrote: A Block-sized read will not be broken up. But if you're reading ina size bigger than the underlying systems block sizes then it can get broken up. So yes a sequential read will get broken up. A single read request for a block may or may not get broken up. If

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-25 Thread Bruce Momjian
Actually, this brings up a different point. We use 8k blocks now because at the time PostgreSQL was developed, it used BSD file systems, and those prefer 8k blocks, and there was some concept that an 8k write was atomic, though with 512 byte disk blocks, that was incorrect. (We knew that at

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-25 Thread mlw
Bruce Momjian wrote: Now, with larger RAM and disk sizes, it may be time to consider larger page sizes, like 32k pages. That reduces the granularity of the cache, but it may have other performance advantages that would be worth it. What people are actually suggesting with the read-ahead

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-25 Thread Curt Sampson
On Thu, 25 Apr 2002, Bruce Momjian wrote: Also keep in mind most disks have 512 byte blocks, so even if the file system is 8k, the disk block sizes are different. A given 8k or 1k file system block may not even be all in the same cylinder. Right. Though much of the time they will be in the

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-25 Thread Curt Sampson
On Thu, 25 Apr 2002, Bruce Momjian wrote: Actually, this brings up a different point. We use 8k blocks now because at the time PostgreSQL was developed, it used BSD file systems, and those prefer 8k blocks, and there was some concept that an 8k write was atomic, though with 512 byte disk

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-24 Thread mlw
Curt Sampson wrote: On Tue, 23 Apr 2002, mlw wrote: On a system that has neither read-ahead nor sorting of I/O requests, yes. Which systems are you using that provide neither of these facilities? This only happens if the OS can organize the I/O requests in such a manner. It is

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-24 Thread Luis Alberto Amigo Navarro
] Index Scans become Seq Scans after VACUUM ANALYSE Luis Alberto Amigo Navarro wrote: Hi All. I've been reading all the thread and I want to add a few points: You can set enable_seqscan=off in small or easy queries, but in large queries index can speed parts of the query and slow other

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-24 Thread Curt Sampson
On Wed, 24 Apr 2002, mlw wrote: I am not arguing about whether or not they do it, I am saying it is not always possible. I/O requests do not remain in queue waiting for reordering indefinitely. It doesn't matter. When they go out to the disk they go out in order. On every Unix-based OS I

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-24 Thread Michael Loftis
A Block-sized read will not be rboken up. But if you're reading ina size bigger than the underlying systems block sizes then it can get broken up. So yes a sequential read will get broken up. A single read request for a block may or may not get broken up. If you're freading with set block

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-24 Thread Curt Sampson
On Wed, 24 Apr 2002, Michael Loftis wrote: A Block-sized read will not be rboken up. But if you're reading ina size bigger than the underlying systems block sizes then it can get broken up. In which operating systems, and under what circumstances? I'll agree that some OSs may not coalesce

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-23 Thread Bruce Momjian
Lincoln Yeoh wrote: At 10:48 AM 4/18/02 -0400, mlw wrote: Bruce Momjian wrote: Have you tried reducing 'random_page_cost' in postgresql.conf. That should solve most of your problems if you would like more index scans. My random page cost is 1 :-) What happens when you set random

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-23 Thread Bruce Momjian
Luis Alberto Amigo Navarro wrote: Hi All. I've been reading all the thread and I want to add a few points: You can set enable_seqscan=off in small or easy queries, but in large queries index can speed parts of the query and slow other, so I think it is neccesary if you want Postgres to

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-23 Thread mlw
Lincoln Yeoh wrote: At 10:48 AM 4/18/02 -0400, mlw wrote: Bruce Momjian wrote: Have you tried reducing 'random_page_cost' in postgresql.conf. That should solve most of your problems if you would like more index scans. My random page cost is 1 :-) What happens when you set

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-23 Thread mlw
Curt Sampson wrote: On Thu, 18 Apr 2002, Michael Loftis wrote: mlw wrote: The supposed advantage of a sequential read over an random read, in an active multitasking system, is a myth. If you are executing one query and the system is doing only that query, you may be right.

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-23 Thread Curt Sampson
On Tue, 23 Apr 2002, mlw wrote: On a system that has neither read-ahead nor sorting of I/O requests, yes. Which systems are you using that provide neither of these facilities? This only happens if the OS can organize the I/O requests in such a manner. It is a non-trivial function.

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-22 Thread Luis Alberto Amigo Navarro
Hi All. I've been reading all the thread and I want to add a few points: You can set enable_seqscan=off in small or easy queries, but in large queries index can speed parts of the query and slow other, so I think it is neccesary if you want Postgres to become a Wide-used DBMS that the planner

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-19 Thread Curt Sampson
On Thu, 18 Apr 2002, mlw wrote: The days when head movement is relevant are long over. Not a single drive sold today, or in the last 5 years, is a simple spindle/head system. The assumption that sequentially reading a file from a modern disk drive means that the head will move less

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-19 Thread Curt Sampson
On Thu, 18 Apr 2002, Michael Loftis wrote: mlw wrote: The supposed advantage of a sequential read over an random read, in an active multitasking system, is a myth. If you are executing one query and the system is doing only that query, you may be right. Execute a number of queries at the

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-19 Thread Maarten . Boekhold
On 04/18/2002 12:41:15 PM tycho wrote: Don't know if the optimizer takes this into consideration, but a query that uses a primary and/or unique key in the where-clause, should always choose to use the related indices (assuming the table size is above a certain threshold). Since a

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-19 Thread Lincoln Yeoh
At 10:48 AM 4/18/02 -0400, mlw wrote: Bruce Momjian wrote: Have you tried reducing 'random_page_cost' in postgresql.conf. That should solve most of your problems if you would like more index scans. My random page cost is 1 :-) What happens when you set random page cost to 1? Between an

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-19 Thread Tom Lane
Lincoln Yeoh [EMAIL PROTECTED] writes: ...By the way, are updates treated the same as selects by the optimizer? Yeah. The writes must occur in any case, so I see no reason why the optimizer should worry about them. All it needs to consider are the cycles used by the various alternatives for

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Luis Alberto Amigo Navarro
Look at the pgbench utility. I can't run that program without a +- 10% variation from run to run, no mater how many times I run vacuum and checkpoint. It's pgbench's fault, TPC-B was replaced with TPC-C because it is not accurate enough, we run a pseudo TPC-H and it has almost no variations

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread tycho
On Thu, 18 Apr 2002 [EMAIL PROTECTED] wrote: On 04/17/2002 01:44:46 PM Michael Loftis wrote: In many of the cases where it is a primary key it is also there to ensure fast lookups when referenced as a foreign key.  Or for joins. Don't know if the optimizer takes this into

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Mark Pritchard
Apologies for the naff double post, but I meant to add that obviously the figures for the solaris box are bogus after the first run...imagine a file system cache of an entire 2gb file. I tried creating a file of 4gb on this box, but it bombed with a file too large error. Unfortunately, I can't

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Tom Lane
mlw [EMAIL PROTECTED] writes: For instance: say we have two similarly performing plans, close to one another, say within 20%, one plan uses an index, and one does not. It is unlikely that the index plan will perform substantially worse than the non-index plan, right? This seems to be the crux

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Tom Lane
mlw [EMAIL PROTECTED] writes: should we not just allow the developer to place hints in the SQL, as: select /*+ INDEX(a_id, b_id) */ * from a, b where a.id = b.id; itch People have suggested that sort of thing from time to time, but I have a couple of problems with it: 1. It's unobvious

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread mlw
Tom Lane wrote: mlw [EMAIL PROTECTED] writes: For instance: say we have two similarly performing plans, close to one another, say within 20%, one plan uses an index, and one does not. It is unlikely that the index plan will perform substantially worse than the non-index plan, right?

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Adrian 'Dagurashibanipal' von Bidder
On Wed, 2002-04-17 at 19:43, Tom Lane wrote: Hannu Krosing [EMAIL PROTECTED] writes: OTOH, it is also important where the file is on disk. As seen from disk speed test graphs on http://www.tomshardware.com , the speed difference of sequential reads is 1.5 to 2.5 between inner and outer

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread mlw
Tom Lane wrote: By and large this argument reminds me of the compiler versus hand- programmed assembler argument. Which was pretty much a dead issue when I was an undergrad, more years ago than I care to admit in a public forum. Yes, a competent programmer who's willing to work hard can

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Bruce Momjian
mlw wrote: I don't think we will agree, we have seen different behaviors, and our experiences seem to conflict. This however does not mean that either of us is in error, it just may mean that we use data with very different characteristics. This thread is kind of frustrating for me because

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Bruce Momjian
Tom Lane wrote: 2. The tag approach presumes that the query programmer is smarter than the planner. This might be true under ideal circumstances, but I have a hard time crediting that the planner looking at today's stats is dumber than the junior programmer who left two years ago, and no

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Michael Loftis
Finally someone writes down whats been itching at my brain for a while. In a multi-tasking system it's always cheaper to fetch less blocks, no matter where they are. Because, as you said, it will end up more or less random onf a system experiencing a larger number of queries. mlw wrote:

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Michael Loftis
Somethings wrong with the random numbers from the sun... re-run them, that first sample is insane Caching looks like it's affecctign your results alot... Mark Pritchard wrote: I threw together the attached program (compiles fine with gcc 2.95.2 on Solaris 2.6 and egcs-2.91.66 on RedHat

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread mlw
Bruce Momjian wrote: mlw wrote: I don't think we will agree, we have seen different behaviors, and our experiences seem to conflict. This however does not mean that either of us is in error, it just may mean that we use data with very different characteristics. This thread is kind

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Tom Lane
Michael Loftis [EMAIL PROTECTED] writes: Somethings wrong with the random numbers from the sun... re-run them, that first sample is insane Caching looks like it's affecctign your results alot... Yeah; it looks like the test case is not large enough to swamp out caching effects on the

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Thomas Lockhart
... My one most important experience (I've had more than one) with this whole topic is DMN's music database, when PostgreSQL uses the index, the query executes in a fraction of a second. When enable_seqscan=true PostgreSQL refuses to use the index, and the query takes a about a minute. No

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Bruce Momjian
mlw wrote: Bruce Momjian wrote: mlw wrote: I don't think we will agree, we have seen different behaviors, and our experiences seem to conflict. This however does not mean that either of us is in error, it just may mean that we use data with very different characteristics.

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Tom Lane
mlw [EMAIL PROTECTED] writes: My one most important experience (I've had more than one) with this whole topic is DMN's music database, when PostgreSQL uses the index, the query executes in a fraction of a second. When enable_seqscan=true PostgreSQL refuses to use the index, and the query

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-18 Thread Michael Loftis
Got some numbers now... You'll notice the Random reads are *really* slow. The reason for htis is the particular read sizes that are ebing used are the absolute worst-case for my particular configuration. (wiht a 32kb or 64kb block size I generally achieve much higher performance even on

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Luis Alberto Amigo Navarro
On Wed, 2002-04-17 at 06:51, mlw wrote: I just think there is sufficient evidence to suggest that if a DBA creates an index, there is strong evidence (better than statistics) that the index need be used. In the event that an index exists, there is a strong indication that, without

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Christopher Kings-Lynne
If the DBA notices that there is a problem with a query, he adds an index, he notices that there is no difference, then he notices that PostgreSQL is not using his index. First and foremost he gets mad at PostgreSQL for not using his index. If PostgreSQL decided to use an index which

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Michael Loftis
My opinion. Expose some of the cost factors via run-time settings (or start-time settings). This would allow those who wanted to 'tweak' the planner to do so and those that felt the defaults were fine or didn't know to leave them alone. Comments? ---(end of

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Michael Loftis
Oliver Elphick wrote: On Wed, 2002-04-17 at 06:51, mlw wrote: I just think there is sufficient evidence to suggest that if a DBA creates an index, there is strong evidence (better than statistics) that the index need be used. In the event that an index exists, there is a strong indication

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Tom Lane
Hannu Krosing [EMAIL PROTECTED] writes: OTOH, it is also important where the file is on disk. As seen from disk speed test graphs on http://www.tomshardware.com , the speed difference of sequential reads is 1.5 to 2.5 between inner and outer tracks. True. But if we use the same test file

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Tom Lane
mlw [EMAIL PROTECTED] writes: ... I have seen many instances of when PostgreSQL refuses to use an index because the data distribution is uneven. This is fixed, or at least attacked, in 7.2. Again, I do not see this as an argument for making the planner stupider instead of smarter.

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread mlw
Tom Lane wrote: mlw [EMAIL PROTECTED] writes: ... I have seen many instances of when PostgreSQL refuses to use an index because the data distribution is uneven. This is fixed, or at least attacked, in 7.2. Again, I do not see this as an argument for making the planner stupider instead

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Andrew Sullivan
On Wed, Apr 17, 2002 at 12:35:13PM -0400, mlw wrote: about a 10 vs 8 second difference. I have seen many instances of when PostgreSQL refuses to use an index because the data distribution is uneven. Making it more difficult for the planer to ignore an index would solve practically all the

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Peter Eisentraut
mlw writes: Adding huristics, such as weighting for index scans, is not making the planner stupider. It is making it smarter and more flexable. If life was as simple as index or no index then this might make some sense. But in general the planner has a whole bunch of choices of join plans,

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread mlw
Andrew Sullivan wrote: Given the apparent infrequency of docs-consultation, I am considerably less sanguine than you are about the correctness of the choices many DBAs make. Poking at the planner to make it use an index more often strikes me as at least as likely to cause worse performance.

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread mlw
Peter Eisentraut wrote: mlw writes: Adding huristics, such as weighting for index scans, is not making the planner stupider. It is making it smarter and more flexable. If life was as simple as index or no index then this might make some sense. But in general the planner has a whole

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Rod Taylor
Lane [EMAIL PROTECTED]; Bruce Momjian [EMAIL PROTECTED]; Louis-David Mitterrand [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Wednesday, April 17, 2002 10:31 AM Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE Thomas Lockhart wrote: Systems which have optimizing planners

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread mlw
Andrew Sullivan wrote: You haven't shown anything except a couple of anecdotal reports as evidence against his view. Anyone who asks you for more evidence gets treated to a remark that statistics won't do everything in this case. I do not, currently, have access to systems which exhibit

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Bruce Momjian
mlw wrote: Now, given the choice of the two strategies on a table, both pretty close to one another, the risk of poor performance for using the index scan is minimal based on the statistics, but the risk of poor performance for using the sequential scan is quite high on a large table. Wow,

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread mlw
Andrew Sullivan wrote: Now, given the choice of the two strategies on a table, both pretty close to one another, the risk of poor performance for using the index scan is minimal based on the statistics, but the risk of poor performance for using the sequential scan is quite high on a

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Bruce Momjian
mlw wrote: Andrew Sullivan wrote: Now, given the choice of the two strategies on a table, both pretty close to one another, the risk of poor performance for using the index scan is minimal based on the statistics, but the risk of poor performance for using the sequential scan is

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread mlw
Bruce Momjian wrote: mlw wrote: Now, given the choice of the two strategies on a table, both pretty close to one another, the risk of poor performance for using the index scan is minimal based on the statistics, but the risk of poor performance for using the sequential scan is quite

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Bill Cunningham
Tom Lane wrote: mlw [EMAIL PROTECTED] writes: That is the difference, in another post Tom said he could not get excited about 10.9 second execution time over a 7.96 execution time. Damn!!! I would. That is wrong. Sure. Show us how to make the planner's estimates 2x more accurate (on average)

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Bruce Momjian
mlw wrote: Bruce Momjian wrote: mlw wrote: Now, given the choice of the two strategies on a table, both pretty close to one another, the risk of poor performance for using the index scan is minimal based on the statistics, but the risk of poor performance for using the

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Bruce Momjian
Michael Loftis wrote: As far as the 'planner benchmark suite' so we cans tart gathering more statistical data about what costs should be, or are better at, that's an excellent idea. People with different hardware have different random page costs, clearly. Even different workloads affect

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread mlw
Bruce Momjian wrote: My second point, that index scan is more risky than sequential scan, is outlined above. A sequential scan reads each page once, and uses the file system read-ahead code to prefetch the disk buffers. Index scans are random, and could easily re-read disk pages to plow

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Bruce Momjian
mlw wrote: Bruce Momjian wrote: My second point, that index scan is more risky than sequential scan, is outlined above. A sequential scan reads each page once, and uses the file system read-ahead code to prefetch the disk buffers. Index scans are random, and could easily re-read disk

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Doug McNaught
mlw [EMAIL PROTECTED] writes: It took a bike ride to think about this one. The supposed advantage of a sequential read over an random read, in an active multitasking system, is a myth. Disagree. Execute a number of queries at the same time, the expected benefit of a sequential scan goes

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread D'Arcy J.M. Cain
On April 17, 2002 05:44 pm, mlw wrote: It took a bike ride to think about this one. The supposed advantage of a sequential read over an random read, in an active multitasking system, is a myth. If you are executing one query and the system is doing only that query, you may be right. Execute

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread mlw
Bruce Momjian wrote: mlw wrote: Bruce Momjian wrote: mlw wrote: Now, given the choice of the two strategies on a table, both pretty close to one another, the risk of poor performance for using the index scan is minimal based on the statistics, but the risk of poor

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Dann Corbit
-Original Message- From: Bruce Momjian [mailto:[EMAIL PROTECTED]] Sent: Wednesday, April 17, 2002 2:39 PM To: mlw Cc: Andrew Sullivan; PostgreSQL-development; Tom Lane Subject: Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE mlw wrote: Bruce Momjian wrote: mlw wrote

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread mlw
Bruce Momjian wrote: OK, yes, sequential scan _can_ be as slow as index scan, but sometimes it is faster. Can you provide reasoning why index scan should be preferred, other than the admin created it, which I already addressed? If you have a choice between two or more sub-plans, similar

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Bruce Momjian
mlw wrote: Bruce Momjian wrote: OK, yes, sequential scan _can_ be as slow as index scan, but sometimes it is faster. Can you provide reasoning why index scan should be preferred, other than the admin created it, which I already addressed? If you have a choice between two or more

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Bruce Momjian
D'Arcy J.M. Cain wrote: On April 17, 2002 05:44 pm, mlw wrote: It took a bike ride to think about this one. The supposed advantage of a sequential read over an random read, in an active multitasking system, is a myth. If you are executing one query and the system is doing only that

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Hannu Krosing
On Wed, 2002-04-17 at 22:43, Tom Lane wrote: Hannu Krosing [EMAIL PROTECTED] writes: OTOH, it is also important where the file is on disk. As seen from disk speed test graphs on http://www.tomshardware.com , the speed difference of sequential reads is 1.5 to 2.5 between inner and outer

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Sean Chittenden
... I have seen many instances of when PostgreSQL refuses to use an index because the data distribution is uneven. This is fixed, or at least attacked, in 7.2. Again, I do not see this as an argument for making the planner stupider instead of smarter. Could someone fork out some decent

Re: [HACKERS] Index Scans become Seq Scans after VACUUM ANALYSE

2002-04-17 Thread Mark Pritchard
I threw together the attached program (compiles fine with gcc 2.95.2 on Solaris 2.6 and egcs-2.91.66 on RedHat Linux 6.2) and ran it a few times. Data is below. Usual disclaimers about hastily written code etc :) Machine = ghoul (generic intel, 384mb ram, dual p3-800, ide disk running dma)

  1   2   >