Re: [PERFORM] performance of implicit join vs. explicit conditions on inet queries?

2005-10-31 Thread Qingqing Zhou
Robert Edmonds [EMAIL PROTECTED] wrote EXPLAIN ANALYZE SELECT * FROM inet_addresses WHERE addr inet('10.2.0.0/24') OR addr inet('10.4.0.0/24') OR addr inet('10.8.0.0/24'); Bitmap Heap Scan on inet_addresses (cost=6.51..324.48 rows=1792335 width=11) (actual time=0.350..1.104

Re: [PERFORM] Best way to check for new data.

2005-10-31 Thread Merlin Moncure
There are a few ways to do this...thinking about it a bit, I would add a timestamp column to your log table (indexed) and keep a control table which keeps track of the last log print sweep operation. The print operation would just do select * from log where logtime (select lastlogtime());

Re: [PERFORM] performance of implicit join vs. explicit conditions on inet queries?

2005-10-31 Thread Tom Lane
Qingqing Zhou [EMAIL PROTECTED] writes: Robert Edmonds [EMAIL PROTECTED] wrote Instead of specifying explicit address ranges in the query, I'd like to store the ranges in a table: Good illustration. I guess we have a problem of the historgram statistical information. No, that's completely

[PERFORM] 8.x index insert performance

2005-10-31 Thread Kelly Burkhart
Greetings, We are running some performance tests in which we are attempting to insert about 100,000,000 rows in a database at a sustained rate. About 50M rows in, our performance drops dramatically. This test is with data that we believe to be close to what we will encounter in production.

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Merlin Moncure
Kelly wrote: We are running some performance tests in which we are attempting to insert about 100,000,000 rows in a database at a sustained rate. About 50M rows in, our performance drops dramatically. This test is with data that we believe to be close to what we will encounter in

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Kelly Burkhart
On Mon, 2005-10-31 at 12:32 -0500, Merlin Moncure wrote: well, can you defer index generation until after loading the set (or use COPY?) I cannot defer index generation. We are using the copy API. Copying 1 rows in a batch. if that index is causing the problem, you may want to

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Claus Guttesen
We are running some performance tests in which we are attempting to insert about 100,000,000 rows in a database at a sustained rate. About 50M rows in, our performance drops dramatically. This test is with data that we believe to be close to what we will encounter in production. However in

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Merlin Moncure
if that index is causing the problem, you may want to consider setting up partial index to exclude null values. This is a single column index. I assumed that null column values were not indexed. Is my assumption incorrect? -K It turns out it is, or it certainly seems to be. I didn't

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread mark
On Mon, Oct 31, 2005 at 12:32:03PM -0500, Merlin Moncure wrote: if that index is causing the problem, you may want to consider setting up partial index to exclude null values. Hey all. Pardon my ignorance. :-) I've been trying to figure out whether null values are indexed or not from the

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Merlin Moncure
On Mon, Oct 31, 2005 at 12:32:03PM -0500, Merlin Moncure wrote: if that index is causing the problem, you may want to consider setting up partial index to exclude null values. Hey all. Pardon my ignorance. :-) I've been trying to figure out whether null values are indexed or not from

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Tom Lane
[EMAIL PROTECTED] writes: I've been trying to figure out whether null values are indexed or not from the documentation. I was under the impression, that null values are not stored in the index. You're mistaken, at least with regard to btree indexes. regards, tom lane

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Kelly Burkhart
On Mon, 2005-10-31 at 15:30 -0500, Tom Lane wrote: [EMAIL PROTECTED] writes: I've been trying to figure out whether null values are indexed or not from the documentation. I was under the impression, that null values are not stored in the index. You're mistaken, at least with regard to

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Merlin Moncure
[EMAIL PROTECTED] writes: I've been trying to figure out whether null values are indexed or not from the documentation. I was under the impression, that null values are not stored in the index. You're mistaken, at least with regard to btree indexes. hmm. I tried several different ways

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Scott Marlowe
On Mon, 2005-10-31 at 13:13, Merlin Moncure wrote: if that index is causing the problem, you may want to consider setting up partial index to exclude null values. This is a single column index. I assumed that null column values were not indexed. Is my assumption incorrect? -K

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Merlin Moncure
select * from sometable where somefield IS NULL won't work because IS is not a nomally indexible operator. Ah, I didn't know that. So there is no real reason not to exclude null values from all your indexes :). Reading Tom's recent comments everything is clear now. Instead of using your two

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Tom Lane
Merlin Moncure [EMAIL PROTECTED] writes: You're mistaken, at least with regard to btree indexes. hmm. I tried several different ways to filter/extract null values from an indexed key and got a seq scan every time. I said they were stored, not that you could query against them ;-) IS NULL

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread Tom Lane
Kelly Burkhart [EMAIL PROTECTED] writes: Ha! So I'm creating an index 98% full of nulls! Looks like this is easily fixed with partial indexes. Still, though, it's not immediately clear why you'd be seeing a severe dropoff in insert performance after 50M rows. Even though there are lots of

[PERFORM] 8.1beta3 performance

2005-10-31 Thread PostgreSQL
We're running 8.1beta3 on one server and are having ridiculous performance issues. This is a 2 cpu Opteron box and both processors are staying at 98 or 99% utilization processing not-that-complex queries. Prior to the upgrade, our I/O wait time was about 60% and cpu utilization rarely got

Re: [PERFORM] 8.x index insert performance

2005-10-31 Thread mark
On Mon, Oct 31, 2005 at 03:27:31PM -0500, Merlin Moncure wrote: On Mon, Oct 31, 2005 at 12:32:03PM -0500, Merlin Moncure wrote: if that index is causing the problem, you may want to consider setting up partial index to exclude null values. Hey all. Pardon my ignorance. :-) I've been

Re: [PERFORM] 8.1beta3 performance

2005-10-31 Thread Steinar H. Gunderson
On Mon, Oct 31, 2005 at 05:16:46PM -0600, PostgreSQL wrote: We're running 8.1beta3 on one server and are having ridiculous performance issues. This is a 2 cpu Opteron box and both processors are staying at 98 or 99% utilization processing not-that-complex queries. Prior to the upgrade,