Re: [PERFORM] Query Plan choice with timestamps

2008-08-07 Thread Giorgio Valoti
On 07/ago/08, at 23:01, Tom Lane wrote: Giorgio Valoti <[EMAIL PROTECTED]> writes: On 07/ago/08, at 17:50, Tom Lane wrote: These numbers seem pretty bogus: there is hardly any scenario in which a full-table indexscan should be costed as significantly cheaper than a seqscan. Have you put i

Re: [PERFORM] query planner not using the correct index

2008-08-07 Thread Tom Lane
"Joshua Shanks" <[EMAIL PROTECTED]> writes: > How do I increase the stats target for just one column? Look under ALTER TABLE. regards, tom lane -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.

Re: [PERFORM] query planner not using the correct index

2008-08-07 Thread Joshua Shanks
Yeah with default_statistics_target at 500 most_common_vals had 4 values with the fourth having a frequency of 1.5% and distinct have 250+ in it. How do I increase the stats target for just one column? On Thu, Aug 7, 2008 at 6:48 PM, Tom Lane <[EMAIL PROTECTED]> wrote: > "Joshua Shanks" <[EMAIL P

Re: [PERFORM] query planner not using the correct index

2008-08-07 Thread Tom Lane
"Joshua Shanks" <[EMAIL PROTECTED]> writes: > SELECT null_frac, n_distinct, most_common_vals, most_common_freqs FROM > pg_stats WHERE tablename = 'bars' AND attname='bars_id'; > null_frac | n_distinct | most_common_vals | most_common_freqs > ---++--+

Re: [PERFORM] query planner not using the correct index

2008-08-07 Thread Joshua Shanks
On Thu, Aug 7, 2008 at 5:38 PM, Gregory Stark <[EMAIL PROTECTED]> wrote: > Measuring n_distinct from a sample is inherently difficult and unreliable. > When 98% of your table falls into those categories it's leaving very few > chances for the sample to find many other distinct values. > > I haven't

Re: [PERFORM] query planner not using the correct index

2008-08-07 Thread Gregory Stark
"Joshua Shanks" <[EMAIL PROTECTED]> writes: > Those 3 values in reality and in the stats account for 98% of the > rows. actual distinct values are around 350 Measuring n_distinct from a sample is inherently difficult and unreliable. When 98% of your table falls into those categories it's leaving

Re: [PERFORM] file system and raid performance

2008-08-07 Thread Mark Mielke
Andrej Ricnik-Bay wrote: 2008/8/8 Scott Marlowe <[EMAIL PROTECTED]>: noatime turns off the atime write behaviour. Or did you already know that and I missed some weird post where noatime somehow managed to slow down performance? Scott, I'm quite aware of what noatime does ... you didn'

Re: [PERFORM] file system and raid performance

2008-08-07 Thread Scott Marlowe
On Thu, Aug 7, 2008 at 3:57 PM, Andrej Ricnik-Bay <[EMAIL PROTECTED]> wrote: > 2008/8/8 Scott Marlowe <[EMAIL PROTECTED]>: >> noatime turns off the atime write behaviour. Or did you already know >> that and I missed some weird post where noatime somehow managed to >> slow down performance? > > Sco

Re: [PERFORM] query planner not using the correct index

2008-08-07 Thread Joshua Shanks
On Thu, Aug 7, 2008 at 4:12 PM, Tom Lane <[EMAIL PROTECTED]> wrote: > Well, you haven't told us how big any of these tables are, so it's > hard to tell if the n_distinct value is wrong or not ... but in > any case I don't think that the stats on attr1 have anything to do > with your problem. The r

Re: [PERFORM] file system and raid performance

2008-08-07 Thread Gregory S. Youngblood
> -Original Message- > From: Mark Wong [mailto:[EMAIL PROTECTED] > Sent: Thursday, August 07, 2008 12:37 PM > To: Mario Weilguni > Cc: Mark Kirkwood; [EMAIL PROTECTED]; [EMAIL PROTECTED]; pgsql- > [EMAIL PROTECTED]; Gabrielle Roth > Subject: Re: [PERFORM] file system and raid performance >

Re: [PERFORM] file system and raid performance

2008-08-07 Thread Andrej Ricnik-Bay
2008/8/8 Scott Marlowe <[EMAIL PROTECTED]>: > noatime turns off the atime write behaviour. Or did you already know > that and I missed some weird post where noatime somehow managed to > slow down performance? Scott, I'm quite aware of what noatime does ... you didn't miss a post, but if you look

Re: [PERFORM] file system and raid performance

2008-08-07 Thread Scott Marlowe
On Thu, Aug 7, 2008 at 2:59 PM, Andrej Ricnik-Bay <[EMAIL PROTECTED]> wrote: > To me it still boggles the mind that noatime should actually slow down > activities on ANY file-system ... has someone got an explanation for > that kind of behaviour? As far as I'm concerned this means that even > to a

Re: [PERFORM] query planner not using the correct index

2008-08-07 Thread Tom Lane
"Joshua Shanks" <[EMAIL PROTECTED]> writes: > It seems obvious that the stats on attr1 at the current level are > inaccurate as there are over 100,000 unique enteries in the table. Well, you haven't told us how big any of these tables are, so it's hard to tell if the n_distinct value is wrong or n

Re: [PERFORM] Query Plan choice with timestamps

2008-08-07 Thread Tom Lane
Giorgio Valoti <[EMAIL PROTECTED]> writes: > On 07/ago/08, at 17:50, Tom Lane wrote: >> These numbers seem pretty bogus: there is hardly any scenario in >> which a >> full-table indexscan should be costed as significantly cheaper than a >> seqscan. Have you put in silly values for random_page_co

Re: [PERFORM] file system and raid performance

2008-08-07 Thread Andrej Ricnik-Bay
To me it still boggles the mind that noatime should actually slow down activities on ANY file-system ... has someone got an explanation for that kind of behaviour? As far as I'm concerned this means that even to any read I'll add the overhead of a write - most likely in a disk-location slightly of

Re: [PERFORM] file system and raid performance

2008-08-07 Thread Mark Wong
On Thu, Aug 7, 2008 at 3:21 AM, Mario Weilguni <[EMAIL PROTECTED]> wrote: > Mark Kirkwood schrieb: >> >> Mark Kirkwood wrote: >>> >>> You are right, it does (I may be recalling performance from my other >>> machine that has a 3Ware card - this was a couple of years ago...) Anyway, >>> I'm thinking

Re: [PERFORM] Query Plan choice with timestamps

2008-08-07 Thread Giorgio Valoti
On 07/ago/08, at 20:37, Giorgio Valoti wrote: […] If you haven't mucked with the cost parameters, the only way I can think of to get this result is to have an enormously bloated table that's mostly empty. Maybe you need to review your vacuuming procedures. I’ll review them. I’ve ma

Re: [PERFORM] Query Plan choice with timestamps

2008-08-07 Thread Giorgio Valoti
On 07/ago/08, at 14:36, Richard Huxton wrote: Giorgio Valoti wrote: On 07/ago/08, at 10:35, Richard Huxton wrote: Giorgio Valoti wrote: Hi, I have a timestamptz field that I want to use with a query, but I don’t need the full timestamp resolution, so I’ve created a day_trunc(timestamptz)

Re: [PERFORM] Query Plan choice with timestamps

2008-08-07 Thread Giorgio Valoti
On 07/ago/08, at 17:50, Tom Lane wrote: Giorgio Valoti <[EMAIL PROTECTED]> writes: GroupAggregate (cost=98431.58..119773.92 rows=74226 width=8) -> Sort (cost=98431.58..99050.92 rows=247736 width=8) Sort Key: (day_trunc(ts)) -> Seq Scan on blackbox (cost=0.00..72848.3

Re: [PERFORM] Query Plan choice with timestamps

2008-08-07 Thread Tom Lane
Giorgio Valoti <[EMAIL PROTECTED]> writes: > GroupAggregate (cost=98431.58..119773.92 rows=74226 width=8) > -> Sort (cost=98431.58..99050.92 rows=247736 width=8) > Sort Key: (day_trunc(ts)) > -> Seq Scan on blackbox (cost=0.00..72848.36 rows=247736 width=8) > Group

Re: [PERFORM] query planner not using the correct index

2008-08-07 Thread Joshua Shanks
On Wed, Aug 6, 2008 at 10:24 PM, Craig Ringer <[EMAIL PROTECTED]> wrote: > OK, that's interesting. There are ways to examine Pg's statistics on > columns, get an idea of which stats might be less than accurate, etc, > but I'm not really familiar enough with it all to give you any useful > advice on

Re: [PERFORM] Plz Heeeelp! performance settings

2008-08-07 Thread Richard Huxton
dforums wrote: > The delete is global, the procedure is called for each line/tracks. > > So - are you calling this function 14000 times to inject your data? > > You're doing this in one transaction, yes? > NO I have to make it 14000 times cause, I use some inserted information > for other insert

Re: [PERFORM] Plz Heeeelp! performance settings

2008-08-07 Thread Merlin Moncure
On Thu, Aug 7, 2008 at 9:30 AM, dforums <[EMAIL PROTECTED]> wrote: > The performance problem is really only on the insertion and even more on the > treatment for the aggregation. > > To treat the 3000 entrances and to insert, or update the tables it needs 10 > minutes. > > As I told you I inject 14

[PERFORM] Another index related question....

2008-08-07 Thread ries van Twisk
Hey all, I have two tables that look like this: CREATE TABLE details ( cust_code character varying(6) NOT NULL, cust_po character varying(20) NOT NULL, date_ordd date NOT NULL, item_nbr integer NOT NULL, orig_qty_ordd integer, CONSTRAINT details_pkey PRIMARY KEY (cust_code, cust_po,

Re: [PERFORM] Unexpectedly Long DELETE Wait

2008-08-07 Thread Volkan YAZICI
On Thu, 07 Aug 2008, Richard Huxton <[EMAIL PROTECTED]> writes: > Volkan YAZICI wrote: >> DELETE FROM mugpsreglog >> WHERE NOT EXISTS (SELECT 1 >> FROM mueventlog >>WHERE mueventlog.eventlogid = >> mugpsreglog.eventlogid); >> >>

Re: [PERFORM] Plz Heeeelp! performance settings

2008-08-07 Thread Richard Huxton
dforums wrote: The performance problem is really only on the insertion and even more on the treatment for the aggregation. To treat the 3000 entrances and to insert, or update the tables it needs 10 minutes. As I told you I inject 14000 query every 2 minutes, and it needs 10 minutes to trea

Re: [PERFORM] Plz Heeeelp! performance settings

2008-08-07 Thread dforums
The performance problem is really only on the insertion and even more on the treatment for the aggregation. To treat the 3000 entrances and to insert, or update the tables it needs 10 minutes. As I told you I inject 14000 query every 2 minutes, and it needs 10 minutes to treat 3000 of those

Re: [PERFORM] Query Plan choice with timestamps

2008-08-07 Thread Richard Huxton
Giorgio Valoti wrote: On 07/ago/08, at 10:35, Richard Huxton wrote: Giorgio Valoti wrote: Hi, I have a timestamptz field that I want to use with a query, but I don’t need the full timestamp resolution, so I’ve created a day_trunc(timestamptz) immutable function which I’ll use with the query

Re: [PERFORM] Query Plan choice with timestamps

2008-08-07 Thread Giorgio Valoti
On 07/ago/08, at 10:35, Richard Huxton wrote: Giorgio Valoti wrote: Hi, I have a timestamptz field that I want to use with a query, but I don’t need the full timestamp resolution, so I’ve created a day_trunc(timestamptz) immutable function which I’ll use with the query and with a new inde

Re: [PERFORM] Plz Heeeelp! performance settings

2008-08-07 Thread Richard Huxton
dforums wrote: vmstat is giving : procs ---memory-- ---swap-- -io --system-- cpu r b swpd free buff cache si sobibo incs us sy id wa 0 2 1540 47388 41684 757897600 131 2590 1 9 3 82 7 This system is pract

Re: [PERFORM] file system and raid performance

2008-08-07 Thread Mario Weilguni
Mark Kirkwood schrieb: Mark Kirkwood wrote: You are right, it does (I may be recalling performance from my other machine that has a 3Ware card - this was a couple of years ago...) Anyway, I'm thinking for the Hardware raid tests they may need to be specified. FWIW - of course this somewha

[PERFORM] Filesystem setup on new system

2008-08-07 Thread Henrik
Hi list, I'm helping a customer with their new postgresql server and have some questions. The servers is connected to a SAN with dual raid cards which all have 512MB cache with BBU. The configuration they set up is now. 2 SAS 15K drives in RAID 1 on the internal controller for OS. 6 SAS

Re: [PERFORM] file system and raid performance

2008-08-07 Thread Mark Kirkwood
Mark Kirkwood wrote: You are right, it does (I may be recalling performance from my other machine that has a 3Ware card - this was a couple of years ago...) Anyway, I'm thinking for the Hardware raid tests they may need to be specified. FWIW - of course this somewhat academic given that th

Re: [PERFORM] Plz Heeeelp! performance settings

2008-08-07 Thread dforums
Richard Huxton a écrit : > dforum wrote: >> Tx for your reply. >> >> You mean that RAID use fsync method for keeping data's copy. > > No, Merlin means PostgreSQL will issue a sync to force WAL to actual disk. > >> So you invite me to desactivate fsync to increase the performance ? > > He means

Re: [PERFORM] Query Plan choice with timestamps

2008-08-07 Thread Richard Huxton
Giorgio Valoti wrote: Hi, I have a timestamptz field that I want to use with a query, but I don’t need the full timestamp resolution, so I’ve created a day_trunc(timestamptz) immutable function which I’ll use with the query and with a new index: logs=> create index test_idx on blackbox (day_t

Re: [PERFORM] Plz Heeeelp! performance settings

2008-08-07 Thread Richard Huxton
dforum wrote: Tx for your reply. You mean that RAID use fsync method for keeping data's copy. No, Merlin means PostgreSQL will issue a sync to force WAL to actual disk. So you invite me to desactivate fsync to increase the performance ? He means you might have to if you can't afford new ha

Re: [PERFORM] Unexpectedly Long DELETE Wait

2008-08-07 Thread Richard Huxton
Volkan YAZICI wrote: Hi, Below command has been running since ~700 minutes in one of our PostgreSQL servers. DELETE FROM mugpsreglog WHERE NOT EXISTS (SELECT 1 FROM mueventlog WHERE mueventlog.eventlogid = mugpsreglog.eventlogid)

[PERFORM] Query Plan choice with timestamps

2008-08-07 Thread Giorgio Valoti
Hi, I have a timestamptz field that I want to use with a query, but I don’t need the full timestamp resolution, so I’ve created a day_trunc(timestamptz) immutable function which I’ll use with the query and with a new index: logs=> create index test_idx on blackbox (day_trunc(ts)); However,

Re: [PERFORM] Plz Heeeelp! performance settings

2008-08-07 Thread dforum
Tx for your reply. You mean that RAID use fsync method for keeping data's copy. So you invite me to desactivate fsync to increase the performance ? Desactivating fsync. my second disk will not be uptodate, so if the machine crash, I wont be able to get the server working quickly??? But if I u