[PERFORM] performance sol10 zone (fup)

2010-08-04 Thread Heiko L.
Hallo, Im running pg-8,pgpoolII on sol10-zone. After update sol10u7, queries on coltype timestamp are very slow. System: sparc, 2GB RAM This DB is a greylist-DB to fight spam. 500 connections should be easy. But 16 connection consum 10sec/query. On another system (sparc) only 1 sec.i s.

[PERFORM] How to improve: performance of query on postgresql 8.3 takes days

2010-08-04 Thread Dino Vliet
Dear postgresql list, I have some troubles generating data for a analysis task at hand. I have a table (table A) containing 5 million records and 28 number of attributes. This table is 461MB big if I copy it to a csv file. I want to create another table (table B) based

[PERFORM] Re: [GENERAL] How to improve: performance of query on postgresql 8.3 takes days

2010-08-04 Thread John R Pierce
On 07/29/10 2:58 PM, Dino Vliet wrote: Dear postgresql list, I have some troubles generating data for a analysis task at hand. I have a table (table A) containing 5 million records and 28 number of attributes. This table is 461MB big if I copy it to a csv file. I want to create another

[PERFORM] what does initplan operation in explain output mean?

2010-08-04 Thread Mark Rostron
We are running 8.3.10 64bit. Compare the plans below. They all do the same thing and delete from a table named work_active (about 500rows), which is a subset of work_unit (about 50m rows). I want to introduce range-partitions on work_unit.id column (serial pk), and I want constraint

Re: [PERFORM] [GENERAL] How to improve: performance of query on postgresql 8.3 takes days

2010-08-04 Thread Alban Hertroys
On 29 Jul 2010, at 23:58, Dino Vliet wrote: CREATE OR REPLACE FUNCTION agenttype1(a character) RETURNS integer AS LANGUAGE 'plpgsql' VOLATILE COST 100; CREATE OR REPLACE FUNCTION agenttype2(a character) RETURNS integer AS LANGUAGE 'plpgsql' VOLATILE COST 100; CREATE OR REPLACE

Re: [PERFORM] Testing Sandforce SSD

2010-08-04 Thread Hannu Krosing
On Tue, 2010-08-03 at 10:40 +0200, Yeb Havinga wrote: se note that the 10% was on a slower CPU. On a more recent CPU the difference was 47%, based on tests that ran for an hour. I am not surprised at all that reading and writing almost twice as much data from/to disk takes 47% longer. If less

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Hannu Krosing
On Mon, 2010-08-02 at 14:23 -0700, Peter Hussey wrote: I already had effective_cache_size set to 500MB. I experimented with lowering random_page_cost to 3 then 2. In case of fully cached database it is closer to 1. 2) Why is the setting of work_mem something left to the admin and/or

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Robert Haas
On Tue, Aug 3, 2010 at 3:03 AM, Hannu Krosing ha...@2ndquadrant.com wrote: In case of fully cached database it is closer to 1. In the case of a fully cached database I believe the correct answer begins with a decimal point. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The

Re: [PERFORM] Strange explain on partitioned tables

2010-08-04 Thread Robert Haas
On Mon, Jul 26, 2010 at 5:26 PM, Gerald Fontenay gvfonte...@lbl.gov wrote: Thank you for your response. So if I query only my target child table, this should be just like any other single table wrt planning right? I have thousands of these tables. (I suppose that I'm only using inheritance for

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Kevin Grittner
Peter Hussey pe...@labkey.com wrote: I already had effective_cache_size set to 500MB. That seems awfully small. You do realize that this setting does not cause PostgreSQL to allocate any memory; it merely advises how much disk space is likely to be cached. It should normally be set to the

[PERFORM] LIKE without wildcard different from =

2010-08-04 Thread Kevin Grittner
We have been using the C locale for everything at our site, but there is occasionally talk of supporting characters outside the ASCII7 set. In playing around with indexing, to see what the impact of that would be, I stumbled across something which was mildly surprising. In the C locale, if you

Re: [PERFORM] LIKE without wildcard different from =

2010-08-04 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: We have been using the C locale for everything at our site, but there is occasionally talk of supporting characters outside the ASCII7 set. In playing around with indexing, to see what the impact of that would be, I stumbled across something

Re: [PERFORM] LIKE without wildcard different from =

2010-08-04 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Kevin Grittner kevin.gritt...@wicourts.gov writes: We have been using the C locale for everything at our site, but there is occasionally talk of supporting characters outside the ASCII7 set. In playing around with indexing, to see what the impact of that

Re: [PERFORM] LIKE without wildcard different from =

2010-08-04 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: To be clear, though, the problem isn't that it didn't turn a LIKE with no wildcard characters into an equality test, it's that it would have been three orders of magnitude faster (because of an available index with an opclass specification) if

Re: [PERFORM] LIKE without wildcard different from =

2010-08-04 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote: Ah. Well, the real fix for that is also in 8.4: we got rid of the separate ~=~ operator, so a text_pattern_ops index is now usable for plain =. Nice! Thanks, -Kevin -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Hannu Krosing
On Wed, 2010-08-04 at 09:14 -0400, Robert Haas wrote: On Tue, Aug 3, 2010 at 3:03 AM, Hannu Krosing ha...@2ndquadrant.com wrote: In case of fully cached database it is closer to 1. In the case of a fully cached database I believe the correct answer begins with a decimal point. The number 1

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Tom Lane
Hannu Krosing ha...@2ndquadrant.com writes: Of course there are more variables than just *_page_cost, so if you nail down any other one, you may end with less than 1 for both page costs. I have always used seq_page_cost = 1 in my thinking and adjusted others relative to it. Right,

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Hannu Krosing
On Wed, 2010-08-04 at 14:00 -0400, Tom Lane wrote: Hannu Krosing ha...@2ndquadrant.com writes: Of course there are more variables than just *_page_cost, so if you nail down any other one, you may end with less than 1 for both page costs. I have always used seq_page_cost = 1 in my thinking

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Hannu Krosing
On Wed, 2010-08-04 at 21:41 +0300, Hannu Krosing wrote: On Wed, 2010-08-04 at 14:00 -0400, Tom Lane wrote: regression=# select name, setting from pg_settings where name like '%cost'; name | setting --+- cpu_index_tuple_cost | 0.005

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Greg Smith
Hannu Krosing wrote: There was ample space for keeping the indexes in linux cache (it has 1GB cached currently) though the system may have decided to start writing it to disk, so I suspect that most of the time was spent copying random index pages back and forth between shared buffers and disk

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Greg Smith
This time with attachment... -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us fsync-stress.sh Description: Bourne shell script -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Tom Lane
Hannu Krosing ha...@2ndquadrant.com writes: There was ample space for keeping the indexes in linux cache (it has 1GB cached currently) though the system may have decided to start writing it to disk, so I suspect that most of the time was spent copying random index pages back and forth between

Re: [PERFORM] Testing Sandforce SSD

2010-08-04 Thread Scott Carey
On Jul 26, 2010, at 12:45 PM, Greg Smith wrote: Yeb Havinga wrote: I did some ext3,ext4,xfs,jfs and also ext2 tests on the just-in-memory read/write test. (scale 300) No real winners or losers, though ext2 isn't really faster and the manual need for fix (y) during boot makes it

Re: [PERFORM] Testing Sandforce SSD

2010-08-04 Thread Scott Carey
On Aug 2, 2010, at 7:26 AM, Merlin Moncure wrote: On Fri, Jul 30, 2010 at 11:01 AM, Yeb Havinga yebhavi...@gmail.com wrote: After a week testing I think I can answer the question above: does it work like it's supposed to under PostgreSQL? YES The drive I have tested is the $435,- 50GB

Re: [PERFORM] Testing Sandforce SSD

2010-08-04 Thread Scott Carey
On Aug 3, 2010, at 9:27 AM, Merlin Moncure wrote: 2) I've heard that some SSD have utilities that you can use to query the write cycles in order to estimate lifespan. Does this one, and is it possible to publish the output (an approximation of the amount of work along with this would be

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Kevin Grittner
Greg Smith g...@2ndquadrant.com wrote: What I do to quantify that is...well, the attached shows it better than I can describe; only works on 9.0 or later as it depends on a feature I added for this purpose there. It measures exactly how much buffer cache churn happened during a test, in this

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Hannu Krosing
On Wed, 2010-08-04 at 22:03 +0300, Hannu Krosing wrote: On Wed, 2010-08-04 at 21:41 +0300, Hannu Krosing wrote: On Wed, 2010-08-04 at 14:00 -0400, Tom Lane wrote: regression=# select name, setting from pg_settings where name like '%cost'; name | setting

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Greg Smith
Kevin Grittner wrote: Assuming this is real, and not just in the noise -- what conclusions would you draw from this? Was trying to demonstrate the general ability of pg_stat_bgwriter snapshots at points in time to directly measure the buffer activity Hannu was theorizing about, not

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Greg Smith
Hannu Krosing wrote: Do you mean written to disk, or written out from shared_buffers to disk cache ? The later turns into the former eventually, so both really. The kernel will do some amount of write combining for you if you're lucky. But not in all cases; it may decide to write

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Kevin Grittner
Hannu Krosing ha...@2ndquadrant.com wrote: This seems to indicate about 1 ms for moving pages over user/system boundary. (Intel Core2 Duo T7500 @ 2.20GHz, Ubuntu 9.10, 4GB RAM) Using Greg's test script on a box with two cores like this: Intel(R) Pentium(R) D CPU 3.40GHz Linux

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Kevin Grittner
Greg Smith g...@2ndquadrant.com wrote: Was trying to demonstrate the general ability of pg_stat_bgwriter snapshots at points in time to directly measure the buffer activity Hannu was theorizing about, not necessarily show a useful benchmark of any sort with that. Ah, OK. Sorry I didn't

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Hannu Krosing
On Wed, 2010-08-04 at 15:16 -0400, Greg Smith wrote: Hannu Krosing wrote: There was ample space for keeping the indexes in linux cache (it has 1GB cached currently) though the system may have decided to start writing it to disk, so I suspect that most of the time was spent copying random

Re: [PERFORM] Questions on query planner, join types, and work_mem

2010-08-04 Thread Hannu Krosing
On Wed, 2010-08-04 at 22:03 +0300, Hannu Krosing wrote: On Wed, 2010-08-04 at 21:41 +0300, Hannu Krosing wrote: On Wed, 2010-08-04 at 14:00 -0400, Tom Lane wrote: regression=# select name, setting from pg_settings where name like '%cost'; name | setting

Re: [PERFORM] Testing Sandforce SSD

2010-08-04 Thread Chris Browne
g...@2ndquadrant.com (Greg Smith) writes: Yeb Havinga wrote: * What filesystem to use on the SSD? To minimize writes and maximize chance for seeing errors I'd choose ext2 here. I don't consider there to be any reason to deploy any part of a PostgreSQL database on ext2. The potential for

Re: [PERFORM] Testing Sandforce SSD

2010-08-04 Thread Chris Browne
j...@commandprompt.com (Joshua D. Drake) writes: On Sat, 2010-07-24 at 16:21 -0400, Greg Smith wrote: Greg Smith wrote: Note that not all of the Sandforce drives include a capacitor; I hope you got one that does! I wasn't aware any of the SF drives with a capacitor on them were even