Re: [PERFORM] Kernel Resources and max_connections

2005-05-03 Thread Mark Kirkwood
Chris Hebrard wrote: kern.ipc.shmmax and kern.ipc.shmmin will not stay to what I set them to. What am I doing wrong or not doing at all? These need to go in /etc/sysctl.conf. You might need to set shmall as well. (This not-very-clear distinction between what is sysctl'abe and what is a kernel tuna

Re: [PERFORM] Kernel Resources and max_connections

2005-05-03 Thread Mark Kirkwood
Chris Hebrard wrote: I set the values in etc/sysctl.conf: # $FreeBSD: src/etc/sysctl.conf,v 1.1.2.3 2002/04/15 00:44:13 dougb Exp $ # # This file is read when going to multi-user and its contents piped thru # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details. # # Added by IMP

Re: [PERFORM] tuning

2005-05-30 Thread Mark Kirkwood
list wrote: hi- i would like to see if someone could recommend something to make my query run faster. Values in postgresql.conf: shared_buffers = 1000 sort_mem is commented out effective_cache_size is commented out random_page_cost is commented out I would increase shared_buffers (say 5000

Re: [PERFORM] postgresql-8.0.1 performance tuning

2005-05-31 Thread Mark Kirkwood
Cosimo Streppone wrote: # Config /etc/sysctl.conf: kernel.shmall = 786432000 kernel.shmmax = 786432000 I think you have a problem here. kernel.shmmax should *not* be set to an amount of RAM, but to maximum number of shared memory pages, which on a typical linux system is

Re: [PERFORM] postgresql-8.0.1 performance tuning

2005-06-01 Thread Mark Kirkwood
Cosimo Streppone wrote: Mark Kirkwood ha scritto: Cosimo Streppone wrote: # Config /etc/sysctl.conf: kernel.shmall = 786432000 kernel.shmmax = 786432000 I think you have a problem here. kernel.shmmax should *not* be set to an amount of RAM, but Sorry, I thought

Re: SHMMAX / SHMALL Was (Re: [PERFORM] postgresql-8.0.1 performance

2005-06-02 Thread Mark Kirkwood
Paul McGarry wrote: Based on the powerpostgresql.com Performance Checklist [1] and Annotated Postgresql.conf [2] I understand that: -I should have less than 1/3 of my total memory as shared_buffers -For my server 15000 is a fairly reasonable starting point for shared_buffers which is ~120MB -I

Re: SHMMAX / SHMALL Was (Re: [PERFORM] postgresql-8.0.1 performance

2005-06-03 Thread Mark Kirkwood
Martin Fandel wrote: Aah ok :) I've set my values now as follow (2GB RAM): SHMMAX=`cat /proc/meminfo | grep MemTotal | cut -d: -f 2 | awk '{print $1*1024/3}'` echo kernel.shmmax=${SHMMAX} >> /etc/sysctl.conf SHMALL=`expr ${SHMALL} / 4096 \* \( 4096 / 16 \)` echo kernel.shmall=${SHMALL} >> /etc

Re: [PERFORM] Filesystem

2005-06-03 Thread Mark Kirkwood
Martin Fandel wrote: Hi @ all, i have only a little question. Which filesystem is preferred for postgresql? I'm plan to use xfs (before i used reiserfs). The reason is the xfs_freeze Tool to make filesystem-snapshots. Is the performance better than reiserfs, is it reliable? I used postgre

Re: [PERFORM] Updates on large tables are extremely slow

2005-06-12 Thread Mark Kirkwood
Yves Vindevogel wrote: I'm trying to update a table that has about 600.000 records. The update query is very simple : update mytable set pagesdesc = - pages ; The query takes about half an hour to an hour to execute. I have tried a lot of things. Half an hour seem a bit long - I would expe

Re: [PERFORM] Updates on large tables are extremely slow

2005-06-13 Thread Mark Kirkwood
19076 pages, 3000 rows sampled, 588209 estimated total rows ANALYZE On 13 Jun 2005, at 04:43, Mark Kirkwood wrote: Yves Vindevogel wrote: I'm trying to update a table that has about 600.000 records. The update query is very simple : update mytable set pagesdesc =

Re: [PERFORM] QRY seems not using indexes

2005-08-08 Thread Mark Kirkwood
Qingqing Zhou wrote: <[EMAIL PROTECTED]> writes so, if I do a qry like "EXPLAIN ANALYZE select * from pridecdr where idsede=8977758488" it tooks a lot of time before i get back any result: Index Scan using prd_id_sede on pridecdr (cost=0.00..699079.90 rows=181850 width=138) (actual time=51.2

Re: [PERFORM] Limit + group + join

2005-08-25 Thread Mark Kirkwood
Tobias, Interesting example: The 'desc' seems to be the guy triggering the sort, e.g: explain select c.id from c join b on c_id=c.id group by c.id order by c.id limit 5; QUERY PLAN ---

Re: [PERFORM] Limit + group + join

2005-08-26 Thread Mark Kirkwood
Tom Lane wrote: Mark Kirkwood <[EMAIL PROTECTED]> writes: What is interesting is why this plan is being rejected... Which PG version are you using exactly? That mistake looks like an artifact of the 8.0 "fuzzy plan cost" patch, which we fixed recently: http://archives.post

Re: [PERFORM] Limit + group + join

2005-08-26 Thread Mark Kirkwood
Interestingly enough, 7.4.8 and 8.1devel-2005-08-23 all behave the same as 8.0.3 for me (tables freshly ANALYZEd): joinlimit=# SELECT version(); version ---

Re: [PERFORM] Limit + group + join

2005-08-27 Thread Mark Kirkwood
Tom Lane wrote: I looked into this and found that indeed the desirable join plan was getting generated, but it wasn't picked because query_planner didn't have an accurate idea of how much of the join needed to be scanned to satisfy the GROUP BY step. I've committed some changes that hopefully w

Re: [PERFORM] Massive performance issues

2005-09-01 Thread Mark Kirkwood
It would be good to see EXPLAIN ANALYZE output for the three queries below (the real vs. estimated row counts being of interest). The number of pages in your address table might be interesting to know too. regards Mark Matthew Sackman wrote (with a fair bit of snippage): explain select local

Re: [PERFORM] Massive performance issues

2005-09-01 Thread Mark Kirkwood
Matthew Sackman wrote: I need to get to the stage where I can run queries such as: > select street, locality_1, locality_2, city from address where (city = 'Nottingham' or locality_2 = 'Nottingham' or locality_1 = 'Nottingham') and upper(substring(street from 1 for 1)) = 'A' group b

Re: [PERFORM] Massive performance issues

2005-09-02 Thread Mark Kirkwood
Mark Kirkwood wrote: Matthew Sackman wrote: I need to get to the stage where I can run queries such as: > select street, locality_1, locality_2, city from address where (city = 'Nottingham' or locality_2 = 'Nottingham' or locality_1 = 'Nottingham&#x

Re: [PERFORM] SELECT LIMIT 1 VIEW Performance Issue

2005-09-23 Thread Mark Kirkwood
K C Lau wrote: Thank you all for your suggestions. I' tried, with some variations too, but still no success. The times given are the best of a few repeated tries on an 8.1 beta 2 db freshly migrated from 8.0.3 on Windows. A small denormalization, where you mark the row with the latest atdate

Re: [PERFORM] SELECT LIMIT 1 VIEW Performance Issue

2005-09-23 Thread Mark Kirkwood
K C Lau wrote: I'm wondering if this performance issue is common enough for other users to merit a fix in pg, especially as it seems that with MVCC, each of the data records need to be accessed in addition to scanning the index. Yes - there are certainly cases where index only access (or so

Re: [PERFORM] Bytea poor performance

2005-10-15 Thread Mark Kirkwood
NSO wrote: Hello, Yes, I can understand that, but then why the same app on the server machine is done in 4 seconds? (big difference from 20-30 seconds). I tryed to monitor network traffic and it is used only for 1-2% of total 100mbit. Is this a web app? If so, then check you are using the s

Re: [PERFORM] Bytea poor performance

2005-10-17 Thread Mark Kirkwood
NSO wrote: Well, no. Delphi isn't better, same time just for downloading data... But as I told before, if for ex. pgAdminIII is running on server machine it is a lot faster, I do not know why, I was monitoring network connection between client and server and it is using only up to 2% of full spe

Re: [PERFORM] Used Memory

2005-10-23 Thread Mark Kirkwood
Christian Paul B. Cosinas wrote: Here is the result of “free” command” I am talking about. What does this result mean? I seem to recall the Linux man page for 'free' being most unenlightening, so have a look at: http://gentoo-wiki.com/FAQ_Linux_Memory_Management (For Gentoo, but should

Re: [PERFORM] Used Memory

2005-10-24 Thread Mark Kirkwood
Christian Paul B. Cosinas wrote: Hi To all those who replied. Thank You. I monitor my database server a while ago and found out that memory is used extensively when I am fetching records from the database. I use the command "fetch all" in my VB Code and put it in a recordset.Also in this command

Re: [PERFORM] Used Memory

2005-10-24 Thread Mark Kirkwood
Christian Paul B. Cosinas wrote: Hi mark I have so many functions, more than 100 functions in the database :) And I am dealing about 3 million of records in one database. And about 100 databases :) LOL - sorry, mis-understood your previous message to mean you had identified *one* query where

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-18 Thread Mark Kirkwood
Luke Lonergan wrote: (mass snippage) time psql -c "select count(*) from ivp.bigtable1" dgtestdb [EMAIL PROTECTED] IVP]$ cat sysout3 count -- 8000 (1 row) real1m9.875s user0m0.000s sys 0m0.004s [EMAIL PROTECTED] IVP]$ !

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-18 Thread Mark Kirkwood
Luke Lonergan wrote: Mark, On 11/18/05 3:46 PM, "Mark Kirkwood" <[EMAIL PROTECTED]> wrote: If you alter this to involve more complex joins (e.g 4. way star) and (maybe add a small number of concurrent executors too) - is it still the case? I may not have listened

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-19 Thread Mark Kirkwood
Luke Lonergan wrote: Mark, On 11/18/05 3:46 PM, "Mark Kirkwood" <[EMAIL PROTECTED]> wrote: If you alter this to involve more complex joins (e.g 4. way star) and (maybe add a small number of concurrent executors too) - is it still the case? 4-way star, same result, that&#x

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-20 Thread Mark Kirkwood
Mark Kirkwood wrote: - I am happy that seqscan is cpu bound after ~110M/s (It's cpu bound on my old P3 system even earlier than that) Ahem - after reading Alan's postings I am not so sure, ISTM that there is some more investigation require

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-20 Thread Mark Kirkwood
Alan Stange wrote: Another data point. We had some down time on our system today to complete some maintenance work. It took the opportunity to rebuild the 700GB file system using XFS instead of Reiser. One iostat output for 30 seconds is avg-cpu: %user %nice%sys %iowait %idle

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-20 Thread Mark Kirkwood
Mark Kirkwood wrote: The test is SELECT 1 FROM table That should read "The test is SELECT count(1) FROM table" ---(end of broadcast)--- TIP 6: explain analyze is your friend

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-21 Thread Mark Kirkwood
Luke Lonergan wrote: So that leaves the question - why not more than 64% of the I/O scan rate? And why is it a flat 64% as the I/O subsystem increases in speed from 333-400MB/s? It might be interesting to see what effect reducing the cpu consumption entailed by the count aggregation has - b

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-23 Thread Mark Kirkwood
Luke Lonergan wrote: Mark, This is an excellent idea – unfortunately I’m in Maui right now (Mahalo!) and I’m not getting to testing with this. My first try was with 8.0.3 and it’s an 8.1 function I presume. Not to be lazy – but any hint as to how to do the same thing for 8.0? Yeah, it's

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-24 Thread Mark Kirkwood
Luke Lonergan wrote: 12.9GB of DBT-3 data from the lineitem table llonergan=# select relpages from pg_class where relname='lineitem'; relpage

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-24 Thread Mark Kirkwood
Luke Lonergan wrote: Mark, It would be nice to put some tracers into the executor and see where the time is going. I'm also curious about the impact of the new 8.1 virtual tuples in reducing the executor overhead. In this case my bet's on the agg node itself, what do you think? Yeah - it's

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-24 Thread Mark Kirkwood
Luke Lonergan wrote: That says it's something else in the path. As you probably know there is a page lock taken, a copy of the tuple from the page, lock removed, count incremented for every iteration of the agg node on a count(*). Is the same true of a count(1)? Sorry Luke - message 3 - I s

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-24 Thread Mark Kirkwood
Luke Lonergan wrote: Mark, Time: 197870.105 ms So 198 seconds is the uncached read time with count (Just for clarity, did you clear the Pg and filesystem caches or unmount / remount the filesystem?) Nope - the longer time is due to the "second write" known issue with Postgres - it writes

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-24 Thread Mark Kirkwood
Tom Lane wrote: Greg Stark <[EMAIL PROTECTED]> writes: Last I heard the reason count(*) was so expensive was because its state variable was a bigint. That means it doesn't fit in a Datum and has to be alloced and stored as a pointer. And because of the Aggregate API that means it has to be allo

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-28 Thread Mark Kirkwood
Merlin Moncure wrote: It certainly makes quite a difference as I measure it: doing select(1) from a 181000 page table (completely uncached) on my PIII: 8.0 : 32 s 8.1 : 25 s Note that the 'fastcount()' function takes 21 s in both cases - so all the improvement seems to be from the count ove

Re: [PERFORM] SAN/NAS options

2005-12-13 Thread Mark Kirkwood
Charles Sprickman wrote: Hello all, It seems that I'm starting to outgrow our current Postgres setup. We've been running a handful of machines as standalone db servers. This is all in a colocation environment, so everything is stuffed into 1U Supermicro boxes. Our standard build looks like

Re: [PERFORM] Simple Join

2005-12-14 Thread Mark Kirkwood
Kevin Brown wrote: I'll just start by warning that I'm new-ish to postgresql. I'm running 8.1 installed from source on a Debian Sarge server. I have a simple query that I believe I've placed the indexes correctly for, and I still end up with a seq scan. It makes sense, kinda, but it should b

Re: [PERFORM] Simple Join

2005-12-14 Thread Mark Kirkwood
Kevin Brown wrote: I only had two explicit indexes. One was on to_ship.ordered_product_id and the other was on ordered_products.paid. ordered_products.id is a primary key. This is on your query with an index added on suspended_sub: dli=# explain analyze SELECT ordered_products.product_i

Re: [PERFORM] Simple Join

2005-12-15 Thread Mark Kirkwood
Kevin Brown wrote: On Wednesday 14 December 2005 18:36, you wrote: Well - that had no effect at all :-) You don't have and index on to_ship.ordered_product_id do you? - try adding one (ANALYZE again), and let use know what happens (you may want to play with SET enable_seqscan=off as well). I

Re: [PERFORM] Simple Join

2005-12-15 Thread Mark Kirkwood
Mitch Skinner wrote: I saw that; what I'm suggesting is that that you try creating a 3-column index on ordered_products using the paid, suspended_sub, and id columns. In that order, I think, although you could also try the reverse. It may or may not help, but it's worth a shot--the fact that all

Re: [PERFORM] Overriding the optimizer

2005-12-15 Thread Mark Kirkwood
Craig A. James wrote: I asked a while back if there were any plans to allow developers to override the optimizer's plan and force certain plans, and received a fairly resounding "No". The general feeling I get is that a lot of work has gone into the optimizer, and by God we're going to use it!

Re: [PERFORM] Overriding the optimizer

2005-12-15 Thread Mark Kirkwood
Craig A. James wrote: What would be cool would be some way the developer could alter the plan, but they way of doing so would strongly encourage the developer to send the information to this mailing list. Postgres would essentially say, "Ok, you can do that, but we want to know why!" Ye

Re: [PERFORM] Simple Join

2005-12-16 Thread Mark Kirkwood
David Lang wrote: On Fri, 16 Dec 2005, Mark Kirkwood wrote: Right on. Some of these "coerced" plans may perform much better. If so, we can look at tweaking your runtime config: e.g. effective_cache_size random_page_cost default_statistics_target to see if said plans can

Re: [PERFORM] SAN/NAS options

2005-12-16 Thread Mark Kirkwood
Jim C. Nasby wrote: On Wed, Dec 14, 2005 at 08:28:56PM +1300, Mark Kirkwood wrote: Another interesting thing to try is rebuilding the database ufs filesystem(s) with 32K blocks and 4K frags (as opposed to 8K/1K or 16K/2K - can't recall the default on 4.x). I found this to give a factor

Re: [PERFORM] Should Oracle outperform PostgreSQL on a complex

2005-12-16 Thread Mark Kirkwood
Bruce Momjian wrote: How are star joins different from what we do now? --- Recall that a "star" query with n tables means a query where there are (n - 1) supposedly small tables (dimensions) and 1 large table (fact) - w

Re: [PERFORM] Should Oracle outperform PostgreSQL on a complex

2005-12-17 Thread Mark Kirkwood
Tom Lane wrote: Simon Riggs <[EMAIL PROTECTED]> writes: On Fri, 2005-12-16 at 23:28 -0500, Bruce Momjian wrote: How are star joins different from what we do now? Methods: 1. join all N small tables together in a cartesian product, then join to main Large table once (rather than N times)

Re: [PERFORM] Should Oracle outperform PostgreSQL on a complex

2005-12-17 Thread Mark Kirkwood
Tom Lane wrote: 2. transform joins into subselects, then return subselect rows via an index bitmap. Joins are performed via a bitmap addition process. Looks like 8.1 pretty much does this right now: First the basic star: EXPLAIN ANALYZE SELECT d0.dmth, d1.dat, count(f

Re: [PERFORM] Should Oracle outperform PostgreSQL on a complex

2005-12-18 Thread Mark Kirkwood
Simon Riggs wrote: On Sun, 2005-12-18 at 17:07 +1300, Mark Kirkwood wrote: Tom Lane wrote: 2. transform joins into subselects, then return subselect rows via an index bitmap. Joins are performed via a bitmap addition process. Looks like 8.1 pretty much does this right now: Good

Re: [PERFORM] Should Oracle outperform PostgreSQL on a complex

2005-12-18 Thread Mark Kirkwood
Simon Riggs wrote: On Sun, 2005-12-18 at 15:02 +1300, Mark Kirkwood wrote: Yeah - the quoted method of "make a cartesian product of the dimensions and then join to the fact all at once" is not actually used (as written) in many implementations But it is used in some, whic

Re: [PERFORM] Should Oracle outperform PostgreSQL on a complex

2005-12-18 Thread Mark Kirkwood
Simon Riggs wrote: On Sat, 2005-12-17 at 13:13 -0500, Tom Lane wrote: Simon Riggs <[EMAIL PROTECTED]> writes: On Fri, 2005-12-16 at 23:28 -0500, Bruce Momjian wrote: How are star joins different from what we do now? Methods: 1. join all N small tables together in a cartesian product, the

Re: [PERFORM] Simple Join

2005-12-20 Thread Mark Kirkwood
Mark Kirkwood wrote: Kevin Brown wrote: I'll just start by warning that I'm new-ish to postgresql. I'm running 8.1 installed from source on a Debian Sarge server. I have a simple query that I believe I've placed the indexes correctly for, and I still end up with a seq s

Re: [PERFORM] What's the best hardver for PostgreSQL 8.1?

2005-12-23 Thread Mark Kirkwood
Vivek Khera wrote: and only the opteron boxes needed to come from sun. add a zero return policy and you wonder how they expect to keep in business sorry, i had to vent. Just out of interest - why did the opterons need to come from Sun? ---(end of broadcast

Re: RES: [PERFORM] Priority to a mission critical transaction

2006-11-28 Thread Mark Kirkwood
Josh Berkus wrote: All, The Bizgres project is working on resource management for PostgreSQL. So far, however, they have been able to come up with schemes that work for BI/DW at the expense of OLTP. Becuase of O^N lock checking issues, resource management for OLTP which doesn't greatly redu

Re: [PERFORM] RES: Priority to a mission critical transaction

2006-11-28 Thread Mark Kirkwood
Ron Mayer wrote: Short summary: * Papers studying priority inversion issues with databases including PosgreSQL and realistic workloads conclude setpriority() helps even in the presence of priority inversion issues for TCP-C and TCP-W like workloads. * Avoiding priority inversi

Re: [PERFORM] Bad iostat numbers

2006-11-30 Thread Mark Kirkwood
Carlos H. Reimer wrote: While collecting performance data I discovered very bad numbers in the I/O subsystem and I would like to know if I´m thinking correctly. Here is a typical iostat -x: avg-cpu: %user %nice %system %iowait %idle 50.400.000.501.10 48.00

Re: [PERFORM] Bad iostat numbers

2006-11-30 Thread Mark Kirkwood
David Boreham wrote: These number look a bit strange. I am wondering if there is a hardware problem on one of the drives or on the controller. Check in syslog for messages about disk timeouts etc. 100% util but 6 writes/s is just wrong (unless the drive is a 1980's vintage floppy). Agreed

Re: [PERFORM] SQL_CALC_FOUND_ROWS in POSTGRESQL / Some one can

2006-12-10 Thread Mark Kirkwood
Chris wrote: It's the same as doing a select count(*) type query using the same clauses, but all in one query instead of two. It doesn't return any extra rows on top of the limit query so it's better than using pg_numrows which runs the whole query and returns it to php (in this example).

Re: [PERFORM] New to PostgreSQL, performance considerations

2006-12-11 Thread Mark Kirkwood
Daniel van Ham Colchete wrote: On 12/11/06, Steinar H. Gunderson <[EMAIL PROTECTED]> wrote: On Mon, Dec 11, 2006 at 11:31:48AM -0200, Daniel van Ham Colchete wrote: > What PostgreSQL benchmark software should I use??? Look up the list archives; search for "TPC". > I'll test PostgreSQL 8.1 on a

Re: [PERFORM] EXPLAIN ANALYZE on 8.2

2006-12-14 Thread Mark Kirkwood
Kelly Burkhart wrote: I hope this isn't a "crummy mainboard" but I can't seem to affect things by changing clock source (kernel 2.6.16 SLES10). I tried kernel command option clock=XXX where XXX in (cyclone,hpet,pmtmr,tsc,pit), no option was significantly better than the default. Anyone know ho

Re: [PERFORM] slow result

2007-01-23 Thread Mark Kirkwood
Laurent Manchon wrote: Hi, I have a slow response of my PostgreSQL database 7.4 using this query below on a table with 80 rows: select count(*)from tbl; PostgreSQL return result in 28 sec every time. Can you post the results of: analyze verbose tbl; explain analyze select count(*) from

Re: [PERFORM] extract(field from timestamp) vs date dimension

2007-01-23 Thread Mark Kirkwood
Tobias Brox wrote: I suppose the strongest argument for introducing date dimensions already now is that I probably will benefit from having conform and well-designed dimensions when I will be introducing more data marts. As for now I have only one fact table and some few dimensions in the syst

Re: [PERFORM] How long should it take to insert 200,000 records?

2007-02-06 Thread Mark Kirkwood
Karen Hill wrote: The postgres version is 8.2.1 on Windows. The pl/pgsql function is inserting to an updatable view (basically two tables). CREATE TABLE foo1 ( ) ; CREATE TABLE foo2 ( ); CREATE VIEW viewfoo AS ( ); CREATE RULE ruleFoo AS ON INSERT TO viewfoo DO INSTEAD ( ); CREATE OR

Re: [PERFORM] quad or dual core Intel CPUs

2007-02-13 Thread Mark Kirkwood
Kenji Morishige wrote: Please comment on any issues you may see with this box and my assumptions. Also any FreeBSD kernel issues or tweaks you could recommend. I would recommend posting to freebsd-hardware or freebsd-stable and asking if there are any gotchas with the X7DBE+ and 6.2 (for ins

Re: [PERFORM] Postgres performance Linux vs FreeBSD

2007-02-21 Thread Mark Kirkwood
Jacek Zarêba wrote: Hello, I've set up 2 identical machines, hp server 1ghz p3, 768mb ram, 18gb scsi3 drive. On the first one I've installed Debian/GNU 4.0 Linux, on the second FreeBSD 6.2. On both machines I've installed Postgresql 8.2.3 from sources. Now the point :)) According to my tests post

Re: [PERFORM] increasing database connections

2007-02-28 Thread Mark Kirkwood
Joshua D. Drake wrote: Jonah H. Harris wrote: On 3/1/07, Shiva Sarna <[EMAIL PROTECTED]> wrote: I am sorry if it is a repeat question but I want to know if database performance will decrease if I increase the max-connections to 2000. At present it is 100. Most certainly. Adding connections ov

Re: [PERFORM] SCSI vs SATA

2007-04-06 Thread Mark Kirkwood
Ron wrote: I read them as soon as they were available. Then I shrugged and noted YMMV to myself. 1= Those studies are valid for =those= users under =those= users' circumstances in =those= users' environments. How well do those circumstances and environments mimic anyone else's? Exactly,

Re: [PERFORM] index structure for 114-dimension vector

2007-04-20 Thread Mark Kirkwood
Jeff Davis wrote: On Fri, 2007-04-20 at 12:07 -0700, Andrew Lazarus wrote: I have a table with 2.5 million real[] arrays. (They are points in a time series.) Given a new array X, I'd like to find, say, the 25 closest to X in some sense--for simplification, let's just say in the usual vector norm

Re: [PERFORM] index structure for 114-dimension vector

2007-04-20 Thread Mark Kirkwood
Andrew Lazarus wrote: Because I know the 25 closest are going to be fairly close in each coordinate, I did try a multicolumn index on the last 6 columns and used a +/- 0.1 or 0.2 tolerance on each. (The 25 best are very probably inside that hypercube on the distribution of data in question.) Thi

Re: [PERFORM] postgres: 100% CPU utilization

2007-04-23 Thread Mark Kirkwood
Scott Marlowe wrote: (snippage) that's the kinda hardware I was running 7.0.2 on, so you might as well get retro in your hardware department while you're at it. Notice he's running FreeBSD 4.4(!), so it could well be a very old machine... Cheers Mark ---(end of b

Re: [PERFORM] postgres: 100% CPU utilization

2007-04-26 Thread Mark Kirkwood
Sergey Tsukinovsky wrote: Just for the record - the hardware that was used for the test has the following parameters: AMD Opteron 2GHZ 2GB RAM LSI Logic SCSI And you ran FreeBSD 4.4 on it right? This may be a source of high cpu utilization in itself if the box is SMP or dual core, as multi-c

Re: [PERFORM] Feature Request --- was: PostgreSQL Performance Tuning

2007-05-05 Thread Mark Kirkwood
Josh Berkus wrote: Sebastian, Before inventing a hyper tool, we might consider to provide 3-5 example szenarios for common hardware configurations. This consumes less time and be discussed and defined in a couple of days. This is of course not the correct option for a brandnew 20 spindle Sata 1

Re: [PERFORM] Disk Fills Up and fsck "Compresses" it

2007-05-16 Thread Mark Kirkwood
Jim C. Nasby wrote: No, it's part of FreeBSD's UFS. google FreeBSD softupdates and you should get plenty of info. As I said, it's probably not worth worrying about. On Wed, May 16, 2007 at 08:21:23AM -0700, Y Sidhu wrote: What do you mean by "softupdates?" Is that a parameter in what I am gues

Re: [PERFORM] Weird 8.2.4 performance

2007-06-06 Thread Mark Kirkwood
Kurt Overberg wrote: Explain Outputs: -- 8.2 -> Bitmap Heap Scan on taskinstance (cost=20.71..2143.26 rows=556 width=8) (actual time=421.423..5655.745 rows=98 loops=9) Recheck Cond: (taskinstance.taskid = task.id) -> Bitmap Index Scan on taskid_taskinstance_key (

Re: [PERFORM] Weird 8.2.4 performance

2007-06-07 Thread Mark Kirkwood
Steinar H. Gunderson wrote: On Thu, Jun 07, 2007 at 11:35:27AM +0200, Steinar H. Gunderson wrote: If that table doesn't fit in the cache on your Mac, you pretty much lose. From the EXPLAIN output, it looks like it fits very nicely in cache on your server. Thus, I don't think the difference is be

Re: [PERFORM] Weird 8.2.4 performance

2007-06-07 Thread Mark Kirkwood
Kurt Overberg wrote: work_mem = 100MB# when I ran the original query, this was set to 1MB, increased on Mark Kirkwood's advice, seemed to help a bit but not really For future reference, be careful with this parameter, as *every* connection will use this much memory f

Re: [PERFORM] disk filling up

2007-07-26 Thread Mark Kirkwood
Brandon Shalton wrote: Hello all, My hard disk is filling up in the /base directory to where it has consumed all 200gig of that drive. All the posts that i see keep saying move to a bigger drive, but at some point a bigger drive would just get consumed. How can i keep the disk from filli

Re: [PERFORM] When/if to Reindex

2007-08-23 Thread Mark Kirkwood
Tom Lane wrote: The fly in the ointment is that after collecting the pg_index definition of the index, plancat.c also wants to know how big it is --- it calls RelationGetNumberOfBlocks. And that absolutely does look at the physical storage, which means it absolutely is unsafe to do in parallel

[PERFORM] File system choice for Red Hat systems

2010-06-01 Thread Mark Kirkwood
I'm helping set up a Red Hat 5.5 system for Postgres. I was going to recommend xfs for the filesystem - however it seems that xfs is supported as a technology preview "layered product" for 5.5. This apparently means that the xfs tools are only available via special channels. What are Red Hat

Re: [PERFORM] File system choice for Red Hat systems

2010-06-01 Thread Mark Kirkwood
On 02/06/10 15:26, Tom Lane wrote: What's your time horizon? RHEL6 will have full support for xfs. On RHEL5 I really wouldn't consider anything except ext3. Yeah, RHEL6 seems like the version we would prefer - unfortunately time frame is the next few days. Awesome - thanks for the quick r

Re: [PERFORM] File system choice for Red Hat systems

2010-06-01 Thread Mark Kirkwood
On 02/06/10 17:17, Devrim GÜNDÜZ wrote: For xfs, you may want to read this: http://blog.2ndquadrant.com/en/2010/04/the-return-of-xfs-on-linux.html Thanks - yes RHEL6 is the version we would have liked to use I suspect! Regards Mark -- Sent via pgsql-performance mailing list (pgsql-p

Re: [PERFORM] File system choice for Red Hat systems

2010-06-02 Thread Mark Kirkwood
On 03/06/10 02:53, Alan Hodgson wrote: On Tuesday 01 June 2010, Mark Kirkwood wrote: I'm helping set up a Red Hat 5.5 system for Postgres. I was going to recommend xfs for the filesystem - however it seems that xfs is supported as a technology preview "layered product"

Re: [PERFORM] Weird XFS WAL problem

2010-06-02 Thread Mark Kirkwood
On 03/06/10 11:30, Craig James wrote: I'm testing/tuning a new midsize server and ran into an inexplicable problem. With an RAID10 drive, when I move the WAL to a separate RAID1 drive, TPS drops from over 1200 to less than 90! I've checked everything and can't find a reason. Are the 2 n

Re: [PERFORM] PostgreSQL as a local in-memory cache

2010-06-16 Thread Mark Kirkwood
On 16/06/10 18:30, jgard...@jonathangardner.net wrote: On Jun 15, 4:18 pm, j...@agliodbs.com (Josh Berkus) wrote: On 6/15/10 10:37 AM, Chris Browne wrote: I'd like to see some figures about WAL on RAMfs vs. simply turning off fsync and full_page_writes. Per Gavin's tests, PostgreSQL is alr

[PERFORM] wal_synch_method = open_sync safe on RHEL 5.5?

2010-06-17 Thread Mark Kirkwood
Some more on the RHEL 5.5 system I'm helping to setup. Some benchmarking using pgbench appeared to suggest that wal_sync_method=open_sync was a little faster than fdatasync [1]. Now I recall some discussion about this enabling direct io and the general flakiness of this on Linux, so is the opti

Re: [PERFORM] wal_synch_method = open_sync safe on RHEL 5.5?

2010-06-17 Thread Mark Kirkwood
On 18/06/10 15:29, Greg Smith wrote: P.S. Be wary of expecting pgbench to give you useful numbers on a single run. For the default write-heavy test, I recommend three runs of 10 minutes each (-T 600 on recent PostgreSQL versions) before I trust any results it gives. You can get useful data

Re: [PERFORM] [Slony1-general] WAL partition overloaded--by autovacuum?

2010-07-07 Thread Mark Kirkwood
On 07/07/10 13:10, Richard Yen wrote: This leads me to believe that there was a sudden flurry of write activity that occurred, and the process that would flush WAL files to /db/data/ couldn't keep up, thereby filling up the disk. I'm wondering if anyone else out there might be able to give m

Re: [PERFORM] Need help in performance tuning.

2010-07-09 Thread Mark Kirkwood
On 10/07/10 00:56, Brad Nicholson wrote: On Fri, 2010-07-09 at 00:42 -0400, Tom Lane wrote: Perhaps not, but there's no obvious benefit either. Since there's More Than One Way To Do It, it seems more practical to keep that as a separate problem that can be solved by a choice of add-on pack

[PERFORM] Linux Filesystems again - Ubuntu this time

2010-07-26 Thread Mark Kirkwood
I'm involved with the setup of replacement hardware for one of our systems. It is going to be using Ubuntu Lucid Server (kernel 2.6.32 I believe). The issue of filesystems has raised its head again. I note that ext4 is now the default for Lucid, what do folks think about using it: stable enoug

Re: [PERFORM] Advice configuring ServeRAID 8k for performance

2010-08-05 Thread Mark Kirkwood
On 06/08/10 06:28, Kenneth Cox wrote: I am using PostgreSQL 8.3.7 on a dedicated IBM 3660 with 24GB RAM running CentOS 5.4 x86_64. I have a ServeRAID 8k controller with 6 SATA 7500RPM disks in RAID 6, and for the OLAP workload it feels* slow. I have 6 more disks to add, and the RAID has to be

Re: [PERFORM] Advice configuring ServeRAID 8k for performance

2010-08-05 Thread Mark Kirkwood
On 06/08/10 11:58, Alan Hodgson wrote: On Thursday, August 05, 2010, Mark Kirkwood wrote: Normally I'd agree with the others and recommend RAID10 - but you say you have an OLAP workload - if it is *heavily* read biased you may get better performance with RAID5 (more effective disks to

Re: [PERFORM] Advice configuring ServeRAID 8k for performance

2010-08-05 Thread Mark Kirkwood
On 06/08/10 12:31, Mark Kirkwood wrote: On 06/08/10 11:58, Alan Hodgson wrote: On Thursday, August 05, 2010, Mark Kirkwood wrote: Normally I'd agree with the others and recommend RAID10 - but you say you have an OLAP workload - if it is *heavily* read biased you may get better perfor

Re: [PERFORM] Very poor performance

2010-08-17 Thread Mark Kirkwood
On 18/08/10 06:19, Kevin Grittner wrote: Since you haven't set effective_cache_size, you're discouraging some types of plans which might be worth considering. This should normally be set to the sum of your shared_buffers setting and whatever is cached by the OS; try setting effective_cache_siz

Re: [PERFORM] Memory usage - indexes

2010-09-23 Thread Mark Kirkwood
On 24/09/10 09:50, Tobias Brox wrote: We've come to a tipping point with one of our database servers, it's generally quite loaded but up until recently it was handling the load well - but now we're seeing that it struggles to process all the selects fast enough. Sometimes we're observing some we

Re: [PERFORM] Memory usage - indexes

2010-09-29 Thread Mark Kirkwood
On 29/09/10 19:41, Tobias Brox wrote: I just got this crazy, stupid or maybe genius idea :-) Now, my idea is to drop that fat index and replace it with conditional indexes for a dozen of heavy users - like those: acc_trans(trans_type, created) where customer_id=224885; acc_trans(trans_ty

Re: [PERFORM] Memory usage - indexes

2010-09-30 Thread Mark Kirkwood
On 30/09/10 01:09, Tobias Brox wrote: With the most popular trans type it chose another plan and it took more than 3s (totally unacceptable): Try tweeking effective_cache_size up a bit and see what happens - I've found these bitmap plans to be sensitive to it sometimes. regards Mark

Re: [PERFORM] Slow count(*) again...

2010-10-13 Thread Mark Kirkwood
On 13/10/10 19:47, Neil Whelchel wrote: Nope... So, possible conclusions are: 1. Even with VACUUM database table speed degrades as tables are updated. 2. Time testing on a freshly INSERTed table gives results that are not real- world. 3. Filesystem defragmentation helps (some). 4. Cache only mak

  1   2   3   4   5   >