Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread James Mansion
Right --- the point is not the interface, but whether the drive is built for reliability or to hit a low price point. Personally I take the marketing mublings about the enterprise drives with a pinch of salt. The low-price drives HAVE TO be reliable too, because a non-negligible failure rate

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Tom Lane
James Mansion [EMAIL PROTECTED] writes: Right --- the point is not the interface, but whether the drive is built for reliability or to hit a low price point. Personally I take the marketing mublings about the enterprise drives with a pinch of salt. The low-price drives HAVE TO be reliable

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Arjen van der Meijden
If the 3U case has a SAS-expander in its backplane (which it probably has?) you should be able to connect all drives to the Adaptec controller, depending on the casing's exact architecture etc. That's another two advantages of SAS, you don't need a controller port for each harddisk (we have a

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Heikki Linnakangas
[EMAIL PROTECTED] wrote: In a perhaps fitting compromise, I have decide to go with a hybrid solution: 8*73GB 15k SAS drives hooked up to Adaptec 4800SAS PLUS 6*150GB SATA II drives hooked up to mobo (for now) All wrapped in a 16bay 3U server. My reasoning is that the extra SATA drives are

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Ron
BE VERY WARY OF USING AN ADAPTEC RAID CONTROLLER! IME, they are usually the worst of the commodity RAID controllers available. I've often seen SW RAID outperform them. If you are going to use this config, Tyan's n3600M (AKA S2932) MB has a variant that comes with 8 SAS + 6 SATA II connectors.

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Scott Marlowe
On Wed, 2007-04-04 at 09:12, [EMAIL PROTECTED] wrote: On Apr 3, 2007, at 6:54 PM, Geoff Tolley wrote: But what's likely to make the largest difference in the OP's case (many inserts) is write caching, and a battery-backed cache would be needed for this. This will help mask write

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Scott Marlowe
On Thu, 2007-04-05 at 00:32, Tom Lane wrote: James Mansion [EMAIL PROTECTED] writes: Right --- the point is not the interface, but whether the drive is built for reliability or to hit a low price point. Personally I take the marketing mublings about the enterprise drives with a pinch of

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Jeff Frost
On Thu, 5 Apr 2007, Scott Marlowe wrote: I've read some recent contrary advice. Specifically advising the sharing of all files (pg_xlogs, indices, etc..) on a huge raid array and letting the drives load balance by brute force. The other, at first almost counter-intuitive result was that

Re: [PERFORM] postgres 7.4 vs 8.x redux: query plans

2007-04-05 Thread Alex Deucher
Ok, well, I dropped the DB and reloaded it and now all seems to be fine and performing well. I'm not sure what was going on before. Thanks for everyone's help! Alex On 4/3/07, Alex Deucher [EMAIL PROTECTED] wrote: On 4/3/07, Tom Lane [EMAIL PROTECTED] wrote: Alex Deucher [EMAIL PROTECTED]

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread [EMAIL PROTECTED]
On Apr 5, 2007, at 4:09 AM, Ron wrote: BE VERY WARY OF USING AN ADAPTEC RAID CONTROLLER! Thanks - I received similar private emails with the same advice. I will change the controller to a LSI MegaRAID SAS 8408E -- any feedback on this one? IME, they are usually the worst of the

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Joshua D. Drake
[EMAIL PROTECTED] wrote: On Apr 5, 2007, at 4:09 AM, Ron wrote: BE VERY WARY OF USING AN ADAPTEC RAID CONTROLLER! Thanks - I received similar private emails with the same advice. I will change the controller to a LSI MegaRAID SAS 8408E -- any feedback on this one? LSI makes a good

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Alex Deucher
On 4/5/07, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: On Apr 5, 2007, at 4:09 AM, Ron wrote: BE VERY WARY OF USING AN ADAPTEC RAID CONTROLLER! Thanks - I received similar private emails with the same advice. I will change the controller to a LSI MegaRAID SAS 8408E -- any feedback on this

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Ron
At 11:19 AM 4/5/2007, Scott Marlowe wrote: On Thu, 2007-04-05 at 00:32, Tom Lane wrote: James Mansion [EMAIL PROTECTED] writes: Right --- the point is not the interface, but whether the drive is built for reliability or to hit a low price point. Personally I take the marketing mublings

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Arjen van der Meijden
On 5-4-2007 17:58 [EMAIL PROTECTED] wrote: On Apr 5, 2007, at 4:09 AM, Ron wrote: BE VERY WARY OF USING AN ADAPTEC RAID CONTROLLER! Thanks - I received similar private emails with the same advice. I will change the controller to a LSI MegaRAID SAS 8408E -- any feedback on this one? We

[PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Xiaoning Ding
Hi, A page may be double buffered in PG's buffer pool and in OS's buffer cache. Other DBMS like DB2 and Oracle has provided Direct I/O option to eliminate double buffering. I noticed there were discusses on the list. But I can not find similar option in PG. Does PG support direct I/O now? The

[PERFORM] What do the adminpack functions do? (8.2.3)

2007-04-05 Thread Michael Dengler
Hi, Ive looked through the README in the contrib/adminpack dir but it doesn't really say what these functions do and how to use them Any help? Thanks miguel

Re: [PERFORM] Equivalents in PostgreSQL of MySQL's ENGINE=MEMORY MAX_ROWS=1000

2007-04-05 Thread Josh Berkus
Dimitri, Probably another helpful solution may be to implement: ALTER TABLE LOGGING OFF/ON; just to disable/enable WAL? Actually, a patch similar to this is currently in the queue for 8.3. -- Josh Berkus PostgreSQL @ Sun San Francisco ---(end of

Re: [PERFORM] What do the adminpack functions do? (8.2.3)

2007-04-05 Thread Joshua D. Drake
Michael Dengler wrote: Hi, Ive looked through the README in the contrib/adminpack dir but it doesn't really say what these functions do and how to use them It is for use with Pgadmin. J Any help? Thanks miguel -- === The PostgreSQL Company: Command Prompt, Inc. ===

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Erik Jones
On Apr 5, 2007, at 12:09 PM, Xiaoning Ding wrote: Hi, A page may be double buffered in PG's buffer pool and in OS's buffer cache. Other DBMS like DB2 and Oracle has provided Direct I/O option to eliminate double buffering. I noticed there were discusses on the list. But I can not find

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Xiaoning Ding
Erik Jones wrote: On Apr 5, 2007, at 12:09 PM, Xiaoning Ding wrote: Hi, A page may be double buffered in PG's buffer pool and in OS's buffer cache. Other DBMS like DB2 and Oracle has provided Direct I/O option to eliminate double buffering. I noticed there were discusses on the list. But I

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Mark Lewis
Not to hijack this thread, but has anybody here tested the behavior of PG on a file system with OS-level caching disabled via forcedirectio or by using an inherently non-caching file system such as ocfs2? I've been thinking about trying this setup to avoid double-caching now that the 8.x series

[PERFORM] Premature view materialization in 8.2?

2007-04-05 Thread Jonathan Ellis
I have a query hitting a view that takes 0.15s on 8.1 but 6s on 8.2.3: select party_id from clan_members_v cm, clans c where cm.clan_id = c.id and c.type = 'standard' The problem seems to be that clan_members_v contains a call to an expensive function: create or replace view

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Erik Jones
On Apr 5, 2007, at 1:22 PM, Xiaoning Ding wrote: Erik Jones wrote: On Apr 5, 2007, at 12:09 PM, Xiaoning Ding wrote: Hi, A page may be double buffered in PG's buffer pool and in OS's buffer cache. Other DBMS like DB2 and Oracle has provided Direct I/O option to eliminate double

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Alex Deucher
On 4/5/07, Erik Jones [EMAIL PROTECTED] wrote: On Apr 5, 2007, at 1:22 PM, Xiaoning Ding wrote: Erik Jones wrote: On Apr 5, 2007, at 12:09 PM, Xiaoning Ding wrote: Hi, A page may be double buffered in PG's buffer pool and in OS's buffer cache. Other DBMS like DB2 and Oracle has provided

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Erik Jones
On Apr 5, 2007, at 1:27 PM, Mark Lewis wrote: On Thu, 2007-04-05 at 13:09 -0500, Erik Jones wrote: On Apr 5, 2007, at 12:09 PM, Xiaoning Ding wrote: Hi, A page may be double buffered in PG's buffer pool and in OS's buffer cache. Other DBMS like DB2 and Oracle has provided Direct I/O option

Re: [PERFORM] Equivalents in PostgreSQL of MySQL's ENGINE=MEMORY MAX_ROWS=1000

2007-04-05 Thread Dimitri
Wow, it's excellent! :)) probably the next step is: ALTER TABLE CACHE ON/OFF; just to force keeping any table in the cache. What do you think?... Rgds, -Dimitri On 4/5/07, Josh Berkus josh@agliodbs.com wrote: Dimitri, Probably another helpful solution may be to implement: ALTER

[PERFORM] High Load on Postgres 7.4.16 Server

2007-04-05 Thread John Allgood
Hello All I sent this message to the admin list and it never got through so I am trying the performance list. We moved our application to a new machine last night. It is a Dell PowerEdge 6950 2X Dual Core. AMD Opteron 8214 2.2Ghz. 8GB Memory. The machine is running

Re: [PERFORM] High Load on Postgres 7.4.16 Server

2007-04-05 Thread Jeff Frost
On Thu, 5 Apr 2007, John Allgood wrote: Hello All I sent this message to the admin list and it never got through so I am trying the performance list. We moved our application to a new machine last night. It is a Dell PowerEdge 6950 2X Dual Core. AMD Opteron 8214 2.2Ghz. 8GB

Re: [PERFORM] High Load on Postgres 7.4.16 Server

2007-04-05 Thread C. Bergström
John Allgood wrote: Hello All I sent this message to the admin list and it never got through so I am trying the performance list. We moved our application to a new machine last night. It is a Dell PowerEdge 6950 2X Dual Core. AMD Opteron 8214 2.2Ghz. 8GB Memory.

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread James Mansion
Server drives are generally more tolerant of higher temperatures. I.e. the failure rate for consumer and server class HDs may be about the same at 40 degrees C, but by the time the internal case temps get up to 60-70 degrees C, the consumer grade drives will likely be failing at a much higher

Re: [PERFORM] High Load on Postgres 7.4.16 Server

2007-04-05 Thread John Allgood
The hard thing about running multiple postmasters is that you have to tune each one separate. Most of the databases I have limited the max-connections to 30-50 depending on the database. What would reasonable values for effective_cache_size and random_page_cost. I think I have these default. Also

Re: [PERFORM] High Load on Postgres 7.4.16 Server

2007-04-05 Thread Jeff Frost
On Thu, 5 Apr 2007, John Allgood wrote: The hard thing about running multiple postmasters is that you have to tune each one separate. Most of the databases I have limited the max-connections to 30-50 depending on the database. What would reasonable values for effective_cache_size and

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Mark Lewis
... [snipped for brevity] ... Not to hijack this thread, but has anybody here tested the behavior of PG on a file system with OS-level caching disabled via forcedirectio or by using an inherently non-caching file system such as ocfs2? I've been thinking about trying this setup

Re: [PERFORM] High Load on Postgres 7.4.16 Server

2007-04-05 Thread Dave Cramer
On 5-Apr-07, at 3:33 PM, John Allgood wrote: The hard thing about running multiple postmasters is that you have to tune each one separate. Most of the databases I have limited the max- connections to 30-50 depending on the database. What would reasonable values for effective_cache_size and

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread david
On Thu, 5 Apr 2007, [EMAIL PROTECTED] wrote: I'm curious to know why you're on xfs (i've been too chicken to stray from ext3). better support for large files (although postgres does tend to try and keep the file size down by going with multiple files) and also for more files the

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread david
On Thu, 5 Apr 2007, Ron wrote: At 11:19 AM 4/5/2007, Scott Marlowe wrote: On Thu, 2007-04-05 at 00:32, Tom Lane wrote: James Mansion [EMAIL PROTECTED] writes: Right --- the point is not the interface, but whether the drive is built for reliability or to hit a low price point.

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Erik Jones
On Apr 5, 2007, at 2:56 PM, Mark Lewis wrote: ... [snipped for brevity] ... Not to hijack this thread, but has anybody here tested the behavior of PG on a file system with OS-level caching disabled via forcedirectio or by using an inherently non-caching file system such as ocfs2? I've been

Re: [PERFORM] High Load on Postgres 7.4.16 Server

2007-04-05 Thread John Allgood
We run multiple postmasters because we can shutdown one postmaster/database without affecting the other postmasters/databases. Each database is a division in our company. If we had everything under one postmaster if something happened to the one the whole company would be down. -Original

Re: [PERFORM] High Load on Postgres 7.4.16 Server

2007-04-05 Thread Dave Cramer
The problem with this is that it doesn't leverage shared buffers and kernel buffers well. Anyways, my bet is that your SAN isn't performing as you expect on the new hardware. Dave On 5-Apr-07, at 4:13 PM, John Allgood wrote: We run multiple postmasters because we can shutdown one

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread david
On Thu, 5 Apr 2007, Xiaoning Ding wrote: To the best of my knowledge, Postgres itself does not have a direct IO option (although it would be a good addition). So, in order to use direct IO with postgres you'll need to consult your filesystem docs for how to set the forcedirectio mount

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Xiaoning Ding
Alex Deucher wrote: On 4/5/07, Erik Jones [EMAIL PROTECTED] wrote: On Apr 5, 2007, at 1:22 PM, Xiaoning Ding wrote: Erik Jones wrote: On Apr 5, 2007, at 12:09 PM, Xiaoning Ding wrote: Hi, A page may be double buffered in PG's buffer pool and in OS's buffer cache. Other DBMS like DB2 and

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Alex Deucher
On 4/5/07, Xiaoning Ding [EMAIL PROTECTED] wrote: Alex Deucher wrote: On 4/5/07, Erik Jones [EMAIL PROTECTED] wrote: On Apr 5, 2007, at 1:22 PM, Xiaoning Ding wrote: Erik Jones wrote: On Apr 5, 2007, at 12:09 PM, Xiaoning Ding wrote: Hi, A page may be double buffered in PG's buffer pool

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Scott Marlowe
On Thu, 2007-04-05 at 14:30, James Mansion wrote: Server drives are generally more tolerant of higher temperatures. I.e. the failure rate for consumer and server class HDs may be about the same at 40 degrees C, but by the time the internal case temps get up to 60-70 degrees C, the consumer

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread Erik Jones
On Apr 5, 2007, at 3:33 PM, [EMAIL PROTECTED] wrote: On Thu, 5 Apr 2007, Xiaoning Ding wrote: To the best of my knowledge, Postgres itself does not have a direct IO option (although it would be a good addition). So, in order to use direct IO with postgres you'll need to consult your

Re: [PERFORM] Weird performance drop

2007-04-05 Thread Vincenzo Romano
On Friday 30 March 2007 16:34 Dave Dutcher wrote: From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Vincenzo Romano Is there any workaround? In my opinion the later the query planner decisions are taken the more effective they can be. It could be an option for the

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread david
On Thu, 5 Apr 2007, Xiaoning Ding wrote: Xiaoning Looks like it. I just did a cursory search of the archives and it seems that others have looked at this before so you'll probably want to start there if your up to it. Linux used to have (still does?) a RAW interface which might also

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread david
On Thu, 5 Apr 2007, Scott Marlowe wrote: On Thu, 2007-04-05 at 14:30, James Mansion wrote: Server drives are generally more tolerant of higher temperatures. I.e. the failure rate for consumer and server class HDs may be about the same at 40 degrees C, but by the time the internal case temps

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Ron
At 10:07 PM 4/5/2007, [EMAIL PROTECTED] wrote: On Thu, 5 Apr 2007, Scott Marlowe wrote: Server class drives are designed with a longer lifespan in mind. Server class hard drives are rated at higher temperatures than desktop drives. these two I question. David Lang Both statements are the

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread david
On Thu, 5 Apr 2007, Ron wrote: At 10:07 PM 4/5/2007, [EMAIL PROTECTED] wrote: On Thu, 5 Apr 2007, Scott Marlowe wrote: Server class drives are designed with a longer lifespan in mind. Server class hard drives are rated at higher temperatures than desktop drives. these two I question.

Re: [PERFORM] a question about Direct I/O and double buffering

2007-04-05 Thread david
On Thu, 5 Apr 2007, Erik Jones wrote: On Apr 5, 2007, at 3:33 PM, [EMAIL PROTECTED] wrote: On Thu, 5 Apr 2007, Xiaoning Ding wrote: To the best of my knowledge, Postgres itself does not have a direct IO option (although it would be a good addition). So, in order to use direct

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Richard Troy
On Thu, 5 Apr 2007 [EMAIL PROTECTED] wrote: On Thu, 5 Apr 2007, Ron wrote: At 10:07 PM 4/5/2007, [EMAIL PROTECTED] wrote: On Thu, 5 Apr 2007, Scott Marlowe wrote: Server class drives are designed with a longer lifespan in mind. Server class hard drives are rated at higher

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Greg Smith
On Thu, 5 Apr 2007, Scott Marlowe wrote: On Thu, 2007-04-05 at 14:30, James Mansion wrote: Can you cite any statistical evidence for this? Logic? OK, everyone who hasn't already needs to read the Google and CMU papers. I'll even provide links for you:

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread Ron
At 11:40 PM 4/5/2007, [EMAIL PROTECTED] wrote: On Thu, 5 Apr 2007, Ron wrote: At 10:07 PM 4/5/2007, [EMAIL PROTECTED] wrote: On Thu, 5 Apr 2007, Scott Marlowe wrote: Server class drives are designed with a longer lifespan in mind. Server class hard drives are rated at higher temperatures

Re: [PERFORM] Premature view materialization in 8.2?

2007-04-05 Thread Tom Lane
Jonathan Ellis [EMAIL PROTECTED] writes: I have a query hitting a view that takes 0.15s on 8.1 but 6s on 8.2.3: ... Is this a regression, or a feature of 8.2? Hard to say without EXPLAIN ANALYZE output to compare. regards, tom lane ---(end of

Re: [PERFORM] SCSI vs SATA

2007-04-05 Thread david
On Fri, 6 Apr 2007, Ron wrote: Bear in mind that Google was and is notorious for pushing their environmental factors to the limit while using the cheapest PoS HW they can get their hands on. Let's just say I'm fairly sure every piece of HW they were using for those studies was operating

Re: [PERFORM] High Load on Postgres 7.4.16 Server

2007-04-05 Thread Tom Lane
John Allgood [EMAIL PROTECTED] writes: ... The other configuration was RHEL3 and Postgres 7.4.13 and Redhat Cluster Suite. The application seemed to run much faster on the older equipment. While I agree with the other comments that you should think about moving to something newer than 7.4.x,

Re: [PERFORM] Premature view materialization in 8.2?

2007-04-05 Thread Jonathan Ellis
On 4/5/07, Tom Lane [EMAIL PROTECTED] wrote: Jonathan Ellis [EMAIL PROTECTED] writes: I have a query hitting a view that takes 0.15s on 8.1 but 6s on 8.2.3: ... Is this a regression, or a feature of 8.2? Hard to say without EXPLAIN ANALYZE output to compare. To my eye they are identical