Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-14 Thread Scott Marlowe
On Tue, Dec 11, 2012 at 6:03 PM, Craig Ringer wrote: > On 12/12/2012 06:44 AM, Evgeny Shishkin wrote: > > > On Dec 12, 2012, at 2:41 AM, Niels Kristian Schjødt > wrote: > > Are you using a hardware based raid controller with them? > > Yes, of course. Hardware raid with cache and bbu is a must. Yo

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-13 Thread Jeff Janes
On Wed, Dec 12, 2012 at 8:46 AM, Niels Kristian Schjødt wrote: > > Den 11/12/2012 kl. 18.25 skrev Jeff Janes : > >> On Tue, Dec 11, 2012 at 2:04 AM, Niels Kristian Schjødt >> wrote: >> >>> Maybe I should mention, that I never see more than max 5Gb out of my total >>> 32Gb being in use on the ser

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-12 Thread Craig Ringer
On 13/12/2012 12:22 AM, Niels Kristian Schjødt wrote: > Well, In fact I do (as you can see from my configuration). I have a > similar server running with hot standby replication - and it runs two > 3T HDD in a RAID1 array. > > So - is it still very bad if I choose to put four intel 520 disks in a >

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-12 Thread Niels Kristian Schjødt
Den 11/12/2012 kl. 18.25 skrev Jeff Janes : > On Tue, Dec 11, 2012 at 2:04 AM, Niels Kristian Schjødt > wrote: >> Den 11/12/2012 kl. 00.58 skrev Jeff Janes : >> >>> >>> The fact that there is much more writing than reading tells me that >>> most of your indexes are in RAM. The amount of index

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-12 Thread Niels Kristian Schjødt
Well, In fact I do (as you can see from my configuration). I have a similar server running with hot standby replication - and it runs two 3T HDD in a RAID1 array. So - is it still very bad if I choose to put four intel 520 disks in a RAID10 array on the other production server? Den 12/12/2012

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread David Boreham
On 12/11/2012 8:11 PM, Evgeny Shishkin wrote: Quoting http://www.storagereview.com/intel_ssd_dc_s3700_series_enterprise_ssd_review Heh. A fine example of the kind of hand-waving of which I spoke ;) Higher performance is certainly a benefit, although at present we can't saturate even a single

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Evgeny Shishkin
On Dec 12, 2012, at 7:05 AM, David Boreham wrote: > On 12/11/2012 7:49 PM, Evgeny Shishkin wrote: >> Yeah, s3700 looks promising, but sata interface is limiting factor for this >> drive. >> I'm looking towards SMART ssd >> http://www.storagereview.com/smart_storage_systems_optimus_sas_enterpri

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread David Boreham
On 12/11/2012 7:49 PM, Evgeny Shishkin wrote: Yeah, s3700 looks promising, but sata interface is limiting factor for this drive. I'm looking towards SMART ssd http://www.storagereview.com/smart_storage_systems_optimus_sas_enterprise_ssd_review What don't you like about SATA ? I prefer to avo

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Mark Kirkwood
On 12/12/12 15:41, David Boreham wrote: On 12/11/2012 7:38 PM, Evgeny Shishkin wrote: Which drives would you recommend? Besides intel 320 and 710. Those are the only drive types we have deployed in servers at present (almost all 710, but we have some 320 for less mission-critical machines). Th

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Evgeny Shishkin
On Dec 12, 2012, at 6:41 AM, David Boreham wrote: > On 12/11/2012 7:38 PM, Evgeny Shishkin wrote: >> Which drives would you recommend? Besides intel 320 and 710. > Those are the only drive types we have deployed in servers at present (almost > all 710, but we have some 320 for less mission-crit

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Craig Ringer
On 12/12/2012 10:13 AM, Evgeny Shishkin wrote: > > Yes, i am aware of this issue. Never experienced this neither on intel > 520, no ocz vertex 3. > I wouldn't trust either of those drives. The 520 doesn't have Intel's " Enhanced Power Loss Data Protection"; it's going to lose its buffers if it los

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread David Boreham
On 12/11/2012 7:38 PM, Evgeny Shishkin wrote: Which drives would you recommend? Besides intel 320 and 710. Those are the only drive types we have deployed in servers at present (almost all 710, but we have some 320 for less mission-critical machines). The new DC-S3700 Series looks nice too, but

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Evgeny Shishkin
On Dec 12, 2012, at 6:26 AM, David Boreham wrote: > On 12/11/2012 7:20 PM, Evgeny Shishkin wrote: >> Oh, there is no 100% safe system. > In this case we're discussing specifically "safety in the event of power loss > shortly after the drive indicates to the controller that it has committed a >

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread David Boreham
On 12/11/2012 7:20 PM, Evgeny Shishkin wrote: Oh, there is no 100% safe system. In this case we're discussing specifically "safety in the event of power loss shortly after the drive indicates to the controller that it has committed a write operation". Some drives do provide 100% safety against

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Evgeny Shishkin
On Dec 12, 2012, at 6:15 AM, David Boreham wrote: > On 12/11/2012 7:13 PM, Evgeny Shishkin wrote: >> Yes, i am aware of this issue. Never experienced this neither on intel 520, >> no ocz vertex 3. >> Have you heard of them on this list? > People have done plug-pull tests and reported the result

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread David Boreham
On 12/11/2012 7:13 PM, Evgeny Shishkin wrote: Yes, i am aware of this issue. Never experienced this neither on intel 520, no ocz vertex 3. Have you heard of them on this list? People have done plug-pull tests and reported the results on the list (sometime in the past couple of years). But you

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Evgeny Shishkin
On Dec 12, 2012, at 6:02 AM, Craig Ringer wrote: > On 12/12/2012 09:44 AM, Evgeny Shishkin wrote: >> So far, more than a year already, i bought consumer ssds with 300-400$ hw >> raid. Cost effective and fast, may be not very safe, but so far so good. All >> data protection measures from postgr

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Craig Ringer
On 12/12/2012 09:44 AM, Evgeny Shishkin wrote: > So far, more than a year already, i bought consumer ssds with 300-400$ > hw raid. Cost effective and fast, may be not very safe, but so far so > good. All data protection measures from postgresql are on, of course. You're aware that many low end SSD

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Rosser Schwarz
On Tue, Dec 11, 2012 at 5:17 PM, Evgeny Shishkin wrote: > Actually most of low-end SSDs don't do write caching, they do not have > enough ram for that. > AIUI, *all* SSDs do write-caching of a sort: writes are actually flushed to the NAND media by erasing, and then overwriting the erased space, a

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Evgeny Shishkin
On Dec 12, 2012, at 5:29 AM, Craig Ringer wrote: > On 12/12/2012 09:17 AM, Evgeny Shishkin wrote: >> >> Actually most of low-end SSDs don't do write caching, they do not have >> enough ram for that. Sandforce for example. >> > Or, worse, some of them do limited write caching but don't protect

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Craig Ringer
On 12/12/2012 09:17 AM, Evgeny Shishkin wrote: > > Actually most of low-end SSDs don't do write caching, they do not have > enough ram for that. Sandforce for example. > Or, worse, some of them do limited write caching but don't protect their write cache from power loss. Instant data corruption! I

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Evgeny Shishkin
On Dec 12, 2012, at 5:03 AM, Craig Ringer wrote: > On 12/12/2012 06:44 AM, Evgeny Shishkin wrote: >> >> On Dec 12, 2012, at 2:41 AM, Niels Kristian Schjødt >> wrote: >> >>> Are you using a hardware based raid controller with them? >>> >> Yes, of course. Hardware raid with cache and bbu is a

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Craig Ringer
On 12/12/2012 06:44 AM, Evgeny Shishkin wrote: > > On Dec 12, 2012, at 2:41 AM, Niels Kristian Schjødt > mailto:nielskrist...@autouncle.com>> wrote: > >> Are you using a hardware based raid controller with them? >> > Yes, of course. Hardware raid with cache and bbu is a must. You can't > get fast f

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Evgeny Shishkin
On Dec 12, 2012, at 2:41 AM, Niels Kristian Schjødt wrote: > Are you using a hardware based raid controller with them? > Yes, of course. Hardware raid with cache and bbu is a must. You can't get fast fsync without it. Also mdadm is a pain in the ass and is suitable only on amazon and other cl

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Niels Kristian Schjødt
Are you using a hardware based raid controller with them? Den 11/12/2012 20.11 skrev "Evgeny Shishkin" : > > On Dec 11, 2012, at 10:54 PM, Niels Kristian Schjødt < > nielskrist...@autouncle.com> wrote: > > And what is your experience so far? > > Increased tps by a factor of 10, database no longer

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Evgeny Shishkin
On Dec 11, 2012, at 10:54 PM, Niels Kristian Schjødt wrote: > And what is your experience so far? > Increased tps by a factor of 10, database no longer a limiting factor of application. And it is cheaper than brand rotating drives. > Den 11/12/2012 18.16 skrev "Evgeny Shishkin" : > > On De

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Niels Kristian Schjødt
And what is your experience so far? Den 11/12/2012 18.16 skrev "Evgeny Shishkin" : > > On Dec 11, 2012, at 5:35 PM, Niels Kristian Schjødt < > nielskrist...@autouncle.com> wrote: > > > > > Den 11/12/2012 kl. 14.29 skrev Craig Ringer : > > > >> On 12/11/2012 06:04 PM, Niels Kristian Schjødt wrote:

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Jeff Janes
On Tue, Dec 11, 2012 at 2:04 AM, Niels Kristian Schjødt wrote: > Den 11/12/2012 kl. 00.58 skrev Jeff Janes : > >> >> The fact that there is much more writing than reading tells me that >> most of your indexes are in RAM. The amount of index you are rapidly >> reading and dirtying is large enough

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Evgeny Shishkin
On Dec 11, 2012, at 5:35 PM, Niels Kristian Schjødt wrote: > > Den 11/12/2012 kl. 14.29 skrev Craig Ringer : > >> On 12/11/2012 06:04 PM, Niels Kristian Schjødt wrote: >>> >>> Maybe I should mention, that I never see more than max 5Gb out of my total >>> 32Gb being in use on the server… Can

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Niels Kristian Schjødt
Den 11/12/2012 kl. 14.29 skrev Craig Ringer : > On 12/11/2012 06:04 PM, Niels Kristian Schjødt wrote: >> >> Maybe I should mention, that I never see more than max 5Gb out of my total >> 32Gb being in use on the server… Can I somehow utilize more of it? > For an update-mostly workload it probabl

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Craig Ringer
On 12/11/2012 06:04 PM, Niels Kristian Schjødt wrote: > > Maybe I should mention, that I never see more than max 5Gb out of my total > 32Gb being in use on the server… Can I somehow utilize more of it? For an update-mostly workload it probably won't do you tons of good so long as all your indexes

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-11 Thread Niels Kristian Schjødt
Den 11/12/2012 kl. 00.58 skrev Jeff Janes : > On Mon, Dec 10, 2012 at 2:51 PM, Niels Kristian Schjødt > wrote: > >> synchronous_commit = off >> >> The pg_xlog folder has been moved onto the SSD array (md3), and symlinked >> back into the postgres dir. > > With synchronous_commit = off, or with

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-10 Thread Jeff Janes
On Mon, Dec 10, 2012 at 2:51 PM, Niels Kristian Schjødt wrote: > synchronous_commit = off > > The pg_xlog folder has been moved onto the SSD array (md3), and symlinked > back into the postgres dir. With synchronous_commit = off, or with large transactions, there is probably no advantage to movin

Re: [PERFORM] Do I have a hardware or a software problem?

2012-12-10 Thread Evgeny Shishkin
On Dec 11, 2012, at 2:51 AM, Niels Kristian Schjødt wrote: > Pitch > ## > I previously posted this question > http://archives.postgresql.org/pgsql-performance/2012-11/msg00289.php about a > performance i

[PERFORM] Do I have a hardware or a software problem?

2012-12-10 Thread Niels Kristian Schjødt
Pitch ## I previously posted this question http://archives.postgresql.org/pgsql-performance/2012-11/msg00289.php about a performance issue with an update query. The question evolved into a more general disc