e: [PERFORM] Hardware recommendations
>
>
>
> > -Original Message-
> > From: Greg Smith [mailto:g...@2ndquadrant.com]
> > Sent: Saturday, December 11, 2010 2:18 AM
> > To: Benjamin Krajmalnik
> > Cc: pgsql-performance@postgresql.org
> > Subject: R
> -Original Message-
> From: Greg Smith [mailto:g...@2ndquadrant.com]
> Sent: Saturday, December 11, 2010 2:18 AM
> To: Benjamin Krajmalnik
> Cc: pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] Hardware recommendations
>
>
> What sort of total read
Benjamin Krajmalnik wrote:
I am already having serious I/O bottlenecks with iostat -x showing extended
periods where the disk subsystem on the data partition (the one with all the
random i/o) at over 85% busy. The system is running FreeBSD 7.2 amd64 and
PostgreSQL 8.4.4 on amd64-portbld-freeb
John W Strange wrote:
http://www.fusionio.com/products/iodrive/ - BEST in slot currently IMHO.
http://www.intel.com/design/flash/nand/extreme/index.htm?wapkw=(X25-E) - not a
bad alternative.
The FusionIO drives are OK, so long as you don't mind the possibility
that your system will be down
> The "common knowledge" you based that comment on, may
> actually not be very up-to-date anymore. Current
> consumer-grade SSD's can achieve up to 200MB/sec when
> writing sequentially and they can probably do that a lot
> more consistent than a hard disk.
>
> Have a look here: http://www.anandt
On Fri, Dec 10, 2010 at 11:05 AM, Arjen van der Meijden
wrote:
>
> On 10-12-2010 18:57 Arjen van der Meijden wrote:
>>
>> Have a look here: http://www.anandtech.com/show/2829/21
>> The sequential writes-graphs consistently put several SSD's at twice the
>> performance of the VelociRaptor 300GB 10k
On 10-12-2010 18:57 Arjen van der Meijden wrote:
Have a look here: http://www.anandtech.com/show/2829/21
The sequential writes-graphs consistently put several SSD's at twice the
performance of the VelociRaptor 300GB 10k rpm disk and that's a test
from over a year old, current SSD's have increase
On 10-12-2010 14:58 Andy wrote:
We use ZFS and use SSDs for both the log device and L2ARC. All
disks and SSDs are behind a 3ware with BBU in single disk mode.
Out of curiosity why do you put your log on SSD? Log is all
sequential IOs, an area in which SSD is not any faster than HDD. So
I'd thi
> We use ZFS and use SSDs for both the log device and
> L2ARC. All disks
> and SSDs are behind a 3ware with BBU in single disk
> mode.
Out of curiosity why do you put your log on SSD? Log is all sequential IOs, an
area in which SSD is not any faster than HDD. So I'd think putting log on SSD
On Wed, Dec 8, 2010 at 3:03 PM, Benjamin Krajmalnik wrote:
> I need to build a new high performance server to replace our current
> production database server.
We run FreeBSD 8.1 with PG 8.4 (soon to upgrade to PG 9). Hardware is:
Supermicro 2u 6026T-NTR+
2x Intel Xeon E5520 Nehalem 2.26GHz
6:09 AM
To: Scott Marlowe
Cc: Benjamin Krajmalnik; John W Strange; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Hardware recommendations
On Thu, Dec 9, 2010 at 04:28, Scott Marlowe wrote:
> On Wed, Dec 8, 2010 at 5:03 PM, Benjamin Krajmalnik
> wrote:
>> My biggest concern wi
On Thu, Dec 9, 2010 at 04:28, Scott Marlowe wrote:
> On Wed, Dec 8, 2010 at 5:03 PM, Benjamin Krajmalnik
> wrote:
>> My biggest concern with SSD drives is their life expectancy,
>
> Generally that's not a big issue, especially as the SSDs get larger.
> Being able to survive a power loss without
-Original Message-
From: pgsql-performance-ow...@postgresql.org
[mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Andy
Sent: Wednesday, December 08, 2010 5:24 PM
To: Marti Raudsepp
Cc: pgsql-performance@postgresql.org; Benjamin Krajmalnik
Subject: Re: [PERFORM] Hardware
On Wed, Dec 8, 2010 at 5:03 PM, Benjamin Krajmalnik wrote:
> John,
>
> The platform is a network monitoring system, so we have quite a lot of
> inserts/updates (every data point has at least one record insert as well as
> at least 3 record updates). The management GUI has a lot of selects. We
Sent from my android device.
-Original Message-
From: Benjamin Krajmalnik
To: pgsql-performance@postgresql.org
Sent: Wed, 08 Dec 2010 17:14
Subject: [PERFORM] Hardware recommendations
Received: from mx2.hub.org [200.46.204.254] by mail.pengdows.com with SMTP
(EHLO mx2.hub.org)
(ArGo
Sent from my android device.
-Original Message-
From: Benjamin Krajmalnik
To: pgsql-performance@postgresql.org
Sent: Wed, 08 Dec 2010 17:14
Subject: [PERFORM] Hardware recommendations
Received: from mx2.hub.org [200.46.204.254] by mail.pengdows.com with SMTP
(EHLO mx2.hub.org)
(ArGo
Sent from my android device.
-Original Message-
From: Benjamin Krajmalnik
To: pgsql-performance@postgresql.org
Sent: Wed, 08 Dec 2010 17:14
Subject: [PERFORM] Hardware recommendations
Received: from mx2.hub.org [200.46.204.254] by mail.pengdows.com with SMTP
(EHLO mx2.hub.org)
(ArGo
Sent from my android device.
-Original Message-
From: Benjamin Krajmalnik
To: pgsql-performance@postgresql.org
Sent: Wed, 08 Dec 2010 17:14
Subject: [PERFORM] Hardware recommendations
Received: from mx2.hub.org [200.46.204.254] by mail.pengdows.com with SMTP
(EHLO mx2.hub.org)
(ArGo
> > If you are IO-bound, you might want to consider using
> SSD.
> >
> > A single SSD could easily give you more IOPS than 16
> 15k SAS in RAID 10.
>
> Are there any that don't risk your data on power loss, AND
> are cheaper
> than SAS RAID 10?
>
Vertex 2 Pro has a built-in supercapacitor to s
Sent from my android device.
-Original Message-
From: Benjamin Krajmalnik
To: pgsql-performance@postgresql.org
Sent: Wed, 08 Dec 2010 17:14
Subject: [PERFORM] Hardware recommendations
Received: from mx2.hub.org [200.46.204.254] by mail.pengdows.com with SMTP
(EHLO mx2.hub.org)
(ArGo
On Thu, Dec 9, 2010 at 01:26, Andy wrote:
> If you are IO-bound, you might want to consider using SSD.
>
> A single SSD could easily give you more IOPS than 16 15k SAS in RAID 10.
Are there any that don't risk your data on power loss, AND are cheaper
than SAS RAID 10?
Regards,
Marti
--
Sent vi
Ben,
It would help if you could tell us a bit more about the read/write mix and
transaction requirements. *IF* you are heavy writes I would suggest moving off
the RAID1 configuration to a RAID10 setup. I would highly suggest looking at
SLC based solid state drives or if your budget has legs, l
John,
The platform is a network monitoring system, so we have quite a lot of
inserts/updates (every data point has at least one record insert as well as at
least 3 record updates). The management GUI has a lot of selects. We are
refactoring the database to some degree to aid in the performanc
Sent from my android device.
-Original Message-
From: Benjamin Krajmalnik
To: pgsql-performance@postgresql.org
Sent: Wed, 08 Dec 2010 17:14
Subject: [PERFORM] Hardware recommendations
Received: from mx2.hub.org [200.46.204.254] by mail.pengdows.com with SMTP
(EHLO mx2.hub.org)
(ArGo
If you are IO-bound, you might want to consider using SSD.
A single SSD could easily give you more IOPS than 16 15k SAS in RAID 10.
--- On Wed, 12/8/10, Benjamin Krajmalnik wrote:
> From: Benjamin Krajmalnik
> Subject: [PERFORM] Hardware recommendations
> To: pgsql-performance@postgresql.org
Gregory S. Williamson wrote:
> Nitpicking --
>
> Perhaps the 4th data line is meant to be:
> Inserts in separate transactions 2500 inserts/second
>^^^
Oh, yes, sorry. It is:
> Sorry to be replying late. Here is what I found.
>
> fsync o
: Matt Clark
Cc: Ron Johnson; PgSQL Performance ML
Subject:Re: [PERFORM] Hardware recommendations to scale to silly load
Matt Clark wrote:
> > Just a data point, but on my Dual Xeon 2.4Gig machine with a 10k SCSI
> > drive I can do 4k inserts/second if I turn fsync off.
Matt Clark wrote:
> > Just a data point, but on my Dual Xeon 2.4Gig machine with a 10k SCSI
> > drive I can do 4k inserts/second if I turn fsync off. If you have a
> > battery-backed controller, you should be able to do the same. (You will
> > not need to turn fsync off --- fsync will just be fas
On Tue, 2003-09-02 at 11:01, Vivek Khera wrote:
> > "AS" == Andrew Sullivan <[EMAIL PROTECTED]> writes:
>
> AS> On Fri, Aug 29, 2003 at 12:05:03AM -0700, William Yu wrote:
> >> We should see a boost when we move to 64-bit Linux and hopefully another
> >> one when NUMA for Linux is production-
Vivek Khera <[EMAIL PROTECTED]> writes:
> The front-end small processes get to deal with your dialup customers
> trickling down the data since it buffers your backend for you.
Huh. Well, I used to think this. But I think I was wrong. I used to have
apache proxy servers running in front of the mo
> "MC" == Matt Clark <[EMAIL PROTECTED]> writes:
MC> And concurrency is very high, because it's a web app, and each
MC> httpd has one connection to PG, and there can be hundreds of
MC> active httpd processes. Some kind of connection pooling scheme
MC> might be in order when there are that man
> "AS" == Andrew Sullivan <[EMAIL PROTECTED]> writes:
AS> On Fri, Aug 29, 2003 at 12:05:03AM -0700, William Yu wrote:
>> We should see a boost when we move to 64-bit Linux and hopefully another
>> one when NUMA for Linux is production-stable.
AS> According to the people who've worked with SG
> Just a data point, but on my Dual Xeon 2.4Gig machine with a 10k SCSI
> drive I can do 4k inserts/second if I turn fsync off. If you have a
> battery-backed controller, you should be able to do the same. (You will
> not need to turn fsync off --- fsync will just be fast because of the
> disk dr
> SELECT
> IF
> BEGIN
> INSERT
>or
> UPDATE
> COMMIT;
>
> He says his current h/w peaks at 1/10th that rate.
>
> My question is: is that current peak rate ("300 inserts/updates
> *or* 2500 selects") based upon 1 connection, or many c
> My question is: is that current peak rate ("300 inserts/updates
> *or* 2500 selects") based upon 1 connection, or many connections?
> With 4 CPUs, and a 4 disk RAID10, I wouldn't be surprised if 4 con-
> current connections gives the optimum speed.
Optimum number of active workers is probably
On Fri, 2003-08-29 at 21:44, Bruce Momjian wrote:
> matt wrote:
> > > Are you *sure* about that 3K updates/inserts per second xlates
> > > to 10,800,000 per hour. That, my friend, is a WHOLE HECK OF A LOT!
> >
> > Yup, I know!
>
> Just a data point, but on my Dual Xeon 2.4Gig machine with
matt wrote:
> > Are you *sure* about that 3K updates/inserts per second xlates
> > to 10,800,000 per hour. That, my friend, is a WHOLE HECK OF A LOT!
>
> Yup, I know!
Just a data point, but on my Dual Xeon 2.4Gig machine with a 10k SCSI
drive I can do 4k inserts/second if I turn fsync off
On Fri, 2003-08-29 at 11:33, William Yu wrote:
> Shridhar Daithankar wrote:
[snip]
> > I am sure. But is 64 bit environment, Xeon is not the compitition. It's PA-RSC-
> > 8700, ultraSparcs, Power series and if possible itanium.
>
> Well, just because the Opteron is 64-bit doesn't mean it's direct
Shridhar Daithankar wrote:
Just a guess here but does a precompiled postgresql for x86 and a x86-64
optimized one makes difference?
>
> Opteron is one place on earth you can watch difference between 32/64
> bit on same machine. Can be handy at times..
I don't know yet. I tried building a 64-bit ke
On Fri, Aug 29, 2003 at 12:05:03AM -0700, William Yu wrote:
> We should see a boost when we move to 64-bit Linux and hopefully another
> one when NUMA for Linux is production-stable.
According to the people who've worked with SGIs, NUMA actually seems
to make things worse. It has something to do
On Fri, 2003-08-29 at 03:18, Shridhar Daithankar wrote:
> On 29 Aug 2003 at 0:05, William Yu wrote:
>
> > Shridhar Daithankar wrote:
[snip]
> > As for performance, the scaling is magnificient -- even when just using
> > PAE instead of 64-bit addressing. At low transaction counts, it's only
> > ~
On 29 Aug 2003 at 0:05, William Yu wrote:
> Shridhar Daithankar wrote:
> >> Be careful here, we've seen that with the P4 Xeon's that are
> >>hyper-threaded and a system that has very high disk I/O causes the
> >>system to be sluggish and slow. But after disabling the hyper-threading
> >>itself, ou
> We should see a boost when we move to 64-bit Linux and hopefully another
> one when NUMA for Linux is production-stable.
Assuming SCO doesn't make them remove it :P
Chris
---(end of broadcast)---
TIP 8: explain analyze is your friend
Shridhar Daithankar wrote:
Be careful here, we've seen that with the P4 Xeon's that are
hyper-threaded and a system that has very high disk I/O causes the
system to be sluggish and slow. But after disabling the hyper-threading
itself, our system flew..
Anybody has opteron working? Hows' the perform
On Thu, 2003-08-28 at 12:37, Matt Clark wrote:
> > Ok.. I would be surprised if you needed much more actual CPU power. I
> > suspect they're mostly idle waiting on data -- especially with a Quad
> > Xeon (shared memory bus is it not?).
>
> In reality the CPUs get pegged: about 65% PG and 35% syste
> Ok.. I would be surprised if you needed much more actual CPU power. I
> suspect they're mostly idle waiting on data -- especially with a Quad
> Xeon (shared memory bus is it not?).
In reality the CPUs get pegged: about 65% PG and 35% system. But I agree that memory
throughput and latency is an
> Just how big do you expect your DB to grow? For a 1GB disk-space
> database, I'd probably just splurge for an SSD hooked up either via
> SCSI or FibreChannel. Heck, up to about 5Gb or so it is not that
> expensive (about $25k) and adding another 5Gb should set you back
> probably another $20k.
On Wed, Aug 27, 2003 at 05:49:25PM +0100, matt wrote:
>
> I'm also looking at renting equipment, or even trying out IBM/HP's
> 'on-demand' offerings.
To handle that kind of load, you're not going to be able to do it
with cheap hardware. Renting may be your answer.
a
--
Andrew Sullivan
On 27 Aug 2003, matt wrote:
> > You probably, more than anything, should look at some kind of
> > superfast, external storage array
>
> Yeah, I think that's going to be a given. Low end EMC FibreChannel
> boxes can do around 20,000 IOs/sec, which is probably close to good
> enough.
>
> You men
> I need to increase the overall performance by a factor of 10, while at
> the same time the DB size increases by a factor of 50. e.g. 3000
> inserts/updates or 25,000 selects per second, over a 25GB database with
> most used tables of 5,000,000 and 1,000,000 rows.
Ok.. I would be surprised if yo
sm> On 27 Aug 2003, matt wrote:
>> My app is likely to come under some serious load in the next 6 months,
>> but the increase will be broadly predictable, so there is time to throw
>> hardware at the problem.
>>
>> Currently I have a ~1GB DB, with the largest (and most commonly accessed
>> and u
On 28 Aug 2003 at 11:05, Chris Bowlby wrote:
> On Tue, 2003-08-26 at 23:59, Ron Johnson wrote:
>
> > What a fun exercises. Ok, lets see:
> > Postgres 7.3.4
> > RH AS 2.1
> > 12GB RAM
> > motherboard with 64 bit 66MHz PCI slots
> > 4 - Xenon 3.0GHz (1MB cache) CPUs
> > 8 - 36GB 15K RPM as RAID10
On Tue, 2003-08-26 at 23:59, Ron Johnson wrote:
> What a fun exercises. Ok, lets see:
> Postgres 7.3.4
> RH AS 2.1
> 12GB RAM
> motherboard with 64 bit 66MHz PCI slots
> 4 - Xenon 3.0GHz (1MB cache) CPUs
> 8 - 36GB 15K RPM as RAID10 on a 64 bit 66MHz U320 controller
> having 512MB cache (for
On Thu, 2003-08-28 at 03:17, matt wrote:
> > Are you *sure* about that 3K updates/inserts per second xlates
> > to 10,800,000 per hour. That, my friend, is a WHOLE HECK OF A LOT!
>
> Yup, I know!
>
> > During the 1 hour surge, will SELECTs at 10 minutes after the
> > hour depend on INSER
> Are you *sure* about that 3K updates/inserts per second xlates
> to 10,800,000 per hour. That, my friend, is a WHOLE HECK OF A LOT!
Yup, I know!
> During the 1 hour surge, will SELECTs at 10 minutes after the
> hour depend on INSERTs at 5 minutes after the hour?
Yes, they do. It's a
2003-08-27 ragyogó napján Bill Moran ezt üzente:
> With FreeBSD, you have jails, which allow multiple users to share
> hardware without having to worry about user A looking at user B's
> stuff. Does such a paradigm exist on any heavy iron? I have no
Of course. All IBM hw can do this, because on
2003-08-27 ragyogó napján matt ezt üzente:
> Yeah, I can imagine getting 5% extra from a slim kernel and
> super-optimised PG.
Hm, about 20%, but only for the correctness - 20% not help you also :(
> The FS is ext3, metadata journaling (the default), mounted noatime.
Worst fs under linux :) Try
On Wed, 2003-08-27 at 21:26, Bill Moran wrote:
> Christopher Browne wrote:
> > Martha Stewart called it a Good Thing [EMAIL PROTECTED] (matt)wrote:
[snip]
> With FreeBSD, you have jails, which allow multiple users to share
> hardware without having to worry about user A looking at user B's
> stuff.
matt wrote:
I'm wondering if the good people out there could perhaps give me some
pointers on suitable hardware to solve an upcoming performance issue.
I've never really dealt with these kinds of loads before, so any
experience you guys have would be invaluable. Apologies in advance for
the amoun
On Tue, 2003-08-26 at 20:35, matt wrote:
> I'm wondering if the good people out there could perhaps give me some
> pointers on suitable hardware to solve an upcoming performance issue.
> I've never really dealt with these kinds of loads before, so any
> experience you guys have would be invaluable
Christopher Browne wrote:
Martha Stewart called it a Good Thing [EMAIL PROTECTED] (matt)wrote:
I'm also looking at renting equipment, or even trying out IBM/HP's
'on-demand' offerings.
You're assuming that this is likely to lead to REAL savings, and that
seems unlikely.
During the recent power out
After takin a swig o' Arrakan spice grog, [EMAIL PROTECTED]
("scott.marlowe") belched out... :-):
> whether you like it or not, you're gonna need heavy iron if you need
> to do this all in one hour once a week.
The other thing worth considering is trying to see if there is a way
of partitioning th
On Wed, Aug 27, 2003 at 02:35:13AM +0100, matt wrote:
> I need to increase the overall performance by a factor of 10, while at
> the same time the DB size increases by a factor of 50. e.g. 3000
> inserts/updates or 25,000 selects per second, over a 25GB database with
> most used tables of 5,000,0
> Are you sure? Have you tested the overall application to see if possibly
> you gain more on insert performance than you lose on select performanc?
Unfortunately dropping any of the indexes results in much worse select
performance that is not remotely clawed back by the improvement in
insert per
Martha Stewart called it a Good Thing [EMAIL PROTECTED] (matt)wrote:
> I'm also looking at renting equipment, or even trying out IBM/HP's
> 'on-demand' offerings.
You're assuming that this is likely to lead to REAL savings, and that
seems unlikely.
During the recent power outage in the NorthEast,
> Don't know how "cheap" they are.
>
> I have an app that does large batch updates. I found that if I dropped
> the indexes, did the updates and recreated the indexes, it was faster
> than doing the updates while the indexes were intact.
Yeah, unfortunately it's not batch work, but real time fina
On 27 Aug 2003, matt wrote:
> I'm wondering if the good people out there could perhaps give me some
> pointers on suitable hardware to solve an upcoming performance issue.
> I've never really dealt with these kinds of loads before, so any
> experience you guys have would be invaluable. Apologies
> You probably, more than anything, should look at some kind of
> superfast, external storage array
Yeah, I think that's going to be a given. Low end EMC FibreChannel
boxes can do around 20,000 IOs/sec, which is probably close to good
enough.
You mentioned using multiple RAID controllers as a b
matt wrote:
Are you sure? Have you tested the overall application to see if possibly
you gain more on insert performance than you lose on select performanc?
Unfortunately dropping any of the indexes results in much worse select
performance that is not remotely clawed back by the improvement in
ins
69 matches
Mail list logo