Re: [PERFORM] Comparative performance

2005-10-03 Thread Joe
PFC wrote: Even though this query isn't that optimized, it's still only 16 milliseconds. Why does it take this long for PHP to get the results ? Can you try pg_query'ing this exact same query, FROM PHP, and timing it with getmicrotime() ? That query took about 27 msec in actual

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Ron Peacetree
OK, change "performance" to "single thread performance" and we still have a valid starting point for a discussion. Ron -Original Message- From: Gregory Maxwell <[EMAIL PROTECTED]> Sent: Oct 3, 2005 8:19 PM To: Ron Peacetree <[EMAIL PROTECTED]> Subject: Re: [HACKERS] [PERFORM] A Better Ex

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Ron Peacetree
Let's pretend we get a 24HD HW RAID solution like that J Baker says he has access to and set it up as a RAID 10. Assuming it uses two 64b 133MHz PCI-X busses and has the fastest HDs available on it, Jeff says he can hit ~1GBps of XFS FS IO rate with that set up (12*83.3MBps= 1GBps). Josh says th

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Josh Berkus
Jeffrey, > I guess database reads are different, but I remain unconvinced that they > are *fundamentally* different. After all, a tab-delimited file (my sort > workload) is a kind of database. Unfortunately, they are ... because of CPU overheads. I'm basing what's "reasonable" for data writes

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Michael Stone
On Mon, Oct 03, 2005 at 01:34:01PM -0700, Josh Berkus wrote: >Realistically, you can't do better than about 25MB/s on a > single-threaded I/O on current Linux machines, What on earth gives you that idea? Did you drop a zero? Nope, LOTS of testing, at OSDL, GreenPlum and Sun. For comparison,

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Luke Lonergan
Hannu, On 10/3/05 2:43 PM, "Hannu Krosing" <[EMAIL PROTECTED]> wrote: > Just FYI, I run a count(*) on a 15.6GB table on a lightly loaded db and > it run in 163 sec. (Dual opteron 2.6GHz, 6GB RAM, 6 x 74GB 15k disks in > RAID10, reiserfs). A little less than 100MB sec. This confirms our findings

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Josh Berkus
Michael, > >Nope, LOTS of testing, at OSDL, GreenPlum and Sun. For comparison, A > >Big-Name Proprietary Database doesn't get much more than that either. > > You seem to be talking about database IO, which isn't what you said. Right, well, it was what I meant. I failed to specify, that's all.

Re: [PERFORM] database bloat, but vacuums are done, and fsm seems

2005-10-03 Thread Simon Riggs
On Wed, 2005-09-28 at 09:07 +0200, hubert depesz lubaczewski wrote: > database has quite huge load of updates, but i thought that vacum will > guard me from database bloat, but from what i observed it means that > vacuuming of b-tree indices is somewhat faulty. No, thats perfectly normal. Indices

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Jeffrey W. Baker
On Mon, 2005-10-03 at 14:16 -0700, Josh Berkus wrote: > Jeff, > > > > Nope, LOTS of testing, at OSDL, GreenPlum and Sun. For comparison, A > > > Big-Name Proprietary Database doesn't get much more than that either. > > > > I find this claim very suspicious. I get single-threaded reads in > > ex

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Luke Lonergan
Jeff, Josh, On 10/3/05 2:16 PM, "Josh Berkus" wrote: > Jeff, > >>> Nope, LOTS of testing, at OSDL, GreenPlum and Sun. For comparison, A >>> Big-Name Proprietary Database doesn't get much more than that either. >> >> I find this claim very suspicious. I get single-threaded reads in >> excess

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Ron Peacetree
Jeff, are those _burst_ rates from HD buffer or _sustained_ rates from actual HD media? Rates from IO subsystem buffer or cache are usually considerably higher than Average Sustained Transfer Rate. Also, are you measuring _raw_ HD IO (bits straight off the platters, no FS or other overhead) or _c

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Josh Berkus
Jeff, > > Nope, LOTS of testing, at OSDL, GreenPlum and Sun. For comparison, A > > Big-Name Proprietary Database doesn't get much more than that either. > > I find this claim very suspicious. I get single-threaded reads in > excess of 1GB/sec with XFS and > 250MB/sec with ext3. Database reads?

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Jeffrey W. Baker
On Mon, 2005-10-03 at 13:34 -0700, Josh Berkus wrote: > Michael, > > > >Realistically, you can't do better than about 25MB/s on a > > > single-threaded I/O on current Linux machines, > > > > What on earth gives you that idea? Did you drop a zero? > > Nope, LOTS of testing, at OSDL, GreenPlum and

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Josh Berkus
Tom, > Raising work_mem to a gig should result in about five runs, needing only > one pass, which is really going to be as good as it gets. If you could > not see any difference then I see little hope for the idea that reducing > the number of merge passes will help. Right. It *should have*, bu

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-03 Thread Josh Berkus
Michael, > >Realistically, you can't do better than about 25MB/s on a > > single-threaded I/O on current Linux machines, > > What on earth gives you that idea? Did you drop a zero? Nope, LOTS of testing, at OSDL, GreenPlum and Sun. For comparison, A Big-Name Proprietary Database doesn't get mu

Re: [PERFORM] Query seem to slow if table have more than 200 million rows

2005-10-03 Thread Qingqing Zhou
""Ahmad Fajar"" <[EMAIL PROTECTED]> wrote > Hi Qingqing, > > I don't know whether the statistic got is bad or good, this is the > statistic: Please do it in this way: 1. Start postmaster with "stats_start_collector=true" and "stats_block_level=true". 2. Use psql connect it, do something like t

Re: [PERFORM] Ultra-cheap NVRAM device

2005-10-03 Thread Vivek Khera
On Oct 3, 2005, at 7:02 AM, Steinar H. Gunderson wrote: Anybody know a good reason why you can't put a WAL on this, and enjoy a hefty speed boost for a fraction of the price of a traditional SSD? (Yes, it's SATA, not PCI, so the throughput is not all that impressive -- but still, it's got

Re: [PERFORM] Ultra-cheap NVRAM device

2005-10-03 Thread Ron Peacetree
Nah. It's still not right. It needs: 1= full PCI, preferably at least 64b 133MHz PCI-X, bandwidth. A RAM card should blow the doors off the fastest commodity RAID setup you can build. 2= 8-16 DIMM slots 3= a standard battery type that I can pick up spares for easily 4= ECC support If it had all

Re: [PERFORM] Ultra-cheap NVRAM device

2005-10-03 Thread Jeffrey W. Baker
On Mon, 2005-10-03 at 11:15 -0600, Dan Harris wrote: > On Oct 3, 2005, at 5:02 AM, Steinar H. Gunderson wrote: > > > I thought this might be interesting, not the least due to the > > extremely low > > price ($150 + the price of regular DIMMs): > > > > > > > > This has been posted before, and th

Re: [PERFORM] Ultra-cheap NVRAM device

2005-10-03 Thread Dan Harris
On Oct 3, 2005, at 5:02 AM, Steinar H. Gunderson wrote: I thought this might be interesting, not the least due to the extremely low price ($150 + the price of regular DIMMs): Replying before my other post came through.. It looks like their benchmarks are markedly improved since the last

Re: [PERFORM] Ultra-cheap NVRAM device

2005-10-03 Thread Dan Harris
On Oct 3, 2005, at 5:02 AM, Steinar H. Gunderson wrote: I thought this might be interesting, not the least due to the extremely low price ($150 + the price of regular DIMMs): This has been posted before, and the main reason nobody got very excited is that: a) it only uses the PCI bus

Re: [PERFORM] Alternative to a temporary table

2005-10-03 Thread Steve Atkins
On Mon, Oct 03, 2005 at 11:47:52AM -0400, Steven Rosenstein wrote: > I currently create a temporary table to hold the selected server_id's and > characteristics. I then join this temp table with other data tables to > produce my reports. My reason for using the temporary table method is that > t

[PERFORM] Alternative to a temporary table

2005-10-03 Thread Steven Rosenstein
I have a PHP web-based application where a temporary list of servers and their characteristics (each represented by a unique numeric server_id) is extracted from a master server list based on a number of dynamic and user-selected criteria (can the user view the server, is it on-line, is it a me

Re: [PERFORM] URGENT: pg_statistic_relid_att_index has gone

2005-10-03 Thread Tom Lane
Harald Lau <[EMAIL PROTECTED]> writes: > on a 7.2 System (Suse-Linux) I got an error "duplicate key in unique > index pg_statistic_relid_att_index" (think it was while vacuuming) > I REINDEXd the database. > Now the table pg_statistic_relid_att_index is completely gone. > Has anybody an advise? Du

Re: [PERFORM] URGENT: pg_statistic_relid_att_index has gone

2005-10-03 Thread Harald Lau
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > on a 7.2 System (Suse-Linux) I got an error "duplicate key in unique > index pg_statistic_relid_att_index" (think it was while vacuuming) > > I REINDEXd the database. > Now the table pg_statistic_relid_att_index is completely gone. go searching the

[PERFORM] URGENT: pg_statistic_relid_att_index has gone

2005-10-03 Thread Harald Lau
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, on a 7.2 System (Suse-Linux) I got an error "duplicate key in unique index pg_statistic_relid_att_index" (think it was while vacuuming) I REINDEXd the database. Now the table pg_statistic_relid_att_index is completely gone. Has anybody an advise

[PERFORM] Ultra-cheap NVRAM device

2005-10-03 Thread Steinar H. Gunderson
I thought this might be interesting, not the least due to the extremely low price ($150 + the price of regular DIMMs): http://www.tomshardware.com/storage/20050907/index.html Anybody know a good reason why you can't put a WAL on this, and enjoy a hefty speed boost for a fraction of the price of