Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-01 Thread Michael Stone
On Fri, Sep 30, 2005 at 01:41:22PM -0700, Josh Berkus wrote: Realistically, you can't do better than about 25MB/s on a single-threaded I/O on current Linux machines, What on earth gives you that idea? Did you drop a zero? Mike Stone ---(end of

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-05 Thread Michael Stone
On Mon, Oct 03, 2005 at 01:34:01PM -0700, Josh Berkus wrote: Realistically, you can't do better than about 25MB/s on a single-threaded I/O on current Linux machines, What on earth gives you that idea? Did you drop a zero? Nope, LOTS of testing, at OSDL, GreenPlum and Sun. For comparison, A

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-05 Thread Michael Stone
On Sat, Oct 01, 2005 at 06:19:41PM +0200, Martijn van Oosterhout wrote: COPY TO /dev/null WITH binary 13MB/s55% user 45% system (ergo, CPU bound) [snip] the most expensive. But it does point out that the whole process is probably CPU bound more than anything else. Note that 45% of that

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-05 Thread Michael Stone
On Tue, Oct 04, 2005 at 12:43:10AM +0300, Hannu Krosing wrote: Just FYI, I run a count(*) on a 15.6GB table on a lightly loaded db and it run in 163 sec. (Dual opteron 2.6GHz, 6GB RAM, 6 x 74GB 15k disks in RAID10, reiserfs). A little less than 100MB sec. And none of that 15G table is in the

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-05 Thread Michael Stone
On Wed, Oct 05, 2005 at 11:24:07AM -0400, Luke Lonergan wrote: Nope - it would be disk wait. I said I/O overhead; i.e., it could be the overhead of calling the kernel for I/O's. E.g., the following process is having I/O problems: time dd if=/dev/sdc of=/dev/null bs=1 count=1000

Re: [HACKERS] [PERFORM] A Better External Sort?

2005-10-06 Thread Michael Stone
On Wed, Oct 05, 2005 at 04:55:51PM -0700, Luke Lonergan wrote: You've proven my point completely. This process is bottlenecked in the CPU. The only way to improve it would be to optimize the system (libc) functions like fread where it is spending most of it's time. Or to optimize its IO