Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-27 Thread Stephan Szabo
On Sun, 27 Nov 2005, Luke Lonergan wrote: > Stephan, > > On 11/27/05 7:48 AM, "Stephan Szabo" <[EMAIL PROTECTED]> wrote: > > > On Sun, 27 Nov 2005, Luke Lonergan wrote: > > > >> Has anyone done the math.on the original post? 5TB takes how long to > >> scan once? If you want to wait less than a

Re: [PERFORM] Open request for benchmarking input

2005-11-27 Thread David Lang
On Sun, 27 Nov 2005, Andreas Pflug wrote: David Lang wrote: Postgres needs to work on the low end stuff as well as the high end stuff or people will write their app to work with things that DO run on low end hardware and they spend much more money then is needed to scale the hardware up rat

Re: [PERFORM] Hardware/OS recommendations for large databases

2005-11-27 Thread Ron
At 02:11 PM 11/27/2005, Luke Lonergan wrote: Ron, On 11/27/05 9:10 AM, "Ron" <[EMAIL PROTECTED]> wrote: > Clever use of RAM can get a 5TB sequential scan down to ~17mins. > > Yes, it's a lot of data. But sequential scan times should be in the > mins or low single digit hours, not days. Partic

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-27 Thread Luke Lonergan
Stephan, On 11/27/05 7:48 AM, "Stephan Szabo" <[EMAIL PROTECTED]> wrote: > On Sun, 27 Nov 2005, Luke Lonergan wrote: > >> Has anyone done the math.on the original post? 5TB takes how long to >> scan once? If you want to wait less than a couple of days just for a >> seq scan, you'd better be in

Re: [PERFORM] Hardware/OS recommendations for large databases

2005-11-27 Thread Luke Lonergan
Ron, On 11/27/05 9:10 AM, "Ron" <[EMAIL PROTECTED]> wrote: > Clever use of RAM can get a 5TB sequential scan down to ~17mins. > > Yes, it's a lot of data. But sequential scan times should be in the > mins or low single digit hours, not days. Particularly if you use > RAM to maximum advantage.

Re: [PERFORM] Hardware/OS recommendations for large databases

2005-11-27 Thread Ron
At 01:18 AM 11/27/2005, Luke Lonergan wrote: For data warehousing its pretty well open and shut. To use all cpus and io channels on each query you will need mpp. Has anyone done the math.on the original post? 5TB takes how long to scan once? If you want to wait less than a couple of days ju

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-27 Thread Stephan Szabo
On Sun, 27 Nov 2005, Luke Lonergan wrote: > Has anyone done the math.on the original post? 5TB takes how long to > scan once? If you want to wait less than a couple of days just for a > seq scan, you'd better be in the multi-gb per second range. Err, I get about 31 megabytes/second to do 5TB in

Re: [PERFORM] Open request for benchmarking input

2005-11-27 Thread Andreas Pflug
David Lang wrote: These boxes don't look like being designed for a DB server. The first are very CPU bound, and the third may be a good choice for very large amounts of streamed data, but not optimal for TP random access. I don't know what you mean when you say that the first ones are CPU bo

Re: [PERFORM] Open request for benchmarking input

2005-11-27 Thread Steinar H. Gunderson
On Sat, Nov 26, 2005 at 01:57:47PM -0500, Qingqing Zhou wrote: > Don't forget TPCC (data > memory, with intensive updates). So the benchmarks > in my mind include TPCC, TPCH and TPCW. I'm lost in all those acronyms, but am I right in assuming that none of these actually push the planner very hard

Re: [PERFORM] Hardware/OS recommendations for large databases (

2005-11-27 Thread Luke Lonergan
Have you factored in how long it takes to build an index on 5TB? And the index size? Really, it's a whole different world at multi-TB, everything has to scale. Btw we don't just scan in parallel, we do all in parallel, check the sort number on this thread. Mpp is for the god box too. And you