Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
Jeffrey, On 1/31/06 8:09 PM, "Jeffrey W. Baker" <[EMAIL PROTECTED]> wrote: >> ... Prove it. > I think I've proved my point. Software RAID1 read balancing provides > 0%, 300%, 100%, and 100% speedup on 1, 2, 4, and 8 threads, > respectively. In the presence of random I/O, the results are even > b

Re: [PERFORM] partitioning and locking problems

2006-01-31 Thread Tom Lane
"Marc Morin" <[EMAIL PROTECTED]> writes: > Would like to understand the implications of changing postgres' > code/locking for rule changes and truncate to not require locking out > select statements? It won't work... regards, tom lane ---(end of

Re: [PERFORM] Sequential scan being used despite indexes

2006-01-31 Thread Michael Fuhr
On Wed, Feb 01, 2006 at 01:33:08PM +0900, James Russell wrote: > Reading about this issue further in the FAQ, it seems that I should ensure > that Postgres has adequate and accurate information about the tables in > question by regularly running VACUUM ANALYZE, something I don't do > currently. Ma

Re: [PERFORM] Sequential scan being used despite indexes

2006-01-31 Thread Christopher Kings-Lynne
Reading about this issue further in the FAQ, it seems that I should ensure that Postgres has adequate and accurate information about the tables in question by regularly running VACUUM ANALYZE, something I don't do currently. Well then you'll get rubbish performance always in PostgreSQL... I s

Re: [PERFORM] Sequential scan being used despite indexes

2006-01-31 Thread James Russell
[Sorry, my last reply didn't go to the list] Reading about this issue further in the FAQ, it seems that I should ensure that Postgres has adequate and accurate information about the tables in question by regularly running VACUUM ANALYZE, something I don't do currently. I disabled SeqScan as per

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Jeffrey W. Baker
On Tue, 2006-01-31 at 12:47 -0800, Luke Lonergan wrote: > Jeffrey, > > On 1/31/06 12:03 PM, "Jeffrey W. Baker" <[EMAIL PROTECTED]> wrote: > > Linux does balanced reads on software > > mirrors. I'm not sure why you think this can't improve bandwidth. It > > does improve streaming bandwidth as lon

Re: [PERFORM] Sequential scan being used despite indexes

2006-01-31 Thread Michael Fuhr
On Tue, Jan 31, 2006 at 07:29:51PM -0800, Joshua D. Drake wrote: > > Any ideas? > > What does explain analyze say? Also, have the tables been vacuumed and analyzed? -- Michael Fuhr ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster

Re: [PERFORM] Sequential scan being used despite indexes

2006-01-31 Thread Joshua D. Drake
Explaining: Hash Join (cost=337.93..1267.54 rows=180 width=35) Hash Cond: ("outer".message_id = "inner".message_id) -> Seq Scan on message_meta_data (cost=0.00..739.19 rows=37719 width=30) -> Hash (cost=337.79..337.79 rows=57 width=13) -> Nested Loop (cost=0.00..337.79 rows=57 width=

[PERFORM] Sequential scan being used despite indexes

2006-01-31 Thread James Russell
Hi there, I'm running a simple query with 2 inner joins (say A, B and C). Each of the join columns has indexes. If I run queries that join just A and B, or just B and C, postgres uses indexes. But if I run "A join B join C" then the "B join C" part starts using a sequential scan and I can't figure

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Jim C. Nasby
On Tue, Jan 31, 2006 at 03:19:38PM -0800, Luke Lonergan wrote: > > Well, the only problem with that is if the machine crashes for any > > reason you risk having the database corrupted (or at best losing some > > committed transactions). > > So, do you routinely turn off Linux write caching? If no

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
Steinar, On 1/31/06 5:26 PM, "Steinar H. Gunderson" <[EMAIL PROTECTED]> wrote: > cassarossa:~# grep md1 /proc/mdstat > md1 : active raid1 sdf6[1] sda6[0] > cassarossa:~# dd if=/dev/sda6 of=/dev/null bs=8k count=40 > [system at about 35% wait for I/O and 15% system, according to top] > 40+

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Steinar H. Gunderson
On Tue, Jan 31, 2006 at 12:47:10PM -0800, Luke Lonergan wrote: >> Linux does balanced reads on software >> mirrors. I'm not sure why you think this can't improve bandwidth. It >> does improve streaming bandwidth as long as the platter STR is more than >> the bus STR. > ... Prove it. FWIW, this i

Re: [PERFORM] Storing Digital Video

2006-01-31 Thread Matt Davies | Postgresql List
Rodrigo Madera wrote: I am concerned with performance issues involving the storage of DV on a database. I though of some options, which would be the most advised for speed? 1) Pack N frames inside a "container" and store the container to the db. 2) Store each frame in a separate record in the

[PERFORM] Storing Digital Video

2006-01-31 Thread Rodrigo Madera
I am concerned with performance issues involving the storage of DV on a database. I though of some options, which would be the most advised for speed? 1) Pack N frames inside a "container" and store the container to the db. 2) Store each frame in a separate record in the table "frames". 3) (type

[PERFORM] partitioning and locking problems

2006-01-31 Thread Marc Morin
We have a large database system designed around partitioning. Our application is characterized with - terabytes of data - billions of rows in dozens of base tables (and 100s of paritions) - 24x7 insert load of new data that cannot be stopped, data is time sensitive. - periodic reports that can h

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
Jim, On 1/31/06 3:12 PM, "Jim C. Nasby" <[EMAIL PROTECTED]> wrote: >> The alternating technique in mirroring might improve rotational latency for >> random seeking - a trick that Tandem exploited, but it won't improve >> bandwidth. > > Or just work in multiples of tracks, which would greatly red

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Jeffrey W. Baker
On Tue, 2006-01-31 at 09:00 -0800, Luke Lonergan wrote: > Jim, > > On 1/30/06 12:25 PM, "Jim C. Nasby" <[EMAIL PROTECTED]> wrote: > > > Why divide by 2? A good raid controller should be able to send read > > requests to both drives out of the mirrored set to fully utilize the > > bandwidth. Of co

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
PFC, On 1/31/06 3:11 PM, "PFC" <[EMAIL PROTECTED]> wrote: >> ... Prove it. >> > > (I have a software RAID1 on this desktop machine) > > It's a lot faster than a single disk for random reads when more than 1 > thread hits the disk, because it distributes reads to both disks. Thus, > application

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Jim C. Nasby
On Tue, Jan 31, 2006 at 02:52:57PM -0800, Luke Lonergan wrote: > It's because your alternating reads are skipping in chunks across the > platter. Disks work at their max internal rate when reading sequential > data, and the cache is often built to buffer a track-at-a-time, so > alternating pieces

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread PFC
Linux does balanced reads on software mirrors. I'm not sure why you think this can't improve bandwidth. It does improve streaming bandwidth as long as the platter STR is more than the bus STR. ... Prove it. (I have a software RAID1 on this desktop machine) It's a lot faster than

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
Jim, On 1/31/06 11:21 AM, "Jim C. Nasby" <[EMAIL PROTECTED]> wrote: > (BTW, I did some testing that seems to confirm this) > > Why couldn't you double the bandwidth? If you're doing a largish read > you should be able to do something like have drive a read the first > track, drive b the second,

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
Jeffrey, On 1/31/06 12:03 PM, "Jeffrey W. Baker" <[EMAIL PROTECTED]> wrote: > Then you've not seen Linux. :-D > Linux does balanced reads on software > mirrors. I'm not sure why you think this can't improve bandwidth. It > does improve streaming bandwidth as long as the platter STR is more th

Re: [PERFORM] Delete me

2006-01-31 Thread Jim C. Nasby
On Tue, Jan 31, 2006 at 06:33:14PM +, Richard Huxton wrote: > [EMAIL PROTECTED] wrote: > >Can you delete me from the mail list Please? > > Go to the website. > Click "community" > Click "mailing lists" > On the left-hand side click "Subscribe" > Fill in the form, changing action to "unsubscrib

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Jim C. Nasby
On Tue, Jan 31, 2006 at 09:00:30AM -0800, Luke Lonergan wrote: > Jim, > > On 1/30/06 12:25 PM, "Jim C. Nasby" <[EMAIL PROTECTED]> wrote: > > > Why divide by 2? A good raid controller should be able to send read > > requests to both drives out of the mirrored set to fully utilize the > > bandwidth

Re: [PERFORM] Delete me

2006-01-31 Thread Richard Huxton
[EMAIL PROTECTED] wrote: Can you delete me from the mail list Please? Go to the website. Click "community" Click "mailing lists" On the left-hand side click "Subscribe" Fill in the form, changing action to "unsubscribe" HTH -- Richard Huxton Archonet Ltd ---(end o

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Kevin
Luke Lonergan wrote: Jim, On 1/30/06 12:25 PM, "Jim C. Nasby" <[EMAIL PROTECTED]> wrote: Why divide by 2? A good raid controller should be able to send read requests to both drives out of the mirrored set to fully utilize the bandwidth. Of course, that probably won't come into play unless t

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
Jim, On 1/30/06 12:25 PM, "Jim C. Nasby" <[EMAIL PROTECTED]> wrote: > Why divide by 2? A good raid controller should be able to send read > requests to both drives out of the mirrored set to fully utilize the > bandwidth. Of course, that probably won't come into play unless the OS > decides that

[PERFORM] Delete me

2006-01-31 Thread 2386429
Can you delete me from the mail list Please? ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq