Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
Jim, On 1/30/06 12:25 PM, Jim C. Nasby [EMAIL PROTECTED] wrote: Why divide by 2? A good raid controller should be able to send read requests to both drives out of the mirrored set to fully utilize the bandwidth. Of course, that probably won't come into play unless the OS decides that it's

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Kevin
Luke Lonergan wrote: Jim, On 1/30/06 12:25 PM, Jim C. Nasby [EMAIL PROTECTED] wrote: Why divide by 2? A good raid controller should be able to send read requests to both drives out of the mirrored set to fully utilize the bandwidth. Of course, that probably won't come into play unless the

Re: [PERFORM] Delete me

2006-01-31 Thread Richard Huxton
[EMAIL PROTECTED] wrote: Can you delete me from the mail list Please? Go to the website. Click community Click mailing lists On the left-hand side click Subscribe Fill in the form, changing action to unsubscribe HTH -- Richard Huxton Archonet Ltd ---(end of

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Jim C. Nasby
On Tue, Jan 31, 2006 at 09:00:30AM -0800, Luke Lonergan wrote: Jim, On 1/30/06 12:25 PM, Jim C. Nasby [EMAIL PROTECTED] wrote: Why divide by 2? A good raid controller should be able to send read requests to both drives out of the mirrored set to fully utilize the bandwidth. Of course,

Re: [PERFORM] Delete me

2006-01-31 Thread Jim C. Nasby
On Tue, Jan 31, 2006 at 06:33:14PM +, Richard Huxton wrote: [EMAIL PROTECTED] wrote: Can you delete me from the mail list Please? Go to the website. Click community Click mailing lists On the left-hand side click Subscribe Fill in the form, changing action to unsubscribe Or take a

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
Jeffrey, On 1/31/06 12:03 PM, Jeffrey W. Baker [EMAIL PROTECTED] wrote: Then you've not seen Linux. :-D Linux does balanced reads on software mirrors. I'm not sure why you think this can't improve bandwidth. It does improve streaming bandwidth as long as the platter STR is more than the

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread PFC
Linux does balanced reads on software mirrors. I'm not sure why you think this can't improve bandwidth. It does improve streaming bandwidth as long as the platter STR is more than the bus STR. ... Prove it. (I have a software RAID1 on this desktop machine) It's a lot faster

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Jim C. Nasby
On Tue, Jan 31, 2006 at 02:52:57PM -0800, Luke Lonergan wrote: It's because your alternating reads are skipping in chunks across the platter. Disks work at their max internal rate when reading sequential data, and the cache is often built to buffer a track-at-a-time, so alternating pieces

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
PFC, On 1/31/06 3:11 PM, PFC [EMAIL PROTECTED] wrote: ... Prove it. (I have a software RAID1 on this desktop machine) It's a lot faster than a single disk for random reads when more than 1 thread hits the disk, because it distributes reads to both disks. Thus, applications start

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
Jim, On 1/31/06 3:12 PM, Jim C. Nasby [EMAIL PROTECTED] wrote: The alternating technique in mirroring might improve rotational latency for random seeking - a trick that Tandem exploited, but it won't improve bandwidth. Or just work in multiples of tracks, which would greatly reduce the

[PERFORM] partitioning and locking problems

2006-01-31 Thread Marc Morin
We have a large database system designed around partitioning. Our application is characterized with - terabytes of data - billions of rows in dozens of base tables (and 100s of paritions) - 24x7 insert load of new data that cannot be stopped, data is time sensitive. - periodic reports that can

[PERFORM] Storing Digital Video

2006-01-31 Thread Rodrigo Madera
I am concerned with performance issues involving the storage of DV on a database. I though of some options, which would be the most advised for speed? 1) Pack N frames inside a container and store the container to the db. 2) Store each frame in a separate record in the table frames. 3) (type

Re: [PERFORM] Storing Digital Video

2006-01-31 Thread Matt Davies | Postgresql List
Rodrigo Madera wrote: I am concerned with performance issues involving the storage of DV on a database. I though of some options, which would be the most advised for speed? 1) Pack N frames inside a container and store the container to the db. 2) Store each frame in a separate record in the

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Steinar H. Gunderson
On Tue, Jan 31, 2006 at 12:47:10PM -0800, Luke Lonergan wrote: Linux does balanced reads on software mirrors. I'm not sure why you think this can't improve bandwidth. It does improve streaming bandwidth as long as the platter STR is more than the bus STR. ... Prove it. FWIW, this is on

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Jim C. Nasby
On Tue, Jan 31, 2006 at 03:19:38PM -0800, Luke Lonergan wrote: Well, the only problem with that is if the machine crashes for any reason you risk having the database corrupted (or at best losing some committed transactions). So, do you routinely turn off Linux write caching? If not, then

[PERFORM] Sequential scan being used despite indexes

2006-01-31 Thread James Russell
Hi there, I'm running a simple query with 2 inner joins (say A, B and C). Each of the join columns has indexes. If I run queries that join just A and B, or just B and C, postgres uses indexes. But if I run A join B join C then the B join C part starts using a sequential scan and I can't figure

Re: [PERFORM] Sequential scan being used despite indexes

2006-01-31 Thread Joshua D. Drake
Explaining: Hash Join (cost=337.93..1267.54 rows=180 width=35) Hash Cond: (outer.message_id = inner.message_id) - Seq Scan on message_meta_data (cost=0.00..739.19 rows=37719 width=30) - Hash (cost=337.79..337.79 rows=57 width=13) - Nested Loop (cost=0.00..337.79 rows=57 width=13)

Re: [PERFORM] Sequential scan being used despite indexes

2006-01-31 Thread Michael Fuhr
On Tue, Jan 31, 2006 at 07:29:51PM -0800, Joshua D. Drake wrote: Any ideas? What does explain analyze say? Also, have the tables been vacuumed and analyzed? -- Michael Fuhr ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster

Re: [PERFORM] Sequential scan being used despite indexes

2006-01-31 Thread James Russell
[Sorry, my last reply didn't go to the list] Reading about this issue further in the FAQ, it seems that I should ensure that Postgres has adequate and accurate information about the tables in question by regularly running VACUUM ANALYZE, something I don't do currently. I disabled SeqScan as

Re: [PERFORM] Sequential scan being used despite indexes

2006-01-31 Thread Michael Fuhr
On Wed, Feb 01, 2006 at 01:33:08PM +0900, James Russell wrote: Reading about this issue further in the FAQ, it seems that I should ensure that Postgres has adequate and accurate information about the tables in question by regularly running VACUUM ANALYZE, something I don't do currently. Many

Re: [PERFORM] partitioning and locking problems

2006-01-31 Thread Tom Lane
Marc Morin [EMAIL PROTECTED] writes: Would like to understand the implications of changing postgres' code/locking for rule changes and truncate to not require locking out select statements? It won't work... regards, tom lane ---(end of

Re: [PERFORM] Huge Data sets, simple queries

2006-01-31 Thread Luke Lonergan
Jeffrey, On 1/31/06 8:09 PM, Jeffrey W. Baker [EMAIL PROTECTED] wrote: ... Prove it. I think I've proved my point. Software RAID1 read balancing provides 0%, 300%, 100%, and 100% speedup on 1, 2, 4, and 8 threads, respectively. In the presence of random I/O, the results are even better.