Re: [PERFORM] extremly low memory usage

2005-08-23 Thread John A Meinel
Jeremiah Jahn wrote: > On Sun, 2005-08-21 at 16:13 -0400, Ron wrote: > >>At 10:54 AM 8/21/2005, Jeremiah Jahn wrote: >> ... >>So you have 2 controllers each with 2 external slots? But you are >>currently only using 1 controller and only one external slot on that >>controller? > > > Sorry, no. I

Re: [PERFORM] extremly low memory usage

2005-08-22 Thread William Yu
Ron wrote: PERC4eDC-PCI Express, 128MB Cache, 2-External Channels Looks like they are using the LSI Logic MegaRAID SCSI 320-2E controller. IIUC, you have 2 of these, each with 2 external channels? A lot of people have mentioned Dell's versions of the LSI cards can be WAY slower than the on

Re: [PERFORM] extremly low memory usage

2005-08-22 Thread Jeremiah Jahn
On Sun, 2005-08-21 at 16:13 -0400, Ron wrote: > At 10:54 AM 8/21/2005, Jeremiah Jahn wrote: > >On Sat, 2005-08-20 at 21:32 -0500, John A Meinel wrote: > > > Ron wrote: > > > > > > Well, since you can get a read of the RAID at 150MB/s, that means that > > > it is actual I/O speed. It may not be cach

Re: [PERFORM] extremly low memory usage

2005-08-21 Thread Ron
At 10:54 AM 8/21/2005, Jeremiah Jahn wrote: On Sat, 2005-08-20 at 21:32 -0500, John A Meinel wrote: > Ron wrote: > > Well, since you can get a read of the RAID at 150MB/s, that means that > it is actual I/O speed. It may not be cached in RAM. Perhaps you could > try the same test, only using say

Re: [PERFORM] extremly low memory usage

2005-08-21 Thread Ron
I'm resending this as it appears not to have made it to the list. At 10:54 AM 8/21/2005, Jeremiah Jahn wrote: On Sat, 2005-08-20 at 21:32 -0500, John A Meinel wrote: > Ron wrote: > > Well, since you can get a read of the RAID at 150MB/s, that means that > it is actual I/O speed. It may not be ca

Re: [PERFORM] extremly low memory usage

2005-08-21 Thread John A Meinel
Jeremiah Jahn wrote: On Sat, 2005-08-20 at 21:32 -0500, John A Meinel wrote: Ron wrote: At 02:53 PM 8/20/2005, Jeremiah Jahn wrote: Well, since you can get a read of the RAID at 150MB/s, that means that it is actual I/O speed. It may not be cached in RAM. Perhaps you could try the same te

Re: [PERFORM] extremly low memory usage

2005-08-21 Thread Jeremiah Jahn
On Sat, 2005-08-20 at 21:32 -0500, John A Meinel wrote: > Ron wrote: > > At 02:53 PM 8/20/2005, Jeremiah Jahn wrote: > > > >> On Fri, 2005-08-19 at 16:03 -0500, John A Meinel wrote: > >> > Jeremiah Jahn wrote: > >> > > On Fri, 2005-08-19 at 12:18 -0500, John A Meinel wrote: > >> > > > >> > >> > >

Re: [PERFORM] extremly low memory usage

2005-08-21 Thread Marko Ristola
I'm Sorry, that I wrote that the option would risk the LOG persistency with PostgreSQL. I should have asked instead, that how you have taken this into account. Tom Lane's email below convinces me, that you have taken the metadata only journalling into account and still fulfill the persistency of

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Ron Mayer
Ron wrote: Oops. There's a misconception here. ... OTOH, access time is _latency_, and that is not changed. Access time for a RAID set is equal to that of the slowest access time, AKA highest latency, HD in the RAID set. You're overgeneralizing from one specific type of raid, aren't you

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread John A Meinel
Ron wrote: At 02:53 PM 8/20/2005, Jeremiah Jahn wrote: On Fri, 2005-08-19 at 16:03 -0500, John A Meinel wrote: > Jeremiah Jahn wrote: > > On Fri, 2005-08-19 at 12:18 -0500, John A Meinel wrote: > > > > > it's cached alright. I'm getting a read rate of about 150MB/sec. I would > > have thought

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread John A Meinel
Jeremiah Jahn wrote: I'm just watching gnome-system-monoitor. Which after careful consideration.and looking at dstat means I'm on CRACKGSM isn't showing cached memory usageI asume that the cache memory usage is where data off of the disks would be cached...? Well a simple "free" al

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Ron
At 02:16 PM 8/20/2005, Jeremiah Jahn wrote: I'm just watching gnome-system-monoitor. Which after careful consideration.and looking at dstat means I'm on CRACKGSM isn't showing cached memory usageI asume that the cache memory usage is where data off of the disks would be cached...?

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Ron
I'm reposting this because my mailer hiccuped when I sent it the first time. If this results in a double post, I apologize. At 02:53 PM 8/20/2005, Jeremiah Jahn wrote: On Fri, 2005-08-19 at 16:03 -0500, John A Meinel wrote: > Jeremiah Jahn wrote: > > On Fri, 2005-08-19 at 12:18 -0500, John A M

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Ron
At 02:53 PM 8/20/2005, Jeremiah Jahn wrote: On Fri, 2005-08-19 at 16:03 -0500, John A Meinel wrote: > Jeremiah Jahn wrote: > > On Fri, 2005-08-19 at 12:18 -0500, John A Meinel wrote: > > > > > it's cached alright. I'm getting a read rate of about 150MB/sec. I would > > have thought is would be f

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Jeremiah Jahn
On Fri, 2005-08-19 at 16:03 -0500, John A Meinel wrote: > Jeremiah Jahn wrote: > > On Fri, 2005-08-19 at 12:18 -0500, John A Meinel wrote: > > > >>Jeremiah Jahn wrote: > >> > > > ... > > >> > >>Well, in general, 3ms for a single lookup seems really long. Maybe your > >>index is bloated by not va

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Jeremiah Jahn
On Sat, 2005-08-20 at 11:59 -0400, Ron wrote: > At 04:11 PM 8/19/2005, Jeremiah Jahn wrote: > >On Fri, 2005-08-19 at 14:23 -0500, John A Meinel wrote: > > > Ron wrote: > > > > At 01:18 PM 8/19/2005, John A Meinel wrote: > > > > > > > >> Jeremiah Jahn wrote: > > > >> > Sorry about the formatting. >

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Jeremiah Jahn
I'm just watching gnome-system-monoitor. Which after careful consideration.and looking at dstat means I'm on CRACKGSM isn't showing cached memory usageI asume that the cache memory usage is where data off of the disks would be cached...? memory output from dstat is this for a few se

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Ron
At 04:11 PM 8/19/2005, Jeremiah Jahn wrote: On Fri, 2005-08-19 at 14:23 -0500, John A Meinel wrote: > Ron wrote: > > At 01:18 PM 8/19/2005, John A Meinel wrote: > > > >> Jeremiah Jahn wrote: > >> > Sorry about the formatting. > >> > > >> > On Thu, 2005-08-18 at 12:55 -0500, John Arbash Meinel wro

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Tom Lane
Michael Stone <[EMAIL PROTECTED]> writes: > On Sat, Aug 20, 2005 at 02:17:54PM +0300, Marko Ristola wrote: >> Based on my knoledge, Ext3 is good with keeping filesystem integrity >> AND data integrity while pressing the reset button. However, by >> selecting data=writeback, you gain more speed, bu

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Michael Stone
On Sat, Aug 20, 2005 at 02:17:54PM +0300, Marko Ristola wrote: Based on my knoledge, Ext3 is good with keeping filesystem integrity AND data integrity while pressing the reset button. However, by selecting data=writeback, you gain more speed, but you risk the data integrity during a crash: Ext3

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Michael Stone
On Sat, Aug 20, 2005 at 01:12:15AM -0600, Dan Harris wrote: XFS seems to be a trusted choice, followed by Reiser and JFS both with the occasional controversy when the comparisons pop up. And don't put the xlog on a journaled filesystem. There is no advantage to doing so, and it will slow thing

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Marko Ristola
Dan Harris wrote: > From my recent experiences, I can say ext3 is probably not a great > choice for Pg databases. If you check the archives you'll see > there's a lot of discussion about various journalling filesystems and > ext3 usually(always?) comes up on the bottom as far as performance >

Re: [PERFORM] extremly low memory usage

2005-08-20 Thread Dan Harris
On Aug 19, 2005, at 3:01 PM, Jeremiah Jahn wrote: Rebuild in progress with just ext3 on the raid array...will see if this helps the access times. From my recent experiences, I can say ext3 is probably not a great choice for Pg databases. If you check the archives you'll see there's a l

Re: [PERFORM] extremly low memory usage

2005-08-19 Thread John A Meinel
Jeremiah Jahn wrote: > Rebuild in progress with just ext3 on the raid array...will see if this > helps the access times. If it doesn't I'll mess with the stripe size. I > have REINDEXED, CLUSTERED, tablespaced and cached with 'cat table/index > >>/dev/null' none of this seems to have helped, or eve

Re: [PERFORM] extremly low memory usage

2005-08-19 Thread John A Meinel
Jeremiah Jahn wrote: > On Fri, 2005-08-19 at 12:18 -0500, John A Meinel wrote: > >>Jeremiah Jahn wrote: >> ... >> >>Well, in general, 3ms for a single lookup seems really long. Maybe your >>index is bloated by not vacuuming often enough. Do you tend to get a lot >>of updates to litigant_details?

Re: [PERFORM] extremly low memory usage

2005-08-19 Thread Jeremiah Jahn
Rebuild in progress with just ext3 on the raid array...will see if this helps the access times. If it doesn't I'll mess with the stripe size. I have REINDEXED, CLUSTERED, tablespaced and cached with 'cat table/index > /dev/null' none of this seems to have helped, or even increased my memory usage.

Re: [PERFORM] extremly low memory usage

2005-08-19 Thread Jeremiah Jahn
On Fri, 2005-08-19 at 14:23 -0500, John A Meinel wrote: > Ron wrote: > > At 01:18 PM 8/19/2005, John A Meinel wrote: > > > >> Jeremiah Jahn wrote: > >> > Sorry about the formatting. > >> > > >> > On Thu, 2005-08-18 at 12:55 -0500, John Arbash Meinel wrote: > >> > > >> >>Jeremiah Jahn wrote: > >> >>

Re: [PERFORM] extremly low memory usage

2005-08-19 Thread Jeremiah Jahn
On Fri, 2005-08-19 at 12:18 -0500, John A Meinel wrote: > Jeremiah Jahn wrote: > > Sorry about the formatting. > > > > On Thu, 2005-08-18 at 12:55 -0500, John Arbash Meinel wrote: > > > >>Jeremiah Jahn wrote: > >> > >> > > ... > > >>The expensive parts are the 4915 lookups into the litigant_detai

Re: [PERFORM] extremly low memory usage

2005-08-19 Thread John A Meinel
Ron wrote: > At 01:18 PM 8/19/2005, John A Meinel wrote: > >> Jeremiah Jahn wrote: >> > Sorry about the formatting. >> > >> > On Thu, 2005-08-18 at 12:55 -0500, John Arbash Meinel wrote: >> > >> >>Jeremiah Jahn wrote: >> >> >> >> >> >> ... >> >> >>The expensive parts are the 4915 lookups into the l

Re: [PERFORM] extremly low memory usage

2005-08-19 Thread Ron
At 01:18 PM 8/19/2005, John A Meinel wrote: Jeremiah Jahn wrote: > Sorry about the formatting. > > On Thu, 2005-08-18 at 12:55 -0500, John Arbash Meinel wrote: > >>Jeremiah Jahn wrote: >> >> ... >>The expensive parts are the 4915 lookups into the litigant_details (each >>one takes approx 4ms fo

Re: [PERFORM] extremly low memory usage

2005-08-19 Thread John A Meinel
Jeremiah Jahn wrote: > Sorry about the formatting. > > On Thu, 2005-08-18 at 12:55 -0500, John Arbash Meinel wrote: > >>Jeremiah Jahn wrote: >> >> ... >>The expensive parts are the 4915 lookups into the litigant_details (each >>one takes approx 4ms for a total of ~20s). >>And then you do it again

Re: [PERFORM] extremly low memory usage

2005-08-19 Thread Jeremiah Jahn
Sorry about the formatting. On Thu, 2005-08-18 at 12:55 -0500, John Arbash Meinel wrote: > Jeremiah Jahn wrote: > > >here's an example standard query. Ireally have to make the first hit go > >faster. The table is clustered as well on full_name as well. 'Smith%' > >took 87 seconds on the first hi

Re: [PERFORM] extremly low memory usage

2005-08-18 Thread Ron
At 01:55 PM 8/18/2005, John Arbash Meinel wrote: Jeremiah Jahn wrote: >here's an example standard query. Ireally have to make the first hit go >faster. The table is clustered as well on full_name as well. 'Smith%' >took 87 seconds on the first hit. I wonder if I set up may array wrong. >I remebe

Re: [PERFORM] extremly low memory usage

2005-08-18 Thread John Arbash Meinel
Jeremiah Jahn wrote: >here's an example standard query. Ireally have to make the first hit go >faster. The table is clustered as well on full_name as well. 'Smith%' >took 87 seconds on the first hit. I wonder if I set up may array wrong. >I remeber see something about DMA access versus something e

Re: [PERFORM] extremly low memory usage

2005-08-18 Thread Jeremiah Jahn
here's an example standard query. Ireally have to make the first hit go faster. The table is clustered as well on full_name as well. 'Smith%' took 87 seconds on the first hit. I wonder if I set up may array wrong. I remeber see something about DMA access versus something else, and choose DMA access

Re: [PERFORM] extremly low memory usage

2005-08-18 Thread John Arbash Meinel
Jeremiah Jahn wrote: >On Wed, 2005-08-17 at 21:21 -0500, John A Meinel wrote: > > >>Jeremiah Jahn wrote: >> >> >>>I just put together a system with 6GB of ram on a 14 disk raid 10 array. >>>When I run my usual big painful queries, I get very little to know >>>memory usage. My production box

Re: [PERFORM] extremly low memory usage

2005-08-18 Thread Jeremiah Jahn
On Wed, 2005-08-17 at 21:21 -0500, John A Meinel wrote: > Jeremiah Jahn wrote: > > I just put together a system with 6GB of ram on a 14 disk raid 10 array. > > When I run my usual big painful queries, I get very little to know > > memory usage. My production box (raid 5 4GB ram) hovers at 3.9GB use

Re: [PERFORM] extremly low memory usage

2005-08-18 Thread Jeff Trout
On Aug 17, 2005, at 10:11 PM, Jeremiah Jahn wrote: I just put together a system with 6GB of ram on a 14 disk raid 10 array. When I run my usual big painful queries, I get very little to know memory usage. My production box (raid 5 4GB ram) hovers at 3.9GB used most of the time. the new devel

Re: [PERFORM] extremly low memory usage

2005-08-17 Thread John A Meinel
Jeremiah Jahn wrote: > I just put together a system with 6GB of ram on a 14 disk raid 10 array. > When I run my usual big painful queries, I get very little to know > memory usage. My production box (raid 5 4GB ram) hovers at 3.9GB used > most of the time. the new devel box sits at around 250MB. >

[PERFORM] extremly low memory usage

2005-08-17 Thread Jeremiah Jahn
I just put together a system with 6GB of ram on a 14 disk raid 10 array. When I run my usual big painful queries, I get very little to know memory usage. My production box (raid 5 4GB ram) hovers at 3.9GB used most of the time. the new devel box sits at around 250MB. I've switched to an 8.0 syste