Re: [PERFORM] Need for speed 2
I have found that while the OS may flush to the controller fast with fsync=true, the controller does as it pleases (it has BBU, so I'm not too worried), so you get great performance because your controller is determine read/write sequence outside of what is being demanded by an fsync. Alex Turner NetEconomistOn 8/25/05, Kelly Burkhart <[EMAIL PROTECTED]> wrote: On Thu, 2005-08-25 at 11:16 -0400, Ron wrote:> ># - Settings -> >> >fsync = false # turns forced synchronization on or off> >#wal_sync_method = fsync# the default varies across platforms:> > # fsync, fdatasync, open_sync, or>> I hope you have a battery backed write buffer!Battery backed write buffer will do nothing here, because the OS istaking it's sweet time flushing to the controller's battery backed write buffer!Isn't the reason for batter backed controller cache to make fsync()sfast?-K---(end of broadcast)---TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PERFORM] Need for speed 2
On Thu, 2005-08-25 at 11:16 -0400, Ron wrote: > ># - Settings - > > > >fsync = false # turns forced synchronization on or off > >#wal_sync_method = fsync# the default varies across platforms: > > # fsync, fdatasync, open_sync, or > > I hope you have a battery backed write buffer! Battery backed write buffer will do nothing here, because the OS is taking it's sweet time flushing to the controller's battery backed write buffer! Isn't the reason for batter backed controller cache to make fsync()s fast? -K ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PERFORM] Need for speed 2
> Putting pg_xlog on the IDE drives gave about 10% performance > improvement. Would faster disks give more performance? > > What my application does: > > Every five minutes a new logfile will be imported. Depending on the > source of the request it will be imported in one of three "raw click" > tables. (data from two months back, to be able to verify customer > complains) > For reporting I have a set of tables. These contain data from the last > two years. My app deletes all entries from today and reinserts updated > data calculated from the raw data tables. > > The queries contain no joins only aggregates. I have several indexes to > speed different kinds of queries. > > My problems occur when one users does a report that contains to much old > data. In that case all cache mechanisms will fail and disc io is the > limiting factor. It seems like you are pushing limit of what server can handle. This means: 1. expensive server upgrade. or 2. make software more efficient. Since you sound I/O bound, you can tackle 1. by a. adding more memory or b. increasing i/o throughput. Unfortunately, you already have a pretty decent server (for x86) so 1. means 64 bit platform and 2. means more expensive hard drives. The archives is full of information about this... Is your data well normalized? You can do tricks like: if table has fields a,b,c,d,e,f with a is primary key, and d,e,f not frequently queried or missing, move d,e,f to seprate table. well normalized structures are always more cache efficient. Do you have lots of repeating and/or empty data values in your tables? Make your indexes and data as small as possible to reduce pressure on the cache, here are just a few tricks: 1. use int2/int4 instead of numeric 2. know when to use char and varchar 3. use functional indexes to reduce index expression complexity. This can give extreme benefits if you can, for example, reduce double field index to Boolean. Merlin ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PERFORM] Need for speed 2
At 03:10 AM 8/25/2005, Ulrich Wisser wrote: I realize I need to be much more specific. Here is a more detailed description of my hardware and system design. Pentium 4 2.4GHz Memory 4x DIMM DDR 1GB PC3200 400MHZ CAS3, KVR Motherboard chipset 'I865G', two IDE channels on board First suggestion: Get better server HW. AMD Opteron based dual processor board is the current best in terms of price/performance ratio, _particularly_ for DB applications like the one you have described. Such mainboards cost ~$400-$500. RAM will cost about $75-$150/GB. Opteron 2xx are ~$200-$700 apiece. So a 2P AMD system can be had for as little as ~$850 + the cost of the RAM you need. In the worst case where you need 24GB of RAM (~$3600), the total comes in at ~$4450. As you can see from the numbers, buying only what RAM you actually need can save you a great deal on money. Given what little you said about how much of your DB is frequently accessed, I'd suggest buying a server based around the 2P 16 DIMM slot IWill DK88 mainboard (Tyan has announced a 16 DIMM slot mainboard, but I do not think it is actually being sold yet.). Then fill it with the minimum amount of RAM that will allow the "working set" of the DB to be cached in RAM. In the worst case where DB access is essentially uniform and essentially random, you will need 24GB of RAM to hold the 22GB DB + OS + etc. That worst case is _rare_. Usually DB's have a working set that is smaller than the entire DB. You want to keep that working set in RAM. If you can't identify the working set, buy enough RAM to hold the entire DB. In particular, you want to make sure that any frequently accessed read only tables or indexes are kept in RAM. The "read only" part is very important. Tables (and their indexes) that are frequently written to _have_ to access HD. Therefore you get much less out of having them in RAM. Read only tables and their indexes can be loaded into tmpfs at boot time thereby keeping out of the way of the file system buffer cache. tmpfs does not save data if the host goes down so it is very important that you ONLY use this trick with read only tables. The other half of the trick is to make sure that the file system buffer cache does _not_ cache whatever you have loaded into tmpfs. 2x SEAGATE BARRACUDA 7200.7 80GB 7200RPM ATA/100 (software raid 1, system, swap, pg_xlog) ADAPTEC SCSI RAID 2100S ULTRA160 32MB 1-CHANNEL 2x SEAGATE CHEETAH 15K.3 73GB ULTRA320 68-PIN WIDE (raid 1, /var/lib/pgsql) Second suggestion: you need a MUCH better IO subsystem. In fact, given that you have described this system as being primarily OLTP like, this is more important that the above server HW. Best would be to upgrade everything, but if you are strapped for cash, upgrade the IO subsystem first. You need many more spindles and a decent RAID card or cards. You want 15Krpm (best) or 10Krpm HDs. As long as all of the HD's are at least 10Krpm, more spindles is more important than faster spindles. If it's a choice between more 10Krpm discs or fewer 15Krpm discs, buy the 10Krpm discs. Get the spindle count as high as you RAID cards can handle. Whatever RAID cards you get should have as much battery backed write buffer as possible. In the commodity market, presently the highest performance RAID cards I know of, and the ones that support the largest battery backed write buffer, are made by Areca. Database size on disc is 22GB. (without pg_xlog) Find out what the working set, ie the most frequently accessed portion, of this 22GB is and you will know how much RAM is worth having. 4GB is definitely too little! Please find my postgresql.conf below. Third suggestion: make sure you are running a 2.6 based kernel and at least PG 8.0.3. Helping beta test PG 8.1 might be an option for you as well. Putting pg_xlog on the IDE drives gave about 10% performance improvement. Would faster disks give more performance? What my application does: Every five minutes a new logfile will be imported. Depending on the source of the request it will be imported in one of three "raw click" tables. (data from two months back, to be able to verify customer complains) For reporting I have a set of tables. These contain data from the last two years. My app deletes all entries from today and reinserts updated data calculated from the raw data tables. The raw data tables seem to be read only? If so, you should buy enough RAM to load them into tmpfs at boot time and have them be completely RAM resident in addition to having enough RAM for the OS to cache an appropriate amount of the rest of the DB. The queries contain no joins only aggregates. I have several indexes to speed different kinds of queries. My problems occur when one users does a report that contains too much old data. In that case all cache mechanisms will fail and disc io is the limiting factor. If one query contains so much data, that a f
Re: [PERFORM] Need for speed 2
On Thu, 25 Aug 2005 09:10:37 +0200 Ulrich Wisser <[EMAIL PROTECTED]> wrote: > Pentium 4 2.4GHz > Memory 4x DIMM DDR 1GB PC3200 400MHZ CAS3, KVR > Motherboard chipset 'I865G', two IDE channels on board > 2x SEAGATE BARRACUDA 7200.7 80GB 7200RPM ATA/100 > (software raid 1, system, swap, pg_xlog) > ADAPTEC SCSI RAID 2100S ULTRA160 32MB 1-CHANNEL > 2x SEAGATE CHEETAH 15K.3 73GB ULTRA320 68-PIN WIDE > (raid 1, /var/lib/pgsql) > > Database size on disc is 22GB. (without pg_xlog) > > Please find my postgresql.conf below. > > Putting pg_xlog on the IDE drives gave about 10% performance > improvement. Would faster disks give more performance? Faster as in RPM on your pg_xlog partition probably won't make much of a difference. However, if you can get a drive with better overall write performance then it would be a benefit. Another thing to consider on this setup is whether or not you're hitting swap often and/or logging to that same IDE RAID set. For optimal insertion benefit you want the heads of your disks to essentially be only used for pg_xlog. If you're having to jump around the disk in the following manner: write to pg_xlog read from swap write syslog data write to pg_xlog ... ... You probably aren't getting anywhere near the benefit you could. One thing you could easily try is to break your IDE RAID set and put OS/swap on one disk and pg_xlog on the other. > If one query contains so much data, that a full table scan is needed, > I do not care if it takes two minutes to answer. But all other > queries with less data (at the same time) still have to be fast. > > I can not stop users doing that kind of reporting. :( > > I need more speed in orders of magnitude. Will more disks / more > memory do that trick? More disk and more memory always helps out. Since you say these queries are mostly on not-often-used data I would lean toward more disks in your SCSI RAID-1 setup than maxing out available RAM based on the size of your database. - Frank Wiles <[EMAIL PROTECTED]> http://www.wiles.org - ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
