Re: [PERFORM] Performance on SUSE w/ reiserfs
On Tue, 2005-10-11 at 09:41 +0200, Claus Guttesen wrote: I have a postgresql 7.4.8-server with 4 GB ram. snip #effective_cache_size = 1000# typically 8KB each This is computed by sysctl -n vfs.hibufspace / 8192 (on FreeBSD). So I changed it to: effective_cache_size = 27462# typically 8KB each Apparently this formula is no longer relevant on the FreeBSD systems as it can cache up to almost all the available RAM. With 4GB of RAM, one could specify most of the RAM as being available for caching, assuming that nothing but PostgreSQL runs on the server -- certainly 1/2 the RAM would be a reasonable value to tell the planner. (This was verified by using dd: dd if=/dev/zero of=/usr/local/pgsql/iotest bs=128k count=16384 to create a 2G file then dd if=/usr/local/pgsql/iotest of=/dev/null If you run systat -vmstat 2 you will see 0% diskaccess during the read of the 2G file indicating that it has, in fact, been cached) Sven ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PERFORM] Performance on SUSE w/ reiserfs
Realise also that unless you are running the 1.5 x86-64 build, java will not use more than 1Gig, and if the app server requests more than 1gig, Java will die (I've been there) with an out of memory error, even though there is plenty of free mem available. This can easily be cause by a lazy GC thread if the applicaiton is running high on CPU usage. The kernel will not report memory used for caching pages as being unavailable, if a program calls a malloc, the kernel will just swap out the oldest disk page and give the memory to the application. Your free -mo shows 3 gig free even with cached disk pages. It looks to me more like either a Java problem, or a kernel problem... Alex Turner NetEconomistOn 10/10/05, Jon Brisbin [EMAIL PROTECTED] wrote: Tom Lane wrote: Are you sure it's not cached data pages, rather than cached inodes? If so, the above behavior is *good*. People often have a mistaken notion that having near-zero free RAM means they have a problem.In point of fact, that is the way it is supposed to be (at least on Unix-like systems).This is just a reflection of the kernel doing what it is supposed to do, which is to use all spare RAM for caching recently accessed disk pages.If you're not swapping then you do not have a problem.Except for the fact that my Java App server crashes when all theavailable memory is being used by caching and not reclaimed :-) If it wasn't for the app server going down, I probably wouldn't care.--Jon BrisbinWebmasterNPC International, Inc.---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PERFORM] Performance on SUSE w/ reiserfs
Well - to each his own I guess - we did extensive testing on 1.4, and it refused to allocate much past 1gig on both Linux x86/x86-64 and Windows. AlexOn 10/11/05, Alan Stange [EMAIL PROTECTED] wrote: Alex Turner wrote: Perhaps this is true for 1.5 on x86-32 (I've only used it on x86-64) but I was more thinking 1.4 which many folks are still using.The 1.4.x JVM's will also work just fine with much more than 1GB of memory. Perhaps you'd like to try again?-- Alan On 10/11/05, *Alan Stange* [EMAIL PROTECTED] mailto: [EMAIL PROTECTED] wrote: Alex Turner wrote: Realise also that unless you are running the 1.5 x86-64 build, java will not use more than 1Gig, and if the app server requests more than 1gig, Java will die (I've been there) with an out of memory error, even though there is plenty of free mem available.This can easily be cause by a lazy GC thread if the applicaiton is running high on CPU usage. On my side of Planet Earth, the standard non-x64 1.5 JVM will happily use more than 1G of memory (on linux and Solaris, can't speak for Windows).If you're running larger programs, it's probably a good idea to use the -server compiler in the JVM as well.I regularly run with -Xmx1800m and regularly have 1GB heap sizes. The standard GC will not cause on OOM error if space remains for the requested object.The GC thread blocks all other threads during its activity, whatever else is happening on the machine. The newer/experimental GC's did have some potential race conditions, but I believe those have been resolved in the 1.5 JVMs. Finally, note that the latest _05 release of the 1.5 JVM also now supports large page sizes on Linux and Windows: -XX:+UseLargePages this can be quite beneficial depending on the memory patterns in your programs. -- Alan
Re: [PERFORM] Performance on SUSE w/ reiserfs
I have a SUSE 9 box that is running Postgres 8.0.1 compiled from source. Over time, I see the memory usage of the box go way way up (it's got 8GBs in it and by the end of the day, it'll be all used up) with what looks like cached inodes relating to the extreme IO generated by I was wondering if there is something I'm doing wrong with my default settings of postgres that is keeping all that stuff cached, or if I just need to switch to XFS or if there is some setting in postgres that I can tweak that will make this problem go away. It's gone beyond an annoyance and is now into the realm of getting me in trouble if I can't keep this DB server up and running. Even a minute or two of downtime in a restart is often too much. Any help you can give in this would be extrememly helpful as I'm very far from an expert on Linux filesystems and postgres tuning. You may want to submit your postgresql.conf. Upgrading to the latest stable version may also help, although my experience is related to FreeBSD and postgresql 7.4.8. regards Claus ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PERFORM] Performance on SUSE w/ reiserfs
Jon Brisbin [EMAIL PROTECTED] writes: I have a SUSE 9 box that is running Postgres 8.0.1 compiled from source. Over time, I see the memory usage of the box go way way up (it's got 8GBs in it and by the end of the day, it'll be all used up) with what looks like cached inodes relating to the extreme IO generated by postgres. We replicate about 10GBs of data every day from our AS/400 into postgres, and it is the main database for our intranet portal, which will server 40,000 pages on a good day. Are you sure it's not cached data pages, rather than cached inodes? If so, the above behavior is *good*. People often have a mistaken notion that having near-zero free RAM means they have a problem. In point of fact, that is the way it is supposed to be (at least on Unix-like systems). This is just a reflection of the kernel doing what it is supposed to do, which is to use all spare RAM for caching recently accessed disk pages. If you're not swapping then you do not have a problem. You should be looking at swap I/O rates (see vmstat or iostat) to determine if you have memory pressure, not free RAM. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PERFORM] Performance on SUSE w/ reiserfs
Tom Lane wrote: Are you sure it's not cached data pages, rather than cached inodes? If so, the above behavior is *good*. People often have a mistaken notion that having near-zero free RAM means they have a problem. In point of fact, that is the way it is supposed to be (at least on Unix-like systems). This is just a reflection of the kernel doing what it is supposed to do, which is to use all spare RAM for caching recently accessed disk pages. If you're not swapping then you do not have a problem. Except for the fact that my Java App server crashes when all the available memory is being used by caching and not reclaimed :-) If it wasn't for the app server going down, I probably wouldn't care. -- Jon Brisbin Webmaster NPC International, Inc. ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PERFORM] Performance on SUSE w/ reiserfs
Jon Brisbin wrote: Tom Lane wrote: Are you sure it's not cached data pages, rather than cached inodes? If so, the above behavior is *good*. People often have a mistaken notion that having near-zero free RAM means they have a problem. In point of fact, that is the way it is supposed to be (at least on Unix-like systems). This is just a reflection of the kernel doing what it is supposed to do, which is to use all spare RAM for caching recently accessed disk pages. If you're not swapping then you do not have a problem. Except for the fact that my Java App server crashes when all the available memory is being used by caching and not reclaimed :-) Ah, so you have a different problem. What you should be asking is why the appserver crashes. You still seem to have a lot of free swap, judging by a nearby post. But maybe the problem is that the swap is completely used too, and so the OOM killer (is this Linux?) comes around and kills the appserver. Certainly the problem is not the caching. You should be monitoring when and why the appserver dies. -- Alvaro Herrera Architect, http://www.EnterpriseDB.com On the other flipper, one wrong move and we're Fatal Exceptions (T.U.X.: Term Unit X - http://www.thelinuxreview.com/TUX/) ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PERFORM] Performance on SUSE w/ reiserfs
Jon Brisbin [EMAIL PROTECTED] writes: Tom Lane wrote: If you're not swapping then you do not have a problem. Except for the fact that my Java App server crashes when all the available memory is being used by caching and not reclaimed :-) That's a kernel bug (or possibly a Java bug ;-)). I concur with Josh's suggestion that you need a newer kernel. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
