Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
On Mon, 17 Jan 2005 18:36:35 -0500 Dave Cramer [EMAIL PROTECTED] wrote: The *only* way to avoid this is to go to a 64 bit processor (opteron) and then for greater performance use a linux distribution compiled for a 64bit processor. Or NetBSD (http://www.NetBSD.org/) which has been 64 bit clean since 1995 and has had the Opteron port integrated in its main tree (not as patches to or a separate tree) since April 2003. -- D'Arcy J.M. Cain [EMAIL PROTECTED] http://www.NetBSD.org/ ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
[EMAIL PROTECTED] wrote: I would like to upgrade both OS kernel and PGsql version , so in my opinion the best way to handle it is to *backup* the data in .tar Just remember if you're going from 7.3.2 = 7.4.x or 8.0 then you'll need to use pg_dump not just tar up the directories. If you do use tar, remember to backup *all* the directories. -- Richard Huxton Archonet Ltd ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
You can *not* go from any major release to another major release using any kind of file backup. Please use pg_dump. Additionally there are known issues dumping and restoring from 7.3 - 7.4 if you use the default copy command. Use the pg_dump --inserts option. I would still tar the directory just in case you *have* to fall back to 7.3 for some reason (Better safe than sorry ) Dave Richard Huxton wrote: [EMAIL PROTECTED] wrote: I would like to upgrade both OS kernel and PGsql version , so in my opinion the best way to handle it is to *backup* the data in .tar Just remember if you're going from 7.3.2 = 7.4.x or 8.0 then you'll need to use pg_dump not just tar up the directories. If you do use tar, remember to backup *all* the directories. -- Richard Huxton Archonet Ltd ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match -- Dave Cramer http://www.postgresintl.com 519 939 0336 ICQ#14675561 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
Why dont you just grab the latest stable kernel from kernel.org, customize it, compile it and the see what happens? On Tue, 18 Jan 2005 09:35:12 +0700, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: I understand that the 2.6.* kernels are much better at large memory support (with respect to performance issues), so unless you have a 64-bit machine lying around - this is probably worth a try. You may need to build a new kernel with the various parameters : CONFIG_NOHIGHMEM CONFIG_HIGHMEM4G CONFIG_HIGHMEM64G set appropriately (or even upgrade to the latest 2.6.10). I would expect that some research and experimentation will be required to get the best out of it - (e.g. the 'bounce buffers' issue). In the standard rpm FC 2-3 with a newly install server , would it automatically detect and config it if I use the mechine with 4 Gb [6Gb.] or should I manually config it? Amrit Thailand ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
This must be a linux'ism because to my knowledge FreeBSD does not keep the os-cache mapped into the kernel address space unless it have active objects associated with the data. And FreeBSD also have a default split of 3GB userspace and 1GB. kernelspace when running with a default configuration. Linux people might want to try other os'es to compare the performance. Best regards, Nicolai Petri Ps. Sorry for my lame MS mailer - quoting is not something it knows how to do. :) - Original Message - From: William Yu [EMAIL PROTECTED] I inferred this from reading up on the compressed vm project. It can be higher or lower depending on what devices you have in your system -- however, I've read messages from kernel hackers saying Linux is very aggressive in reserving memory space for devices because it must be allocated at boottime. Josh Berkus wrote: William, The theshold for using PAE is actually far lower than 4GB. 4GB is the total memory address space -- split that in half for 2GB for userspace, 2GB for kernel. The OS cache resides in kernel space -- after you take alway the memory allocation for devices, you're left with a window of roughly 900MB. I'm curious, how do you get 1.1GB for memory allocation for devices? ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
I inferred this from reading up on the compressed vm project. It can be higher or lower depending on what devices you have in your system -- however, I've read messages from kernel hackers saying Linux is very aggressive in reserving memory space for devices because it must be allocated at boottime. Josh Berkus wrote: William, The theshold for using PAE is actually far lower than 4GB. 4GB is the total memory address space -- split that in half for 2GB for userspace, 2GB for kernel. The OS cache resides in kernel space -- after you take alway the memory allocation for devices, you're left with a window of roughly 900MB. I'm curious, how do you get 1.1GB for memory allocation for devices? ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
[EMAIL PROTECTED] wrote: Since the optimal state is to allocate a small amount of memory to Postgres and leave a huge chunk to the OS cache, this means you are already hitting the PAE penalty at 1.5GB of memory. How could I chang this hitting? Upgrade to 64-bit processors + 64-bit linux. ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
Amrit, It's not useless, it's just not optimal. All operating systems, FC2, FC3, will have the same problem with greater than 4G of memory on a 32 bit processor. The *only* way to avoid this is to go to a 64 bit processor (opteron) and then for greater performance use a linux distribution compiled for a 64bit processor. Have you identified and optimized the queries, are you sure you need more memory? Dave [EMAIL PROTECTED] wrote: Since the optimal state is to allocate a small amount of memory to Postgres and leave a huge chunk to the OS cache, this means you are already hitting the PAE penalty at 1.5GB of memory. How could I change this hitting? Upgrade to 64-bit processors + 64-bit linux. Does the PAE help linux in handling the memory of more than 4 Gb limit in 32 bit archetech ? My intel server board could handle the mem of 12 Gb [according to intel spec.] and if I use Fedora C2 with PAE , will it useless for mem of more than 4Gb.? Any comment please? Amrit Thailand ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED]) -- Dave Cramer http://www.postgresintl.com 519 939 0336 ICQ#14675561
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
[EMAIL PROTECTED] wrote: Does the PAE help linux in handling the memory of more than 4 Gb limit in 32 bit archetech ? My intel server board could handle the mem of 12 Gb [according to intel spec.] and if I use Fedora C2 with PAE , will it useless for mem of more than 4Gb.? Any comment please? I understand that the 2.6.* kernels are much better at large memory support (with respect to performance issues), so unless you have a 64-bit machine lying around - this is probably worth a try. You may need to build a new kernel with the various parameters : CONFIG_NOHIGHMEM CONFIG_HIGHMEM4G CONFIG_HIGHMEM64G set appropriately (or even upgrade to the latest 2.6.10). I would expect that some research and experimentation will be required to get the best out of it - (e.g. the 'bounce buffers' issue). regards Mark ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
[EMAIL PROTECTED] wrote: In the standard rpm FC 2-3 with a newly install server , would it automatically detect and config it if I use the mechine with 4 Gb [6Gb.] or should I manually config it? Amrit Thailand Good question. I dont have FC2-3 here to check. I recommend firing off a question to [EMAIL PROTECTED] (you need to subscribe first): http://www.redhat.com/mailman/listinfo/fedora-list best wishes Mark ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
My experience is RH9 auto detected machines = 2GB of RAM and installs the PAE bigmem kernel by default. I'm pretty sure the FC2/3 installer will do the same. [EMAIL PROTECTED] wrote: I understand that the 2.6.* kernels are much better at large memory support (with respect to performance issues), so unless you have a 64-bit machine lying around - this is probably worth a try. You may need to build a new kernel with the various parameters : CONFIG_NOHIGHMEM CONFIG_HIGHMEM4G CONFIG_HIGHMEM64G set appropriately (or even upgrade to the latest 2.6.10). I would expect that some research and experimentation will be required to get the best out of it - (e.g. the 'bounce buffers' issue). In the standard rpm FC 2-3 with a newly install server , would it automatically detect and config it if I use the mechine with 4 Gb [6Gb.] or should I manually config it? Amrit Thailand ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
The theshold for using PAE is actually far lower than 4GB. 4GB is the total memory address space -- split that in half for 2GB for userspace, 2GB for kernel. The OS cache resides in kernel space -- after you take alway the memory allocation for devices, you're left with a window of roughly 900MB. I set shammax = [EMAIL PROTECTED] /]# cat /proc/sys/kernel/shmmax 40 shmall = [EMAIL PROTECTED] /]# cat /proc/sys/kernel/shmall 134217728 Is that ok for 4 Gb. mechine? Since the optimal state is to allocate a small amount of memory to Postgres and leave a huge chunk to the OS cache, this means you are already hitting the PAE penalty at 1.5GB of memory. How could I chang this hitting? Thanks Amrit Thailand ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
William, The theshold for using PAE is actually far lower than 4GB. 4GB is the total memory address space -- split that in half for 2GB for userspace, 2GB for kernel. The OS cache resides in kernel space -- after you take alway the memory allocation for devices, you're left with a window of roughly 900MB. I'm curious, how do you get 1.1GB for memory allocation for devices? -- Josh Berkus Aglio Database Solutions San Francisco ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
Gavin Sherry wrote: There is no problem with free Linux distros handling 4 GB of memory. The problem is that 32 hardware must make use of some less than efficient mechanisms to be able to address the memory. The theshold for using PAE is actually far lower than 4GB. 4GB is the total memory address space -- split that in half for 2GB for userspace, 2GB for kernel. The OS cache resides in kernel space -- after you take alway the memory allocation for devices, you're left with a window of roughly 900MB. Since the optimal state is to allocate a small amount of memory to Postgres and leave a huge chunk to the OS cache, this means you are already hitting the PAE penalty at 1.5GB of memory. ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
On Wed, 12 Jan 2005 [EMAIL PROTECTED] wrote: I wonder if I would like to increase more RAM from 4 Gb. to 6 Gb. [which I hope to increase more performance ] and I now I used RH 9 and Pgsql 7.3.2 ON DUAL Xeon 3.0 server thay has the limtation of 4 Gb. ram, I should use which OS between FC 2-3 or redhat EL 3 [which was claimed to support 64 Gb.ram] .May I use FC 2 [which is freely downloaded] with 6 Gb. and PGsql 7.4 ? Amrit Thailand Try 7.4 before the memory upgrade. If you still have performance issues, try optimising your queries. As I mentioned before, you can join the #postgresql channel on irc.freenode.net and we can assist. Gavin ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
Is that 4GB limit a hardware limitation? If it is, then there is not much you can do except upgrading the server. If the server is capable of handling more than 4GB of ram then you can just upgrade the kernel and enable high memory support (up to 64GB of memory in kernel 2.6.9). There is no need to migrate your distro, but if you do I recommend upgrading your Pgsql too. Martin On Wed, 12 Jan 2005 10:49:28 +0700, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: I wonder if I would like to increase more RAM from 4 Gb. to 6 Gb. [which I hope to increase more performance ] and I now I used RH 9 and Pgsql 7.3.2 ON DUAL Xeon 3.0 server thay has the limtation of 4 Gb. ram, I should use which OS between FC 2-3 or redhat EL 3 [which was claimed to support 64 Gb.ram] .May I use FC 2 [which is freely downloaded] with 6 Gb. and PGsql 7.4 ? Amrit Thailand ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
On Wed, 12 Jan 2005 [EMAIL PROTECTED] wrote: I wonder if I would like to increase more RAM from 4 Gb. to 6 Gb. [which I hope to increase more performance ] and I now I used RH 9 and Pgsql 7.3.2 ON DUAL Xeon 3.0 server thay has the limtation of 4 Gb. ram, I should use which OS between FC 2-3 or redhat EL 3 [which was claimed to support 64 Gb.ram] .May I use FC 2 [which is freely downloaded] with 6 Gb. and PGsql 7.4 ? There is no problem with free Linux distros handling 4 GB of memory. The problem is that 32 hardware must make use of some less than efficient mechanisms to be able to address the memory. So, try 7.4 before the memory upgrade. If you still have performance issues, try optimising your queries. As I mentioned before, you can join the #postgresql channel on irc.freenode.net and we can assist. Gavin Amrit Thailand Gavin ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
There is no problem with free Linux distros handling 4 GB of memory. The problem is that 32 hardware must make use of some less than efficient mechanisms to be able to address the memory. So, try 7.4 before the memory upgrade. If you still have performance issues, try optimising your queries. As I mentioned before, you can join the #postgresql channel on irc.freenode.net and we can assist. Yes , of course I must try to upgrade PGsql to 7.4 and may be OS to FC 2-3 too. My server products are intel based [mainboard , CPU ,Case , Power supply ,RAID Network card] dual Xeon 32 bit 3.0 Ghz which I consulted Thai intel supervisor and they told me that increasing the ram for more than 4 Gb. may be possible depending on the OS. I ask the programmer who wrote that huge query and they told me that it was the query generated by Delphi 6.0 component and not written by themselve. Amrit Thailand ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [PERFORM] Increasing RAM for more than 4 Gb. using postgresql
Yes , of course I must try to upgrade PGsql to 7.4 and may be OS to FC 2-3 too. My server products are intel based [mainboard , CPU ,Case , Power supply ,RAID Network card] dual Xeon 32 bit 3.0 Ghz which I consulted Thai intel supervisor and they told me that increasing the ram for more than 4 Gb. may be possible depending on the OS. I never tried FC before, but I recommend using Debian (with custom kernel) or if you have the patience: Gentoo. Those are strictly business distros, no unnecesary stuffs running after installation. Providing a good base for you to focus on performance tweaks. Others may have different opinions, though... ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
