Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-03-04 Thread Christian Kruse
Hi, On 03/03/14 21:03, Heikki Linnakangas wrote: I spotted this in section 17.4.1 Shared Memory and Semaphores: Linux The default maximum segment size is 32 MB, and the default maximum total size is 2097152 pages. A page is almost always 4096 bytes except in unusual kernel

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-03-03 Thread Christian Kruse
Hi, Attached is a patch with the updated documentation (now uses consistently huge pages) as well as a renamed GUC, consistent wording (always use huge pages) as well as renamed variables. Hmm, I wonder if that could now be misunderstood to have something to do with the PostgreSQL page

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-03-03 Thread Christian Kruse
Hi, On 28/02/14 17:58, Peter Geoghegan wrote: On Fri, Feb 28, 2014 at 9:43 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Hmm, I wonder if that could now be misunderstood to have something to do with the PostgreSQL page size? Maybe add the word memory or operating system in the

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-03-03 Thread Heikki Linnakangas
On 03/03/2014 11:34 AM, Christian Kruse wrote: Hi, Attached is a patch with the updated documentation (now uses consistently huge pages) as well as a renamed GUC, consistent wording (always use huge pages) as well as renamed variables. Hmm, I wonder if that could now be misunderstood to have

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-28 Thread Heikki Linnakangas
On 02/27/2014 09:34 AM, Christian Kruse wrote: Hi, On 26/02/14 13:13, Alvaro Herrera wrote: There's one thing that rubs me the wrong way about all this functionality, which is that we've named it huge TLB pages. That is wrong -- the TLB pages are not huge. In fact, as far as I understand,

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-28 Thread Peter Geoghegan
On Fri, Feb 28, 2014 at 9:43 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Hmm, I wonder if that could now be misunderstood to have something to do with the PostgreSQL page size? Maybe add the word memory or operating system in the first sentence in the docs, like this: Enables/disables

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-27 Thread Christian Kruse
Hi, On 27/02/14 08:35, Christian Kruse wrote: Hi Peter, Sorry, Stephen of course – it was definitely to early. Best regards, -- Christian Kruse http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services pgpm6lYpan4Df.pgp Description: PGP signature

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-26 Thread Christian Kruse
Hi, On 25/02/14 19:28, Andres Freund wrote: I think all that's needed is to cut the first paragraphs that generally explain what huge pages are in some detail from the text and make sure the later paragraphs don't refer to the earlier ones. Attached you will find a new version of the patch.

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-26 Thread Christian Kruse
Hi Peter, after a night of sleep I'm still not able to swallow the pill. To be honest I'm a little bit angry about this accusation. I didn't mean to copy from the Debian wiki and after re-checking the text again I'm still convinced that I didn't. Of course the text SAYS something similar, but

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-26 Thread Heikki Linnakangas
On 02/26/2014 10:35 AM, Christian Kruse wrote: On 25/02/14 19:28, Andres Freund wrote: I think all that's needed is to cut the first paragraphs that generally explain what huge pages are in some detail from the text and make sure the later paragraphs don't refer to the earlier ones. Attached

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-26 Thread Christian Kruse
Hi, On 26/02/14 14:34, Heikki Linnakangas wrote: That still says The setting is ignored on other systems. That's not quite true: as explained later in the section, if you set huge_tlb_pages=on and the platform doesn't support it, the server will refuse to start. I added a sentence about it.

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-26 Thread Alvaro Herrera
There's one thing that rubs me the wrong way about all this functionality, which is that we've named it huge TLB pages. That is wrong -- the TLB pages are not huge. In fact, as far as I understand, the TLB doesn't have pages at all. It's the pages that are huge, but those pages are not TLB

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-26 Thread Heikki Linnakangas
On 02/26/2014 06:13 PM, Alvaro Herrera wrote: There's one thing that rubs me the wrong way about all this functionality, which is that we've named it huge TLB pages. That is wrong -- the TLB pages are not huge. In fact, as far as I understand, the TLB doesn't have pages at all. It's the

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-26 Thread Stephen Frost
Christian, Thanks for working on all of this and dealing with the requests for updates and changes, as well as for dealing very professionally with an inappropriate and incorrect remark. Unfortunately, mailing lists can make communication difficult and someone's knee-jerk reaction (not referring

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-26 Thread Christian Kruse
Hi, On 26/02/14 13:13, Alvaro Herrera wrote: There's one thing that rubs me the wrong way about all this functionality, which is that we've named it huge TLB pages. That is wrong -- the TLB pages are not huge. In fact, as far as I understand, the TLB doesn't have pages at all. It's the

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-26 Thread Christian Kruse
Hi Peter, thank you for your nice words, much appreciated. I'm sorry that I was so whiny about this in the last post. Best regards, -- Christian Kruse http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services pgpxVj8SJRDQS.pgp Description: PGP

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-25 Thread Peter Eisentraut
On 1/30/14, 2:28 AM, Christian Kruse wrote: after I finally got documentation compilation working I updated the patch to be syntactically correct. You will find it attached. I don't think we should be explaining the basics of OS memory management in our documentation. And if we did, we

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-25 Thread Andres Freund
On 2014-02-25 10:29:32 -0500, Peter Eisentraut wrote: On 1/30/14, 2:28 AM, Christian Kruse wrote: after I finally got documentation compilation working I updated the patch to be syntactically correct. You will find it attached. I don't think we should be explaining the basics of OS memory

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-25 Thread Christian Kruse
Hi, On 25/02/14 10:29, Peter Eisentraut wrote: I don't think we should be explaining the basics of OS memory management in our documentation. Well, I'm confused. I thought that's exactly what has been asked. And if we did, we shouldn't copy it verbatim from the Debian wiki without

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-25 Thread Christian Kruse
Hi, On 25/02/14 17:01, Andres Freund wrote: And if we did, we shouldn't copy it verbatim from the Debian wiki without attribution. Is it actually? A quick comparison doesn't show that many similarities? Christian? Not as far as I know. But of course, as I wrote the text I _also_ (that's

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-25 Thread Peter Eisentraut
On 2/25/14, 11:08 AM, Christian Kruse wrote: Hi, On 25/02/14 17:01, Andres Freund wrote: And if we did, we shouldn't copy it verbatim from the Debian wiki without attribution. Is it actually? A quick comparison doesn't show that many similarities? Christian? Not as far as I know. But

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-25 Thread Bruce Momjian
On Tue, Feb 25, 2014 at 12:18:02PM -0500, Peter Eisentraut wrote: On 2/25/14, 11:08 AM, Christian Kruse wrote: Hi, On 25/02/14 17:01, Andres Freund wrote: And if we did, we shouldn't copy it verbatim from the Debian wiki without attribution. Is it actually? A quick comparison

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-25 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: On Tue, Feb 25, 2014 at 12:18:02PM -0500, Peter Eisentraut wrote: As I mentioned, I would just cut those introductory parts out. Should we link to the Debian wiki content? -1. We generally don't link to our *own* wiki in our SGML docs, let alone things

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-02-25 Thread Andres Freund
On 2014-02-25 13:21:46 -0500, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: On Tue, Feb 25, 2014 at 12:18:02PM -0500, Peter Eisentraut wrote: As I mentioned, I would just cut those introductory parts out. Should we link to the Debian wiki content? -1. We generally don't link

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Heikki Linnakangas
On 01/28/2014 06:11 PM, Christian Kruse wrote: Hi, attached you will find a new version of the patch, ported to HEAD, fixed the mentioned bug and - hopefully - dealing the the remaining issues. Thanks, I have committed this now. The documentation is still lacking. We should explain somewhere

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Vik Fearing
On 01/29/2014 01:12 PM, Heikki Linnakangas wrote: On 01/28/2014 06:11 PM, Christian Kruse wrote: Hi, attached you will find a new version of the patch, ported to HEAD, fixed the mentioned bug and - hopefully - dealing the the remaining issues. Thanks, I have committed this now. The

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Heikki Linnakangas
On 01/29/2014 04:01 PM, Vik Fearing wrote: On 01/29/2014 01:12 PM, Heikki Linnakangas wrote: The documentation is still lacking. The documentation is indeed lacking since it breaks the build. doc/src/sgml/config.sgml contains the line normal allocation if that fails. With

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Merlin Moncure
On Tue, Jan 28, 2014 at 5:58 AM, Christian Kruse christ...@2ndquadrant.com wrote: Hi, On 28/01/14 13:51, Heikki Linnakangas wrote: Oh darn, I remembered we had already committed this, but clearly not. I'd love to still get this into 9.4. The latest patch (hugepages-v5.patch) was pretty much

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Jeff Janes
On Wed, Jan 29, 2014 at 4:12 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/28/2014 06:11 PM, Christian Kruse wrote: Hi, attached you will find a new version of the patch, ported to HEAD, fixed the mentioned bug and - hopefully - dealing the the remaining issues. Thanks, I

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Christian Kruse
Hi, On 29/01/14 14:12, Heikki Linnakangas wrote: The documentation is still lacking. We should explain somewhere how to set nr.hugepages, for example. The Managing Kernel Resources section ought to mention setting. Could I ask you to work on that, please? Of course! Attached you will find a

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Christian Kruse
Hi, On 29/01/14 10:11, Jeff Janes wrote: I'm getting this warning now with gcc (GCC) 4.4.7: Interesting. I don't get that warning. But the compiler is (formally) right. pg_shmem.c: In function 'PGSharedMemoryCreate': pg_shmem.c:332: warning: 'allocsize' may be used uninitialized in this

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Heikki Linnakangas
On 01/29/2014 09:18 PM, Christian Kruse wrote: Hi, On 29/01/14 10:11, Jeff Janes wrote: I'm getting this warning now with gcc (GCC) 4.4.7: Interesting. I don't get that warning. But the compiler is (formally) right. pg_shmem.c: In function 'PGSharedMemoryCreate': pg_shmem.c:332: warning:

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Christian Kruse
Hi, On 29/01/14 21:36, Heikki Linnakangas wrote: […] Fix pushed. You are right. Thanks. But there is another bug, see 20140128154307.gc24...@defunct.ch ff. Attached you will find a patch fixing that. Best regards, -- Christian Kruse http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Heikki Linnakangas
On 01/29/2014 09:59 PM, Christian Kruse wrote: Hi, On 29/01/14 21:36, Heikki Linnakangas wrote: […] Fix pushed. You are right. Thanks. But there is another bug, see 20140128154307.gc24...@defunct.ch ff. Attached you will find a patch fixing that. Thanks. There are more cases of that in

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Christian Kruse
Hi, On 29/01/14 22:17, Heikki Linnakangas wrote: Thanks. There are more cases of that in InternalIpcMemoryCreate, they ought to be fixed as well. And should also grep the rest of the codebase for more instances of that. And this needs to be back-patched. I'm way ahead of you ;-) Working on

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-29 Thread Christian Kruse
Hi, after I finally got documentation compilation working I updated the patch to be syntactically correct. You will find it attached. Best regards, -- Christian Kruse http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services diff --git

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-28 Thread Heikki Linnakangas
On 01/27/2014 09:20 PM, Alvaro Herrera wrote: Heikki Linnakangas wrote: I spent some time whacking this around, new patch version attached. I moved the mmap() code into a new function, that leaves the PGSharedMemoryCreate more readable. Did this patch go anywhere? Oh darn, I remembered we

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-28 Thread Christian Kruse
Hi, On 28/01/14 13:51, Heikki Linnakangas wrote: Oh darn, I remembered we had already committed this, but clearly not. I'd love to still get this into 9.4. The latest patch (hugepages-v5.patch) was pretty much ready for commit, except for documentation. I'm working on it. I ported it to HEAD

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-28 Thread Christian Kruse
Hi, On 15/11/13 15:17, Heikki Linnakangas wrote: I spent some time whacking this around, new patch version attached. I moved the mmap() code into a new function, that leaves the PGSharedMemoryCreate more readable. I think there's a bug in this version of the patch. Have a look at this: +

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-28 Thread Christian Kruse
Hi, attached you will find a new version of the patch, ported to HEAD, fixed the mentioned bug and - hopefully - dealing the the remaining issues. Best regards, -- Christian Kruse http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services diff --git

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2014-01-27 Thread Alvaro Herrera
Heikki Linnakangas wrote: I spent some time whacking this around, new patch version attached. I moved the mmap() code into a new function, that leaves the PGSharedMemoryCreate more readable. Did this patch go anywhere? Someone just pinged me about a kernel scalability problem in Linux with

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-11-24 Thread Abhijit Menon-Sen
At 2013-11-21 22:14:35 +0100, and...@2ndquadrant.com wrote: I'd certainly want a setting that errors out if it cannot get the memory using hugetables. OK, then the current try/on/off settings are fine. I'm better today, so I'll read the patch Heikki posted and see what more needs to be done

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-11-21 Thread Andres Freund
On 2013-11-21 18:09:38 -0300, Alvaro Herrera wrote: Abhijit Menon-Sen wrote: At 2013-11-15 15:17:32 +0200, hlinnakan...@vmware.com wrote: But I'm not wedded to the idea if someone objects; a log message might also be reasonable: LOG: huge TLB pages are not supported on this platform,

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-11-21 Thread Andres Freund
On 2013-11-21 16:24:56 -0500, Robert Haas wrote: What about huge_tlb_pages={off,try} Or maybe huge_tlb_pages={off,try,require} I'd spell require as on, or at least accept that as a synonym. That's off,try, on is what the patch currently implements, Abhijit just was arguing for

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-11-21 Thread Alvaro Herrera
Abhijit Menon-Sen wrote: At 2013-11-15 15:17:32 +0200, hlinnakan...@vmware.com wrote: But I'm not wedded to the idea if someone objects; a log message might also be reasonable: LOG: huge TLB pages are not supported on this platform, but huge_tlb_pages was 'on' Put that way, I have to

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-11-21 Thread Robert Haas
On Thu, Nov 21, 2013 at 4:09 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Abhijit Menon-Sen wrote: At 2013-11-15 15:17:32 +0200, hlinnakan...@vmware.com wrote: But I'm not wedded to the idea if someone objects; a log message might also be reasonable: LOG: huge TLB pages are not

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-11-17 Thread Abhijit Menon-Sen
At 2013-11-15 15:17:32 +0200, hlinnakan...@vmware.com wrote: I spent some time whacking this around, new patch version attached. Thanks. But I'm not wedded to the idea if someone objects; a log message might also be reasonable: LOG: huge TLB pages are not supported on this platform, but

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-11-16 Thread Sameer Kumar
I was recently running some tests with huge page tables. I ran them on two different architectures: x86 and PPC64. I saw some discussion going on over here so thought of sharing. I was using 3 Cores, 8GB RAM, 2 LUN for filesystem (1 for dbfiles and 1 for logfiles) for these tests... I had

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-11-15 Thread Heikki Linnakangas
On 30.10.2013 19:11, Andres Freund wrote: On 2013-10-30 22:39:20 +0530, Abhijit Menon-Sen wrote: At 2013-10-30 11:04:36 -0400, t...@sss.pgh.pa.us wrote: As a compromise, perhaps we can unconditionally round the size up to be a multiple of 2MB? […] That sounds reasonably painless to me.

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Sergey Konoplev
On Tue, Oct 29, 2013 at 9:31 PM, Tom Lane t...@sss.pgh.pa.us wrote: Sergey Konoplev gray...@gmail.com writes: On Wed, Oct 23, 2013 at 11:03 PM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: This is a slightly reworked version of the patch submitted by Richard Poole last month, which was

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread David Fetter
On Tue, Oct 29, 2013 at 11:08:05PM -0700, Sergey Konoplev wrote: On Tue, Oct 29, 2013 at 9:31 PM, Tom Lane t...@sss.pgh.pa.us wrote: Sergey Konoplev gray...@gmail.com writes: On Wed, Oct 23, 2013 at 11:03 PM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: This is a slightly reworked

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread David Fetter
On Wed, Oct 30, 2013 at 10:16:57AM +0530, Abhijit Menon-Sen wrote: At 2013-10-24 16:06:19 +0300, hlinnakan...@vmware.com wrote: Let's get rid of the rounding. I share Andres's concern that the bug is present in various recent kernels that are going to stick around for quite some time.

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Abhijit Menon-Sen
At 2013-10-30 00:10:39 -0700, da...@fetter.org wrote: How about documenting that 2MB is the quantum (OK, we'll say indivisible unit or smallest division or something) and failing with a message to that effect if someone tries to set it otherwise? I don't think you understand the problem.

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Tom Lane
Abhijit Menon-Sen a...@2ndquadrant.com writes: As a compromise, perhaps we can unconditionally round the size up to be a multiple of 2MB? That way, we can use huge pages more often, but also avoid putting in a lot of code and effort into the workaround and waste only a little space (if any at

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Tom Lane
Sergey Konoplev gray...@gmail.com writes: On Tue, Oct 29, 2013 at 9:31 PM, Tom Lane t...@sss.pgh.pa.us wrote: Say what? There's never been any hugepages support in Postgres. There were an ability to back shared memory with hugepages when using =9.2. I use it on ~30 servers for several years

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Abhijit Menon-Sen
At 2013-10-30 11:04:36 -0400, t...@sss.pgh.pa.us wrote: As a compromise, perhaps we can unconditionally round the size up to be a multiple of 2MB? […] That sounds reasonably painless to me. Here's a patch that does that and adds a DEBUG1 log message when we try with MAP_HUGETLB and fail

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Andres Freund
On 2013-10-30 22:39:20 +0530, Abhijit Menon-Sen wrote: At 2013-10-30 11:04:36 -0400, t...@sss.pgh.pa.us wrote: As a compromise, perhaps we can unconditionally round the size up to be a multiple of 2MB? […] That sounds reasonably painless to me. Here's a patch that does that and

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Sergey Konoplev
On Wed, Oct 30, 2013 at 8:11 AM, Tom Lane t...@sss.pgh.pa.us wrote: Sergey Konoplev gray...@gmail.com writes: On Tue, Oct 29, 2013 at 9:31 PM, Tom Lane t...@sss.pgh.pa.us wrote: Say what? There's never been any hugepages support in Postgres. There were an ability to back shared memory with

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Alvaro Herrera
Sergey Konoplev escribió: On Wed, Oct 30, 2013 at 8:11 AM, Tom Lane t...@sss.pgh.pa.us wrote: Sergey Konoplev gray...@gmail.com writes: There were an ability to back shared memory with hugepages when using =9.2. I use it on ~30 servers for several years and it brings 8-17% of performance

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Sergey Konoplev
On Wed, Oct 30, 2013 at 11:50 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: There were an ability to back shared memory with hugepages when using =9.2. I use it on ~30 servers for several years and it brings 8-17% of performance depending on the memory size. Here you will find several

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Alvaro Herrera
Alvaro Herrera escribió: Sergey Konoplev escribió: I wasn't talking about a built-in support. It was about an ability (a way) to back sh_buf with hugepages. Then what you need is to set dynamic_shared_memory_type = sysv in postgresql.conf. The above is mistaken -- there's no way to

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Sergey Konoplev
On Wed, Oct 30, 2013 at 12:17 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I wasn't talking about a built-in support. It was about an ability (a way) to back sh_buf with hugepages. Then what you need is to set dynamic_shared_memory_type = sysv in postgresql.conf. The above is

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-30 Thread Sergey Konoplev
On Wed, Oct 30, 2013 at 12:51 PM, Sergey Konoplev gray...@gmail.com wrote: On Wed, Oct 30, 2013 at 12:17 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I wasn't talking about a built-in support. It was about an ability (a way) to back sh_buf with hugepages. Then what you need is to set

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-29 Thread Sergey Konoplev
Hi, On Wed, Oct 23, 2013 at 11:03 PM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: This is a slightly reworked version of the patch submitted by Richard Poole last month, which was based on Christian Kruse's earlier patch. Is it possible that this patch will be included in a minor version of

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-29 Thread Tom Lane
Sergey Konoplev gray...@gmail.com writes: On Wed, Oct 23, 2013 at 11:03 PM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: This is a slightly reworked version of the patch submitted by Richard Poole last month, which was based on Christian Kruse's earlier patch. Is it possible that this

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-29 Thread Abhijit Menon-Sen
At 2013-10-24 16:06:19 +0300, hlinnakan...@vmware.com wrote: Let's get rid of the rounding. I share Andres's concern that the bug is present in various recent kernels that are going to stick around for quite some time. Given the rather significant performance gain, I think it's worth doing

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-29 Thread Abhijit Menon-Sen
At 2013-10-24 19:00:28 +0200, and...@2ndquadrant.com wrote: I think we should log when we tried to use hugepages but fell back to plain mmap, currently it's hard to see whether they are used. Good idea, thanks. I'll do this in the next patch I post (which will be after we reach some consensus

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-24 Thread Abhijit Menon-Sen
At 2013-10-24 11:33:13 +0530, a...@2ndquadrant.com wrote: From /proc/$pid/status, VmPTE was 2880kb with huge_tlb_pages=off, and 56kb with it turned on. (VmPTE is the size of the process's page tables.) -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-24 Thread Heikki Linnakangas
On 24.10.2013 09:03, Abhijit Menon-Sen wrote: This is a slightly reworked version of the patch submitted by Richard Poole last month, which was based on Christian Kruse's earlier patch. Thanks. With huge_tlb_pages=off, this is the best result I got: tps = 8680.771068 (including

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-24 Thread Robert Haas
On Thu, Oct 24, 2013 at 9:06 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: * the documentation should perhaps mention that the setting only has an effect if POSIX shared memory is used. That's the default on Linux, but we will try to fall back to SystemV shared memory if it fails. This

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-24 Thread Andres Freund
On 2013-10-24 16:06:19 +0300, Heikki Linnakangas wrote: On 24.10.2013 09:03, Abhijit Menon-Sen wrote: One open question is what to do about rounding up the size. It should not be necessary, but for the fairly recent bug described at the link in the comment

Re: [HACKERS] [PATCH] Use MAP_HUGETLB where supported (v3)

2013-10-24 Thread Robert Haas
On Thu, Oct 24, 2013 at 1:00 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-10-24 16:06:19 +0300, Heikki Linnakangas wrote: On 24.10.2013 09:03, Abhijit Menon-Sen wrote: One open question is what to do about rounding up the size. It should not be necessary, but for the fairly recent