Re: [HACKERS] Supporting huge pages on Windows

From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Ashutosh Sharma > I have once again tested the latest patch (v14 patch) on Windows and the > results looked fine to me. Basically I have repeated the test cases which > I had done earlier on v8

Re: [HACKERS] Supporting huge pages on Windows

On Wed, Sep 13, 2017 at 3:41 AM, Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> wrote: > Hi Thomas, Magnus > > From: pgsql-hackers-ow...@postgresql.org > > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Thomas Munro > > Since it only conflicts with c7b8998e because of pgindent

Re: [HACKERS] Supporting huge pages on Windows

On Wed, Sep 13, 2017 at 7:11 AM, Tsunakawa, Takayuki wrote: > Hi Thomas, Magnus > > From: pgsql-hackers-ow...@postgresql.org >> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Thomas Munro >> Since it only conflicts with c7b8998e because of pgindent

Re: [HACKERS] Supporting huge pages on Windows

Hi Thomas, Magnus From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Thomas Munro > Since it only conflicts with c7b8998e because of pgindent whitespace > movement, I applied it with "patch -p1 --ignore-whitespace" and created > a new patch. See

Re: [HACKERS] Supporting huge pages on Windows

On Thu, Aug 17, 2017 at 2:11 PM, Thomas Munro wrote: > On Wed, Apr 12, 2017 at 7:08 PM, Tsunakawa, Takayuki > wrote: >> Oh, I got it now. Thanks. The revised patch is attached. The only >> modified file is pg_ctl.c. The patch

Re: [HACKERS] Supporting huge pages on Windows

On Wed, Apr 12, 2017 at 7:08 PM, Tsunakawa, Takayuki wrote: > Oh, I got it now. Thanks. The revised patch is attached. The only modified > file is pg_ctl.c. The patch worked as expected. > > It is regrettable that I could not make it in time for PG 10, but I'd

Re: [HACKERS] Supporting huge pages on Windows

Hello, Magnus cc: Andres From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Magnus Hagander > I think what you'd need to do is enumerate what privileges the user has > *before* calling CreateRestrictedToken(), using GetTokenInformation(). > And

Re: [HACKERS] Supporting huge pages on Windows

On 2017-04-07 13:57:07 +0200, Magnus Hagander wrote: > On Wed, Apr 5, 2017 at 9:15 AM, Tsunakawa, Takayuki < > tsunakawa.ta...@jp.fujitsu.com> wrote: > > > From: pgsql-hackers-ow...@postgresql.org > > > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Andres Freund > > > As I asked

Re: [HACKERS] Supporting huge pages on Windows

On Wed, Apr 5, 2017 at 9:15 AM, Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> wrote: > From: pgsql-hackers-ow...@postgresql.org > > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Andres Freund > > As I asked before, why can't we delete all privs and add the explicitly > > needed

Re: [HACKERS] Supporting huge pages on Windows

From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Andres Freund > As I asked before, why can't we delete all privs and add the explicitly > needed once back (using AdjustTokenPrivileges)? I tried it with pg_ctl.c attached to an earlier mail today,

Re: [HACKERS] Supporting huge pages on Windows

On 2017-04-05 04:25:41 +, Tsunakawa, Takayuki wrote: > From: Craig Ringer [mailto:craig.rin...@2ndquadrant.com] > > On 5 April 2017 at 10:37, Tsunakawa, Takayuki > > wrote: > > > > OTOH, I tried again to leave the DISABLE_MAX_PRIVILEGE as is and add Lock > >

Re: [HACKERS] Supporting huge pages on Windows

From: Craig Ringer [mailto:craig.rin...@2ndquadrant.com] > On 5 April 2017 at 10:37, Tsunakawa, Takayuki > wrote: > > OTOH, I tried again to leave the DISABLE_MAX_PRIVILEGE as is and add Lock > Pages in Memory, using the attached pg_ctl.c. Please see >

Re: [HACKERS] Supporting huge pages on Windows

On 5 April 2017 at 10:37, Tsunakawa, Takayuki wrote: > Good point! And I said earlier in this thread, I think managing privileges > (adding/revoking privileges from the user account) is the DBA's or sysadmin's > duty, and PG's removing all privileges feels

Re: [HACKERS] Supporting huge pages on Windows

From: Craig Ringer [mailto:craig.rin...@2ndquadrant.com] > TBH, anyone who cares about security and runs Win7 or Win2k8 or newer should > be using virtual service accounts and managed service accounts. > > https://technet.microsoft.com/en-us/library/dd548356 > > > Those are more like Unix

Re: [HACKERS] Supporting huge pages on Windows

On 4 Apr. 2017 14:22, "Andres Freund" wrote: On 2017-01-05 03:12:09 +, Tsunakawa, Takayuki wrote: > From: pgsql-hackers-ow...@postgresql.org > > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Magnus Hagander > > For the pg_ctl changes, we're going from removing

Re: [HACKERS] Supporting huge pages on Windows

On 2017-01-05 03:12:09 +, Tsunakawa, Takayuki wrote: > From: pgsql-hackers-ow...@postgresql.org > > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Magnus Hagander > > For the pg_ctl changes, we're going from removing all privilieges from the > > token, to removing none. Are there any

Re: [HACKERS] Supporting huge pages on Windows

From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Andres Freund > I don't think the errdetail is quite right - OpenProcessToken isn't really > a syscall, is it? But then it's a common pattern already in wind32_shmem.c... Yes, "Win32 API function"

Re: [HACKERS] Supporting huge pages on Windows

On 2017-04-03 04:56:45 +, Tsunakawa, Takayuki wrote: > +/* > + * EnableLockPagesPrivilege > + * > + * Try to acquire SeLockMemoryPrivilege so we can use large pages. > + */ > +static bool > +EnableLockPagesPrivilege(int elevel) > +{ > + HANDLE hToken; > + TOKEN_PRIVILEGES tp; > +

Re: [HACKERS] Supporting huge pages on Windows

From: Michael Paquier [mailto:michael.paqu...@gmail.com] > Amit is right, you had better use %zu as we do in all the other places of > the code dealing with Size variables that are printed. When compiling with > Visual Studio, Postgres falls back to the implementation of sprintf in >

Re: [HACKERS] Supporting huge pages on Windows

On Mon, Apr 3, 2017 at 1:12 PM, Tsunakawa, Takayuki wrote: > From: pgsql-hackers-ow...@postgresql.org >> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Amit Kapila >> You should use %zu instead of %llu to print Size type of variable. > > I did so at

Re: [HACKERS] Supporting huge pages on Windows

From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Amit Kapila > You should use %zu instead of %llu to print Size type of variable. I did so at first, but it didn't work with Visual Studio 2010 at hand. It doesn't support %z which is added in

Re: [HACKERS] Supporting huge pages on Windows

On Fri, Mar 31, 2017 at 8:05 AM, Tsunakawa, Takayuki wrote: > From: Amit Kapila [mailto:amit.kapil...@gmail.com] >> The latest patch looks good to me apart from one Debug message, so I have >> marked it as Ready For Committer. > > Thank you so much! > > >> +

Re: [HACKERS] Supporting huge pages on Windows

From: Amit Kapila [mailto:amit.kapil...@gmail.com] > The latest patch looks good to me apart from one Debug message, so I have > marked it as Ready For Committer. Thank you so much! > + ereport(DEBUG1, > + (errmsg("disabling huge pages"))); > > I think this should be similar to what we display

Re: [HACKERS] Supporting huge pages on Windows

On Thu, Mar 9, 2017 at 7:06 AM, Tsunakawa, Takayuki wrote: > From: pgsql-hackers-ow...@postgresql.org >> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Ashutosh Sharma >> To start with, I ran the regression test-suite and didn't find any failures. >> But,

Re: [HACKERS] Supporting huge pages on Windows

From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of David Steele > It's not clear to me what state this patch should be in. Is there more > review that needs to be done or is it ready for a committer? I would most appreciate it if Magnus could

Re: [HACKERS] Supporting huge pages on Windows

On 3/24/17 12:51 PM, Ashutosh Sharma wrote: Hi, On Fri, Mar 24, 2017 at 9:00 PM, David Steele wrote: Hi Ashutosh, On 3/22/17 8:52 AM, Amit Kapila wrote: On Wed, Mar 22, 2017 at 12:07 AM, David Steele wrote: Amit, Magnus, you are signed up as

Re: [HACKERS] Supporting huge pages on Windows

Hi, On Fri, Mar 24, 2017 at 9:00 PM, David Steele wrote: > Hi Ashutosh, > > On 3/22/17 8:52 AM, Amit Kapila wrote: >> >> On Wed, Mar 22, 2017 at 12:07 AM, David Steele >> wrote: >>> >>> >>> Amit, Magnus, you are signed up as reviewers for this patch.

Re: [HACKERS] Supporting huge pages on Windows

Hi Ashutosh, On 3/22/17 8:52 AM, Amit Kapila wrote: On Wed, Mar 22, 2017 at 12:07 AM, David Steele wrote: Amit, Magnus, you are signed up as reviewers for this patch. Do you know when you'll have a chance to take a look? I have provided my feedback and I could not

Re: [HACKERS] Supporting huge pages on Windows

On Wed, Mar 22, 2017 at 12:07 AM, David Steele wrote: > On 3/8/17 8:36 PM, Tsunakawa, Takayuki wrote: >> >> From: pgsql-hackers-ow...@postgresql.org >>> >>> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Ashutosh Sharma >>> To start with, I ran the regression

Re: [HACKERS] Supporting huge pages on Windows

On 3/8/17 8:36 PM, Tsunakawa, Takayuki wrote: From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Ashutosh Sharma To start with, I ran the regression test-suite and didn't find any failures. But, then I am not sure if huge_pages are getting used or

Re: [HACKERS] Supporting huge pages on Windows

From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Ashutosh Sharma > To start with, I ran the regression test-suite and didn't find any failures. > But, then I am not sure if huge_pages are getting used or not. However, > upon checking the settings

Re: [HACKERS] Supporting huge pages on Windows

On Thu, Feb 23, 2017 at 8:29 PM, Tsunakawa, Takayuki wrote: > [win_large_pages_v8.patch] +Huge pages are known as large pages on Windows. To use them, you need to +assign the user right Lock Pages in Memory to the Windows user account +

Re: [HACKERS] Supporting huge pages on Windows

Hi, I tried to test v8 version of patch. Firstly, I was able to start the postgresql server process with 'huge_pages' set to on. I had to follow the instructions given in MSDN[1] to enable lock pages in memory option and also had to start the postgresql server process as admin user. test=# show

Re: [HACKERS] Supporting huge pages on Windows

From: Amit Kapila [mailto:amit.kapil...@gmail.com] > > Hmm, the large-page requires contiguous memory for each page, so this > error could occur on a long-running system where the memory is heavily > fragmented. For example, please see the following page and check the memory > with RAMMap program

Re: [HACKERS] Supporting huge pages on Windows

On Mon, Jan 30, 2017 at 10:46 AM, Tsunakawa, Takayuki wrote: > From: Amit Kapila [mailto:amit.kapil...@gmail.com] >> I think it is better to document in some way if we decide to go-ahead with >> the patch. > > Sure, I added these sentences. Patch has been moved to

Re: [HACKERS] Supporting huge pages on Windows

Magnus Hagander wrote: > Taking a look at the actual code here, and a few small nitpicks: > > +errdetail("Failed system call was %s, > error code %lu", "LookupPrivilegeValue", GetLastError(; > > this seems to be a new pattern of code -- for other similar

Re: [HACKERS] Supporting huge pages on Windows

On Mon, Jan 30, 2017 at 7:16 AM, Tsunakawa, Takayuki wrote: > From: Amit Kapila [mailto:amit.kapil...@gmail.com] >> Hmm. It doesn't work even on a command prompt with administrative >> privileges. It gives below error: >> >> waiting for server to

Re: [HACKERS] Supporting huge pages on Windows

From: Amit Kapila [mailto:amit.kapil...@gmail.com] > Hmm. It doesn't work even on a command prompt with administrative > privileges. It gives below error: > > waiting for server to start2017-01-17 11:20:13.780 IST [4788] FATAL: > could not create shared memory segment: error code 1450 >

Re: [HACKERS] Supporting huge pages on Windows

On Tue, Jan 10, 2017 at 8:49 AM, Tsunakawa, Takayuki wrote: > Hello, Amit, Magnus, > > I'm sorry for my late reply. Yesterday was a national holiday in Japan. > > > From: pgsql-hackers-ow...@postgresql.org >> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf

Re: [HACKERS] Supporting huge pages on Windows

Hello, Amit, Magnus, I'm sorry for my late reply. Yesterday was a national holiday in Japan. From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Amit Kapila > PGSharedMemoryReAttach is called after the startup of new process whereas >

Re: [HACKERS] Supporting huge pages on Windows

On Thu, Jan 5, 2017 at 8:42 AM, Tsunakawa, Takayuki wrote: >> Your version of the patch looks better than the previous one. Don't you >> need to consider MEM_LARGE_PAGES in VirtualAllocEx call (refer >> pgwin32_ReserveSharedMemoryRegion)? At least that is what is

Re: [HACKERS] Supporting huge pages on Windows

From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com] > In the Microsoft documentation I've seen, the privilege's name is always > written as "Lock Pages in Memory" (note: "Pages" plural, and with initial > capital letters). It's quite hard to parse the sentence otherwise! How > about this?

Re: [HACKERS] Supporting huge pages on Windows

On Thu, Jan 5, 2017 at 4:12 PM, Tsunakawa, Takayuki wrote: > [win_large_pages_v4.patch] Just a small suggestion about the wording in this patch: +This feature uses the large-page support on Windows. To use the large-page +support, you need to

Re: [HACKERS] Supporting huge pages on Windows

From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Magnus Hagander > For the pg_ctl changes, we're going from removing all privilieges from the > token, to removing none. Are there any other privileges that we should be > worried about? I think you

Re: [HACKERS] Supporting huge pages on Windows

On Sat, Dec 31, 2016 at 7:04 PM, Magnus Hagander wrote: > > > On Wed, Sep 28, 2016 at 2:26 AM, Tsunakawa, Takayuki > wrote: >> >> > From: pgsql-hackers-ow...@postgresql.org >> > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert

Re: [HACKERS] Supporting huge pages on Windows

On Wed, Sep 28, 2016 at 2:26 AM, Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> wrote: > > From: pgsql-hackers-ow...@postgresql.org > > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas > > On Sun, Sep 25, 2016 at 10:45 PM, Tsunakawa, Takayuki > >

Re: [HACKERS] Supporting huge pages on Windows

On Tue, Oct 11, 2016 at 1:36 PM, Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> wrote: > > Moved to next CF with "needs review" status. Regards, Hari Babu Fujitsu Australia

Re: [HACKERS] Supporting huge pages on Windows

From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com] > Your ~2.4% number is similar to what was reported for Linux with 4GB > shared_buffers: > > https://www.postgresql.org/message-id/20130913234125.GC13697%40roobarb > .crazydogs.org I'm relieved to know that a similar figure was gained on

Re: [HACKERS] Supporting huge pages on Windows

On 2016-10-11 09:57:48 +1300, Thomas Munro wrote: > Later in that thread there was a report of a dramatic ~15% increase in > "best result" TPS, but that was with 60GB of shared_buffers on a > machine with 256GB of RAM: > > https://www.postgresql.org/message-id/20131024060313.GA21888%40toroid.org

Re: [HACKERS] Supporting huge pages on Windows

On Wed, Sep 28, 2016 at 7:32 PM, Tsunakawa, Takayuki wrote: > From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com] >> > huge_pages=off: 70412 tps >> > huge_pages=on : 72100 tps >> >> Hmm. I guess it could be noise or random code rearrangement effects. > >

Re: [HACKERS] Supporting huge pages on Windows

From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com] > > huge_pages=off: 70412 tps > > huge_pages=on : 72100 tps > > Hmm. I guess it could be noise or random code rearrangement effects. I'm not the difference was a random noise, because running multiple set of three runs of pgbench

Re: [HACKERS] Supporting huge pages on Windows

On Wed, Sep 28, 2016 at 1:26 PM, Tsunakawa, Takayuki wrote: >> From: pgsql-hackers-ow...@postgresql.org >> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas >> On Sun, Sep 25, 2016 at 10:45 PM, Tsunakawa, Takayuki >>

Re: [HACKERS] Supporting huge pages on Windows

> From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas > On Sun, Sep 25, 2016 at 10:45 PM, Tsunakawa, Takayuki > wrote: > > Credit: This patch is based on Thomas Munro's one. > > How are they different?

Re: [HACKERS] Supporting huge pages on Windows

On Sun, Sep 25, 2016 at 10:45 PM, Tsunakawa, Takayuki wrote: > Credit: This patch is based on Thomas Munro's one. How are they different? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers

[HACKERS] Supporting huge pages on Windows

Hello, The attached patch implements huge_pages on Windows. I'll add this to the CommitFest. The performance improvement was about 2% with the following select-only pgbench. The scaling factor is 200, where the database size is roughly 3GB. I ran the benchmark on my Windows 10 PC with 6