Re: [HACKERS] Supporting huge pages on Windows

[Andrew Dunstan]

On 01/22/2018 04:16 AM, Magnus Hagander wrote:
>
>
>
>
> I'll be quite happy to retire the XP machine running brolga, currawong
> and frogmouth, if that's the consensus. XP is now long out of support.
> OTOH I have personal experience of it running in many potentially
> critical situations, still (hospitals, for example). 
>
>
> But do they really run PostgreSQL 11 (or 10..) on that? In my
> experience they usually run an old business application on it only.
> That is a problem in itself of course, but that is not our problem in
> this case :)
>
>  
>
> I can, if people
> want, keep the machine running just building the back branches.
>
>
> That's what I suggest we do. Removing the builds of back branches
> would be the equivalent of de-supporting it on a still supported
> branch, and I don't like that idea. But removing the master branch
> means we desupport in 11, which I think is the right thing to do.


OK, I have left the machine running but these three animals will no
longer build 11, only the back branches.

>
>
>  
>
> I should probably look at setting up a modern 32-bit replacement (say
> Windows 10 Pro-32).
>
>
> Unless we want to desupport 32-bit Windows completely. But unless we
> have an actual reason to do so, I think we shouldn't. So yeah if you
> can get a box like that up and running, that'd be much welcome.


It's probably going to have to wait a couple of months, and at least a
couple of weeks.

It's worth noting that the last Windows Server edition that supported
342bit architectures was WS2008. That's quite old now. I wonder how long
they will continue to support it in the consumer-grade Windows versions.

cheers

andrew

-- 
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Supporting huge pages on Windows

[Magnus Hagander]

On Mon, Jan 22, 2018 at 4:24 AM, Thomas Munro  wrote:

> On Mon, Jan 22, 2018 at 3:45 AM, Magnus Hagander 
> wrote:
> > I got myself a working build env now so I can at least verify it builds,
> > which it does.
> >
> > With that, I'm pushing this. Let's see what the buildfarm thinks of it.
> And
> > if others end up complaining about the platform drop, but I doubt that.
> >
> > Once again, apologies for the delay, and thanks for the patch!
>
> +To start the database server on the command prompt as a
> standalone process,
> +not as a Windows service, the command prompt must be run as
> an administrator
> +User Access Control (UAC) must be disabled. When the UAC is
> enabled, the normal
> +command prompt revokes the user right Lock Pages in Memory
> when started.
>
> Is that first sentence  missing the word "and" after "administrator"?
>
>
Nope.

But it is missing the word "or".

Fixed, thanks for the spot!

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ 
 Work: https://www.redpill-linpro.com/ 

Re: [HACKERS] Supporting huge pages on Windows

[Magnus Hagander]

On Mon, Jan 22, 2018 at 12:13 AM, Andrew Dunstan <
andrew.duns...@2ndquadrant.com> wrote:

>
>
> On 01/21/2018 01:02 PM, Andres Freund wrote:
> > Hi,
> >
> > On 2018-01-21 13:42:13 +0200, Magnus Hagander wrote:
> >> To add some more notes on this. Again, the API appears in Vista/2003.
> >> Windows Vista went EOL (out of extended support even) in April 2017,
> >> Windows 2003 did so in July 2015. Those are the versions that it's *in*
> --
> >> obviously the versions without it are even older.
> >>
> >> Our binaries only support Windows 2008 and up (at least the ones by EDB,
> >> which are the ones that have a supported-version matrix documented on
> our
> >> site).
> >>
> >> We have traditionally supported older versions of Windows as long as
> people
> >> build from source. But perhaps I'm way overreading that and we should
> just
> >> bite the bullet, commit this patch, and declare those platforms as
> >> completely dead by PostgreSQL 11?
> > Yea, I think it's beyond time that we declare some old windows versions
> > dead. There's enough weird behaviour in supported versions of windows
> > (especially its socket API) that I really don't want to support more
> > than necessary. And supporting versions that've been out of date for a
> > while seems more than unnecessary.
> >
>
>
> I'll be quite happy to retire the XP machine running brolga, currawong
> and frogmouth, if that's the consensus. XP is now long out of support.
> OTOH I have personal experience of it running in many potentially
> critical situations, still (hospitals, for example).


But do they really run PostgreSQL 11 (or 10..) on that? In my experience
they usually run an old business application on it only. That is a problem
in itself of course, but that is not our problem in this case :)



> I can, if people
> want, keep the machine running just building the back branches.
>

That's what I suggest we do. Removing the builds of back branches would be
the equivalent of de-supporting it on a still supported branch, and I don't
like that idea. But removing the master branch means we desupport in 11,
which I think is the right thing to do.




> I should probably look at setting up a modern 32-bit replacement (say
> Windows 10 Pro-32).
>

Unless we want to desupport 32-bit Windows completely. But unless we have
an actual reason to do so, I think we shouldn't. So yeah if you can get a
box like that up and running, that'd be much welcome.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ 
 Work: https://www.redpill-linpro.com/ 

Re: [HACKERS] Supporting huge pages on Windows

[Thomas Munro]

On Mon, Jan 22, 2018 at 3:45 AM, Magnus Hagander  wrote:
> I got myself a working build env now so I can at least verify it builds,
> which it does.
>
> With that, I'm pushing this. Let's see what the buildfarm thinks of it. And
> if others end up complaining about the platform drop, but I doubt that.
>
> Once again, apologies for the delay, and thanks for the patch!

+To start the database server on the command prompt as a
standalone process,
+not as a Windows service, the command prompt must be run as
an administrator
+User Access Control (UAC) must be disabled. When the UAC is
enabled, the normal
+command prompt revokes the user right Lock Pages in Memory
when started.

Is that first sentence  missing the word "and" after "administrator"?

-- 
Thomas Munro
http://www.enterprisedb.com

Re: [HACKERS] Supporting huge pages on Windows

[Andrew Dunstan]

On 01/21/2018 01:02 PM, Andres Freund wrote:
> Hi,
>
> On 2018-01-21 13:42:13 +0200, Magnus Hagander wrote:
>> To add some more notes on this. Again, the API appears in Vista/2003.
>> Windows Vista went EOL (out of extended support even) in April 2017,
>> Windows 2003 did so in July 2015. Those are the versions that it's *in* --
>> obviously the versions without it are even older.
>>
>> Our binaries only support Windows 2008 and up (at least the ones by EDB,
>> which are the ones that have a supported-version matrix documented on our
>> site).
>>
>> We have traditionally supported older versions of Windows as long as people
>> build from source. But perhaps I'm way overreading that and we should just
>> bite the bullet, commit this patch, and declare those platforms as
>> completely dead by PostgreSQL 11?
> Yea, I think it's beyond time that we declare some old windows versions
> dead. There's enough weird behaviour in supported versions of windows
> (especially its socket API) that I really don't want to support more
> than necessary. And supporting versions that've been out of date for a
> while seems more than unnecessary.
>


I'll be quite happy to retire the XP machine running brolga, currawong
and frogmouth, if that's the consensus. XP is now long out of support.
OTOH I have personal experience of it running in many potentially
critical situations, still (hospitals, for example). I can, if people
want, keep the machine running just building the back branches.

I should probably look at setting up a modern 32-bit replacement (say
Windows 10 Pro-32).

cheers

andrew

-- 
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Supporting huge pages on Windows

[Andres Freund]

Hi,

On 2018-01-21 13:42:13 +0200, Magnus Hagander wrote:
> To add some more notes on this. Again, the API appears in Vista/2003.
> Windows Vista went EOL (out of extended support even) in April 2017,
> Windows 2003 did so in July 2015. Those are the versions that it's *in* --
> obviously the versions without it are even older.
> 
> Our binaries only support Windows 2008 and up (at least the ones by EDB,
> which are the ones that have a supported-version matrix documented on our
> site).
> 
> We have traditionally supported older versions of Windows as long as people
> build from source. But perhaps I'm way overreading that and we should just
> bite the bullet, commit this patch, and declare those platforms as
> completely dead by PostgreSQL 11?

Yea, I think it's beyond time that we declare some old windows versions
dead. There's enough weird behaviour in supported versions of windows
(especially its socket API) that I really don't want to support more
than necessary. And supporting versions that've been out of date for a
while seems more than unnecessary.

Greetings,

Andres Freund

Re: [HACKERS] Supporting huge pages on Windows

[Magnus Hagander]

On Sun, Jan 21, 2018 at 6:11 PM, Tom Lane  wrote:

> Magnus Hagander  writes:
> > With that, I'm pushing this. Let's see what the buildfarm thinks of it.
> And
> > if others end up complaining about the platform drop, but I doubt that.
>
> frogmouth:
>
> pg_shmem.c: In function 'PGSharedMemoryCreate':
> pg_shmem.c:205:3: warning: implicit declaration of function
> 'GetLargePageMinimum'
> pg_shmem.c:222:38: error: 'SEC_LARGE_PAGES' undeclared (first use in this
> function)
> pg_shmem.c:222:38: note: each undeclared identifier is reported only once
> for each function it appears in
> make[3]: *** [pg_shmem.o] Error 1
>
> so you were right to guess that this functionality isn't in XP.
>
> I wonder whether this could be band-aided around by using "#ifdef
> SEC_LARGE_PAGES" to protect the new code.  I have no particular desire to
> spend effort on supporting old Windows versions, but if there's someone
> out there who does, they could be asked to look into that.
>

I think that is not actually XP, in this case it's a case of the SDK being
too old. Which *might* happen on mingw on a modern platform as well.

The question is do we care enough. Or do we just declare XP as unsupported,
in which case frogmouth should stop building master.

I think the second one is OK, as long as it's only things that are as old
as XP. But I think we have to wait for a modre modern mingw (jacana?) to
complete before we can be sure.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ 
 Work: https://www.redpill-linpro.com/ 

Re: [HACKERS] Supporting huge pages on Windows

[Tom Lane]

Magnus Hagander  writes:
> With that, I'm pushing this. Let's see what the buildfarm thinks of it. And
> if others end up complaining about the platform drop, but I doubt that.

frogmouth:

pg_shmem.c: In function 'PGSharedMemoryCreate':
pg_shmem.c:205:3: warning: implicit declaration of function 
'GetLargePageMinimum'
pg_shmem.c:222:38: error: 'SEC_LARGE_PAGES' undeclared (first use in this 
function)
pg_shmem.c:222:38: note: each undeclared identifier is reported only once for 
each function it appears in
make[3]: *** [pg_shmem.o] Error 1

so you were right to guess that this functionality isn't in XP.

I wonder whether this could be band-aided around by using "#ifdef
SEC_LARGE_PAGES" to protect the new code.  I have no particular desire to
spend effort on supporting old Windows versions, but if there's someone
out there who does, they could be asked to look into that.

regards, tom lane

Re: [HACKERS] Supporting huge pages on Windows

[Magnus Hagander]

On Sun, Jan 21, 2018 at 2:30 PM, Michael Paquier 
wrote:

> On Sun, Jan 21, 2018 at 01:42:13PM +0200, Magnus Hagander wrote:
> > We have traditionally supported older versions of Windows as long as
> people
> > build from source. But perhaps I'm way overreading that and we should
> just
> > bite the bullet, commit this patch, and declare those platforms as
> > completely dead by PostgreSQL 11?
>
> Yeah, I'd like to think that this is a good idea for HEAD. Microsoft is
> pushing hard to have all its users move to newer versions like 10
> anyway.
>

I got myself a working build env now so I can at least verify it builds,
which it does.

With that, I'm pushing this. Let's see what the buildfarm thinks of it. And
if others end up complaining about the platform drop, but I doubt that.

Once again, apologies for the delay, and thanks for the patch!

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ 
 Work: https://www.redpill-linpro.com/ 

Re: [HACKERS] Supporting huge pages on Windows

[Michael Paquier]

On Sun, Jan 21, 2018 at 01:42:13PM +0200, Magnus Hagander wrote:
> We have traditionally supported older versions of Windows as long as people
> build from source. But perhaps I'm way overreading that and we should just
> bite the bullet, commit this patch, and declare those platforms as
> completely dead by PostgreSQL 11?

Yeah, I'd like to think that this is a good idea for HEAD. Microsoft is
pushing hard to have all its users move to newer versions like 10
anyway.
--
Michael


signature.asc
Description: PGP signature

Re: [HACKERS] Supporting huge pages on Windows

[Magnus Hagander]

 Sat, Jan 20, 2018 at 2:33 PM, Magnus Hagander  wrote:

>
>
> On Fri, Dec 1, 2017 at 5:02 AM, Michael Paquier  > wrote:
>
>> On Fri, Nov 24, 2017 at 9:28 AM, Tsunakawa, Takayuki
>>  wrote:
>> > From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com]
>> >> I hope Tsunakawa-san doesn't mind me posting another rebased version of
>> >> his patch.  The last version conflicted with the change from SGML to
>> XML
>> >> that just landed in commit 3c49c6fa.
>> >
>> > Thank you very much for keeping it fresh.  I hope the committer could
>> have some time.
>>
>> I have moved this patch to next CF. Magnus, you are registered as a
>> reviewer of this patch. Are you planning to look at it and potentially
>> commit it?
>>
>
> Apologies for the delays here. Yes, I was and am.
>
> I took a look at this patch again. I've made some small updates to
> comments etc. There was also from what I can tell one actual bug in the
> code -- a missing free() of delPrivs.
>
> The debug message for ERROR_NO_SYSTEM_RESOURCES said it turned off huge
> pages because of not enough huge pages, but AIUI it can be because of other
> resources as well. So I dropped that specific.
>
>
> However, reading through a few things -- we now use the
> API GetLargePageMinimum() unconditionally. This API appeared in Windows
> Vista and Windows 2003. That means that if we apply this patch, those are
> now our minimum versions of Windows. At least unless Microsoft backpatched
> things.
>
> This is probably equally true of some other things we do, but those are
> runtime, and we would just give an error on old platforms. If I'm thinking
> right, then direct linking to GetLargePageMinimum() will simply make it
> impossible to start a PostgreSQL backend with or without huge pages on
> previous versions, because it will give a linker error.
>
> I wonder if we need to do something similar to what we already do for
> CreateRestrictedToken() in pg_ctl.c for example. That one actually is done
> for compatibility with NT4/2000 -- CreateRestrictedToken was added in XP.
> So while we could consider getting rid of that workaround now, perhaps we
> need to put a similar one in place for this?
>
> I don't have a Windows build box around ATM, or a Windows XP, so if
> somebody could try the attached version, build a postgres and see if it
> even starts on a Windows XP machine, that would be useful input!
>
> If we can confirm/deny the situation around XP and decide what to do about
> it, I am now happy to commit the rest of this patch.
>
>
To add some more notes on this. Again, the API appears in Vista/2003.
Windows Vista went EOL (out of extended support even) in April 2017,
Windows 2003 did so in July 2015. Those are the versions that it's *in* --
obviously the versions without it are even older.

Our binaries only support Windows 2008 and up (at least the ones by EDB,
which are the ones that have a supported-version matrix documented on our
site).

We have traditionally supported older versions of Windows as long as people
build from source. But perhaps I'm way overreading that and we should just
bite the bullet, commit this patch, and declare those platforms as
completely dead by PostgreSQL 11?


-- 
 Magnus Hagander
 Me: https://www.hagander.net/ 
 Work: https://www.redpill-linpro.com/ 

Re: [HACKERS] Supporting huge pages on Windows

[Magnus Hagander]

On Fri, Dec 1, 2017 at 5:02 AM, Michael Paquier 
wrote:

> On Fri, Nov 24, 2017 at 9:28 AM, Tsunakawa, Takayuki
>  wrote:
> > From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com]
> >> I hope Tsunakawa-san doesn't mind me posting another rebased version of
> >> his patch.  The last version conflicted with the change from SGML to XML
> >> that just landed in commit 3c49c6fa.
> >
> > Thank you very much for keeping it fresh.  I hope the committer could
> have some time.
>
> I have moved this patch to next CF. Magnus, you are registered as a
> reviewer of this patch. Are you planning to look at it and potentially
> commit it?
>

Apologies for the delays here. Yes, I was and am.

I took a look at this patch again. I've made some small updates to comments
etc. There was also from what I can tell one actual bug in the code -- a
missing free() of delPrivs.

The debug message for ERROR_NO_SYSTEM_RESOURCES said it turned off huge
pages because of not enough huge pages, but AIUI it can be because of other
resources as well. So I dropped that specific.


However, reading through a few things -- we now use the
API GetLargePageMinimum() unconditionally. This API appeared in Windows
Vista and Windows 2003. That means that if we apply this patch, those are
now our minimum versions of Windows. At least unless Microsoft backpatched
things.

This is probably equally true of some other things we do, but those are
runtime, and we would just give an error on old platforms. If I'm thinking
right, then direct linking to GetLargePageMinimum() will simply make it
impossible to start a PostgreSQL backend with or without huge pages on
previous versions, because it will give a linker error.

I wonder if we need to do something similar to what we already do for
CreateRestrictedToken() in pg_ctl.c for example. That one actually is done
for compatibility with NT4/2000 -- CreateRestrictedToken was added in XP.
So while we could consider getting rid of that workaround now, perhaps we
need to put a similar one in place for this?

I don't have a Windows build box around ATM, or a Windows XP, so if
somebody could try the attached version, build a postgres and see if it
even starts on a Windows XP machine, that would be useful input!

If we can confirm/deny the situation around XP and decide what to do about
it, I am now happy to commit the rest of this patch.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ 
 Work: https://www.redpill-linpro.com/ 
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 37a61a13c8..cc156c6385 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1369,14 +1369,26 @@ include_dir 'conf.d'

 

-At present, this feature is supported only on Linux. The setting is
-ignored on other systems when set to try.
+At present, this feature is supported only on Linux and Windows. The
+setting is ignored on other systems when set to try.

 

 The use of huge pages results in smaller page tables and less CPU time
-spent on memory management, increasing performance. For more details,
-see .
+spent on memory management, increasing performance. For more details about
+using huge pages on Linux, see .
+   
+
+   
+Huge pages are known as large pages on Windows.  To use them, you need to
+assign the user right Lock Pages in Memory to the Windows user account
+that runs PostgreSQL.
+You can use Windows Group Policy tool (gpedit.msc) to assign the user right
+Lock Pages in Memory.
+To start the database server on the command prompt as a standalone process,
+not as a Windows service, the command prompt must be run as an administrator
+User Access Control (UAC) must be disabled. When the UAC is enabled, the normal
+command prompt revokes the user right Lock Pages in Memory when started.

 

diff --git a/src/backend/port/win32_shmem.c b/src/backend/port/win32_shmem.c
index 4991ed46f1..9362f1c4a4 100644
--- a/src/backend/port/win32_shmem.c
+++ b/src/backend/port/win32_shmem.c
@@ -21,6 +21,7 @@ HANDLE		UsedShmemSegID = INVALID_HANDLE_VALUE;
 void	   *UsedShmemSegAddr = NULL;
 static Size UsedShmemSegSize = 0;
 
+static bool EnableLockPagesPrivilege(int elevel);
 static void pgwin32_SharedMemoryDelete(int status, Datum shmId);
 
 /*
@@ -103,6 +104,66 @@ PGSharedMemoryIsInUse(unsigned long id1, unsigned long id2)
 	return true;
 }
 
+/*
+ * EnableLockPagesPrivilege
+ *
+ * Try to acquire SeLockMemoryPrivilege so we can use large pages.
+ */
+static bool
+EnableLockPagesPrivilege(int elevel)
+{
+	HANDLE hToken;
+	TOKEN_PRIVILEGES tp;
+	LUID luid;
+
+	if (!OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, &hToken))
+	{
+		ereport(elevel,
+(errmsg("could not enable Lock Pages in Memory user right: error code %lu", GetL

Re: [HACKERS] Supporting huge pages on Windows

[Michael Paquier]

On Fri, Nov 24, 2017 at 9:28 AM, Tsunakawa, Takayuki
 wrote:
> From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com]
>> I hope Tsunakawa-san doesn't mind me posting another rebased version of
>> his patch.  The last version conflicted with the change from SGML to XML
>> that just landed in commit 3c49c6fa.
>
> Thank you very much for keeping it fresh.  I hope the committer could have 
> some time.

I have moved this patch to next CF. Magnus, you are registered as a
reviewer of this patch. Are you planning to look at it and potentially
commit it?
-- 
Michael

RE: [HACKERS] Supporting huge pages on Windows

[Tsunakawa, Takayuki]

From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com]
> I hope Tsunakawa-san doesn't mind me posting another rebased version of
> his patch.  The last version conflicted with the change from SGML to XML
> that just landed in commit 3c49c6fa.

Thank you very much for keeping it fresh.  I hope the committer could have some 
time.

Regards
Takayuki Tsunakawa

Re: [HACKERS] Supporting huge pages on Windows

[Thomas Munro]

On Thu, Sep 14, 2017 at 12:32 AM, Magnus Hagander  wrote:
> It's my plan to get to this patch during this commitfest. I've been
> travelling for open and some 24/7 work so far, but hope to get CFing soon.

I hope Tsunakawa-san doesn't mind me posting another rebased version
of his patch.  The last version conflicted with the change from SGML
to XML that just landed in commit 3c49c6fa.

-- 
Thomas Munro
http://www.enterprisedb.com


win_large_pages_v15.patch
Description: Binary data