Re: [HACKERS] Supporting huge pages on Windows

2017-09-13 Thread Tsunakawa, Takayuki 
From: pgsql-hackers-ow...@postgresql.org
> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Ashutosh Sharma
> I have once again tested the latest patch (v14 patch) on Windows and the
> results looked fine to me. Basically I have repeated the test cases which
> I had done earlier on v8 patch. For more details, on the tests that i have
> re-executed, please refer to - [1]. Thanks.

Thanks so much.  I'm relieved to know that the patch still works.

Regards
Takayuki Tsunakawa


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-09-13 Thread Magnus Hagander 
On Wed, Sep 13, 2017 at 3:41 AM, Tsunakawa, Takayuki <
tsunakawa.ta...@jp.fujitsu.com> wrote:

> Hi Thomas, Magnus
>
> From: pgsql-hackers-ow...@postgresql.org
> > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Thomas Munro
> > Since it only conflicts with c7b8998e because of pgindent whitespace
> > movement, I applied it with "patch -p1 --ignore-whitespace" and created
> > a new patch.  See attached.
>
> Thanks, Thomas.  I've added your name in the CF entry so that your name
> will also be listed on the release note, because my patch is originally
> based on your initial try.  Please remove your name just in case you mind
> it.  BTW, your auto-reviewer looks very convenient.  Thank you again for
> your great work.
>
> Magnus, it would be grateful if you could review and commit the patch
> while your memory is relatively fresh.
>
> I've been in a situation which keeps me from doing development recently,
> but I think I can gradually rejoin the community activity soon.
>
>
Hi!

It's my plan to get to this patch during this commitfest. I've been
travelling for open and some 24/7 work so far, but hope to get CFing soon.



-- 
 Magnus Hagander
 Me: https://www.hagander.net/ 
 Work: https://www.redpill-linpro.com/

Re: [HACKERS] Supporting huge pages on Windows

2017-09-13 Thread Ashutosh Sharma 
On Wed, Sep 13, 2017 at 7:11 AM, Tsunakawa, Takayuki
 wrote:
> Hi Thomas, Magnus
>
> From: pgsql-hackers-ow...@postgresql.org
>> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Thomas Munro
>> Since it only conflicts with c7b8998e because of pgindent whitespace
>> movement, I applied it with "patch -p1 --ignore-whitespace" and created
>> a new patch.  See attached.
>
> Thanks, Thomas.  I've added your name in the CF entry so that your name will 
> also be listed on the release note, because my patch is originally based on 
> your initial try.  Please remove your name just in case you mind it.  BTW, 
> your auto-reviewer looks very convenient.  Thank you again for your great 
> work.
>
> Magnus, it would be grateful if you could review and commit the patch while 
> your memory is relatively fresh.
>
> I've been in a situation which keeps me from doing development recently, but 
> I think I can gradually rejoin the community activity soon.
>

I have once again tested the latest patch (v14 patch) on Windows and
the results looked fine to me. Basically I have repeated the test
cases which I had done earlier on v8 patch. For more details, on the
tests that i have re-executed, please refer to - [1]. Thanks.

[1]-   
https://www.postgresql.org/message-id/CAE9k0Pkz%2BtOiPmx2LrVePM7cZydTLNbQ6R3GqgeivurfsXyZ5w%40mail.gmail.com

--
With Regards,
Ashutosh Sharma
EnterpriseDB:http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-09-12 Thread Tsunakawa, Takayuki 
Hi Thomas, Magnus

From: pgsql-hackers-ow...@postgresql.org
> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Thomas Munro
> Since it only conflicts with c7b8998e because of pgindent whitespace
> movement, I applied it with "patch -p1 --ignore-whitespace" and created
> a new patch.  See attached.

Thanks, Thomas.  I've added your name in the CF entry so that your name will 
also be listed on the release note, because my patch is originally based on 
your initial try.  Please remove your name just in case you mind it.  BTW, your 
auto-reviewer looks very convenient.  Thank you again for your great work.

Magnus, it would be grateful if you could review and commit the patch while 
your memory is relatively fresh.

I've been in a situation which keeps me from doing development recently, but I 
think I can gradually rejoin the community activity soon.

Regards
Takayuki Tsunakawa



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-09-12 Thread Thomas Munro 
On Thu, Aug 17, 2017 at 2:11 PM, Thomas Munro
 wrote:
> On Wed, Apr 12, 2017 at 7:08 PM, Tsunakawa, Takayuki
>  wrote:
>> Oh, I got it now.  Thanks.  The revised patch is attached.  The only 
>> modified file is pg_ctl.c.  The patch worked as expected.
>>
>> It is regrettable that I could not make it in time for PG 10, but I'd 
>> appreciate it if you could review and commit this patch early in PG 11 while 
>> our memory is fresh.  Thank you for your patience.  I'll create an entry in 
>> the next CF soon.
>
> win_large_pages_v13.patch seems to be the latest version posted, but
> it no longer applies.  Could you please post a rebased version?

Since it only conflicts with c7b8998e because of pgindent whitespace
movement, I applied it with "patch -p1 --ignore-whitespace" and
created a new patch.  See attached.

-- 
Thomas Munro
http://www.enterprisedb.com


win_large_pages_v14.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-08-16 Thread Thomas Munro 
On Wed, Apr 12, 2017 at 7:08 PM, Tsunakawa, Takayuki
 wrote:
> Oh, I got it now.  Thanks.  The revised patch is attached.  The only modified 
> file is pg_ctl.c.  The patch worked as expected.
>
> It is regrettable that I could not make it in time for PG 10, but I'd 
> appreciate it if you could review and commit this patch early in PG 11 while 
> our memory is fresh.  Thank you for your patience.  I'll create an entry in 
> the next CF soon.

Hi Takayuki-san,

win_large_pages_v13.patch seems to be the latest version posted, but
it no longer applies.  Could you please post a rebased version?

Thanks!

-- 
Thomas Munro
http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-12 Thread Tsunakawa, Takayuki 
Hello, Magnus
cc: Andres

From: pgsql-hackers-ow...@postgresql.org
> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Magnus Hagander
> I think what you'd need to do is enumerate what privileges the user has
> *before* calling CreateRestrictedToken(), using GetTokenInformation().
> And then pass those into PrivilegesToDelete (except for
> SeChangeNotifyPrivilege) in the call to CreateRestrictedToken(), instead
> of using DISABLE_MAX_PRIVILEGE. (and add the privilege needed for huge pages
> before you start that whole process -- that needs to be added in the token
> used *before* we create the restricted one).
> 
> At least that's my guess from reading the docs and trying to remember :)

Oh, I got it now.  Thanks.  The revised patch is attached.  The only modified 
file is pg_ctl.c.  The patch worked as expected.

It is regrettable that I could not make it in time for PG 10, but I'd 
appreciate it if you could review and commit this patch early in PG 11 while 
our memory is fresh.  Thank you for your patience.  I'll create an entry in the 
next CF soon.

Regards
Takayuki Tsunakawa





win_large_pages_v13.patch
Description: win_large_pages_v13.patch

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-07 Thread Andres Freund 
On 2017-04-07 13:57:07 +0200, Magnus Hagander wrote:
> On Wed, Apr 5, 2017 at 9:15 AM, Tsunakawa, Takayuki <
> tsunakawa.ta...@jp.fujitsu.com> wrote:
> 
> > From: pgsql-hackers-ow...@postgresql.org
> > > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Andres Freund
> > > As I asked before, why can't we delete all privs and add the explicitly
> > > needed once back (using AdjustTokenPrivileges)?
> >
> > I tried it with pg_ctl.c attached to an earlier mail today, i.e. delete
> > all privs with CreateRestrictedToken(DISABLE_ALL_PRIVILEGE) and enable
> > Lock Pages in Memory with AdjustTokenPrivileges().  But it didn't work;
> > AdjustTokenPrivileges() failed to enable the priv.  It's probably that
> > CreateRestrictedToken() deletes (unassigns?) the privs from the access
> > token, so subsequent AdjustTokenPrivileges() can no longer enable the priv.
> >
> >
> Once you have used CreateRestrictedToken(), you can no longer add
> *anything* to it. It's not just removed privileges, there's a special flag
> on the token that says it's restricted (can be checked with
> IsTokenRestricted()).

:/


> I think what you'd need to do is enumerate what privileges the user has
> *before* calling CreateRestrictedToken(), using GetTokenInformation(). And
> then pass those into PrivilegesToDelete (except for
> SeChangeNotifyPrivilege) in the call to CreateRestrictedToken(), instead of
> using DISABLE_MAX_PRIVILEGE. (and add the privilege needed for huge pages
> before you start that whole process -- that needs to be added in the token
> used *before* we create the restricted one).
> 
> At least that's my guess from reading the docs and trying to remember :)

Yea, seems that way.  Therefore I propose returning this patch with
feedback.

Andres


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-07 Thread Magnus Hagander 
On Wed, Apr 5, 2017 at 9:15 AM, Tsunakawa, Takayuki <
tsunakawa.ta...@jp.fujitsu.com> wrote:

> From: pgsql-hackers-ow...@postgresql.org
> > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Andres Freund
> > As I asked before, why can't we delete all privs and add the explicitly
> > needed once back (using AdjustTokenPrivileges)?
>
> I tried it with pg_ctl.c attached to an earlier mail today, i.e. delete
> all privs with CreateRestrictedToken(DISABLE_ALL_PRIVILEGE) and enable
> Lock Pages in Memory with AdjustTokenPrivileges().  But it didn't work;
> AdjustTokenPrivileges() failed to enable the priv.  It's probably that
> CreateRestrictedToken() deletes (unassigns?) the privs from the access
> token, so subsequent AdjustTokenPrivileges() can no longer enable the priv.
>
>
Once you have used CreateRestrictedToken(), you can no longer add
*anything* to it. It's not just removed privileges, there's a special flag
on the token that says it's restricted (can be checked with
IsTokenRestricted()).

I think what you'd need to do is enumerate what privileges the user has
*before* calling CreateRestrictedToken(), using GetTokenInformation(). And
then pass those into PrivilegesToDelete (except for
SeChangeNotifyPrivilege) in the call to CreateRestrictedToken(), instead of
using DISABLE_MAX_PRIVILEGE. (and add the privilege needed for huge pages
before you start that whole process -- that needs to be added in the token
used *before* we create the restricted one).

At least that's my guess from reading the docs and trying to remember :)

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [HACKERS] Supporting huge pages on Windows

2017-04-05 Thread Tsunakawa, Takayuki 
From: pgsql-hackers-ow...@postgresql.org
> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Andres Freund
> As I asked before, why can't we delete all privs and add the explicitly
> needed once back (using AdjustTokenPrivileges)?

I tried it with pg_ctl.c attached to an earlier mail today, i.e. delete all 
privs with CreateRestrictedToken(DISABLE_ALL_PRIVILEGE) and enable Lock Pages 
in Memory with AdjustTokenPrivileges().  But it didn't work; 
AdjustTokenPrivileges() failed to enable the priv.  It's probably that 
CreateRestrictedToken() deletes (unassigns?) the privs from the access token, 
so subsequent AdjustTokenPrivileges() can no longer enable the priv.

Regards
Takayuki Tsunakawa




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-05 Thread Andres Freund 
On 2017-04-05 04:25:41 +, Tsunakawa, Takayuki wrote:
> From: Craig Ringer [mailto:craig.rin...@2ndquadrant.com]
> > On 5 April 2017 at 10:37, Tsunakawa, Takayuki
> >  wrote:
> > 
> > OTOH, I tried again to leave the DISABLE_MAX_PRIVILEGE as is and add Lock
> > Pages in Memory, using the attached pg_ctl.c.  Please see
> > EnableLockPagesPrivilege() and its call site.  But pg_ctl -w start fails
> > emitting the following message:
> > 
> > That won't work. You'd have to pass 0 to the flags of CreateRestrictedToken
> > and instead supply a PrivilegesToDelete array.
> > You'd probably GetTokenInformation and AND with a mask of ones you wanted
> > to retain.
> 
> Uh, that's inconvenient.  We can't determine what privileges to delete, and 
> we must be aware of new privileges added in the future version of Windows.
> 
> Then, I have to say the last patch (v12) is the final answer.

As I asked before, why can't we delete all privs and add the explicitly
needed once back (using AdjustTokenPrivileges)?

- Andres


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-04 Thread Tsunakawa, Takayuki 
From: Craig Ringer [mailto:craig.rin...@2ndquadrant.com]
> On 5 April 2017 at 10:37, Tsunakawa, Takayuki
>  wrote:
> 
> OTOH, I tried again to leave the DISABLE_MAX_PRIVILEGE as is and add Lock
> Pages in Memory, using the attached pg_ctl.c.  Please see
> EnableLockPagesPrivilege() and its call site.  But pg_ctl -w start fails
> emitting the following message:
> 
> That won't work. You'd have to pass 0 to the flags of CreateRestrictedToken
> and instead supply a PrivilegesToDelete array.
> You'd probably GetTokenInformation and AND with a mask of ones you wanted
> to retain.

Uh, that's inconvenient.  We can't determine what privileges to delete, and we 
must be aware of new privileges added in the future version of Windows.

Then, I have to say the last patch (v12) is the final answer.

Regards
Takayuki Tsunakawa


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-04 Thread Craig Ringer 
On 5 April 2017 at 10:37, Tsunakawa, Takayuki
 wrote:

> Good point!  And I said earlier in this thread, I think managing privileges 
> (adding/revoking privileges from the user account) is the DBA's or sysadmin's 
> duty, and PG's removing all privileges feels overkill.

I think it's a sensible alternative to refusing to run as a highly
privileged role, which is what we used to do IIRC.

> OTOH, I tried again to leave the DISABLE_MAX_PRIVILEGE as is and add Lock 
> Pages in Memory, using the attached pg_ctl.c.  Please see 
> EnableLockPagesPrivilege() and its call site.  But pg_ctl -w start fails 
> emitting the following message:

That won't work. You'd have to pass 0 to the flags of
CreateRestrictedToken and instead supply a PrivilegesToDelete array.
You'd probably GetTokenInformation and AND with a mask of ones you
wanted to retain.

-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-04 Thread Tsunakawa, Takayuki 
From: Craig Ringer [mailto:craig.rin...@2ndquadrant.com]
> TBH, anyone who cares about security and runs Win7 or Win2k8 or newer should
> be using virtual service accounts and managed service accounts.
> 
> https://technet.microsoft.com/en-us/library/dd548356
> 
> 
> Those are more like Unix service accounts. Notably they don't need a password,
> getting rid of some of the management pain that led us to abandon the
> 'postgres' system user on Windows.
> 
> Now that older platforms are EoL and even the oldest that added this feature
> are also near EoL or in extended maintenance, I think installers should
> switch to these by default instead of using NETWORK SERVICE.
> 
> Then the issue of priv dropping would be a lesser concern anyway.

Good point!  And I said earlier in this thread, I think managing privileges 
(adding/revoking privileges from the user account) is the DBA's or sysadmin's 
duty, and PG's removing all privileges feels overkill.

OTOH, I tried again to leave the DISABLE_MAX_PRIVILEGE as is and add Lock Pages 
in Memory, using the attached pg_ctl.c.  Please see EnableLockPagesPrivilege() 
and its call site.  But pg_ctl -w start fails emitting the following message:

error code 1300
waiting for server to startFATAL:  could not enable "Lock pages in memory" 
privilege
HINT:  Assign "Lock pages in memory" privilege to the Windows user account 
which runs PostgreSQL.
LOG:  database system is shut down
 stopped waiting
pg_ctl: could not start server
Examine the log output.

error code 1300 is ERROR_NOT_ALL_ASSIGNED, which means AdjustTokenPrivileges() 
cound not enable Lock Pages in Memory privilege.  It seems that the privilege 
cannot be enabled once it was removed with 
CreateRestrictedToken(DISABLE_MAX_PRIVILEGE).

Regards
Takayuki Tsunakawa


/*-
 *
 * pg_ctl --- start/stops/restarts the PostgreSQL server
 *
 * Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
 *
 * src/bin/pg_ctl/pg_ctl.c
 *
 *-
 */

#ifdef WIN32
/*
 * Need this to get defines for restricted tokens and jobs. And it
 * has to be set before any header from the Win32 API is loaded.
 */
#define _WIN32_WINNT 0x0501
#endif

#include "postgres_fe.h"

#include "libpq-fe.h"
#include "pqexpbuffer.h"

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

#ifdef HAVE_SYS_RESOURCE_H
#include 
#include 
#endif

#include "getopt_long.h"
#include "miscadmin.h"

/* PID can be negative for standalone backend */
typedef long pgpid_t;


typedef enum
{
SMART_MODE,
FAST_MODE,
IMMEDIATE_MODE
} ShutdownMode;


typedef enum
{
NO_COMMAND = 0,
INIT_COMMAND,
START_COMMAND,
STOP_COMMAND,
RESTART_COMMAND,
RELOAD_COMMAND,
STATUS_COMMAND,
PROMOTE_COMMAND,
KILL_COMMAND,
REGISTER_COMMAND,
UNREGISTER_COMMAND,
RUN_AS_SERVICE_COMMAND
} CtlCommand;

#define DEFAULT_WAIT60

static bool do_wait = false;
static bool del_service_rid = false;
static bool wait_set = false;
static int  wait_seconds = DEFAULT_WAIT;
static bool wait_seconds_arg = false;
static bool silent_mode = false;
static ShutdownMode shutdown_mode = FAST_MODE;
static int  sig = SIGINT;   /* default */
static CtlCommand ctl_command = NO_COMMAND;
static char *pg_data = NULL;
static char *pg_config = NULL;
static char *pgdata_opt = NULL;
static char *post_opts = NULL;
static const char *progname;
static char *log_file = NULL;
static char *exec_path = NULL;
static char *event_source = NULL;
static char *register_servicename = "PostgreSQL";   /* FIXME: + 
version ID? */
static char *register_username = NULL;
static char *register_password = NULL;
static char *argv0 = NULL;
static bool allow_core_files = false;
static time_t start_time;

static char postopts_file[MAXPGPATH];
static char version_file[MAXPGPATH];
static char pid_file[MAXPGPATH];
static char backup_file[MAXPGPATH];
static char recovery_file[MAXPGPATH];
static char promote_file[MAXPGPATH];

#ifdef WIN32
static DWORD pgctl_start_type = SERVICE_AUTO_START;
static SERVICE_STATUS status;
static SERVICE_STATUS_HANDLE hStatus = (SERVICE_STATUS_HANDLE) 0;
static HANDLE shutdownHandles[2];
static pid_t postmasterPID = -1;

#define shutdownEvent shutdownHandles[0]
#define postmasterProcess shutdownHandles[1]
#endif


static void write_stderr(const char *fmt,...) pg_attribute_printf(1, 2);
static void do_advice(void);
static void do_help(void);
static void set_mode(char *modeopt);
static void set_sig(char *signame);
static void do_init(void);
static void do_start(void);
static void do_stop(void);
static void do_restart(void);
static void do_reload(void);
static void do_status(void);
static void do_promote(void);
static void do_kill(pgpid_t pid);
static void print_msg(const char *msg);
static void

Re: [HACKERS] Supporting huge pages on Windows

2017-04-04 Thread Craig Ringer 
On 4 Apr. 2017 14:22, "Andres Freund"  wrote:

On 2017-01-05 03:12:09 +, Tsunakawa, Takayuki wrote:
> From: pgsql-hackers-ow...@postgresql.org
> > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Magnus Hagander
> > For the pg_ctl changes, we're going from removing all privilieges from
the
> > token, to removing none. Are there any other privileges that we should
be
> > worried about? I think you may be correct in that it's overkill to do
it,
> > but I think we need some more specifics to decide that.
>
> This page lists the privileges.  Is there anyhing you are concerned about?
>
> https://msdn.microsoft.com/ja-jp/library/windows/desktop/
bb530716(v=vs.85).aspx

Aren't like nearly all of them a concern?  We gone from having some
containment (not being to create users, shut the system down, ...), to
none.  I do however think there's a fair argument to be made that other
platforms do not have a similar containment (no root, but sudo etc is
still possible), and that the containment isn't super strong.


TBH, anyone who cares about security and runs Win7 or Win2k8 or newer
should be using virtual service accounts and managed service accounts.

https://technet.microsoft.com/en-us/library/dd548356

Those are more like Unix service accounts. Notably they don't need a
password, getting rid of some of the management pain that led us to abandon
the 'postgres' system user on Windows.

Now that older platforms are EoL and even the oldest that added this
feature are also near EoL or in extended maintenance, I think installers
should switch to these by default instead of using NETWORK SERVICE.

Then the issue of priv dropping would be a lesser concern anyway.

Re: [HACKERS] Supporting huge pages on Windows

2017-04-04 Thread Andres Freund 
On 2017-01-05 03:12:09 +, Tsunakawa, Takayuki wrote:
> From: pgsql-hackers-ow...@postgresql.org
> > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Magnus Hagander
> > For the pg_ctl changes, we're going from removing all privilieges from the
> > token, to removing none. Are there any other privileges that we should be
> > worried about? I think you may be correct in that it's overkill to do it,
> > but I think we need some more specifics to decide that.
> 
> This page lists the privileges.  Is there anyhing you are concerned about?
> 
> https://msdn.microsoft.com/ja-jp/library/windows/desktop/bb530716(v=vs.85).aspx

Aren't like nearly all of them a concern?  We gone from having some
containment (not being to create users, shut the system down, ...), to
none.  I do however think there's a fair argument to be made that other
platforms do not have a similar containment (no root, but sudo etc is
still possible), and that the containment isn't super strong.

Can't we, to reduce the size of the behavioural change, just use
AdjustTokenPrivileges() to re-add the privileges we want?

Regards,

Andres


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-04 Thread Tsunakawa, Takayuki 
From: pgsql-hackers-ow...@postgresql.org
> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Andres Freund
> I don't think the errdetail is quite right - OpenProcessToken isn't really
> a syscall, is it? But then it's a common pattern already in wind32_shmem.c...

Yes, "Win32 API function" would be correct, but I followed the existing code...


> > +errdetail("Failed system call was %s.",
> > +"LookupPrivilegeValue")));
> 
> Other places in the file actually log the arguments to the function...

The only place is CreateFileMapping.  Other places (DuplicateHandle and 
MapViewOfFileEx) don't log arguments.  I guess the original developer thought 
that size and name arguments to CreateFileMapping() might be useful for 
troubleshooting.


> Wonder if we should quote "Lock Pages in Memory" or add dashes, to make
> sure it's clear that that's the right?

I saw several Microsoft pages, including a page someone introduced me here, and 
they didn't quote the user right.  I'm comfortable with the current code, but I 
don't mind if the committer adds some quotation.


> > +   flProtect = PAGE_READWRITE | SEC_COMMIT |
> SEC_LARGE_PAGES;
> 
> Why don't we just add the relevant flag, instead of overwriting the previous
> contents?

I don't have strong opinion on this...

> Uh - isn't that a behavioural change?  Was this discussed?

Yes, I described this in the first mail.  Magnus asked about this later and I 
replied.

Regards
Takayuki Tsunakawa



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-03 Thread Andres Freund 
On 2017-04-03 04:56:45 +, Tsunakawa, Takayuki wrote:
> +/*
> + * EnableLockPagesPrivilege
> + *
> + * Try to acquire SeLockMemoryPrivilege so we can use large pages.
> + */
> +static bool
> +EnableLockPagesPrivilege(int elevel)
> +{
> + HANDLE hToken;
> + TOKEN_PRIVILEGES tp;
> + LUID luid;
> +
> + if (!OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | 
> TOKEN_QUERY, ))
> + {
> + ereport(elevel,
> + (errmsg("could not enable Lock Pages in Memory 
> user right: error code %lu", GetLastError()),
> +  errdetail("Failed system call was %s.", 
> "OpenProcessToken")));
> + return FALSE;
> + }

I don't think the errdetail is quite right - OpenProcessToken isn't
really a syscall, is it? But then it's a common pattern already in
wind32_shmem.c...


> + if (!LookupPrivilegeValue(NULL, SE_LOCK_MEMORY_NAME, ))
> + {
> + ereport(elevel,
> + (errmsg("could not enable Lock Pages in Memory 
> user right: error code %lu", GetLastError()),
> +  errdetail("Failed system call was %s.", 
> "LookupPrivilegeValue")));

Other places in the file actually log the arguments to the function...

Wonder if we should quote "Lock Pages in Memory" or add dashes, to make
sure it's clear that that's the right?


> + if (huge_pages == HUGE_PAGES_ON || huge_pages == HUGE_PAGES_TRY)
> + {
> + /* Does the processor support large pages? */
> + largePageSize = GetLargePageMinimum();
> + if (largePageSize == 0)
> + {
> + ereport(huge_pages == HUGE_PAGES_ON ? FATAL : DEBUG1,
> + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> +  errmsg("the processor does not support 
> large pages")));
> + ereport(DEBUG1,
> + (errmsg("disabling huge pages")));
> + }
> + else if (!EnableLockPagesPrivilege(huge_pages == HUGE_PAGES_ON 
> ? FATAL : DEBUG1))
> + {
> + ereport(DEBUG1,
> + (errmsg("disabling huge pages")));
> + }
> + else
> + {
> + /* Huge pages available and privilege enabled, so turn 
> on */
> + flProtect = PAGE_READWRITE | SEC_COMMIT | 
> SEC_LARGE_PAGES;

Why don't we just add the relevant flag, instead of overwriting the
previous contents?




> diff --git a/src/bin/pg_ctl/pg_ctl.c b/src/bin/pg_ctl/pg_ctl.c
> index c63819b..2358ed0 100644
> --- a/src/bin/pg_ctl/pg_ctl.c
> +++ b/src/bin/pg_ctl/pg_ctl.c
> @@ -1708,11 +1708,6 @@ typedef BOOL (WINAPI * __SetInformationJobObject) 
> (HANDLE, JOBOBJECTINFOCLASS, L
>  typedef BOOL (WINAPI * __AssignProcessToJobObject) (HANDLE, HANDLE);
>  typedef BOOL (WINAPI * __QueryInformationJobObject) (HANDLE, 
> JOBOBJECTINFOCLASS, LPVOID, DWORD, LPDWORD);
>  
> -/* Windows API define missing from some versions of MingW headers */
> -#ifndef  DISABLE_MAX_PRIVILEGE
> -#define DISABLE_MAX_PRIVILEGE0x1
> -#endif
> -

>  /*
>   * Create a restricted token, a job object sandbox, and execute the specified
>   * process with it.
> @@ -1794,7 +1789,7 @@ CreateRestrictedProcess(char *cmd, PROCESS_INFORMATION 
> *processInfo, bool as_ser
>   }
>  
>   b = _CreateRestrictedToken(origToken,
> -
> DISABLE_MAX_PRIVILEGE,
> +0,
>  sizeof(dropSids) / 
> sizeof(dropSids[0]),
>  dropSids,
>  0, NULL,

Uh - isn't that a behavioural change?  Was this discussed?

Greetings,

Andres Freund


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-02 Thread Tsunakawa, Takayuki 
From: Michael Paquier [mailto:michael.paqu...@gmail.com]
> Amit is right, you had better use %zu as we do in all the other places of
> the code dealing with Size variables that are printed. When compiling with
> Visual Studio, Postgres falls back to the implementation of sprintf in
> src/port/snprintf.c so that's not something to worry about.

Thanks, fixed.

Regards
Takayuki Tsunakawa



win_large_pages_v12.patch
Description: win_large_pages_v12.patch

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-02 Thread Michael Paquier 
On Mon, Apr 3, 2017 at 1:12 PM, Tsunakawa, Takayuki
 wrote:
> From: pgsql-hackers-ow...@postgresql.org
>> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Amit Kapila
>> You should use %zu instead of %llu to print Size type of variable.
>
> I did so at first, but it didn't work with Visual Studio 2010 at hand.  It 
> doesn't support %z which is added in C99.
>
> But "I" (capital i) instead of "ll" seems appropriate as the following page 
> says.  I chose this.
>
> https://en.wikipedia.org/wiki/Printf_format_string
>
> I For signed integer types, causes printf to expect ptrdiff_t-sized integer 
> argument; for unsigned integer types, causes printf to expect size_t-sized 
> integer argument. Commonly found in Win32/Win64 platforms.

+elog(DEBUG1, "CreateFileMapping(%Iu) with
SEC_LARGE_PAGES failed "
+ "due to insufficient large pages, huge pages disabled",
+ size);
Amit is right, you had better use %zu as we do in all the other places
of the code dealing with Size variables that are printed. When
compiling with Visual Studio, Postgres falls back to the
implementation of sprintf in src/port/snprintf.c so that's not
something to worry about.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-04-02 Thread Tsunakawa, Takayuki 
From: pgsql-hackers-ow...@postgresql.org
> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Amit Kapila
> You should use %zu instead of %llu to print Size type of variable.

I did so at first, but it didn't work with Visual Studio 2010 at hand.  It 
doesn't support %z which is added in C99.

But "I" (capital i) instead of "ll" seems appropriate as the following page 
says.  I chose this.

https://en.wikipedia.org/wiki/Printf_format_string

I For signed integer types, causes printf to expect ptrdiff_t-sized integer 
argument; for unsigned integer types, causes printf to expect size_t-sized 
integer argument. Commonly found in Win32/Win64 platforms. 

Regards
Takayuki Tsunakawa




win_large_pages_v11.patch
Description: win_large_pages_v11.patch

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-31 Thread Amit Kapila 
On Fri, Mar 31, 2017 at 8:05 AM, Tsunakawa, Takayuki
 wrote:
> From: Amit Kapila [mailto:amit.kapil...@gmail.com]
>> The latest patch looks good to me apart from one Debug message, so I have
>> marked it as Ready For Committer.
>
> Thank you so much!
>
>
>> + ereport(DEBUG1,
>> + (errmsg("disabling huge pages")));
>>
>> I think this should be similar to what we display in sysv_shmem.c as below:
>>
>> elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
>> allocsize);
>
> I understood you suggested this to make the reason clear for disabling huge 
> pages.  OK, done as follows.
>
> +   elog(DEBUG1, "CreateFileMapping(%llu) with SEC_LARGE_PAGES 
> failed "
> +"due to insufficient large pages, huge pages disabled",
> +size);
>

You should use %zu instead of %llu to print Size type of variable.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-30 Thread Tsunakawa, Takayuki 
From: Amit Kapila [mailto:amit.kapil...@gmail.com]
> The latest patch looks good to me apart from one Debug message, so I have
> marked it as Ready For Committer.

Thank you so much!


> + ereport(DEBUG1,
> + (errmsg("disabling huge pages")));
> 
> I think this should be similar to what we display in sysv_shmem.c as below:
> 
> elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
> allocsize);

I understood you suggested this to make the reason clear for disabling huge 
pages.  OK, done as follows.

+   elog(DEBUG1, "CreateFileMapping(%llu) with SEC_LARGE_PAGES 
failed "
+"due to insufficient large pages, huge pages disabled",
+size);

I hope this will be committed soon.


Regards
Takayuki Tsunakawa



win_large_pages_v10.patch
Description: win_large_pages_v10.patch

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-30 Thread Amit Kapila 
On Thu, Mar 9, 2017 at 7:06 AM, Tsunakawa, Takayuki
 wrote:
> From: pgsql-hackers-ow...@postgresql.org
>> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Ashutosh Sharma
>> To start with, I ran the regression test-suite and didn't find any failures.
>> But, then I am not sure if huge_pages are getting used or not. However,
>> upon checking the settings for huge_pages and I found it as 'on'. I am
>> assuming, if huge pages is not being used due to shortage of large pages,
>> it should have fallen back to non-huge pages.
>
> You are right, the server falls back to non-huge pages when the large pages 
> run short.
>

The latest patch looks good to me apart from one Debug message, so I
have marked it as Ready For Committer.

+ ereport(DEBUG1,
+ (errmsg("disabling huge pages")));

I think this should be similar to what we display in sysv_shmem.c as below:

elog(DEBUG1, "mmap(%zu) with MAP_HUGETLB failed, huge pages disabled: %m",
allocsize);


-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-28 Thread Tsunakawa, Takayuki 
From: pgsql-hackers-ow...@postgresql.org
> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of David Steele
> It's not clear to me what state this patch should be in.  Is there more
> review that needs to be done or is it ready for a committer?

I would most appreciate it if Magnus could review the code, because he gave me 
the most critical comment that the value of huge_page variable should not be 
changed on the fly, and I did so.  I hope that will be the final review.

Regards
Takayuki Tsunakawa



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-28 Thread David Steele 

On 3/24/17 12:51 PM, Ashutosh Sharma wrote:

Hi,

On Fri, Mar 24, 2017 at 9:00 PM, David Steele  wrote:

Hi Ashutosh,

On 3/22/17 8:52 AM, Amit Kapila wrote:


On Wed, Mar 22, 2017 at 12:07 AM, David Steele 
wrote:



Amit, Magnus, you are signed up as reviewers for this patch.  Do you know
when you'll have a chance to take a look?



I have provided my feedback and I could not test it on my machine.  I
think Ashutosh Sharma has tested it.  I can give it another look, but
not sure if it helps.



I know you are not signed up as a reviewer, but have you take a look at this
patch?


Well, I had a quick look into the patch just because I wanted the test
it as I am having windows setup. But, I never looked into the patch
from reviewer's perspective. The intention was to reverify the test
results shared by Tsunawaka.


Thanks, Ashutosh.

It's not clear to me what state this patch should be in.  Is there more 
review that needs to be done or is it ready for a committer?


--
-David
da...@pgmasters.net


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-24 Thread Ashutosh Sharma 
Hi,

On Fri, Mar 24, 2017 at 9:00 PM, David Steele  wrote:
> Hi Ashutosh,
>
> On 3/22/17 8:52 AM, Amit Kapila wrote:
>>
>> On Wed, Mar 22, 2017 at 12:07 AM, David Steele 
>> wrote:
>>>
>>>
>>> Amit, Magnus, you are signed up as reviewers for this patch.  Do you know
>>> when you'll have a chance to take a look?
>>>
>>
>> I have provided my feedback and I could not test it on my machine.  I
>> think Ashutosh Sharma has tested it.  I can give it another look, but
>> not sure if it helps.
>
>
> I know you are not signed up as a reviewer, but have you take a look at this
> patch?

Well, I had a quick look into the patch just because I wanted the test
it as I am having windows setup. But, I never looked into the patch
from reviewer's perspective. The intention was to reverify the test
results shared by Tsunawaka.

--
With Regards,
Ashutosh Sharma
EnterpriseDB:http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-24 Thread David Steele 

Hi Ashutosh,

On 3/22/17 8:52 AM, Amit Kapila wrote:

On Wed, Mar 22, 2017 at 12:07 AM, David Steele  wrote:


Amit, Magnus, you are signed up as reviewers for this patch.  Do you know
when you'll have a chance to take a look?



I have provided my feedback and I could not test it on my machine.  I
think Ashutosh Sharma has tested it.  I can give it another look, but
not sure if it helps.


I know you are not signed up as a reviewer, but have you take a look at 
this patch?


Thanks,
--
-David
da...@pgmasters.net


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-22 Thread Amit Kapila 
On Wed, Mar 22, 2017 at 12:07 AM, David Steele  wrote:
> On 3/8/17 8:36 PM, Tsunakawa, Takayuki wrote:
>>
>> From: pgsql-hackers-ow...@postgresql.org
>>>
>>> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Ashutosh Sharma
>>> To start with, I ran the regression test-suite and didn't find any
>>> failures.
>>> But, then I am not sure if huge_pages are getting used or not. However,
>>> upon checking the settings for huge_pages and I found it as 'on'. I am
>>> assuming, if huge pages is not being used due to shortage of large pages,
>>> it should have fallen back to non-huge pages.
>>
>>
>> You are right, the server falls back to non-huge pages when the large
>> pages run short.
>>
>>> I also ran the pgbench tests on read-only workload and here are the
>>> results
>>> I got.
>>>
>>> pgbench -c 4 -j 4 - T 600 bench
>>>
>>> huge_pages=on, TPS = 21120.768085
>>> huge_pages=off, TPS = 20606.288995
>>
>>
>> Thanks.  It's about 2% improvement, which is the same as what I got.
>>
>>
>> From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com]
>>>
>>> The line beginning 'Huge pages are known as...' has been accidentally
>>> duplicated.
>>
>>
>> Oops, how careless I was.  Fixed.  As Ashutosh referred, I added a very
>> simple suggestion to use Windows Group Policy tool.
>
>
> Amit, Magnus, you are signed up as reviewers for this patch.  Do you know
> when you'll have a chance to take a look?
>

I have provided my feedback and I could not test it on my machine.  I
think Ashutosh Sharma has tested it.  I can give it another look, but
not sure if it helps.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-21 Thread David Steele 

On 3/8/17 8:36 PM, Tsunakawa, Takayuki wrote:

From: pgsql-hackers-ow...@postgresql.org

[mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Ashutosh Sharma
To start with, I ran the regression test-suite and didn't find any failures.
But, then I am not sure if huge_pages are getting used or not. However,
upon checking the settings for huge_pages and I found it as 'on'. I am
assuming, if huge pages is not being used due to shortage of large pages,
it should have fallen back to non-huge pages.


You are right, the server falls back to non-huge pages when the large pages run 
short.


I also ran the pgbench tests on read-only workload and here are the results
I got.

pgbench -c 4 -j 4 - T 600 bench

huge_pages=on, TPS = 21120.768085
huge_pages=off, TPS = 20606.288995


Thanks.  It's about 2% improvement, which is the same as what I got.


From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com]

The line beginning 'Huge pages are known as...' has been accidentally
duplicated.


Oops, how careless I was.  Fixed.  As Ashutosh referred, I added a very simple 
suggestion to use Windows Group Policy tool.


Amit, Magnus, you are signed up as reviewers for this patch.  Do you 
know when you'll have a chance to take a look?


Thanks,
--
-David
da...@pgmasters.net


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-08 Thread Tsunakawa, Takayuki 
From: pgsql-hackers-ow...@postgresql.org
> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Ashutosh Sharma
> To start with, I ran the regression test-suite and didn't find any failures.
> But, then I am not sure if huge_pages are getting used or not. However,
> upon checking the settings for huge_pages and I found it as 'on'. I am
> assuming, if huge pages is not being used due to shortage of large pages,
> it should have fallen back to non-huge pages.

You are right, the server falls back to non-huge pages when the large pages run 
short.

> I also ran the pgbench tests on read-only workload and here are the results
> I got.
> 
> pgbench -c 4 -j 4 - T 600 bench
> 
> huge_pages=on, TPS = 21120.768085
> huge_pages=off, TPS = 20606.288995

Thanks.  It's about 2% improvement, which is the same as what I got.


From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com]
> The line beginning 'Huge pages are known as...' has been accidentally
> duplicated.

Oops, how careless I was.  Fixed.  As Ashutosh referred, I added a very simple 
suggestion to use Windows Group Policy tool.

Regards
Takayuki Tsunakawa



win_large_pages_v9.patch
Description: win_large_pages_v9.patch

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-08 Thread Thomas Munro 
On Thu, Feb 23, 2017 at 8:29 PM, Tsunakawa, Takayuki
 wrote:
> [win_large_pages_v8.patch]

+Huge pages are known as large pages on Windows.  To use them,
you need to
+assign the user right Lock Pages in Memory to the Windows user account
+that runs PostgreSQL.
+Huge pages are known as large pages on Windows.  To use them,
you need to
+To start the database server on the command prompt as a
standalone process,
+not as a Windows service, run the command prompt as an administrator or
+disable the User Access Control (UAC). When the UAC is
enabled, the normal
+command prompt revokes the user right Lock Pages in Memory.

The line beginning 'Huge pages are known as...' has been accidentally
duplicated.

-- 
Thomas Munro
http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-03-08 Thread Ashutosh Sharma 
Hi,

I tried to test v8 version of patch. Firstly, I was able to start the
postgresql server process  with 'huge_pages' set to on. I had to
follow the instructions given in MSDN[1] to enable lock pages in
memory option and also had to start the postgresql server process as
admin user.

test=# show huge_pages;
 huge_pages

 on
(1 row)

To start with, I ran the regression test-suite and didn't find any
failures. But, then I am not sure if huge_pages are getting used or
not. However, upon checking the settings for huge_pages and I found it
as 'on'. I am assuming, if huge pages is not being used due to
shortage of large pages, it should have fallen back to non-huge pages.

I also ran the pgbench tests on read-only workload and here are the
results I got.

pgbench -c 4 -j 4 - T 600 bench

huge_pages=on, TPS = 21120.768085
huge_pages=off, TPS = 20606.288995

[1] - https://msdn.microsoft.com/en-IN/library/ms190730.aspx

--
With Regards,
Ashutosh Sharma
EnterpriseDB:http://www.enterprisedb.com

On Thu, Feb 23, 2017 at 12:59 PM, Tsunakawa, Takayuki
 wrote:
> From: Amit Kapila [mailto:amit.kapil...@gmail.com]
>> > Hmm, the large-page requires contiguous memory for each page, so this
>> error could occur on a long-running system where the memory is heavily
>> fragmented.  For example, please see the following page and check the memory
>> with RAMMap program referred there.
>> >
>>
>> I don't have RAMMap and it might take some time to investigate what is going
>> on, but I think in such a case even if it works we should keep the default
>> value of huge_pages as off on Windows.  I request somebody else having
>> access to Windows m/c to test this patch and if it works then we can move
>> forward.
>
> You are right.  I modified the patch so that the code falls back to the 
> non-huge page when CreateFileMapping() fails due to the shortage of large 
> pages.  That's what the Linux version does.
>
> The other change is to parameterize the Win32 function names in the messages 
> in EnableLockPagePrivileges().  This is to avoid adding almost identical 
> messages unnecessarily.  I followed Alvaro's comment.  I didn't touch the two 
> existing sites that embed Win32 function names.  I'd like to leave it up to 
> the committer to decide whether to change as well, because changing them 
> might make it a bit harder to apply some bug fixes to earlier releases.
>
> FYI, I could reproduce the same error as Amit on 32-bit Win7, where the total 
> RAM is 3.5 GB and available RAM is 2 GB.  I used the attached largepage.c.  
> Immediately after the system boot, I could only allocate 8 large pages.  When 
> I first tried to allocate 32 large pages, the test program produced:
>
> large page size = 2097152
> allocating 32 large pages...
> CreateFileMapping failed: error code = 1450
>
> You can build the test program as follows:
>
> cl largepage.c advapi32.lib
>
> Regards
> Takayuki Tsunakawa
>
>
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-02-22 Thread Tsunakawa, Takayuki 
From: Amit Kapila [mailto:amit.kapil...@gmail.com]
> > Hmm, the large-page requires contiguous memory for each page, so this
> error could occur on a long-running system where the memory is heavily
> fragmented.  For example, please see the following page and check the memory
> with RAMMap program referred there.
> >
> 
> I don't have RAMMap and it might take some time to investigate what is going
> on, but I think in such a case even if it works we should keep the default
> value of huge_pages as off on Windows.  I request somebody else having
> access to Windows m/c to test this patch and if it works then we can move
> forward.

You are right.  I modified the patch so that the code falls back to the 
non-huge page when CreateFileMapping() fails due to the shortage of large 
pages.  That's what the Linux version does.

The other change is to parameterize the Win32 function names in the messages in 
EnableLockPagePrivileges().  This is to avoid adding almost identical messages 
unnecessarily.  I followed Alvaro's comment.  I didn't touch the two existing 
sites that embed Win32 function names.  I'd like to leave it up to the 
committer to decide whether to change as well, because changing them might make 
it a bit harder to apply some bug fixes to earlier releases.

FYI, I could reproduce the same error as Amit on 32-bit Win7, where the total 
RAM is 3.5 GB and available RAM is 2 GB.  I used the attached largepage.c.  
Immediately after the system boot, I could only allocate 8 large pages.  When I 
first tried to allocate 32 large pages, the test program produced:

large page size = 2097152
allocating 32 large pages...
CreateFileMapping failed: error code = 1450

You can build the test program as follows:

cl largepage.c advapi32.lib

Regards
Takayuki Tsunakawa



#include 
#include 
#include 

static void EnableLockPagesPrivilege(void);

void main(int argc, char *argv[])
{
SIZE_T  largePageSize = 0;
HANDLE hmap;
int pages = 1;

largePageSize = GetLargePageMinimum();
printf("large page size = %u\n", largePageSize);

EnableLockPagesPrivilege();

if (argc > 1)
pages = atoi(argv[1]);
printf("allocating %d large pages...\n", pages);

hmap = CreateFileMapping(INVALID_HANDLE_VALUE, NULL,
PAGE_READWRITE | SEC_COMMIT | SEC_LARGE_PAGES,
0, largePageSize * pages,
"myshmem");
if (hmap)
printf("allocated large pages successfully\n");
else
printf("CreateFileMapping failed: error code = %u", 
GetLastError());
}

static void
EnableLockPagesPrivilege(void)
{
HANDLE hToken;
TOKEN_PRIVILEGES tp;
LUID luid;

if (!OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | 
TOKEN_QUERY, ))
{
printf("OpenProcessToken failed: error code = %u", 
GetLastError());
exit(1);
}

if (!LookupPrivilegeValue(NULL, SE_LOCK_MEMORY_NAME, ))
{
printf("LookupPrivilegeValue failed: error code = %u", 
GetLastError());
exit(1);
}
tp.PrivilegeCount = 1;
tp.Privileges[0].Luid = luid;
tp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;

if (!AdjustTokenPrivileges(hToken, FALSE, , 0, NULL, NULL))
{
printf("AdjustTokenPrivileges failed: error code = %u", 
GetLastError());
exit(1);
}

if (GetLastError() != ERROR_SUCCESS)
{
if (GetLastError() == ERROR_NOT_ALL_ASSIGNED)
printf("could not enable Lock Pages in Memory user 
right");
else
printf("AdjustTokenPrivileges failed: error code = %u", 
GetLastError());
exit(1);
}

CloseHandle(hToken);
}


win_large_pages_v8.patch
Description: win_large_pages_v8.patch

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-01-31 Thread Michael Paquier 
On Mon, Jan 30, 2017 at 10:46 AM, Tsunakawa, Takayuki
 wrote:
> From: Amit Kapila [mailto:amit.kapil...@gmail.com]
>> I think it is better to document in some way if we decide to go-ahead with
>> the patch.
>
> Sure, I added these sentences.

Patch has been moved to CF 2017-03. There is a recent new patch, and
the discussion is moving on.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-01-30 Thread Alvaro Herrera 
Magnus Hagander wrote:

> Taking a look at the actual code here, and a few small nitpicks:
> 
> +errdetail("Failed system call was %s,
> error code %lu", "LookupPrivilegeValue", GetLastError(;
> 
> this seems to be a new pattern of code -- for other similar cases it just
> writes LookupPrivilegeValue inside the patch itself. I'm guessing the idea
> was for translatability, but I think it's better we stick to the existing
> pattern.

There are two reasons for doing things this way.  One is that you reduce
the chances of a translator making a mistake with the function name (say
just a typo, or in egregious cases they may even translate the function
name).  The other is that if you have many of these messages, you only
translate the generic part once instead of having the same message
a handful of times, exactly identical but for the function name.

So please do apply that kind of pattern wherever possible.  We already
have the proposed error message, twice.  No need for two more
occurrences of it.

-- 
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-01-30 Thread Amit Kapila 
On Mon, Jan 30, 2017 at 7:16 AM, Tsunakawa, Takayuki
 wrote:
> From: Amit Kapila [mailto:amit.kapil...@gmail.com]
>> Hmm.  It doesn't work even on a command prompt with administrative
>> privileges. It gives below error:
>>
>> waiting for server to start2017-01-17 11:20:13.780 IST [4788] FATAL:
>> could not create shared memory segment: error code 1450
>> 2017-01-17 11:20:13.780 IST [4788] DETAIL:  Failed system call was
>> CreateFileMap ping(size=148897792,
>> name=Global/PostgreSQL:E:/WorkSpace/PostgreSQL/master/Data)
>> .
>> 2017-01-17 11:20:13.780 IST [4788] LOG:  database system is shut down
>> stopped waiting
>> pg_ctl: could not start server
>> Examine the log output.
>>
>>
>> Now, error code 1450 can occur due to insufficient system resources, so
>> I have tried by increasing the size of shared memory (higher value of
>> shared_buffers) without your patch and it works.  This indicates some
>> problem with the patch.
>
> Hmm, the large-page requires contiguous memory for each page, so this error 
> could occur on a long-running system where the memory is heavily fragmented.  
> For example, please see the following page and check the memory with RAMMap 
> program referred there.
>

I don't have RAMMap and it might take some time to investigate what is
going on, but I think in such a case even if it works we should keep
the default value of huge_pages as off on Windows.  I request somebody
else having access to Windows m/c to test this patch and if it works
then we can move forward.

> http://blog.dbi-services.com/large-pages-and-memory_target-on-windows/
>
> BTW, is your OS or PostgreSQL 32-bit?
>

both 64-bit.

-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-01-29 Thread Tsunakawa, Takayuki 
From: Amit Kapila [mailto:amit.kapil...@gmail.com]
> Hmm.  It doesn't work even on a command prompt with administrative
> privileges. It gives below error:
> 
> waiting for server to start2017-01-17 11:20:13.780 IST [4788] FATAL:
> could not create shared memory segment: error code 1450
> 2017-01-17 11:20:13.780 IST [4788] DETAIL:  Failed system call was
> CreateFileMap ping(size=148897792,
> name=Global/PostgreSQL:E:/WorkSpace/PostgreSQL/master/Data)
> .
> 2017-01-17 11:20:13.780 IST [4788] LOG:  database system is shut down
> stopped waiting
> pg_ctl: could not start server
> Examine the log output.
> 
> 
> Now, error code 1450 can occur due to insufficient system resources, so
> I have tried by increasing the size of shared memory (higher value of
> shared_buffers) without your patch and it works.  This indicates some
> problem with the patch.

Hmm, the large-page requires contiguous memory for each page, so this error 
could occur on a long-running system where the memory is heavily fragmented.  
For example, please see the following page and check the memory with RAMMap 
program referred there.

http://blog.dbi-services.com/large-pages-and-memory_target-on-windows/

BTW, is your OS or PostgreSQL 32-bit?


> >  It seems that Windows removes many privileges, including "Lock Pages
> in Memory", when starting the normal command prompt.  As its evidence, you
> can use the attached priv.c to see what privileges are assigned and and
> enabled/disabled.  Build it like "cl priv.c" and just run priv.exe on each
> command prompt.  Those runs show different privileges.
> >
> 
> This is bad.
> 
> > Should I need to do something, e.g. explain in the document that the user
> should use the command prompt with administrative privileges when he uses
> huge_pages?
> >
> 
> I think it is better to document in some way if we decide to go-ahead with
> the patch.

Sure, I added these sentences. 

+To start the database server on the command prompt as a standalone 
process,
+not as a Windows service, run the command prompt as an administrator or
+disable the User Access Control (UAC). When the UAC is enabled, the 
normal
+command prompt revokes the user right Lock Pages in Memory.

Regards
Takayuki Tsunakawa



win_large_pages_v7.patch
Description: win_large_pages_v7.patch

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-01-16 Thread Amit Kapila 
On Tue, Jan 10, 2017 at 8:49 AM, Tsunakawa, Takayuki
 wrote:
> Hello, Amit, Magnus,
>
> I'm sorry for my late reply.  Yesterday was a national holiday in Japan.
>
>
> From: pgsql-hackers-ow...@postgresql.org
>> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Amit Kapila
>> PGSharedMemoryReAttach is called after the startup of new process whereas
>> pgwin32_ReserveSharedMemoryRegion is called before the new process could
>> actually start.  Basically, pgwin32_ReserveSharedMemoryRegion is used to
>> reserve shared memory for each backend, so calling VirtualAlloc there should
>> follow spec for huge pages.  If you have some reason for not using, then
>> it is not clear from your reply, can you try to explain in detail.
>
> OK.  The processing takes place in the following order:
>
> 1. postmaster calls CreateProcess() to start a child postgres in a suspended 
> state.
> 2. postmaster calls VirtualAlloc(MEM_RESERVE) in 
> pgwin32_ReserveSharedMemoryRegion() to reserve the virtual address space in 
> the child to secure space for the shared memory.  This call just affects the 
> virtual address space and does not allocate physical memory.  So the large 
> page is still irrelevant.

Okay, today again reading VirtualAlloc specs, I could see that
MEM_LARGE_PAGES is not not required to reserve the memory region.  It
is only required during allocation.


>
> I succeeded by following the same procedure using secpol.msc on Win10, 
> running 64-bit PostgreSQL.  I started PostgreSQL as a Windows service because 
> it's the normal way, with the service account being a regular Windows user 
> account(i.e. not LocalSystem).
>
> But... I failed to start PostgreSQL by running "pg_ctl start" from a command 
> prompt, receiving the same error message as you.  On the other hand, I could 
> start PostgreSQL on a command prompt with administrative privileges 
> (right-click on "Command prompt" from the Start menu and select "Run as an 
> administrator" in the menu.


Hmm.  It doesn't work even on a command prompt with administrative
privileges. It gives below error:

waiting for server to start2017-01-17 11:20:13.780 IST [4788] FATAL:  could
not create shared memory segment: error code 1450
2017-01-17 11:20:13.780 IST [4788] DETAIL:  Failed system call was CreateFileMap
ping(size=148897792, name=Global/PostgreSQL:E:/WorkSpace/PostgreSQL/master/Data)
.
2017-01-17 11:20:13.780 IST [4788] LOG:  database system is shut down
 stopped waiting
pg_ctl: could not start server
Examine the log output.


Now, error code 1450 can occur due to insufficient system resources,
so I have tried by increasing the size of shared memory (higher value
of shared_buffers) without your patch and it works.  This indicates
some problem with the patch.

>  It seems that Windows removes many privileges, including "Lock Pages in 
> Memory", when starting the normal command prompt.  As its evidence, you can 
> use the attached priv.c to see what privileges are assigned and and 
> enabled/disabled.  Build it like "cl priv.c" and just run priv.exe on each 
> command prompt.  Those runs show different privileges.
>

This is bad.

> Should I need to do something, e.g. explain in the document that the user 
> should use the command prompt with administrative privileges when he uses 
> huge_pages?
>

I think it is better to document in some way if we decide to go-ahead
with the patch.


-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-01-09 Thread Tsunakawa, Takayuki 
Hello, Amit, Magnus,

I'm sorry for my late reply.  Yesterday was a national holiday in Japan.


From: pgsql-hackers-ow...@postgresql.org
> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Amit Kapila
> PGSharedMemoryReAttach is called after the startup of new process whereas
> pgwin32_ReserveSharedMemoryRegion is called before the new process could
> actually start.  Basically, pgwin32_ReserveSharedMemoryRegion is used to
> reserve shared memory for each backend, so calling VirtualAlloc there should
> follow spec for huge pages.  If you have some reason for not using, then
> it is not clear from your reply, can you try to explain in detail.

OK.  The processing takes place in the following order:

1. postmaster calls CreateProcess() to start a child postgres in a suspended 
state.
2. postmaster calls VirtualAlloc(MEM_RESERVE) in 
pgwin32_ReserveSharedMemoryRegion() to reserve the virtual address space in the 
child to secure space for the shared memory.  This call just affects the 
virtual address space and does not allocate physical memory.  So the large page 
is still irrelevant.
3. postmaster resumes execution of the child postgres.
4. The child postgres calls VirtualFree(MEM_RESERVE)  in 
PGSharedMemoryReAttach() to release the reserved virtual address space.  Here, 
the effect of VirtualAlloc() is invalidated anyway.
5. The child process calls MapViewOfFile() in PGSharedMemoryReAttach() to map 
the shared memory at the same address.  Hereafter, the large page option 
specified in CreateFileMapping() call is relevant.


> + if (!LookupPrivilegeValue(NULL, SE_LOCK_MEMORY_NAME, )) {
> + CloseHandle(hToken); ereport(elevel, (errmsg("could not enable Lock
> + pages in memory user right: error
> code %lu", GetLastError()),
> + errdetail("Failed system call was LookupPrivilegeValue."))); return
> + FALSE; }
> 
> The order of closing handle and ereport is different here than other places
> in the same function.  If there is no specific reason for doing so, then
> keep the order consistent.

You are right, I modified the patch to call CloseHandle() after ereport() so 
that ereport() CloseHandle() wouldn't change the error code for ereport().  
That's the order used in postmaster.c.


> I have tried to test v4 version of the patch and it is always failing in
> below error after call to AdjustTokenPrivileges:
> 
> + if (GetLastError() != ERROR_SUCCESS)
> + {
> + if (GetLastError() == ERROR_NOT_ALL_ASSIGNED) ereport(elevel,
> + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> + errmsg("could not enable Lock pages in memory user right"),
> + errhint("Assign Lock pages in memory user right to the Windows user
> account which runs PostgreSQL.")));
> 
> I have ensured that user used to run PostgreSQL has "Lock pages in memory"
> privilege/rights.  I have followed msdn tips [1] to do that (restarted the
> m/c after assigning privilege).  I am using Win7. Can you share the steps
> you have followed to test and your windows m/c details?
> 
> [1] -
> https://msdn.microsoft.com/en-us/library/windows/desktop/ff961911(v=vs
> .85).aspx

I succeeded by following the same procedure using secpol.msc on Win10, running 
64-bit PostgreSQL.  I started PostgreSQL as a Windows service because it's the 
normal way, with the service account being a regular Windows user account(i.e. 
not LocalSystem).

But... I failed to start PostgreSQL by running "pg_ctl start" from a command 
prompt, receiving the same error message as you.  On the other hand, I could 
start PostgreSQL on a command prompt with administrative privileges 
(right-click on "Command prompt" from the Start menu and select "Run as an 
administrator" in the menu.  It seems that Windows removes many privileges, 
including "Lock Pages in Memory", when starting the normal command prompt.  As 
its evidence, you can use the attached priv.c to see what privileges are 
assigned and and enabled/disabled.  Build it like "cl priv.c" and just run 
priv.exe on each command prompt.  Those runs show different privileges.

Should I need to do something, e.g. explain in the document that the user 
should use the command prompt with administrative privileges when he uses 
huge_pages?

Regards
Takayuki Tsunakawa



win_large_pages_v6.patch
Description: win_large_pages_v6.patch
#include 
#include 

int main()
{
HANDLE hToken;
PTOKEN_PRIVILEGES pTokenPrivileges;
char szPrivilegeName[256];
char szDisplayName[256];
DWORD dwLength;
DWORD dwLanguageId;
DWORD i;

if (!OpenProcessToken(GetCurrentProcess(), TOKEN_QUERY, )) {
return 1;
}

GetTokenInformation(hToken, TokenPrivileges, NULL, 0, );

pTokenPrivileges = (PTOKEN_PRIVILEGES)LocalAlloc(LPTR, dwLength);
if (pTokenPrivileges == NULL) {
CloseHandle(hToken);
return 1;
}

GetTokenInformation(hToken, TokenPrivileges, pTokenPrivileges, dwLength, 
);


for (i = 0; i < pTokenPrivileges->PrivilegeCount; i++) {

dwLength =

Re: [HACKERS] Supporting huge pages on Windows

2017-01-05 Thread Amit Kapila 
On Thu, Jan 5, 2017 at 8:42 AM, Tsunakawa, Takayuki
 wrote:
>> Your version of the patch looks better than the previous one.  Don't you
>> need to consider MEM_LARGE_PAGES in VirtualAllocEx call (refer
>> pgwin32_ReserveSharedMemoryRegion)?  At least that is what is mentioned
>> in MSDN [1].  Another point worth considering is that currently for
>> VirtualAllocEx() we use PAGE_READWRITE as flProtect value, shouldn't it
>> be same as used in CreateFileMapping() by patch.
>>
>>
>> [1] -
>> https://msdn.microsoft.com/en-us/library/windows/desktop/aa366720(v=vs
>> .85).aspx
>
> No, it's not necessary.  Please see PGSharedMemoryReAttach(), where 
> VirtualFree() is called to free the reserved address space and then call 
> MapViewOfFile() to allocate the already created shared memory to that area.
>

PGSharedMemoryReAttach is called after the startup of new process
whereas pgwin32_ReserveSharedMemoryRegion is called before the new
process could actually start.  Basically,
pgwin32_ReserveSharedMemoryRegion is used to reserve shared memory for
each backend, so calling VirtualAlloc there should follow spec for
huge pages.  If you have some reason for not using, then it is not
clear from your reply, can you try to explain in detail.

I have tried to test v4 version of the patch and it is always failing
in below error after call to AdjustTokenPrivileges:

+ if (GetLastError() != ERROR_SUCCESS)
+ {
+ if (GetLastError() == ERROR_NOT_ALL_ASSIGNED)
+ ereport(elevel,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("could not enable Lock pages in memory user right"),
+ errhint("Assign Lock pages in memory user right to the Windows user
account which runs PostgreSQL.")));

I have ensured that user used to run PostgreSQL has "Lock pages in
memory" privilege/rights.  I have followed msdn tips [1] to do that
(restarted the m/c after assigning privilege).  I am using Win7. Can
you share the steps you have followed to test and your windows m/c
details?

+ if (!LookupPrivilegeValue(NULL, SE_LOCK_MEMORY_NAME, ))
+ {
+ CloseHandle(hToken);
+ ereport(elevel,
+ (errmsg("could not enable Lock pages in memory user right: error
code %lu", GetLastError()),
+ errdetail("Failed system call was LookupPrivilegeValue.")));
+ return FALSE;
+ }

The order of closing handle and ereport is different here than other
places in the same function.  If there is no specific reason for doing
so, then keep the order consistent.


[1] - 
https://msdn.microsoft.com/en-us/library/windows/desktop/ff961911(v=vs.85).aspx


-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-01-05 Thread Tsunakawa, Takayuki 
From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com]
> In the Microsoft documentation I've seen, the privilege's name is always
> written as "Lock Pages in Memory" (note: "Pages" plural, and with initial
> capital letters).  It's quite hard to parse the sentence otherwise!  How
> about this?
> 
>  Huge pages are known as large pages on Windows.  To use them, you
> need to
>  assign the user right Lock Pages in Memory to the Windows user
> account
>  that runs PostgreSQL.
> 
> + ereport(elevel,
> + (errmsg("could not enable Lock pages in memory user right: error
> code %lu", GetLastError()),
> + errdetail("Failed system call was OpenProcessToken.")));
> 
> Same comment about capitalisation of the privilege name in this and other
> error messages.

Thanks, your sentences are surely better.  Fixed.

Regards
Takayuki Tsunakawa



win_large_pages_v5.patch
Description: win_large_pages_v5.patch

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-01-05 Thread Thomas Munro 
On Thu, Jan 5, 2017 at 4:12 PM, Tsunakawa, Takayuki
 wrote:
> [win_large_pages_v4.patch]

Just a small suggestion about the wording in this patch:

+This feature uses the large-page support on Windows. To use
the large-page
+support, you need to assign Lock page in memory user right to
the Windows
+user account which runs PostgreSQL.

In the Microsoft documentation I've seen, the privilege's name is
always written as "Lock Pages in Memory" (note: "Pages" plural, and
with initial capital letters).  It's quite hard to parse the sentence
otherwise!  How about this?

 Huge pages are known as large pages on Windows.  To use them,
you need to
 assign the user right Lock Pages in Memory to the Windows user account
 that runs PostgreSQL.

+ ereport(elevel,
+ (errmsg("could not enable Lock pages in memory user right: error
code %lu", GetLastError()),
+ errdetail("Failed system call was OpenProcessToken.")));

Same comment about capitalisation of the privilege name in this and
other error messages.

-- 
Thomas Munro
http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-01-04 Thread Tsunakawa, Takayuki 
From: pgsql-hackers-ow...@postgresql.org
> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Magnus Hagander
> For the pg_ctl changes, we're going from removing all privilieges from the
> token, to removing none. Are there any other privileges that we should be
> worried about? I think you may be correct in that it's overkill to do it,
> but I think we need some more specifics to decide that.

This page lists the privileges.  Is there anyhing you are concerned about?

https://msdn.microsoft.com/ja-jp/library/windows/desktop/bb530716(v=vs.85).aspx



> Also, what happens with privileges that were granted to the groups that
> were removed? Are they retained or lost?

Are you referring to the privileges of Administrators and PowerUsers that 
pg_ctl removes?  They are lost.  The Windows user account who actually runs 
PostgreSQL needs SeLockMemory privilege.


> Should we perhaps consider having pg_ctl instead *disable* all the
> privileges (rather than removing them), using AdjustTokenPrivileges? As
> a middle ground?

Sorry, I may misunderstand what you are suggesting, but AdjustTokenPrivilege() 
cannot enable the privilege which is not assigned to the user.  Anyway, I think 
it's the user's responsibility (and freedom) to assign desired privileges, and 
pg_ctl's disabling all privileges is overkill.


> +errdetail("Failed system call was %s,
> error code %lu", "LookupPrivilegeValue", GetLastError(;
> 
> this seems to be a new pattern of code -- for other similar cases it just
> writes LookupPrivilegeValue inside the patch itself. I'm guessing the idea
> was for translatability, but I think it's better we stick to the existing
> pattern.

OK, modified.

> When AdjustTokenPrivileges() returns, you explicitly check for
> ERROR_NOT_ALL_ASSIGNED, which is good. But we should probably also
> explicitly check for ERROR_SUCCESS for that case. Right now that's the only
> two possible options that can be returned, but in a future version other
> result codes could be added and we'd miss them. Basically, "there should
> be an else branch".

OK, modified.

> Is there a reason the error messages for AdjustTokenPrivileges() returning
> false and ERROR_NOT_ALL_ASSIGNED is different?

As mentioned in the following page, the error cause is clearly defined.  So, I 
thought it'd be better to give a specific hint message to help users 
troubleshoot the error.


https://msdn.microsoft.com/ja-jp/library/windows/desktop/aa375202(v=vs.85).aspx

ERROR_NOT_ALL_ASSIGNED 
The token does not have one or more of the privileges specified in the NewState 
parameter. The function may succeed with this error value even if no privileges 
were adjusted. The PreviousState parameter indicates the privileges that were 
adjusted.


> There are three repeated blocks of
> +   if (huge_pages == HUGE_PAGES_ON || huge_pages == HUGE_PAGES_TRY)
> 
> It threw me off in the initial reading, until I realized the upper levels
> of them can change the value of huge_pages.

OK, I like your code.


> I don't think changing the global variable huge_pages like that is a very
> good idea.
 
Yes, actually, I was afraid that it might be controversial to change the GUC 
value.  But I thought it may be better for "SHOW huge_pages" to reflect whether 
the huge_pages feature is effective.  Otherwise, users don't know about that.  
For example, wal_buffers behaves similarly; if it's set to -1 (default), "SHOW 
wal_buffers" displays the actual wal buffer size, not -1.  What do you think?  
Surely, the Linux code for huge_pages doesn't do that.  I'm OK with either.


From: Amit Kapila [mailto:amit.kapil...@gmail.com]
> Your version of the patch looks better than the previous one.  Don't you
> need to consider MEM_LARGE_PAGES in VirtualAllocEx call (refer
> pgwin32_ReserveSharedMemoryRegion)?  At least that is what is mentioned
> in MSDN [1].  Another point worth considering is that currently for
> VirtualAllocEx() we use PAGE_READWRITE as flProtect value, shouldn't it
> be same as used in CreateFileMapping() by patch.
> 
> 
> [1] -
> https://msdn.microsoft.com/en-us/library/windows/desktop/aa366720(v=vs
> .85).aspx

No, it's not necessary.  Please see PGSharedMemoryReAttach(), where 
VirtualFree() is called to free the reserved address space and then call 
MapViewOfFile() to allocate the already created shared memory to that area.

Regards
Takayuki Tsunakawa




win_large_pages_v4.patch
Description: win_large_pages_v4.patch

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2017-01-03 Thread Amit Kapila 
On Sat, Dec 31, 2016 at 7:04 PM, Magnus Hagander  wrote:
>
>
> On Wed, Sep 28, 2016 at 2:26 AM, Tsunakawa, Takayuki
>  wrote:
>>
>> > From: pgsql-hackers-ow...@postgresql.org
>> > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas
>> > On Sun, Sep 25, 2016 at 10:45 PM, Tsunakawa, Takayuki
>> >  wrote:
>> > > Credit: This patch is based on Thomas Munro's one.
>> >
>> > How are they different?
>>
>> As Thomas mentioned, his patch (only win32_shmem.c) might not have been
>> able to compile (though I didn't try.)  And it didn't have error processing
>> or documentation.  I added error handling, documentation, comments, and a
>> little bit of structural change.  The possibly biggest change, though it's
>> only one-liner in pg_ctl.c, is additionally required.  I failed to include
>> it in the first patch.  The attached patch includes that.
>>
>
>
> For the pg_ctl changes, we're going from removing all privilieges from the
> token, to removing none. Are there any other privileges that we should be
> worried about? I think you may be correct in that it's overkill to do it,
> but I think we need some more specifics to decide that.
>
> Also, what happens with privileges that were granted to the groups that were
> removed? Are they retained or lost?
>
> Should we perhaps consider having pg_ctl instead *disable* all the
> privileges (rather than removing them), using AdjustTokenPrivileges? As a
> middle ground?
>

Sounds like a good idea.

>
>
>
> Taking a look at the actual code here, and a few small nitpicks:
>
> +errdetail("Failed system call was %s, error
> code %lu", "LookupPrivilegeValue", GetLastError(;
>
> this seems to be a new pattern of code -- for other similar cases it just
> writes LookupPrivilegeValue inside the patch itself. I'm guessing the idea
> was for translatability, but I think it's better we stick to the existing
> pattern.
>
>
> When AdjustTokenPrivileges() returns, you explicitly check for
> ERROR_NOT_ALL_ASSIGNED, which is good. But we should probably also
> explicitly check for ERROR_SUCCESS for that case. Right now that's the only
> two possible options that can be returned, but in a future version other
> result codes could be added and we'd miss them. Basically, "there should be
> an else branch".
>
>
> Is there a reason the error messages for AdjustTokenPrivileges() returning
> false and ERROR_NOT_ALL_ASSIGNED is different?
>
>
> There are three repeated blocks of
> +   if (huge_pages == HUGE_PAGES_ON || huge_pages == HUGE_PAGES_TRY)
>
> It threw me off in the initial reading, until I realized the upper levels of
> them can change the value of huge_pages.
>
> I don't think changing the global variable huge_pages like that is a very
> good idea. Wouldn't something like the attached be cleaner (not tested)? At
> least I find that one easier to read.
>

Your version of the patch looks better than the previous one.  Don't
you need to consider MEM_LARGE_PAGES in VirtualAllocEx call (refer
pgwin32_ReserveSharedMemoryRegion)?  At least that is what is
mentioned in MSDN [1].  Another point worth considering is that
currently for VirtualAllocEx() we use PAGE_READWRITE as flProtect
value, shouldn't it be same as used in CreateFileMapping() by patch.


[1] - 
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366720(v=vs.85).aspx


-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2016-12-31 Thread Magnus Hagander 
On Wed, Sep 28, 2016 at 2:26 AM, Tsunakawa, Takayuki <
tsunakawa.ta...@jp.fujitsu.com> wrote:

> > From: pgsql-hackers-ow...@postgresql.org
> > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas
> > On Sun, Sep 25, 2016 at 10:45 PM, Tsunakawa, Takayuki
> >  wrote:
> > > Credit: This patch is based on Thomas Munro's one.
> >
> > How are they different?
>
> As Thomas mentioned, his patch (only win32_shmem.c) might not have been
> able to compile (though I didn't try.)  And it didn't have error processing
> or documentation.  I added error handling, documentation, comments, and a
> little bit of structural change.  The possibly biggest change, though it's
> only one-liner in pg_ctl.c, is additionally required.  I failed to include
> it in the first patch.  The attached patch includes that.
>
>

For the pg_ctl changes, we're going from removing all privilieges from the
token, to removing none. Are there any other privileges that we should be
worried about? I think you may be correct in that it's overkill to do it,
but I think we need some more specifics to decide that.

Also, what happens with privileges that were granted to the groups that
were removed? Are they retained or lost?

Should we perhaps consider having pg_ctl instead *disable* all the
privileges (rather than removing them), using AdjustTokenPrivileges? As a
middle ground?




Taking a look at the actual code here, and a few small nitpicks:

+errdetail("Failed system call was %s,
error code %lu", "LookupPrivilegeValue", GetLastError(;

this seems to be a new pattern of code -- for other similar cases it just
writes LookupPrivilegeValue inside the patch itself. I'm guessing the idea
was for translatability, but I think it's better we stick to the existing
pattern.


When AdjustTokenPrivileges() returns, you explicitly check for
ERROR_NOT_ALL_ASSIGNED, which is good. But we should probably also
explicitly check for ERROR_SUCCESS for that case. Right now that's the only
two possible options that can be returned, but in a future version other
result codes could be added and we'd miss them. Basically, "there should be
an else branch".


Is there a reason the error messages for AdjustTokenPrivileges() returning
false and ERROR_NOT_ALL_ASSIGNED is different?


There are three repeated blocks of
+   if (huge_pages == HUGE_PAGES_ON || huge_pages == HUGE_PAGES_TRY)

It threw me off in the initial reading, until I realized the upper levels
of them can change the value of huge_pages.

I don't think changing the global variable huge_pages like that is a very
good idea. Wouldn't something like the attached be cleaner (not tested)? At
least I find that one easier to read.



-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 8d7b3bf..e1c3c66 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1335,17 +1335,23 @@ include_dir 'conf.d'

 

-At present, this feature is supported only on Linux. The setting is
-ignored on other systems when set to try.
+At present, this feature is supported only on Linux and Windows. The
+setting is ignored on other systems when set to try.

 

 The use of huge pages results in smaller page tables and less CPU time
-spent on memory management, increasing performance. For more details,
+spent on memory management, increasing performance. For more details on Linux,
 see .

 

+This feature uses the large-page support on Windows. To use the large-page
+support, you need to assign Lock page in memory user right to the Windows
+user account which runs PostgreSQL.
+   
+
+   
 With huge_pages set to try,
 the server will try to use huge pages, but fall back to using
 normal allocation if that fails. With on, failure
diff --git a/src/backend/port/win32_shmem.c b/src/backend/port/win32_shmem.c
index 0ff2c7e..88b195a 100644
--- a/src/backend/port/win32_shmem.c
+++ b/src/backend/port/win32_shmem.c
@@ -21,6 +21,7 @@ HANDLE		UsedShmemSegID = INVALID_HANDLE_VALUE;
 void	   *UsedShmemSegAddr = NULL;
 static Size UsedShmemSegSize = 0;
 
+static bool EnableLockPagesPrivilege(int elevel);
 static void pgwin32_SharedMemoryDelete(int status, Datum shmId);
 
 /*
@@ -103,6 +104,61 @@ PGSharedMemoryIsInUse(unsigned long id1, unsigned long id2)
 	return true;
 }
 
+/*
+ * EnableLockPagesPrivilege
+ *
+ * Try to acquire SeLockMemoryPrivilege so we can use large pages.
+ */
+static bool
+EnableLockPagesPrivilege(int elevel)
+{
+	HANDLE hToken;
+	TOKEN_PRIVILEGES tp;
+	LUID luid;
+
+	if (!OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, ))
+	{
+		ereport(elevel,
+(errmsg("could not enable Lock pages in memory user right"),
+ errdetail("Failed

Re: [HACKERS] Supporting huge pages on Windows

2016-12-02 Thread Haribabu Kommi 
On Tue, Oct 11, 2016 at 1:36 PM, Tsunakawa, Takayuki <
tsunakawa.ta...@jp.fujitsu.com> wrote:

>
>
Moved to next CF with "needs review" status.


Regards,
Hari Babu
Fujitsu Australia

Re: [HACKERS] Supporting huge pages on Windows

2016-10-10 Thread Tsunakawa, Takayuki 
From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com]
> Your ~2.4% number is similar to what was reported for Linux with 4GB
> shared_buffers:
> 
> https://www.postgresql.org/message-id/20130913234125.GC13697%40roobarb
> .crazydogs.org

I'm relieved to know that a similar figure was gained on Linux.  Thanks for the 
info.


> Later in that thread there was a report of a dramatic ~15% increase in "best
> result" TPS, but that was with 60GB of shared_buffers on a machine with
> 256GB of RAM:
> 
> https://www.postgresql.org/message-id/20131024060313.GA21888%40toroid.
> org

From: Andres Freund [mailto:and...@anarazel.de]
> FWIW, I've seen 2-3x increases with ~60GB of s_b.

Wow, nice figures.  It's unfortunate that I don't have such a big machine 
available at hand.

Regards
Takayuki Tsunakawa



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2016-10-10 Thread Andres Freund 
On 2016-10-11 09:57:48 +1300, Thomas Munro wrote:
> Later in that thread there was a report of a dramatic ~15% increase in
> "best result" TPS, but that was with 60GB of shared_buffers on a
> machine with 256GB of RAM:
> 
> https://www.postgresql.org/message-id/20131024060313.GA21888%40toroid.org

FWIW, I've seen 2-3x increases with ~60GB of s_b.

Andres


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2016-10-10 Thread Thomas Munro 
On Wed, Sep 28, 2016 at 7:32 PM, Tsunakawa, Takayuki
 wrote:
> From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com]
>> >  huge_pages=off: 70412 tps
>> >  huge_pages=on : 72100 tps
>>
>> Hmm.  I guess it could be noise or random code rearrangement effects.
>
> I'm not the difference was a random noise, because running multiple set of 
> three runs of pgbench (huge_pages = on, off, on, off, on...) produced similar 
> results.  But I expected a bit greater improvement, say, +10%.  There may be 
> better benchmark model where the large page stands out, but I think pgbench 
> is not so bad because its random data access would cause TLB cache misses.

Your ~2.4% number is similar to what was reported for Linux with 4GB
shared_buffers:

https://www.postgresql.org/message-id/20130913234125.GC13697%40roobarb.crazydogs.org

Later in that thread there was a report of a dramatic ~15% increase in
"best result" TPS, but that was with 60GB of shared_buffers on a
machine with 256GB of RAM:

https://www.postgresql.org/message-id/20131024060313.GA21888%40toroid.org

-- 
Thomas Munro
http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2016-09-28 Thread Tsunakawa, Takayuki 
From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com]
> >  huge_pages=off: 70412 tps
> >  huge_pages=on : 72100 tps
> 
> Hmm.  I guess it could be noise or random code rearrangement effects.

I'm not the difference was a random noise, because running multiple set of 
three runs of pgbench (huge_pages = on, off, on, off, on...) produced similar 
results.  But I expected a bit greater improvement, say, +10%.  There may be 
better benchmark model where the large page stands out, but I think pgbench is 
not so bad because its random data access would cause TLB cache misses.



> I saw your recent post[2] proposing to remove the sentence about the 512MB
> effective limit and I wondered why you didn't go to larger sizes with a
> larger database and more run time.  But I will let others with more
> benchmarking experience comment on the best approach to investigate Windows
> shared_buffers performance.

Yes, I could have gone to 8GB of shared_buffers because my PC has 16GB of RAM, 
but I felt the number of variations was sufficient.  Anyway, positive comments 
on benchmarking would be appreciated.

Regards
Takayuki Tsunakawa



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2016-09-27 Thread Thomas Munro 
On Wed, Sep 28, 2016 at 1:26 PM, Tsunakawa, Takayuki
 wrote:
>> From: pgsql-hackers-ow...@postgresql.org
>> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas
>> On Sun, Sep 25, 2016 at 10:45 PM, Tsunakawa, Takayuki
>>  wrote:
>> > Credit: This patch is based on Thomas Munro's one.
>>
>> How are they different?
>
> As Thomas mentioned, his patch (only win32_shmem.c) might not have been able 
> to compile (though I didn't try.)  And it didn't have error processing or 
> documentation.  I added error handling, documentation, comments, and a little 
> bit of structural change.  The possibly biggest change, though it's only 
> one-liner in pg_ctl.c, is additionally required.  I failed to include it in 
> the first patch.  The attached patch includes that.

Right, my patch[1] was untested sketch code.  I had heard complaints
about poor performance with large shared_buffers on Windows, and then
I bumped into some recommendations to turn large pages on for a couple
of other RDBMSs if using large buffer pool.  So I wrote that patch
based on the documentation to try some time in the future if I ever
got trapped in a room with Windows, but I posted it just in case the
topic would interest other hackers.  Thanks for picking it up!

>  huge_pages=off: 70412 tps
>  huge_pages=on : 72100 tps

Hmm.  I guess it could be noise or random code rearrangement effects.
I saw your recent post[2] proposing to remove the sentence about the
512MB effective limit and I wondered why you didn't go to larger sizes
with a larger database and more run time.  But I will let others with
more benchmarking experience comment on the best approach to
investigate Windows shared_buffers performance.

[1] 
https://www.postgresql.org/message-id/CAEepm=075-bghi_vdt4scamt+o_+1xarap2zh7xwfzvt294...@mail.gmail.com
[2] 
https://www.postgresql.org/message-id/0A3221C70F24FB45833433255569204D1F5EE995@G01JPEXMBYT05

-- 
Thomas Munro
http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2016-09-27 Thread Tsunakawa, Takayuki 
> From: pgsql-hackers-ow...@postgresql.org
> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas
> On Sun, Sep 25, 2016 at 10:45 PM, Tsunakawa, Takayuki
>  wrote:
> > Credit: This patch is based on Thomas Munro's one.
> 
> How are they different?

As Thomas mentioned, his patch (only win32_shmem.c) might not have been able to 
compile (though I didn't try.)  And it didn't have error processing or 
documentation.  I added error handling, documentation, comments, and a little 
bit of structural change.  The possibly biggest change, though it's only 
one-liner in pg_ctl.c, is additionally required.  I failed to include it in the 
first patch.  The attached patch includes that.


Regards
Takayuki Tsunakawa



win_large_page_v2.patch
Description: win_large_page_v2.patch

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Supporting huge pages on Windows

2016-09-27 Thread Robert Haas 
On Sun, Sep 25, 2016 at 10:45 PM, Tsunakawa, Takayuki
 wrote:
> Credit: This patch is based on Thomas Munro's one.

How are they different?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Supporting huge pages on Windows

2016-09-25 Thread Tsunakawa, Takayuki 
Hello,

The attached patch implements huge_pages on Windows.  I'll add this to the 
CommitFest.

The performance improvement was about 2% with the following select-only 
pgbench.  The scaling factor is 200, where the database size is roughly 3GB.  I 
ran the benchmark on my Windows 10 PC with 6 CPU cores and 16GB of RAM.

　　pgbench -c18 -j6 -T60 -S bench

Before running pgbench, I used pg_prewarm to cache all pgbench tables and 
indexes (excluding the history table) in the 4GB shared buffers.  The averages 
of running pgbench three times are:

  huge_pages=off: 70412 tps
  huge_pages=on : 72100 tps

The purpose of pg_ctl.c modification is to retain "Lock pages in memory" 
Windows user right in postgres.  That user right is necessary for the 
large-page support.  The current pg_ctl removes all privileges when spawning 
postgres, which is overkill.  The system administrator should be able to assign 
appropriate privileges to the PostgreSQL service account.

Credit: This patch is based on Thomas Munro's one.


Regards
Takayuki Tsunakawa



win_large_page.patch
Description: win_large_page.patch

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers