Re: [HACKERS] Add doc advice about systemd RemoveIPC

[Peter Eisentraut]

On 2/14/17 11:49 AM, Magnus Hagander wrote:
> +   
> +If systemd is in use, some care must be
> taken
> +that IPC resources (shared memory and semaphores) are not prematurely
> +removed by the operating system.  This is especially of concern when
> +installing PostgreSQL from source.  Users of distribution packages of
> +PostgreSQL are less likely to be affected.
> +   
> 
> I would add "are less likely to be affected as the postgres user is
> normally created as a system user" or something like that -- to indicate
> *why* they are less likely to be affected (and it also tells people that
> if they change the user, then they might become affected again).

Committed with that addition, thanks!

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add doc advice about systemd RemoveIPC

[Magnus Hagander]

On Fri, Feb 10, 2017 at 10:36 PM, Peter Eisentraut <
peter.eisentr...@2ndquadrant.com> wrote:

> On 12/31/16 11:43 AM, Tom Lane wrote:
> > Magnus Hagander  writes:
> >> I still think that some wording in the direction of the fact that the
> >> majority of all users won't actually have this problem is the right
> thing
> >> to do (regardless of our previous history in the area as pointed out by
> >> Craig)
> >
> > +1.  The use-a-system-user solution is the one that's in place on the
> > ground for most current PG users on affected platforms.  We should
> explain
> > that one first and make clear that platform-specific packages attempt to
> > set it up that way for you.  The RemoveIPC technique should be documented
> > as a fallback to be used if you can't/won't use a system userid.
>
> How about this version, which shifts the emphasis a bit, as suggested?
>
>
Looks much better.

+   
+If systemd is in use, some care must be
taken
+that IPC resources (shared memory and semaphores) are not prematurely
+removed by the operating system.  This is especially of concern when
+installing PostgreSQL from source.  Users of distribution packages of
+PostgreSQL are less likely to be affected.
+   

I would add "are less likely to be affected as the postgres user is
normally created as a system user" or something like that -- to indicate
*why* they are less likely to be affected (and it also tells people that if
they change the user, then they might become affected again).



-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [HACKERS] Add doc advice about systemd RemoveIPC

[Peter Eisentraut]

On 12/31/16 11:43 AM, Tom Lane wrote:
> Magnus Hagander  writes:
>> I still think that some wording in the direction of the fact that the
>> majority of all users won't actually have this problem is the right thing
>> to do (regardless of our previous history in the area as pointed out by
>> Craig)
> 
> +1.  The use-a-system-user solution is the one that's in place on the
> ground for most current PG users on affected platforms.  We should explain
> that one first and make clear that platform-specific packages attempt to
> set it up that way for you.  The RemoveIPC technique should be documented
> as a fallback to be used if you can't/won't use a system userid.

How about this version, which shifts the emphasis a bit, as suggested?

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>From 5b4ba436d3882bfa2ce0e6243b9ab2ece66a4da4 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut 
Date: Fri, 10 Feb 2017 16:34:20 -0500
Subject: [PATCH v2] doc: Add advice about systemd RemoveIPC

---
 doc/src/sgml/runtime.sgml | 77 +++
 1 file changed, 77 insertions(+)

diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml
index 130c386462..25c57192db 100644
--- a/doc/src/sgml/runtime.sgml
+++ b/doc/src/sgml/runtime.sgml
@@ -1165,6 +1165,83 @@ System V IPC Parameters
 
   
 
+  
+   systemd RemoveIPC
+
+   
+systemd
+RemoveIPC
+   
+
+   
+If systemd is in use, some care must be taken
+that IPC resources (shared memory and semaphores) are not prematurely
+removed by the operating system.  This is especially of concern when
+installing PostgreSQL from source.  Users of distribution packages of
+PostgreSQL are less likely to be affected.
+   
+
+   
+The setting RemoveIPC
+in logind.conf controls whether IPC objects are
+removed when a user fully logs out.  System users are exempt.  This
+setting defaults to on in stock systemd, but
+some operating system distributions default it to off.
+   
+
+   
+A typical observed effect when this setting is on is that the semaphore
+objects used by a PostgreSQL server are removed at apparently random
+times, leading to the server crashing with log messages like
+
+LOG: semctl(1234567890, 0, IPC_RMID, ...) failed: Invalid argument
+
+Different types of IPC objects (shared memory vs. semaphores, System V
+vs. POSIX) are treated slightly differently
+by systemd, so one might observe that some IPC
+resources are not removed in the same way as others.  But it is not
+advisable to rely on these subtle differences.
+   
+
+   
+A user logging out might happen as part of a maintenance
+job or manually when an administrator logs in as
+the postgres user or similar, so it is hard to prevent
+in general.
+   
+
+   
+What is a system user is determined
+at systemd compile time from
+the SYS_UID_MAX setting
+in /etc/login.defs.
+   
+
+   
+Packaging and deployment scripts should be careful to create
+the postgres user as a system user by
+using useradd -r, adduser --system,
+or equivalent.
+   
+
+   
+Alternatively, if the user account was created incorrectly or cannot be
+changed, it is recommended to set
+
+RemoveIPC=no
+
+in /etc/systemd/logind.conf or another appropriate
+configuration file.
+   
+
+   
+
+ At least one of these two things have to be ensured, or the PostgreSQL
+ server will be very unreliable.
+
+   
+  
+
   
Resource Limits
 
-- 
2.11.1


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add doc advice about systemd RemoveIPC

[Tom Lane]

Magnus Hagander  writes:
> I still think that some wording in the direction of the fact that the
> majority of all users won't actually have this problem is the right thing
> to do (regardless of our previous history in the area as pointed out by
> Craig)

+1.  The use-a-system-user solution is the one that's in place on the
ground for most current PG users on affected platforms.  We should explain
that one first and make clear that platform-specific packages attempt to
set it up that way for you.  The RemoveIPC technique should be documented
as a fallback to be used if you can't/won't use a system userid.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add doc advice about systemd RemoveIPC

[Magnus Hagander]

On Sat, Dec 31, 2016 at 6:30 AM, Peter Eisentraut <
peter.eisentr...@2ndquadrant.com> wrote:

> On 12/30/16 3:59 AM, Magnus Hagander wrote:
> > I wonder if I missed part of the discussions around this, so maybe my
> > understanding of the cases where this occurs is wrong, but isn't it the
> > case of pretty much all (or actually) all the packaged versions of
> > postgresql out there (debian, redhat etc) that they do the right thing,
> > as in that they create "postgres" as a system user?
>
> If you install a package but the user already exists, then the package
> will just use that user.  So just using a package is not a guarantee
> that everything will be alright.
>
>
Good point.

I still think that some wording in the direction of the fact that the
majority of all users won't actually have this problem is the right thing
to do (regardless of our previous history in the area as pointed out by
Craig)

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [HACKERS] Add doc advice about systemd RemoveIPC

[Peter Eisentraut]

On 12/30/16 3:59 AM, Magnus Hagander wrote:
> I wonder if I missed part of the discussions around this, so maybe my
> understanding of the cases where this occurs is wrong, but isn't it the
> case of pretty much all (or actually) all the packaged versions of
> postgresql out there (debian, redhat etc) that they do the right thing,
> as in that they create "postgres" as a system user?

If you install a package but the user already exists, then the package
will just use that user.  So just using a package is not a guarantee
that everything will be alright.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add doc advice about systemd RemoveIPC

[Craig Ringer]

On 30 December 2016 at 16:59, Magnus Hagander  wrote:
> On Wed, Dec 28, 2016 at 4:34 AM, Peter Eisentraut
>  wrote:
>>
>> Here is a patch to add some information about the systemd RemoveIPC
>> issue to the documentation, sort of in the spirit of the OOM discussion
>> nearby.
>
>
> I wonder if I missed part of the discussions around this, so maybe my
> understanding of the cases where this occurs is wrong, but isn't it the case
> of pretty much all (or actually) all the packaged versions of postgresql out
> there (debian, redhat etc) that they do the right thing, as in that they
> create "postgres" as a system user?

Yes.

The postgres docs do tend to ignore the reality of most actual
postgres users, though, and talk as if you installed it from source
code under your own user account. I see people bewildered by this
regularly, since we have no discussion at all of common things like
"sudo -u postgres psql" on default packaged installs. Sure, there are
many platforms, but still.



> I like the text in general, but if the above is true, then I think we should
> put a note at the beginning of it with something along the line (not using
> those words) of "if you have installed postgresql using packages, the
> packager should have taken care of this already"? So as not to scare people
> unnecessarily?

You need to have not only installed it with packages, but be running
it under the package-provided postgres user account. This is not
always the case. I see installs from packages that are then manually
initdb'd in /srv/wtf/why all the time, sadly, and often launched by
manual pg_ctl invocations under surprising user accounts.

"If you have installed postgres from distribution or
postgresql.org-provided packages and use the scripts or commands
provided by the packages to start and stop PostgreSQL, this issue is
unlikely to affect you."

?

-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add doc advice about systemd RemoveIPC

[Magnus Hagander]

On Wed, Dec 28, 2016 at 4:34 AM, Peter Eisentraut <
peter.eisentr...@2ndquadrant.com> wrote:

> Here is a patch to add some information about the systemd RemoveIPC
> issue to the documentation, sort of in the spirit of the OOM discussion
> nearby.
>

I wonder if I missed part of the discussions around this, so maybe my
understanding of the cases where this occurs is wrong, but isn't it the
case of pretty much all (or actually) all the packaged versions of
postgresql out there (debian, redhat etc) that they do the right thing, as
in that they create "postgres" as a system user?

I like the text in general, but if the above is true, then I think we
should put a note at the beginning of it with something along the line (not
using those words) of "if you have installed postgresql using packages, the
packager should have taken care of this already"? So as not to scare people
unnecessarily?

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

[HACKERS] Add doc advice about systemd RemoveIPC

[Peter Eisentraut]

Here is a patch to add some information about the systemd RemoveIPC
issue to the documentation, sort of in the spirit of the OOM discussion
nearby.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>From eaf0eda3f4c402a2e8b7f2f2395a8536f38aafbc Mon Sep 17 00:00:00 2001
From: Peter Eisentraut 
Date: Tue, 27 Dec 2016 12:00:00 -0500
Subject: [PATCH] doc: Add advice about systemd RemoveIPC

---
 doc/src/sgml/runtime.sgml | 82 +++
 1 file changed, 82 insertions(+)

diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml
index 787cfce987..fee0d65d90 100644
--- a/doc/src/sgml/runtime.sgml
+++ b/doc/src/sgml/runtime.sgml
@@ -1165,6 +1165,88 @@ System V IPC Parameters
 
   
 
+  
+   systemd RemoveIPC
+
+   
+systemd
+RemoveIPC
+   
+
+   
+
+ If systemd is in use, the advice in this
+ section must be followed, or your PostgreSQL server will be very
+ unreliable.
+
+   
+
+   
+If systemd is in use, special care must be
+taken that IPC resources (shared memory and semaphores) are not
+prematurely removed by the operating system.  Although the issues
+described here are most commonly known to happen if the PostgreSQL server
+is started through a systemd service unit, they
+can also happen in other setups.
+   
+
+   
+The setting RemoveIPC
+in logind.conf controls whether IPC objects are
+removed when a user fully logs out.  System users are exempt.  This
+setting defaults to on in stock systemd, but
+some operating system distributions default it to off.
+   
+
+   
+A typical observed effect when this setting is on is that the semaphore
+objects used by a PostgreSQL server are removed at apparently random
+times, leading to the server crashing with log messages like
+
+LOG: semctl(1234567890, 0, IPC_RMID, ...) failed: Invalid argument
+
+Different types of IPC objects (shared memory vs. semaphores, System V
+vs. POSIX) are treated slightly differently
+by systemd, so one might observe that some IPC
+resources are not removed in the same way as others.  But it is not
+advisable to rely on these subtle differences.
+   
+
+   
+A user logging out might happen as part of a maintenance
+job or manually when an administrator logs in as
+the postgres user or similar, so it is hard to prevent
+in general.
+   
+
+   
+What is a system user is determined
+at systemd compile time from
+the SYS_UID_MAX setting
+in /etc/login.defs.
+   
+
+   
+It is recommended to set
+
+RemoveIPC=no
+
+on all server hosts used for PostgreSQL.
+   
+
+   
+Also, packaging and deployment scripts should be careful to create
+the postgres user as a system user by
+using useradd -r, adduser --system,
+or equivalent.
+   
+
+   
+At least one of these two things have to be ensured, or the
+PostgreSQL server will be very unreliable.
+   
+  
+
   
Resource Limits
 
-- 
2.11.0


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers