Re: [HACKERS] Shared memory changes in 9.4?
Uh, this patch was never applied. Do we want it? --- On Thu, Jun 12, 2014 at 03:39:14PM +0200, Christoph Berg wrote: Re: Andres Freund 2014-06-12 20140612094112.gz8...@alap3.anarazel.de * Make initdb determine the best shm type for this platform and write it into postgresql.conf as it does now. * If no dynamic_shared_memory_type is found in the config, default to none. * Modify the three identical error messages concerned about shm segments to include the shm type instead of always just saying FATAL: could not open shared memory segment * Add a HINT to the POSIX error message: HINT: This might indicate that /dev/shm is not mounted, or its permissions do not allow the database user to create files there Sounds like a sane plan to me. Here are two patches, one that implements the annotated error messages, and one that selects none as default. It might also make sense to add a Note that POSIX depends on /dev/shm, and also a Note that dynamic_shared_memory_type is not related to the shared_buffers shm segments, which I didn't include here. Christoph -- c...@df7cb.de | http://www.df7cb.de/ diff --git a/src/backend/storage/ipc/dsm_impl.c b/src/backend/storage/ipc/dsm_impl.c new file mode 100644 index 0819641..780e3a5 *** a/src/backend/storage/ipc/dsm_impl.c --- b/src/backend/storage/ipc/dsm_impl.c *** dsm_impl_posix(dsm_op op, dsm_handle han *** 289,296 if (errno != EEXIST) ereport(elevel, (errcode_for_dynamic_shared_memory(), ! errmsg(could not open shared memory segment \%s\: %m, ! name))); return false; } --- 289,299 if (errno != EEXIST) ereport(elevel, (errcode_for_dynamic_shared_memory(), ! errmsg(could not open POSIX shared memory segment \%s\: %m, ! name), ! errhint(This error usually means that /dev/shm is not mounted, or its ! permissions do not allow the database user to create files ! there.))); return false; } *** dsm_impl_posix(dsm_op op, dsm_handle han *** 313,319 ereport(elevel, (errcode_for_dynamic_shared_memory(), ! errmsg(could not stat shared memory segment \%s\: %m, name))); return false; } --- 316,322 ereport(elevel, (errcode_for_dynamic_shared_memory(), ! errmsg(could not stat POSIX shared memory segment \%s\: %m, name))); return false; } *** dsm_impl_posix(dsm_op op, dsm_handle han *** 332,338 ereport(elevel, (errcode_for_dynamic_shared_memory(), ! errmsg(could not resize shared memory segment %s to %zu bytes: %m, name, request_size))); return false; } --- 335,341 ereport(elevel, (errcode_for_dynamic_shared_memory(), ! errmsg(could not resize POSIX shared memory segment %s to %zu bytes: %m, name, request_size))); return false; } *** dsm_impl_posix(dsm_op op, dsm_handle han *** 358,364 ereport(elevel, (errcode_for_dynamic_shared_memory(), !errmsg(could not unmap shared memory segment \%s\: %m, name))); return false; } --- 361,367 ereport(elevel, (errcode_for_dynamic_shared_memory(), !errmsg(could not unmap POSIX shared memory segment \%s\: %m, name))); return false; } *** dsm_impl_posix(dsm_op op, dsm_handle han *** 382,388 ereport(elevel, (errcode_for_dynamic_shared_memory(), ! errmsg(could not map shared memory segment
Re: [HACKERS] Shared memory changes in 9.4?
[redirecting to -hackers] Re: Robert Haas 2014-05-28 CA+TgmoaTcAd48zW3auWzGdHi_V+QwA5HVCTabqgTq=zlwpy...@mail.gmail.com On Tue, May 27, 2014 at 8:22 PM, Maciek Sakrejda m.sakre...@gmail.com wrote: On Mon, May 26, 2014 at 12:24 AM, Andres Freund and...@2ndquadrant.com wrote: Any chance you're using a 9.3 configuration file instead of the one generated by initdb? dynamic_shared_memory_type defaults to 'posix' if not specified in the config file (on platforms supporting it). If initdb detects that 'posix' can't be used it'll emit a different value. If you're copying the config from 9.3 and your environment doesn't support posix shm that'll cause the above error. I still think dynamic_shared_memory_type should default to 'none' because of such problems It works with 'none' and 'sysv'--I think the issue is that technically our environment does support 'posix', but '/dev/shm' is indeed not mounted in the LXC container, leading to a discrepancy between what initdb decides and what's actually possible. Thanks for your help. I think it would be good to understand why initdb isn't getting this right. Did you run initdb outside the LXC container, where /dev/shm would have worked, but then run postgres inside the LXC container, where /dev/shm does not work? I ask because initdb is supposed to be doing the same thing that postgres does, so it really ought to come to the same conclusion about what will and won't work. With regard to Andres' proposal, I'm not that keen on setting dynamic_shared_memory_type='none' by default. Would we leave it that way until we get in-core users of the facility, and then change it? I guess that'd be OK, but frankly if enabling dynamic_shared_memory_type by default is causing us many problems, then we'd better reconsider the design of the facility now, before we start adding more dependencies on it. We've already fixed a bunch of DSM-related issues as a result of the fact that the default *isn't* none, and I dunno how many of those we would have found if the default had been none. I tend to think DSM is an important facility that we're going to be wanting to build on in future releases, so I'm keen to have it available by default so that we can iron out any kinks before we get too far down that path. Hi, I've just run into the same problem. I am running the Debian postgresql-common testsuite, which includes upgrade tests. On upgrades, the old postgresql.conf is copied to the new server (after initdb and/or pg_upgrade), and deprecated options are removed/renamed. [*] In that chroot environment, 9.4 is running fine, except that upgrades failed because /dev/shm wasn't mounted, and the old 9.3 postgresql.conf doesn't contain dynamic_shared_memory_type = 'sysv'. To me, the following should be done: * Make initdb determine the best shm type for this platform and write it into postgresql.conf as it does now. * If no dynamic_shared_memory_type is found in the config, default to none. * Modify the three identical error messages concerned about shm segments to include the shm type instead of always just saying FATAL: could not open shared memory segment * Add a HINT to the POSIX error message: HINT: This might indicate that /dev/shm is not mounted, or its permissions do not allow the database user to create files there Despite /dev/shm having been around for quite some time, many people seem to be unaware what POSIX shared memory is, so the HINT is really needed there. It certainly took me weeks after seeing the error message the first time till I found the time to dig deeper to the issure - it should just have been oh yes, /dev/shm isn't mounted there, that's why. Christoph [*] This might not be the smartest thing to do, but it's a simple default approach to get the new cluster running with as much user config from the old config as possible. -- c...@df7cb.de | http://www.df7cb.de/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Shared memory changes in 9.4?
Hi, On 2014-06-12 11:07:31 +0200, Christoph Berg wrote: Re: Robert Haas 2014-05-28 CA+TgmoaTcAd48zW3auWzGdHi_V+QwA5HVCTabqgTq=zlwpy...@mail.gmail.com On Tue, May 27, 2014 at 8:22 PM, Maciek Sakrejda m.sakre...@gmail.com wrote: On Mon, May 26, 2014 at 12:24 AM, Andres Freund and...@2ndquadrant.com wrote: Any chance you're using a 9.3 configuration file instead of the one generated by initdb? dynamic_shared_memory_type defaults to 'posix' if not specified in the config file (on platforms supporting it). If initdb detects that 'posix' can't be used it'll emit a different value. If you're copying the config from 9.3 and your environment doesn't support posix shm that'll cause the above error. I still think dynamic_shared_memory_type should default to 'none' because of such problems With regard to Andres' proposal, I'm not that keen on setting dynamic_shared_memory_type='none' by default. Note that I'm not proposing to disable the whole thing. Just that a unset dynamic_shared_memory_type doesn't configure dsm. Initdb would still configure it after probing. To me, the following should be done: * Make initdb determine the best shm type for this platform and write it into postgresql.conf as it does now. * If no dynamic_shared_memory_type is found in the config, default to none. * Modify the three identical error messages concerned about shm segments to include the shm type instead of always just saying FATAL: could not open shared memory segment * Add a HINT to the POSIX error message: HINT: This might indicate that /dev/shm is not mounted, or its permissions do not allow the database user to create files there Sounds like a sane plan to me. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Shared memory changes in 9.4?
Re: Andres Freund 2014-06-12 20140612094112.gz8...@alap3.anarazel.de
* Make initdb determine the best shm type for this platform and write
it into postgresql.conf as it does now.
* If no dynamic_shared_memory_type is found in the config, default to
none.
* Modify the three identical error messages concerned about shm
segments to include the shm type instead of always just saying
FATAL: could not open shared memory segment
* Add a HINT to the POSIX error message:
HINT: This might indicate that /dev/shm is not mounted, or its
permissions do not allow the database user to create files there
Sounds like a sane plan to me.
Here are two patches, one that implements the annotated error
messages, and one that selects none as default.
It might also make sense to add a Note that POSIX depends on /dev/shm,
and also a Note that dynamic_shared_memory_type is not related to
the shared_buffers shm segments, which I didn't include here.
Christoph
--
c...@df7cb.de | http://www.df7cb.de/
diff --git a/src/backend/storage/ipc/dsm_impl.c b/src/backend/storage/ipc/dsm_impl.c
new file mode 100644
index 0819641..780e3a5
*** a/src/backend/storage/ipc/dsm_impl.c
--- b/src/backend/storage/ipc/dsm_impl.c
*** dsm_impl_posix(dsm_op op, dsm_handle han
*** 289,296
if (errno != EEXIST)
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not open shared memory segment \%s\: %m,
! name)));
return false;
}
--- 289,299
if (errno != EEXIST)
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not open POSIX shared memory segment \%s\: %m,
! name),
! errhint(This error usually means that /dev/shm is not mounted, or its
! permissions do not allow the database user to create files
! there.)));
return false;
}
*** dsm_impl_posix(dsm_op op, dsm_handle han
*** 313,319
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not stat shared memory segment \%s\: %m,
name)));
return false;
}
--- 316,322
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not stat POSIX shared memory segment \%s\: %m,
name)));
return false;
}
*** dsm_impl_posix(dsm_op op, dsm_handle han
*** 332,338
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not resize shared memory segment %s to %zu bytes: %m,
name, request_size)));
return false;
}
--- 335,341
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not resize POSIX shared memory segment %s to %zu bytes: %m,
name, request_size)));
return false;
}
*** dsm_impl_posix(dsm_op op, dsm_handle han
*** 358,364
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not unmap shared memory segment \%s\: %m,
name)));
return false;
}
--- 361,367
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not unmap POSIX shared memory segment \%s\: %m,
name)));
return false;
}
*** dsm_impl_posix(dsm_op op, dsm_handle han
*** 382,388
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not map shared memory segment \%s\: %m,
name)));
return false;
}
--- 385,391
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not map POSIX shared memory segment \%s\: %m,
name)));
return false;
}
*** dsm_impl_sysv(dsm_op op, dsm_handle hand
*** 512,518
errno = save_errno;
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not get shared memory segment: %m)));
}
return false;
}
--- 515,521
errno = save_errno;
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not get System V shared memory segment: %m)));
}
return false;
}
*** dsm_impl_sysv(dsm_op op, dsm_handle hand
*** 530,536
{
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not unmap shared memory segment \%s\: %m,
name)));
return false;
}
--- 533,539
{
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not unmap System V shared memory segment \%s\: %m,
name)));
return false;
}
*** dsm_impl_sysv(dsm_op op, dsm_handle hand
*** 540,546
{
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could not remove shared memory segment \%s\: %m,
name)));
return false;
}
--- 543,549
{
ereport(elevel,
(errcode_for_dynamic_shared_memory(),
! errmsg(could
Re: [HACKERS] Shared memory changes in 9.4?
On Thu, Jun 12, 2014 at 5:41 AM, Andres Freund and...@2ndquadrant.com wrote: With regard to Andres' proposal, I'm not that keen on setting dynamic_shared_memory_type='none' by default. Note that I'm not proposing to disable the whole thing. Just that a unset dynamic_shared_memory_type doesn't configure dsm. Initdb would still configure it after probing. OK, I misunderstood your position; thanks for clarifying. I think that would be OK with me. To some degree, I think that the test setup is broken by design: while we try to maintain backward-compatibility for postgresql.conf files, there's never been any guarantee that an old postgresql.conf file will work on a newer server. For example, a whole lot of pre-8.4 users probably had max_fsm_pages and max_fsm_relations configured, and with 8.4, those settings went away. Fixing that kind of thing is an essential part of the upgrade process. That having been said, in this particular case, we can probably ease the pain without much downside by doing as you suggest. The only thing I'm worried about is that people will discover much later that they don't have working dynamic shared memory, and be unhappy about that. Sometimes it's better to complain loudly at the beginning than to leave buried problems for later. But I'll defer to the majority on what to do in his instance. To me, the following should be done: * Make initdb determine the best shm type for this platform and write it into postgresql.conf as it does now. * If no dynamic_shared_memory_type is found in the config, default to none. * Modify the three identical error messages concerned about shm segments to include the shm type instead of always just saying FATAL: could not open shared memory segment * Add a HINT to the POSIX error message: HINT: This might indicate that /dev/shm is not mounted, or its permissions do not allow the database user to create files there Sounds like a sane plan to me. +1 to the rest of this. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Shared memory changes in 9.4?
Hi, On 2014-06-12 11:08:35 -0400, Robert Haas wrote: On Thu, Jun 12, 2014 at 5:41 AM, Andres Freund and...@2ndquadrant.com wrote: With regard to Andres' proposal, I'm not that keen on setting dynamic_shared_memory_type='none' by default. Note that I'm not proposing to disable the whole thing. Just that a unset dynamic_shared_memory_type doesn't configure dsm. Initdb would still configure it after probing. OK, I misunderstood your position; thanks for clarifying. I think that would be OK with me. To some degree, I think that the test setup is broken by design: while we try to maintain backward-compatibility for postgresql.conf files, there's never been any guarantee that an old postgresql.conf file will work on a newer server. For example, a whole lot of pre-8.4 users probably had max_fsm_pages and max_fsm_relations configured, and with 8.4, those settings went away. Fixing that kind of thing is an essential part of the upgrade process. While I think I see where you're coming from I don't think these cases are comparable. In this case commenting out the GUC can prevent the server from starting. That's pretty odd imo. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
