Re: [HACKERS] Incorrect formula for SysV IPC parameters
Thank you for pushing this. At Tue, 16 Feb 2016 15:07:32 +0900, Fujii Masao wrote in > > Attached patch uses the above formula for 9.3. I'm thinking to push your > > patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3. > > Comments? > > Pushed. Thanks for the report and patches! regards, -- Kyotaro Horiguchi NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Incorrect formula for SysV IPC parameters
On Fri, Feb 12, 2016 at 11:19 PM, Fujii Masao wrote: > On Fri, Feb 5, 2016 at 2:17 PM, Kyotaro HORIGUCHI > wrote: >> At Thu, 4 Feb 2016 21:43:04 +0900, Fujii Masao wrote >> in >>> On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI >>> wrote: >>> > Hello, I found that the formulas to calculate SEMMNI and SEMMNS >>> > are incorrect in 9.2 and later. >>> > >>> > http://www.postgresql.org/docs/9.5/static/kernel-resources.html >>> > >>> > But actually the number of semaphores PostgreSQL needs is >>> > calculated as following in 9.4 and later. >> ... >>> > So, the formula for SEMMNI should be >>> > >>> > ceil((max_connections + autovacuum_max_workers + max_worker_processes + >>> > 5) / 16) >>> > >>> > and SEMMNS should have the same fix. >>> > >>> > >>> > In 9.3 and 9.2, the documentation says the same thing but >> ... >>> > ceil((max_connections + autovacuum_max_workers + 5) / 16) >>> > >>> > In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct. >>> >>> Good catch! >> >> Thanks. >> >>> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS >>> under the Table 17-1. >> >> Oops! Thank you for pointing it out. >> >> The original description doesn't mention the magic-number ('5' in >> the above formulas, which previously was '4') so I haven't added >> anything about it. >> >> Process of which the number is limited by max_worker_processes is >> called 'background process' (not 'backend worker') in the >> documentation so I used the name to mention it in the additional >> description. >> >> The difference in the body text for 9.2, 9.3 is only a literal >> '4' to '5' in the formula. > > Thanks for updating the patches! > > They look good to me except that the formulas don't include the number of > background processes requesting shared memory access, i.e., > GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following > formula in 9.3? > > ceil((max_connections + autovacuum_max_workers + number of > background proceses + 5) / 16) > > Attached patch uses the above formula for 9.3. I'm thinking to push your > patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3. > Comments? Pushed. Thanks for the report and patches! Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Incorrect formula for SysV IPC parameters
On Mon, Feb 15, 2016 at 2:31 PM, Kyotaro HORIGUCHI wrote: > Thanks for looking at this. > > At Fri, 12 Feb 2016 23:19:45 +0900, Fujii Masao wrote > in >> >> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS >> >> under the Table 17-1. >> > >> > Oops! Thank you for pointing it out. >> > >> > The original description doesn't mention the magic-number ('5' in >> > the above formulas, which previously was '4') so I haven't added >> > anything about it. >> > >> > Process of which the number is limited by max_worker_processes is >> > called 'background process' (not 'backend worker') in the >> > documentation so I used the name to mention it in the additional >> > description. >> > >> > The difference in the body text for 9.2, 9.3 is only a literal >> > '4' to '5' in the formula. >> >> Thanks for updating the patches! >> >> They look good to me except that the formulas don't include the number of >> background processes requesting shared memory access, i.e., >> GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following >> formula in 9.3? >> >> ceil((max_connections + autovacuum_max_workers + number of >> background proceses + 5) / 16) >> >> Attached patch uses the above formula for 9.3. I'm thinking to push your >> patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3. >> Comments? > > One concern is that users don't have any simple way to know how > many bg-workers a server runs in their current configuration. Users need to read the document of the extensions they want to load, to see the number of background worker processes which will be running. > The formula donsn't make sense without it. IMO, documenting "incorrect" formula can cause more troubles. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Incorrect formula for SysV IPC parameters
Thanks for looking at this. At Fri, 12 Feb 2016 23:19:45 +0900, Fujii Masao wrote in > >> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS > >> under the Table 17-1. > > > > Oops! Thank you for pointing it out. > > > > The original description doesn't mention the magic-number ('5' in > > the above formulas, which previously was '4') so I haven't added > > anything about it. > > > > Process of which the number is limited by max_worker_processes is > > called 'background process' (not 'backend worker') in the > > documentation so I used the name to mention it in the additional > > description. > > > > The difference in the body text for 9.2, 9.3 is only a literal > > '4' to '5' in the formula. > > Thanks for updating the patches! > > They look good to me except that the formulas don't include the number of > background processes requesting shared memory access, i.e., > GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following > formula in 9.3? > > ceil((max_connections + autovacuum_max_workers + number of > background proceses + 5) / 16) > > Attached patch uses the above formula for 9.3. I'm thinking to push your > patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3. > Comments? One concern is that users don't have any simple way to know how many bg-workers a server runs in their current configuration. The formula donsn't make sense without it. regards, -- Kyotaro Horiguchi NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Incorrect formula for SysV IPC parameters
On Fri, Feb 5, 2016 at 2:17 PM, Kyotaro HORIGUCHI wrote: > At Thu, 4 Feb 2016 21:43:04 +0900, Fujii Masao wrote > in >> On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI >> wrote: >> > Hello, I found that the formulas to calculate SEMMNI and SEMMNS >> > are incorrect in 9.2 and later. >> > >> > http://www.postgresql.org/docs/9.5/static/kernel-resources.html >> > >> > But actually the number of semaphores PostgreSQL needs is >> > calculated as following in 9.4 and later. > ... >> > So, the formula for SEMMNI should be >> > >> > ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) >> > / 16) >> > >> > and SEMMNS should have the same fix. >> > >> > >> > In 9.3 and 9.2, the documentation says the same thing but > ... >> > ceil((max_connections + autovacuum_max_workers + 5) / 16) >> > >> > In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct. >> >> Good catch! > > Thanks. > >> ISTM that you also need to change the descriptions about SEMMNI and SEMMNS >> under the Table 17-1. > > Oops! Thank you for pointing it out. > > The original description doesn't mention the magic-number ('5' in > the above formulas, which previously was '4') so I haven't added > anything about it. > > Process of which the number is limited by max_worker_processes is > called 'background process' (not 'backend worker') in the > documentation so I used the name to mention it in the additional > description. > > The difference in the body text for 9.2, 9.3 is only a literal > '4' to '5' in the formula. Thanks for updating the patches! They look good to me except that the formulas don't include the number of background processes requesting shared memory access, i.e., GetNumShmemAttachedBgworkers(), in 9.3. Isn't it better to use the following formula in 9.3? ceil((max_connections + autovacuum_max_workers + number of background proceses + 5) / 16) Attached patch uses the above formula for 9.3. I'm thinking to push your patches to 9.2, 9.4, 9.5, master, also push the attached one to 9.3. Comments? Regards, -- Fujii Masao fix_doc_sysvipc_93.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Incorrect formula for SysV IPC parameters
At Thu, 4 Feb 2016 21:43:04 +0900, Fujii Masao wrote in > On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI > wrote: > > Hello, I found that the formulas to calculate SEMMNI and SEMMNS > > are incorrect in 9.2 and later. > > > > http://www.postgresql.org/docs/9.5/static/kernel-resources.html > > > > But actually the number of semaphores PostgreSQL needs is > > calculated as following in 9.4 and later. ... > > So, the formula for SEMMNI should be > > > > ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) > > / 16) > > > > and SEMMNS should have the same fix. > > > > > > In 9.3 and 9.2, the documentation says the same thing but ... > > ceil((max_connections + autovacuum_max_workers + 5) / 16) > > > > In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct. > > Good catch! Thanks. > ISTM that you also need to change the descriptions about SEMMNI and SEMMNS > under the Table 17-1. Oops! Thank you for pointing it out. The original description doesn't mention the magic-number ('5' in the above formulas, which previously was '4') so I haven't added anything about it. Process of which the number is limited by max_worker_processes is called 'background process' (not 'backend worker') in the documentation so I used the name to mention it in the additional description. The difference in the body text for 9.2, 9.3 is only a literal '4' to '5' in the formula. regards, -- Kyotaro Horiguchi NTT Open Source Software Center >From 6ed2f296cc5899f75a2e817f470e5da07bcb0d2c Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Wed, 3 Feb 2016 11:35:43 +0900 Subject: [PATCH] Fix the formula to calculate SEMMNI and SEMMNS in documentation --- doc/src/sgml/runtime.sgml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml index 0db3807..2bc0a91 100644 --- a/doc/src/sgml/runtime.sgml +++ b/doc/src/sgml/runtime.sgml @@ -645,13 +645,13 @@ psql: could not connect to server: No such file or directory SEMMNI Maximum number of semaphore identifiers (i.e., sets) -at least ceil((max_connections + autovacuum_max_workers + 4) / 16) +at least ceil((max_connections + autovacuum_max_workers + 5) / 16) SEMMNS Maximum number of semaphores system-wide -ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 plus room for other applications +ceil((max_connections + autovacuum_max_workers + 5) / 16) * 17 plus room for other applications @@ -712,7 +712,7 @@ psql: could not connect to server: No such file or directory linkend="sysvipc-parameters">). The parameter SEMMNI determines the limit on the number of semaphore sets that can exist on the system at one time. Hence this parameter must be at -least ceil((max_connections + autovacuum_max_workers + 4) / 16). +least ceil((max_connections + autovacuum_max_workers + 5) / 16). Lowering the number of allowed connections is a temporary workaround for failures, which are usually confusingly worded No space -- 1.8.3.1 >From c196c82665e4081a6f90938ac86bb63e4e60e45c Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Wed, 3 Feb 2016 11:07:12 +0900 Subject: [PATCH] Fix the formula to calculate SEMMNI and SEMMNS in documentation --- doc/src/sgml/runtime.sgml | 14 -- 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml index cda05f5..dd42dd0 100644 --- a/doc/src/sgml/runtime.sgml +++ b/doc/src/sgml/runtime.sgml @@ -645,13 +645,13 @@ psql: could not connect to server: No such file or directory SEMMNI Maximum number of semaphore identifiers (i.e., sets) -at least ceil((max_connections + autovacuum_max_workers + 4) / 16) +at least ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16) SEMMNS Maximum number of semaphores system-wide -ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 plus room for other applications +ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16) * 17 plus room for other applications @@ -699,20 +699,22 @@ psql: could not connect to server: No such file or directory PostgreSQL uses one semaphore per allowed connection -() and allowed autovacuum worker -process (), in sets of 16. +(), allowed autovacuum worker process +() and allowed background +process (), in sets of 16. Each such set will also contain a 17th semaphore which contains a magic number, to detect collision with semaphore sets used by other applications. The maximum number of semaphores in the system is set by SEMMNS, which consequently must be at least as high as max_connections plus -autovacuum_max_workers, plus one
Re: [HACKERS] Incorrect formula for SysV IPC parameters
On Wed, Feb 3, 2016 at 12:51 PM, Kyotaro HORIGUCHI wrote: > Hello, I found that the formulas to calculate SEMMNI and SEMMNS > are incorrect in 9.2 and later. > > http://www.postgresql.org/docs/9.5/static/kernel-resources.html > > All of them say that the same thing as following, > > | SEMMNI Maximum number of semaphore identifiers (i.e., sets) > | > | at least ceil((max_connections + autovacuum_max_workers + 4) / 16) > | > | SEMMNSMaximum number of semaphores system-wide > | > | ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 > |plus room for other applications > > But actually the number of semaphores PostgreSQL needs is > calculated as following in 9.4 and later. > > numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4) > MaxConnections = max_connections + autovacuum_max_workers + 1 + >max_worker_processes > > So, the formula for SEMMNI should be > > ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / > 16) > > and SEMMNS should have the same fix. > > > In 9.3 and 9.2, the documentation says the same thing but > actually it is calculated as following, > > numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4) > MaxConnections = max_connections + autovacuum_max_workers + 1 + >GetNumShmemAttachedBgworkers() > > Omitting GetNumShmemAttachedBgworkers, the actual formula is > > ceil((max_connections + autovacuum_max_workers + 5) / 16) > > > In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct. > > > I attached two patches for 9.2-9.3 and 9.4-9.6dev > respectively. patch command complains a bit on applying it on > 9.2. Good catch! ISTM that you also need to change the descriptions about SEMMNI and SEMMNS under the Table 17-1. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Incorrect formula for SysV IPC parameters
Hello, I found that the formulas to calculate SEMMNI and SEMMNS are incorrect in 9.2 and later. http://www.postgresql.org/docs/9.5/static/kernel-resources.html All of them say that the same thing as following, | SEMMNI Maximum number of semaphore identifiers (i.e., sets) | | at least ceil((max_connections + autovacuum_max_workers + 4) / 16) | | SEMMNSMaximum number of semaphores system-wide | | ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 |plus room for other applications But actually the number of semaphores PostgreSQL needs is calculated as following in 9.4 and later. numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4) MaxConnections = max_connections + autovacuum_max_workers + 1 + max_worker_processes So, the formula for SEMMNI should be ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16) and SEMMNS should have the same fix. In 9.3 and 9.2, the documentation says the same thing but actually it is calculated as following, numSemas = MaxConnections + NUM_AUXILIARY_PROCS(=4) MaxConnections = max_connections + autovacuum_max_workers + 1 + GetNumShmemAttachedBgworkers() Omitting GetNumShmemAttachedBgworkers, the actual formula is ceil((max_connections + autovacuum_max_workers + 5) / 16) In 9.1, NUM_AUXILIARY_PROCS is 3 so the documentations is correct. I attached two patches for 9.2-9.3 and 9.4-9.6dev respectively. patch command complains a bit on applying it on 9.2. On the platforforms that doesn't have tas operation needs additional 1024 + 64 semaphores but I understand it as out of scope of the documentation. One concern is that 'at least' and 'plus room for other applications' are mixed in the table 17-1, especially for SEMMNI and SEMMNS. It seems to me that they should be in the same wording, but it is not an actual problem. regards, -- Kyotaro Horiguchi NTT Open Source Software Center >From 6da5ad413dff4724fee75f1ba09013b6033f76ca Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Wed, 3 Feb 2016 11:35:43 +0900 Subject: [PATCH] Fix the formula to calculate SEMMNI and SEMMNS in documentation --- doc/src/sgml/runtime.sgml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml index 0db3807..45579da 100644 --- a/doc/src/sgml/runtime.sgml +++ b/doc/src/sgml/runtime.sgml @@ -645,13 +645,13 @@ psql: could not connect to server: No such file or directory SEMMNI Maximum number of semaphore identifiers (i.e., sets) -at least ceil((max_connections + autovacuum_max_workers + 4) / 16) +at least ceil((max_connections + autovacuum_max_workers + 5) / 16) SEMMNS Maximum number of semaphores system-wide -ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 plus room for other applications +ceil((max_connections + autovacuum_max_workers + 5) / 16) * 17 plus room for other applications -- 1.8.3.1 >From aaf51a50e942edfa35af20a8a1d8b8719155664b Mon Sep 17 00:00:00 2001 From: Kyotaro Horiguchi Date: Wed, 3 Feb 2016 11:19:58 +0900 Subject: [PATCH] Fix the formula to calculate SEMMNI and SEMMNS in documentation --- doc/src/sgml/runtime.sgml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/runtime.sgml b/doc/src/sgml/runtime.sgml index 650d455..4b2e403 100644 --- a/doc/src/sgml/runtime.sgml +++ b/doc/src/sgml/runtime.sgml @@ -645,13 +645,13 @@ psql: could not connect to server: No such file or directory SEMMNI Maximum number of semaphore identifiers (i.e., sets) -at least ceil((max_connections + autovacuum_max_workers + 4) / 16) +at least ceil((max_connections + autovacuum_max_workers max_worker_processes + 5) / 16) SEMMNS Maximum number of semaphores system-wide -ceil((max_connections + autovacuum_max_workers + 4) / 16) * 17 plus room for other applications +ceil((max_connections + autovacuum_max_workers + max_worker_processes + 5) / 16) * 17 plus room for other applications -- 1.8.3.1 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers