[DOCS] Shared memory usage calculations
I've had a look at the documentation for how much shared memory (in bytes) Postgres uses: http://www.postgresql.org/docs/8.4/static/kernel-resources.html#SHARED-MEMORY-PARAMETERS However, after using these calculations to work out the shared memory usage for my own setup, the numbers I came up with are clearly wrong. Here's what I put my values down as: max_locks_per_transaction = 64 max_connections = 220 autovacuum_max_workers = 3 max_prepared_transactions = 0 block_size = 8192 shared_buffers = 19600 (196MB) wal_block_size = 8192 wal_buffers = 800 (8MB) Using the calculations on that page, it puts my shared memory requirements up to 1.7 terabytes, but my server is running along fine with these settings. If I convert the sizes to kilobytes instead of bytes, it shows a total value of 47 megabytes, which, while not extreme, looks too low. And I am surprised that max_connections has relatively little bearing on the shared memory requirements. Is this right, or should is it more a case of it affecting semaphores? I was under the impression that the maximum number of connections played a large role in deciding shared memory limits. My SHMMAX is currently set at 268435456, which I thought was just enough to cover my config settings. Is the documentation flawed, or am I using the wrong units? Thanks Thom
Re: [DOCS] Shared memory usage calculations
Thom Brown writes: > I've had a look at the documentation for how much shared memory (in bytes) > Postgres uses: > http://www.postgresql.org/docs/8.4/static/kernel-resources.html#SHARED-MEMORY-PARAMETERS > However, after using these calculations to work out the shared memory usage > for my own setup, the numbers I came up with are clearly wrong. Clearly wrong compared to what --- ie, what's the actual size of your shared memory segment? (See ipcs) > If I convert the sizes to kilobytes instead of bytes, it shows a total value > of 47 megabytes, which, while not extreme, looks too low. And I am > surprised that max_connections has relatively little bearing on the shared > memory requirements. Is this right, or should is it more a case of it > affecting semaphores? I was under the impression that the maximum number of > connections played a large role in deciding shared memory limits. No, it doesn't really ... shared_buffers is the first-order component. Also, I'm not sure whether you realize that the native unit for these numbers is mostly *not* bytes. > shared_buffers = 19600 (196MB) > wal_buffers = 800 (8MB) Neither of those parenthetical remarks are correct if that's exactly what you wrote in postgresql.conf. It might be worth checking the way these values are displayed in pg_settings. regards, tom lane -- Sent via pgsql-docs mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-docs
Re: [DOCS] Shared memory usage calculations
2009/10/10 Tom Lane > Thom Brown writes: > > I've had a look at the documentation for how much shared memory (in > bytes) > > Postgres uses: > > > http://www.postgresql.org/docs/8.4/static/kernel-resources.html#SHARED-MEMORY-PARAMETERS > > However, after using these calculations to work out the shared memory > usage > > for my own setup, the numbers I came up with are clearly wrong. > > Clearly wrong compared to what --- ie, what's the actual size of your > shared memory segment? (See ipcs) > I don't actually know the size of my shared memory segment. "sysctl -a | grep -i shmseg" doesn't return anything as it hasn't been set, and not sure how to find out the default value. I didn't see that mentioned in the calculations so I didn't take it into account anywhere. But 1.7TB shared memory can't be right as the entire system memory is only 720MB. > > > If I convert the sizes to kilobytes instead of bytes, it shows a total > value > > of 47 megabytes, which, while not extreme, looks too low. And I am > > surprised that max_connections has relatively little bearing on the > shared > > memory requirements. Is this right, or should is it more a case of it > > affecting semaphores? I was under the impression that the maximum number > of > > connections played a large role in deciding shared memory limits. > > No, it doesn't really ... shared_buffers is the first-order component. > Also, I'm not sure whether you realize that the native unit for these > numbers is mostly *not* bytes. > > > shared_buffers = 19600 (196MB) > > wal_buffers = 800 (8MB) > > Neither of those parenthetical remarks are correct if that's exactly > what you wrote in postgresql.conf. It might be worth checking the > way these values are displayed in pg_settings. > The parenthesised values are what I have in postgresql.conf. I take it these are supposed to be mebibytes then? When I check the value in pg_settings, it's set at 25088 with the unit set at 8kB is 196 mebibytes. It also shows wal_buffers at 1024 with 8kB units, which would be 8 mebibytes. I gather I'm calculating the final value wrong though. :/ Thom
Re: [DOCS] Shared memory usage calculations
Thom Brown writes: > 2009/10/10 Tom Lane >> Clearly wrong compared to what --- ie, what's the actual size of your >> shared memory segment? (See ipcs) > I don't actually know the size of my shared memory segment. "sysctl -a | > grep -i shmseg" doesn't return anything as it hasn't been set, and not sure > how to find out the default value. There is no "default value", we are talking about the actual size of an actual memory object. Which you find out with ipcs, not sysctl. For instance $ ipcs -a -- Shared Memory Segments keyshmid owner perms bytes nattch status 0x00530201 3080195tgl 60037904384 4 -- Semaphore Arrays keysemid owner perms nsems 0x00530201 20873216 tgl 60017 0x00530202 20905985 tgl 60017 0x00530203 20938754 tgl 60017 0x00530204 20971523 tgl 60017 0x00530205 21004292 tgl 60017 0x00530206 21037061 tgl 60017 0x00530207 21069830 tgl 60017 -- Message Queues keymsqid owner perms used-bytes messages (I'm running this postmaster as "tgl", not the more usual "postgres". On some platforms you might need to be the postgres user or root to see shared objects belonging to postgres.) regards, tom lane -- Sent via pgsql-docs mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-docs
Re: [DOCS] Shared memory usage calculations
2009/10/10 Tom Lane > > There is no "default value", we are talking about the actual size of an > actual memory object. Which you find out with ipcs, not sysctl. > For instance > > $ ipcs -a > > -- Shared Memory Segments > keyshmid owner perms bytes nattch status > 0x00530201 3080195tgl 60037904384 4 > > -- Semaphore Arrays > keysemid owner perms nsems > 0x00530201 20873216 tgl 60017 > 0x00530202 20905985 tgl 60017 > 0x00530203 20938754 tgl 60017 > 0x00530204 20971523 tgl 60017 > 0x00530205 21004292 tgl 60017 > 0x00530206 21037061 tgl 60017 > 0x00530207 21069830 tgl 60017 > > -- Message Queues > keymsqid owner perms used-bytes messages > > (I'm running this postmaster as "tgl", not the more usual "postgres". > On some platforms you might need to be the postgres user or root to see > shared objects belonging to postgres.) > >regards, tom lane > My mistake. I've never seen that before. Running it on mine returns the following: -- Shared Memory Segments keyshmid owner perms bytes nattch status 0x0052e2c1 491520 postgres 600235511808 4 -- Semaphore Arrays keysemid owner perms nsems 0x0052e2c1 8585218postgres 60017 0x0052e2c2 8617987postgres 60017 0x0052e2c3 8650756postgres 60017 0x0052e2c4 8683525postgres 60017 0x0052e2c5 8716294postgres 60017 0x0052e2c6 8749063postgres 60017 0x0052e2c7 8781832postgres 60017 0x0052e2c8 8814601postgres 60017 0x0052e2c9 8847370postgres 60017 0x0052e2ca 8880139postgres 60017 0x0052e2cb 8912908postgres 60017 0x0052e2cc 8945677postgres 60017 0x0052e2cd 8978446postgres 60017 0x0052e2ce 9011215postgres 60017 0x0052e2cf 9043984postgres 60017 0x0052e2d0 9076753postgres 60017 0x0052e2d1 9109522postgres 60017 0x0052e2d2 9142291postgres 60017 0x0052e2d3 9175060postgres 60017 0x0052e2d4 9207829postgres 60017 0x0052e2d5 9240598postgres 60017 0x0052e2d6 9273367postgres 60017 0x0052e2d7 9306136postgres 60017 0x0052e2d8 9338905postgres 60017 0x0052e2d9 9371674postgres 60017 0x0052e2da 9404443postgres 60017 0x0052e2db 9437212postgres 60017 0x0052e2dc 9469981postgres 60017 0x0052e2dd 9502750postgres 60017 0x0052e2de 9535519postgres 60017 0x0052e2df 9568288postgres 60017 0x0052e2e0 9601057postgres 60017 0x0052e2e1 9633826postgres 60017 0x0052e2e2 9666595postgres 60017 0x0052e2e3 9699364postgres 60017 0x0052e2e4 9732133postgres 60017 0x0052e2e5 9764902postgres 60017 0x0052e2e6 9797671postgres 60017 0x0052e2e7 9830440postgres 60017 0x0052e2e8 9863209postgres 60017 0x0052e2e9 9895978postgres 60017 0x0052e2ea 9928747postgres 60017 0x0052e2eb 9961516postgres 60017 0x0052e2ec 9994285postgres 60017 0x0052e2ed 10027054 postgres 60017 0x0052e2ee 10059823 postgres 60017 0x0052e2ef 10092592 postgres 60017 0x0052e2f0 10125361 postgres 60017 0x0052e2f1 10158130 postgres 60017 0x0052e2f2 10190899 postgres 60017 0x0052e2f3 10223668 postgres 60017 0x0052e2f4 10256437 postgres 60017 0x0052e2f5 10289206 postgres 60017 0x0052e2f6 10321975 postgres 60017 0x0052e2f7 10354744 postgres 60017 0x0052e2f8 10387513 postgres 60017 0x0052e2f9 10420282 postgres 60017 0x0052e2fa 10453051 postgres 60017 0x0052e2fb 10485820 postgres 60017 0x0052e2fc 10518589 postgres 60017 -- Message Queues keymsqid owner perms used-bytes messages In any case, I gather that the documentation isn't inaccurate, rather my interpretation is. And having put my number of connections right up to 1500, Postgres still starts fine, so I'm not sure why I thought it would cause the shared memory limit to be reached so easily. Thanks Thom
Re: [DOCS] Shared memory usage calculations
Thom Brown writes: > My mistake. I've never seen that before. Running it on mine returns the > following: > -- Shared Memory Segments > keyshmid owner perms bytes nattch status > 0x0052e2c1 491520 postgres 600235511808 4 Okay, so your actual shmem size is 235MB, of which 196 is shared_buffers and 8 is wal_buffers, and the rest is related to max_connections and various third-order parameters. I'm not sure how precise the values given in the docs are, but last I checked they weren't hugely wrong. regards, tom lane -- Sent via pgsql-docs mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-docs
