Re: [GENERAL] Memory issue on FreeBSD
FYI
http://freebsd.1045724.n5.nabble.com/Postgresql-related-memory-question-td5759467.html
Am 2012-11-07 10:28, schrieb Achilleas Mantzios:
On Τετ 07 Νοε 2012 09:42:47 Frank Broniewski wrote:
Hey, this is really cool. I directly tried the script and there's a line
from the output that caught my eye:
mem_gap_vm: + 8812892160 ( 8404MB) [ 26%] Memory gap: UNKNOWN
is this the shared buffers? I guess so, but I want to confirm my guess ...
Hmm, that would be ideal, (from an understanding perspective) but at least in
my system (FreeBSD-8.3), no.
psql -q -t -c show shared_buffers | grep -v -e '^$' | awk '{print $1}'
3840MB
SYSTEM MEMORY INFORMATION:
mem_gap_vm: +996843520 (950MB) [ 5%] Memory gap: UNKNOWN
$mem_gap_vm = $mem_all - ($mem_wire + $mem_active + $mem_inactive + $mem_cache
+ $mem_free);
mem_all is some rounded and more rationalized version less than hw.physmem : $mem_all =
$sysctl-{vm.stats.vm.v_page_count} * $sysctl-{hw.pagesize};
Anyway, this is not so postgresql related at the moment. The correct thing to
do (since you run production servers on FreeBSD) is to post to the relevant
FreeBSD list and/or forum. freebsd-questi...@freebsd.org and
freebsd-sta...@freebsd.org would be a good start.
Also the forums : http://forums.freebsd.org/forumdisplay.php?f=3
Only after gathering substantial info from there, would it make sense to come
back here and maybe ask more questions.
And since we are observing different percentages of gaps (mine is 5%, yours is
26%), i think maybe you should look into it on the FreeBSD camp.
Please drop the link to the relevant thread there, if you decide to do so.
I would like to follow this.
Thanx!
Frank
Am 2012-11-07 09:26, schrieb Achilleas Mantzios:
Vick,
fantastic script, thanx! FreeBSD sysctl system is awesome!
On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote:
On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski b...@metrico.lu wrote:
and this is after a few hours of running:
Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free
Swap: 4096M Total, 828K Used, 4095M Free
For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with
Postgres 9.0.7 running since June 10, and is heavily pounded on 24x7. The data
+ indexes are about 240GB on disk. This server only runs postgres aside from
the basic system processes.
Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free
Swap: 4096M Total, 272K Used, 4096M Free
I agree with the conclusion that the shared memory segments are confusing the output of
top. There are no memory leaks, and FreeBSD doesn't lose any memory.
There are some scripts floating around that read values from sysctl vm.stats.vm
and format them nicely to tell you how much memory is used up and free. Try
the one referenced here:
http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/
-
Achilleas Mantzios
IT DEPT
-
Achilleas Mantzios
IT DEPT
--
Frank BRONIEWSKI
METRICO s.à r.l.
géomètres
technologies d'information géographique
rue des Romains 36
L-5433 NIEDERDONVEN
tél.: +352 26 74 94 - 28
fax.: +352 26 74 94 99
http://www.metrico.lu
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Memory issue on FreeBSD
Thanx for the link.
I just think that it would be a good idea, instead of posting the links at this
list, to include a
dense but detailed summary of the situation in your machine, and give as much
data as possible.
In short, you might do a quantitative compilation of this thread, and present
it in a nice way
in order to gain more attention.
Also, i think posting to -stable would be a better idea, -questions is for
noobs.
On Παρ 09 Νοε 2012 09:37:14 Frank Broniewski wrote:
FYI
http://freebsd.1045724.n5.nabble.com/Postgresql-related-memory-question-td5759467.html
Am 2012-11-07 10:28, schrieb Achilleas Mantzios:
On Τετ 07 Νοε 2012 09:42:47 Frank Broniewski wrote:
Hey, this is really cool. I directly tried the script and there's a line
from the output that caught my eye:
mem_gap_vm: + 8812892160 ( 8404MB) [ 26%] Memory gap: UNKNOWN
is this the shared buffers? I guess so, but I want to confirm my guess ...
Hmm, that would be ideal, (from an understanding perspective) but at least
in my system (FreeBSD-8.3), no.
psql -q -t -c show shared_buffers | grep -v -e '^$' | awk '{print $1}'
3840MB
SYSTEM MEMORY INFORMATION:
mem_gap_vm: +996843520 (950MB) [ 5%] Memory gap: UNKNOWN
$mem_gap_vm = $mem_all - ($mem_wire + $mem_active + $mem_inactive +
$mem_cache + $mem_free);
mem_all is some rounded and more rationalized version less than hw.physmem
: $mem_all = $sysctl-{vm.stats.vm.v_page_count} *
$sysctl-{hw.pagesize};
Anyway, this is not so postgresql related at the moment. The correct thing
to do (since you run production servers on FreeBSD) is to post to the
relevant
FreeBSD list and/or forum. freebsd-questi...@freebsd.org and
freebsd-sta...@freebsd.org would be a good start.
Also the forums : http://forums.freebsd.org/forumdisplay.php?f=3
Only after gathering substantial info from there, would it make sense to
come back here and maybe ask more questions.
And since we are observing different percentages of gaps (mine is 5%, yours
is 26%), i think maybe you should look into it on the FreeBSD camp.
Please drop the link to the relevant thread there, if you decide to do so.
I would like to follow this.
Thanx!
Frank
Am 2012-11-07 09:26, schrieb Achilleas Mantzios:
Vick,
fantastic script, thanx! FreeBSD sysctl system is awesome!
On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote:
On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski b...@metrico.lu wrote:
and this is after a few hours of running:
Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free
Swap: 4096M Total, 828K Used, 4095M Free
For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server,
with Postgres 9.0.7 running since June 10, and is heavily pounded on
24x7. The data + indexes are about 240GB on disk. This server only runs
postgres aside from the basic system processes.
Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M
Free
Swap: 4096M Total, 272K Used, 4096M Free
I agree with the conclusion that the shared memory segments are confusing
the output of top. There are no memory leaks, and FreeBSD doesn't lose
any memory.
There are some scripts floating around that read values from sysctl
vm.stats.vm and format them nicely to tell you how much memory is used up
and free. Try the one referenced here:
http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/
-
Achilleas Mantzios
IT DEPT
-
Achilleas Mantzios
IT DEPT
-
Achilleas Mantzios
IT DEPT
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Memory issue on FreeBSD
Vick, fantastic script, thanx! FreeBSD sysctl system is awesome! On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote: On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski b...@metrico.lu wrote: and this is after a few hours of running: Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free Swap: 4096M Total, 828K Used, 4095M Free For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with Postgres 9.0.7 running since June 10, and is heavily pounded on 24x7. The data + indexes are about 240GB on disk. This server only runs postgres aside from the basic system processes. Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free Swap: 4096M Total, 272K Used, 4096M Free I agree with the conclusion that the shared memory segments are confusing the output of top. There are no memory leaks, and FreeBSD doesn't lose any memory. There are some scripts floating around that read values from sysctl vm.stats.vm and format them nicely to tell you how much memory is used up and free. Try the one referenced here: http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/ - Achilleas Mantzios IT DEPT
Re: [GENERAL] Memory issue on FreeBSD
Hey, this is really cool. I directly tried the script and there's a line from the output that caught my eye: mem_gap_vm: + 8812892160 ( 8404MB) [ 26%] Memory gap: UNKNOWN is this the shared buffers? I guess so, but I want to confirm my guess ... Frank Am 2012-11-07 09:26, schrieb Achilleas Mantzios: Vick, fantastic script, thanx! FreeBSD sysctl system is awesome! On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote: On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski b...@metrico.lu wrote: and this is after a few hours of running: Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free Swap: 4096M Total, 828K Used, 4095M Free For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with Postgres 9.0.7 running since June 10, and is heavily pounded on 24x7. The data + indexes are about 240GB on disk. This server only runs postgres aside from the basic system processes. Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free Swap: 4096M Total, 272K Used, 4096M Free I agree with the conclusion that the shared memory segments are confusing the output of top. There are no memory leaks, and FreeBSD doesn't lose any memory. There are some scripts floating around that read values from sysctl vm.stats.vm and format them nicely to tell you how much memory is used up and free. Try the one referenced here: http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/ - Achilleas Mantzios IT DEPT -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Memory issue on FreeBSD
On Τετ 07 Νοε 2012 09:42:47 Frank Broniewski wrote:
Hey, this is really cool. I directly tried the script and there's a line
from the output that caught my eye:
mem_gap_vm: + 8812892160 ( 8404MB) [ 26%] Memory gap: UNKNOWN
is this the shared buffers? I guess so, but I want to confirm my guess ...
Hmm, that would be ideal, (from an understanding perspective) but at least in
my system (FreeBSD-8.3), no.
psql -q -t -c show shared_buffers | grep -v -e '^$' | awk '{print $1}'
3840MB
SYSTEM MEMORY INFORMATION:
mem_gap_vm: +996843520 (950MB) [ 5%] Memory gap: UNKNOWN
$mem_gap_vm = $mem_all - ($mem_wire + $mem_active + $mem_inactive + $mem_cache
+ $mem_free);
mem_all is some rounded and more rationalized version less than hw.physmem :
$mem_all = $sysctl-{vm.stats.vm.v_page_count} * $sysctl-{hw.pagesize};
Anyway, this is not so postgresql related at the moment. The correct thing to
do (since you run production servers on FreeBSD) is to post to the relevant
FreeBSD list and/or forum. freebsd-questi...@freebsd.org and
freebsd-sta...@freebsd.org would be a good start.
Also the forums : http://forums.freebsd.org/forumdisplay.php?f=3
Only after gathering substantial info from there, would it make sense to come
back here and maybe ask more questions.
And since we are observing different percentages of gaps (mine is 5%, yours is
26%), i think maybe you should look into it on the FreeBSD camp.
Please drop the link to the relevant thread there, if you decide to do so.
I would like to follow this.
Thanx!
Frank
Am 2012-11-07 09:26, schrieb Achilleas Mantzios:
Vick,
fantastic script, thanx! FreeBSD sysctl system is awesome!
On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote:
On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski b...@metrico.lu wrote:
and this is after a few hours of running:
Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free
Swap: 4096M Total, 828K Used, 4095M Free
For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with
Postgres 9.0.7 running since June 10, and is heavily pounded on 24x7. The
data + indexes are about 240GB on disk. This server only runs postgres
aside from the basic system processes.
Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free
Swap: 4096M Total, 272K Used, 4096M Free
I agree with the conclusion that the shared memory segments are confusing
the output of top. There are no memory leaks, and FreeBSD doesn't lose
any memory.
There are some scripts floating around that read values from sysctl
vm.stats.vm and format them nicely to tell you how much memory is used up
and free. Try the one referenced here:
http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/
-
Achilleas Mantzios
IT DEPT
-
Achilleas Mantzios
IT DEPT
Re: [GENERAL] Memory issue on FreeBSD
Vick, fantastic script, thanx! FreeBSD sysctl system is awesome! On Τρι 06 Νοε 2012 14:33:43 Vick Khera wrote: On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski b...@metrico.lu wrote: and this is after a few hours of running: Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free Swap: 4096M Total, 828K Used, 4095M Free For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with Postgres 9.0.7 running since June 10, and is heavily pounded on 24x7. The data + indexes are about 240GB on disk. This server only runs postgres aside from the basic system processes. Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free Swap: 4096M Total, 272K Used, 4096M Free I agree with the conclusion that the shared memory segments are confusing the output of top. There are no memory leaks, and FreeBSD doesn't lose any memory. There are some scripts floating around that read values from sysctl vm.stats.vm and format them nicely to tell you how much memory is used up and free. Try the one referenced here: http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/ - Achilleas Mantzios IT DEPT
Re: [GENERAL] Memory issue on FreeBSD
On Mon, Nov 5, 2012 at 10:11 AM, Frank Broniewski b...@metrico.lu wrote: and this is after a few hours of running: Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free Swap: 4096M Total, 828K Used, 4095M Free For comparison, here is the output of a 32GB FreeBSD 9.0/amd64 server, with Postgres 9.0.7 running since June 10, and is heavily pounded on 24x7. The data + indexes are about 240GB on disk. This server only runs postgres aside from the basic system processes. Mem: 231M Active, 21G Inact, 3777M Wired, 1009M Cache, 3285M Buf, 191M Free Swap: 4096M Total, 272K Used, 4096M Free I agree with the conclusion that the shared memory segments are confusing the output of top. There are no memory leaks, and FreeBSD doesn't lose any memory. There are some scripts floating around that read values from sysctl vm.stats.vm and format them nicely to tell you how much memory is used up and free. Try the one referenced here: http://www.cyberciti.biz/faq/freebsd-command-to-get-ram-information/
[GENERAL] Memory issue on FreeBSD
Hi, I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. Usually I use top to examine the memory usage of the system. After a while, a part, approximately 5GB, vanish from top, so that the memory rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again available, but then it's already slightly decreasing. It's a standalone database server. It has an OpenStreetMap world database running with 353GB data (with indices). Some system information: # uname -r 9.0-RELEASE-p3 # pg_ctl --version pg_ctl (PostgreSQL) 9.1.6 # cat /boot/loader.conf ... kern.ipc.semmni=256 kern.ipc.semmns=512 kern.ipc.semmnu=256 kern.ipc.semumr=200 vm.pmap.shpgperproc=400 vm.pmap.pv_entry_max=50331648 ... # cat /pgdata/data/postgresql.conf ... default_statistics_target = 50 # pgtune wizard 2012-04-04 maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 constraint_exclusion = on # pgtune wizard 2012-04-04 checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 effective_cache_size = 24GB # pgtune wizard 2012-04-04 work_mem = 768MB # pgtune wizard 2012-04-04 wal_buffers = 16MB # pgtune wizard 2012-04-04 checkpoint_segments = 60 # 20 shared_buffers = 8GB # pgtune wizard 2012-04-04 max_connections = 100 synchronous_commit = off So any help finding out why my system looses some RAM is greatly appreciated :-) If more information is needed I will gladly provide it. Frank -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Memory issue on FreeBSD
ipcs in FreeBSD is a little ... tricky. ipcs -M ipcs -m ipcs -am could be your friends On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: Hi, I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. Usually I use top to examine the memory usage of the system. After a while, a part, approximately 5GB, vanish from top, so that the memory rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again available, but then it's already slightly decreasing. It's a standalone database server. It has an OpenStreetMap world database running with 353GB data (with indices). Some system information: # uname -r 9.0-RELEASE-p3 # pg_ctl --version pg_ctl (PostgreSQL) 9.1.6 # cat /boot/loader.conf ... kern.ipc.semmni=256 kern.ipc.semmns=512 kern.ipc.semmnu=256 kern.ipc.semumr=200 vm.pmap.shpgperproc=400 vm.pmap.pv_entry_max=50331648 ... # cat /pgdata/data/postgresql.conf ... default_statistics_target = 50 # pgtune wizard 2012-04-04 maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 constraint_exclusion = on # pgtune wizard 2012-04-04 checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 effective_cache_size = 24GB # pgtune wizard 2012-04-04 work_mem = 768MB # pgtune wizard 2012-04-04 wal_buffers = 16MB # pgtune wizard 2012-04-04 checkpoint_segments = 60 # 20 shared_buffers = 8GB # pgtune wizard 2012-04-04 max_connections = 100 synchronous_commit = off So any help finding out why my system looses some RAM is greatly appreciated :-) If more information is needed I will gladly provide it. Frank - Achilleas Mantzios IT DEPT -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Memory issue on FreeBSD
(scrap my previous internal email (hence fake) address this one is correct : sorry for that) You can stop pgsql, start it and then watch out for the increase in SEGSZ values. I pretty much think they are in bytes. I am pretty confident that this value depicts the shared_buffers size in bytes. On Δευ 05 Νοε 2012 13:14:37 Achilleas Mantzios wrote: ipcs in FreeBSD is a little ... tricky. ipcs -M ipcs -m ipcs -am could be your friends On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: Hi, I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. Usually I use top to examine the memory usage of the system. After a while, a part, approximately 5GB, vanish from top, so that the memory rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again available, but then it's already slightly decreasing. It's a standalone database server. It has an OpenStreetMap world database running with 353GB data (with indices). Some system information: # uname -r 9.0-RELEASE-p3 # pg_ctl --version pg_ctl (PostgreSQL) 9.1.6 # cat /boot/loader.conf ... kern.ipc.semmni=256 kern.ipc.semmns=512 kern.ipc.semmnu=256 kern.ipc.semumr=200 vm.pmap.shpgperproc=400 vm.pmap.pv_entry_max=50331648 ... # cat /pgdata/data/postgresql.conf ... default_statistics_target = 50 # pgtune wizard 2012-04-04 maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 constraint_exclusion = on # pgtune wizard 2012-04-04 checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 effective_cache_size = 24GB # pgtune wizard 2012-04-04 work_mem = 768MB # pgtune wizard 2012-04-04 wal_buffers = 16MB # pgtune wizard 2012-04-04 checkpoint_segments = 60 # 20 shared_buffers = 8GB # pgtune wizard 2012-04-04 max_connections = 100 synchronous_commit = off So any help finding out why my system looses some RAM is greatly appreciated :-) If more information is needed I will gladly provide it. Frank - Achilleas Mantzios IT DEPT - Achilleas Mantzios IT DEPT -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Memory issue on FreeBSD
Hi, thank you for your feedback. I had a look at those commands and their output, especially in conjunction with the SEGSZ value from icps -am Here's an example output: # ipcs -am Shared Memory: T ID KEY MODEOWNERGROUPCREATOR CGROUP NATTCHSEGSZ CPID LPID ATIME DTIMECTIME m 262144 5432001 --rw--- pgsqlpgsqlpgsqlpgsql 12 88139939844551245512 13:49:28 14:31:29 13:49:28 but frankly this tells me nothing. I can tell that the value SEGSZ is right from the start 8813993984 and it doesn't change anymore. The only value that changes is the NATTCH value, I observed a range from 8 to 36 there. I agree that the SEGSZ value matches the 8GB shared buffer, but how can I make the connection of my 5GB missing in top? I wonder if this might be the maintenance_work_mem, which is set to 4GB? Many thanks, Frank Am 2012-11-05 12:14, schrieb Achilleas Mantzios: ipcs in FreeBSD is a little ... tricky. ipcs -M ipcs -m ipcs -am could be your friends On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: Hi, I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. Usually I use top to examine the memory usage of the system. After a while, a part, approximately 5GB, vanish from top, so that the memory rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again available, but then it's already slightly decreasing. It's a standalone database server. It has an OpenStreetMap world database running with 353GB data (with indices). Some system information: # uname -r 9.0-RELEASE-p3 # pg_ctl --version pg_ctl (PostgreSQL) 9.1.6 # cat /boot/loader.conf ... kern.ipc.semmni=256 kern.ipc.semmns=512 kern.ipc.semmnu=256 kern.ipc.semumr=200 vm.pmap.shpgperproc=400 vm.pmap.pv_entry_max=50331648 ... # cat /pgdata/data/postgresql.conf ... default_statistics_target = 50 # pgtune wizard 2012-04-04 maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 constraint_exclusion = on # pgtune wizard 2012-04-04 checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 effective_cache_size = 24GB # pgtune wizard 2012-04-04 work_mem = 768MB # pgtune wizard 2012-04-04 wal_buffers = 16MB # pgtune wizard 2012-04-04 checkpoint_segments = 60 # 20 shared_buffers = 8GB # pgtune wizard 2012-04-04 max_connections = 100 synchronous_commit = off So any help finding out why my system looses some RAM is greatly appreciated :-) If more information is needed I will gladly provide it. Frank - Achilleas Mantzios IT DEPT -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Memory issue on FreeBSD
How do you measure that smth is missing from top? What values do you add? I am currently running 8.3 but we shouldn't be so far apart top-wise. What is the reading under SIZE and RES in top for all postgresql processes? Take note that shared mem should be recorded for each and every postmaster running. On Δευ 05 Νοε 2012 14:36:44 Frank Broniewski wrote: Hi, thank you for your feedback. I had a look at those commands and their output, especially in conjunction with the SEGSZ value from icps -am Here's an example output: # ipcs -am Shared Memory: T ID KEY MODEOWNERGROUPCREATOR CGROUP NATTCHSEGSZ CPID LPID ATIME DTIMECTIME m 262144 5432001 --rw--- pgsqlpgsqlpgsqlpgsql 12 88139939844551245512 13:49:28 14:31:29 13:49:28 but frankly this tells me nothing. I can tell that the value SEGSZ is right from the start 8813993984 and it doesn't change anymore. The only value that changes is the NATTCH value, I observed a range from 8 to 36 there. I agree that the SEGSZ value matches the 8GB shared buffer, but how can I make the connection of my 5GB missing in top? I wonder if this might be the maintenance_work_mem, which is set to 4GB? Many thanks, Frank Am 2012-11-05 12:14, schrieb Achilleas Mantzios: ipcs in FreeBSD is a little ... tricky. ipcs -M ipcs -m ipcs -am could be your friends On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: Hi, I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. Usually I use top to examine the memory usage of the system. After a while, a part, approximately 5GB, vanish from top, so that the memory rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again available, but then it's already slightly decreasing. It's a standalone database server. It has an OpenStreetMap world database running with 353GB data (with indices). Some system information: # uname -r 9.0-RELEASE-p3 # pg_ctl --version pg_ctl (PostgreSQL) 9.1.6 # cat /boot/loader.conf ... kern.ipc.semmni=256 kern.ipc.semmns=512 kern.ipc.semmnu=256 kern.ipc.semumr=200 vm.pmap.shpgperproc=400 vm.pmap.pv_entry_max=50331648 ... # cat /pgdata/data/postgresql.conf ... default_statistics_target = 50 # pgtune wizard 2012-04-04 maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 constraint_exclusion = on # pgtune wizard 2012-04-04 checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 effective_cache_size = 24GB # pgtune wizard 2012-04-04 work_mem = 768MB # pgtune wizard 2012-04-04 wal_buffers = 16MB # pgtune wizard 2012-04-04 checkpoint_segments = 60 # 20 shared_buffers = 8GB # pgtune wizard 2012-04-04 max_connections = 100 synchronous_commit = off So any help finding out why my system looses some RAM is greatly appreciated :-) If more information is needed I will gladly provide it. Frank - Achilleas Mantzios IT DEPT - Achilleas Mantzios IT DEPT -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Memory issue on FreeBSD
Hi, I just add the different memory values together (minus the buffers). Usually this sums up (+/-) to the installed memory size, at least on my other machines. I found a thread similar to my problem here [1], but no solution. I don't mind top showing false values, but if there's a larger problem behind this, then I really want to solve it. Top is really just an indicator for this issue, it's also visible in my munin stats [2] Below is a output _without_ postgresql running: Mem: 59M Active, 17G Inact, 3953M Wired, 1325M Cache, 3283M Buf, 8663M Free Swap: 4096M Total, 828K Used, 4095M Free and this is after a few hours of running: Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free Swap: 4096M Total, 828K Used, 4095M Free some memory related sysctl values: hw.realmem: 34879832064 hw.physmem: 34322804736 hw.usermem: 30161108992 # sysctl vm.vmtotal vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) === Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 70) Virtual Memory: (Total: 1084659688K Active: 10400940K) Real Memory:(Total: 1616176K Active: 1349052K) Shared Virtual Memory: (Total: 60840K Active: 14132K) Shared Real Memory: (Total: 11644K Active: 8388K) Free Memory Pages: 7263972K [1] http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html [2] http://www.gis-hosting.lu/monitor/munin/metrico/bilbo.metrico/memory.html Am 2012-11-05 15:21, schrieb Achilleas Mantzios: How do you measure that smth is missing from top? What values do you add? I am currently running 8.3 but we shouldn't be so far apart top-wise. What is the reading under SIZE and RES in top for all postgresql processes? Take note that shared mem should be recorded for each and every postmaster running. On Δευ 05 Νοε 2012 14:36:44 Frank Broniewski wrote: Hi, thank you for your feedback. I had a look at those commands and their output, especially in conjunction with the SEGSZ value from icps -am Here's an example output: # ipcs -am Shared Memory: T ID KEY MODEOWNERGROUPCREATOR CGROUP NATTCHSEGSZ CPID LPID ATIME DTIMECTIME m 262144 5432001 --rw--- pgsqlpgsqlpgsqlpgsql 12 88139939844551245512 13:49:28 14:31:29 13:49:28 but frankly this tells me nothing. I can tell that the value SEGSZ is right from the start 8813993984 and it doesn't change anymore. The only value that changes is the NATTCH value, I observed a range from 8 to 36 there. I agree that the SEGSZ value matches the 8GB shared buffer, but how can I make the connection of my 5GB missing in top? I wonder if this might be the maintenance_work_mem, which is set to 4GB? Many thanks, Frank Am 2012-11-05 12:14, schrieb Achilleas Mantzios: ipcs in FreeBSD is a little ... tricky. ipcs -M ipcs -m ipcs -am could be your friends On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: Hi, I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. Usually I use top to examine the memory usage of the system. After a while, a part, approximately 5GB, vanish from top, so that the memory rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again available, but then it's already slightly decreasing. It's a standalone database server. It has an OpenStreetMap world database running with 353GB data (with indices). Some system information: # uname -r 9.0-RELEASE-p3 # pg_ctl --version pg_ctl (PostgreSQL) 9.1.6 # cat /boot/loader.conf ... kern.ipc.semmni=256 kern.ipc.semmns=512 kern.ipc.semmnu=256 kern.ipc.semumr=200 vm.pmap.shpgperproc=400 vm.pmap.pv_entry_max=50331648 ... # cat /pgdata/data/postgresql.conf ... default_statistics_target = 50 # pgtune wizard 2012-04-04 maintenance_work_mem = 4GB # pgtune wizard 2012-04-04 constraint_exclusion = on # pgtune wizard 2012-04-04 checkpoint_completion_target = 0.9 # pgtune wizard 2012-04-04 effective_cache_size = 24GB # pgtune wizard 2012-04-04 work_mem = 768MB # pgtune wizard 2012-04-04 wal_buffers = 16MB # pgtune wizard 2012-04-04 checkpoint_segments = 60 # 20 shared_buffers = 8GB # pgtune wizard 2012-04-04 max_connections = 100 synchronous_commit = off So any help finding out why my system looses some RAM is greatly appreciated :-) If more information is needed I will gladly provide it. Frank - Achilleas Mantzios IT DEPT - Achilleas Mantzios IT DEPT -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
Re: [GENERAL] Memory issue on FreeBSD
Since the top reporting goes back to normal when postgresql is stopped , and since postgresql is special due to the use of IPC, i would be inclined to think that the culprit here is the shared memory. I don't know where maintenance_work_mem really lives (process normal address space or IPC shared mem) and if that makes any difference. If it is possible you might play with those two values and see if anything changes. Currently i have : maintenance_work_mem = 960MB # pgtune wizard 2012-11-01 shared_buffers = 3840MB # pgtune wizard 2012-11-01 top: last pid: 74896; load averages: 0.02, 0.08, 0.08 up 4+06:20:31 18:14:19 187 processes: 1 running, 172 sleeping, 14 zombie CPU: % user, % nice, % system, % interrupt, % idle Mem: 4064M Active, 8111M Inact, 2014M Wired, 322M Cache, 1645M Buf, 1106M Free Swap: 8000M Total, 608K Used, 7999M Free hw.physmem: 17144205312 hw.usermem: 15028662272 hw.realmem: 17985175552 top (excluding Buf) amounts to 15617 Megs while physmem shows as 16349 Megs but as i said i run 8.3 on AMD64 and pgsql 9.2.1 On Δευ 05 Νοε 2012 16:11:39 Frank Broniewski wrote: Hi, I just add the different memory values together (minus the buffers). Usually this sums up (+/-) to the installed memory size, at least on my other machines. I found a thread similar to my problem here [1], but no solution. I don't mind top showing false values, but if there's a larger problem behind this, then I really want to solve it. Top is really just an indicator for this issue, it's also visible in my munin stats [2] Below is a output _without_ postgresql running: Mem: 59M Active, 17G Inact, 3953M Wired, 1325M Cache, 3283M Buf, 8663M Free Swap: 4096M Total, 828K Used, 4095M Free and this is after a few hours of running: Mem: 91M Active, 17G Inact, 3983M Wired, 1526M Cache, 3283M Buf, 155M Free Swap: 4096M Total, 828K Used, 4095M Free some memory related sysctl values: hw.realmem: 34879832064 hw.physmem: 34322804736 hw.usermem: 30161108992 # sysctl vm.vmtotal vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) === Processes:(RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 70) Virtual Memory: (Total: 1084659688K Active: 10400940K) Real Memory: (Total: 1616176K Active: 1349052K) Shared Virtual Memory:(Total: 60840K Active: 14132K) Shared Real Memory: (Total: 11644K Active: 8388K) Free Memory Pages:7263972K [1] http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html [2] http://www.gis-hosting.lu/monitor/munin/metrico/bilbo.metrico/memory.html Am 2012-11-05 15:21, schrieb Achilleas Mantzios: How do you measure that smth is missing from top? What values do you add? I am currently running 8.3 but we shouldn't be so far apart top-wise. What is the reading under SIZE and RES in top for all postgresql processes? Take note that shared mem should be recorded for each and every postmaster running. On Δευ 05 Νοε 2012 14:36:44 Frank Broniewski wrote: Hi, thank you for your feedback. I had a look at those commands and their output, especially in conjunction with the SEGSZ value from icps -am Here's an example output: # ipcs -am Shared Memory: T ID KEY MODEOWNERGROUPCREATOR CGROUP NATTCHSEGSZ CPID LPID ATIME DTIMECTIME m 262144 5432001 --rw--- pgsqlpgsqlpgsqlpgsql 12 88139939844551245512 13:49:28 14:31:29 13:49:28 but frankly this tells me nothing. I can tell that the value SEGSZ is right from the start 8813993984 and it doesn't change anymore. The only value that changes is the NATTCH value, I observed a range from 8 to 36 there. I agree that the SEGSZ value matches the 8GB shared buffer, but how can I make the connection of my 5GB missing in top? I wonder if this might be the maintenance_work_mem, which is set to 4GB? Many thanks, Frank Am 2012-11-05 12:14, schrieb Achilleas Mantzios: ipcs in FreeBSD is a little ... tricky. ipcs -M ipcs -m ipcs -am could be your friends On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: Hi, I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. Usually I use top to examine the memory usage of the system. After a while, a part, approximately 5GB, vanish from top, so that the memory rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again available, but then it's already slightly decreasing. It's a standalone database server. It has an OpenStreetMap world database running with 353GB data (with indices). Some system information: # uname -r 9.0-RELEASE-p3 # pg_ctl --version pg_ctl (PostgreSQL) 9.1.6 # cat /boot/loader.conf ...
