Re: Swap Usage
Some more testing indicates that the problem is most likely in close, not mmap.
I modified the program to the following:
zool# more test.c
#include stdio.h
#include fcntl.h
#include string.h
#include stdlib.h
#include errno.h
#include sys/mman.h
int main (int argc, char *argv[])
{
int rc, pid, fd;
char *cp;
char cmd[1024];
pid = getpid ();
sprintf (cmd, procstat -v %d | grep df, pid);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf (--\n);
fd = open (./test.c, O_RDWR);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf (--\n);
close (fd);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
}
The output is:
zool# ./test
85860 0x8049000 0x804a000 rw-10 1 0 CN-- df
85860 0x2805f000 0x28069000 rw- 100 1 0 C--- df
85860 0x28186000 0x281ad000 rw- 130 1 0 CN-- df
85860 0xbfbdf000 0xbfbff000 rwx30 1 0 C--D df
--
85860 0x8049000 0x804a000 rw-10 1 0 CN-- df
85860 0x804a000 0x840 rw-10 1 0 CN-- df
85860 0x2805f000 0x28069000 rw- 100 1 0 CN-- df
85860 0x28186000 0x281ad000 rw- 140 1 0 CN-- df
85860 0x2840 0x2880 rw-60 1 0 CN-- df
85860 0xbfbdf000 0xbfbff000 rwx30 1 0 C--D df
--
85860 0x8049000 0x804a000 rw-10 1 0 CN-- df
85860 0x804a000 0x840 rw-10 1 0 CN-- df
85860 0x2805f000 0x28069000 rw- 100 1 0 CN-- df
85860 0x28186000 0x281ad000 rw- 140 1 0 CN-- df
85860 0x2840 0x2880 rw-60 1 0 CN-- df
85860 0xbfbdf000 0xbfbff000 rwx30 1 0 C--D df
Open creates 2 memory allocations, one in low memory and one in high. Close
does not remove them. Somewhere in the source I found a note that said that
close on files didn’t require any action. However, those two memory
allocations do need to get freed.
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Swap Usage
On 30 July 2015, at 01:39, Doug Hardie bc...@lafn.org wrote:
On 29 July 2015, at 23:44, Peter Jeremy pe...@rulingia.com wrote:
[reformatted]
On 2015-Jul-29 17:41:33 -0700, Doug Hardie bc...@lafn.org wrote:
I have several FreeBSD 9.3 systems that are using swap and I can’t
figure out what is doing it. The key system has 6GB swap and
currently it has over 2GB in use.
Is the system currently paging (top(1) and systat -v will show
this)? If not, this just means that at some time in the past, the
system was under memory pressure and paged some process memory out.
Since then, that memory hasn't been touched so the system hasn't paged
it in.
ps shows only a kernel module
[intr] with a W status.
'W' means the whole process is 'swapped' out - this will only occur
under severe RAM pressure. Normally, the system will just page out
inactive parts of a processes address space - and none of the ps flags
will show this.
How do I figure out what that swap space is being used for?
I don't think this can be trivially done. procstat -v will show
the number of resident pages within each swap-backed region, any
pages in that region that have been touched but are not resident
are on the swap device but any pages that have never been touched
aren't counted at all.
Bingo. procstat shows the problem. The process that I suspected has a large
number of entries like:
6500x834c00x83580 rw-00 1 0 sw
6500x835800x835c0 rw-00 1 0 sw
6500x835c00x837c0 rw-10 1 0 sw
I don’t know whats in those areas yet. If I were to kill the process with
SIGABRT would the core dump show those areas? I might be able to figure out
what they are from that.
Thanks for the pointer.
I believe I have found a bug in FreeBSD 9.3 mmap. The following test program
(named test.c) does an mmap to itself and monitors the memory objects created
with procstat. After the mmap, munamp is called and the objects are checked
again. The objects created from the mmap do not get purged, but remain.
Eventually, after enough have been created, they start becoming type sw and
eventually the system runs out of swap.
The output:
zool# ./test
31459 0x8049000 0x804a000 rw-10 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 100 1 0 C--- df
31459 0x28186000 0x281ad000 rw- 130 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx30 1 0 C--D df
--
31459 0x8049000 0x804a000 rw-10 1 0 CN-- df
31459 0x804a000 0x840 rw-10 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 100 1 0 CN-- df
31459 0x28186000 0x281ad000 rw- 140 1 0 CN-- df
31459 0x2840 0x2880 rw-60 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx30 1 0 C--D df
--
31459 0x8049000 0x804a000 rw-10 1 0 CN-- df
31459 0x804a000 0x840 rw-10 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 100 1 0 CN-- df
31459 0x28186000 0x281ad000 rw- 140 1 0 CN-- df
31459 0x2840 0x2880 rw-60 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx30 1 0 C--D df
--
31459 0x8049000 0x804a000 rw-10 1 0 CN-- df
31459 0x804a000 0x840 rw-10 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 100 1 0 CN-- df
31459 0x28186000 0x281ad000 rw- 140 1 0 CN-- df
31459 0x2840 0x2880 rw-60 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx30 1 0 C--D df
The program:
int main (int argc, char *argv[])
{
int rc, pid, fd;
char *cp;
char cmd[1024];
pid = getpid ();
sprintf (cmd, procstat -v %d | grep df, pid);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf (--\n);
fd = open (./test.c, O_RDWR);
cp = mmap (0, 100, PROT_READ|PROT_WRITE, MAP_FILE, fd, 0);
if (cp == MAP_FAILED)
{
printf (mmap error %s\n, strerror(errno));
exit (1);
}
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf (--\n);
rc = munmap (cp, 100);
if (rc == -1)
{
printf (munmap error %s\n, strerror(errno));
exit (1);
}
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf (--\n);
close (fd);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
}
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to
Re: Swap Usage
On 29 July 2015, at 23:44, Peter Jeremy pe...@rulingia.com wrote: [reformatted] On 2015-Jul-29 17:41:33 -0700, Doug Hardie bc...@lafn.org wrote: I have several FreeBSD 9.3 systems that are using swap and I can’t figure out what is doing it. The key system has 6GB swap and currently it has over 2GB in use. Is the system currently paging (top(1) and systat -v will show this)? If not, this just means that at some time in the past, the system was under memory pressure and paged some process memory out. Since then, that memory hasn't been touched so the system hasn't paged it in. ps shows only a kernel module [intr] with a W status. 'W' means the whole process is 'swapped' out - this will only occur under severe RAM pressure. Normally, the system will just page out inactive parts of a processes address space - and none of the ps flags will show this. How do I figure out what that swap space is being used for? I don't think this can be trivially done. procstat -v will show the number of resident pages within each swap-backed region, any pages in that region that have been touched but are not resident are on the swap device but any pages that have never been touched aren't counted at all. Bingo. procstat shows the problem. The process that I suspected has a large number of entries like: 6500x834c00x83580 rw-00 1 0 sw 6500x835800x835c0 rw-00 1 0 sw 6500x835c00x837c0 rw-10 1 0 sw I don’t know whats in those areas yet. If I were to kill the process with SIGABRT would the core dump show those areas? I might be able to figure out what they are from that. Thanks for the pointer. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Swap Usage
On 29 July 2015, at 17:41, Doug Hardie bc...@lafn.org wrote: I have several FreeBSD 9.3 systems that are using swap and I can’t figure out what is doing it. The key system has 6GB swap and currently it has over 2GB in use. ps shows only a kernel module [intr] with a W status. Obviously that isn’t using the space. No other process shows a W in its status. I suspect this is somewhat related to the use of mmap in one application. However, all of the mmaps in that application are file backed and thus shouldn’t use swap. How do I figure out what that swap space is being used for? UFS although that shouldn’t matter. Swap does not use a file system. If it did, it would be easy to figure out. Top doesn’t show much. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Swap Usage
[reformatted] On 2015-Jul-29 17:41:33 -0700, Doug Hardie bc...@lafn.org wrote: I have several FreeBSD 9.3 systems that are using swap and I can’t figure out what is doing it. The key system has 6GB swap and currently it has over 2GB in use. Is the system currently paging (top(1) and systat -v will show this)? If not, this just means that at some time in the past, the system was under memory pressure and paged some process memory out. Since then, that memory hasn't been touched so the system hasn't paged it in. ps shows only a kernel module [intr] with a W status. 'W' means the whole process is 'swapped' out - this will only occur under severe RAM pressure. Normally, the system will just page out inactive parts of a processes address space - and none of the ps flags will show this. How do I figure out what that swap space is being used for? I don't think this can be trivially done. procstat -v will show the number of resident pages within each swap-backed region, any pages in that region that have been touched but are not resident are on the swap device but any pages that have never been touched aren't counted at all. -- Peter Jeremy pgp_HZj6rM0Rp.pgp Description: PGP signature
Re: Swap Usage
On 29 July 2015, at 18:57, Chris H bsd-li...@bsdforge.com wrote: On Wed, 29 Jul 2015 17:41:33 -0700 Doug Hardie bc...@lafn.org wrote I have several FreeBSD 9.3 systems that are using swap and I can’t figure out what is doing it. The key system has 6GB swap and currently it has over 2GB in use. ps shows only a kernel module [intr] with a W status. Obviously that isn’t using the space. No other process shows a W in its status. I suspect this is somewhat related to the use of mmap in one application. However, all of the mmaps in that application are file backed and thus shouldn’t use swap. How do I figure out what that swap space is being used for? Maybe top(1)? top -P for example. At least you could see who's chewing all your memory. Which should be a good clue as to who's responsible for swap usage. UFS although I don’t see how that could make a difference. Swap doesn’t use a regular file system. However, the kernel must track the actual usage using something like a simple file system. There must be a way to investigate it. Top doesn’t show anything unusual. The most VM used is by the process that uses a lot of mmap space that is all file backed. If it was another process that was partially swapped then the ps status should show a W. Only the one does. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: Swap Usage
On Wed, 29 Jul 2015 17:41:33 -0700 Doug Hardie bc...@lafn.org wrote I have several FreeBSD 9.3 systems that are using swap and I can’t figure out what is doing it. The key system has 6GB swap and currently it has over 2GB in use. ps shows only a kernel module [intr] with a W status. Obviously that isn’t using the space. No other process shows a W in its status. I suspect this is somewhat related to the use of mmap in one application. However, all of the mmaps in that application are file backed and thus shouldn’t use swap. How do I figure out what that swap space is being used for? Maybe top(1)? top -P for example. At least you could see who's chewing all your memory. Which should be a good clue as to who's responsible for swap usage. --Chris ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
