On Thu, Jun 2, 2016 at 12:04 PM, <[email protected]> wrote: > > I'm dealing with a very long-running simple real-time program (weeks, even > months) which opens and closes sockets continually to a variety of > locations, for very short transactions. There is very slow growth of > virtual and real memory use (especially the real memory) of this program, > on a variety of *NIX types and apparently proportional to the activity > level, that raises the following questions: >
Does it have the ability to restart and continue periodically? That's the easiest way to give memory back. > - I assume that the kernel has to allocate buffer space for each socket > created, which presumably it charges to the pid of the socket requester > (the problem perl program in this case)? > I would expect both kernel and proc socket buffers would be managed better but don't know. - I once ran across a comment somewhere in perldoc or code comments that > when perl requests memory from the system it *never* gets released back to > the system -- perl may reuse it itself after getting it garbage collected, > but it doesn't get returned to the kernel. Could have been a very old > piece of doc -- is this still true? > It's (almost) true of anything built with the C library, which Perl is; and Perl does NOT try to. Perl does garbage collect its own heap. But if virtual memory increases more than real albeit both slowly, that suggests garbage collection on the heap is not solving all fragmentation and/or memory leaks. Perl code can 'leak' memory such that the G/C doesn't free it by creating circular data structures; solution is weak refs. These leaks could be in core or libs not just your code. The underlying C code and runtime can also leak ... and the malloc heap does NOT g/c; if it stays fragmented, it will have to grow the process. > - Strictly speaking perl isn't directly requesting the buffer memory, just > the socket, so I'm not sure that even if the statement above is true that > the buffer space doesn't go back to the system when the socket closes. > Depends how much socket memory is in kernel vs process ... But it's possible it's the string copy buffers of processing not the underthehood sockets and heap fragmentation. Step one is to find out if it's Perl's heap or the rest of the process that is growing. If it's fragmentation in the Perl heap, the classic solution for no-growth, run-forever is to take over managing whatever object is causing the problem by allocating a fixed ring buffer of them. (Applying that process to the C Malloc heap would mean re-writing the Perl runtime. Nope.) > - Sooooo, it's a long way of asking if closing the socket in perl returns > the buffer memory back to the system? > > - And finally, if the buffer memory does stay with the perl pid, is perl > actually able to use that memory later for other purposes (since it seems > like an indirect memory allocation) ? > > PS I realize that socket work is not generally done this way (i.e. so > long-running) but would like to better understand the situation before > undertaking restructuring. > and that may require understanding OS and your C lib socket libraries. This is Unix/Linux with Gnu C ? versions? Are the sockets TCP, UDP, or Berkley (local) sockets ? (But i suspect it's Perl &/or Malloc heap fragmentation &/or leakage. You could re-build Perl from source with an instrumented Malloc to get Malloc to confess.) -- Bill Ricker [email protected] https://www.linkedin.com/in/n1vux _______________________________________________ Boston-pm mailing list [email protected] http://mail.pm.org/mailman/listinfo/boston-pm
