On 1/10/13 1:41 AM, Alfred Perlstein wrote:
On 12/23/12 12:28 PM, Jason Evans wrote:
On Dec 21, 2012, at 7:37 PM, Alfred Perlstein <bri...@mu.org> wrote:
So the other day in an effort to debug a memory leak I decided to
take a look at malloc+utrace(2) and decided to make a tool to debug
where leaks are coming from.
A few hours later I have:
1) a new version of utrace(2) (utrace2(2)) that uses structured data
to prevent overloading of data. (utrace2.diff)
2) changes to ktrace and kdump to decode the new format. (also in
utrace2.diff)
3) changes to jemalloc to include the new format AND the function
caller so it's easy to get the source of the leaks. (also in
utrace2.diff)
4) a program that can take a pipe of kdump(1) and figure out what
memory has leaked. (alloctrace.py)
5) simple test program (test_utrace.c)
[…]
Have you looked at the heap profiling functionality built into
jemalloc? It's not currently enabled on FreeBSD, but as far as I
know, the only issue keeping it from being useful is the absence of a
Linux-compatible /proc/<pid>/maps (and the gperftools folks may
already have a solution for that; I haven't looked). I think it
makes more sense to get that sorted out than to develop a separate
trace-based leak checker. The problem with tracing is that it
doesn't scale beyond some relatively small number of allocator events.
I have looked at some of this functionality (heap profiling) but alas
it is not implemented yet. In addition the dtrace work appears to be
quite away from a workable solution with too many performance
penalties until some serious hacking is done.
I am just not sure how to proceed, on one hand I do not really have
the skill to fix the /proc/pid/maps problem, nor figure out how to get
dtrace into the system in any time frame that is reasonable.
All a few of us need is the addition of the trace back into the
existing utrace framework.
Is it time to start installing with some form of debug symbols? This
would help us also with dtrace.
Re: debug symbols, frame pointers, etc. necessary to make userland
dtrace work by default, IMO we should strongly prefer such defaults.
It's more reasonable to expect people who need every last bit of
performance to remove functionality than to expect people who want to
figure out what the system is doing to figure out what functionality
to turn on.
This is very true. I'm going to continue to work towards this end
with a few people and get up to speed on it so that hopefully we can
get to this point hopefully in the next release cycle or two.
If you have a few moments, can you have a look at the "utrace2"
branches here:
https://github.com/alfredperlstein/freebsd/tree/utrace2
This branch contains the addition of the utrace2 system call which is
needed to structure data via utrace(2). The point of this is to avoid
kdump(1) needing to discern type of ktrace records based on arbitrary
size or other parameters and introduces an extensible protocol for new
types of utrace data.
The utrace2 branch here augments jemalloc to use utrace2 to pass the
old utrace records, but in addition to pass the return address along
with the type and size of the allocation:
https://github.com/alfredperlstein/jemalloc/tree/utrace2
-Alfred
Jason,
Here are more convenient links that give diffs against FreeBSD and
jemalloc for the proposed changes:
FreeBSD:
https://github.com/alfredperlstein/freebsd/compare/13e7228d5b83c8fcfc63a0803a374212018f6b68~1...utrace2
jemalloc:
https://github.com/alfredperlstein/jemalloc/compare/master...utrace2
-Alfred
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
