I still digging into this issue, and have hacked memleak/bcc to show addresses
when they cant be resolved. In other places, I'm seeing jit complied java stack
frames showing up alongside C ones, which is the expected behaviour when using
Java with -XX:+PreserveFramePointer.
I'm still seeing these allocation traces with only two frames per the below.
11534336 bytes in 11 allocations from stack
[7efeebb115d4 unknown] [jna7632521838566054573.tmp ]
[7efec27f7440 unknown] [perf-15445.map]
14680064 bytes in 14 allocations from stack
[7efeebb115d4 unknown] [jna7632521838566054573.tmp ]
[7efec27f7380 unknown] [perf-15445.map]
I suspect the lower frame above is a libffi generated trampoline. Are there any
known issues with the eBPF stack tracing infrastructure and such trampolines?
Workarounds?
Thanks,
Bradley
On 7/4/21, 9:04 pm, "Bradley Schatz" <[email protected]> wrote:
> What does '[unknown] [perf-18047.map]' mean? Does this mean
> perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
> symbolization won't be possible. Maybe you want to double check this?
The file perf-18047.map is there and from other parts of the stack trace I
can see it being used to successfully resolve symbols.
Thanks!
On 7/4/21, 4:44 am, "Y Song" <[email protected]> wrote:
On Mon, Apr 5, 2021 at 10:08 PM Bradley Schatz
<[email protected]> wrote:
>
> Thanks for the suggestion. I found a tunable to keep the JNI shared
library in memory after loading. As you can see below, it is no longer showing
as deleted.
>
> 13238272 bytes in 404 allocations from stack
> [unknown] [jna2576903844543447777.tmp]
> [unknown] [perf-18047.map]
I have no experience with perf-map-agent, but the following is what I
guess:
[perf-18047.map] is used to find the mapping between address and
symbol.
What does '[unknown] [perf-18047.map]' mean? Does this mean
perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
symbolization won't be possible. Maybe you want to double check this?
>
> No improvement in granularity though.
>
> In the VM I'm using -XX:+PreserveFramePointer
-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints. In perf_maps_agent, I'm
using "unfoldall"
>
> Any other suggestions?
>
> Thanks!
>
>
>
>
> On 3/4/21, 2:42 am, "Y Song" <[email protected]> wrote:
>
> On Wed, Mar 31, 2021 at 11:25 PM Bradley Schatz
> <[email protected]> wrote:
> >
> > Hi,
> >
> >
> >
> > I’m just starting to come to grips with bcc & perf-map-agent
for introspecting java on linux, with the goal of identifying what appears to
be an off-heap memory leak (using memleak).
> >
> >
> >
> > I appear to be getting reliable stack decoding for jvm library
code and for jit’ed java methods (see below for an example of the former).
However I am seeing some very short stack traces which don’t seem to decode
(the latter three stacks of below).
> >
> >
> >
> > It’s looking to me like the frame starting with “jna…” is
likely the native JNI shared library for the FFI library “JNA”.
> >
> >
> >
> > Any suggestions as to why these latter three are so brief
and/or how I can increase the resolution?
>
> I can see the file has been marked as deleted.
>
> 34603008 bytes in 33 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> 96468992 bytes in 92 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> So the file has been removed in userspace and current bcc won't be
> pass to parse it since it takes the file name as
> "jna9005484735610534564.tmp (deleted)"
> The file name is actually taken from /proc/<pid>/maps.
>
> I am not sure whether you can hack to parse
"jna9005484735610534564.tmp" or not.
> But I would consider it is unsafe to do that as the original file
> related info may just
> exist in kernel and there is a reference to it. For user space,
it is
> either gone or
> could be replaced by something else. So the safest way is to find
a place to
> do symbolization before file is gone or keep tmp file a little
bit longer.
> >
> >
> >
> > Apologies if this is the wrong place for such a question. Thank
you for your help.
> >
> >
> >
> > Kind regards,
> >
> > Bradley
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 119408 bytes in 71 allocations from stack
> >
> > os::malloc(unsigned long, MemoryType,
NativeCallStack const&)+0xb5 [libjvm.so]
> >
> > CodeBlob::set_oop_maps(OopMapSet*) [clone
.part.5]+0x75 [libjvm.so]
> >
> > CodeBlob::CodeBlob(char const*, CodeBuffer*, int,
int, int, int, OopMapSet*)+0xe3 [libjvm.so]
> >
> > nmethod::nmethod(Method*, int, int, int,
CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int,
OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*,
int)+0x4d [libjvm.so]
> >
> > nmethod::new_nmethod(methodHandle, int, int,
CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int,
OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*,
int)+0x219 [libjvm.so]
> >
> > ciEnv::register_method(ciMethod*, int,
CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*,
ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool, RTMState)+0x1b1
[libjvm.so]
> >
> > Compile::Compile(ciEnv*, C2Compiler*, ciMethod*,
int, bool, bool, bool)+0xe60 [libjvm.so]
> >
> > C2Compiler::compile_method(ciEnv*, ciMethod*,
int)+0xa3 [libjvm.so]
> >
> >
CompileBroker::invoke_compiler_on_method(CompileTask*)+0x808 [libjvm.so]
> >
> > CompileBroker::compiler_thread_loop()+0x6d8
[libjvm.so]
> >
> > JavaThread::thread_main_inner()+0x1c7 [libjvm.so]
> >
> > JavaThread::run()+0x2fa [libjvm.so]
> >
> > java_start(Thread*)+0x102 [libjvm.so]
> >
> > start_thread+0xf3 [libpthread-2.28.so]
> >
> > 34603008 bytes in 33 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> > 96468992 bytes in 92 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> > 295698432 bytes in 282 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> >
> >
> >
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#1987): https://lists.iovisor.org/g/iovisor-dev/message/1987
Mute This Topic: https://lists.iovisor.org/mt/82136333/21656
Group Owner: [email protected]
Unsubscribe: https://lists.iovisor.org/g/iovisor-dev/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
