Hello,
I'm having trouble profiling our application with Callgrind.
Basically what I'm doing is to start from the 'core' application, profile
it, add functionality, profile again etc.
The application sniffs traffic from the wire and does TCP stream
reassembly. Everything works fine until the TCP reassembly stage.
The error:
vex amd64->IR: unhandled instruction bytes: 0xF 0xD 0x8F 0x40 0x3 0x0 0x0
0xF
==4616== valgrind: Unrecognised instruction at address 0x616aeee.
==4616== at 0x616AEEE: memcpy (memcpy.S:376)
==4616== by 0x4F4F5C: tcp::HalfStream::dropReassembledData(int)
If we start the application while we're already sending data (see below)
then the problem might occur here:
vex amd64->IR: unhandled instruction bytes: 0xF 0xD 0x8F 0x40 0x3 0x0 0x0
0xF
==4223== valgrind: Unrecognised instruction at address 0x616aeee.
==4223== at 0x616AEEE: memcpy (memcpy.S:376)
==4223== by 0x42F80E: tcp::HalfStream::addPayload(char const*, unsigned
int, unsigned int, timeval)
So far, Callgrind is the only tool giving this error (but I suppose the
other tools work fundamentally different).
There are no warnings about bad jumps (or any other warning/message for
that matter) when running MemCheck.
Bounds are checked with assertions, but I've attached GDB (target remote +
vgdb) to look at the memcpy calls and check the bounds myself, they seem
fine:
(gdb) up
#1 0x00000000004f4f5d in tcp::HalfStream::dropReassembledData
(this=0x4242bf8, lenToDrop=1552) at < <blanked> >HalfStream.cc:426
426 memcpy( buf_, data_ + lenToDrop, dataLen_ - lenToDrop );
(gdb) p bufSize_
$1 = 36864
(gdb) p dataSize_
$2 = 36864
(gdb) p dataLen_ - lenToDrop
$3 = 33120
So buf_ is large enough (36864 (bufSize) > 33120 (dataLen_ - lenToDrop))
and data_: 33120 (dataLen_ - lenToDrop) + 1552 (lenToDrop) < 36864
(dataSize)
Disassembling memcpy shows:
0x000000000616aee6 <+598>: mov QWORD PTR [rdi+0x30],r13
0x000000000616aeea <+602>: mov QWORD PTR [rdi+0x38],r14
=> 0x000000000616aeee <+606>: prefetchw BYTE PTR [rdi+0x340]
0x000000000616aef5 <+613>: prefetchw BYTE PTR [rdi+0x380]
0x000000000616aefc <+620>: lea rsi,[rsi+0x40]
0x000000000616af00 <+624>: lea rdi,[rdi+0x40]
So it seems the location and instructions are valid.
However, if I instruct gdb to break at memcpy.S:376 then this breakpoint
does not get hit until the SIGILL.
(gdb) b memcpy.S:376
Breakpoint 1 at 0x616a5fe: file ../sysdeps/x86_64/memcpy.S, line 376. (2
locations)
(gdb) c
Continuing.
[New Thread 4839]
[Switching to Thread 4839]
Breakpoint 1, memcpy () at ../sysdeps/x86_64/memcpy.S:376
376 ../sysdeps/x86_64/memcpy.S: No such file or directory.
in ../sysdeps/x86_64/memcpy.S
(gdb) n
Program received signal SIGILL, Illegal instruction.
memcpy () at ../sysdeps/x86_64/memcpy.S:376
376 in ../sysdeps/x86_64/memcpy.S
My question is, how do I go about identifying the problem?
To discover if this was a Callgrind problem or something in our application
I tried to run with different pcaps (We're replaying a pcap from one
machine to another which is running our application and sniffing the
traffic). So far, the problem does not happen when using a different pcap.
The number of pcaps with relevant data that I have available at the moment
is limited, so this is only a small indication.
Some more notes:
- There are memcpy calls in previous parts of the code that work fine.
- The same memcpy that causes the problem is called many times
successfully, with different data (backing the theory that this is a bug in
our application).
- We do not observe that the traffic is reassembled incorrectly, but this
is difficult to verify beyond doubt.
- The problem does not occur when I change memcpy to memmove in addPayload
and dropReassembledData. So while this is a workaround for me to be able to
profile the rest of the application, the SIGILL still scares me ;)
System details:
testbox3:~/app$ uname -a
Linux testbox3 2.6.32-38-server #83-Ubuntu SMP Wed Jan 4 11:26:59 UTC 2012
x86_64 GNU/Linux
I hope this is enough detail.
Any help is greatly appreciated.
Regards,
Steve
------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
Valgrind-users mailing list
Valgrind-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/valgrind-users
