While using our network traffic monitoring tool (which uses WinPcap on
Windows) on a single processor Win2000 machine running an appserver under
load, we've run into a problem which one of our developers has subsequently
analyzed and formulated a theory as to the possible cause:
------------------------------------------------------
An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : EXCEPTION_ACCESS_VIOLATION occurred at PC=0xa1a975f
Function name=pcap_read
Library=C:\WINNT\System32\wpcap.dll
Current Java thread:
at
com.sitraka.alto.agent.plugin.NetworkTrafficMonitorPlugIn$DeviceMonit
or.readPackets(Native Method)
at
com.sitraka.alto.agent.plugin.NetworkTrafficMonitorPlugIn$DeviceMonit
or.run(NetworkTrafficMonitorPlugIn.java:413)
at java.lang.Thread.run(Thread.java:484)
Dynamic libraries:
0x00400000 - 0x00405000 c:\bea61sp2\jdk131\bin\java.exe
0x77F80000 - 0x77FFB000 C:\WINNT\System32\ntdll.dll
0x77DB0000 - 0x77E0B000 C:\WINNT\system32\ADVAPI32.dll
0x77E80000 - 0x77F35000 C:\WINNT\system32\KERNEL32.DLL
0x77D40000 - 0x77DB0000 C:\WINNT\system32\RPCRT4.DLL
0x78000000 - 0x78046000 C:\WINNT\system32\MSVCRT.dll
0x6D420000 - 0x6D4EE000 c:\bea61sp2\jdk131\jre\bin\hotspot\jvm.dll
0x77E10000 - 0x77E74000 C:\WINNT\system32\USER32.dll
0x77F40000 - 0x77F7C000 C:\WINNT\system32\GDI32.DLL
0x77570000 - 0x775A0000 C:\WINNT\System32\WINMM.dll
0x6D220000 - 0x6D227000 c:\bea61sp2\jdk131\jre\bin\hpi.dll
0x6D3B0000 - 0x6D3BD000 c:\bea61sp2\jdk131\jre\bin\verify.dll
0x6D250000 - 0x6D266000 c:\bea61sp2\jdk131\jre\bin\java.dll
0x6D3C0000 - 0x6D3CD000 c:\bea61sp2\jdk131\jre\bin\zip.dll
0x6D340000 - 0x6D348000 C:\bea61sp2\jdk131\jre\bin\net.dll
0x75050000 - 0x75058000 C:\WINNT\System32\WSOCK32.dll
0x75030000 - 0x75043000 C:\WINNT\System32\WS2_32.DLL
0x75020000 - 0x75028000 C:\WINNT\System32\WS2HELP.DLL
0x785C0000 - 0x785CC000 C:\WINNT\System32\rnr20.dll
0x77980000 - 0x779A4000 C:\WINNT\System32\DNSAPI.DLL
0x77340000 - 0x77353000 C:\WINNT\System32\iphlpapi.dll
0x77520000 - 0x77525000 C:\WINNT\System32\ICMP.DLL
0x77320000 - 0x77337000 C:\WINNT\System32\MPRAPI.DLL
0x75150000 - 0x75160000 C:\WINNT\System32\SAMLIB.DLL
0x75170000 - 0x751BF000 C:\WINNT\System32\NETAPI32.DLL
0x77BE0000 - 0x77BEF000 C:\WINNT\System32\SECUR32.DLL
0x751C0000 - 0x751C6000 C:\WINNT\System32\NETRAP.DLL
0x77950000 - 0x77979000 C:\WINNT\system32\WLDAP32.DLL
0x77A50000 - 0x77B46000 C:\WINNT\system32\OLE32.DLL
0x779B0000 - 0x77A4B000 C:\WINNT\system32\OLEAUT32.DLL
0x773B0000 - 0x773DE000 C:\WINNT\System32\ACTIVEDS.DLL
0x77380000 - 0x773A2000 C:\WINNT\System32\ADSLDPC.DLL
0x77830000 - 0x7783E000 C:\WINNT\System32\RTUTILS.DLL
0x77880000 - 0x7790D000 C:\WINNT\System32\SETUPAPI.DLL
0x77C10000 - 0x77C6D000 C:\WINNT\System32\USERENV.DLL
0x774E0000 - 0x77512000 C:\WINNT\System32\RASAPI32.DLL
0x774C0000 - 0x774D1000 C:\WINNT\System32\RASMAN.DLL
0x77530000 - 0x77552000 C:\WINNT\system32\TAPI32.DLL
0x716F0000 - 0x7177A000 C:\WINNT\system32\COMCTL32.DLL
0x70BD0000 - 0x70C1C000 C:\WINNT\system32\SHLWAPI.DLL
0x77360000 - 0x77379000 C:\WINNT\System32\DHCPCSVC.DLL
0x775A0000 - 0x77625000 C:\WINNT\System32\CLBCATQ.DLL
0x777E0000 - 0x777E8000 C:\WINNT\System32\winrnr.dll
0x777F0000 - 0x777F5000 C:\WINNT\System32\rasadhlp.dll
0x74FD0000 - 0x74FEF000 C:\WINNT\system32\msafd.dll
0x75010000 - 0x75017000 C:\WINNT\System32\wshtcpip.dll
0x10000000 - 0x10011000
C:\PerformaSure1.6\CAPE-20020703.0400\bin\Window
sPDH.dll
0x692E0000 - 0x69307000 C:\WINNT\System32\pdh.dll
0x76B30000 - 0x76B6E000 C:\WINNT\system32\comdlg32.dll
0x782F0000 - 0x78532000 C:\WINNT\system32\SHELL32.DLL
0x78280000 - 0x782B3000 C:\WINNT\system32\kerberos.dll
0x76670000 - 0x7667E000 C:\WINNT\System32\CRYPTDLL.DLL
0x77430000 - 0x77440000 C:\WINNT\System32\MSASN1.DLL
0x692B0000 - 0x692B8000 C:\WINNT\System32\perfdisk.dll
0x69280000 - 0x69289000 C:\WINNT\System32\perfos.dll
0x0A180000 - 0x0A195000
C:\PerformaSure1.6\CAPE-20020703.0400\bin\Networ
kTrafficMonitorPlugIn.dll
0x0A1A0000 - 0x0A1C9000 C:\WINNT\System32\wpcap.dll
0x0A1D0000 - 0x0A1D8000 C:\WINNT\System32\packet.dll
0x0BB60000 - 0x0BB65000 C:\bea61sp2\wlserver6.1\bin\wlntio.dll
0x6D240000 - 0x6D246000 C:\bea61sp2\jdk131\jre\bin\ioser12.dll
0x77920000 - 0x77943000 C:\WINNT\system32\imagehlp.dll
0x72A00000 - 0x72A2D000 C:\WINNT\system32\DBGHELP.dll
0x690A0000 - 0x690AB000 C:\WINNT\System32\PSAPI.DLL
Local Time = Wed Jul 03 17:08:15 2002
Elapsed Time = 1957
#
# The exception above was detected in native code outside the VM
#
# Java VM: Java HotSpot(TM) Client VM (1.3.1_01 mixed mode)
#
# An error report file has been saved as hs_err_pid824.log.
# Please refer to the file for further information.
#
------------------------------------------------------
After digging around in the WinPcap source code the following theory was
proposed:
------------------------------------------------------
1. The description of the system exception (EXCEPTION_ACCESS_VIOLATION
occurred at PC=0xa1a975f
Function name=pcap_read) points to the following code fragment (from the
disasm of wpcap.dll -- wpcap.dll.dump):
pcap_read:
10009700: 51 push ecx
10009701: 53 push ebx
10009702: 55 push ebp
10009703: 56 push esi
10009704: 57 push edi
10009705: 8B 7C 24 18 mov edi,dword ptr [esp+18h]
10009709: 33 ED xor ebp,ebp
1000970B: 8B 47 5C mov eax,dword ptr [edi+5Ch]
1000970E: 85 C0 test eax,eax
10009710: 75 3A jne 1000974C
10009712: 8B 47 04 mov eax,dword ptr [edi+4]
10009715: 8B 0F mov ecx,dword ptr [edi]
10009717: 6A 01 push 1
10009719: 50 push eax
1000971A: 51 push ecx
1000971B: E8 F4 1B 00 00 call 1000B314
10009720: 83 C4 0C add esp,0Ch
10009723: 84 C0 test al,al
10009725: 75 1A jne 10009741
10009727: 83 C7 6C add edi,6Ch
1000972A: 68 9C CE 01 10 push 1001CE9Ch
1000972F: 57 push edi
10009730: E8 3E 1C 00 00 call 1000B373
10009735: 83 C4 08 add esp,8
10009738: 83 C8 FF or eax,0FFFFFFFFh
1000973B: 5F pop edi
1000973C: 5E pop esi
1000973D: 5D pop ebp
1000973E: 5B pop ebx
1000973F: 59 pop ecx
10009740: C3 ret
10009741: 8B 4F 04 mov ecx,dword ptr [edi+4]
10009744: 8B 41 20 mov eax,dword ptr [ecx+20h]
10009747: 8B 71 18 mov esi,dword ptr [ecx+18h]
1000974A: EB 03 jmp 1000974F
1000974C: 8B 77 58 mov esi,dword ptr [edi+58h]
1000974F: 03 C6 add eax,esi
10009751: 3B F0 cmp esi,eax
10009753: 89 44 24 10 mov dword ptr [esp+10h],eax
10009757: 73 3A jae 10009793
10009759: 8B 44 24 24 mov eax,dword ptr [esp+24h]
1000975D: 33 FF xor edi,edi
**1000975F: 66 8B 7E 10 mov di,word ptr [esi+10h]
10009763: 8B 5E 08 mov ebx,dword ptr [esi+8]
10009766: 8D 14 37 lea edx,[edi+esi]
10009769: 52 push edx
1000976A: 56 push esi
1000976B: 50 push eax
1000976C: FF 54 24 2C call dword ptr [esp+2Ch]
10009770: 8B 44 24 28 mov eax,dword ptr [esp+28h]
10009774: 8D 4C 1F 03 lea ecx,[edi+ebx+3]
10009778: 83 E1 FC and ecx,0FFFFFFFCh
1000977B: 83 C4 0C add esp,0Ch
1000977E: 03 F1 add esi,ecx
10009780: 45 inc ebp
10009781: 3B E8 cmp ebp,eax
10009783: 7C 04 jl 10009789
10009785: 85 C0 test eax,eax
10009787: 7F 19 jg 100097A2
10009789: 3B 74 24 10 cmp esi,dword ptr [esp+10h]
1000978D: 72 CA jb 10009759
1000978F: 8B 7C 24 18 mov edi,dword ptr [esp+18h]
10009793: C7 47 5C 00 00 00 mov dword ptr [edi+5Ch],0
00
1000979A: 5F pop edi
1000979B: 8B C5 mov eax,ebp
1000979D: 5E pop esi
1000979E: 5D pop ebp
1000979F: 5B pop ebx
100097A0: 59 pop ecx
100097A1: C3 ret
100097A2: 8B 44 24 10 mov eax,dword ptr [esp+10h]
100097A6: 8B 4C 24 18 mov ecx,dword ptr [esp+18h]
100097AA: 2B C6 sub eax,esi
100097AC: 5F pop edi
100097AD: 89 71 58 mov dword ptr [ecx+58h],esi
100097B0: 89 41 5C mov dword ptr [ecx+5Ch],eax
100097B3: 8B C5 mov eax,ebp
100097B5: 5E pop esi
100097B6: 5D pop ebp
100097B7: 5B pop ebx
100097B8: 59 pop ecx
100097B9: C3 ret
When compared to the source (Pcap-win32.c, lines 69--116) it most certainly
corresponds to the following C statement:
#define bhp ((struct bpf_hdr *)bp)
hdrlen = bhp->bh_hdrlen;
^^^^^^^^^^^^^^^
Register mapping around line 100975f: esi = bp; edi = (short) hdrlen; edx =
bp + hdrlen
In this context the error indicates that the address in esi was invalid. But
on the other hand the value of bp is taken from
(*pcap_t)->Buffer. That buffer is allocated only once and is never
reallocated for the duration of pcap session, and data is not moved or
copied by wpcap.dll. Consequently this makes me assume that the exception is
not caused by an error in wpcap.dll, but instead there's an error in the
driver that caused the region of memory containing pcap_t structure to be
overwritten with 'bad' data.
2. Let's take a look at the driver's code (packetNtx/DRIVER/Read.c)
This file apparently contains logic to deal with transferring captured
packets from the underlying system network driver to the user memory (i.e.
wpcap.dll). The driver maintains a circular buffer for packet data: the
incoming packets are put the 'end' of it causing Btail and BlastByte to be
updated; when the data is sent to the dll Bhead is updated thus freeing
buffer space.
Read.c has two functions of interest: Packet_tap, which obviously retrieves
a packed from network driver and adds it to the circular buffer, and
PacketRead, which copies the accumulated packets to the supplied dll buffer.
Now the problem:
The following fragment is executed when a new packet needs to be added to
the buffer.
Packet_tap (lines 331-341):
maxbufspace=fres+sizeof(struct bpf_hdr);
if(Ttail+maxbufspace>=Open->BufSize){
if(Thead<=maxbufspace)
{
//the buffer is full: the packet is lost
Open->Dropped++;
return NDIS_STATUS_NOT_ACCEPTED;
}
else{
Ttail=0;
}
}
Let's consider the situation when 0 < Ttail < Thead < TLastByte <
BufferLength
plus (Thead - Ttail) is small and (BufferLength - Ttail) is small (*), that
is the buffer is almost full and wraps around the end.
Buffer occupancy:
X X X X X X * * * X X
^ ^
| Thead
Ttail
where X indicates occupied space, * - free space
This situation seems possible according to the code of Packet_tap and
PacketRead.
Then it is quite possible that for the incoming packet
"Ttail+maxbufspace>=Open->BufSize" is true
and "Thead<=maxbufspace" is false
Normally this means that the buffer was not wrapped and we are getting close
to its end,
but under conditions *) it may occur with a wrapped buffer too.
Consequently Ttail will be reset to the beginning of the buffer and the old
data there will be overwritten with new packet:
!
Y Y Y x X X * * * X X
^ ^
| Thead
Ttail
where Y - new data, X - old data, * - free space
Apparently nothing can guarantee that the bytes stating with small 'x'
contain valid struct bpf_hdr now.
Let's consider PacketRead.
Only the read access to Bhead, Btail and BLastByte is synchronized with
Packet_tap, so it can happen that the two functions get called at about the
same time, retrieve the same values for Bhead, Btail and BLastByte and then
while Packet_tap overwrites the beginning of the buffer, PacketRead sends
the (corrupt) data to the DLL:
Ttail Thead
PacketRead: ! !
Y Y Y x X X * * * X X
Packet_tap: ^ ^
| Thead
Ttail
Who knows what will happen in the following fragment (PacketRead, lines
204--231) if it is processing invalid data.
//the buffer must be scannned to determine the number of bytes to copy
CpStart=Thead;
while(TRUE){
if(Thead==Ttail)break;
if(Thead==TLastByte){
PacketMoveMem(packp,CurrBuff+CpStart,Thead-CpStart,&(Open->Bhead));
packp+=(Thead-CpStart);
Open->Bhead=0;
Thead=0;
CpStart=0;
}
cplen=((struct bpf_hdr*)(CurrBuff+Thead))->bh_caplen+sizeof(struct
bpf_hdr);
if((i+cplen > Input_Buffer_Length)){//no more space in the application's
buffer
PacketMoveMem(packp,CurrBuff+CpStart,Thead-CpStart,&(Open->Bhead));
Irp->IoStatus.Information = i;
Irp->IoStatus.Status = STATUS_SUCCESS;
IoCompleteRequest(Irp, IO_NO_INCREMENT);
return(STATUS_SUCCESS);
}
cplen=Packet_WORDALIGN(cplen);
i+=cplen;
Thead+=cplen;
}
It seems to me that the situation described above can lead to DLL memory
corruption as a result of PacketRead's sending more data than the size of
DLL buffer.
Please note that winpcap sources that I used may not be the latest since
http://netgroup-serv.polito.it/winpcap/ seems to be down today so I could
not update.
------------------------------------------------------
Any thoughts?
Ciao,
Gordon
Sitraka -- the Java advantage
http://www.sitraka.com/
