Bart Van Assche wrote:
> On Thu, Aug 6, 2009 at 12:02 PM,
>> Any opinions of whether this error could be causing my glibc double free
>> errors, or whether it is a red-herring?
>
> It is unlikely that the above message printed by Valgrind is related
> to the double free reported by glibc.
>
> Try to trigger as much code as you can in your application and in
> Net-SNMP and see whether this makes Valgrind report more errors.
>
> Bart.
Hi,
I left valgrind running for over 20 hours with the application doing
some simple snmpget-ing. After 8.5 hours and 29,664 SNMP gets - I got
the first valgrind errors - which were all from a single call to
snmp_free_pdu with a pdu that had already been freed after a
snmp_synch_response returned a STAT_ERROR. No other valgrind errors were
received.
Normally the application would crash, but it seems valgrind allows it to
continue - and the application ran for another 9 hours 22 minutes, and
32,767 SNMP gets with no valgrind errors, and then I got the same
valgrind errors complaining that the pdu had already been freed. The
only valgrind errors I could find where as a direct result of memory
being freed twice.
I have also noticed that if I do enough localhost snmpwalks - each
around 3000 SNMP gets - I can reproduce the same problem. It seems after
around 10 attempts on average, sometimes more, sometimes less. This may
be just a fluke, or my imagination...
The problem seems to come down to the PDU being freed once when a
STAT_ERROR is generated in snmp_synch_response_cb (snmp_client.c:1000):
999 if ((state->reqid = snmp_send(ss, pdu)) == 0) {
1000 snmp_free_pdu(pdu);
1001 state->status = STAT_ERROR;
1002 } else
1003 state->waiting = 1;
And once again when snmp_close() is called as a result of this error. In
snmp_sess_close() - the PDU that was freed above at snmp_client.c:1000
is still on the input request list associated with the session at
snmp_api.c:1886:
1847 /*
1848 * Close the input session. Frees all data allocated for the session,
1849 * dequeues any pending requests, and closes any sockets allocated for
1850 * the session. Returns 0 on error, 1 otherwise.
1851 */
1852 int
1853 snmp_sess_close(void *sessp)
1854 {
1855 struct session_list *slp = (struct session_list *) sessp;
1856 netsnmp_transport *transport;
1857 struct snmp_internal_session *isp;
1858 netsnmp_session *sesp = NULL;
1859 struct snmp_secmod_def *sptr;
1860
1861 if (slp == NULL) {
1862 return 0;
1863 }
1864
1865 if (slp->session != NULL &&
1866 (sptr = find_sec_mod(slp->session->securityModel)) != NULL &&
1867 sptr->session_close != NULL) {
1868 (*sptr->session_close) (slp->session);
1869 }
1870
1871 isp = slp->internal;
1872 slp->internal = 0;
1873
1874 if (isp) {
1875 netsnmp_request_list *rp, *orp;
1876
1877 SNMP_FREE(isp->packet);
1878
1879 /*
1880 * Free each element in the input request list.
1881 */
1882 rp = isp->requests;
1883 while (rp) {
1884 orp = rp;
1885 rp = rp->next_request;
1886 snmp_free_pdu(orp->pdu);
1887 free((char *) orp);
1888 }
1889
1890 free((char *) isp);
1891 }
...
The following is a dbg session for a standard snmpwalk - where you can
see the pdu 0x9050960 was freed at snmp_client.c:1000 and was still on
the request list at snmp_api.c:1885, and was freed again at
snmp_api.c:1886, causing the double free.
Is this a bug, or a symptom of some other issue?
Any ideas, suggestions for other things to try would be much appreciated
- as this problem is slowly driving me mad!
Many thanks,
Craig
# gdb /usr/local/bin/snmpwalk
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(gdb) set args -v2c -cpublic localhost
(gdb) b snmp_client.c:1000
No source file named snmp_client.c.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (snmp_client.c:1000) pending.
(gdb) run
Starting program: /usr/local/bin/snmpwalk -v2c -cpublic localhost
SNMPv2-MIB::sysDescr.0 = STRING: Linux *** 2.6.18-128.1.10.el5 #1 SMP
Thu May 7 10:35:59 EDT 2009 x86_64
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (7619945) 21:09:59.45
SNMPv2-MIB::sysContact.0 = STRING: ***
SNMPv2-MIB::sysName.0 = STRING: ***
SNMPv2-MIB::sysLocation.0 = STRING: Unknown
...
...
...
IP-MIB::ipSystemStatsInUnknownProtos.ipv4 = Counter32: 0
IP-MIB::ipSystemStatsInForwDatagrams.ipv4 = Counter32: 0
IP-MIB::ipSystemStatsHCInForwDatagrams.ipv4 = Counter64: 0
IP-MIB::ipSystemStatsReasmReqds.ipv4 = Counter32: 0
IP-MIB::ipSystemStatsReasmOKs.ipv4 = Counter32: 0
IP-MIB::ipSystemStatsReasmFails.ipv4 = Counter32: 0
IP-MIB::ipSystemStatsInDiscards.ipv4 = Counter32: 0
Breakpoint 1, snmp_synch_response_cb (ss=0x8fffab0, pdu=0x9050960,
response=0x7fff36a5a2d8, pcb=0x2b347407cf6c <snmp_synch_input>) at
snmp_client.c:1000
1000 snmp_free_pdu(pdu);
(gdb) list
995 cbmagsav = ss->callback_magic;
996 ss->callback = pcb;
997 ss->callback_magic = (void *) state;
998
999 if ((state->reqid = snmp_send(ss, pdu)) == 0) {
1000 snmp_free_pdu(pdu);
1001 state->status = STAT_ERROR;
1002 } else
1003 state->waiting = 1;
1004
(gdb) print state->reqid
$1 = 0
(gdb) print pdu
$2 = (netsnmp_pdu *) 0x9050960
(gdb) print *pdu
$3 = {version = 1, command = 161, reqid = 0, msgid = 32703, transid = 0,
sessid = 0, errstat = 0, errindex = 0, time = 0, flags = 512,
securityModel = -1,
securityLevel = 0, msgParseModel = 0, transport_data = 0x0,
transport_data_length = 0, tDomain = 0x0, tDomainLen = 0, variables =
0x9050a80,
community = 0x9050320 "public", community_len = 6, enterprise = 0x0,
enterprise_length = 0, trap_type = 0, specific_type = 0,
agent_addr = "\000\000\000", contextEngineID = 0x0,
contextEngineIDLen = 0, contextName = 0x0, contextNameLen = 0,
securityEngineID = 0x0,
securityEngineIDLen = 0, securityName = 0x0, securityNameLen = 0,
priority = 0, range_subid = 0, securityStateRef = 0x0}
(gdb) n
1001 state->status = STAT_ERROR;
(gdb) print pdu
$4 = (netsnmp_pdu *) 0x9050960
(gdb) n
1005 while (state->waiting) {
(gdb) print state->waiting
$5 = 0
(gdb) s
1052 *response = state->pdu;
(gdb) bt
#0 snmp_synch_response_cb (ss=0x8fffab0, pdu=0x9050960,
response=0x7fff36a5a2d8, pcb=0x2b347407cf6c <snmp_synch_input>) at
snmp_client.c:1052
#1 0x00002b347407e4e9 in snmp_synch_response (ss=0x8fffab0,
pdu=0x9050960, response=0x7fff36a5a2d8) at snmp_client.c:1062
#2 0x0000000000401360 in main (argc=<value optimized out>, argv=0x0) at
snmpwalk.c:279
(gdb) s
1053 ss->callback = cbsav;
(gdb) s
1054 ss->callback_magic = cbmagsav;
(gdb) s
1055 return state->status;
(gdb) s
1056 }
(gdb) s
snmp_synch_response (ss=0x8fffab0, pdu=0x9050960,
response=0x7fff36a5a2d8) at snmp_client.c:1063
1063 }
(gdb) s
main (argc=<value optimized out>, argv=0x0) at snmpwalk.c:280
280 if (status == STAT_SUCCESS) {
(gdb) s
279 status = snmp_synch_response(ss, pdu, &response);
(gdb) print status
$6 = 0
(gdb) s
280 if (status == STAT_SUCCESS) {
(gdb) n
350 } else if (status == STAT_TIMEOUT) {
(gdb) n
356 snmp_sess_perror("snmpwalk", ss);
(gdb) n
snmpwalk:
360 if (response)
(gdb) s
269 while (running) {
(gdb) s
363 if (netsnmp_ds_get_boolean(NETSNMP_DS_APPLICATION_ID,
(gdb) s
netsnmp_ds_get_boolean (storeid=1, which=4) at default_store.c:240
240 if (storeid < 0 || storeid >= NETSNMP_DS_MAX_IDS ||
(gdb) s
245 return (netsnmp_ds_booleans[storeid][which/8] & (1 << (which
% 8))) ? 1:0;
(gdb) s
246 }
(gdb) s
main (argc=<value optimized out>, argv=<value optimized out>) at
snmpwalk.c:367
367 if (numprinted == 0 && status == STAT_SUCCESS) {
(gdb) s
377 snmp_close(ss);
(gdb) s
snmp_close (session=0x8fffab0) at snmp_api.c:1931
1931 struct session_list *slp = NULL, *oslp = NULL;
(gdb) list
1926 }
1927
1928 int
1929 snmp_close(netsnmp_session * session)
1930 {
1931 struct session_list *slp = NULL, *oslp = NULL;
1932
1933 { /*MTCRITICAL_RESOURCE */
1934 snmp_res_lock(MT_LIBRARY_ID, MT_LIB_SESSION);
1935 if (Sessions && Sessions->session == session) { /* If
first entry */
(gdb) s
1935 if (Sessions && Sessions->session == session) { /* If
first entry */
(gdb) s
1936 slp = Sessions;
(gdb) s
1937 Sessions = slp->next;
(gdb) s
1935 if (Sessions && Sessions->session == session) { /* If
first entry */
(gdb) s
1950 if (slp == NULL) {
(gdb) s
1953 return snmp_sess_close((void *) slp);
(gdb) s
snmp_sess_close (sessp=0x90419a0) at snmp_api.c:1855
1855 struct session_list *slp = (struct session_list *) sessp;
(gdb) s
1858 netsnmp_session *sesp = NULL;
(gdb) s
1861 if (slp == NULL) {
(gdb) s
1865 if (slp->session != NULL &&
(gdb) s
find_sec_mod (secmod=3) at snmp_secmod.c:145
145 for (sptr = registered_services; sptr; sptr = sptr->next) {
(gdb) n
146 if (sptr->securityModel == secmod) {
(gdb) n
147 return sptr->secDef;
(gdb) n
154 }
(gdb) n
snmp_sess_close (sessp=0x90419a0) at snmp_api.c:1871
1871 isp = slp->internal;
(gdb) s
1872 slp->internal = 0;
(gdb) s
1874 if (isp) {
(gdb) s
1877 SNMP_FREE(isp->packet);
(gdb) n
1882 rp = isp->requests;
(gdb) print isp->requests
$7 = (netsnmp_request_list *) 0x90503f0
(gdb) print *isp->requests
$8 = {next_request = 0x0, request_id = 0, message_id = 32703, callback =
0, cb_data = 0x0, retries = 0, timeout = 1000000, time = {tv_sec =
1250062211,
tv_usec = 355904}, expire = {tv_sec = 1250062212, tv_usec =
355904}, session = 0x0, pdu = 0x9050960}
(gdb) s
1883 while (rp) {
(gdb) s
1884 orp = rp;
(gdb) s
1885 rp = rp->next_request;
(gdb) print orp
$9 = (netsnmp_request_list *) 0x90503f0
(gdb) print *orp
$10 = {next_request = 0x0, request_id = 0, message_id = 32703, callback
= 0, cb_data = 0x0, retries = 0, timeout = 1000000, time = {tv_sec =
1250062211,
tv_usec = 355904}, expire = {tv_sec = 1250062212, tv_usec =
355904}, session = 0x0, pdu = 0x9050960}
(gdb) print orp->pdu
$11 = (netsnmp_pdu *) 0x9050960
(gdb) print *orp->pdu
$12 = {version = 0, command = 0, reqid = 0, msgid = 0, transid = 0,
sessid = 0, errstat = 0, errindex = 0, time = 0, flags = 0,
securityModel = 0,
securityLevel = 0, msgParseModel = 0, transport_data = 0x0,
transport_data_length = 0, tDomain = 0x0, tDomainLen = 0, variables =
0x0, community = 0x0,
community_len = 0, enterprise = 0x0, enterprise_length = 0, trap_type
= 0, specific_type = 0, agent_addr = "\000\000\000", contextEngineID = 0x0,
contextEngineIDLen = 0, contextName = 0x0, contextNameLen = 0,
securityEngineID = 0x0, securityEngineIDLen = 0, securityName = 0x0,
securityNameLen = 0,
priority = 0, range_subid = 0, securityStateRef = 0x0}
(gdb) s
1886 snmp_free_pdu(orp->pdu);
(gdb) s
snmp_free_pdu (pdu=0x9050960) at snmp_api.c:5007
5007 if (!pdu)
(gdb) s
5030 if ((sptr = find_sec_mod(pdu->securityModel)) != NULL &&
(gdb) n
5034 snmp_free_varbind(pdu->variables);
(gdb) n
5035 SNMP_FREE(pdu->enterprise);
(gdb) n
5036 SNMP_FREE(pdu->community);
(gdb) n
5037 SNMP_FREE(pdu->contextEngineID);
(gdb) n
5038 SNMP_FREE(pdu->securityEngineID);
(gdb) n
5039 SNMP_FREE(pdu->contextName);
(gdb) n
5040 SNMP_FREE(pdu->securityName);
(gdb) n
5041 SNMP_FREE(pdu->transport_data);
(gdb) n
5042 memset(pdu, 0, sizeof(netsnmp_pdu));
(gdb) n
5043 free((char *) pdu);
(gdb) n
*** glibc detected *** /usr/local/bin/snmpwalk: double free or
corruption (top): 0x0000000009050960 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3d0de71ce2]
/lib64/libc.so.6(cfree+0x8c)[0x3d0de7590c]
/usr/local/lib/libnetsnmp.so.15(snmp_free_pdu+0x1b8)[0x2b34740aae86]
/usr/local/lib/libnetsnmp.so.15(snmp_sess_close+0xeb)[0x2b347409eea0]
/usr/local/lib/libnetsnmp.so.15(snmp_close+0xc8)[0x2b347409f0e7]
/usr/local/bin/snmpwalk[0x4013f9]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3d0de1d974]
/usr/local/bin/snmpwalk[0x400e19]
======= Memory map: ========
00400000-00402000 r-xp 00000000 fd:00 14915774
/usr/local/bin/snmpwalk
00602000-00603000 rw-p 00002000 fd:00 14915774
/usr/local/bin/snmpwalk
08fcd000-09071000 rw-p 08fcd000 00:00 0
[heap]
3d0da00000-3d0da1c000 r-xp 00000000 fd:00 44236802
/lib64/ld-2.5.so
3d0dc1b000-3d0dc1c000 r--p 0001b000 fd:00 44236802
/lib64/ld-2.5.so
3d0dc1c000-3d0dc1d000 rw-p 0001c000 fd:00 44236802
/lib64/ld-2.5.so
3d0de00000-3d0df4c000 r-xp 00000000 fd:00 44236805
/lib64/libc-2.5.so
3d0df4c000-3d0e14c000 ---p 0014c000 fd:00 44236805
/lib64/libc-2.5.so
3d0e14c000-3d0e150000 r--p 0014c000 fd:00 44236805
/lib64/libc-2.5.so
3d0e150000-3d0e151000 rw-p 00150000 fd:00 44236805
/lib64/libc-2.5.so
3d0e151000-3d0e156000 rw-p 3d0e151000 00:00 0
3d0e200000-3d0e202000 r-xp 00000000 fd:00 44236812
/lib64/libdl-2.5.so
3d0e202000-3d0e402000 ---p 00002000 fd:00 44236812
/lib64/libdl-2.5.so
3d0e402000-3d0e403000 r--p 00002000 fd:00 44236812
/lib64/libdl-2.5.so
3d0e403000-3d0e404000 rw-p 00003000 fd:00 44236812
/lib64/libdl-2.5.so
3d0ee00000-3d0ee14000 r-xp 00000000 fd:00 14916464
/usr/lib64/libz.so.1.2.3
3d0ee14000-3d0f013000 ---p 00014000 fd:00 14916464
/usr/lib64/libz.so.1.2.3
3d0f013000-3d0f014000 rw-p 00013000 fd:00 14916464
/usr/lib64/libz.so.1.2.3
3d0fe00000-3d0fe0d000 r-xp 00000000 fd:00 44236827
/lib64/libgcc_s-4.1.2-20080825.so.1
3d0fe0d000-3d1000d000 ---p 0000d000 fd:00 44236827
/lib64/libgcc_s-4.1.2-20080825.so.1
3d1000d000-3d1000e000 rw-p 0000d000 fd:00 44236827
/lib64/libgcc_s-4.1.2-20080825.so.1
3d15600000-3d1572d000 r-xp 00000000 fd:00 44236859
/lib64/libcrypto.so.0.9.8e
3d1572d000-3d1592c000 ---p 0012d000 fd:00 44236859
/lib64/libcrypto.so.0.9.8e
3d1592c000-3d1594d000 rw-p 0012c000 fd:00 44236859
/lib64/libcrypto.so.0.9.8e
3d1594d000-3d15951000 rw-p 3d1594d000 00:00 0
2b347404d000-2b347404e000 rw-p 2b347404d000 00:00 0
2b3474066000-2b3474067000 rw-p 2b3474066000 00:00 0
2b3474067000-2b347410d000 r-xp 00000000 fd:00 14915178
/usr/local/lib/libnetsnmp.so.15.1.2
2b347410d000-2b347430d000 ---p 000a6000 fd:00 14915178
/usr/local/lib/libnetsnmp.so.15.1.2
2b347430d000-2b3474311000 rw-p 000a6000 fd:00 14915178
/usr/local/lib/libnetsnmp.so.15.1.2
2b3474311000-2b3474348000 rw-p 2b3474311000 00:00 0
2b3474348000-2b3477916000 r--p 00000000 fd:00 14929693
/usr/lib/locale/locale-archive
2b3477916000-2b3477917000 rw-p 2b3477916000 00:00 0
2b347792e000-2b3477938000 r-xp 00000000 fd:00 44237085
/lib64/libnss_files-2.5.so
2b3477938000-2b3477b37000 ---p 0000a000 fd:00 44237085
/lib64/libnss_files-2.5.so
2b3477b37000-2b3477b38000 r--p 00009000 fd:00 44237085
/lib64/libnss_files-2.5.so
2b3477b38000-2b3477b39000 rw-p 0000a000 fd:00 44237085
/lib64/libnss_files-2.5.so
2b3478000000-2b3478021000 rw-p 2b3478000000 00:00 0
2b3478021000-2b347c000000 ---p 2b3478021000 00:00 0
7fff36a47000-7fff36a5c000 rw-p 7ffffffea000 00:00 0
[stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0
Program received signal SIGABRT, Aborted.
0x0000003d0de30215 in raise () from /lib64/libc.so.6
(gdb)
__________ Information from ESET Smart Security, version of virus signature
database 4328 (20090812) __________
The message was checked by ESET Smart Security.
http://www.eset.com
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Net-snmp-coders mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/net-snmp-coders
