Greetings,
In testing agentx sub-agent notifications I have run into problems. I
have a subagent with an alarm callback which wakes every 2 seconds to
send a notification.
I consistently see the agent hang within a few hours. In the agent
trace I see what I believe is a ping from the subagent. I believe it is
a ping because 1) it is time for the periodic subagent to agent ping,
and 2) byte count received = 20.
BUT, the data shown for the notification by the agent trace is
identical to the previous notification (same packet IDs, system up time,
...) the agent processes this as a notification sending it out to snmptrapd.
The subagent follows this by sending another ping which appeared to be
handled correctly by agent. This is the last transaction in the trace.
The subagent and snmptrapd both timeout and give up. Agent will not
respond to snmpget.
I am including the trace records of the final transactions and other
info below.
Has anyone experienced this and know of a fix? Should this be logged as
a bug?
Thanks for advice,
Dean
The final transactions in trace
===============================
SUBAGENT --- notif 1 --> SNMPD
SNMPD --- notif 1 --> SNMPTRAPD
SNMPD <-- resp 1 --- SNMPTRAPD
SUBAGENT <-- resp 1 --- SNMPD
SUBAGENT --- notif 2 --> SNMPD
SNMPD --- notif 2 --> SNMPTRAPD
SUBAGENT --- ping ? --> SNMPD
* IT APPEARS TO BE A PING
SNMPD --- notif 2 --> SNMPTRAPD
* ITS SEEN AS THE
PRIOR NOTIFICATION 2
SNMPD <-- resp 2 --- SNMPTRAPD
SNMPD <-- resp 2 --- SNMPTRAPD
SUBAGENT <-- resp 2 --- SNMPD
SUBAGENT --- ping --> SNMPD
SUBAGENT <-- ping rsp --- SNMPD
Trace records that appear to me to be the problem
(Subagent appears to be at fd 14, and
snmptrapd on fd 11)
=================================================
* AGENT COMPLETES SEN*DING NOTIF TO HANDLER
netsnmp_udp: send 109 bytes from 0x80f60a3 to UDP: [127.0.0.1]:162
on fd 11
* AGENT RECEIVES 20 BYTES FROM SUBAGENT (I BELIEVE A PING)
* AND THINKS ITS A NOTIFICATION
trace: snmp_sess_select_info(): snmp_api.c, 5700:
sess_select: for all sessions: 14 13 12 10 11 (to in
1126558735.358313 sec) 8 6 4
sess_select: next alarm 0.0 sec
verbose:sess_select: timer due *real* soon. 100 usec
verbose:sess_select: setting timer to 0.100 sec, clear block (was 1)
trace: netsnmp_unix_recv(): snmpUnixDomain.c, 145:
netsnmp_unix: recv fd 14 got 20 bytes
dumpx_recv:8C 00 00 00
dumpv_recv: Integer: 140 (0x8C)
trace: _sess_read(): snmp_api.c, 5471:
sess_read: loop packet_len 180, PDU length 160
trace: _sess_process_packet(): snmp_api.c, 4971:
sess_process_packet: session 0x8106540 fd 14 pkt 0x80f5910
length 160
trace: agentx_parse_header(): mibgroup/agentx/protocol.c, 1481:
dumph_recv: AgentX Header
trace: agentx_parse_header(): mibgroup/agentx/protocol.c, 1482:
dumph_recv: Version
dumpx_recv: 01
dumpv_recv: Version: 1
trace: agentx_parse_header(): mibgroup/agentx/protocol.c, 1489:
dumph_recv: Command
dumpx_recv: 0C
dumpv_recv: Command: 12 (Notify)
trace: agentx_parse_header(): mibgroup/agentx/protocol.c, 1497:
dumph_recv: Flags
dumpx_recv: 00
-
-
-
The final records in the trace - following the ping response
============================================================
dumpv_send: errstat: 0
trace: _agentx_realloc_build(): mibgroup/agentx/protocol.c, 920:
dumpv_send: errindex: 0
trace: _agentx_realloc_build(): mibgroup/agentx/protocol.c, 932:
dumph_send: Get* Variable List
dumpx_send:08 00 00 00
dumpv_send: Integer: 8 (0x08)
trace: _agentx_realloc_build(): mibgroup/agentx/protocol.c, 1004:
agentx_build: packet built okay
trace: netsnmp_unix_send(): snmpUnixDomain.c, 159:
netsnmp_unix: send 28 bytes to 0x8111940 on fd 14
trace: free_agent_snmp_session(): snmp_agent.c, 1270:
snmp_agent: agent_session 0x81116d0 released
trace: free_agent_snmp_session(): snmp_agent.c, 1274:
verbose:asp: asp 0x81116d0 reqinfo 0x810e170 freed
trace: _sess_read(): snmp_api.c, 5289:
sess_read: not reading 13 (fdset 0xbfffe370 set 0)
trace: _sess_read(): snmp_api.c, 5289:
sess_read: not reading 12 (fdset 0xbfffe370 set 0)
trace: _sess_read(): snmp_api.c, 5289:
sess_read: not reading 10 (fdset 0xbfffe370 set 0)
SUBAGENT CONSOLE OUTPUT
=======================
2005-09-12 16:52:10 example-demon is up and running.
2005-09-12 16:59:16 AgentX master agent failed to respond to ping.
Attempting to re-register.
2005-09-12 16:59:28 snmpd: send_trap: Timeout
2005-09-12 16:59:30 snmpd: send_trap: Timeout
2005-09-12 16:59:32 snmpd: send_trap: Timeout
2005-09-12 16:59:34 snmpd: send_trap: Timeout
2005-09-12 16:59:36 snmpd: send_trap: Timeout
2005-09-12 16:59:38 snmpd: send_trap: Timeout
2005-09-12 16:59:40 snmpd: send_trap: Timeout
2005-09-12 16:59:42 snmpd: send_trap: Timeout
2005-09-12 16:59:49 snmpd: send_trap: Timeout
2005-09-12 16:59:51 snmpd: send_trap: Timeout
2005-09-12 16:59:53 snmpd: send_trap: Timeout
2005-09-12 16:59:55 snmpd: send_trap: Timeout
2005-09-12 16:59:57 snmpd: send_trap: Timeout
2005-09-12 17:00:04 snmpd: send_trap: Timeout
2005-09-12 17:00:06 snmpd: send_trap: Timeout
2005-09-12 17:00:08 snmpd: send_trap: Timeout
2005-09-12 17:00:10 snmpd: send_trap: Timeout
2005-09-12 17:00:12 snmpd: send_trap: Timeout
SNMPTRAPD CONSOLE OUTPUT (all appears normal until)
====================================================
AgentX master agent failed to respond to ping. Attempting to re-register.
STACK TRACE FOR SUBAGENT AND TRAP HANDLER BOTH SHOW
===================================================
#0 0x420e8042 in connect () from /lib/i686/libc.so.6
#1 0x4015fbbb in netsnmp_unix_transport (addr=0xbffff520, local=0) at
snmpUnixDomain.c:361
#2 0x4015fd46 in netsnmp_unix_create_tstring (string=0x80a6578
"/var/agentx/master", local=0) at snmpUnixDomain.c:406
#3 0x4015bf8c in netsnmp_tdomain_transport (string=0x80a6560
"/var/agentx/master", local=0, default_domain=0x4017707e "tcp") at
snmp_transport.c:351
#4 0x40131ab1 in _sess_open (in_session=0xbffff710) at snmp_api.c:1387
#5 0x40131ed5 in snmp_sess_open (pss=0xbffff710) at snmp_api.c:1565
#6 0x40130cd0 in snmp_open_ex (session=0xbffff710, fpre_parse=0,
fparse=0x400411c8 <agentx_parse>, fpost_parse=0, fbuild=0,
frbuild=0x4003edf0 <agentx_realloc_build>, fcheck=0x40042a24
<agentx_check_packet>) at snmp_api.c:908
#7 0x40038153 in subagent_open_master_session () at
mibgroup/agentx/subagent.c:719
#8 0x40038624 in agentx_reopen_session (clientreg=14, clientarg=0x0) at
mibgroup/agentx/subagent.c:838
#9 0x40158f22 in run_alarms () at snmp_alarm.c:251
-
-
-
STACK TRACE FOR AGENT
=====================
(gdb) where
#0 0x420e81c2 in recvfrom () from /lib/i686/libc.so.6
#1 0x401514d0 in netsnmp_udp_recv (t=0x80ebdb8, buf=0x81652b8,
size=65536, opaque=0xbfffe2c4, olength=0xbfffe2c8) at snmpUDPDomain.c:124
#2 0x40133b2e in _sess_read (sessp=0x80f3010, fdset=0xbfffe370) at
snmp_api.c:5415
#3 0x40134374 in snmp_sess_read (sessp=0x80f3010, fdset=0xbfffe370) at
snmp_api.c:5624
#4 0x401335ee in snmp_read (fdset=0xbfffe370) at snmp_api.c:5260
#5 0x4010fd82 in snmp_synch_response_cb (ss=0x80f4d90, pdu=0x81aa378,
response=0xbfffe458, pcb=0x4010f32c <snmp_synch_input>) at snmp_client.c:813
#6 0x4010fe3e in snmp_synch_response (ss=0x80f4d90, pdu=0x81aa378,
response=0xbfffe458) at snmp_client.c:851
#7 0x400cd57b in send_trap_to_sess (sess=0x80f4d90,
template_pdu=0x8176c20) at agent_trap.c:829
#8 0x4003ea10 in send_notifications (major=1, minor=7,
serverarg=0x8176c20, clientarg=0x0) at notification/snmpNotifyTable.c:127
#9 0x4014c5eb in snmp_call_callbacks (major=1, minor=7,
caller_arg=0x8176c20) at callback.c:224
#10 0x400cd434 in netsnmp_send_traps (trap=-1, specific=-1,
enterprise=0x400e8bf0, enterprise_length=10, vars=0x8104b40,
context=0x0, flags=0) at agent_trap.c:778
#11 0x400cd47a in send_enterprise_trap_vars (trap=-1, specific=-1,
enterprise=0x400e8bf0, enterprise_length=10, vars=0x8104b40) at
agent_trap.c:792
#12 0x400cd61f in send_trap_vars (trap=-1, specific=-1, vars=0x8104b40)
at agent_trap.c:849
#13 0x400e1fe9 in agentx_notify (session=0x810bda8, pdu=0x8105160) at
mibgroup/agentx/master_admin.c:440
#14 0x400e2366 in handle_master_agentx_packet (operation=1,
session=0x810bda8, reqid=917286845, pdu=0x8105160, magic=0x0) at
mibgroup/agentx/master_admin.c:532
#15 0x401334a5 in _sess_process_packet (sessp=0x810bd58, sp=0x810bda8,
isp=0x810bd70, transport=0x810bd10, opaque=0x8105cd0, olength=110,
packetptr=0x81652b8
"h\uffff\022Bh\uffff\022Bublic\uffff^\002\004=\uffff=}\002\001",
length=160) at snmp_api.c:5208
#16 0x4013404e in _sess_read (sessp=0x810bd58, fdset=0xbfffe910) at
snmp_api.c:5526
#17 0x40134374 in snmp_sess_read (sessp=0x810bd58, fdset=0xbfffe910) at
snmp_api.c:5624
#18 0x401335ee in snmp_read (fdset=0xbfffe910) at snmp_api.c:5260
#19 0x0804b6f6 in receive () at snmpd.c:1149
#20 0x0804b151 in main (argc=4, argv=0xbffffaf4) at snmpd.c:993
#21 0x42017499 in __libc_start_main () from /lib/i686/libc.so.6
STRACE FOR AGENT ( 11 == SNMPTRAPD)
================
recvfrom(11,
STRACE FOR SNMPTRAPD
====================
#> strace -p 18949
connect(8, {sin_family=AF_UNIX, path="/var/agentx/master"}, 110
STRACE FOR SUBAGENT
===================
#> strace -p 18965
connect(5, {sin_family=AF_UNIX, path="/var/agentx/master"}, 110
-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server.
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Net-snmp-coders mailing list
Net-snmp-coders@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/net-snmp-coders
