From my SNMP guy:
UDP port 161 is the default port on which SNMP listens for incoming snmp get/set requests. My guess is that somewhere on the network is an SNMP manager that is sending requests to that host. That being said, it looks to me like the host's SNMP master agent is not reading from the port and that is why the receive q is filling up. This could happen if a request results in snmpd getting into a deadlock situation.
I think I'd start trying to figure out the problem by using tcpdump to see if I could determine if, in fact, there are SNMP requests being sent to this host and if so, where they are coming from.
Regardless of whether or not there are SNMP requests, enable snmpd logging of everything and check that information when the receive q starts to fill up.
Altermately, tcpdump may tell you that the host is being flooded with SNMP requests. If this is the case, it may explain the system performance impact because for the most part, SNMP has to query for information from the kernel and/or kernel modules via ioctl/etc. If the host is being flooded with SNMP requests, it's possible that it may be doing nothing but servicing those requests.
At 02:44 PM 3/1/2005, Biggs, Eric J [ITS] wrote:
We are using RHEL AS 3.0 (update 3) on a z990, under z/VM 4.4 and have been experiencing a strange problem related to snmp and various other programs and utilities. Take a look at this netstat output for port 161:
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State udp 130928 0 0.0.0.0:161 0.0.0.0:*
Notice the huge amount of data in Recv-Q. This situation just builds over time. We haven't been able to determine anything that might be causing it. Here's the strange thing: Other programs and utilities get "stuck" when they are used whenever this "queue" exists for port 161. If we restart the snmpd service, this queue clears up and the other programs free up. As an example, if I enter a sudo command it just hangs the curser. I have to close the session. Same thing with the ifconfig command, it just hangs. This was also observed using Red Hat's "up2date" command for accessing Red Hat Network. The two issues are obviously related because I can open another session (after I've hung one of those commands), restart the snmpd service, then go back to the first window and the command completes. Weird. There are probably other commands that are impacted too, we just haven't found them yet.
The vendor support that we pay for (I won't mention any names) has not been able to shed much light on this issue. Does anyone have any ideas, or have you experienced a similar situation?
Eric Biggs
---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
