Hi, Issue: SNMPv3 bulkwalk()/callback blocking during background EngineID queries
I have been successfully using Perl SNMP package/Net-SNMP library for years polling 3000+ SNMPv2. At any one time, there are parallel queries to around 200 devices, and the non-blocking+callback solution works perfectly. I am currently running Ubuntu 22.04.5 LTS, Perl 5.34.0, SNMP.pm 5.0901 (5.9.1+dfsg-1ubuntu2.9 Perl module libsnmp-base modules)
However, the devices are now reconfigured for SNMPv3, and whilst my code still works fine, it runs very slowly when there are a number of devices unavailable (say 1%) at startup. (Such as when a remote site is offline/many devices under maintenance). My code consists of a loop with SNMP::MainLoop(), which periodically fires off various device bulkwalk()s with callback refs, (setting up a SNMP session beforehand, if none exists). The callbacks process the returned data, and undef the session, if necessary. I can provide a bare minimum code to demonstrate the issue.
From packet analysis, I see that when bulkwalk() is called, the SNMP library first queries that device's EngineID if this is not already cached. Despite using callbacks, bulkwalk() unexpectedly blocks until either the EngineID is returned and cached, or until the request to the device for the EngineID times out. If the device is unavailable, this blocks the running of SNMP::MainLoop() for up to the device's timeout/retry duration. Given that some devices are on some very slow links, the timeout to an unavailable device may be longer than that of a device that is part way through a large table bulkwalk(). When SNMP::MainLoop() is eventually called, the latter device may find itself timed-out, through no fault of its own - as any bulkwalk()'s blocking duration will add to their timeout.
To compound this, I appear to receive no callback in the event an unavailable device times-out (no EngineID received). I am also unable to close(undef) such a session, and the OS-allocated UDP session socket remains until the Perl program exits. I periodically retry unavailable devices to see if they have become available (I have no other way to tell), which blocks all other devices being polled if the device is still unavailable, and eats up a UDP socket on each occasion. At any point in time, for various reasons, I may have ~30 devices that don't respond. Each device is set to 3 retries of 2 seconds each, and each device will of course block sequentially (as its call to bulkwalk() blocks). I attempt to poll each of my devices every 5 minutes, and this means that the 30 devices block for around 3 minutes of that 5 minutes, and eat 360 UDP sockets per hour (or 8640 sockets per day).
Whilst I could implement some form of exponential back-off for polling, this doesn't escape from the fact I can only back-off for a short maximum period, merely slowing the issue. As UDP sockets are chewed-up, I need to restart the process to release them anyway, which just restarts the issue (and not withstanding the fact that ALL devices will then need to re-cache their EngineID's, and each of these devices will block sequentially on their first bulkwalk() ).
Should a single device's EngineID request block like this (or is there a way to make it non-blocking)? Even if I decrease the retries and timeouts, I'm unlikely to go below 1 second each, which will still impact around 30 seconds in each poll period. Is there a way to ensure that failed EngineID requests result in a callback, such that I can close and release the UDP socket back to the OS?
This aside, I have to say the library has been rock-solid for me for decades. Very many thanks indeed.
Regards, Paul S _______________________________________________ Net-snmp-coders mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/net-snmp-coders
