SNMP async polling was also specifically what I wanted to rework the api
for ^_^. I actually never implemented any of the synchronous API
functions, I ran out of time during our 'hack week.'
Perhaps this is a better link of my specific work;
https://sourceforge.net/u/hcf64/net-snmp/ci/python_api/tree/python/netsnmp/snmp_api.c
I've left the rest of the netsnmp/ python module basically untouched.
I think we've done the same things with different approaches.
Whereas your get_async.c internalizes a select loop, the snmp_api approach
lets you perform select in the (python) caller. This is a tradeoff -
obvious performance advantages in native C versus including more business
logic inside the polling loop in Python. My personal (and totally
subjective) opinion is that I would probably just write the entire
application in C if I were considering including the select loop there (and
I have done that at previous jobs ^_^).
I guess that's the root of my (hopefully fair) criticism of get_async.c;
you are basically using python to supply an iterable of hosts, and it is
not doing much else for you. This can just as easily be done in C with not
very much additional effort, having tightly coupled the rest of the
application in C...and that's perfectly fine and fair thing to do, my last
large network SNMP poller for billing data was all C from config (lex/yacc)
to storage.
Just as a contrast, in snmp_api.c's approach you would get the typified
model of async high resolution large scale polling this way;
from __future__ import print_function
> import ipaddr # undocumented requirement :(
> from netsnmp import snmp_api
> import select
> class SnmpEx(object):
> OIDS = [[1, 3, 6, 1, 2, 1, 1,
> idx
> , 0] for
> idx
> in range(1, 9)]
> # sys MIB for ex
>
> ERR_TO_NAME = {value: key for key, value in snmp_api.errorcodes.items()}
> def __init__(self, hostname, community, queries, oids):
> sess_cfg = snmp_api.snmp_sess_init()
> sess_cfg['peername'] = hostname
> sess_cfg['timeout'] = 250000
> sess_cfg['retries'] = 0
> sess_cfg['community'] = community
> sess_cfg['version'] = snmp_api.versions['2c']
> sess_cfg['callback'] = self.callback
> self._max_in_flight = 5
> self._in_flight = 0
> self._session = snmp_api.snmp_open(sess_cfg)
> def callback(self, operation, reqid, pdu, raw_pdu):
> if pdu['errstat'] != 0:
> errstat = pdu['errstat']
> print('SNMP Error: %s/%d %s' %
> (operation, reqid, self.ERR_TO_NAME.get(errstat,
> str(errstat))),
> file=sys.stderr)
> return 1
> if operation == 'RECEIVED_MESSAGE':
> delta = time.time() - self.send_time
> self.continue_sending()
> return 1
> def continue_
> sending
> (self):
> # Or whatever your work flow is.
> while self._in_flight < self._max_in_flight:
> pdu = snmp_api.snmp_pdu_create(snmp_api.commands['GET'])
> for oid in self.OIDS:
> snmp_api.snmp_add_null_var(pdu, oid)
> if snmp_api.snmp_send(self._session, pdu) == 0:
> snmp_api.snmp_perror('snmp_bench: snmp_send(session, pdu):')
> break
> else:
> self._in_flight += 1
> def run(self):
> self.continue_
> sending
> ()
> while True:
> (fd_set, timeout) = snmp_api.snmp_select_info(self._session)
> # Be very carfeful here. 'timeout' can be 0.0 with pending queries!
> if timeout is not None:
> break
> (rset, wset, eset) = select.select(fd_set, [], [], timeout)
> if rset:
> snmp_api.snmp_read(rset)
> else:
> snmp_api.snmp_timeout()
> snmp_api.snmp_close(self._session)
Sorry for formatting. And my thinking is that this could be the root of a
new net-snmp/python/netsnmp/netsnmp.py. Which you can then sub-class on
that day when you have to, and do a piece differently ... include your own
event timers to send out the queries in constant time, etc etc. Again just
contrasting that with getting out the C source code and building new static
wheels or eggs.
The Net-SNMP code has grown organically over many years (and I've been
writing for it since CMU-SNMP, so, it's a little easier to navigate for me
maybe). I wouldn't say it's disgusting, but it has certainly grown ugly.
I'd just rather deal with its idiosyncracies and warts on Python's side,
where I have more tools at my disposal to smooth its edges and where I'm
not as "locked in" to one application model. It's easier on the python
side to quickly change how the application works.
I don't really IRC, but stay in touch!
On Thu, Mar 17, 2016 at 11:00 AM, Gabe <r...@un1x.su> wrote:
> Hi David,
>
> Glad to see some shared interest in this!
>
> I've made a lot of updates to my project since my posting to the mailing
> list, including Python 2 (tested with 2.7 at least) support.
>
> The only thing I needed to add to facilitate both Py2 and Py3
> functionality was add a macro in the C code to interface with the PyString
> vs PyUnicode as seen on the top here:
> https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/_api.h
>
> I took a lazier route in terms of interfacing with C from Python. I didn't
> implement the PDU structures, and only partially implemented the session
> structure. Currently my project lacks all the stuff required for SNMPv3,
> for example.
>
> I only focused on implementing the basic (v1/v2c) polling functions
> (GET/GETNEXT/WALK), but basic (v1/v2c) SET should just be a matter of
> cloning one of the existing functions, adding a value variable somewhere,
> and updating snmp_pdu_create(<type>) call.
>
> --
>
> Something you may be interested in looking at is the asynchronous polling
> methodology I developed;
>
> https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/get_async.c
> https://github.com/xstaticxgpx/netsnmp-py3/blob/master/test_async.py
>
> There's some cruft in that test_async.py you can ignore, but it boils down
> to this:
>
> Using Python multiprocessing, spawn ZeroMQ IPC receivers, then pass 4096
> chunks of devices to multiple get_async() C function instances, all of
> which communicate back via the ZeroMQ IPC socket. I was stuffing the
> returned data into redis in real time, but there are other ways to deal
> with it.
>
> There's a lot going on there and I'm having a hard time typing up a decent
> overview. Basically what this allows is to do MASSIVE amounts of SNMP
> polling, very efficiently.
>
> For example, we had a use case where we needed to poll (SNMPGET) 8-12 OIDs
> from upwards of 14 million devices. This get_async methodology could poll,
> from a single Dell R710 (dual xeon, single 1GB nic), all that information
> in under 20 minutes. This was all to unique, remote, devices.
>
> If you were just polling basic sysDesc or sysUptime, it could do ~14
> million unique polls in 7 minutes if I'm remember correctly. The memory
> utilization was totally reasonable during the entire run too, somehow I
> managed to avoid any serious memory leaks (at least in my experience)...
>
> I had left the position where I was working on this use case, however I
> was looking to extend this to be distributable across more than 1 host, but
> I hadn't made any progress on that front.
>
> One of the last things I did however, was *try* to reduce overhead during
> PDU creation.
>
> https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/interface.c#L148
> Here you would pass a OID string, and it would return the void pointer for
> that object as a PyLong object, so you can then re-pass that into C...as
> seen here:
>
> https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/get_async.c#L140
>
> https://github.com/xstaticxgpx/netsnmp-py3/blob/master/netsnmp/get_async.c#L147
>
> In practice, I believe I saw the snmp_clone_pdu() function was doing the
> same process as creating a brand new PDU object, so I don't believe that
> had any beneficial impact on performance.
>
> --
>
> Depending on what you're trying to accomplish, the above may or may not
> make any sense, or seem appealing. I would love to speak with you further
> regarding it if you're interested, perhaps on IRC somewhere?
>
> Personally, I found the net-snmp source code disgustingly convoluted and
> hard to work with. My opinion is that trying to replicate it 1:1 into
> python would not be very beneficial.
>
> Other than that, best of luck in your endeavors!! Hope some of this wall
> of text helped, please feel free to dig through my source code on github...
>
> -Gabe
>
> On Thu, Mar 17, 2016 at 1:05 PM, David Hankins <dhank...@twitter.com>
> wrote:
>
>> Hi Gabe,
>>
>> I'm also working on improving the python bindings (tho I'm not targeting
>> python 3 specifically, I'm trying to be future proof there);
>>
>> https://sourceforge.net/u/hcf64/net-snmp/ci/python_api/tree/python/
>>
>> My approach was also a little different; I made the new python bindings
>> (snmp_api.c) as close to a 1:1 map with Net-SNMP API as possible. My
>> strategy there is to make the "pythonic" SNMP library on python's side
>> (either a rewrite of netsnmp.py, or a different module).
>>
>> In retrospect one thing I think I would do differently would be to use
>> custom types for the PDU and the two different types of Session objects
>> (snmp_open() vs snmp_sess_open()), rather than the Capsule. The
>> implementation I have requires I transform the PDU structure into a
>> dictionary to return to the caller. This creates many python objects and
>> causes a GC storm later in heavy use. A custom type wrapping the Net-SNMP
>> structs would construct objects on demand, so I don't think it would have
>> that GC penalty, and it would be more reverse compatible (to 2.6 and
>> earlier).
>>
>> I also think I should consider a new strategy for freeing and taking the
>> GIL, to handle threaded python applications.
>>
>> On Thu, Mar 3, 2016 at 9:48 AM, Wes Hardaker <w...@net-snmp-pro.com>
>> wrote:
>>
>>> Gabe <r...@un1x.su> writes:
>>>
>>> > I've been working on a Python3 C Extension for the net-snmp API. Still
>>> very much a work in progress, but I feel like
>>> > it's at a point now that I can share:
>>> >
>>> > https://github.com/xstaticxgpx/netsnmp-py3
>>> >
>>> > Mostly just wanted to gauge interest on this, however any feedback
>>> > would be much appreciated.
>>>
>>> Glad to hear someone is plugging away at it. Did you eventually want to
>>> contribute it back to the Net-SNMP project, or keep it as a separate
>>> piece of work?
>>> --
>>> Wes Hardaker
>>> Please mail all replies to net-snmp-coders@lists.sourceforge.net
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Transform Data into Opportunity.
>>> Accelerate data analysis in your applications with
>>> Intel Data Analytics Acceleration Library.
>>> Click to learn more.
>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
>>> _______________________________________________
>>> Net-snmp-coders mailing list
>>> Net-snmp-coders@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/net-snmp-coders
>>>
>>
>>
>>
>> --
>> Network Tools & Automation
>> Twitter, Inc.
>>
>
>
--
Network Tools & Automation
Twitter, Inc.
------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
_______________________________________________
Net-snmp-coders mailing list
Net-snmp-coders@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/net-snmp-coders
