Hi, This is sort of a long story so grab a cup of coffee :)
I've been using ActiveHTTP::Request to issue HTTP requests from Bro scripts so far, but as it's using shell invocation of curl it is a) slow due to execution of external process b) unable to make use of HTTP keep-alive and c) hence unable to multiplex requests to same server. This causes whole lot of overhead with current ActiveHTTP::Request making it unusable for various interesting usecases, eg. querying data from ElasticSearch ;) I eventually started tweaking the code to incorporate libcurl functionality into Bro runtime and the blocking version came along just fine. Now as I am trying to make the libcurl function calls asynchronous, it's becoming really complex. There is curl_multi_fdset(multi_handle, &fdread, &fdwrite, &fdexcep, &maxfd); [1] which can be used to fetch socket descriptors associated with libcurl's connections. It is designed to populate existing fd_set struct with proper file handles to be spoon-fed to select(). Note that fd_set struct internal structure is not well defined (different on Windows and POSIX); macros FD_SET, FD_CLR, FD_ISSET and FD_ZERO are designed for fd_set manipulation and there is no macro to extract socket descriptors associated with particular fd_set. Trying to work with and around Bro's FD_Set and IOSource classes makes it nearly impossible to integrate libcurl as it is without a lot of dirty code. I also noticed that you've implemented asynchronous DNS client on top of your event loop handler mechanism. I assume this was built much earlier than any event handling library was conceived so it's sort of a legacy. Now looking how stuff is done nowadays you see there are a lot of event loop libraries out there which make more efficient use of kernel such as epoll() and kqueue in contrast to select() and already have built-in methods for asynchronous file input/output. Note that libev [3] also has built-in async DNS client. My conclusion at this very early stage is that it would make sense to substitute Bro's event loop and DNS client with libev. This should make it significantly easier to integrate with other libraries such as libcurl, take a look at libuv example of libcurl [4]. I am not sure how this would affect Bro runtime logic. Comments, questions and feedback on the ideas presented above are very much welcome :) 1. http://curl.haxx.se/libcurl/c/curl_multi_fdset.html 2. http://www.mkssoftware.com/docs/man3/select.3.asp 3. https://github.com/libuv/libuv 4. http://curl.haxx.se/libcurl/c/multi-uv.html -- Lauri Võsandi tel: +372 53329412 e-mail: [email protected] blog: http://lauri.vosandi.com/ _______________________________________________ bro-dev mailing list [email protected] http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
