Hi Chris, Great news about nodeflow. Let me know when is appropriate to put a link on the pmacct webpage about it.
What you describe for timestamps seems a good match for NetFlow, ie. cast packets into flows and handle these via a flow-aware cache (so active/passive expiration timers, max lifetime, etc.). All described is already part of the nfprobe plugin. Collecting back such data via nfacctd (on the same box where NetFlow is exported or ship it to some central location) enables to use timestamp_start, timestamp_end aggregation primitives - which should be precisely what you want to achieve. The beauty is that you can have all time references possible at once: timetamp_start, timestamp_end, stamp_inserted, stamp_updated. Don't know how much you like/dislike the solution but i'd encourage to run a proof-of-concept with these tools (which are all available already) so to see we are in line with your requirements and hence take it from there. About sql_refresh_time less than one second. I've not considered it for a simple reason: it seems to me like forcing an existing caching mechanism towards a real-time use-case. Then better to disable it at all and stream flows as they arrive onto the AMQP exchange. I have this on my todo list - does it seem what you are looking for? Cheers, Paolo On Mon, Jun 02, 2014 at 03:31:32PM +0300, Chris Wilson wrote: > Hi Paolo, > > I've started work on a pure JavaScript replacement for pmGraph, so > it should be easier to install and more lightweight: > https://github.com/qris/nodeflow > > I have a small server-side component in Node which connect to RabbitMQ, > receives JSON flow data from pmacct, and relays it to connected > WebSocket clients (browsers). This is working. I'm currently working > on the client-side data processing, aggregation and graphing. > > I noticed that we don't currently have any timestamps in the JSON > objects that pmGraph sends to RabbitMQ: > > {"ip_dst": "86.30.131.x", "ip_proto": "tcp", "tos": 0, "ip_src": > "196.200.217.x", "bytes": 984, "port_dst": 2201, "packets": 12, > "port_src": 37201} > > Would you consider adding two timestamps, both in seconds (possibly > fractional) since epoch in UTC: start of bucket (what the timestamp > is rounded down to by sql_history_roundoff) and end of bucket > (stamp_inserted/updated?) so that we can know over what period the > data was collected for graphing purposes. > > We can guess a single timestamp based on the time that the data is > received, but it's not very reliable and requires us to know or > assume the sql_refresh_time and sql_history values configured in > pmacct. > > Also, have you given any thought to supporting sub-second timestamps > (e.g. sql_refresh_time/sql_history = 0.1s)? > > Thanks for your consideration! > > Cheers, Chris. > -- > Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838 > Citylife House, Sturton Street, Cambridge, CB1 2QF, UK > > Aptivate is a not-for-profit company registered in England and Wales > with company number 04980791. > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
