Hi Nathan, Thanks for elaborating. With this extra info i was actually able to reproduce the issue and fix it - code in the CVS:
http://www.mail-archive.com/[email protected]/msg00851.html Cheers, Paolo On Wed, Oct 09, 2013 at 02:09:17AM +0000, Nathan Kennedy wrote: > Hi Paolo, > > Thanks for your assistance. > > Capturing method is NetFlow, and in this case I'm using a Netflow simulator > (http://www.paessler.com/tools/netflowgenerator), however I have > nfacctd_time_new set to true. > > I compiled using --enable-rabbitmq and --enable-jansson flags. > > The first cache purge looks to be correct, it's the subsequent ones that seem > to have the trouble. Below is the JSON debug output from the console. I > started the process at approximately 2013-10-09 01:47:00 GMT: > {"port_dst": 80, "peer_ip_src": "192.168.24.100", "port_src": 1000, "ip_dst": > "10.0.2.254", "ip_src": "10.0.1.1", "ip_proto": "tcp", "stamp_updated": > "2013-10-09 01:50:01", "stamp_inserted": "2013-10-09 01:45:00", "packets": 0, > "bytes": 137} > {"port_dst": 80, "peer_ip_src": "192.168.24.100", "port_src": 1000, "ip_dst": > "10.0.2.254", "ip_src": "10.0.1.1", "ip_proto": "tcp", "stamp_updated": > "2013-10-09 01:55:01", "stamp_inserted": "2013-10-09 01:45:00", "packets": 0, > "bytes": 300} > {"port_dst": 80, "peer_ip_src": "192.168.24.100", "port_src": 1000, "ip_dst": > "10.0.2.254", "ip_src": "10.0.1.1", "ip_proto": "tcp", "stamp_updated": > "2013-10-09 02:00:01", "stamp_inserted": "2013-10-09 01:50:00", "packets": 0, > "bytes": 300} > {"port_dst": 80, "peer_ip_src": "192.168.24.100", "port_src": 1000, "ip_dst": > "10.0.2.254", "ip_src": "10.0.1.1", "ip_proto": "tcp", "stamp_updated": > "2013-10-09 02:05:01", "stamp_inserted": "2013-10-09 01:55:00", "packets": 0, > "bytes": 300} > > Complete config is as follows: > debug: true > > daemonize: false > nfacctd_port: 9996 > > nfacctd_disable_checks: true > nfacctd_time_new: true > > ! AMQP connection details > amqp_host: 127.0.0.1 > amqp_user: guest > amqp_passwd: guest > amqp_exchange: pmacct.direct > amqp_exchange_type: direct > > plugins: amqp[5m_ipip] > > ! 5 minutely IP to IP > aggregate[5m_ipip]: src_host, dst_host, src_port, dst_port, proto, peer_src_ip > amqp_routing_key[5m_ipip]: 5m_ipip > amqp_history[5m_ipip]: 5m > amqp_time_roundoff[5m_ipip]: m > amqp_refresh_time[5m_ipip]: 300 > > Thanks, > Nathan. > > -----Original Message----- > From: pmacct-discussion [mailto:[email protected]] On > Behalf Of Paolo Lucente > Sent: Wednesday, 9 October 2013 6:13 a.m. > To: [email protected] > Subject: Re: [pmacct-discussion] amqp_plugin and the stamp_inserted timestamp > > Hi Nathan, > > I did try to reproduce your issue but with no luck. Can you say what > capturing method are you using (libpcap, NetFlow, sFlow, or ..?). Is it that > all data is mistakenly placed into the previous timeslot or it is possible > you might be effectively receiving only bits of data belonging to the > previous timeslot and bulk of the data belongs to the current timeslot? I can > imagine this can happen a lot with NetFlow if nfacctd_time_new is set to > false, which is default, or with libpcap and/or sFlow in case buffering is > enabled. Let me know. > > Cheers, > Paolo > > On Fri, Oct 04, 2013 at 12:45:49AM +0000, Nathan Kennedy wrote: > > Hi all, > > > > I've managed to compile with rabbitmq enabled, and publish data to an > > exchange - works great! > > > > I have noticed some strange behaviour with the stamp_inserted value. > > When amqp_history is set, the stamp_inserted value is being set to the > > previous slot rather than the current slot. > > For example, stamp_updated may be 2013-10-04 00:37:11 but stamp_inserted is > > 2013-10-04 00:30:00. > > > > My config for this aggregation is as follows: > > > > ! 5 minutely IP to IP > > aggregate[5m_ipip]: src_host, dst_host > > amqp_routing_key[5m_ipip]: 5m_ipip > > amqp_history[5m_ipip]: 5m > > amqp_time_roundoff[5m_ipip]: m > > amqp_refresh_time[5m_ipip]: 10 > > > > I managed to get the expected results by switching lines 194 and 195 of > > amqp_plugin.c, but would like to confirm that this is the correct fix and > > that it won't affect anything else? > > > > Thanks again! > > Nathan. > > > _______________________________________________ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
