Re: [pmacct-discussion] pmacct and pnda.io integration

2018-03-27 Thread Paolo Lucente
e > pmacct directly with pnda.io so we can increase the overall throughput of > the system and maximize the efficiency. > > If there's any interest in the details, I can share some of the > documentation we are working on right now. > > thank you > > --Jaim

Re: [pmacct-discussion] Dynamic SQL data

2018-03-26 Thread Paolo Lucente
Hi Andrey, 'nfacctd_account_options: true' is likely causing this. You should split flow records from options - since you would need different schemas for each. See the 'Va. NetFlow daemon & accounting NetFlow v9/IPFIX options' chapter in https://github.com/pmacct/pmacct/blob/master/QUICKSTART

Re: [pmacct-discussion] pmacct and pnda.io integration

2018-03-24 Thread Paolo Lucente
t;, \"timestamp_end\": \"2018-03-16 22:17:29.856000\", > \"timestamp_arrival\": \"2018-03-16 22:18:30.893573\", \"stamp_inserted\": > \"2018-03-16 22:17:00\", \"stamp_updated\": \"2018-03-16 22:19:01\", > \"

Re: [pmacct-discussion] Dynamic SQL data

2018-03-24 Thread Paolo Lucente
Hi Andrey, So there are some rows that contain empty ip addresses and this is causing the issue. I don't have a full vision of your config so i can not guess what could be the cause - please send it over. It could also be not a config issue but rather some flow spit by the exporting IPFIX

Re: [pmacct-discussion] Dynamic SQL data

2018-03-20 Thread Paolo Lucente
Hi Andrey, Yes what i meant is: any chance you can try to understand what precisely is the problematic value? If it's a specific one or there is a set of values that give problems? Having this info would put us in pole position for troubleshooting the issue. Since the field type is 'inet' i'm

Re: [pmacct-discussion] Dynamic SQL data

2018-03-18 Thread Paolo Lucente
Hi Andrey, Data is purged to the database each sql_refresh_time, by default 60 secs so this is expected behaviour. You are free to redefine this to a different value of course. On the error from your other email: any chance you can post what is the problematic column (since you said it's always

Re: [pmacct-discussion] Avro support issues

2018-03-18 Thread Paolo Lucente
Hi Jaime, Is it possible you are launching a different nfacctd executable (since you are not using the path), maybe in /usr/local/sbin? The CL switches you pass to the configure script do not match with those compiled-in the executable you launch, ie.: INFO ( default/core ): NetFlow Accounting

Re: [pmacct-discussion] Dynamic SQL data

2018-03-16 Thread Paolo Lucente
Hi Andrey, Inline: On Fri, Mar 16, 2018 at 10:43:45AM +0200, Andrey Koblyuk wrote: > > Except for the standard fields IPFIX in the flow there is nothing. template > from router in attach. > How can I make collector on the information from the template when creating a > table? > I do not

Re: [pmacct-discussion] Dynamic SQL data

2018-03-15 Thread Paolo Lucente
Hi Andrey, To your questions: 1) 01011970 corresponds to a UNIX timestamp of zero. The easiest way to mitigate it is to use 'nfacctd_time_new: true' so to consider the time of arrival at the collector (that is, nfacctd) as timestamp for bucketing instead of the flow start time;

Re: [pmacct-discussion] losing environment variables when setting sql_startup_delay

2018-03-09 Thread Paolo Lucente
Hi Johannes, Do you see that behaviour - the reduced amount of environment variables being set - in coincidence with empty purge events? That is zero entries pushed to the database? If yes: that actually was intended behaviour and it still kind of makes sense to me; maybe i would refine it by

Re: [pmacct-discussion] Wait for BGP peering and massiv ipfix data

2018-03-06 Thread Paolo Lucente
Hi Andrey, Inline: On Mon, Mar 05, 2018 at 08:51:34AM +0200, Andrey Koblyuk wrote: > > I would like to create a table once a week with a new name (for > example,acct_%w%Y), which would store non-historical data from > aggregate[storage] : >

Re: [pmacct-discussion] pmacct + ELK made easy?

2018-03-04 Thread Paolo Lucente
> > From: "Mike Hammett" <pmacct-discuss...@ics-il.net> > To: "Paolo Lucente" <pa...@pmacct.net>, pmacct-discussion@pmacct.net > Sent: Saturday, March 3, 2018 4:34:15 PM > Subject: Re: [pmacct-discussion] pmacct + ELK made easy? > >

[pmacct-discussion] MongoDB plugin

2018-03-03 Thread Paolo Lucente
Dearests, I'm trying to sense how much interest there is (still) around a MongoDB plugin since the current plans are to phase it out with the 1.7 train this year. If you have a GitHub account, can you please +1 the following issue to show interest? https://github.com/pmacct/pmacct/issues/187

Re: [pmacct-discussion] pmacct + ELK made easy?

2018-03-03 Thread Paolo Lucente
. I am currently working on a post that describes how I am using > pmacct to process about 100Billion records a day and storing it for > visualization with superset. > > On Sat, Mar 3, 2018 at 11:15 AM Paolo Lucente <pa...@pmacct.net> wrote: > > > > > Anthony is corr

Re: [pmacct-discussion] Export Netflow with BGP?

2018-03-03 Thread Paolo Lucente
Hi Mike, If you are collecting with nfacctd, you can enrich NetFlow data once collecting (not replicating); then you can export to 3rd party elements via files (print plugin) or Kafka/RabbitMQ. Enrichment of BGP data at the replicator is not supported and i doubt it will be in future since it is

Re: [pmacct-discussion] Wait for BGP peering and massiv ipfix data

2018-03-03 Thread Paolo Lucente
Hi Andrey, Inline: On Fri, Mar 02, 2018 at 02:06:30PM +0200, Andrey Koblyuk wrote: > 1) sql_startup_delay does not work for me . I would like to postpone the > first data processing/cache purging before BGP peering is up. otherwise the > table contains data without information from BGP

Re: [pmacct-discussion] Ability to filter by as_src, net_src in tee plugin

2018-03-03 Thread Paolo Lucente
Hi Stanislaw, Such a feature exists for sFlow/sfacctd in tee mode but not (yet) for NetFlow/IPFIX/nfacctd. There are definitely plans to introduce it already for some time (complexities of the porting include template management, options, etc.), it should happen later in the year with 1.7.2.

Re: [pmacct-discussion] Juniper IPFIX (as_src)

2018-03-03 Thread Paolo Lucente
Hi Andrey, Nice solution using bgp_stdcomm_pattern_to_asn to fit the bill, thanks for your feedback. Paolo On Thu, Mar 01, 2018 at 02:21:49PM +0200, Andrey Koblyuk wrote: > Hi, Paolo! > > Thanks for your reply! > > Unfortunately, the configuration you proposed is only partially suitable. >

Re: [pmacct-discussion] Juniper IPFIX (as_src)

2018-03-01 Thread Paolo Lucente
Hi Andrey, That is because you are establishing an iBGP session. You have two possible alternatives: 1) establish an eBGP session by specifying an ASN different than your own via bgp_daemon_as or 2) compose a networks_file with your own prefixes where you specify which ASN to assign them to

Re: [pmacct-discussion] print plugin

2018-02-14 Thread Paolo Lucente
Hi Fabien, It smells like a crash. Please follow: https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2031-L2060 We can then follow-up 1:1 with any output you may be able to gather. Cheers, Paolo On Tue, Feb 13, 2018 at 10:21:54PM +0100, Fabien VINCENT wrote: > Hi all, > > Trying to

Re: [pmacct-discussion] IP Version Field Aggregate

2018-02-11 Thread Paolo Lucente
Hi Tim, Yes, you are looking for the 'etype' primitive which will be populated with 800 (v4) or 86dd (v6). Paolo On Mon, Feb 12, 2018 at 04:58:46PM +1100, Tim Raphael wrote: > Hi All, > > I can’t seem to find anywhere in the docs the pmacct aggregate keyword that > would aggregate traffic on

Re: [pmacct-discussion] nfacctd problems when using pgsql plugin with as_path and sql_use_copy

2018-02-02 Thread Paolo Lucente
> a écrit : > > > > That did it! My apologies for not seeing the sql_delimiter directive > > while reading through the documentation. I will try to rtfm better in > > the future. > > > > Pat > > > > On 12/03/2014 11:21 PM, Paolo Lucente wrote:

Re: [pmacct-discussion] IXP BGP and sFlow Question

2018-02-02 Thread Paolo Lucente
Hi Tim, Thanks for your email. Yes, you can map a BGP speaker to a flow exporter via bgp_agent_map; since you speak of route servers, that is plural, would they all return an omogeneous view in term of routes and more of them are in place for redundancy? Or routes would be different at each of

Re: [pmacct-discussion] netflow packet size

2018-01-18 Thread Paolo Lucente
Hi Steve, This is because the default maximum size of a NetFlow v9/IPFIX packet is set to 512 bytes - in order to be reasonably safe wrt MTU and not enter PMTU stuff. Currently the default value cannot be changed but adding a config option to do so is very easy. You can check what would happen

Re: [pmacct-discussion] How to differentiate Kafka "update" messages -- nfacctd, historic accounting, no nfacctd_time_new

2018-01-16 Thread Paolo Lucente
Hi Georgios, Thanks for the interesting email. Inline: On Fri, Jan 12, 2018 at 07:29:32PM +0100, Georgios Kaklamanos wrote: > > I'm using nfacctd, with Kafka, and the data are written in InfluxDB, and > I'd like some help to clarify an issue that I encountered. > > As far as I've understood

Re: [pmacct-discussion] Updating Timestamps without historical accounting

2017-11-22 Thread Paolo Lucente
Hi Mehul, Can you please elaborate more on what you are trying to accomplish? An example would be fine. Paolo On Wed, Nov 22, 2017 at 09:03:35AM +, Mehul Prajapati wrote: > Hi, > > I want to do accounting in which timestamp is updated at sql_refresh > interval. (No historical accounting)

Re: [pmacct-discussion] nDPI with nfacctd

2017-11-22 Thread Paolo Lucente
Hi Grimur, You may be mixing two unrelated things, nDPI and NetFlow. nDPI applies to actual traffic (libpcap, NFLOG); typical NetFlow exports do report only some elements of the packet headers (further summarised in flows) in its records so a DPI tecnique can't be applied to it; Cisco provides

Re: [pmacct-discussion] sfacctd, mysql

2017-11-21 Thread Paolo Lucente
Hi Nick, I guess you are executing sfacctd -f -P mysql .. and that makes it complain with that cryptic message (action item on me make it more readable) since it already instantiated the plugin. Just removing the -P mysql will make all go smoothly. I also checked i achieve the same behaviour

Re: [pmacct-discussion] pmacct performance

2017-11-21 Thread Paolo Lucente
ime kafka_history runs no matter what > interval you put it on it will crash pmacct and restart the service. > > On Sat, Nov 18, 2017 at 9:27 AM, Anthony Caiafa <2600...@gmail.com> wrote: > > Sounds good. I’ll be sending out some data to you. > > > > On Sat, Nov 1

Re: [pmacct-discussion] Traffic accounting of BGP Subnets and Hosts

2017-11-21 Thread Paolo Lucente
Hi Mehul, I think an aggregation method like the following is a good start for your task: pmacctd_net: bgp aggregate: src_host, src_net, dst_host, dst_net This assumes you are using pmacctd, that is libpcap, for traffic accounting. It will work from 1.6.0 onwards (as before such release

Re: [pmacct-discussion] pmacct performance

2017-11-18 Thread Paolo Lucente
tor > is the best way forward. > > On Fri, Nov 17, 2017 at 9:47 AM, Paolo Lucente <pa...@pmacct.net> wrote: > > > > Hi Anthony, > > > > I map the word 'message' to 'flow' and not to NetFlow packet, please > > correct me if this assumption is wrong. 55m

Re: [pmacct-discussion] pmacct performance

2017-11-17 Thread Paolo Lucente
Hi Anthony, I map the word 'message' to 'flow' and not to NetFlow packet, please correct me if this assumption is wrong. 55m flows/min makes it roughly 1m flows/sec. I would not recommend stretching a single nfacctd daemon beyond beyond 200K flows/sec and the beauty of NetFlow, being UDP, is

Re: [pmacct-discussion] Apply label to IPs, based on IP range (not subnets).

2017-11-17 Thread Paolo Lucente
> Best, > George > > On 11/11/2017 01:48 PM, Paolo Lucente wrote: > > > > Hi Georgios, > > > > Very cool, thanks for sharing this. I think there is also good material > > for me for extra documentation here. > > > > Paolo > > > &

Re: [pmacct-discussion] Apply label to IPs, based on IP range (not subnets).

2017-11-11 Thread Paolo Lucente
of the ranges we have, do not fit into subnets. > > > > For example: > > > > labelA: 192.168.0.1 - 192.168.0.100 > > labelB: 192.168.0.101 - 192.168.0.200 > > > > That is why I was trying to play around with the less than / greater > > than operators, combined

Re: [pmacct-discussion] Apply label to IPs, based on IP range (not subnets).

2017-11-10 Thread Paolo Lucente
Hi Georgios, The 'filter' keyword in pre_tag_map accepts a libpcap/tcpdump filter syntax - what you would find working as a filter in tcpdump, should work here too. To express IP ranges, you should use IP subnets, for example: set_label=labelAfilter='net 192.168.0.0/17' set_label=labelB

Re: [pmacct-discussion] Monitor IP fragmentation

2017-11-09 Thread Paolo Lucente
Hi Hidde, Yes, there is plenty of defragmentation code and you are right that there is no 'external visibility' into it. I'm curious what you'd have in mind to give such visibility, a bool like fragmented traffic yes/no of some sort? Paolo On Thu, Nov 09, 2017 at 04:26:37PM +0100, Hidde van

Re: [pmacct-discussion] Empty bgp attributes ( src/dst_as, as_path, med, etc)

2017-11-09 Thread Paolo Lucente
t; bgp_daemon_max_peers: 20 > bgp_table_dump_file: /tmp/bgp-$peer_src_ip-%H%M.log > bgp_table_dump_refresh_time: 36000 > bgp_follow_default: 5 > bgp_agent_map: bgp_agents > nfacctd_as: bgp > pmacctd_as: bgp > nfacctd_net: bgp > pmacctd_net: bgp > bgp_aspath_radius: 3 > b

Re: [pmacct-discussion] oom-killer with pmacct 1.7.0

2017-11-08 Thread Paolo Lucente
ne for IPv6 traffic ... which could > probably be changed to very small unfortunately) > > I could share config privately if needed/wanted. > > > Regards, > > Wouter > > -Oorspronkelijk bericht- > Van: pmacct-discussion [mailto:pmacct-discussion-boun...@

Re: [pmacct-discussion] Empty bgp attributes ( src/dst_as, as_path, med, etc)

2017-11-08 Thread Paolo Lucente
Hi Tamas, >From your outputs definitely looks everything is in order. I wonder though, since you use the IMT plugin, if those entries are created before the BGP session is successfully established. Any chance, keeping things simple and for the sake of a test, you can try the same with the print

Re: [pmacct-discussion] pmacct and AS4 BGP capability issue/bug

2017-11-08 Thread Paolo Lucente
Dears, To wrap this thread up: https://github.com/pmacct/pmacct/issues/167 https://github.com/pmacct/pmacct/commit/069f8b815e224b8fad7fa8cb8cac5c89fd3ed8df Thanks very much Radu for your support. Cheers, Paolo On Tue, Nov 07, 2017 at 01:05:52AM +, Paolo Lucente wrote: > > H

Re: [pmacct-discussion] pmacct and AS4 BGP capability issue/bug

2017-11-06 Thread Paolo Lucente
Hi Radu, Thanks for reporting this. Any chance you could send me by unicast email a pcap trace with the two BGP OPEN? Reviewing the code there is nothing that would make this happen and hence the deleted part could reveal more info about the condition. Thanks in advance, Paolo On Mon, Nov 06,

Re: [pmacct-discussion] oom-killer with pmacct 1.7.0

2017-11-06 Thread Paolo Lucente
Hi Wouter, If i understand correctly you are using 1.7.0 with ZMQ on all of your boxes but only 2 of them present an issue, while the others run fine, is that correct? Please, yes, either recompile without ZeroMQ or revert to the home-grown circular buffer for those two boxes - for the sake of

Re: [pmacct-discussion] pmacct 1.7.0, can't connect to mysql

2017-11-03 Thread Paolo Lucente
Hi Nick, Inline: On Thu, Nov 02, 2017 at 05:10:31PM -0500, Nicholas Geovanis wrote: > > INSERT INTO `acct_v8` (vlan, ip_src, ip_dst, as_src, as_dst, port_src, > port_dst, tcp_flags, tos, ip_proto, agent_id, class_id, mac_src, mac_dst, > packets, bytes, flows) VALUES (0,

Re: [pmacct-discussion] pmacct 1.7.0, can't connect to mysql

2017-11-02 Thread Paolo Lucente
Hi Nicholas, For the user permission issue i would recommend trying one of the following things: 1) adding a 'sql_host: 127.0.0.1' to your config or 2) configuring your user as 'pmacct'@'%'. If i recall correctly, when connecting locally you could opt for the loopback interface or the UNIX

Re: [pmacct-discussion] nfacctd and IPFIX custom fields

2017-11-02 Thread Paolo Lucente
Hi Eddi, Thanks for the kind words and the interesting email. Any chance you can send me a brief sample of such data via unicast email so to be able to reproduce things at my end? Thanks, Paolo On Wed, Nov 01, 2017 at 06:24:26PM +0200, edd! wrote: > Hi, > > After wasting a couple of months on

Re: [pmacct-discussion] error compiling v1.7.0 with rabbitmq

2017-10-28 Thread Paolo Lucente
START that I didn't fully read. Lesson > learned: RTFD. > > Thanks for your help. > > > > On Fri, Oct 27, 2017 at 6:50 AM, Paolo Lucente <pa...@pmacct.net> wrote: > > > > > Hi Hrothgar, > > > > I wonder whether it is possible you did b

Re: [pmacct-discussion] error compiling v1.7.0 with rabbitmq

2017-10-27 Thread Paolo Lucente
Hi Hrothgar, I wonder whether it is possible you did build only the shared object version of the rabbitmq-c library and the location where it's installed, /usr/local/rabbitmq-c-0.8.0/lib64 , is not in the path for ldconfig / ld.so.conf and this is creating issues. I can confirm that I'm

Re: [pmacct-discussion] RE on pretag.map

2017-10-22 Thread Paolo Lucente
Hi Fabien, Unfortunately not, no support for regex. Paolo On Fri, Oct 20, 2017 at 04:58:57PM +0200, Fabien VINCENT wrote: > Hi All, > > Is it possible to have regex on pretag.map ? > > i.e. I want to match some fwdstatus and tag them in the same manner. > > Today I do : > > set_tag2=1

Re: [pmacct-discussion] Get TCP SYN totals per IP

2017-10-22 Thread Paolo Lucente
Hi Vaggelis, Which capturing method are you using, libpcap or NetFlow/IPFIX/sFlow? And also are you looking for a dedicated solution for this or this is going to be yet another activity for an existing pmacct deployment? Taking the simplest scenario: you using libpcap, so pmacctd, and want to

[pmacct-discussion] pmacct 1.7.0 released !

2017-10-21 Thread Paolo Lucente
VERSION. 1.7.0 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect infrastructure data

Re: [pmacct-discussion] Dual Stack

2017-10-13 Thread Paolo Lucente
Hi Paolo, > > > > Happy to give you more work ;) It's not "needed", but it could be great I > > can do it. > > > > I thought about AF in another one, will try to implement / configure it on > > ASR and give the trick here for the community/list ;) > &

Re: [pmacct-discussion] wrong value in Start Time and End Time

2017-10-11 Thread Paolo Lucente
Hi Mohsen, Please make a capture of your flows with the tool of your choice (tcpdump, wireshark, tshark, etc.) and send it over to me via private email. It will help me assessing better where the problem could be. Cheers, Paolo On Wed, Oct 11, 2017 at 07:42:55PM +0330, mohsen Abbaspour wrote:

Re: [pmacct-discussion] A tool to build bgp_peer_src_as_map: mac-to-peer

2017-10-02 Thread Paolo Lucente
Ciao PC, First of all, let me thank you very much for this second valuable contribution to the pmacct ecosystem (first being the successful pmacct-to-elasticsearch project (*)). I've included mac-to-peer as part of pmacct documentation:

Re: [pmacct-discussion] Wrong timestamp for netflow streams in sql history

2017-09-26 Thread Paolo Lucente
Hi Eythor, Your config looks simple and correct. I would have told you to check time on the box where you are running pmacct but you confirmed all is good there so i'm not sure. I'm willing to take a look myself; if that is an option please follow-up by unicast email. As a workaround i can

Re: [pmacct-discussion] fswstatus

2017-09-24 Thread Paolo Lucente
Hi Fabien, What version are you running? You can confirm this with a 'nfacctd -V'; the feature was added in 1.7.0 (that is, master code on GitHub). I can also confirm you, on your original question, that an atoi() is performed on the input value - so you should express values in decimal. Paolo

Re: [pmacct-discussion] nfacctd and amqp_multi_values

2017-09-20 Thread Paolo Lucente
lt (possible) to implement? > I didn't look into the code yet, I didn't program in C since high school > and my skills are quite rusty. > > Cheers, > > Yann > > On Fri, Sep 15, 2017 at 6:38 PM Paolo Lucente <pa...@pmacct.net> wrote: > > > > > Hi Yann, >

Re: [pmacct-discussion] nfacctd and amqp_multi_values

2017-09-15 Thread Paolo Lucente
Hi Yann, I confirm you can't do that with AMQP as the only knob pmacct gives you is the size-based amqp_multi_values. Although not part of your question, with Kafka and you may choose not to leverage the pmacct knob, kafka_multi_values, and use instead batch.num.messages (ie. amount of messages

Re: [pmacct-discussion] null padded packet issue

2017-09-08 Thread Paolo Lucente
Hi Steffen, Thanks for this report. I'd be indeed interested in seeing a capture of the packets myself in order to see whether there is anything that can be improved on the pmacct side of the things. Should it be possible for you to send a brief pcap trace over, please get in touch via unicast

Re: [pmacct-discussion] MySQL plugin and dynamic table names

2017-09-06 Thread Paolo Lucente
Hi Mathias, Inline: On Tue, Sep 05, 2017 at 03:05:55PM +0200, Mathias Gumz wrote: > Exactly that is what I switched to and it does its job. :) Great :) > Last question so far: Why am I not seeing data in the database when > using "sql_history: 1" (or "10")? I have both "sql_history" and >

Re: [pmacct-discussion] Best way to output ip addresses as integers

2017-09-04 Thread Paolo Lucente
Hi Yann, Depending on the plugin you are using there could already be a built-in or not. For the SQL plugins you have it: 'sql_num_hosts: true'; for all the others you don't but actually adapting a feature like this would be relatively easy/quick. If you use SQL plugins, specifically MySQL,

Re: [pmacct-discussion] [RFC] Patches to allow capturing specific PF_PACKET protocols

2017-08-30 Thread Paolo Lucente
I missing something? :) > > > Cheers, > Lennert > > > On Wed, Aug 30, 2017 at 02:14:48PM +, Paolo Lucente wrote: > > > Hi Lennert, > > > > That would indeed work too, yes. Go for it :) > > > > Paolo > > > > On Wed

Re: [pmacct-discussion] [RFC] Patches to allow capturing specific PF_PACKET protocols

2017-08-30 Thread Paolo Lucente
rs, > Lennert > > > On Wed, Aug 30, 2017 at 01:58:48PM +, Paolo Lucente wrote: > > > Hi Lennert, > > > > Fantastic, please proceed. I guess the patch should be wrapped around > > some version checking of libpcap a-la if greater or equal than c

Re: [pmacct-discussion] [RFC] Patches to allow capturing specific PF_PACKET protocols

2017-08-30 Thread Paolo Lucente
Hi Lennert, Fantastic, please proceed. I guess the patch should be wrapped around some version checking of libpcap a-la if greater or equal than current release in master then compile the code. I did jus review configure.ac and libpcap version is not captured plus i seem to read libpcap folks

Re: [pmacct-discussion] MySQL plugin and dynamic table names

2017-08-29 Thread Paolo Lucente
Hi Mathias, Inline: On Tue, Aug 29, 2017 at 05:00:34PM +0200, Mathias Gumz wrote: > > Currently I have set the "sql_history" and "sql_refresh_time" to 60s. I > > wonder, > > how the algorithm works. "sql_refresh_time" seems to scan the cache and, if > > needed, writes/updates an entry in the

Re: [pmacct-discussion] MySQL plugin and dynamic table names

2017-08-28 Thread Paolo Lucente
Hi Mathias, Can you please post your config? Gut feeling says you may be missing the sql_history directive (essentially indicate what is the time-binning period). Paolo On Mon, Aug 28, 2017 at 03:36:02PM +0200, Mathias Gumz wrote: > Hi, > > we are using nfacctd to collect NAT events (event

Re: [pmacct-discussion] overriding src_as

2017-08-26 Thread Paolo Lucente
Hi Daryl, You should be looking for: nfacctd_net: longest nfacctd_as: longest Can you give this a try and see if it works as expected? Longest match of file entries vs default route in BGP should achieve your desired "override" behaviour. Paolo On Fri, Aug 25, 2017 at 04:27:52PM +0930,

Re: [pmacct-discussion] Segfault 1.6.1 and current master

2017-08-23 Thread Paolo Lucente
Hi Steve, Spot on. Just committed the fix to GitHub code with credits to you: https://github.com/pmacct/pmacct/commit/7afe854b627a31de764d1567038181e2eec16640 Thanks, Paolo On Wed, Aug 23, 2017 at 08:03:08AM -0400, Stephen Clark wrote: > Hi Paolo, > > After doing some more investigation this

Re: [pmacct-discussion] Add pmacct hostname to SQL schema

2017-08-22 Thread Paolo Lucente
Hi Franz, Are you interested in the pmacct server hostname or the IP address of the NetFlow/IPFIX/sFlow exporter? Would peer_src_ip, the IP address of the flow exporter do it? Or you are collecting via libpcap or NFLOG? Paolo On Mon, Aug 21, 2017 at 05:23:34PM +0200, fboehm wrote: > Hi, > >

Re: [pmacct-discussion] pre_tag_map not being applied?

2017-08-17 Thread Paolo Lucente
Hi Brian, Thanks for getting in touch. Is the flow sample pointed in your email an example of one that does not get properly tagged? Besides, we should try to make the issue reproducible: if you could make a pcap trace containing some flow samples that should be tagged and they are not, and

[pmacct-discussion] nDPI integration

2017-07-23 Thread Paolo Lucente
Dearests, A first round of coding to integrate packet classification via nDPI in pmacct is now available on the GitHub code for all those souls that would like to contribute helping out testing this. I recall a few of you that have been waiting this: please reach out to me if i don't reach out

Re: [pmacct-discussion] Dynamic Labelling from Exporters IP Address.

2017-07-21 Thread Paolo Lucente
Hi Yann, Agree with you, Harry seems to be looking precisely for peer_src_ip. Paolo On Fri, Jul 21, 2017 at 04:18:44PM +0200, Yann Belin wrote: > Hi Harry, > > Unless I am mistaken, the IP of the flow exporter can be obtained by > adding 'peer_src_ip' in your aggregate list. > > Kind

Re: [pmacct-discussion] sfacctd_as and ASN 0

2017-07-21 Thread Paolo Lucente
+, Mike Jager wrote: > Hi Paolo, > > On 18 Jul 2017, at 2:28, Paolo Lucente wrote: > > > The version the post refers to is very old and, yes, the issue was > resolved back then. I would start from scratch investigating what your > issue may be. What version are you

Re: [pmacct-discussion] [RFC] Patches to allow capturing specific PF_PACKET protocols

2017-07-02 Thread Paolo Lucente
Hi Lennert, I'm familiar with the context and the patch, on the pmacct side of the things, looks sane. The only thing i noticed is that a default value for config.pcap_protocol is never imposed. I'd be curious what is your strategy to move this forward, especially on the libpcap side. If it

Re: [pmacct-discussion] Flow filter

2017-06-24 Thread Paolo Lucente
Hi Alex, What you are looking for is already supported for sFlow tee but not (yet) for NetFlow/IPFIX tee. Would you mind reaching out privately on this? I'd be more than willing to make this happen and in order to do so i would like to review your use-case, possibly ask for a testbed (so to test

Re: [pmacct-discussion] Strange behavior with protocol field on nfacctd

2017-06-20 Thread Paolo Lucente
A quick note to say we did troubleshoot the issue further with Yann and, as a result of that, we have this commit in the master code: https://github.com/pmacct/pmacct/commit/9fa8779344854ab876e3bcc6ff6f25c51c6df226 Cheers, Paolo On Wed, Jun 14, 2017 at 05:11:42PM +0200, Yann Belin wrote: >

Re: [pmacct-discussion] Mikrotik Netflow: strange difference between nfacctd and nfcapd

2017-06-17 Thread Paolo Lucente
Hi Michael, I would suggest to comment out buffering, especially if the volume of NetFlow packets is not sustained (plugin_pipe_size, plugin_buffer_size); that should return a more accurate comparison. Should differences still persist, the course of action would be a unicast email to me with the

Re: [pmacct-discussion] Timestamps and sample rates with L2 netflow v9

2017-06-17 Thread Paolo Lucente
Hi Stephen, You can make nfacctd use as timestamp the time the flow is received by setting 'nfacctd_time_new: true'. I'm not entirely clear on the second question though; in an ideal world, where you have a different sampling rate per port and your device does report that correctly via NetFlow

Re: [pmacct-discussion] Configuring pmacct/nfacctd as a Proxy/Ingest node & Visualize Data

2017-06-14 Thread Paolo Lucente
Hi Sami, Let me take inline the only question that went unanswered by previous replies: On Sat, Jun 10, 2017 at 08:37:24PM -0400, Sami wrote: > > What i want to do now is to log NetFlow traffic on files (.csv/.log ..), do > you have any sample configuration for this? > You can refer to

Re: [pmacct-discussion] Matching data to options with nfacctd

2017-06-08 Thread Paolo Lucente
| 8 | 4 > | > | interface input snmp|10 || 12 | 4 > | > | interface output snmp | 14 | | 16 | 4 > | > (...) > > -

Re: [pmacct-discussion] Filtering on custom primitive

2017-05-11 Thread Paolo Lucente
Hi Emil, Thanks for your kind words. I precisely see what you are after with this but unfortunately that is not supported; improving the custom primitives infrastructure is something i'd intend to do soon, including precisely filtering/tagging over custom primitives. Paolo On Tue, May 02,

[pmacct-discussion] pmacct 1.6.2 released !

2017-04-22 Thread Paolo Lucente
VERSION. 1.6.2 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect infrastructure data

Re: [pmacct-discussion] question about sql_refresh

2017-04-22 Thread Paolo Lucente
Hi Farshid, The question is generic. What daemon are you using? Since you mentioned flows, the following holds if you are using NetFlow/IPFIX and hence nfacctd as daemon. Also the assumption is sql_dont_try_update is not set to true (defaults to false). Granted nothing gets lost. What you will

Re: [pmacct-discussion] Sfacctd per-ASN + BGP traffic accounting

2017-04-06 Thread Paolo Lucente
to sfacctd is > incomplete but I announce all the prefixes my routers have got: they > route traffic to uplinks according to it.. I guess pmacct's BGP > daemon ignores or "loses" some announces. The routers sending BGP > feed are Juniper MX480 and MX80. > > I'd b

Re: [pmacct-discussion] sfacctd and tagging with filters

2017-04-06 Thread Paolo Lucente
Hi Aurelien, Is it possible your traffic is VLAN-tagged and/or MPLS-labelled? That may explain why tagging is not working, pcap filters are sensible to that. See for example here: https://www.mail-archive.com/pmacct-discussion@pmacct.net/msg02784.html Paolo On Tue, Apr 04, 2017 at 04:32:13PM

Re: [pmacct-discussion] nfacctd: ASR9k bulk-port-allocation (BPA) for CGN

2017-04-01 Thread Paolo Lucente
Hi Kenneth, You have the aggregate_primitives infrastructure to define new and custom NetFlow/IPFIX fields. Here you can also find some examples: https://github.com/pmacct/pmacct/blob/master/examples/primitives.lst.example Don't hesitate to get in touch privately with a sample of your data in

Re: [pmacct-discussion] nfacctd and pfring

2017-03-22 Thread Paolo Lucente
Hi Steve, Optionally only by pmacctd. Reason being nfacctd is a daemon listening to a port rather than sniffing traffic. Paolo On Wed, Mar 22, 2017 at 08:45:55AM -0400, Stephen Clark wrote: > Hi Paolo, > > Does nfacctd make use of pfring or is it only used by pmacctd? > > Thanks, > Steve >

Re: [pmacct-discussion] BGP ERROR port=179/tcp failed

2017-03-19 Thread Paolo Lucente
Ciao Mario, Error code #99 corresponds to EADDRNOTAVAIL (or address not available); the most intuitive reason for that would be that 10.39.11.34 is actually not configured on the box (and that would explain why if you change it to being the localhost, 127.0.0.1, it all works). That not being the

Re: [pmacct-discussion] sql_dont_try_update causes key collisions?

2017-03-07 Thread Paolo Lucente
> > > > > Can you tell me (or point me to the documentation) regarding how to read > > > the 'purging' log line? > > > > > > e.g. > > > > > > Mar 5 13:47:04 server nfacctd[28824]: INFO ( ip_dst/sqlite3 ): *** > > Purging >

Re: [pmacct-discussion] aggregate_filter for non pcap elements

2017-03-07 Thread Paolo Lucente
Hi Timo, 'ip' is to match the IP address of the sender of the NetFlow/IPFIX packet. Paolo On Tue, Mar 07, 2017 at 01:29:11PM +0100, Timo Lindhorst wrote: > Hi Paolo, > > > I would make one exception > > which did not raise from your email: the potential need for IP addresses > > (in the

Re: [pmacct-discussion] sql_dont_try_update causes key collisions?

2017-03-06 Thread Paolo Lucente
regarding how to read > the 'purging' log line? > > e.g. > > Mar 5 13:47:04 server nfacctd[28824]: INFO ( ip_dst/sqlite3 ): *** Purging > cache - END (PID: 28824, QN: 577/284209, ET: 2) *** > > I'm curious what the QN: 577/284209 part means. > > > > On Fri, Mar 3

Re: [pmacct-discussion] Failing to compile amqp

2017-03-06 Thread Paolo Lucente
Hi Ben, Thanks very much for reporting the issue :) It's now fixed: https://github.com/pmacct/pmacct/commit/89da3e2312660f60b9c2dca20c6d60c1156e36d8 Cheers, Paolo On Mon, Mar 06, 2017 at 08:42:27PM +, Ben Wilson wrote: > Hi, > > > I'm trying to build with rabbitmq support, but it's

Re: [pmacct-discussion] sql_dont_try_update causes key collisions?

2017-03-03 Thread Paolo Lucente
+1 on Tristan's feedback. Ed, you can check at this propo also: https://github.com/pmacct/pmacct/wiki/RDBMS:-Customising-the-SQL-database-indexes If commenting out sql_dont_try_update makes things work well then it means the setup is making use of UPDATE queries. Maybe you need a larger

Re: [pmacct-discussion] nfacctd: mysql Plugin -- DB Writer [total]

2017-03-02 Thread Paolo Lucente
Hi Luc, At a glance it looks a case where you are overwhelming the RDBMS 1) you write data to a static table, ie. acct, likely making the table and its index(es) big; 2) you make use of UPDATE queries, which are expensive; whereas you should try to aim at an INSERT-only environment (*); 3) the

Re: [pmacct-discussion] nfacctd - log missed sequence numbers?

2017-02-28 Thread Paolo Lucente
f > syslog: daemon > # fgrep -i expecting /var/log/daemon > # > > That would be my working assumption. > > Thanks, > > Ed > > On Sat, Feb 25, 2017 at 7:19 AM, Paolo Lucente <pa...@pmacct.net> wrote: > > > > > Hi Ed, > > > > T

Re: [pmacct-discussion] nfacctd - log missed sequence numbers?

2017-02-25 Thread Paolo Lucente
Hi Ed, The log message produced is actually very simple: Log([..] expecting flow '%u' but received '%u' collector=%s:%u agent=%s:%u [..]); It's a start for some basic analysis but you can get false positives, for example due to out of order arrival of packets. In recent pmacct releases you

Re: [pmacct-discussion] bgp_daemon wrong next-hop

2017-02-17 Thread Paolo Lucente
Hi Catalin, Any chance we can go unicast and arrange a way for me to troubleshoot this? Like either you point the BGP session towards one of my boxes or you can grant me temporary access to your environment? Happy to support you. Thanks, Paolo On Fri, Feb 17, 2017 at 02:53:25PM +,

Re: [pmacct-discussion] Issues with Kafka/Avro sending schema to Kafka topic

2017-02-17 Thread Paolo Lucente
Hi Aaron, The feature is post 1.6.1. Can you please switch to master code on GitHub? Thanks, Paolo On Thu, Feb 16, 2017 at 10:44:23AM -0800, Aaron Finney wrote: > Hi Paolo/all, > > I've been unable to get nfacctd to send the Avro schema to a Kafka topic - > I receive the following message

Re: [pmacct-discussion] Tee tag mapping for V9/IPFIX

2017-02-09 Thread Paolo Lucente
Hi Chip, Thanks a lot for your feedback on this. It makes sense to port the work done for sFlow on NetFlow v9/IPFIX; i have not planned it yet mainly as i was precisely waiting for gathering interest. Do you think we can continue 1:1 on this thread? I'd be looking for your use-case and, given

Re: [pmacct-discussion] pmacct max active plugins

2017-02-06 Thread Paolo Lucente
Hi Alex, Yes, that is OK. See also the thread here: https://github.com/pmacct/pmacct/issues/63 It would be great to know also your use-case for instantiating more than 32 plugins. Keep me posted if it works. Paolo On Mon, Feb 06, 2017 at 11:53:18AM +0200, Abi Askushi wrote: > HI All, > > I

Re: [pmacct-discussion] nfacctd aggregate_filters not working correctly when defined in the same config file

2017-01-23 Thread Paolo Lucente
to flow collection > in our networks, most likely using Riak as a back-end data store. > > Aaron > > > On Sun, Jan 22, 2017 at 10:16 AM, Paolo Lucente <pa...@pmacct.net> wrote: > > > > > Hi Aaron, > > > > Thanks for the feedback. I'm unfortunately

<    1   2   3   4   5   6   7   8   9   10   >