Re: [pmacct-discussion] pmacctd with nfprobe and pretag.map

2024-04-24 Thread Paolo Lucente
that label. Regards, Bruno On Fri, Apr 19, 2024 at 5:43 PM Paolo Lucente <mailto:pa...@pmacct.net>> wrote: Hi Bruno, Yes you can use labels for that & i have just proofed that working successfully end-to-end (exporter and collector sides) on my local

Re: [pmacct-discussion] pmacctd with nfprobe and pretag.map

2024-04-19 Thread Paolo Lucente
Hi Bruno, Yes you can use labels for that & i have just proofed that working successfully end-to-end (exporter and collector sides) on my local environment. In the nfprobe config: pre_tag_map: /path/to/pretag.map aggregate: label, < all other usual suspects here > Then in pretag.map,

Re: [pmacct-discussion] nfacctd pretag.map with mpls_vpn_rd

2024-03-20 Thread Paolo Lucente
:08 AM, Paolo Lucente (pa...@pmacct.net <mailto:pa...@pmacct.net>) wrote: Hi Andy, Amazing! So probably, after all, we may bash Issue 770 on GitHub -- coolio. Let me recap situation so that you can correct me where i am wrong and fill any gaps. You have flows, these (all or some

Re: [pmacct-discussion] nfacctd pretag.map with mpls_vpn_rd

2024-03-19 Thread Paolo Lucente
again, Andy On March 16, 2024 at 1:06:48 AM, Paolo Lucente (pa...@pmacct.net <mailto:pa...@pmacct.net>) wrote: Hi Andy, Thanks for opening the issue on GitHub and the kind words. Thing is all you want to achieve is supported in pmacct when working in collector mode where the proper insp

Re: [pmacct-discussion] nfacctd pretag.map with mpls_vpn_rd

2024-03-15 Thread Paolo Lucente
if you are able to get the VRF ID matching against IPFIX working so we can tee it from there, that will be fantastic. Thanks again for all your help…and also just in general building an awesome product :) Andy On March 15, 2024 at 2:18:01 AM, Paolo Lucente (pa...@pmacct.net <mailto:pa...@p

Re: [pmacct-discussion] nfacctd pretag.map with mpls_vpn_rd

2024-03-15 Thread Paolo Lucente
Hi Andy, mpls_vpn_rd is supported in pre_tag_map however it is not supported when in tee / replication mode (this is documented). For your specific use-case, since you are interested in matching the VRF ID, which in turn is self-consistent as part of an IPFIX record, this is something that

Re: [pmacct-discussion] pmacct with nfprobe_direction / nfprobe_ifindex and pretag.map

2024-01-14 Thread Paolo Lucente
Hi Klaus, Can you confirm what version of pmacct are you using? A 'pmacctd -V' would do. I would like essentially to confirm that, for the first issue you are hitting, you are running either 1.7.8 or a recent code that includes this patch from Dec 15th:

Re: [pmacct-discussion] Traffic Analysis Tool

2023-12-14 Thread Paolo Lucente
Hi Faizan, I guess the underlying question is whether you can build a prefix-to-prefix traffic matrix using sFlow data (and probably attribute it to ASNs automatically using a BGP feed). This is indeed all possible in concept. A proof of concept would be needed in one of your environment for

Re: [pmacct-discussion] pretag.map not working when running nfacctd in a container

2023-12-12 Thread Paolo Lucente
Hi Rich, I was wondering if you had any log availble from nfacctd; for example, is it possible that the file 'pretag.map' with no paths supplied is not found, causing the issue? Paolo On Thu, Dec 07, 2023 at 11:34:56PM +, Compton, Rich A wrote: > Hi, hoping that someone can help me with

Re: [pmacct-discussion] peer_src_as vs src_as

2023-11-24 Thread Paolo Lucente
Hi Benedikt, Yes, fields are directly populated with what is in the NetFlow packet. Super strange the Cisco is putting the Source AS in PeerSrcAS (confirmed also with tcpdump), maybe a bug? You could probably get around it defining a custom primitive but it would be very dirty. I would make

Re: [pmacct-discussion] pmacct+clickhouse

2023-11-21 Thread Paolo Lucente
Hi Sergey, Googling around i could find a couple of documents around the topic. Like for example: https://github.com/kvitex/pmacct-kafka-clickhouse . Not being a user of Clickhouse myself, i can't say if it's complete and actual but maybe it's a starting point & any issues maybe you can report

Re: [pmacct-discussion] filtering in tee mode

2023-08-10 Thread Paolo Lucente
Hi Evgeniy, For a starter, did you have a look to section XVa of the QUICKSTART guide: https://github.com/pmacct/pmacct/blob/0bd518b6fbee4ba286832f07fbf8debf0c3fa925/QUICKSTART#L2198C10-L2198C10 Examples are based on src_mac, dst_mac ; but you could give a try with src_net , dst_net . This

Re: [pmacct-discussion] Cisco NCS - IPFIX 315 - sampling_rate and outer qtag not detected

2023-07-29 Thread Paolo Lucente
Hi Tiago, Great to read from you, about your issues: 1) can you send me a pcap with a data packet and the templates, both data and sampling option? Being able to replay it will give me a chance to understand what may be wrong. 2) vlan_out refers to the vlan after, say, some re-tagging took

Re: [pmacct-discussion] nfprobe/nfacctd communication over TCP

2023-04-29 Thread Paolo Lucente
Hi Eric, Thanks for getting in touch & let me confirm that there are currently no plans. This said, IPFIX RFC does contemplate TCP and it should not be a biggie to implement. Let me put it on my todo list, unless this is a dev that you may take on your side; if this would be on me, can

Re: [pmacct-discussion] IPFix Bi-Flows & docker ARM

2023-03-01 Thread Paolo Lucente
Hi Dain, Thanks very much for your work on the arm docker image. With regards to your question, i guess you are looking into collecting traffic with pmacctd then exporting bi-flows with the nfprobe plugin; bi-flows are currently only supported by nfacctd on collection. Paolo On 24/2/23

Re: [pmacct-discussion] Does pmacct looking glass suppoorts multipath?

2023-02-06 Thread Paolo Lucente
Hi Alexander, Thanks for reporting this. You are hitting onto a (known) limitation of the LG (server / lookup), in fact in bgp_lg_daemon_ip_lookup() there is this note in the code: https://github.com/pmacct/pmacct/blob/4a70a5b41195afc904d77efa61987bcb80023512/src/bgp/bgp_lookup.c#L843 The

Re: [pmacct-discussion] I need help with pre_tag_map and aggregate_filter

2023-01-23 Thread Paolo Lucente
the correctly labeled flows. The pretag.map files were changing over the days, to really only mark the searched traffic and have a clearer configuration. If you need more accurate data, and a flow sample, I can send you by unicast email. Regards, Federico On 16/1/23 16:27, Paolo Lucente wrote: Hi

Re: [pmacct-discussion] I need help with pre_tag_map and aggregate_filter

2023-01-16 Thread Paolo Lucente
Hi Federico, I see the combo pre_tag_label_filter / pre_tag_label_encode_as_map, can you please temporarily disable the latter (pre_tag_label_encode_as_map) and see if the filtering does work as expected? Should it not, can you also disable the filtering and check what you see? Are labels

[pmacct-discussion] pmacct 1.7.8 released !

2022-12-31 Thread Paolo Lucente
VERSION. 1.7.8 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate RPKI

[pmacct-discussion] pmacct 1.7.8 released !

2022-12-31 Thread Paolo Lucente
VERSION. 1.7.8 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate

Re: [pmacct-discussion] Filter destination IP on lists with tens of thousands of entries?

2022-12-16 Thread Paolo Lucente
Hi Rich, Indexed pre_tag_map could fit the bill (and if going down this route i do recommend to perform a proof-of-concept using latest & greatest code in master since it has been a recent area of growth / improvement). Example of how you could populate the pre_tag_map (centric on IPv4 but

Re: [pmacct-discussion] Ballpark pmacctd performance

2022-12-07 Thread Paolo Lucente
/22 07:02, Paolo Lucente wrote: Hi Chander, I am unable to confirm your figure but i can say that you can give a try to PF_RING-enabled libpcap to see if it brings any advantage to the performance you are currently seeing. Also, as explained here ( https://github.com/pmacct/pmacc

Re: [pmacct-discussion] Ballpark pmacctd performance

2022-12-03 Thread Paolo Lucente
Hi Chander, I am unable to confirm your figure but i can say that you can give a try to PF_RING-enabled libpcap to see if it brings any advantage to the performance you are currently seeing. Also, as explained here (

Re: [pmacct-discussion] peer_ip_src 0.0.0.0 on IPv6 netflow

2022-11-13 Thread Paolo Lucente
Hi Federico, This is indeed very strange since, unless your vendor is trying to specify the IP address of the exporter (and this is somehow failing) as part of the flows, the IP address is taken directly from the operating system socket. The feature of using IE #130 (exporterIPv4Address)

Re: [pmacct-discussion] kafka plugin and number of json messages vs number of netflow record

2022-10-25 Thread Paolo Lucente
Improved link: https://github.com/pmacct/pmacct/blob/1.7.8/QUICKSTART#L3065-#L3071 Paolo On 25/10/22 10:24, Paolo Lucente wrote: Hi Wilfrid, Can you please check whether you are dropping any NetFlow packets: https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L3065-#L3071 . Also

Re: [pmacct-discussion] kafka plugin and number of json messages vs number of netflow record

2022-10-25 Thread Paolo Lucente
not asking to troubleshoot, but I would like again confirmation that we should expect from kafka plugin to translate each flow record matching {router, ifindex) into json and sent to kafka. Thanks again Wilfrid -Original Message- From: Paolo Lucente Sent: Friday, 21 October 2022 15

Re: [pmacct-discussion] kafka plugin and number of json messages vs number of netflow record

2022-10-21 Thread Paolo Lucente
Hi Wilfrid, To say whether some aggregation is taking place or not, you should look at the template of the incoming NetFlow records. You can achieve this with Wireshark / tshark or via pmacct, either running it in debug mode - you will find the templates in the log file - or defining a

Re: [pmacct-discussion] Doubt about custom sql tables

2022-10-03 Thread Paolo Lucente
Hi Federico, Thanks for getting in touch and bringing this up. More than a bug, you are running in an aspect about SQL tables that is poorly documented (i will try to improve that as a follow-up). The only vague mentioning of what you are running into is here:

Re: [pmacct-discussion] icmp6 netflow 9 not including type & code sometimes

2022-10-03 Thread Paolo Lucente
Hi, Best would be for me to be able to reproduce the issue; can you make a brief capture in pcap format (ie. with tcpdump) of some of this icmp6 traffic and send it over via unicast email? If you could even compose two traces, one for the interface that is working, one for the one that is

Re: [pmacct-discussion] nfacct accounts traffic twice

2022-07-08 Thread Paolo Lucente
hael Am 04.07.2022 um 21:29 schrieb Paolo Lucente: Hi Michael, Welcome back! :-) What version of pmacct are you using? I see you daemonize but there is no logfile specified: did you check the log on startup to make sure that the filter in 'aggregate_filter' is being accepted and loaded?

Re: [pmacct-discussion] nfacct accounts traffic twice

2022-07-04 Thread Paolo Lucente
Hi Michael, Welcome back! :-) What version of pmacct are you using? I see you daemonize but there is no logfile specified: did you check the log on startup to make sure that the filter in 'aggregate_filter' is being accepted and loaded? Your understanding of how 'aggregate_filter' should

Re: [pmacct-discussion] [docker-doctors] pmacctd in docker

2022-05-10 Thread Paolo Lucente
effectively being required, pcap_ifindex effectively being ignored) . So I'm either making a mistake, e.g. in my config files, misunderstanding the documentation or I'm encountering a bug - which I find difficult to believe given how trivial my setup is. Any Suggestions ? Regards &am

Re: [pmacct-discussion] pmacct.net

2022-05-08 Thread Paolo Lucente
A quick note to thank you Karl for your always good inputs; let me read through and see what actions i can take. Paolo On 4/5/22 13:32, Karl O. Pinc wrote: Hi Paolo, On Wed, 4 May 2022 01:25:23 -0300 Paolo Lucente wrote: Somehow i can't reproduce the problem, both pmacct.net

Re: [pmacct-discussion] [docker-doctors] pmacctd in docker

2022-05-08 Thread Paolo Lucente
Hi Thomas, The simplest thing i may recommend is to check it all working outside a container - this way you can easily isolate whether the issue is somehow related to the container (config or interaction of pmacctd with the container) or with the pmacct config itself. Paolo On 6/5/22

Re: [pmacct-discussion] TimescaleDB

2022-05-03 Thread Paolo Lucente
Hi John, Yes, i can confirm that writing directly from pmacct into a TimescaleDB, you can do it using the 'pgsql' plugin. Should you run into troubles (which you should not!) please let me know. Paolo On 3/5/22 17:33, John Jensen wrote: Hi all, Has anyone successfully used TimescaleDB

Re: [pmacct-discussion] pmacct.net

2022-05-03 Thread Paolo Lucente
Hi Karl, Always great to read from you & thanks for your note. Somehow i can't reproduce the problem, both pmacct.net and www.pmacct.net do actually work for me no problem (http of course, ie. not https, well no https is advertised out nor does it work). Can you please qualify the issue

Re: [pmacct-discussion] Easiest way to ingest nfacctd data into python?

2022-05-03 Thread Paolo Lucente
Hi Rich, While i don't have actual examples and while supporting the answers you already received, i may propose you the following architectural tips: * Write stuff into files with the 'print' plugin; using print_latest_file to point always to the latest finalized file and

Re: [pmacct-discussion] Contributing to the project

2022-04-22 Thread Paolo Lucente
Hi Suphannee, Thanks for this message and for your kind words about the project. The best way to contribute back - making sure to trace every line of code back to you / your company - is to do a Pull Request on GitHub, one per logical feature. Look forward to review your code and thanks in

Re: [pmacct-discussion] set_label=enp1s0_in filter='inbound' doesn't work, while it works against ppp0

2022-04-21 Thread Paolo Lucente
Hi Ruben, Indeed, very strange. And i could easily reproduce the issue on a Linux VM. To be frank i was not even aware of the existence of such inbound vs outbount knob, very convenient indeed. I suspect this is something new that good old bpf_filter() - which accepts filtering instructions,

Re: [pmacct-discussion] Exporting BGP enriched sflow data

2022-01-14 Thread Paolo Lucente
ow internally, maybe this will help to troubleshoot this. If you want, I can send you a pcap of the generated sflow packets. Will try latest git master the next days.  - Marcel Am 12.01.2022 um 04:18 schrieb Paolo Lucente: Hi Marcel, May i ask you one more detail since you looked into the sFlow r

Re: [pmacct-discussion] Exporting BGP enriched sflow data

2022-01-11 Thread Paolo Lucente
d5e336f2d83e0ff8f0b8475238339a557fc3eae8. Kind regards, Marcel Am 10.01.2022 um 02:26 schrieb Paolo Lucente: Hi Marcel, I tried latest & greatest code and i have the ASN info in sFlow using the sfprobe plugin with a config very similar to yours. Can you try to remove peer_dst_as

Re: [pmacct-discussion] Exporting BGP enriched sflow data

2022-01-09 Thread Paolo Lucente
Hi Marcel, I tried latest & greatest code and i have the ASN info in sFlow using the sfprobe plugin with a config very similar to yours. Can you try to remove peer_dst_as from 'aggregate' and give it another try? It is not supported anyway. Should it make the trick, i'll investigate deeper

Re: [pmacct-discussion] src_as and dst_as are always zero -- nfprobe

2021-12-20 Thread Paolo Lucente
Ciao Luca, Apologies for the late answer. I did manage to reproduce your issue and just pushed a fix to master code that seemed to work for me. It is a simple enough one-liner that, if you don't wish to move to master code, you could apply to 1.7.7:

[pmacct-discussion] pmacct 1.7.7 released !

2021-11-07 Thread Paolo Lucente
VERSION. 1.7.7 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate RPKI

Re: [pmacct-discussion] [docker-doctors] docker nfacct ... strange udp source ip !

2021-06-09 Thread Paolo Lucente
Hi Alessandro, (thanks for the kind words, first and foremost) Indeed, the test that Marc proposes is very sound, ie. check the actual packets coming in "on the wire" with tcpdump: do they really change sender IP address? Let me also confirm that what is used to populate peer_ip_src is

Re: [pmacct-discussion] BMP

2021-05-31 Thread Paolo Lucente
on, May 31, 2021 at 7:13 AM Paolo Lucente <mailto:pa...@pmacct.net>> wrote: Hi Edgar, For end-to-end solutions you have two main choices (of course i am excluding the obvious: buy a product or buy consultancy from somebody): 1) Google for them, and you may end up with re

Re: [pmacct-discussion] sql_num_hosts only giving null values in the MySQL database

2021-05-31 Thread Paolo Lucente
Hi Klaas, Do the log provide any hints / error that can put us on the right track? Should that not help, can you please enable debug on the pmacct side (that is: -d or "debug: true") to see if anything more helpful pops up in the logs? A total personal comment: sql_num_hosts is surely

Re: [pmacct-discussion] BMP

2021-05-30 Thread Paolo Lucente
outer -> bmp collector ( pmacct /pmbmpd ) -> kafka -> psql -> grafana." appreciate any help thanks Lipnitsky Edgar On Sat, May 29, 2021 at 9:53 PM Paolo Lucente <mailto:pa...@pmacct.net>> wrote: Hi Edgar, Thanks for your feedback wrt the BMP document

Re: [pmacct-discussion] BMP

2021-05-29 Thread Paolo Lucente
Hi Edgar, Thanks for your feedback wrt the BMP documentation. Let's try to get you up and running and improve docs but, in order to do that, i'd need some more specific question(s) from you. Where are you stuck? What is not working? Paolo On 29/5/21 13:05, edgar lip wrote: Hi pacct team

Re: [pmacct-discussion] IPFIX - bgp_next_hop vs ip_next_hop

2021-05-24 Thread Paolo Lucente
_ip_next_hop" and if set and NF9_IPV4_NEXT_HOP is there it will use it instead of NF9_BGP_IPV4_NEXT_HOP. In my use case this works fine since all bgp peers are on /31 or /30 ptp links and their ip is always equal to the ip of the other side of the ptp link. Andrej On 21.5.2021. 21:18, Paolo Lu

Re: [pmacct-discussion] IPFIX - bgp_next_hop vs ip_next_hop

2021-05-21 Thread Paolo Lucente
Hi Andrej, It is possible that you may find joy with the following combo 'nfacctd_as: bgp' and 'nfacctd_net: flow'. The next-hop for something not intuitive (but that i can explain and is documented) is tied to 'nfacctd_net'. Can you give it a try? If positive, we can take it from there,

Re: [pmacct-discussion] Configuration when sampling from multiple routers.

2021-05-17 Thread Paolo Lucente
them and to sum_host, I haven't tested that yet, will it work or is there something else can I try? Thank you. On Thu, May 13, 2021 at 2:40 AM Paolo Lucente <mailto:pa...@pmacct.net>> wrote: Hi Hendrik, What direction are you sampling NetFlow traffic at y

Re: [pmacct-discussion] Configuration when sampling from multiple routers.

2021-05-12 Thread Paolo Lucente
Hi Hendrik, What direction are you sampling NetFlow traffic at your edges? Is it consistent, are you sampling at both place in the same direction, either ingress (which would make more sense) or egress (which would make slight less sense)? If so, i'd be puzzled why you would get duplicated

Re: [pmacct-discussion] Tee and Kafka plugins

2021-05-12 Thread Paolo Lucente
opic in a data format retrievable from a topic. I hope my question a bit clearer. Thanks for your usual support and for your pmacct swiss knife Wilfrid -Original Message- From: Paolo Lucente Sent: Tuesday, 11 May 2021 04:52 To: pmacct-discussion@pmacct.net; Grassot, Wilfrid

Re: [pmacct-discussion] Tee and Kafka plugins

2021-05-10 Thread Paolo Lucente
Hi Wilfrid, Your understanding is correct although replication and collection are two separate pieces. You can have 1) a nfacctd replicator, that is binary NetFlow to binary NetFlow, where you could fan-out and filter pieces of your original export (to different collector, apps, etc.) and

Re: [pmacct-discussion] Another src_as / dst_as problem

2021-05-06 Thread Paolo Lucente
0505-1 (3edef0c3)) but unfortunately I have the same problem : src_as / dst_as field is still 0 :( Regards Cédric Le mar. 4 mai 2021 à 21:27, Paolo Lucente mailto:pa...@pmacct.net>> a écrit : Hi Cedric, It seems this should work. Can you confirm what version

Re: [pmacct-discussion] Another src_as / dst_as problem

2021-05-04 Thread Paolo Lucente
Hi Cedric, It seems this should work. Can you confirm what version are you using? a "pmacctd -V" would do so that i try to reproduce (and/or encourage you to get to 1.7.6 or master code on GitHub 8-)). Paolo On 4/5/21 14:56, BASSAGET Cédric wrote: Hello, I'm (once again) trying to export

Re: [pmacct-discussion] Kafka purge timing

2021-04-16 Thread Paolo Lucente
Hi Hendrik, You may see these messages appearing in your log (i can spot one in your excerpt in your previous email): "Finished cache entries (ie. print_cache_entries). Purging.". This is the reason for the intermediate purges. You have more entries to store for the 300 seconds interval than

Re: [pmacct-discussion] crashes since update to 1.7.6

2021-04-14 Thread Paolo Lucente
um 14:39 schrieb Paolo Lucente: Hi Goran, Can you please gather more information about the crash following these instructions: https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2876-#L2896 Output from either a gdb back trace or valgrind would be of help. Paolo

Re: [pmacct-discussion] crashes since update to 1.7.6

2021-04-14 Thread Paolo Lucente
Hi Goran, Can you please gather more information about the crash following these instructions: https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2876-#L2896 Output from either a gdb back trace or valgrind would be of help. Paolo On 13/04/2021 19:40, Göran Bruns wrote: Hi there,

Re: [pmacct-discussion] nfprobe vs. print plugin with ESP flows

2021-04-14 Thread Paolo Lucente
to look at the pcap you requested. I am still struggling to set up this netflow accounting for my routers. Thanks! --Sean On Mon, Mar 15, 2021 at 11:51 AM Sean wrote: Thanks for taking a look. I have sent the attachments directly to you. --Sean On Sun, Mar 14, 2021 at 11:16 AM Paolo

Re: [pmacct-discussion] nfprobe vs. print plugin with ESP flows

2021-03-14 Thread Paolo Lucente
Hi Sean, It smells like a bug. May i ask you to send me a brief capture of some of these ESP packets by unicast email? It would allow me to reproduce the issue. You can do that with tcpdump, in case you are not familiar with it something a-la "tcpdump -i -s 0 -n -w esp" should do it;

Re: [pmacct-discussion] Can't add AS number to netflow export

2021-02-15 Thread Paolo Lucente
for your reply. I really was hoping it would work :). Do you think it is still possible with nfacctd and just dumping traffic on the ethernet interface instead of receiving netflow? pon., 15 lut 2021 o 01:07 Paolo Lucente <mailto:pa...@pmacct.net>> napisał(a): Hi Michal, Simi

Re: [pmacct-discussion] Can't add AS number to netflow export

2021-02-14 Thread Paolo Lucente
Hi Michal, Similar topic was discussed recently on the list (*) but, as you can see, the broad generic answer to it is negative. Paolo (*) https://www.mail-archive.com/pmacct-discussion@pmacct.net/msg04028.html On 14/02/2021 22:34, Michał Margula wrote: Hi, I am trying to achieve

[pmacct-discussion] pmacct 1.7.6 released !

2021-02-07 Thread Paolo Lucente
VERSION. 1.7.6 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate RPKI

Re: [pmacct-discussion] sfacctd, aggregation and reexport

2021-01-01 Thread Paolo Lucente
Hi Moo, Unfortunately you are falling in the very same use-case of that message but using sFlow instead of NetFlow/IPFIX. Like you said, the first 3 can be done out of the box but - once sFlow is unpacked, you can't re-pack it with the additional info, ie. ASN information. This is mainly

Re: [pmacct-discussion] Flexible Netflow with Cisco ISR and nfacctd

2020-12-01 Thread Paolo Lucente
Hi Fabien, With prior knowledge of the template, ie. either you start nfacctd with '-d' (debug) so to see the content of templates in the logs or collect some NetFlow in a pcap file and open it with WireShark, you could use the aggregate_primitives framework of pmacct to define custom

Re: [pmacct-discussion] MySQL plugin processes terminating

2020-11-20 Thread Paolo Lucente
Hi Klaas, Is it the main MySQL plugin failing on you or the writer processes (so the main MySQL plugin stays up and running)? Is it possible it is a simple memory issue, a-la you should throw more memory at it? You can collect more info on the crash (which may be useful for debug and

Re: [pmacct-discussion] Fragment/4 buffer full. Skipping fragments.

2020-11-20 Thread Paolo Lucente
Hi Pierre, Maybe you need to increase the pmacctd_frag_buffer_size (by default 4MB and perhaps not sufficient for your traffic footprint): https://github.com/pmacct/pmacct/blob/1.7.5/CONFIG-KEYS Give that a try. Paolo On 20/11/2020 12:53, Pierre Grié wrote: Hello, We are using pmacct to

Re: [pmacct-discussion] pmacctd and OpenVPN

2020-11-20 Thread Paolo Lucente
Hi Erik, Take a capture with tcpdump of some of these packets on the tun interface and send it via unicast email. Let's see what is possible or what is the issue. Paolo On 20/11/2020 11:34, Erik wrote: Hi, I am running a VPN server based on OpenVPN and recently there was a request to

Re: [pmacct-discussion] nfacctd tee - filter subnets before transmit

2020-11-19 Thread Paolo Lucente
ing from the config? Thank you! Eric On Wed, Nov 18, 2020 at 10:46 AM Paolo Lucente mailto:pa...@pmacct.net>> wrote: Hi Eric, You could look at this piece of documentation for what you are trying to do: https://github.com/pmacct/pmacc

Re: [pmacct-discussion] nfacctd tee - filter subnets before transmit

2020-11-18 Thread Paolo Lucente
Hi Eric, You could look at this piece of documentation for what you are trying to do: https://github.com/pmacct/pmacct/blob/1.7.5/QUICKSTART#L2106-#L2200 The example focuses on src_mac and dst_mac, you should be using src_net and dst_net instead. Paolo On 18/11/2020 05:38, eric c wrote:

Re: [pmacct-discussion] 95 percentile (again)

2020-11-03 Thread Paolo Lucente
*Von:* Paolo Lucente *Gesendet:* Dienstag, 3. November 2020 00:42 *An:* pmacct-discussion@pmacct.net ; Klaas Tammling *Betreff:* Re: [pmacct-discussion] 95 percentile (again) Hi Klaas, You are right pmacct does not do 95th percentile calculations

Re: [pmacct-discussion] 95 percentile (again)

2020-11-02 Thread Paolo Lucente
Hi Klaas, You are right pmacct does not do 95th percentile calculations as these are much better suited to be post-process actions (due to the increased data visibility they require) than done in-line at the collector layer. On your question about bits/s. 95th percentile bases on the

Re: [pmacct-discussion] Netflow Iframe Index

2020-10-28 Thread Paolo Lucente
Hi Samir, Not sure, do you mean SNMP interface ifIndex by iframe index in/out? If not, please tell me more - it does not ring a bell; if yes, you have it there populated, 'iface_in' and 'iface_out' fields. Paolo On 28/10/2020 19:41, Samir Faci wrote: I'm using the nfacctd process to

Re: [pmacct-discussion] IPFIX record export with BGP Next Hop added

2020-10-22 Thread Paolo Lucente
Hi Kamiel, Unfortunately, no, this scenario (take IPFIX, massage it & spit it out) is not supported. Paolo On 21/10/2020 09:55, Braet, Kamiel wrote: Hello everyone, Just wanted to know if it is possible to use PMACCT to import IPFIX records and BGP data. After this determine the BGP

Re: [pmacct-discussion] DTLS encrypted flow data

2020-10-13 Thread Paolo Lucente
it to nfacctd_dtls_port. While nfacctd acknowledges that it's receiving DTLS there seem to be some issues that prevent successful parsing of data. Hope I'll be able to find some more time to dig deeper and make it work. Stay safe, Felix Am 09.10.20, 21:49 schrieb "Paolo Lu

Re: [pmacct-discussion] DTLS encrypted flow data

2020-10-09 Thread Paolo Lucente
Hi Felix, Monumental pleasure to read from you, hope all is well. The feature was conceived in conjunction with the great DE-CIX folks, you can see the announcement here: https://twitter.com/thking/status/1292903640877932544 . In the context of pmacct, yes, i have indeed on the roadmap to

Re: [pmacct-discussion] pmbgpd -> Kafka Local Queue Full

2020-09-02 Thread Paolo Lucente
Hi Andy, I may suggest to check Kafka logs and perhaps see if anything useful comes out of librdkafka stats (ie. set "global, statistics.interval.ms, 6" in your librdkafka.conf). Check also that, if you are adding load to existing load, the Kafka broker is not pegging 100% CPU or maxing

Re: [pmacct-discussion] Capturing interface traffic with pmacct and inserting the data in PostgreSQL

2020-08-26 Thread Paolo Lucente
' '--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins'* ** *For suggestions, critics, bugs, contact me: Paolo Lucente .* *[root@pcap pmacct]# pmacctd -V* *Promiscuous Mode Accounting Daemon, pmacctd 1.7.6-git [20200826-0 (57a0334d)]* ** *Arguments:* *'--enable-pgsql' '--enable-l2

Re: [pmacct-discussion] tee plugin ipv6 problem

2020-07-28 Thread Paolo Lucente
Hey Alexander, Can you send me a sample of the IPv6 packets by unicast email? Ideally two tcpdump captures, ie. 'tcpdump -i lo -n -w port ' and 'tcpdump -i -n -w port 2101', taken in parallel. Shall i find you positive on generating a sample, please do not do one single capture with '-i

Re: [pmacct-discussion] master - ndpi on 32bit CentOS 6

2020-07-09 Thread Paolo Lucente
to nDPI. Paolo On 09/07/2020 18:19, Steve Clark wrote: Thanks for checking, could you tell what distro and version you tested on? Also when I compile on 32 bit I get a lot of warning of redefines between ndpi.h and pmacct.h do you get those also? On 07/09/2020 11:55 AM, Paolo Lucente

Re: [pmacct-discussion] master - ndpi on 32bit CentOS 6

2020-07-09 Thread Paolo Lucente
Hi Steve, I do have avail of a i686-based VM. I can't say everything is tested on i686 but i tend to check every now and then that nothing fundamental is broken. I took the example config you used, compiled master code with the same config switches as you did (essentially --enable-ndpi) and

Re: [pmacct-discussion] 1.7.5 with static ndpi

2020-06-24 Thread Paolo Lucente
Hi Steve, Apart from asking the obvious - personal curiosity! - why do you want to link against a static nDPI library. There are a couple main avenues i can point you to depending on your goal: 1) You can supply configure with a --with-ndpi-static-lib knob; guess the static lib and the dynamic

[pmacct-discussion] pmacct & Docker

2020-06-24 Thread Paolo Lucente
Dears, A brief email to say that thanks to the monumental efforts of Marc Sune and Claudio Ortega we could bring pmacct a bit closer to the Docker universe. Since today we are shipping official pmacct containers on Docker Hub ( https://hub.docker.com/u/pmacct ) organized as follows: * A

[pmacct-discussion] pmacct 1.7.5 released !

2020-06-17 Thread Paolo Lucente
VERSION. 1.7.5 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate RPKI

Re: [pmacct-discussion] networks_file reload

2020-06-08 Thread Paolo Lucente
Hi Olaf, To confirm that the file is reloaded. Unfortunately all log messages in loading up a networks_file are related to errors, warnings and debug. No info message to say that simply all went good. So i just added one as an action item for the issue you raised:

Re: [pmacct-discussion] pmacctd and src_std_comm aggregation

2020-05-26 Thread Paolo Lucente
, doing some spare-time work on it, i guess we can converge on this in a week or a couple. Paolo On Mon, May 25, 2020 at 06:21:56PM +0200, Simone Ricci wrote: > Ciao Paolo, > > > Il giorno 25 mag 2020, alle ore 16:03, Paolo Lucente ha > > scritto: > > > > Ciao Simone

Re: [pmacct-discussion] pmacctd and src_std_comm aggregation

2020-05-25 Thread Paolo Lucente
Ciao Simone, If i got it correct you are after static mapping of communities to input traffic - given an input interface / vlan or an ingress router or a source MAC address. It seems doable, like you said, adding a machinery like it exists for the source peer ASN. I'd have one question for you:

Re: [pmacct-discussion] BGP correlation not working with nfacctd, all BGP set to 0

2020-05-19 Thread Paolo Lucente
;a.b.c.d/27", > "rd": "0:ASN:900290024", > "label": "63455" > } > { > "seq": 3, > "timestamp": "2020-05-19 07:15:00", > "peer_ip_src": " w.x.y.z ", > "ip_prefix":

Re: [pmacct-discussion] help configuration cisco 4948E-F netflow-lite

2020-05-19 Thread Paolo Lucente
Hi Ionut, Thanks for getting in touch with this. From the log file you sent apparently the switch sends element #104 (layer2packetSectionData) to include portion of the sampled frame. Unfortunately such element has been "deprecated in favor of 315 dataLinkFrameSection. Layer 2 packet section

Re: [pmacct-discussion] BGP correlation not working with nfacctd, all BGP set to 0

2020-05-19 Thread Paolo Lucente
b.c.d) corresponding to ip_prefix = a.b.c.d ? > > Wilfrid > > > -Original Message- > From: Grassot, Wilfrid > Sent: Monday, 18 May 2020 17:05 > To: Paolo Lucente ; pmacct-discussion@pmacct.net > Subject: RE: [pmacct-discussion] BGP correlation not working w

Re: [pmacct-discussion] BGP correlation not working with nfacctd, all BGP set to 0

2020-05-18 Thread Paolo Lucente
Hi Wilfrid, Thanks for getting in touch. A couple of notes: 1) if you are sending vpnv4 routes - and if that is a requirement - then you will need a flow_to_rd_map to map flows to the right VPN (maybe basing on the input interface at the ingress router? just an idea); 2) Confederations

[pmacct-discussion] pmacct 1.7.5 code freeze

2020-05-10 Thread Paolo Lucente
Dears, pmacct 1.7.5 has entered code freeze today with the outlook of having the official release wrapped up in approx one month. The code has been branched out on GitHub: https://github.com/pmacct/pmacct/tree/1.7.5 Code freeze means that until release time only capital bug fixes will be

Re: [pmacct-discussion] Tracking ingress throughput

2020-04-30 Thread Paolo Lucente
Hi, By sendng a SIGUSR1 to the daemon you are returned some stats informaton in the log. Please see here: https://github.com/pmacct/pmacct/blob/1.7.4/docs/SIGNALS#L17-#L40 Paolo On Wed, Apr 29, 2020 at 10:12:53AM +0530, HEMA CHANDRA YEDDULA wrote: > > Hi paolo, > > Is there any way to

[pmacct-discussion] Test

2020-04-23 Thread Paolo Lucente
Please ignore ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists

[pmacct-discussion] Test

2020-04-23 Thread Paolo Lucente
Please ignore ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists

[pmacct-discussion] Test

2020-04-23 Thread Paolo Lucente
Please ignore ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists

[pmacct-discussion] Test

2020-04-23 Thread Paolo Lucente
Please ignore ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] BGP attributes are empty for almost all the data

2020-04-17 Thread Paolo Lucente
Hi Alexandre, Why don't you try to do a dump of routes received by pmacct? Like: https://github.com/pmacct/pmacct/blob/1.7.4/QUICKSTART#L1780-#L1781 This test may require you compiling pmacct with JSON / Jansson support. Also, for a test you could also add 'dst_host' on your 'aggregate'

  1   2   3   4   5   6   7   8   9   10   >