Re: [pmacct-discussion] pmbgpd -> Kafka Local Queue Full

2020-09-02 Thread Paolo Lucente
Hi Andy, I may suggest to check Kafka logs and perhaps see if anything useful comes out of librdkafka stats (ie. set "global, statistics.interval.ms, 6" in your librdkafka.conf). Check also that, if you are adding load to existing load, the Kafka broker is not pegging 100% CPU or maxing

Re: [pmacct-discussion] Capturing interface traffic with pmacct and inserting the data in PostgreSQL

2020-08-26 Thread Paolo Lucente
' '--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins'* ** *For suggestions, critics, bugs, contact me: Paolo Lucente .* *[root@pcap pmacct]# pmacctd -V* *Promiscuous Mode Accounting Daemon, pmacctd 1.7.6-git [20200826-0 (57a0334d)]* ** *Arguments:* *'--enable-pgsql' '--enable-l2

Re: [pmacct-discussion] tee plugin ipv6 problem

2020-07-28 Thread Paolo Lucente
Hey Alexander, Can you send me a sample of the IPv6 packets by unicast email? Ideally two tcpdump captures, ie. 'tcpdump -i lo -n -w port ' and 'tcpdump -i -n -w port 2101', taken in parallel. Shall i find you positive on generating a sample, please do not do one single capture with '-i

Re: [pmacct-discussion] master - ndpi on 32bit CentOS 6

2020-07-09 Thread Paolo Lucente
to nDPI. Paolo On 09/07/2020 18:19, Steve Clark wrote: Thanks for checking, could you tell what distro and version you tested on? Also when I compile on 32 bit I get a lot of warning of redefines between ndpi.h and pmacct.h do you get those also? On 07/09/2020 11:55 AM, Paolo Lucente

Re: [pmacct-discussion] master - ndpi on 32bit CentOS 6

2020-07-09 Thread Paolo Lucente
Hi Steve, I do have avail of a i686-based VM. I can't say everything is tested on i686 but i tend to check every now and then that nothing fundamental is broken. I took the example config you used, compiled master code with the same config switches as you did (essentially --enable-ndpi) and

Re: [pmacct-discussion] 1.7.5 with static ndpi

2020-06-24 Thread Paolo Lucente
Hi Steve, Apart from asking the obvious - personal curiosity! - why do you want to link against a static nDPI library. There are a couple main avenues i can point you to depending on your goal: 1) You can supply configure with a --with-ndpi-static-lib knob; guess the static lib and the dynamic

[pmacct-discussion] pmacct & Docker

2020-06-24 Thread Paolo Lucente
Dears, A brief email to say that thanks to the monumental efforts of Marc Sune and Claudio Ortega we could bring pmacct a bit closer to the Docker universe. Since today we are shipping official pmacct containers on Docker Hub ( https://hub.docker.com/u/pmacct ) organized as follows: * A

[pmacct-discussion] pmacct 1.7.5 released !

2020-06-17 Thread Paolo Lucente
VERSION. 1.7.5 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate RPKI

Re: [pmacct-discussion] networks_file reload

2020-06-08 Thread Paolo Lucente
Hi Olaf, To confirm that the file is reloaded. Unfortunately all log messages in loading up a networks_file are related to errors, warnings and debug. No info message to say that simply all went good. So i just added one as an action item for the issue you raised:

Re: [pmacct-discussion] pmacctd and src_std_comm aggregation

2020-05-26 Thread Paolo Lucente
, doing some spare-time work on it, i guess we can converge on this in a week or a couple. Paolo On Mon, May 25, 2020 at 06:21:56PM +0200, Simone Ricci wrote: > Ciao Paolo, > > > Il giorno 25 mag 2020, alle ore 16:03, Paolo Lucente ha > > scritto: > > > > Ciao Simone

Re: [pmacct-discussion] pmacctd and src_std_comm aggregation

2020-05-25 Thread Paolo Lucente
Ciao Simone, If i got it correct you are after static mapping of communities to input traffic - given an input interface / vlan or an ingress router or a source MAC address. It seems doable, like you said, adding a machinery like it exists for the source peer ASN. I'd have one question for you:

Re: [pmacct-discussion] BGP correlation not working with nfacctd, all BGP set to 0

2020-05-19 Thread Paolo Lucente
;a.b.c.d/27", > "rd": "0:ASN:900290024", > "label": "63455" > } > { > "seq": 3, > "timestamp": "2020-05-19 07:15:00", > "peer_ip_src": " w.x.y.z ", > "ip_prefix":

Re: [pmacct-discussion] help configuration cisco 4948E-F netflow-lite

2020-05-19 Thread Paolo Lucente
Hi Ionut, Thanks for getting in touch with this. From the log file you sent apparently the switch sends element #104 (layer2packetSectionData) to include portion of the sampled frame. Unfortunately such element has been "deprecated in favor of 315 dataLinkFrameSection. Layer 2 packet section

Re: [pmacct-discussion] BGP correlation not working with nfacctd, all BGP set to 0

2020-05-19 Thread Paolo Lucente
b.c.d) corresponding to ip_prefix = a.b.c.d ? > > Wilfrid > > > -Original Message- > From: Grassot, Wilfrid > Sent: Monday, 18 May 2020 17:05 > To: Paolo Lucente ; pmacct-discussion@pmacct.net > Subject: RE: [pmacct-discussion] BGP correlation not working w

Re: [pmacct-discussion] BGP correlation not working with nfacctd, all BGP set to 0

2020-05-18 Thread Paolo Lucente
Hi Wilfrid, Thanks for getting in touch. A couple of notes: 1) if you are sending vpnv4 routes - and if that is a requirement - then you will need a flow_to_rd_map to map flows to the right VPN (maybe basing on the input interface at the ingress router? just an idea); 2) Confederations

[pmacct-discussion] pmacct 1.7.5 code freeze

2020-05-10 Thread Paolo Lucente
Dears, pmacct 1.7.5 has entered code freeze today with the outlook of having the official release wrapped up in approx one month. The code has been branched out on GitHub: https://github.com/pmacct/pmacct/tree/1.7.5 Code freeze means that until release time only capital bug fixes will be

Re: [pmacct-discussion] Tracking ingress throughput

2020-04-30 Thread Paolo Lucente
Hi, By sendng a SIGUSR1 to the daemon you are returned some stats informaton in the log. Please see here: https://github.com/pmacct/pmacct/blob/1.7.4/docs/SIGNALS#L17-#L40 Paolo On Wed, Apr 29, 2020 at 10:12:53AM +0530, HEMA CHANDRA YEDDULA wrote: > > Hi paolo, > > Is there any way to

[pmacct-discussion] Test

2020-04-23 Thread Paolo Lucente
Please ignore ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists

[pmacct-discussion] Test

2020-04-23 Thread Paolo Lucente
Please ignore ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists

[pmacct-discussion] Test

2020-04-23 Thread Paolo Lucente
Please ignore ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists

[pmacct-discussion] Test

2020-04-23 Thread Paolo Lucente
Please ignore ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] BGP attributes are empty for almost all the data

2020-04-17 Thread Paolo Lucente
Hi Alexandre, Why don't you try to do a dump of routes received by pmacct? Like: https://github.com/pmacct/pmacct/blob/1.7.4/QUICKSTART#L1780-#L1781 This test may require you compiling pmacct with JSON / Jansson support. Also, for a test you could also add 'dst_host' on your 'aggregate'

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-15 Thread Paolo Lucente
mp_start': '2020-04-14 14:15:39.00', > 'timestamp_end': '2020-04-14 14:15:54.00', 'packets': 5, 'bytes': 260, > 'writer_id': 'default_kafka/75091'} > > Did I miss anything ? > > > Thanks ! > > > > On Tue, Apr 14, 2020 at 10:26 AM Paolo Lucente wrote: >

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-14 Thread Paolo Lucente
: > Thank you man, I did this test but I did not see the id being pushed along > with the Netflow info to Kafka topic. Is there the place the information > would show up ? > > > On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente wrote: > > > > > Hi Emanuel, > >

Re: [pmacct-discussion] How to record ICMP and ICMP6 types/codes by pmacctd?

2020-04-14 Thread Paolo Lucente
*generators*, not collectors. My mistake! > > > > Yes, I posted the config that generates > the netflow 9 flows, since I hoped to see if it was missing > something for including the ICMP and ICMP6 types/codes. > > > -Indy &

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-14 Thread Paolo Lucente
ndition. :) > > I am wondering if I could use this one to include a different tag on it > process/collector, but have not yet figured out how. Any thoughts ? > > label: String label, ie. as result of > pre_tag_map evaluation > > > Than

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-13 Thread Paolo Lucente
Hi Emanuel, I think you are looking for (i admit, non-intuitive) 'peer_src_ip' primitive: $ nfacctd -a | grep peer_src_ip peer_src_ip : IP address or identificator of telemetry exporting device Without the grep you can see all supported primitives by the nfacctd release

Re: [pmacct-discussion] How to record ICMP and ICMP6 types/codes by pmacctd?

2020-04-13 Thread Paolo Lucente
Hi, Let me confirm that collecting the ICMP type is partially supported; the native dst_port primitive is locked to UDP and TCP only - making this not suitable for NetFlow v5 kind of scenarios; but if using NetFlow v9 and/or IPFIX you could define your own custom primitive via the

Re: [pmacct-discussion] Looking for value suggestions

2020-04-06 Thread Paolo Lucente
Hi Mark, Since 'nfprobe' plugin would generate sequence numbers on output of IPFIX packets, that would rule out any buffering issue (not saying buffers should not be looked at, just saying buffering considerations are disjoint from the sequencing issue). Do you have any multi-paths between

Re: [pmacct-discussion] Besoin d’aide: Unknown plug-in type: mysql. Ignoring, No plug-in has been activated; defaulting to in memory table.

2020-03-31 Thread Paolo Lucente
d > sql_table_version: 1 > ! > > I would like to know: > 1- how to recover BMP data > 2- how to feed the database (mysql) with BMP data > > thank you in advance for your help > I am in a hurry so that I can move forward thank you. > > Le jeu. 26 mars 2020 à 14:05

Re: [pmacct-discussion] Besoin d’aide: Unknown plug-in type: mysql. Ignoring, No plug-in has been activated; defaulting to in memory table.

2020-03-26 Thread Paolo Lucente
Hi, You need to compile pmacct with MySQL support, --enable-mysql. You may profit from the following section of the QUICKSTART document: https://github.com/pmacct/pmacct/blob/1.7.4/QUICKSTART#L109-#L167 Bu perhaps the whole chapter I and II are good readings to start. Paolo On Wed, Mar 25,

Re: [pmacct-discussion] tee and other plugins simultaneously in nfacctd?

2020-03-25 Thread Paolo Lucente
Hi Jason, Yes, you should use multiple nfacctd instances; one to replicate, one to collect. I intend in future to allow these two distinct functions to run within the same daemon but that's not yet possible at the moment (some coding needed). Paolo On Wed, Mar 25, 2020 at 08:03:39AM -0400,

Re: [pmacct-discussion] TCP segments handling

2020-03-05 Thread Paolo Lucente
Hi, Would you have a packet trace in pcap format to share via unicast email in order to reproduce the issue at my end? If not, i am afraid i can't do much. Paolo On Wed, Mar 04, 2020 at 12:27:36PM +0530, HEMA CHANDRA YEDDULA wrote: > Hi > > We have a case where the packet has the

Re: [pmacct-discussion] Pmacct configuration with direction of traffic

2020-02-27 Thread Paolo Lucente
> Hi Paolo, > > On Tue, Feb 25, 2020 at 6:41 PM Paolo Lucente wrote: > > > > > Hi Alex, > > > > Thanks for your feedback. I see you did run "tcpdump -n -vv -i nflog:1" > > which is equivalent to run uacctd without any filters; as you may know

Re: [pmacct-discussion] Pmacct configuration with direction of traffic

2020-02-25 Thread Paolo Lucente
8.8.8.8 > 192.168.28.11: ICMP echo reply, id 17353, seq 2, length 64 > > The pmacct version I am running is latest master. > Thank you for your assistance. > > Alex > > > On Mon, Feb 24, 2020 at 6:20 PM Alex K wrote: > > > Hi Paolo, > > > > O

Re: [pmacct-discussion] Pmacct configuration with direction of traffic

2020-02-22 Thread Paolo Lucente
Hi Alex, Is it possible with the new setup - the one where pre_tag_map does not match anything - the traffic is VLAN-tagged (or MPLS-labelled)? If so, you should adjust filters accordingly and add 'vlan and', ie. "vlan and src net 192.168.28.0/24 or vlan and src net 192.168.100.0/24". Paolo

Re: [pmacct-discussion] Using large hex values for "label" in pre_tag_map results in strange SQL

2020-02-13 Thread Paolo Lucente
Hey Tim, It should be that the issue you described in the previous email and this one are connected. And it should be that i pin-pointed to a common root cause addressed by this commit: https://github.com/pmacct/pmacct/commit/4e648cc96aae99ee5f4b1c9e135a1afa73b864b3 Which, in turn, is

Re: [pmacct-discussion] Realistic Scaling of pre_tag_map?

2020-02-13 Thread Paolo Lucente
ers that aren't supported? Seems as if '-' in any set_label > operation means the whole string gets ignored.. > > The use-case is just mapping ip+ifIndex -> downstream devices with a label, > but I've got a lot of interfaces to match there.. > > -- > Tim > > &

Re: [pmacct-discussion] Realistic Scaling of pre_tag_map?

2020-02-11 Thread Paolo Lucente
Hey Tim, It really depends whether you can leverage maps_index (*) or not. If yes then computations are O(1) and hence you can scale it as much as you like and i can confirm you there is people building maps of the same magnitude as you have in mind. If not then it's not going to work but then

[pmacct-discussion] pmacct 1.7.4p1 released !

2020-02-09 Thread Paolo Lucente
VERSION. 1.7.4p1 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate

Re: [pmacct-discussion] [PATCH 2/2] * nfprobe: per-interface flows

2020-01-28 Thread Paolo Lucente
Hi Mikhail, Thanks very much also for this contribution. I have passed the patch, it does make full sense. Also in this case i removed the part of the config knob: interfaces, if populated, _must_ be taken into account when comparing flows; if not populated, which is the default case, well it

Re: [pmacct-discussion] Periodically printing flowtrack structure values

2020-01-27 Thread Paolo Lucente
Hi, Unfortunately this is not possible. Paolo On Mon, Jan 27, 2020 at 01:51:03PM +0530, HEMA CHANDRA YEDDULA wrote: > > Hi Paolo, > > Thanks for previous replies. > > Is it possible to log FLOWTRACK structure components like flows_exported, > packets_exported etc., for every 5m and

Re: [pmacct-discussion] pmacct 1.7.4 released !

2020-01-27 Thread Paolo Lucente
/pmacct/issues/356 > > The only changes is I dist-upgrade the machine itself as installed from > > source 1.7.4 release. > > I use print plugin on my side on nfacctd processes. > > Please let me know how I can trouble

Re: [pmacct-discussion] [PATCH 1/2] * pmacctd: allow configuring pcap_setdirection

2020-01-24 Thread Paolo Lucente
Hi Mikhail, Many thanks for your contribution. I have slightly reviewed your patch to test libpcap for pcap_setdirection() in configure.ac (as doing it the way you did would fail compiling for older libpcap versions). See the commit log here (of course with kudos to you):

Re: [pmacct-discussion] sfacctd and nfacctd using the same db tables

2020-01-22 Thread Paolo Lucente
Hi Jordan, Yes, that is a valid scenario. To minimize effects locking (which impact is only increased memory footprint due to the writers sitting there waiting for the lock), you could play with sql_startup_delay so that the two daemons fire up writers with some time offset; but, if the two

Re: [pmacct-discussion] uninitialized req passed to plugin_requests and load_id_file in pm_pcap_cb??

2020-01-21 Thread Paolo Lucente
q : > https://github.com/pmacct/pmacct/blob/d72440dc9a7d0d0a7ed9502f1dd31b90105b1d95/src/nl.c#L51 > , > and noone zeroes it up before using it seems. > > Mikhail > > On Tue, 21 Jan 2020 at 02:25, Paolo Lucente wrote: > > > > > > Hi Mikhail, > >

Re: [pmacct-discussion] uninitialized req passed to plugin_requests and load_id_file in pm_pcap_cb??

2020-01-20 Thread Paolo Lucente
Hi Mikhail, If you see all the daemons that make use of the 'req' structure have a memset() for 'req' shortly after its declaration. For example here in pmacctd: https://github.com/pmacct/pmacct/blob/master/src/pmacctd.c#L360 Paolo On Fri, Jan 17, 2020 at 07:10:13PM +0100, Mikhail Sennikovsky

Re: [pmacct-discussion] Only meaningful custom primitives should pick the value

2020-01-20 Thread Paolo Lucente
and Regards, > Hema Chandra Yeddula > > > > > > > On Thu, 16 Jan 2020 23:48:04 +, Paolo Lucente wrote > Hi, > > If you define certain primitives, those not present in the parsed flow > entry should be indeed left blank. If that is not the case, then it's

Re: [pmacct-discussion] Only meaningful custom primitives should pick the value

2020-01-16 Thread Paolo Lucente
Hi, If you define certain primitives, those not present in the parsed flow entry should be indeed left blank. If that is not the case, then it's a bug and i'd like to ask you for a way to reproduce the issue (so your config along with a brief capture (template + data packets) of your data.

Re: [pmacct-discussion] Processing payload data

2020-01-10 Thread Paolo Lucente
Hi, Matching on regex seems indeed a good use-case for this feature - did you test whether this would work for you in theory? That is, given the full payload, is there a regex that can extract what you are looking for? This said, unfortunately this is not implemented today but it would be a

Re: [pmacct-discussion] effort to relicense pmacct from GPL to a BSD-style license

2020-01-09 Thread Paolo Lucente
Hi Karl and Lennert, Community, Thanks very much for your comments and for the opportunity you are giving me to address them. I'd like to comment publicly - so that we can cast this in stone and have it archvied for time to tell - that there are no hidden reasons, secret plans or favouritism

[pmacct-discussion] pmacct 1.7.4 released !

2019-12-31 Thread Paolo Lucente
), Thomas Graf ( @graf3 , @graf3net ), Paolo Lucente ( @paololucente ). + Introduced support for the 'vxlan' VXLAN/VNI primitive in all traffic daemons (NetFlow/IPFIX, sFlow and libpcap/ULOG). Existing inner tunnel primitives (ie. tunnel_src_host, tunnel_dst_host, tunnel_proto, etc.) have been

Re: [pmacct-discussion] only log_type update in BMP messages

2019-12-10 Thread Paolo Lucente
Hi Rasto, May you try master code on GitHub or the 1.7.4 branch (curently on freeze and due to be released later in the month)? In the last few months there has been plenty of working on the BMP-related code. Should that still not work, it would help if you could generate me a brief capture

Re: [pmacct-discussion] Incorporating GTP fields in IPFIX data

2019-12-06 Thread Paolo Lucente
Hi, Do you have NetFlow/IPFIX data containing such info already and you want to collect it with pmacct? If so, it should be possible to define some custom primitives for that: ping me via unicast email sending a sample trace of such data. If the question is instead to read some GTP traffic and

Re: [pmacct-discussion] pmbgpd looking glass don't start

2019-10-25 Thread Paolo Lucente
Hi Alex, The Looking Glass feature depends on ZeroMQ so you will need to compile pmacct with --enable-zmq. I will produce asap a small patch to output an error if pmacct is not compiled against ZeroMQ and bgp_daemon_lg is set to true: somehow this dependencey is mentioned in both CONFIG-KEYS

Re: [pmacct-discussion] BGP AS values are 0

2019-10-20 Thread Paolo Lucente
om/pmacct/pmacct/blob/93e414f5f34a380281328df58069cc521c33a3c5/CONFIG-KEYS#L1743> > it > appears that `fallback` is not a valid option for `pmacctd_as` and > `pmacctd_net`. Is that right? > > On Sun, Oct 20, 2019 at 9:43 AM Paolo Lucente wrote: > > > > > Hi Brooks, > > > > The

Re: [pmacct-discussion] BGP AS values are 0

2019-10-20 Thread Paolo Lucente
destination IP prefixes can get associated to AS0. > > Is there a way to distinguish between AS0 being my own AS and an unknown > one? > > On Sun, Oct 13, 2019 at 3:39 PM Paolo Lucente wrote: > > > > > Wonderful. Thank you Brooks for sharing your finding. I wil

Re: [pmacct-discussion] BGP map for dual stack IPv4 & IPv6

2019-10-20 Thread Paolo Lucente
ed) > Routes: 0 imported, 74840 exported, 0 preferred > Route change stats: received rejected filteredignored > accepted > Import updates: 0 0 0 0 >0 > Import withdraws:0 0

Re: [pmacct-discussion] peer_src_ip empty

2019-10-19 Thread Paolo Lucente
Hi Brooks, peer_src_ip is definitely the primitive you are looking for. From the previous thread i have a suspect: you may be using the wrong daemon. What daemon are you running? Is it possible you want to collect NetFlow/ IPFIX or sFlow but you are running pmacctd? That would explain. Just in

Re: [pmacct-discussion] BGP map for dual stack IPv4 & IPv6

2019-10-14 Thread Paolo Lucente
eady. > Table master6: > 2602:fe2e:42::/48unicast [static4 2019-10-10] * (200) > via 2602:fe2e:1::135 on ens5 > Type: static univ > ``` > > But there should be an AS present for the source: > > ``` > $ sudo birdc show route for 2607:f8b0:4006:814::20

Re: [pmacct-discussion] BGP map for dual stack IPv4 & IPv6

2019-10-13 Thread Paolo Lucente
On Sun, Oct 13, 2019 at 3:47 PM Paolo Lucente wrote: > > > > > Hi Brooks, > > > > You are in an unsupported use-case, ie. same BGP Agent ID maped onto two > > different entries. You can get out of it in three different ways: 1) my > > top recommendation: tr

Re: [pmacct-discussion] BGP map for dual stack IPv4 & IPv6

2019-10-13 Thread Paolo Lucente
Hi Brooks, You are in an unsupported use-case, ie. same BGP Agent ID maped onto two different entries. You can get out of it in three different ways: 1) my top recommendation: travel both addrress families as part of the same BGP session; 2) use two different BGP Agent ID for ipv4 and for ipv6;

Re: [pmacct-discussion] BGP AS values are 0

2019-10-13 Thread Paolo Lucente
me: 10 > kafka_history: 5m > kafka_history_roundoff: m > ``` > > And in BIRD: > > ``` > protocol bgp AS00v4c1 from monitor46 { > description "pmacctd"; > local 127.0.0.1 as 00; > neighbor 127.0.0.2 port 180 as 00; > rr client; > }

Re: [pmacct-discussion] BGP AS values are 0

2019-10-13 Thread Paolo Lucente
4831, offset 0, flags [DF], proto > > TCP (6), length 52) > > 127.0.0.1.180 > 127.0.0.1.36143: Flags [.], cksum 0xfe28 (incorrect -> > > 0x998d), ack 9274, win 342, options [nop,nop,TS val 1972706308 ecr > > 1972706308], length 0 > > ``` > > > > Which

Re: [pmacct-discussion] BGP AS values are 0

2019-10-13 Thread Paolo Lucente
quot;, > "peer_tcp_port": 39587, "event_type": "dump_close", "entries": 0, "tables": > 1, "seq": 0} > ``` > > But looking at the BIRD side of things, I can see the routes are indeed > being exported: > > ``` > bird

Re: [pmacct-discussion] BGP AS values are 0

2019-10-13 Thread Paolo Lucente
Hi Brooks, +1 to Felix's answer. Also maybe two obvious pointsa: 1) with an iBGP peering setup, AS0 can mean unknown or your own ASN (being a number rather than a string, null is not an option) and 2) until routes are received, source/destination IP prefixes can get associated to AS0. Config

Re: [pmacct-discussion] getting IPv6 traffic per /64 subnet

2019-10-11 Thread Paolo Lucente
_dst, dst_port, ip_proto, mac_src, mac_dst, ip_src, ip_dst, packets, > bytes) VALUES ('::', 0, 'ip', '0:0:0:0:0:0', '0:0:0:0:0:0', '0.0.0.0', > '0.0.0.0', 242934, 335519993) > > > do you know why it behave like this > the version i have is > pmacct -V > pmacct, pmacct cl

Re: [pmacct-discussion] getting IPv6 traffic per /64 subnet

2019-10-10 Thread Paolo Lucente
Hi, Thank you for reporting this. Can show the integral error message you get back from MySQL? It may give relevant additional info; feel free to anonimize any confidential data it may contain (ie. IP addresses). Paolo On Thu, Oct 10, 2019 at 12:34:12PM -0400, moftah moftah wrote: > Hi All,

Re: [pmacct-discussion] nfacctd monthly accounting problem

2019-09-10 Thread Paolo Lucente
ervice dies. > > Terry > > > -Original Message- > From: pmacct-discussion On Behalf Of > Paolo Lucente > Sent: Thursday, September 5, 2019 7:16 AM > To: pmacct-discussion@pmacct.net > Subject: Re: [pmacct-discussion] nfacctd monthly accounting p

Re: [pmacct-discussion] nfacctd monthly accounting problem

2019-09-05 Thread Paolo Lucente
Hi Terry, Thanks for reporting this issue. Can you elaborate a bit more on the 'nfacctd does not start accounting for the new month'? It just stops accounting or it keeps accounting on the old month (ie. it seems like it does not flip the month)? Or some different behaviour? The more details

Re: [pmacct-discussion] Use nfacctd renormalize with tee plugin

2019-09-04 Thread Paolo Lucente
Hi Alexandre, Renormalization is a feature available only with collection (not teeing, since teeing does no or very minimal parsing). I am curious, are you using some sort of variable sampling rate or is it a constant? Perhaps you see where i am going to - if constant, you could perhaps factor

Re: [pmacct-discussion] MySQL SSL/TLS support

2019-08-20 Thread Paolo Lucente
Hi Scott, To confirm SSL/TLS connections to MySQL are not currently supported. While from a coding perspective it does not appear to be a big deal (matter of adding a mysql_set_ssl() call before mysql_real_connect()), i have no infrastructure to test this working properly. Can you help with

Re: [pmacct-discussion] pmbmpd and IPv6

2019-07-15 Thread Paolo Lucente
fic-bins' '--enable-bgp-bins' '--enable-bmp-bins' > '--enable-st-bins' > > Libs: > libpcap version 1.8.1 > PostgreSQL 19 > rabbimq-c 0.8.0 > rdkafka 0.11.3 > jansson 2.11 > > System: > Linux 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 > x8

Re: [pmacct-discussion] sfprobe not sending exports

2019-06-23 Thread Paolo Lucente
Hi Yang, Apparently you are exporting flows to [127.0.0.1]:6343 but you are listening with tcpdump on interface 'eno1'? Paolo On Sun, Jun 23, 2019 at 12:53:59AM -0700, Yang Yu wrote: > I tried to use pmacct to sample local interface and create sFlow > exports. From debug it looks like

Re: [pmacct-discussion] bgp_peer_src_as_map and pmacctd

2019-06-18 Thread Paolo Lucente
Ciao Simone, The config and maps all look good and, to be frank, it should all work. I admit it may be a better tested config with nfacctd/sfacctd (where it should just work) than pmacctd/uacctd. If you have interest in trying to make it work, i'd be more than happy to support you and

Re: [pmacct-discussion] pmacct on ppp interface

2019-05-29 Thread Paolo Lucente
Hi Alex, First thing first 1.6.1 is a release of almost 3 years ago, i can't support that - please upgrade to 1.7.3 or master code. That said i can confirm pmacctd/uacctd should support PPP-encapsulated traffic. Also, you may send me a trace of the NFLOG traffic (as captured by tcpdump) via

Re: [pmacct-discussion] nfacctd crash when using pre_tag_map

2019-05-28 Thread Paolo Lucente
Hi Felix, Thanks for getting in touch. Can you please get more data about the crash by following this section fo the QUICKSTART (i'd need an output of GDB 'bt'): https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2606-#L2635 You can follow up 1:1 so that we don't disturb everybody with

Re: [pmacct-discussion] Tolerating small packet flooding with pmacctd/nfacctd

2019-05-16 Thread Paolo Lucente
Hi Mikhail, For the export (pmacctd) part let me point you to Q7 of the FAQS doc: https://github.com/pmacct/pmacct/blob/master/FAQS#L71-#L101 Specifically PF_RING and ZeroMQ-based internal buffering (for this last part grep 'ZeroMQ' in the QUICKSTART document). For the collection (nfacctd)

[pmacct-discussion] pmacct 1.7.3 released !

2019-05-16 Thread Paolo Lucente
VERSION. 1.7.3 DESCRIPTION. pmacct is a small set of multi-purpose passive network monitoring tools. It can account, classify, aggregate, replicate and export forwarding-plane data, ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP and BMP; collect and correlate RPKI

Re: [pmacct-discussion] IPFIX Periodic Template

2019-04-24 Thread Paolo Lucente
Hi Rajesh, Since templates are sent out periodically (every 18th packet .. is a period :)), do you mean whether templates can be sent out on a time- based interval rather than a packet-based one? If so, currently this is not possible. The choice for each 18th packet was made originally in the

Re: [pmacct-discussion] AMQP exporter error

2019-04-10 Thread Paolo Lucente
Hi Grimur, The issue you did hit was solved last December and will be part of upcoming release 1.7.3: https://github.com/pmacct/pmacct/commit/56c498e60043c868131d64404f3c3a8f338ea406 Until the release will be available, you could use 1.7.3-rc2 or master GitHub code. It is currently left to

Re: [pmacct-discussion] nfacctd Dropping UDP. Buffer Receive Size?

2019-03-09 Thread Paolo Lucente
t. > > When load-balancing, particularly with SO_REUSEPORT, it would be nice to > allow them to communicate the template set to each other. Perhaps another > use for zeromq? > > Brian > > > > ‐‐‐ Original Message ‐‐‐ > On Sunday, February 24,

Re: [pmacct-discussion] Making an RPM out of source code

2019-03-07 Thread Paolo Lucente
Hi Edvinas, For a comprehensive list of files to install (consider some have conditionals, depending of configure time CL switches) you can follow: Makefile.am:all __DATA variables src/Makefile.am:sbin_PROGRAMS and bin_PROGRAMS (*) Paolo (*) I have just committed removing

Re: [pmacct-discussion] nfacctd Dropping UDP. Buffer Receive Size?

2019-03-06 Thread Paolo Lucente
r devices as of yet. > > Original Message ---- > On Feb 25, 2019, 9:28 AM, Paolo Lucente wrote: > > Hi Brian, > > Thanks very much for the nginx config, definitely something to add to > docs as a possible option. QN reads 'Queries Number' (inherited from the >

Re: [pmacct-discussion] nfacctd Dropping UDP. Buffer Receive Size?

2019-02-25 Thread Paolo Lucente
oxy_pass flow_upstreams; > #proxy_timeout 1s; > proxy_responses 0; > # must have user: root in main config > proxy_bind $remote_addr transparent; > error_log /var/log/nginx/stream-flow-err.log; > } > } > > > > >

Re: [pmacct-discussion] nfacctd Dropping UDP. Buffer Receive Size?

2019-02-24 Thread Paolo Lucente
Hi Brian, You are most probably looking for this: https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2644-#L2659 Should that not work, ie. too many input flows for the available resources, you have a couple load-balancing strategies possible: one is to configure a replicator (tee

Re: [pmacct-discussion] dst_as always 0

2019-01-29 Thread Paolo Lucente
Hi Grimur, Any chance you could try this again with some more current code than 1.7.0? Like 1.7.2 or, better, master code in GitHub? Just to make sure you are not hitting something which may have potentially been solved meanwhile (although it does not ring a bell). Also, can you please allow

Re: [pmacct-discussion] Aggregation and filtering

2019-01-28 Thread Paolo Lucente
Hi Christian, Thanks for the kind words, much appreciated, and the very interesting email. In line of principle i'd not be shy to say that you may want to use a different tool for that; but at the very same time i'd like to explore with you (and anybody interested in the conversation) whether

Re: [pmacct-discussion] sending one netflow stream to different NF_PROBE receivers

2019-01-15 Thread Paolo Lucente
Hi Edvinas, You could specify a single receiver in nfprobe. But you can transparent replicate this feed several destinations wish with a simple tee instance. You can get started how to configure a replicator by reading here: https://github.com/pmacct/pmacct/blob/1.7.2/QUICKSTART#L1751-#L1786

Re: [pmacct-discussion] pretag map line length limits

2019-01-10 Thread Paolo Lucente
Hi Inge, Always great to read from you. You are looking for the maps_row_len knob, by default 256 chars. Along with maps_entries it allows to specify the two key dimensions to alloc memory for the map. Paolo On Thu, Jan 10, 2019 at 02:54:09PM +, Inge Bjørnvall Arnesen wrote: > Hi, > >

Re: [pmacct-discussion] pmact to netflow collector

2019-01-06 Thread Paolo Lucente
> >> Maybe is there any quick start guide for first step ? > >> > >> Also - i tried to send all data to other analyzer (Solar Winds) and it > >> errored because of packets which comes with INTERFACE s index 0 (zero) > >> > >> [image: image.png] >

Re: [pmacct-discussion] Custom primitives with netflow

2019-01-06 Thread Paolo Lucente
, engineId, engineType >nfprobe_receiver: 10.40.6.6:17058 > " > > > > > On Wed, Dec 26, 2018 at 12:44 PM RAJESH KUMAR S.R > wrote: > > > Hi Paolo, > > > > Thanks for the fix and suggestion. I'll try tag and label primitives and >

Re: [pmacct-discussion] pmact to netflow collector

2019-01-04 Thread Paolo Lucente
t; no dropped packets occurs. > > > > But the problem exists. Still i see almost 10x lower traffic in > > NFSEN/NFDUMP analyzer than it's really is. What could be the case ? > > > > Thanks > > > > > > > > > > > > > > > >

Re: [pmacct-discussion] pmact to netflow collector

2019-01-03 Thread Paolo Lucente
it lower, but still a lot. > > also noticed strange log message: "INFO ( default/core ): short IPv4 > > packet read (36/38/frags). Snaplen issue ?" > > > > I going to try that PF_RING stuff. > > > > On Thu, Dec 20, 2018 at 10:08 PM Paolo Lucente wrote:

Re: [pmacct-discussion] memory limits - set up question

2018-12-25 Thread Paolo Lucente
Hi Sophie, Let me start with the bad news to conclude with the good ones. Unfortunately there is not a good way to size memory pools given a traffic figure and/or the amount of IP addresses monitored. It really depends on the traffic mix (that is, how big it is the matrix produced by your

Re: [pmacct-discussion] pmact to netflow collector

2018-12-20 Thread Paolo Lucente
gt; > > prod [root@netvpn001prpjay pmacct-1.7.2]# cat > > /proc/sys/net/core/[rw]mem_max > > 212992 > > 212992 > > > > I tried to set the pmacctd_pipe_size: to 20 and later to 212992. > > Seems the drops is still occuring. > > Tomorrow i will try t

Re: [pmacct-discussion] Custom primitives with netflow

2018-12-19 Thread Paolo Lucente
print/print ): *** Purging cache - START (PID: 4443) *** > SRC_IP DST_IP SRC_PORT DST_PORT > PROTOCOLTOS*dummy_byte* PACKETS BYTES > 172.24.1.197 239.255.255.25056940 > 1900 udp

Re: [pmacct-discussion] pmact to netflow collector

2018-12-18 Thread Paolo Lucente
RDec14 2865:35 > pmacctd: Netflow Probe Plugin [default_nfprobe] > > Before starting with your mentioned 'steroid' things, i would like to ask, > is't really worth to go to that kernel "things", or start with techniques > for example like sampling, or like Nikola rec

Re: [pmacct-discussion] Custom primitives with netflow

2018-12-16 Thread Paolo Lucente
Hi Rajesh, Thanks for pointing this out. I've committed some code to unlock field_type also for uacctd/pmacctd daemons precisely for the use case you mentioned. Here the details: https://github.com/pmacct/pmacct/commit/87ebf3a9f907c331f752c96a76ea247e77f99107 You can back port this patch to

Re: [pmacct-discussion] pmact to netflow collector

2018-12-16 Thread Paolo Lucente
.14.101:2101 >nfprobe_version: 9 >! nfprobe_engine: 1:1 >! nfprobe_timeouts: tcp=120:maxlife=3600 >! > ! networks_file: /path/to/networks.lst > > On Thu, Dec 13, 2018 at 4:32 AM Paolo Lucente wrote: > > > > > Hi Nikola, > > > > I see, makes sense. Than

  1   2   3   4   5   6   7   8   9   10   >