Re: [pmacct-discussion] pmacct 1.7.4 released !
Hi ! oom-killer just killed my instance, so «same here»... Some infos: # src/nfacctd -V NetFlow Accounting Daemon, nfacctd 1.7.4-git (20191126-01+c5) Arguments: '--enable-jansson' '--enable-64bit' '--enable-zmq' '--enable-pgsql' '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins' Libs: libpcap version 1.8.1 PostgreSQL 90615 jansson 2.9 ZeroMQ 4.2.1 System: Linux 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64 Compiler: gcc 6.3.0 > Le 7 janv. 2020 à 09:06, Fabien VINCENT a écrit : > > Hi Paolo, > Thanks for this release and enhancements ! > Since upgrade, I see a huge memory leak without any reason. > https://github.com/pmacct/pmacct/issues/356 > The only changes is I dist-upgrade the machine itself as installed from > source 1.7.4 release. > I use print plugin on my side on nfacctd processes. > Please let me know how I can troubleshoot this, I will rollback to 1.7.3 > temporarly > Regards, > > Le 31-12-2019 17:48, Paolo Lucente a écrit : >> VERSION. >> 1.7.4 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] nfacctd problems when using pgsql plugin with as_path and sql_use_copy
Hi guys, hi Paolo,
I resurrect this old thread, but I guess something should be updated in the
code about this.
The default sql_delimiter is ",".
But the "," character is present in recorded AS paths for non-atomic aggregates
and looks like this:
AS1 AS2 AS3 {ASX,ASY,ASZ}
So this causes the whole update to be rejected.
As the default delimiter isn't compatible with the information to record, I
suggest it should be changed to something else (here we have good results with
# for now).
regards,
Olivier Benghozi
Wifirst
> Le 4 déc. 2014 à 19:17, THE MIGHTY VEXORG <vex...@gmail.com> a écrit :
>
> That did it! My apologies for not seeing the sql_delimiter directive
> while reading through the documentation. I will try to rtfm better in
> the future.
>
> Pat
>
> On 12/03/2014 11:21 PM, Paolo Lucente wrote:
>> Hi Pat,
>>
>> Give a try changing the COPY delimiter using the sql_delimiter directive.
>> You can read more about the directive in CONFIG-KEYS. Keep me posted.
>>
>> Cheers,
>> Paolo
>>
>> On Wed, Dec 03, 2014 at 01:30:20AM -0800, THE MIGHTY VEXORG wrote:
>>> Using the pgsql plugin with both sql_use_copy and as_path as part of the
>>> aggregate appears to cause errors during the copy. It seems that there
>>> is an occasional extra comma in the as_path which then causes the copy
>>> to fail. This only appears to happen with the pgsql plugin when both
>>> sql_use_copy is true and as_path is part of the aggregate.
>>>
>>> relevant parts of my nfacctd.conf:
>>> aggregate[bgp_hourly]: src_as, dst_as, peer_src_ip, peer_dst_ip,
>>> as_path, std_comm, local_pref, med, in_iface
>>> aggregate[flat]: src_as, dst_as, peer_src_ip, peer_dst_ip, as_path,
>>> std_comm, local_pref, med, in_iface
>>> plugins: print[flat], pgsql[bgp_hourly]
>>> sql_use_copy[bgp_hourly]: true
>>> print_output[flat]: csv
>>>
>>> "flat" was used to print to csv file to help troubleshoot this problem.
>>> When the error occurs, the postgresql log has the following:
>>>
>>> 2014-12-02 19:50:51 EST ERROR: 22P04: extra data after last expected column
>>> 2014-12-02 19:50:51 EST CONTEXT: COPY acct_bgp_hourly_20141202_1900,
>>> line 28889: "2014-12-02 19:50:51,2014-12-02 19:00:00,0,5,64520,577:55
>>> 577:5606 577:11100 577:22111,174 39743 {322..."
>>> 2014-12-02 19:50:51 EST LOCATION: NextCopyFrom, copy.c:2376
>>> 2014-12-02 19:50:51 EST STATEMENT: COPY acct_bgp_hourly_20141202_1900
>>> (stamp_updated, stamp_inserted, as_src, iface_in, as_dst, comms,
>>> as_path, local_pref, med, peer_ip_src, peer_ip_dst, packets, bytes) FROM
>>> STDIN DELIMITER ','
>>>
>>> and the corresponding line from the csv file:
>>>
>>> SRC_AS,DST_AS,COMMS,AS_PATH,PREF,MED,PEER_SRC_IP,PEER_DST_IP,IN_IFACE,PACKETS,BYTES
>>> 0,64520,577:55_577:5606_577:11100_577:22111,174_39743_{3223,64520},110,0,64.230.195.154,64.230.195.154,5,4,160
>>>
>>> It looks like the extra comma in the as_path field matches the
>>> delimiter, making postgresql think there is an extra field. Is there a
>>> way around this other than setting sql_use_copy to false or not
>>> collecting as_path data when using the pgsql plugin? Is there any other
>>> information I should collect to better troubleshoot this problem? In my
>>> setup, setting sql_use_copy to false makes it hard for nfacct/pgsql to
>>> be able to keep up with the incoming flows, so I would prefer to avoid
>>> that option if possible.
>>>
>>> Pat
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] mySQL ATON/NTOA and pmacct performance
Hi guys, I just wanted to summarize that: It looks like a fine way to address IPs in a DB seems to be the pgsql way. Fast, and you can easily select data using subnets without relying on regexp. While in mysql you either obtain a int value difficult to handle, or a string very slow to search (choose between plague and cholera). For some reasons Scott is migrating from something that works to something that doesn't match the needs and causes problems. So, the DB engine choice was obviously not decided according to the needs. So the reasons were wrong. Rollback. Problem solved: there's no problem :) regards, Olivier Le 18 sept. 2014 à 12:04, itria30...@itri.org.tw a écrit : Hi Paolo and Scott, Your discussion pointed out an important issue - what is the proper design when it comes to high frequently IP-based queries? Inspired by this thread, my idea is to use postgresql and count on its cidr/inet function. (I am using mysql and store ipv4 as string(15)) This way the IP address is readable and processing IP address is also efficient. It seems worthy to try. I will try this approach later. Any comment is appreciated. -Original Message- From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf Of Paolo Lucente Sent: Thursday, September 18, 2014 8:44 AM To: Scott Pettit - Vorco Cc: pmacct-discussion@pmacct.net Subject: Re: [pmacct-discussion] mySQL ATON/NTOA and pmacct performance Hi Scott, Great. Let me know how your testing goes. Should you hit any bugs or gaps, feel free to contact me privately to solve them. Then we can summarize on the list. Cheers, Paolo On Wed, Sep 17, 2014 at 04:35:05AM +, Scott Pettit - Vorco wrote: In MySQL (any SQL actually), string searches are computationally very expensive when compared with integers. If you’re dealing with a relatively small number of rows then the difference isn’t a problem. It’s like asking “show me all the rows where the number is between 1 and 1000” versus “show me all the rows that look like something between one and one thousand in words”. The second statement is the same question but requires a lot more processing because the database has to think about every number as a word. It might only take fractions of a second longer for one row, but multiply that by 300 million rows (which is what I’m trying to analyse) and the difference is huge. PostgreSQL handles this really well as it has support for IP’s as a column type which presumably converts to integers in the back end. I will try pmacct with ipv4 on my test collector with sql_num_hosts and if it works well then having the same for inet6_aton would be great. -Scott On 17/09/14 16:03, itria30...@itri.org.tw itria30...@itri.org.tw wrote: Hi Scott and Paolo, May I ask a question per your good experience? What is the performance difference between using string ipv4 and integer ipv4? And potential impact? I have a PC running sfacctd that collects 2 giga-switches. And it collects around 5M records into mysql per day. I am satisfied with the default string ipv4 for its convenience of being readable. And I would like to know the performance drawback. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Char vs varchar vs text (Postgresql)
Hi all, I just wanted to share our experience with pmacct pushing netflow stuff to a Postgresql DB (9.2). By default the pmacct pgsql schemas use char(xxx) fields (for BGP stuff like AS, AS pathes, etc). After some readings, and some tests, it looks like replacing every char(xx) by text type makes queries really faster; maybe it just makes them use much less memory, claiming less workspace, avoiding to write anything to disk. Anyway, it's now quite faster like that (it appears that internally, char, varchar text are stored the same way in pgsql, just padding char fields with whitespaces). Just my 2 cents. regards, Olivier ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] New features proposal discussion: BGP sync holdtime AS prepend compression
Hi all, I was thinking of some features that may be interesting (at least I think they may be interesting for us, so they may be also for some other people). We use nfacctd with a bgp session to put together netflow and bgp information to a database, only for inbound traffic. 1: what about a feature to prevent any db insertion before a BGP session established within the first minutes of nfacctd life? That would prevent empty AS/BGP related fields in the database. Something like a ldp-synchronization hold-time xx seconds that junos/cisco/others implement for their IGP. 2: what about a feature to shorten or compress the stuff inserted in src_as_path, regarding prepended ASes? Obviously incoming traffic is not linked to the as_path to the source, but it's still interesting to see which kind of link you use to see it. In those conditions, prepended ASes are meaningless in this direction; what about removing prependings in this field ? Or what about replacing prependings by something like AS[55] for a 55 times prepended AS (something that would be easy to remove at the select)? Feel free to comment. -- Olivier Benghozi Wifirst ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] BGP+VPNv4 Address Family Netflow v5
Hi, I'm trying to implement a working config of nfacctd with bgp thread to push some netflow stuff into a PostgreSQL DB. I have some Netflow v5 streams (that is, transporting few information) and an i-MP-BGP reflected session, which sends all the routing table in the VPNv4 address family (nothing in the IPv4 unicast AFI/SAFI), since the traffic is in a VRF. My problem is that I need to have those VPNv4 routes (whose the Route Distinguisher is known) to be successfully matched with the netflow records by nfacctd, to fill all the fancy AS information (as_path, src_as, and so on). In my conf I use things like bgp_agent_map (to match all netflow speakers to my unique BGP session router_id, with bgp_ip=the.rou.ter.idip= 0.0.0.0/0), bgp_peer_src_as_type: bgp, and bgp_src_as_path_type: bgp. But src_as, as_path... sadly stay at 0. Is this working mode possible? Is there something obviously wrong in my parameters? Thanks by advance . -- Olivier Benghozi Wifirst ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
