Re: [pmacct-discussion] effort to relicense pmacct from GPL to a BSD-style license

[Aaron Finney]

Hello,

I happily consent to the relicensing as proposed. Thank you for your
continued dedication to ensuring the long-term viability of the project.
What an amazing community.

Aaron

On Wed, Jan 8, 2020, 5:54 AM Job Snijders  wrote:

> Dear all,
>
> Summary: The pmacct project is looking to relicense its code from the
> current GPL license to a more liberal BSD-style license.
>
> A few weeks ago I had the pleasure to spend some face time with Paolo,
> which allowed for in-depth discussion about pmacct's current trajectory
> and bright future. We concluded it would be in the best interest of the
> pmacct project to attempt to relicense all code under a more permissive
> BSD-style license, for mainly two reasons:
>
> 1) Faced with our own mortality, it became clear that succession
>planning is of paramount importance for this project's continued
>success. We contemplated what happens in context of intellectual
>property rights should one of pmacct's contributors pass away, and
>realized potential heirs won't necessarily desire involvement in this
>open source project, potentially hampering changes to intellectual
>property policies in the project's future.
>
> 2) We suspect there are entities who violate the terms of pmacct's
>current GPL license, but at the same time we don't wish to litigate.
>Instead of getting infringers to change their behavior, relicensing
>the project could be another way to resolve the potential for
>conflict: we see benefits to removing rules we don't plan on
>enforcing anyway.
>
> Going forward, the preferred license under which we encourge people to
> contribute new work is a variant of the ISC license (also used by the
> OpenBSD project). The license template (to be used in file headers) can
> be found here:
>
> https://github.com/pmacct/pmacct/blob/master/LICENSE.template
>
> We need explicit approval from all contributors, and carefully keep
> track of those agreements. If a contributor doesn't agree or answer,
> we'll have to re-implement the contributed functionality or remove the
> contribution from the code base.
>
> REQUEST TO THE PMACCT CONTRIBUTOR COMMUNITY
> ---
>
> If you have contributed to the pmacct project (your name may be listed
> below), please consider a reply-all to this email expressing your
> explicit consent (or disapproval) to change the license governing your
> contributions to the pmacct project, to the following license:
>
> """
> Permission to use, copy, modify, and distribute this software for
> any purpose with or without fee is hereby granted, provided that the
> above copyright notice and this permission notice appear in all
> copies.
>
> THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
> WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
> WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
> AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
> DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA
> OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
> TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
> PERFORMANCE OF THIS SOFTWARE.
> """
>
> ---
>
> The next action in this process will be to individually follow up with
> all contributors who didn't respond to the above request.
>
> Once the relicensing effort has been completed, we'll tag the resulting
> code base as pmacct version 2.0.0 and celebrate! Pmacct has many great
> years ahead of itself; Paolo's enthusiasm to do so is evident in this
> interview video :-) https://www.youtube.com/watch?v=QqmOcMAtGDM
>
> Please feel free to raise any questions you may have on the
> pmacct-discussion@pmacct.net list, or privately with me and/or Paolo.
>
> Kind regards,
>
> Job Snijders
>
>
> DRAFT LIST OF KNOWN PMACCT AUTHORS (based on 'git shortlog -sen')
> =
>
> Commits  Author 
>2921  Paolo Lucente 
>  52  Marc Sune 
>  20  Corentin Néau 
>  17  Vincent Bernat 
>  14  Job Snijders 
>  12  Matthias Arnold 
>   9  Raphaël P. Barazzutti 
>   9  Claudio Ortega 
>   8  Jonas Jensen 
>   8  Matthias Arnold 
>   8  Tim LaBerge 
>   7  Jared Mauch 
>   7  vittoriofoschi 
>   7  Camilo Cardona 
>   5  Aaron Finney 
>   5  Vittorio Foschi 
>   4  vphatarp 
>   3  Alexander Brusilov 
>   3  Emil Palm 
>   3  Dusan Migra 
>   3  Dan Berger 

Re: [pmacct-discussion] New Plugin Pull Request

[Aaron Finney]

Hi Will,

Looks interesting! We accomplished this by using Kafka, but I can see where
having a json-via-udp might be useful as something more lightweight.

Aaron

On Tue, Jan 15, 2019 at 7:59 PM Will Hawkins  wrote:

> Hello great community!
>
> First, I wanted to say that I love the pmacct software -- it's been a
> huge boon to our research lab. Thank you all for developing and
> contributing!
>
> After some usage of the tool, we found that we needed some extra
> functionality that I couldn't find in the included plugins. Namely, we
> needed to send json-formatted output over UDP to a server that would
> perform further processing on the data. I hope that I didn't miss an
> existing plugin to perform such work!
>
> Finding none, I decided to implement a plugin to do just that. I named
> it jsonudp. I implemented it according to the directions in all the
> document (ie, copy existing plugins!). I relied heavily on the
> print_plugin and coded up something that works. I attempted to follow
> the existing code style and meet the project's standards.
>
> I submitted a pull request for the code to the project's repository on
> github. I'd love to hear feedback on the code and what you think I can
> improve if you think that it's a feature worth having in the main
> repository. If you don't think it's worthwhile, I'd love to hear that
> too!
>
> In any event, thank you for building and maintaining such a great tool.
>
> Will
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>


-- 

*Aaron Finney*Infrastructure Engineering | OpenX
888 East Walnut Street, 2nd Floor | Pasadena, CA 91101
o: +1 (626) 466-1141 x6035 | aaron.fin...@openx.com
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Trying to collect NetFlow data from a Cisco router

[Aaron Finney]

And also post the netflow config section from your router.

On Fri, Jul 13, 2018, 2:12 PM Kafui Akyea  wrote:

> I think you need to figure out if nfacctd is receiving any Netflow data at
> all and if it is aggregating it first.
>
> So from a terminal do this
>
> # *nfacctd -l 2100*
>
> where 2100 is the port to listen for netflow records. it will default to
> memory plugin. you should see a few messages printed out.
>
> make sure you see something like this
>
> *OK ( default_memory/memory ): waiting for data on: '/tmp/collect.pipe'*
>
> Then in another terminal do this
>
> # *pmacct -s -p /tmp/collect.pipe*
>
> It should display a nice table of the aggregates it has collected in
> memory for source host.
>
> If everything is ok with the above then you at least know it is getting
> and aggregating Netflow data.
>
> Kafui
>
>
> On Fri, Jul 13, 2018 at 1:00 PM, Tech Support 
> wrote:
>
>> All;
>>
>> I have a Cisco router running IOS and have NetFlow enabled. What I
>> want to do is simply collect that data using nfacctd. No biggie, just
>> collect the data containing the src and dst IP addresses and store it in
>> MySQL. I don’t even need to store it in MySQL, I could simply store it
>> pretty much anywhere. The problem is that no data is being collected. This
>> is my nfacctd.conf file:
>>
>>
>>
>> daemonize: true
>>
>> #debug: true
>>
>> networks_file: /usr/local/etc/pmacct/nfacctd.networks
>>
>> aggregate: src_host,dst_host
>>
>> nfacctd_port: 2100
>>
>> plugins: mysql
>>
>> sql_optimize_clauses: true
>>
>> sql_table_schema: /usr/local/etc/pmacct/pmacct-create-db_v1.sql
>>
>> sql_refresh_time: 60
>>
>> sql_history: 1d
>>
>> sql_history_roundoff: d
>>
>> sql_db: accounting
>>
>> sql_table: tkue_%Y_%m_%d
>>
>> sql_host: localhost
>>
>> sql_passwd: root
>>
>> sql_user: 
>>
>>
>>
>> So, my question is, what am I missing? What am I doing wrong? Any insight
>> at all would be greatly appreciated.
>>
>> Thanks in Advance;
>>
>> John V.
>>
>> ___
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists
>>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Geoloc lat/lon?

[Aaron Finney]

Answering myself...I went ahead and added it to my fork today. :)

If there's any interest in a PR to merge back, I'd be happy to submit it.

Cheers,

Aaron

On Mon, Jun 4, 2018 at 6:20 PM, Aaron Finney  wrote:

> Hi Paolo/all,
>
> Has anyone done (or planned) any work around adding lat/lon data from
> geoipv2 as export primitives?
>
> As I understand it, only the country is currently available for exporting.
> We are adding lat/lon after the fact via etl, but it would be cleaner and
> more efficient to add it at the collector.
>
> Aaron
>



-- 

*Aaron Finney*Network Engineer | OpenX
888 East Walnut Street, 2nd Floor | Pasadena, CA 91101
o: +1 (626) 466-1141 x6035 | aaron.fin...@openx.com
*Advertising Age Best Places to Work
<http://openx.com/press-releases/openx-named-as-one-of-advertising-ages-top-fifty-best-places-to-work-for-2015/>*
*Deloitte's Technology Fast 500™
<http://openx.com/press-releases/openx-ranked-3rd-fastest-growing-software-company-north-america-5th-fastest-overall-deloittes-2013-technology-fast-500/>*
www.openx.com   <http://www.openx.com/>|  Twitter
<http://twitter.com/openx>|  Facebook   <http://www.facebook.com/OpenX>|
LinkedIn   <http://www.linkedin.com/company/openx/products>|  YouTube
<http://www.youtube.com/user/openxvideos>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Geoloc lat/lon?

[Aaron Finney]

Hi Paolo/all,

Has anyone done (or planned) any work around adding lat/lon data from
geoipv2 as export primitives?

As I understand it, only the country is currently available for exporting.
We are adding lat/lon after the fact via etl, but it would be cleaner and
more efficient to add it at the collector.

Aaron
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] pmacct 1.7.1 released !

[Aaron Finney]

Congratulations, Paolo, these are really great updates! Cheers, and thanks
again for all of your hard work for the community.

Aaron

On Sun, May 6, 2018, 6:45 AM Paolo Lucente  wrote:

> VERSION.
> 1.7.1
>
>
> DESCRIPTION.
> pmacct is a small set of multi-purpose passive network monitoring tools. It
> can account, classify, aggregate, replicate and export forwarding-plane
> data,
> ie. IPv4 and IPv6 traffic; collect and correlate control-plane data via BGP
> and BMP; collect infrastructure data via Streaming Telemetry. Each
> component
> works both as a standalone daemon and as a thread of execution for
> correlation
> purposes (ie. enrich NetFlow with BGP data).
>
> A pluggable architecture allows to store collected forwarding-plane data
> into
> memory tables, RDBMS (MySQL, PostgreSQL, SQLite), noSQL databases (MongoDB,
> BerkeleyDB), AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
> pmacct offers customizable historical data breakdown, data enrichments like
> BGP and IGP correlation and GeoIP lookups, filtering, tagging and triggers.
> Libpcap, Linux Netlink/NFLOG, sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX
> are
> all supported as inputs for forwarding-plane data. Replication of incoming
> NetFlow, IPFIX and sFlow datagrams is also available. Statistics can be
> easily exported to time-series databases like ElasticSearch and InfluxDB
> and traditional tools Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc.
>
> Control-plane and infrastructure data, collected via BGP, BMP and Streaming
> Telemetry, can be all logged real-time or dumped at regular time intervals
> to AMQP (RabbitMQ) and Kafka message exchanges and flat-files.
>
>
> HOMEPAGE.
> http://www.pmacct.net/
>
>
> DOWNLOAD.
> http://www.pmacct.net/pmacct-1.7.1.tar.gz
>
>
> CHANGELOG.
> + pmbgpd: introduced a BGP x-connect feature meant to map BGP peers
>   (ie. PE routers) to BGP collectors (ie. nfacctd, sfacctd) via a
>   standalone BGP daemon (pmbgpd). The aim is to facilitate operations
>   when re-sizing/re-balancing the collection infrastructure without
>   impacting (ie. re-configuring) BGP peers. bgp_daemon_xconnect_map
>   expects full pathname to a file where cross-connects are defined;
>   mapping works only against the IP source address and not the BGP
>   Router ID, only 1:1 relationships can be formed (ie. this is about
>   cross-connecting, not replication) and only one session per BGP
>   peer is supported (ie. multiple BGP agents are running on the same
>   IP address or NAT traversal scenarios are not supported [yet]).
>   A sample map is provided in 'examples/bgp_xconnects.map.example'.
> + pmbgpd: introduced a BGP Looking Glass server allowing to perform
>   queries, ie. lookup of IP addresses/prefixes or get the list of BGP
>   peers, against available BGP RIBs. The server is asyncronous and
>   uses ZeroMQ as transport layer to serve incoming queries. Sample
>   C/Python LG clients are available in 'examples/lg'. A sample LG
>   server config is available in QUICKSTART. Request/Reply Looking
>   Glass formats are documented in 'docs/LOOKING_GLASS_FORMAT'.
> + pmacctd: a single daemon can now listen for traffic on multiple
>   interfaces via a polling mechanism. This can be configured via a
>   pcap_interfaces_map feature (interface/pcap_interface can still be
>   used for backward compatiblity to listen on a single interface). The
>   map allows to define also ifindex mapping and capturing direction on
>   a per-interface basis. The map can be reloaded at runtime via a USR2
>   signal and a sample map is in examples/pcap_interfaces.map.example.
> + Kafka plugin: dynamic partitioning via kafka_partition_dynamic and
>   kafka_partition_key knobs is introduced. The Kafka topic can contain
>   variables, ie. $peer_src_ip, $src_host, $dst_port, $tag, etc., which
>   are all computed when data is purged to the backend. This feature is
>   in addition to the existing kafka_partition feature which allows to
>   rely on the built-in Kafka partitioning to assign data statically to
>   one partition or rely dynamically on the default partitioner. The
>   feature is courtesy by Corentin Neau / Codethink ( @weyfonk ).
> + Introduced rfc3339 formatted timestamps: in logs, ie. UTC timezone
>   represented as -MM-ddTHH:mm:ss(.ss)Z; for aggregation primitives
>   the timestamps_rfc3339 knob can be used to enable this feature (left
>   disabled by default for backward compatibility).
> + timestamps_utc: new knob to decode timestamps to UTC timezone even
>   if the Operating System is set to a different timezone. On the goods
>   of running a system set to UTC please read Q18 of FAQS.
> + sfacctd: implemented mpls_label_top, mpls_label_bottom and
>   mpls_stack_depth primitives decoded from sFlow flow sample headers.
>   Thanks to David Barroso ( @dbarrosop ) for his support.
> + nfacctd: added support for IEs 130 (exporterIPv4Address) and 131
>   (exporterIPv6Address) when passed as part of NetFlow v9/IPFIX
>   

Re: [pmacct-discussion] Capture DNS domain and HTTP destinations from incoming netflow packets

[Aaron Finney]

That's pretty vague. The info you're asking about is not exported via
netflow, so you'll need some other process (i.e. ETLs, or stream processing
if your pipeline's resources can handle it) to retrieve/match the
additional data to your flow records - e.g. reverse DNS and mining
aggregated HTTP server logs using the fields you're exporting to match
server transactions with flows.


On Mon, Feb 19, 2018 at 9:26 PM, sadan sohan  wrote:

> Hi,
>
> We have a use case to fetch the DNS domain and the HTTP destination
> requested by the incoming packets from the source host. Can somebody help
> here ?
>
> Thanks & Regards,
> Sadan
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Load balancing nfacctd

[Aaron Finney]

I would think you'd just peer every collector to every device in a full
mesh, unless i'm missing something obvious. Having peering sessions going
up and down constantly between the network devices and one of n collectors
behind a load balancer does not seem feasible.

On Tue, Sep 5, 2017 at 10:46 AM, Paul Mabey <p...@mabey.net> wrote:

> Right…..routers export flow to the VIP, as well as “think” they are BGPing
> with the VIP. The LB then has a static rule that forwards both BGP/flow to
> the correct collector. The goal being that if the collector IP changes for
> some reason, I don’t have to go touch the router configs.
>
>
> On Sep 5, 2017, at 11:35 AM, Aaron Finney <aaron.fin...@openx.com> wrote:
>
> I'm not sure I follow - do you mean setting up BGP peering of the
> collectors to your source devices using the collector VIP as the neighbor
> address?
>
> On Sep 5, 2017 10:11 AM, "Paul Mabey" <p...@mabey.net> wrote:
>
>> Has anyone had success is pushing BGP sessions through an LB along with
>> netflow? Interested in the solution below but would like to have BGP
>> aligned with netflow as well.
>>
>> On Sep 4, 2017, at 9:48 AM, Aaron Finney <aaron.fin...@openx.com> wrote:
>>
>> Great to hear, nice work!
>>
>> Aaron
>>
>> On Sep 4, 2017 1:55 AM, "Yann Belin" <y.belin...@gmail.com> wrote:
>>
>> Hi all,
>>
>> Updating on this, in case someone is interested.
>>
>> Consul was indeed the way to go:
>>
>> * nginx is doing the actual UDP load balancing, based on source IP
>> hash (to optimize aggregation).
>> * consul keeps track of nfacctd collectors, of their health, and of
>> the health of their dependencies (rabbitmq in my case).
>> * consul-template uses the information provided by consul (servers +
>> health) to generate nginx configuration files, and reloads nginx
>> service if needed; if a collector becomes unhealthy (e.g. rabbitmq
>> crashes), it will be removed from nginx configuration and will stop
>> receiving flows.
>>
>> The great thing with consul is that you can write your own checks. For
>> now my checks are relatively basic (process + port binding checks) but
>> I am working on a more advanced one for rabbitmq (e.g. queue length /
>> ram usage). I'm still thinking about more advanced ways to check
>> nfacctd health, if anyone has a suggestion.
>>
>> Cheers,
>>
>> Yann
>>
>>
>> On Mon, Aug 21, 2017 at 4:02 PM, Aaron Finney <aaron.fin...@openx.com>
>> wrote:
>> > Hi Yann
>> >
>> > We use Consul for this, it works very well.
>> >
>> > https://www.consul.io
>> >
>> >
>> > Aaron
>> >
>> >
>> >
>> > On Aug 21, 2017 6:44 AM, "Yann Belin" <y.belin...@gmail.com> wrote:
>> >
>> > Hello,
>> >
>> > I have been looking into solutions to achieve reliable load balancing
>> > of my incoming flows across multiple nfacctd servers / daemons.
>> >
>> > Basic load balancing is relatively easy (see Nginx configuration
>> > below), but *reliable* load balancing (only sending flows to servers
>> > that have a running nfacctd daemon) is quite more complicated. For
>> > instance, Nginx normally monitors UDP responses from the remote
>> > servers to determine if those servers are health, but this approach
>> > will not work in the case of netflow or ipfix.
>> >
>> > Did anybody already managed to solve this? Or has a suggestion perhaps?
>> >
>> > Thanks in advance!
>> >
>> > *-*-*-*-*-*-*-*
>> > stream {
>> > upstream ipfix_traffic {
>> > hash $binary_remote_addr;
>> > server 10.20.10.10:9055;
>> > server 10.20.10.20:9055;
>> > }
>> >
>> > server {
>> > listen 9055 udp;
>> > proxy_responses 0;
>> > proxy_pass ipfix_traffic;
>> > proxy_bind $remote_addr transparent;
>> > error_log /var/log/nginx/ipfix_traffic.error.log;
>> > }
>> > }
>> > *-*-*-*-*-*-*-*
>> >
>> > Kind regards,
>> >
>> > Yann
>> >
>> > ___
>> > pmacct-discussion mailing list
>> > http://www.pmacct.net/#mailinglists
>> >
>> >
>> >
>> > ___
>> > pmacct-discussion mailing list
>> > http://www.pmacct.net

Re: [pmacct-discussion] Load balancing nfacctd

[Aaron Finney]

I'm not sure I follow - do you mean setting up BGP peering of the
collectors to your source devices using the collector VIP as the neighbor
address?

On Sep 5, 2017 10:11 AM, "Paul Mabey" <p...@mabey.net> wrote:

> Has anyone had success is pushing BGP sessions through an LB along with
> netflow? Interested in the solution below but would like to have BGP
> aligned with netflow as well.
>
> On Sep 4, 2017, at 9:48 AM, Aaron Finney <aaron.fin...@openx.com> wrote:
>
> Great to hear, nice work!
>
> Aaron
>
> On Sep 4, 2017 1:55 AM, "Yann Belin" <y.belin...@gmail.com> wrote:
>
> Hi all,
>
> Updating on this, in case someone is interested.
>
> Consul was indeed the way to go:
>
> * nginx is doing the actual UDP load balancing, based on source IP
> hash (to optimize aggregation).
> * consul keeps track of nfacctd collectors, of their health, and of
> the health of their dependencies (rabbitmq in my case).
> * consul-template uses the information provided by consul (servers +
> health) to generate nginx configuration files, and reloads nginx
> service if needed; if a collector becomes unhealthy (e.g. rabbitmq
> crashes), it will be removed from nginx configuration and will stop
> receiving flows.
>
> The great thing with consul is that you can write your own checks. For
> now my checks are relatively basic (process + port binding checks) but
> I am working on a more advanced one for rabbitmq (e.g. queue length /
> ram usage). I'm still thinking about more advanced ways to check
> nfacctd health, if anyone has a suggestion.
>
> Cheers,
>
> Yann
>
>
> On Mon, Aug 21, 2017 at 4:02 PM, Aaron Finney <aaron.fin...@openx.com>
> wrote:
> > Hi Yann
> >
> > We use Consul for this, it works very well.
> >
> > https://www.consul.io
> >
> >
> > Aaron
> >
> >
> >
> > On Aug 21, 2017 6:44 AM, "Yann Belin" <y.belin...@gmail.com> wrote:
> >
> > Hello,
> >
> > I have been looking into solutions to achieve reliable load balancing
> > of my incoming flows across multiple nfacctd servers / daemons.
> >
> > Basic load balancing is relatively easy (see Nginx configuration
> > below), but *reliable* load balancing (only sending flows to servers
> > that have a running nfacctd daemon) is quite more complicated. For
> > instance, Nginx normally monitors UDP responses from the remote
> > servers to determine if those servers are health, but this approach
> > will not work in the case of netflow or ipfix.
> >
> > Did anybody already managed to solve this? Or has a suggestion perhaps?
> >
> > Thanks in advance!
> >
> > *-*-*-*-*-*-*-*
> > stream {
> > upstream ipfix_traffic {
> > hash $binary_remote_addr;
> > server 10.20.10.10:9055;
> > server 10.20.10.20:9055;
> > }
> >
> > server {
> > listen 9055 udp;
> > proxy_responses 0;
> > proxy_pass ipfix_traffic;
> > proxy_bind $remote_addr transparent;
> > error_log /var/log/nginx/ipfix_traffic.error.log;
> > }
> > }
> > *-*-*-*-*-*-*-*
> >
> > Kind regards,
> >
> > Yann
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> >
> >
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Load balancing nfacctd

[Aaron Finney]

Great to hear, nice work!

Aaron

On Sep 4, 2017 1:55 AM, "Yann Belin" <y.belin...@gmail.com> wrote:

Hi all,

Updating on this, in case someone is interested.

Consul was indeed the way to go:

* nginx is doing the actual UDP load balancing, based on source IP
hash (to optimize aggregation).
* consul keeps track of nfacctd collectors, of their health, and of
the health of their dependencies (rabbitmq in my case).
* consul-template uses the information provided by consul (servers +
health) to generate nginx configuration files, and reloads nginx
service if needed; if a collector becomes unhealthy (e.g. rabbitmq
crashes), it will be removed from nginx configuration and will stop
receiving flows.

The great thing with consul is that you can write your own checks. For
now my checks are relatively basic (process + port binding checks) but
I am working on a more advanced one for rabbitmq (e.g. queue length /
ram usage). I'm still thinking about more advanced ways to check
nfacctd health, if anyone has a suggestion.

Cheers,

Yann


On Mon, Aug 21, 2017 at 4:02 PM, Aaron Finney <aaron.fin...@openx.com>
wrote:
> Hi Yann
>
> We use Consul for this, it works very well.
>
> https://www.consul.io
>
>
> Aaron
>
>
>
> On Aug 21, 2017 6:44 AM, "Yann Belin" <y.belin...@gmail.com> wrote:
>
> Hello,
>
> I have been looking into solutions to achieve reliable load balancing
> of my incoming flows across multiple nfacctd servers / daemons.
>
> Basic load balancing is relatively easy (see Nginx configuration
> below), but *reliable* load balancing (only sending flows to servers
> that have a running nfacctd daemon) is quite more complicated. For
> instance, Nginx normally monitors UDP responses from the remote
> servers to determine if those servers are health, but this approach
> will not work in the case of netflow or ipfix.
>
> Did anybody already managed to solve this? Or has a suggestion perhaps?
>
> Thanks in advance!
>
> *-*-*-*-*-*-*-*
> stream {
> upstream ipfix_traffic {
> hash $binary_remote_addr;
> server 10.20.10.10:9055;
> server 10.20.10.20:9055;
> }
>
> server {
> listen 9055 udp;
> proxy_responses 0;
> proxy_pass ipfix_traffic;
> proxy_bind $remote_addr transparent;
> error_log /var/log/nginx/ipfix_traffic.error.log;
> }
> }
> *-*-*-*-*-*-*-*
>
> Kind regards,
>
> Yann
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Load balancing nfacctd

[Aaron Finney]

Hi Yann

We use Consul for this, it works very well.

https://www.consul.io


Aaron



On Aug 21, 2017 6:44 AM, "Yann Belin"  wrote:

Hello,

I have been looking into solutions to achieve reliable load balancing
of my incoming flows across multiple nfacctd servers / daemons.

Basic load balancing is relatively easy (see Nginx configuration
below), but *reliable* load balancing (only sending flows to servers
that have a running nfacctd daemon) is quite more complicated. For
instance, Nginx normally monitors UDP responses from the remote
servers to determine if those servers are health, but this approach
will not work in the case of netflow or ipfix.

Did anybody already managed to solve this? Or has a suggestion perhaps?

Thanks in advance!

*-*-*-*-*-*-*-*
stream {
upstream ipfix_traffic {
hash $binary_remote_addr;
server 10.20.10.10:9055;
server 10.20.10.20:9055;
}

server {
listen 9055 udp;
proxy_responses 0;
proxy_pass ipfix_traffic;
proxy_bind $remote_addr transparent;
error_log /var/log/nginx/ipfix_traffic.error.log;
}
}
*-*-*-*-*-*-*-*

Kind regards,

Yann

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Issues with Kafka/Avro sending schema to Kafka topic

[Aaron Finney]

Hi Paolo,

That's what I get for becoming our internal package maintainer for pmacct
by default - updating it now! :D

One quick sanity check regarding output to Kafka - would setting
kafka_refresh_time to 1 add significant overhead to nfacctd, e.g. related
to buffer flushing process/etc?

My goal is to produce something closer to a live stream, but with
micro-flows aggregated to 1s granularity, then use Spark to do additional
transformations/aggregations against local data along with basic anomaly
detection.

Thanks!

Aaron


On Fri, Feb 17, 2017 at 5:58 AM, Paolo Lucente <pa...@pmacct.net> wrote:

>
> Hi Aaron,
>
> The feature is post 1.6.1. Can you please switch to master code on GitHub?
>
> Thanks,
> Paolo
>
> On Thu, Feb 16, 2017 at 10:44:23AM -0800, Aaron Finney wrote:
> > Hi Paolo/all,
> >
> > I've been unable to get nfacctd to send the Avro schema to a Kafka topic
> -
> > I receive the following message when starting nfacctd:
> >
> > WARN: [/etc/nfacctd.conf:14] Unknown key: kafka_avro_schema_topic.
> Ignored.
> > WARN: [/etc/nfacctd.conf:15] Unknown key: kafka_avro_schema_refresh_
> time.
> > Ignored.
> >
> > Kafka/Avro are otherwise working; the process successfully writes the
> > schema to a local file specified in avro_schema_output_file, but I'd like
> > to have it send to a schema topic in the cluster itself if possible.
> > Version info is below.
> >
> > Thanks!
> >
> > Aaron
> >
> >
> > INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.6.1
> > (20161001-00+c5)
> > INFO ( default/core ):  '--enable-kafka' '--enable-jansson'
> '--enable-avro'
> > 'KAFKA_LIBS=-L/usr/lib64/ -lrdkafka' 'AVRO_CFLAGS=-I/usr/include/avro'
> > 'AVRO_LIBS=-L/usr/lib -lavro'
>
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] nfacctd aggregate_filters not working correctly when defined in the same config file

[Aaron Finney]

Hi Paolo,

It's version 1.6.1:

NetFlow Accounting Daemon, nfacctd 1.6.1 (20161001-00+c5).

Thanks,

Aaron



On Sat, Jan 21, 2017 at 3:57 AM, Paolo Lucente <pa...@pmacct.net> wrote:

>
> Hi Aaron,
>
> Interesting. Can you say what version is this? And if anything before
> 1.6.1 or (much preferrably) master code on GitHub - can you please try
> and confirm you experience the same with any of these?
>
> Paolo
>
> On Fri, Jan 20, 2017 at 07:03:15PM -0800, Aaron Finney wrote:
> > Hello all,
> >
> > I promise I searched the archives exhaustively first...
> >
> > We are trying to separate external ingress/egress traffic using
> > aggregate_filter (config below), but it's not working as expected. When
> we
> > only have one of the sections active and  (xv_ext_in OR xv_ext_out) and
> > comment out the other, we get exactly the data we expect - only external
> > data and either to/from our networks. When we activate both in the
> config,
> > we end up with a mix of both, but not exactly the same data. Any help
> would
> > be greatly appreciated - thanks!
> >
> >
> > Config:
> >
> > daemonize: false
> > nfacctd_port: 2100
> > nfacctd_net: netflow
> > plugins: amqp[xv_ext_in], amqp[xv_ext_out]
> > !
> > amqp_exchange[xv_ext_in]: netflow-in
> > amqp_exchange_type[xv_ext_in]: direct
> > amqp_host[xv_ext_in]: localhost
> > amqp_refresh_time[xv_ext_in]: 5
> > amqp_user[xv_ext_in]: username
> > amqp_passwd[xv_ext_in]: password
> > aggregate[xv_ext_in]: peer_src_ip, src_as, dst_as, src_host, dst_host,
> > src_port, dst_port, in_iface, out_iface, proto, sampling_rate
> > aggregate_filter[xv_ext_in]: not (src net (173.241.240.0/20 or
> 69.6.80.0/20
> > or 199.26.53.0/24 or 209.182.128.0/19)) and not net 10.0.0.0/8
> > amqp_routing_key[xv_ext_in]: xv_in
> > !
> > amqp_exchange[xv_ext_out]: netflow-out
> > amqp_exchange_type[xv_ext_out]: direct
> > amqp_host[xv_ext_out]: localhost
> > amqp_refresh_time[xv_ext_out]: 5
> > amqp_user[xv_ext_out]: username
> > amqp_passwd[xv_ext_out]: password
> > aggregate[xv_ext_out]: peer_src_ip, src_as, dst_as, src_host, dst_host,
> > src_port, dst_port, in_iface, out_iface, proto, sampling_rate
> > aggregate_filter[xv_ext_out]: not (dst net (173.241.240.0/20 or
> 69.6.80.0/20
> > or 199.26.53.0/24 or 209.182.128.0/19)) and not net 10.0.0.0/8
> > amqp_routing_key[xv_ext_out]: xv_out
>
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>



-- 

*Aaron Finney*Network Engineer | OpenX
888 East Walnut Street, 2nd Floor | Pasadena, CA 91101
o: +1 (626) 466-1141 x6035 | aaron.fin...@openx.com
*Advertising Age Best Places to Work
<http://openx.com/press-releases/openx-named-as-one-of-advertising-ages-top-fifty-best-places-to-work-for-2015/>*
*Deloitte's Technology Fast 500™
<http://openx.com/press-releases/openx-ranked-3rd-fastest-growing-software-company-north-america-5th-fastest-overall-deloittes-2013-technology-fast-500/>*
www.openx.com   <http://www.openx.com/>|  Twitter
<http://twitter.com/openx>|  Facebook   <http://www.facebook.com/OpenX>|
LinkedIn   <http://www.linkedin.com/company/openx/products>|  YouTube
<http://www.youtube.com/user/openxvideos>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists