Re: [pmacct-discussion] Centralizing data from multiple nfacct collectors

2017-01-05 Thread Alex 
I have a running setup with remote sites having pmacct + mysql which aggregates 
the data nightly and pushes them to a central hub as json files.

Alex

On January 5, 2017 5:42:45 PM GMT+02:00, Yann Belin  
wrote:
>Thanks all! I spent a couple of hours trough RabbitMQ docs/specs and it
>seems to be exactly what I need.
>
>On Thu, Jan 5, 2017 at 3:15 PM Karl O. Pinc  wrote:
>
>On Thu, 5 Jan 2017 10:57:01 +0100
>Yann Belin  wrote:
>
>> Not strictly a pmacct/nfacct question, but I was wondering if anyone
>> ever built a similar setup.
>
>Not strictly a pmacct/nfacct response, but thought I'd comment
>anyway.  ;-)
>
>> A central side would then gather
>> data from the different locations, and store the that data in a DBMS
>> (currently thinking of MySQL, but I'm not married to it).
>
>If you're already using MySQL it probably makes sense to continue
>to use it.  Otherwise my preference for a db is PostgreSQL.
>
>Regards,
>
>Karl 
>Free Software:  "You don't pay back, you pay forward."
> -- Robert A. Heinlein

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Centralizing data from multiple nfacct collectors

2017-01-05 Thread Karl O. Pinc 
On Thu, 5 Jan 2017 10:57:01 +0100
Yann Belin  wrote:

> Not strictly a pmacct/nfacct question, but I was wondering if anyone
> ever built a similar setup.

Not strictly a pmacct/nfacct response, but thought I'd comment
anyway.  ;-)

> A central side would then gather
> data from the different locations, and store the that data in a DBMS
> (currently thinking of MySQL, but I'm not married to it).

If you're already using MySQL it probably makes sense to continue
to use it.  Otherwise my preference for a db is PostgreSQL.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] pmacct / sflow / bgp

2017-01-05 Thread Paolo Lucente 

Hi Cedric,

43874 is the IANA-assigned enterprise number of pmacct. You have that
as part of your sFlow packets since tag or tag2 are part of your config
directive aggregate. Tags is pmacct-specific information and hence it's
encoded with the pmacct enterprise number. I believe you have two ways
forward: either you remove tags from your aggregate; or you find a way
to make them swallowed (decoded or skipped) by Net::sFlow (btw we should
have Elisa, the author of Net::sFlow, on the list - dunno if she has
anything to comment at this propo).

Cheers,
Paolo

On Wed, Jan 04, 2017 at 03:09:07PM +0100, Cédric ML wrote:
> Hi Paolo,
> sflowtool seems to give good results, but there's is still one
> problem : in each sflow sample, I have this :
> 
> skipping unknown flow_sample_element: 43874:2 len=16
> This causes problems with perl Net::sFlow library, as Flowdata
> enterprise: 43874 is not recognized.
> I'm unable to trace where this "43874" comes from...
> 
> Regards,
> Cédric
> 
> 
> Le 29/12/2016 à 12:38, Paolo Lucente a écrit :
> >Hi Cedric,
> >
> >While i can't say it's the very same issue, it seems related to what i
> >describe in the following comment:
> >
> >https://github.com/pmacct/pmacct/issues/71#issuecomment-265497661
> >
> >The sFlow dissector of Wireshark seems buggy and i recommend using
> >sflowtools for debugging and troubleshooting purposes.
> >
> >Cheers,
> >Paolo
> >
> >On Wed, Dec 28, 2016 at 04:22:19PM +0100, Cédric ML wrote:
> >>Hello,
> >>I'm trying to make pmacct work with a bgp agent (bird).
> >>
> >>pmacct is installed on the bgp router, bgp_agent session is up, and
> >>prefixes are exported to pmacct process.
> >>
> >>This bgp router has three vlans (50,51,52) on interface eth0.
> >>
> >>I'm trying to get correct correct values in incoming/outgoing VLANs,
> >>and source/destination AS (using pretag.map, maybe there is a
> >>simpler way ?)
> >>
> >>My problem, when running "pmacctd -f pmacctd.sflow.conf", is that
> >>wireshark tells me : "Expert Info (Error/Malformed): Malformed
> >>Packet (Exception occurred)"
> >>Agent address & ID are correctly displayed in capture (agent
> >>address=127.0.0.1 & agent_id=0)
> >>
> >>Here's the output of pmacctd :
> >>
> >># pmacctd -f pmacctd.sflow.conf
> >>INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
> >>1.6.2-git (20161222-00)
> >>INFO ( default/core ):
> >>INFO ( default/core ): Reading configuration file
> >>'/usr/local/etc/pmacct/pmacctd.sflow.conf'.
> >>INFO ( sfprobe/sfprobe ): plugin_pipe_size=4096000 bytes
> >>plugin_buffer_size=384 bytes
> >>INFO ( sfprobe/sfprobe ): ctrl channel: obtained=124928 bytes
> >>target=85328 bytes
> >>INFO ( default/core ): [/usr/local/etc/pmacct/pretag.map] (re)loading map.
> >>DEBUG ( sfprobe/sfprobe ): Creating sFlow agent.
> >>INFO ( sfprobe/sfprobe ): Exporting flows to [192.168.156.109]:6343
> >>INFO ( sfprobe/sfprobe ): Sampling at: 1/1000
> >>INFO ( default/core ): [/usr/local/etc/pmacct/pretag.map] map
> >>successfully (re)loaded.
> >>INFO ( default/core ): [/usr/local/etc/pmacct/pretag.map] (re)loading map.
> >>INFO ( default/core ): [/usr/local/etc/pmacct/pretag.map] map
> >>successfully (re)loaded.
> >>INFO ( default/core ): link type is: 1
> >>WARN ( default/core ): eth0: no IPv4 address assigned
> >>INFO ( default/core ): [/usr/local/etc/pmacct/agent_to_peer.map]
> >>(re)loading map.
> >>INFO ( default/core ): [/usr/local/etc/pmacct/agent_to_peer.map] map
> >>successfully (re)loaded.
> >>DEBUG ( default/core/BGP ): 1 thread(s) initialized
> >>INFO ( default/core/BGP ): maximum BGP peers allowed: 2
> >>INFO ( default/core/BGP ): waiting for BGP data on 127.0.0.1:17917
> >>INFO ( default/core/BGP ): [127.0.0.1] BGP peers usage: 1/2
> >>INFO ( default/core/BGP ): [x.x.x.x] Capability: MultiProtocol [1]
> >>AFI [1] SAFI [1]
> >>INFO ( default/core/BGP ): [x.x.x.x] Capability: 4-bytes AS [41] ASN
> >>[203596]
> >>INFO ( default/core/BGP ): [x.x.x.x] BGP_OPEN: Local AS: 203596
> >>Remote AS: 203596 HoldTime: 240
> >>DEBUG ( default/core/BGP ): [x.x.x.x] BGP_KEEPALIVE received
> >>DEBUG ( default/core/BGP ): [x.x.x.x] BGP_KEEPALIVE sent
> >>DEBUG ( sfprobe/sfprobe ): c08c60e112a7 -> 6805ca3dca86 (len = 1478,
> >>captured = 128)
> >>DEBUG ( sfprobe/sfprobe ): 78baf965af1f -> 6805ca3dca86 (len = 64,
> >>captured = 64)
> >>DEBUG ( sfprobe/sfprobe ): 78baf965af1f -> 6805ca3dca86 (len = 64,
> >>captured = 64)
> >>...
> >>
> >>
> >>Can anybody tell me what may be wrong in my config ?
> >>
> >>Best regards,
> >>Cédric
> >>
> >>
> >>== file pmacctd.sflow.conf
> >>debug: true
> >>daemonize: false
> >>interface: eth0
> >>aggregate: tag, src_host, dst_host, src_port, dst_port, proto, tos,
> >>src_as, dst_as
> >>plugins: sfprobe[sfprobe]
> >>sfprobe_receiver: 192.168.156.109:6343
> >>sfprobe_direction[sfprobe]: tag
> >>sfprobe_ifindex[sfprobe]: tag2
> >>sampling_rate: 1000
> >>pmacctd_as: bgp
> >>bgp_daemon: true
> >>bgp_daemon_ip: 127.0.0.1
>

Re: [pmacct-discussion] Unable to specify "bgp_ip=::1" in bgp_agent_map

2017-01-05 Thread Paolo Lucente 

Hi Charlie,

Thanks for following-up. I've also updated the docs with your example
as this or similar cases popped up a few times during last year:

https://github.com/pmacct/pmacct/commit/b3a5dd564983119eba7a481ad571a9d462922897

Cheers,
Paolo
 
On Thu, Jan 05, 2017 at 10:24:10AM +, Charlie Smurthwaite wrote:
> Hi,
> 
> Just to confirm, the following configuration works perfectly, using
> separate router IDs for each protocol, and identifying them by
> router id in the agent map:
> 
> bgp_ip=4.0.0.1   ip=0.0.0.0/0 filter='ip  or (vlan and ip)'
> bgp_ip=6.0.0.1   ip=0.0.0.0/0 filter='ip6 or (vlan and ip6)'
> 
> Now I've done the necessary configuration changes, this solution
> will be fine for my environment.
> 
> Thanks for looking into this for me,
> 
> Charlie
> 
> 
> 
> On 04/01/17 11:33, Paolo Lucente wrote:
> >Hi Charlie,
> >
> >If possible and easy for you, it would be great if you can change the BGP
> >Router ID among the two instances and confirm at least that scenario is
> >working good for you. I'm meanwhile trying to reproduce the scenario to
> >see if the issue you raised can be easily fixed.
> >
> >Cheers,
> >Paolo
> >
> >On Wed, Jan 04, 2017 at 09:57:06AM +, Charlie Smurthwaite wrote:
> >>Hi,
> >>
> >>I just wanted to follow up on this. It this something that could
> >>potentially be fixed, or something I'm doing wrong? I'm afraid I
> >>lack the understanding of the code to understand the nature of the
> >>problem or attempt to patch it myself.
> >>
> >>Thanks again,
> >>
> >>Charlie
> >>
> >>
> >>On 02/01/17 10:19, Charlie Smurthwaite wrote:
> >>>Hi Paolo,
> >>>
> >>>Thanks for the suggestion. I am using bird which has separate
> >>>daemons for v4 and v6, so I don't think there is any way to
> >>>combine the sessions.
> >>>
> >>>I could resolve this by changing the router IDs to be different
> >>>between v4 and v6, but I'd hoped this would not be necessary and I
> >>>could match on the peer IP address instead.
> >>>
> >>>Charlie
> >>>
> >>>
> >>>On 02/01/17 00:21, Paolo Lucente wrote:
> Hi Charlie,
> 
> I see about the same BGP router-id; i then wonder: why don't you just
> travel both v4 and v6 address families inside the very same v4 (or v6)
> session? Why the need for two sessions?
> 
> Cheers,
> Paolo
> 
> On Sat, Dec 31, 2016 at 05:18:18PM +, Charlie Smurthwaite wrote:
> >Thank you very much Paolo!
> >
> >The parser now accepts this configuration, but it still doesn't
> >quite work. Here is  my bgp_agent_map now:
> >
> >bgp_ip=127.0.0.1   ip=0.0.0.0/0 filter='ip  or (vlan and ip)'
> >bgp_ip=::1 ip=0.0.0.0/0 filter='ip6 or (vlan and ip6)'
> >
> >It seems that while ::1 is now accepted, it doesn't actually match
> >the peer. Here is the BGP log output:
> >
> >INFO ( default/core/BGP ): maximum BGP peers allowed: 2
> >INFO ( default/core/BGP ): waiting for BGP data on :::17917
> >INFO ( default/core/BGP ): [127.0.0.1] BGP peers usage: 1/2
> >INFO ( default/core/BGP ): [185.5.34.12] Capability: MultiProtocol
> >[1] AFI [1] SAFI [1]
> >INFO ( default/core/BGP ): [185.5.34.12] Capability: 4-bytes AS [41]
> >ASN [65535]
> >INFO ( default/core/BGP ): [185.5.34.12] BGP_OPEN: Local AS: 65535
> >Remote AS: 65535 HoldTime: 240
> >INFO ( default/core/BGP ): [::1] BGP peers usage: 2/2
> >INFO ( default/core/BGP ): [185.5.34.12] Capability: MultiProtocol
> >[1] AFI [2] SAFI [1]
> >INFO ( default/core/BGP ): [185.5.34.12] Capability: 4-bytes AS [41]
> >ASN [65535]
> >INFO ( default/core/BGP ): [185.5.34.12] BGP_OPEN: Local AS: 65535
> >Remote AS: 65535 HoldTime: 240
> >
> >If i bring up only the IPv6 peering, the following (using the router
> >ID) works and resolves only IPv6 ASNs. Unfortunately, both my IPv4
> >and IPv6 sessions use the same router ID.
> >
> >bgp_ip=185.5.34.12  ip=0.0.0.0/0
> >
> >It seems that something isn't quite right with matching "bgp_id=::1"
> >against the session originating from ::1. Would you mind seeing if
> >you can reproduce this?
> >
> >Thanks!
> >
> >Charlie
> >
> >
> >On 31/12/16 11:34, Paolo Lucente wrote:
> >>Hi Charlie,
> >>
> >>Definitely a bug, yes. Thanks for your report. This is now fixed:
> >>
> >>https://github.com/pmacct/pmacct/commit/ab7d675f1eaa90f753327a07c0184247f5f0517c
> >>
> >>
> >>Cheers,
> >>Paolo
> >>
> >>On Fri, Dec 30, 2016 at 11:37:31PM +, Charlie Smurthwaite wrote:
> >>>Hi,
> >>>
> >>>I am running pmacctd with 2 BGP sessions to a local bird
> >>>instance, one
> >>>for IPv4 and one for IPv6. I have written a bgp_agent_map a follows:
> >>>
> >>>bgp_ip=127.0.0.1 ip=0.0.0.0/0 filter='ip'
> >>>bgp_ip=::1   ip=0.0.0.0/0 filter='ip6'
> >>>
> >>>Unfortunately, the second line fails to parse. The error is:
> >>>
>

Re: [pmacct-discussion] Centralizing data from multiple nfacct collectors

2017-01-05 Thread Paolo Lucente 

+1 on this.

On Thu, Jan 05, 2017 at 10:10:40AM +, Charlie Smurthwaite wrote:
> On 05/01/17 09:57, Yann Belin wrote:
> >the collectors have to store
> >(temporarily) their data locally. A central side would then gather
> >data from the different locations, and store the that data in a DBMS
> >(currently thinking of MySQL, but I'm not married to it).
> 
> Have you considered using a message queue. pmacct supports RabbitMQ, and
> I find this to be an extremely effective way to queue up data to be
> processed later.
> 
> Charlie
> 
> 
> 
> 
> Charlie Smurthwaite
> Technical Director
> 
> tel.  email. charlie@atech.media web. 
> https://atech.media
> 
> This e-mail has been sent by aTech Media Limited (or one of its assoicated 
> group companys, Dial 9 Communications Limited or Viaduct Hosting Limited). 
> Its contents are confidential therefore if you have received this message in 
> error, we would appreciate it if you could let us know and delete the 
> message. aTech Media Limited is a UK limited company, registration number 
> 5523199. Dial 9 Communications Limited is a UK limited company, registration 
> number 7740921. Viaduct Hosting Limited is a UK limited company, registration 
> number 8514362. All companies are registered at Unit 9 Winchester Place, 
> North Street, Poole, Dorset, BH15 1NX.
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Unable to specify "bgp_ip=::1" in bgp_agent_map

2017-01-05 Thread Charlie Smurthwaite 

Hi,

Just to confirm, the following configuration works perfectly, using 
separate router IDs for each protocol, and identifying them by router id 
in the agent map:


bgp_ip=4.0.0.1   ip=0.0.0.0/0 filter='ip  or (vlan and ip)'
bgp_ip=6.0.0.1   ip=0.0.0.0/0 filter='ip6 or (vlan and ip6)'

Now I've done the necessary configuration changes, this solution will be 
fine for my environment.


Thanks for looking into this for me,

Charlie



On 04/01/17 11:33, Paolo Lucente wrote:

Hi Charlie,

If possible and easy for you, it would be great if you can change the BGP
Router ID among the two instances and confirm at least that scenario is
working good for you. I'm meanwhile trying to reproduce the scenario to
see if the issue you raised can be easily fixed.

Cheers,
Paolo

On Wed, Jan 04, 2017 at 09:57:06AM +, Charlie Smurthwaite wrote:

Hi,

I just wanted to follow up on this. It this something that could
potentially be fixed, or something I'm doing wrong? I'm afraid I
lack the understanding of the code to understand the nature of the
problem or attempt to patch it myself.

Thanks again,

Charlie


On 02/01/17 10:19, Charlie Smurthwaite wrote:

Hi Paolo,

Thanks for the suggestion. I am using bird which has separate
daemons for v4 and v6, so I don't think there is any way to
combine the sessions.

I could resolve this by changing the router IDs to be different
between v4 and v6, but I'd hoped this would not be necessary and I
could match on the peer IP address instead.

Charlie


On 02/01/17 00:21, Paolo Lucente wrote:

Hi Charlie,

I see about the same BGP router-id; i then wonder: why don't you just
travel both v4 and v6 address families inside the very same v4 (or v6)
session? Why the need for two sessions?

Cheers,
Paolo

On Sat, Dec 31, 2016 at 05:18:18PM +, Charlie Smurthwaite wrote:

Thank you very much Paolo!

The parser now accepts this configuration, but it still doesn't
quite work. Here is  my bgp_agent_map now:

bgp_ip=127.0.0.1   ip=0.0.0.0/0 filter='ip  or (vlan and ip)'
bgp_ip=::1 ip=0.0.0.0/0 filter='ip6 or (vlan and ip6)'

It seems that while ::1 is now accepted, it doesn't actually match
the peer. Here is the BGP log output:

INFO ( default/core/BGP ): maximum BGP peers allowed: 2
INFO ( default/core/BGP ): waiting for BGP data on :::17917
INFO ( default/core/BGP ): [127.0.0.1] BGP peers usage: 1/2
INFO ( default/core/BGP ): [185.5.34.12] Capability: MultiProtocol
[1] AFI [1] SAFI [1]
INFO ( default/core/BGP ): [185.5.34.12] Capability: 4-bytes AS [41]
ASN [65535]
INFO ( default/core/BGP ): [185.5.34.12] BGP_OPEN: Local AS: 65535
Remote AS: 65535 HoldTime: 240
INFO ( default/core/BGP ): [::1] BGP peers usage: 2/2
INFO ( default/core/BGP ): [185.5.34.12] Capability: MultiProtocol
[1] AFI [2] SAFI [1]
INFO ( default/core/BGP ): [185.5.34.12] Capability: 4-bytes AS [41]
ASN [65535]
INFO ( default/core/BGP ): [185.5.34.12] BGP_OPEN: Local AS: 65535
Remote AS: 65535 HoldTime: 240

If i bring up only the IPv6 peering, the following (using the router
ID) works and resolves only IPv6 ASNs. Unfortunately, both my IPv4
and IPv6 sessions use the same router ID.

bgp_ip=185.5.34.12  ip=0.0.0.0/0

It seems that something isn't quite right with matching "bgp_id=::1"
against the session originating from ::1. Would you mind seeing if
you can reproduce this?

Thanks!

Charlie


On 31/12/16 11:34, Paolo Lucente wrote:

Hi Charlie,

Definitely a bug, yes. Thanks for your report. This is now fixed:

https://github.com/pmacct/pmacct/commit/ab7d675f1eaa90f753327a07c0184247f5f0517c


Cheers,
Paolo

On Fri, Dec 30, 2016 at 11:37:31PM +, Charlie Smurthwaite wrote:

Hi,

I am running pmacctd with 2 BGP sessions to a local bird
instance, one
for IPv4 and one for IPv6. I have written a bgp_agent_map a follows:

bgp_ip=127.0.0.1 ip=0.0.0.0/0 filter='ip'
bgp_ip=::1   ip=0.0.0.0/0 filter='ip6'

Unfortunately, the second line fails to parse. The error is:

WARN ( default/core ): [/etc/pmacct/bgp_agent_map:2] required key
missing. Required keys are: 'id', 'ip'. Line ignored.

I have tried various other IPv6 addresses in place of "::1" and they
work, but "::1" and "0:0:0:0:0:0:0:1" will not work. I
have tested this
on 1.6.1 and master. Is this a bug?

Thanks!
Charlie


Charlie Smurthwaite
Technical Director

tel.  email.
charlie@atech.media web.
https://atech.media

This e-mail has been sent by aTech Media Limited (or one
of its assoicated group companys, Dial 9 Communications
Limited or Viaduct Hosting Limited). Its contents are
confidential therefore if you have received this message
in error, we would appreciate it if you could let us know
and delete the message. aTech Media Limited is a UK
limited company, registration number 5523199. Dial 9
Communications Limited is a UK limited company,
registration number 7740921. Viaduct Hosting Limited is a
UK limited company, registration number 8514362. All
companies are registered at Unit 9 Winchester Place, North
Street, Poole,

Re: [pmacct-discussion] Centralizing data from multiple nfacct collectors

2017-01-05 Thread Charlie Smurthwaite 

On 05/01/17 09:57, Yann Belin wrote:

the collectors have to store
(temporarily) their data locally. A central side would then gather
data from the different locations, and store the that data in a DBMS
(currently thinking of MySQL, but I'm not married to it).


Have you considered using a message queue. pmacct supports RabbitMQ, and
I find this to be an extremely effective way to queue up data to be
processed later.

Charlie




Charlie Smurthwaite
Technical Director

tel.  email. charlie@atech.media web. 
https://atech.media

This e-mail has been sent by aTech Media Limited (or one of its assoicated 
group companys, Dial 9 Communications Limited or Viaduct Hosting Limited). Its 
contents are confidential therefore if you have received this message in error, 
we would appreciate it if you could let us know and delete the message. aTech 
Media Limited is a UK limited company, registration number 5523199. Dial 9 
Communications Limited is a UK limited company, registration number 7740921. 
Viaduct Hosting Limited is a UK limited company, registration number 8514362. 
All companies are registered at Unit 9 Winchester Place, North Street, Poole, 
Dorset, BH15 1NX.

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists