Re: [pmacct-discussion] BMP

[Paolo Lucente]

Hi Edgar,

1) Wonderful stuff!

2) In the logfile you will find basic information of which BMP exporters 
are connected and a recap of how many of them are connected and whether 
any does disconnect; should you go for dumps at regular intervals, you 
will find there a recap as well of how many tables (ie. peers) and 
entries (ie. RIB info) are being sent over to Kafka; if you go for 
message logs, instead, than you can assume that it's message-in / 
message-out, 1:1.


The most beautiful thing about Kafka is that you can have N consumers 
for the very same produced data. So anytime you can have your database 
consuming data but, in parallel, you can "sniff" exactly what would have 
gone in the database, message by message: super useful in 
proof-of-concept phase, then it gets boring. Kafka ships with a console 
consumer, you can find some info here:


https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L1028-#L1033

Should you be looking for yet other summaries of what goes into Kafka, 
you could enable some extra statistics in pmacct, as offered by librdkafka:


https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L1139-#L1144

Just instead of kafka_config_file do use bmp_dump_kafka_config_file or 
bmp_daemon_msglog_kafka_config_file (depending if you are configuring 
dumps or message logs).


Paolo


On 31/5/21 09:19, edgar lip wrote:

Hi Paolo ,

ok , i got it - fair enough.
1. Once I will be able to grasp my head around it and if something will 
come up from this - I will make a doc and share it for sure !!!
2.what about the how to's about only the bmp section - the quick start 
explains how to start it - but not how to drive it ( show commands , 
check proper work , check that kafka send messages , etc ... )




thanks
Lipnitsky Edgar


On Mon, May 31, 2021 at 7:13 AM Paolo Lucente > wrote:



Hi Edgar,

For end-to-end solutions you have two main choices (of course i am
excluding the obvious: buy a product or buy consultancy from somebody):

1) Google for them, and you may end up with results like this one

https://imply.io/post/an-end-to-end-streaming-analytics-stack-for-network-telemetry-data



or

2) look on GitHub for containers, and you may end up with results like
this one https://github.com/kvitex/pmacct-kafka-clickhouse


Essentially with data pipelines, since you enter in a combinatorial
game
of choices, ie. each piece you add to the pipeline to make it
end-to-end
you add more choices that inflate a matrix of options, it gets
difficult
to find an how-to guide for exactly what you are trying to achieve.

So, very possibly, either you stick to one of the solutions you find
documented or you have to gather all the pieces together and cover as
much as possible the pipeline you have in mind; the rest, you have to
fill (and maybe be so kind to document it for others to enjoy :-)). Of
course any specific help needed in filling the gaps you may hit, i'd be
happy to help you with.

Paolo


On 30/5/21 09:19, edgar lip wrote:
 > hi ,
 > from the quick start guide i see that i can set up only the
collector
 > itself , and send what was collected to a kafka , which is a good
start,
 > but = )
 >    - there is no mention of some show commands to check what is
working
 > / if working etc ...
 >    - it is not clear which design is supported ?
 >    - is there a docs on how to  continue from there ... ?
 >   - example how to setup the kafka
 >   -  how to setup the data basse and also connect it to the kafka
 > What i am trying to say is that there is no global guideline to
follow ,
 > i am mostly dealing with network but still have a good overall look
 > For things like this, it will be great if u can point me  to some
docs /
 > howto's to read and learn more about the project.
 >
 > Also saw that someone sent me how to dump logs on the machine
itself (
 > thanks John) , it is nice but not the way that i meant to go with
this.
 > As I mentioned I am trying to go like "MX router -> bmp collector (
 > pmacct /pmbmpd ) -> kafka -> psql -> grafana."
 >
 >
 > appreciate any help
 > thanks
 > Lipnitsky Edgar
 >
 >
 >
 > On Sat, May 29, 2021 at 9:53 PM Paolo Lucente mailto:pa...@pmacct.net>
 > >> wrote:
 >
 >
 >     Hi Edgar,
 >
 >     Thanks for your feedback wrt the BMP documentation. Let's try
to get
 >     you
 >     up and running and improve docs but, in order to do that, i'd
need some
 >     more specific question(s) from you. Where are you stuck? What
is not
 >     working?
 >
 >   

Re: [pmacct-discussion] sql_num_hosts only giving null values in the MySQL database

[Paolo Lucente]

Hi Klaas,

Do the log provide any hints / error that can put us on the right track? 
Should that not help, can you please enable debug on the pmacct side 
(that is: -d or "debug: true") to see if anything more helpful pops up 
in the logs?


A total personal comment: sql_num_hosts is surely good to optimize stuff 
if you are forced into the MySQL / MariaDB world. But if you care about 
optimization and getting the right operands over the IP fields, ie. 
netmask operations and stuff, look into PostgreSQL.


Paolo


On 31/5/21 14:17, Klaas Tammling wrote:

Hi everyone,

I'm having some trouble using sql_num_hosts with nfacctd. I have 
converted all relevant columns (ip_dst, ip_src, net_src, net_dst) to 
VARBINARY(16) in my MariaDB.


However when writing into the database nfacctd only writes NULL values 
into the database. When setting sql_num_hosts to the default I'm getting 
the values back as a string.


Am I missing something additional here?

I was hoping to get a better performance, using sql_num_hosts, when 
searching through the database.


Thanks for your help.

Regards,

Klaas

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists



___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] sql_num_hosts only giving null values in the MySQL database

[Klaas Tammling]

Hi everyone,

I'm having some trouble using sql_num_hosts with nfacctd. I have converted all 
relevant columns (ip_dst, ip_src, net_src, net_dst) to VARBINARY(16) in my 
MariaDB.

However when writing into the database nfacctd only writes NULL values into the 
database. When setting sql_num_hosts to the default I'm getting the values back 
as a string.

Am I missing something additional here?

I was hoping to get a better performance, using sql_num_hosts, when searching 
through the database.

Thanks for your help.

Regards,

Klaas
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] BMP

[edgar lip]

Hi Paolo ,

ok , i got it - fair enough.
1. Once I will be able to grasp my head around it and if something will
come up from this - I will make a doc and share it for sure !!!
2.what about the how to's about only the bmp section - the quick start
explains how to start it - but not how to drive it ( show commands , check
proper work , check that kafka send messages , etc ... )



thanks
Lipnitsky Edgar


On Mon, May 31, 2021 at 7:13 AM Paolo Lucente  wrote:

>
> Hi Edgar,
>
> For end-to-end solutions you have two main choices (of course i am
> excluding the obvious: buy a product or buy consultancy from somebody):
>
> 1) Google for them, and you may end up with results like this one
>
> https://imply.io/post/an-end-to-end-streaming-analytics-stack-for-network-telemetry-data
> or
>
> 2) look on GitHub for containers, and you may end up with results like
> this one https://github.com/kvitex/pmacct-kafka-clickhouse
>
> Essentially with data pipelines, since you enter in a combinatorial game
> of choices, ie. each piece you add to the pipeline to make it end-to-end
> you add more choices that inflate a matrix of options, it gets difficult
> to find an how-to guide for exactly what you are trying to achieve.
>
> So, very possibly, either you stick to one of the solutions you find
> documented or you have to gather all the pieces together and cover as
> much as possible the pipeline you have in mind; the rest, you have to
> fill (and maybe be so kind to document it for others to enjoy :-)). Of
> course any specific help needed in filling the gaps you may hit, i'd be
> happy to help you with.
>
> Paolo
>
>
> On 30/5/21 09:19, edgar lip wrote:
> > hi ,
> > from the quick start guide i see that i can set up only the collector
> > itself , and send what was collected to a kafka , which is a good start,
> > but = )
> >- there is no mention of some show commands to check what is working
> > / if working etc ...
> >- it is not clear which design is supported ?
> >- is there a docs on how to  continue from there ... ?
> >   - example how to setup the kafka
> >   -  how to setup the data basse and also connect it to the kafka
> > What i am trying to say is that there is no global guideline to follow ,
> > i am mostly dealing with network but still have a good overall look
> > For things like this, it will be great if u can point me  to some docs /
> > howto's to read and learn more about the project.
> >
> > Also saw that someone sent me how to dump logs on the machine itself (
> > thanks John) , it is nice but not the way that i meant to go with this.
> > As I mentioned I am trying to go like "MX router -> bmp collector (
> > pmacct /pmbmpd ) -> kafka -> psql -> grafana."
> >
> >
> > appreciate any help
> > thanks
> > Lipnitsky Edgar
> >
> >
> >
> > On Sat, May 29, 2021 at 9:53 PM Paolo Lucente  > > wrote:
> >
> >
> > Hi Edgar,
> >
> > Thanks for your feedback wrt the BMP documentation. Let's try to get
> > you
> > up and running and improve docs but, in order to do that, i'd need
> some
> > more specific question(s) from you. Where are you stuck? What is not
> > working?
> >
> > Paolo
> >
> > On 29/5/21 13:05, edgar lip wrote:
> >  > Hi pacct team / gents
> >  >
> >  > I would like to start using this project with the bmp section ,
> > later
> >  > will check the rpki and then telemetry.
> >  > i am an network dude mostly ( can see this based on my request =)
> )
> >  > but first thing first - bmp
> >  > Can you guys help with a start manual / how to's - i saw the
> > quick start
> >  > but it is very unclear and lacks details.
> >  >
> >  > as of the picture that i see right now:
> >  > MX router -> bmp collector ( pmacct /pmbmpd ) -> kafka -> psql ->
> > grafana.
> >  >
> >  >
> >  > thanks a lot
> >  > Lipnitsky Edgar
> >  >
> >  > ___
> >  > pmacct-discussion mailing list
> >  > http://www.pmacct.net/#mailinglists
> > 
> >  >
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> > 
> >
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> >
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists