Hi Edgar,
1) Wonderful stuff!
2) In the logfile you will find basic information of which BMP exporters
are connected and a recap of how many of them are connected and whether
any does disconnect; should you go for dumps at regular intervals, you
will find there a recap as well of how many tables (ie. peers) and
entries (ie. RIB info) are being sent over to Kafka; if you go for
message logs, instead, than you can assume that it's message-in /
message-out, 1:1.
The most beautiful thing about Kafka is that you can have N consumers
for the very same produced data. So anytime you can have your database
consuming data but, in parallel, you can "sniff" exactly what would have
gone in the database, message by message: super useful in
proof-of-concept phase, then it gets boring. Kafka ships with a console
consumer, you can find some info here:
https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L1028-#L1033
Should you be looking for yet other summaries of what goes into Kafka,
you could enable some extra statistics in pmacct, as offered by librdkafka:
https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L1139-#L1144
Just instead of kafka_config_file do use bmp_dump_kafka_config_file or
bmp_daemon_msglog_kafka_config_file (depending if you are configuring
dumps or message logs).
Paolo
On 31/5/21 09:19, edgar lip wrote:
Hi Paolo ,
ok , i got it - fair enough.
1. Once I will be able to grasp my head around it and if something will
come up from this - I will make a doc and share it for sure !!!
2.what about the how to's about only the bmp section - the quick start
explains how to start it - but not how to drive it ( show commands ,
check proper work , check that kafka send messages , etc ... )
thanks
Lipnitsky Edgar
On Mon, May 31, 2021 at 7:13 AM Paolo Lucente <[email protected]
<mailto:[email protected]>> wrote:
Hi Edgar,
For end-to-end solutions you have two main choices (of course i am
excluding the obvious: buy a product or buy consultancy from somebody):
1) Google for them, and you may end up with results like this one
https://imply.io/post/an-end-to-end-streaming-analytics-stack-for-network-telemetry-data
<https://imply.io/post/an-end-to-end-streaming-analytics-stack-for-network-telemetry-data>
or
2) look on GitHub for containers, and you may end up with results like
this one https://github.com/kvitex/pmacct-kafka-clickhouse
<https://github.com/kvitex/pmacct-kafka-clickhouse>
Essentially with data pipelines, since you enter in a combinatorial
game
of choices, ie. each piece you add to the pipeline to make it
end-to-end
you add more choices that inflate a matrix of options, it gets
difficult
to find an how-to guide for exactly what you are trying to achieve.
So, very possibly, either you stick to one of the solutions you find
documented or you have to gather all the pieces together and cover as
much as possible the pipeline you have in mind; the rest, you have to
fill (and maybe be so kind to document it for others to enjoy :-)). Of
course any specific help needed in filling the gaps you may hit, i'd be
happy to help you with.
Paolo
On 30/5/21 09:19, edgar lip wrote:
> hi ,
> from the quick start guide i see that i can set up only the
collector
> itself , and send what was collected to a kafka , which is a good
start,
> but = )
> - there is no mention of some show commands to check what is
working
> / if working etc ...
> - it is not clear which design is supported ?
> - is there a docs on how to continue from there ... ?
> - example how to setup the kafka
> - how to setup the data basse and also connect it to the kafka
> What i am trying to say is that there is no global guideline to
follow ,
> i am mostly dealing with network but still have a good overall look
> For things like this, it will be great if u can point me to some
docs /
> howto's to read and learn more about the project.
>
> Also saw that someone sent me how to dump logs on the machine
itself (
> thanks John) , it is nice but not the way that i meant to go with
this.
> As I mentioned I am trying to go like "MX router -> bmp collector (
> pmacct /pmbmpd ) -> kafka -> psql -> grafana."
>
>
> appreciate any help
> thanks
> Lipnitsky Edgar
>
>
>
> On Sat, May 29, 2021 at 9:53 PM Paolo Lucente <[email protected]
<mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>
>
> Hi Edgar,
>
> Thanks for your feedback wrt the BMP documentation. Let's try
to get
> you
> up and running and improve docs but, in order to do that, i'd
need some
> more specific question(s) from you. Where are you stuck? What
is not
> working?
>
> Paolo
>
> On 29/5/21 13:05, edgar lip wrote:
> > Hi pacct team / gents
> >
> > I would like to start using this project with the bmp
section ,
> later
> > will check the rpki and then telemetry.
> > i am an network dude mostly ( can see this based on my
request =) )
> > but first thing first - bmp
> > Can you guys help with a start manual / how to's - i saw the
> quick start
> > but it is very unclear and lacks details.
> >
> > as of the picture that i see right now:
> > MX router -> bmp collector ( pmacct /pmbmpd ) -> kafka ->
psql ->
> grafana.
> >
> >
> > thanks a lot
> > Lipnitsky Edgar
> >
> > _______________________________________________
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>
> <http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>>
> >
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>
> <http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>>
>
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>
>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
<http://www.pmacct.net/#mailinglists>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
