Re: [pmacct-discussion] TimescaleDB

[Paolo Lucente]

Hi John,

Yes, i can confirm that writing directly from pmacct into a TimescaleDB, 
you can do it using the 'pgsql' plugin. Should you run into troubles 
(which you should not!) please let me know.


Paolo



On 3/5/22 17:33, John Jensen wrote:

Hi all,

Has anyone successfully used TimescaleDB as a backend for pmacctd? I 
understand that TimescaleDB is essentially just a Postgres extension - 
does this mean that nothing really "changes" in terms of configuring 
pmacct to insert into a traditional SQL database (Postgres)?


Thanks in advance!

-JJ

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] pmacct.net

[Paolo Lucente]

Hi Karl,

Always great to read from you & thanks for your note.

Somehow i can't reproduce the problem, both pmacct.net and 
www.pmacct.net do actually work for me no problem (http of course, ie. 
not https, well no https is advertised out nor does it work).


Can you please qualify the issue better (here or by unicast email).

Thanks,
Paolo



On 3/5/22 16:28, Karl O. Pinc wrote:

FYI.

I notice that "pmacct.net" in my browser's URL bar does
not redirect to "www.pmacct.net".  I get
"unable to connect".

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
  -- Robert A. Heinlein

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Easiest way to ingest nfacctd data into python?

[Paolo Lucente]

Hi Rich,

While i don't have actual examples and while supporting the answers you 
already received, i may propose you the following architectural tips:


* Write stuff into files with the 'print' plugin; using 
print_latest_file to point always to the latest finalized file and 
print_trigger_exec to execute a 3rd party (post-processing) script on 
the file just finalized;


* Whatever language the post-processing is written into, you know that 
you are being invoked by print_trigger_exec because print_latest_file is 
now available. So, not even much env variable sophistication, you know 
which file to read and process with your Python script;


* Should you want to scale up things further you could use the kafka 
plugin, for example, instead of print / files; with a Python script 
consuming from the topic where pmacct is producing. Such a setup would 
allow you to scale things out easily with kafka topic partitions and 
consumer threads / processes.


Paolo


On 3/5/22 15:19, Compton, Rich A wrote:
Hi, I’m trying to take the netflow records from nfacctd and process them 
with a python script.  Can someone suggest how I can do this with python 
without having nfacctd put them into a database and then have my python 
script read it?  Is using kafka the best way?  The netflow collection 
and python script will be on the same instance.  Any example code would 
be very helpful!


Thanks!

signature_2304850901

Rich Compton    |     Principal Eng   |    314.596.2828

8560 Upland Drive,   Suite B  |  Englewood, CO 80112

PGP Key 



The contents of this e-mail message and
any attachments are intended solely for the
addressee(s) and may contain confidential
and/or legally privileged information. If you
are not the intended recipient of this message
or if this message has been addressed to you
in error, please immediately alert the sender
by reply e-mail and then delete this message
and any attachments. If you are not the
intended recipient, you are notified that
any use, dissemination, distribution, copying,
or storage of this message or any attachment
is strictly prohibited.

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] TimescaleDB

[John Jensen]

Hi all,

Has anyone successfully used TimescaleDB as a backend for pmacctd? I
understand that TimescaleDB is essentially just a Postgres extension - does
this mean that nothing really "changes" in terms of configuring pmacct to
insert into a traditional SQL database (Postgres)?

Thanks in advance!

-JJ
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Easiest way to ingest nfacctd data into python?

[Luca Cilloni]

Hi,
I believe that the easiest way is using the json output over a linux pipe, or 
if you prefer over stdin/stdout, which eventually behave like a pipe.
Here an example with pipe:

Python script:
if __name__ == '__main__':
  os.mkfifo(path_to_fifo, mode=0o666)

  while True:
  # One loop == one *acct flush
  with open(path_to_fifo) as fifo:
  timestamp = time.time_ns()

  # Read *acct records from fifo line by line
  for line in fifo:
  buffer = process_json_record(line, timestamp)

  # Process *acct flush
  process_buffer(buffer)

Nfacct config (only relevant parts):
print_output: json
print_output_file_append: true
print_write_empty_file: false
plugins: print[json]
print_output_file[json]: path_to_fifo

Cheers,
luca

> Il giorno 3 mag 2022, alle ore 20:19, Compton, Rich A 
>  ha scritto:
> 
> Hi, I’m trying to take the netflow records from nfacctd and process them with 
> a python script.  Can someone suggest how I can do this with python without 
> having nfacctd put them into a database and then have my python script read 
> it?  Is using kafka the best way?  The netflow collection and python script 
> will be on the same instance.  Any example code would be very helpful!
> Thanks!
>  
> 
>  
> Rich Compton| Principal Eng |314.596.2828
> 8560 Upland Drive,   Suite B  |  Englewood, CO 80112
> PGP Key 
> 
>  
> The contents of this e-mail message and 
> any attachments are intended solely for the 
> addressee(s) and may contain confidential 
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you 
> in error, please immediately alert the sender
> by reply e-mail and then delete this message 
> and any attachments. If you are not the 
> intended recipient, you are notified that 
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment 
> is strictly prohibited.
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Easiest way to ingest nfacctd data into python?

[Tim Jackson]

I've done similar with the IMT and Perl years ago:

https://houstongrackles.com/~tjackson/flows_to_es/

Relevant part in Perl:

sub retrieve_flows {
my $pipe = shift;
my $primitive = shift;
my $filter = shift;
my @flows = `/usr/local/bin/pmacct -p $pipe -l -O json -c
"$primitive" -M "$filter"`;
return @flows;
}

The array there is just JSON and can be decoded afterwards.. Python should
be able to do similar with subprocess + run on 3.5+..

--
Tim

On Tue, May 3, 2022 at 2:30 PM Karl O. Pinc  wrote:

> On Tue, 3 May 2022 18:19:50 +
> "Compton, Rich A"  wrote:
>
> > Hi, I’m trying to take the netflow records from nfacctd and process
> > them with a python script.  Can someone suggest how I can do this
> > with python without having nfacctd put them into a database and then
> > have my python script read it?
>
> You could use the in-memory recording and poll and clear
> out the old data with python.   I don't know how well
> that would work, depending on scale.  I tend to favor
> relational databases and would lean toward using
> sqlite if you're looking for "simple".  (And Postgres
> for everything else.)
>
> You are collecting tabular data, which RDBMSs excel at.
>
> Regards,
>
> Karl 
> Free Software:  "You don't pay back, you pay forward."
>  -- Robert A. Heinlein
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] pmacct.net

[Karl O. Pinc]

FYI.

I notice that "pmacct.net" in my browser's URL bar does
not redirect to "www.pmacct.net".  I get 
"unable to connect".

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Easiest way to ingest nfacctd data into python?

[Karl O. Pinc]

On Tue, 3 May 2022 18:19:50 +
"Compton, Rich A"  wrote:

> Hi, I’m trying to take the netflow records from nfacctd and process
> them with a python script.  Can someone suggest how I can do this
> with python without having nfacctd put them into a database and then
> have my python script read it?  

You could use the in-memory recording and poll and clear
out the old data with python.   I don't know how well
that would work, depending on scale.  I tend to favor
relational databases and would lean toward using
sqlite if you're looking for "simple".  (And Postgres
for everything else.)

You are collecting tabular data, which RDBMSs excel at.

Regards,

Karl 
Free Software:  "You don't pay back, you pay forward."
 -- Robert A. Heinlein

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Easiest way to ingest nfacctd data into python?

[Compton, Rich A]

Hi, I’m trying to take the netflow records from nfacctd and process them with a 
python script.  Can someone suggest how I can do this with python without 
having nfacctd put them into a database and then have my python script read it? 
 Is using kafka the best way?  The netflow collection and python script will be 
on the same instance.  Any example code would be very helpful!
Thanks!

[signature_2304850901]

Rich Compton| Principal Eng |314.596.2828
8560 Upland Drive,   Suite B  |  Englewood, CO 80112
PGP 
Key

E-MAIL CONFIDENTIALITY NOTICE: 
The contents of this e-mail message and any attachments are intended solely for 
the addressee(s) and may contain confidential and/or legally privileged 
information. If you are not the intended recipient of this message or if this 
message has been addressed to you in error, please immediately alert the sender 
by reply e-mail and then delete this message and any attachments. If you are 
not the intended recipient, you are notified that any use, dissemination, 
distribution, copying, or storage of this message or any attachment is strictly 
prohibited.
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists