subject:"\[pmacct\-discussion\] Capturing interface traffic with pmacct and inserting the data in PostgreSQL"

Re: [pmacct-discussion] Capturing interface traffic with pmacct and inserting the data in PostgreSQL

2020-08-26 Thread Paolo Lucente



Hi Arda,

I see that in your config you have 'daemonize: true' but no logfile 
statement set, ie. 'logfile: /tmp/pmacctd.log': this is preventing you 
from seeing any errors / warnings that pmacctd is logging and that may 
put you on the right path - is it an auth issue, is it a schema issue, 
etc. So that would be my first and foremost advice.


A second advice i may give you is, since you ask 'Should I expect the 
same level of detail that I see when I use tshark or tcpdump?', to get 
started with the 'print' plugin and follow 
https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2521-#L2542 . 
For example, given your config:


[..]
!
plugins: print[in], print[out]
aggregate[in]: dst_host
aggregate[out]: src_host
aggregate_filter[in]: dst net 10.10.10.0/24
aggregate_filter[out]: src net 10.10.10.0/24
!
print_refresh_time: 60
print_history: 1h
print_history_roundoff: h
print_output: csv
!
print_output_file[in]: /path/to/file-in-%Y%m%d-%H%M.csv
print_output_file[out]: /path/to/file-out-%Y%m%d-%H%M.csv
!
pcap_interfaces_map: /usr/local/share/pmacct/pcap_interfaces.map

This way, although in a CSV format in a file, playing with 'aggregate' 
you can get an idea what pmacct can get you compared to tcpdump/tshark 
(it will be pretty immediate to realise given the output).


Once you baseline pmacct is the tool for you and you get familiar with 
it, i guess you can complicate things putting a SQL database in the way.


Paolo


On 26/08/2020 19:30, Arda Savran wrote:
I just installed pmacct with postgres support on CentOS8 from GitHub; 
and I think it was a successful installation based on the following:


*[root@pcap pmacct]# pmacct -V*

*pmacct IMT plugin client, pmacct 1.7.6-git (20200826-0 (57a0334d))*

*'--enable-pgsql' '--enable-l2' '--enable-traffic-bins' 
'--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins'*


**

*For suggestions, critics, bugs, contact me: Paolo Lucente 
.*


*[root@pcap pmacct]# pmacctd -V*

*Promiscuous Mode Accounting Daemon, pmacctd 1.7.6-git [20200826-0 
(57a0334d)]*


**

*Arguments:*

*'--enable-pgsql' '--enable-l2' '--enable-traffic-bins' 
'--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins'*


**

*Libs:*

*libpcap version 1.9.0-PRE-GIT (with TPACKET_V3)*

*PostgreSQL 120001*

**

*System:*

*Linux 4.18.0-193.14.2.el8_2.x86_64 #1 SMP Sun Jul 26 03:54:29 UTC 2020 
x86_64*


**

*Compiler:*

*gcc 8.3.1*

**

*For suggestions, critics, bugs, contact me: Paolo Lucente 
.*


My goal is to capture the in/out network traffic on this machine’s 
interfaces and record them in PostgreSQL. I created myself a 
pmacctd.conf file under /usr/local/share/pmacct folder and a 
pcap_interfaces.map under the same folder. Before my question, can 
someone please confirm that my expectations from pmacct is accurate:


  * Pmacct can capture all the network traffic on the local interface
(ens192) and record it in PostgreSQL. Should I expect the same level
detail that I see when I use tshark or tcpdump?
  * Pmacct can store all the packet details in PostgreSQL if needed. If
this is not supported, does this mean that I am obligated to
aggregate the interface traffic before it is inserted into PostgreSQL.

My issue is that I am not seeing any data being written into any of the 
following tables:


*pmacct=# \dt*

*  List of relations*

*Schema |   Name   | Type  |  Owner*

*+--+---+--*

*public | acct | table | postgres*

*public | acct_as  | table | postgres*

*public | acct_uni | table | postgres*

*public | acct_v9  | table | postgres*

*public | proto    | table | postgres*

I started the daemon by running: pmacctd -f pmacctd.conf

My conf file is based on what I read on the WiKi page:

*!*

*daemonize: true*

*plugins: pgsql[in], pgsql[out]*

*aggregate[in]: dst_host*

*aggregate[out]: src_host*

*aggregate_filter[in]: dst net 10.10.10.0/24*

*aggregate_filter[out]: src net 10.10.10.0/24*

*sql_table[in]: acct_in*

*sql_table[out]: acct_out*

*sql_refresh_time: 60*

*sql_history: 1h*

*sql_history_roundoff: h*

*pcap_interfaces_map: /usr/local/share/pmacct/pcap_interfaces.map*

*! ...*

I am not sure how to proceed from here. I don’t know if I am supposed to 
be creating a table on PostgreSQL manually first based on my aggregation 
settings and somehow include that in the config file.


Can some please point me to the right direction.

Thanks,

Sent from Mail  for 
Windows 10



___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists




___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Capturing interface traffic with pmacct and inserting the data in PostgreSQL

2020-08-26 Thread Arda Savran

I just installed pmacct with postgres support on CentOS8 from GitHub; and I 
think it was a successful installation based on the following:
[root@pcap pmacct]# pmacct -V
pmacct IMT plugin client, pmacct 1.7.6-git (20200826-0 (57a0334d))
 '--enable-pgsql' '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins' 
'--enable-bmp-bins' '--enable-st-bins'

For suggestions, critics, bugs, contact me: Paolo Lucente .
[root@pcap pmacct]# pmacctd -V
Promiscuous Mode Accounting Daemon, pmacctd 1.7.6-git [20200826-0 (57a0334d)]

Arguments:
 '--enable-pgsql' '--enable-l2' '--enable-traffic-bins' '--enable-bgp-bins' 
'--enable-bmp-bins' '--enable-st-bins'

Libs:
libpcap version 1.9.0-PRE-GIT (with TPACKET_V3)
PostgreSQL 120001

System:
Linux 4.18.0-193.14.2.el8_2.x86_64 #1 SMP Sun Jul 26 03:54:29 UTC 2020 x86_64

Compiler:
gcc 8.3.1

For suggestions, critics, bugs, contact me: Paolo Lucente .

My goal is to capture the in/out network traffic on this machine’s interfaces 
and record them in PostgreSQL. I created myself a pmacctd.conf file under 
/usr/local/share/pmacct folder and a pcap_interfaces.map under the same folder. 
Before my question, can someone please confirm that my expectations from pmacct 
is accurate:
• Pmacct can capture all the network traffic on the local interface (ens192) 
and record it in PostgreSQL. Should I expect the same level detail that I see 
when I use tshark or tcpdump?
• Pmacct can store all the packet details in PostgreSQL if needed. If this is 
not supported, does this mean that I am obligated to aggregate the interface 
traffic before it is inserted into PostgreSQL.

My issue is that I am not seeing any data being written into any of the 
following tables:
pmacct=# \dt
  List of relations
 Schema |   Name   | Type  |  Owner
+--+---+--
 public | acct | table | postgres
 public | acct_as  | table | postgres
 public | acct_uni | table | postgres
 public | acct_v9  | table | postgres
 public | proto| table | postgres

I started the daemon by running: pmacctd -f pmacctd.conf

My conf file is based on what I read on the WiKi page:

!
daemonize: true
plugins: pgsql[in], pgsql[out]
aggregate[in]: dst_host
aggregate[out]: src_host
aggregate_filter[in]: dst net 10.10.10.0/24
aggregate_filter[out]: src net 10.10.10.0/24
sql_table[in]: acct_in
sql_table[out]: acct_out
sql_refresh_time: 60
sql_history: 1h
sql_history_roundoff: h
pcap_interfaces_map: /usr/local/share/pmacct/pcap_interfaces.map
! ...

I am not sure how to proceed from here. I don’t know if I am supposed to be 
creating a table on PostgreSQL manually first based on my aggregation settings 
and somehow include that in the config file.

Can some please point me to the right direction.

Thanks,

Sent from Mail for Windows 10

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Capturing interface traffic with pmacct and inserting the data in PostgreSQL

[pmacct-discussion] Capturing interface traffic with pmacct and inserting the data in PostgreSQL

2 matches

Site Navigation

Mail list logo

Footer information