[Ntop-misc] Trouble automating daemon startup for ntopng

2016-10-27 Thread Peter Shute 
I've followed the instructions in the nprobe manual to get it running as a 
daemon on system start, but I can't get ntopng to do the same thing.

I've got /etc/ntopng/ntopng.conf configured, and it works ok if I run sudo 
ntopng /etc/ntopng/ntopng.conf.

/etc/ntopng/ntop.start exists, but ntopng doesn't start on boot. What's wrong?

Here's the ntopng.conf file contents:

-G=/var/run/ntopng.pid
--daemon=
-i=tcp://192.168.0.222:2055
-w=3050
-F="mysql;localhost;ntopng;flows;root;admin"
--local-networks="192.168.0.0/23,192.168.2.0/24,192.168.3.0/24,192.168.6.0/24,192.168.7.0/24,192.168.30.0/24,192.168.60.0/24,192.168.32.0/24,192.168.62.0/24,192.168.33.0/24,192.168.63.0/24,192.168.37.0/24,192.168.67.0/24"

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

[Ntop-misc] PF_RING sees DAG but nprobe does not

2016-10-27 Thread John Marshall 
Current situation.
When trying to run nprobe against dag0 I get the following:
-
28/Oct/2016 12:57:02 [util.c:4371] ERROR: Cannot get hw addr for dag0
28/Oct/2016 12:57:02 [pro/pf_ring.c:377] Initializing PF_RING socket on device 
dag0..
28/Oct/2016 12:57:02 [nprobe.c:5573] ERROR: Unable to open interface dag0.
28/Oct/2016 12:57:02 [nprobe.c:7401] ERROR: Unable to open interface dag0 
(dag0: No such device exists (SIOCGIFHWADDR: No such device))
---

Even though the PF_RING pcount utility sees it and can get packet stats from it.

[root@localhost ~]# /usr/local/src/pfring/userland/examples/pcount -h
pcount
(C) 2003-14 Deri Luca 
-h  [Print help]
-i  [Device name]
-f  [pcap filter]
-l [Capture length]
-S  [Do not strip hw timestamps (if present)]
-v[Verbose [1: verbose, 2: very verbose (print packet payload)]]

Available devices (-i):
 0. ens3
 1. ens4
 2. any
 3. lo
 4. dag0
 5. dag0:0
 6. nflog
 7. nfqueue
 8. usbmon1
[root@localhost ~]#
---
What now?

How I got here:
1)compiled PF_RING with dag libraries and confirmed it worked with the pcount 
utility.
2)installed nprobe using yum.  (I did not see how to compile nprobe)

--
John Marshall
Senior Consultant

T +64 9 355 4818 (extn 94818)
M +64 27 819 8366
E john.marsh...@spark.co.nz

Level 1 Green, Spark City | 167 Victoria Street West | Private Bag 92028, 
Auckland 1010
www.sparkdigital.co.nz
--
This communication, including any attachments, is confidential. If you are not 
the intended recipient, you should not read it - please contact me immediately, 
destroy it, and do not copy or use any part of this communication or disclose 
anything about it. Thank you. Please note that this communication does not 
designate an information system for the purposes of the Electronic Transactions 
Act 2002.
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Access to historical charts very slow in ntopng

2016-10-27 Thread Peter Shute 
For what it's worth, this query took 1 minute 37 seconds:
select ip_src_addr from flowsv4 order by ip_src_addr limit 10;

There are about 26,000,000 records in flowsv4 (and none in flowsv6). Is that 
normal?

> -Original Message-
> From: ntop-misc-boun...@listgateway.unipi.it [mailto:ntop-misc-
> boun...@listgateway.unipi.it] On Behalf Of Peter Shute
> Sent: Friday, 28 October 2016 7:56 AM
> To: ntop-misc@listgateway.unipi.it
> Subject: Re: [Ntop-misc] Access to historical charts very slow in ntopng
> 
> Do I have to run "use performance_schema;" first? I'm new to mysql. It took
> 0.02 seconds, is that fast enough?
> 
> *** 1. row
> ***
> 
> SCHEMA_NAME: ntopng
> 
>  DIGEST: 58bffbb800986c1b1147f462d49802e5
> 
> DIGEST_TEXT: INSERT INTO `flowsv4` ( `VLAN_ID` , `L7_PROTO` ,
> `IP_SRC_ADDR` , `L4_SRC_PORT` , `IP_DST_ADDR` , `L4_DST_PORT` ,
> `PROTOCOL` , `IN_BYTES` , `OUT_BYTES` , `PACKETS` , `FIRST_SWITCHED` ,
> `LAST_SWITCHED` , `INFO` , JSON , `NTOPNG_INSTANCE_NAME` ,
> `INTERFACE_ID` , PROFILE ) VALUES ( ?, ... , `COMPRESS` (?) , ?, ... )
> 
>  COUNT_STAR: 1696935
> 
>  SUM_TIMER_WAIT: 6731129622996000
> 
>  MIN_TIMER_WAIT: 220633000
> 
>  AVG_TIMER_WAIT: 3966639000
> 
>  MAX_TIMER_WAIT: 665490453714000
> 
>   SUM_LOCK_TIME: 521387755000
> 
>  SUM_ERRORS: 3
> 
>SUM_WARNINGS: 0
> 
>   SUM_ROWS_AFFECTED: 1696932
> 
>   SUM_ROWS_SENT: 0
> 
>   SUM_ROWS_EXAMINED: 0
> 
> SUM_CREATED_TMP_DISK_TABLES: 0
> 
>  SUM_CREATED_TMP_TABLES: 0
> 
>SUM_SELECT_FULL_JOIN: 0
> 
> SUM_SELECT_FULL_RANGE_JOIN: 0
> 
>SUM_SELECT_RANGE: 0
> 
>  SUM_SELECT_RANGE_CHECK: 0
> 
> SUM_SELECT_SCAN: 0
> 
>   SUM_SORT_MERGE_PASSES: 0
> 
>  SUM_SORT_RANGE: 0
> 
>   SUM_SORT_ROWS: 0
> 
>   SUM_SORT_SCAN: 0
> 
>   SUM_NO_INDEX_USED: 0
> 
>  SUM_NO_GOOD_INDEX_USED: 0
> 
>  FIRST_SEEN: 2016-10-26 14:55:39
> 
>   LAST_SEEN: 2016-10-28 07:51:04
> 
> 1 row in set (0.02 sec)
> 
> 
> 
> 
> 
> From: ntop-misc-boun...@listgateway.unipi.it [mailto:ntop-misc-
> boun...@listgateway.unipi.it] On Behalf Of Luca Deri
> Sent: Friday, 28 October 2016 5:03 AM
> To: ntop-misc@listgateway.unipi.it
> Subject: Re: [Ntop-misc] Access to historical charts very slow in ntopng
> 
> 
> 
> Peter
> 
> it is very likely that your MySQL is not fast enough.
> 
> 
> 
> Try to run the query below on your DB to see how long it took (MySQL 5.6 or
> later)
> 
> 
> 
> Luca
> 
> 
> 
> mysql> select * from  events_statements_summary_by_digest order by
> MAX_TIMER_WAIT desc limit 1 \G
> 
> *** 1. row
> ***
> 
> SCHEMA_NAME: ntopng
> 
>  DIGEST: 79669e73b0e9bcf17c7ebc9c5ba6b8de
> 
> DIGEST_TEXT: SELECT COUNT ( * ) AS `TOT_FLOWS` , SUM (
> `IN_BYTES` + `OUT_BYTES` ) AS `TOT_BYTES` , SUM ( `PACKETS` ) AS
> `TOT_PACKETS` FROM `flowsv4` WHERE `FIRST_SWITCHED` <= ? AND
> `FIRST_SWITCHED` >= ? AND ( `NTOPNG_INSTANCE_NAME` = ? OR
> `NTOPNG_INSTANCE_NAME` IS NULL ) AND ( `INTERFACE_ID` = ? ) AND
> `L7_PROTO` = ? AND ( `IP_SRC_ADDR` = `INET_ATON` (?) OR `IP_DST_ADDR`
> = `INET_ATON` (?) )
> 
>  COUNT_STAR: 37
> 
>  SUM_TIMER_WAIT: 247554300
> 
>  MIN_TIMER_WAIT: 2207200
> 
>  AVG_TIMER_WAIT: 66906567000
> 
>  MAX_TIMER_WAIT: 47117300
> 
>   SUM_LOCK_TIME: 1040700
> 
>  SUM_ERRORS: 0
> 
>SUM_WARNINGS: 0
> 
>   SUM_ROWS_AFFECTED: 0
> 
>   SUM_ROWS_SENT: 37
> 
>   SUM_ROWS_EXAMINED: 817254
> 
> SUM_CREATED_TMP_DISK_TABLES: 0
> 
>  SUM_CREATED_TMP_TABLES: 0
> 
>SUM_SELECT_FULL_JOIN: 0
> 
>  SUM_SELECT_FULL_RANGE_JOIN: 0
> 
>SUM_SELECT_RANGE: 1
> 
>  SUM_SELECT_RANGE_CHECK: 0
> 
> SUM_SELECT_SCAN: 34
> 
>   SUM_SORT_MERGE_PASSES: 0
> 
>  SUM_SORT_RANGE: 0
> 
>   SUM_SORT_ROWS: 0
> 
>   SUM_SORT_SCAN: 0
> 
>   SUM_NO_INDEX_USED: 34
> 
>  SUM_NO_GOOD_INDEX_USED: 0
> 
>  FIRST_SEEN: 2016-10-26 16:11:04
> 
>   LAST_SEEN: 2016-10-26 16:43:11
> 
> 1 row in set (0.00 sec)
> 
> 
> 
> 
> 
> 
> 
>   On 27 Oct 2016, at 01:26, Peter Shute   > wrote:
> 
> 
> 
>   In the charts tab of the Interfaces section, we can choose time
> ranges between 5 minutes and 1 year to display the traffic levels on a chart. 
> I
> haven't been able to get the chart to display more than one week of data. If I
> ask it to display two weeks, it waits for a very long time then seems to give
> up.
> 
>   I assume it's having trouble querying

Re: [Ntop-misc] Access to historical charts very slow in ntopng

2016-10-27 Thread Peter Shute 
Do I have to run "use performance_schema;" first? I'm new to mysql. It took 
0.02 seconds, is that fast enough?
*** 1. row ***
SCHEMA_NAME: ntopng
 DIGEST: 58bffbb800986c1b1147f462d49802e5
DIGEST_TEXT: INSERT INTO `flowsv4` ( `VLAN_ID` , `L7_PROTO` , 
`IP_SRC_ADDR` , `L4_SRC_PORT` , `IP_DST_ADDR` , `L4_DST_PORT` , `PROTOCOL` , 
`IN_BYTES` , `OUT_BYTES` , `PACKETS` , `FIRST_SWITCHED` , `LAST_SWITCHED` , 
`INFO` , JSON , `NTOPNG_INSTANCE_NAME` , `INTERFACE_ID` , PROFILE ) VALUES ( ?, 
... , `COMPRESS` (?) , ?, ... )
 COUNT_STAR: 1696935
 SUM_TIMER_WAIT: 6731129622996000
 MIN_TIMER_WAIT: 220633000
 AVG_TIMER_WAIT: 3966639000
 MAX_TIMER_WAIT: 665490453714000
  SUM_LOCK_TIME: 521387755000
 SUM_ERRORS: 3
   SUM_WARNINGS: 0
  SUM_ROWS_AFFECTED: 1696932
  SUM_ROWS_SENT: 0
  SUM_ROWS_EXAMINED: 0
SUM_CREATED_TMP_DISK_TABLES: 0
 SUM_CREATED_TMP_TABLES: 0
   SUM_SELECT_FULL_JOIN: 0
SUM_SELECT_FULL_RANGE_JOIN: 0
   SUM_SELECT_RANGE: 0
 SUM_SELECT_RANGE_CHECK: 0
SUM_SELECT_SCAN: 0
  SUM_SORT_MERGE_PASSES: 0
 SUM_SORT_RANGE: 0
  SUM_SORT_ROWS: 0
  SUM_SORT_SCAN: 0
  SUM_NO_INDEX_USED: 0
 SUM_NO_GOOD_INDEX_USED: 0
 FIRST_SEEN: 2016-10-26 14:55:39
  LAST_SEEN: 2016-10-28 07:51:04
1 row in set (0.02 sec)


From: ntop-misc-boun...@listgateway.unipi.it 
[mailto:ntop-misc-boun...@listgateway.unipi.it] On Behalf Of Luca Deri
Sent: Friday, 28 October 2016 5:03 AM
To: ntop-misc@listgateway.unipi.it
Subject: Re: [Ntop-misc] Access to historical charts very slow in ntopng

Peter
it is very likely that your MySQL is not fast enough.

Try to run the query below on your DB to see how long it took (MySQL 5.6 or 
later)

Luca

mysql> select * from  events_statements_summary_by_digest order by 
MAX_TIMER_WAIT desc limit 1 \G
*** 1. row ***
SCHEMA_NAME: ntopng
 DIGEST: 79669e73b0e9bcf17c7ebc9c5ba6b8de
DIGEST_TEXT: SELECT COUNT ( * ) AS `TOT_FLOWS` , SUM ( 
`IN_BYTES` + `OUT_BYTES` ) AS `TOT_BYTES` , SUM ( `PACKETS` ) AS `TOT_PACKETS` 
FROM `flowsv4` WHERE `FIRST_SWITCHED` <= ? AND `FIRST_SWITCHED` >= ? AND ( 
`NTOPNG_INSTANCE_NAME` = ? OR `NTOPNG_INSTANCE_NAME` IS NULL ) AND ( 
`INTERFACE_ID` = ? ) AND `L7_PROTO` = ? AND ( `IP_SRC_ADDR` = `INET_ATON` (?) 
OR `IP_DST_ADDR` = `INET_ATON` (?) )
 COUNT_STAR: 37
 SUM_TIMER_WAIT: 247554300
 MIN_TIMER_WAIT: 2207200
 AVG_TIMER_WAIT: 66906567000
 MAX_TIMER_WAIT: 47117300
  SUM_LOCK_TIME: 1040700
 SUM_ERRORS: 0
   SUM_WARNINGS: 0
  SUM_ROWS_AFFECTED: 0
  SUM_ROWS_SENT: 37
  SUM_ROWS_EXAMINED: 817254
SUM_CREATED_TMP_DISK_TABLES: 0
 SUM_CREATED_TMP_TABLES: 0
   SUM_SELECT_FULL_JOIN: 0
 SUM_SELECT_FULL_RANGE_JOIN: 0
   SUM_SELECT_RANGE: 1
 SUM_SELECT_RANGE_CHECK: 0
SUM_SELECT_SCAN: 34
  SUM_SORT_MERGE_PASSES: 0
 SUM_SORT_RANGE: 0
  SUM_SORT_ROWS: 0
  SUM_SORT_SCAN: 0
  SUM_NO_INDEX_USED: 34
 SUM_NO_GOOD_INDEX_USED: 0
 FIRST_SEEN: 2016-10-26 16:11:04
  LAST_SEEN: 2016-10-26 16:43:11
1 row in set (0.00 sec)



On 27 Oct 2016, at 01:26, Peter Shute 
> wrote:

In the charts tab of the Interfaces section, we can choose time ranges between 
5 minutes and 1 year to display the traffic levels on a chart. I haven't been 
able to get the chart to display more than one week of data. If I ask it to 
display two weeks, it waits for a very long time then seems to give up.

I assume it's having trouble querying the mysql database for that much data. 
Are there any tests I can do to prove this, and is there anything I can do to 
speed it up?

The mysql I installed still has all the default configuration settings. It's 
running on a recent version of Ubuntu server.

Peter Shute
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Access to historical charts very slow in ntopng

2016-10-27 Thread Luca Deri 
Peter
it is very likely that your MySQL is not fast enough.

Try to run the query below on your DB to see how long it took (MySQL 5.6 or 
later)

Luca

mysql> select * from  events_statements_summary_by_digest order by 
MAX_TIMER_WAIT desc limit 1 \G
*** 1. row ***
SCHEMA_NAME: ntopng
 DIGEST: 79669e73b0e9bcf17c7ebc9c5ba6b8de
DIGEST_TEXT: SELECT COUNT ( * ) AS `TOT_FLOWS` , SUM ( 
`IN_BYTES` + `OUT_BYTES` ) AS `TOT_BYTES` , SUM ( `PACKETS` ) AS `TOT_PACKETS` 
FROM `flowsv4` WHERE `FIRST_SWITCHED` <= ? AND `FIRST_SWITCHED` >= ? AND ( 
`NTOPNG_INSTANCE_NAME` = ? OR `NTOPNG_INSTANCE_NAME` IS NULL ) AND ( 
`INTERFACE_ID` = ? ) AND `L7_PROTO` = ? AND ( `IP_SRC_ADDR` = `INET_ATON` (?) 
OR `IP_DST_ADDR` = `INET_ATON` (?) )
 COUNT_STAR: 37
 SUM_TIMER_WAIT: 247554300
 MIN_TIMER_WAIT: 2207200
 AVG_TIMER_WAIT: 66906567000
 MAX_TIMER_WAIT: 47117300
  SUM_LOCK_TIME: 1040700
 SUM_ERRORS: 0
   SUM_WARNINGS: 0
  SUM_ROWS_AFFECTED: 0
  SUM_ROWS_SENT: 37
  SUM_ROWS_EXAMINED: 817254
SUM_CREATED_TMP_DISK_TABLES: 0
 SUM_CREATED_TMP_TABLES: 0
   SUM_SELECT_FULL_JOIN: 0
 SUM_SELECT_FULL_RANGE_JOIN: 0
   SUM_SELECT_RANGE: 1
 SUM_SELECT_RANGE_CHECK: 0
SUM_SELECT_SCAN: 34
  SUM_SORT_MERGE_PASSES: 0
 SUM_SORT_RANGE: 0
  SUM_SORT_ROWS: 0
  SUM_SORT_SCAN: 0
  SUM_NO_INDEX_USED: 34
 SUM_NO_GOOD_INDEX_USED: 0
 FIRST_SEEN: 2016-10-26 16:11:04
  LAST_SEEN: 2016-10-26 16:43:11
1 row in set (0.00 sec)



> On 27 Oct 2016, at 01:26, Peter Shute  wrote:
> 
> In the charts tab of the Interfaces section, we can choose time ranges 
> between 5 minutes and 1 year to display the traffic levels on a chart. I 
> haven't been able to get the chart to display more than one week of data. If 
> I ask it to display two weeks, it waits for a very long time then seems to 
> give up.
> 
> I assume it's having trouble querying the mysql database for that much data. 
> Are there any tests I can do to prove this, and is there anything I can do to 
> speed it up?
> 
> The mysql I installed still has all the default configuration settings. It's 
> running on a recent version of Ubuntu server.
> 
> Peter Shute
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc