Paolo, thanks for your work!

> 1) 01011970 corresponds to a UNIX timestamp of zero. The easiest way to
>    mitigate it is to use 'nfacctd_time_new: true' so to consider the
>    time of arrival at the collector (that is, nfacctd) as timestamp for

This parameter is set in the config, but the table is still created with 
timestamp of zero.

>    bucketing instead of the flow start time; alternatively - or also in
>    addition - you can look in your flow data in case there are some with
>    a flow start time of zero (and if not and you can pin point what gets
>    purged to that table, we may be facing a bug or maybe an explanation
>    for it ..).  

Except for the standard fields IPFIX in the flow there is nothing. template 
from router in attach. 
How can I make collector on the information from the template when creating a 
table? 
I do not insist, but it seems to me that when using the parameter 
nfacctd_time_new, the table should be created based on the current date. and 
it's still some kind of bug. :(
  
> 2) It is a simplistic yet harmless approach to blindly send a CREATE
>    TABLE, maybe the message back should be made a warning rather than an
>    error. It's harmless because as you see, data is purged anyway; it's
>    simplistic because it saves the effort to probe if the table exists.

i replace CREATE TABLE in schema file to CREATE TABLE IF NOT EXISTS and get 
NOTICE , not ERROR. it's good :)

also encountered an unknown problem. on every purge flowdata place to table, 
but in postgreesql log there are such error messages
2018-03-16 10:32:20.694 EET [4736] pmacct@pmacct ERROR:  invalid input syntax 
for type inet: ""
2018-03-16 10:32:20.694 EET [4736] pmacct@pmacct CONTEXT:  COPY 
traffic_01011970, line 91719, column ip_src: ""
2018-03-16 10:32:20.694 EET [4736] pmacct@pmacct STATEMENT:  COPY 
traffic_01011970 (ip_src, ip_dst, as_src, iface_in, iface_out, as_dst, 
port_src, port_dst, ip_proto, timestamp_start, timestamp_start_residual, 
timestamp_end, timestamp_end_residual, packets, bytes) FROM STDIN DELIMITER ','
2018-03-16 10:32:30.045 EET [4738] pmacct@pmacct ERROR:  invalid input syntax 
for type inet: ""
2018-03-16 10:32:30.045 EET [4738] pmacct@pmacct CONTEXT:  COPY 
traffic_01011970, line 103998, column ip_src: ""
2018-03-16 10:32:30.045 EET [4738] pmacct@pmacct STATEMENT:  COPY 
traffic_01011970 (ip_src, ip_dst, as_src, iface_in, iface_out, as_dst, 
port_src, port_dst, ip_proto, timestamp_start, timestamp_start_residual, 
timestamp_end, timestamp_end_residual, packets, bytes) FROM STDIN DELIMITER ','
2018-03-16 10:32:39.692 EET [4741] pmacct@pmacct ERROR:  invalid input syntax 
for type inet: ""
2018-03-16 10:32:39.692 EET [4741] pmacct@pmacct CONTEXT:  COPY 
traffic_01011970, line 113034, column ip_src: ""
2018-03-16 10:32:39.692 EET [4741] pmacct@pmacct STATEMENT:  COPY 
traffic_01011970 (ip_src, ip_dst, as_src, iface_in, iface_out, as_dst, 
port_src, port_dst, ip_proto, timestamp_start, timestamp_start_residual, 
timestamp_end, timestamp_end_residual, packets, bytes) FROM STDIN DELIMITER ','

each time a different line number, but the same column. is this of a problem?



> Paolo

> On Tue, Mar 13, 2018 at 04:32:52PM +0200, Andrey Koblyuk wrote:
>> Hello!

>> my config

>> aggregate[storage]                      : 
>> src_host,dst_host,src_port,dst_port,proto,src_as,dst_as,in_iface,out_iface,timestamp_start,timestamp_end
>> sql_optimize_clauses[storage]           : true
>> sql_table[storage]                      : traffic_%d%m%Y
>> sql_table_schema[storage]               : /etc/pmacct/nfacct_schema.psql
>> sql_table_type[storage]                 : bgp
>> sql_dont_try_update[storage]            : true
>> sql_use_copy[storage]                   : true

>> /etc/pmacct/nfacct_schema.psql:
>> CREATE TABLE traffic_%d%m%Y (
>>         id_key BIGSERIAL PRIMARY KEY,
>>         as_src BIGINT NOT NULL DEFAULT 0,
>>         as_dst BIGINT NOT NULL DEFAULT 0,
>>         ip_src inet NOT NULL DEFAULT '0.0.0.0',
>>         ip_dst inet NOT NULL DEFAULT '0.0.0.0',
>>         port_src INT NOT NULL DEFAULT 0,
>>         port_dst INT NOT NULL DEFAULT 0,
>>         ip_proto SMALLINT NOT NULL DEFAULT 0,
>>         iface_in INT NOT NULL,
>>         iface_out INT NOT NULL,
>>         packets INT NOT NULL,
>>         bytes BIGINT NOT NULL,
>>         timestamp_start timestamp without time zone NOT NULL DEFAULT 
>> '0001-01-01 00:00:00',
>>         timestamp_start_residual INT NOT NULL DEFAULT 0,
>>         timestamp_end timestamp without time zone NOT NULL DEFAULT 
>> '0001-01-01 00:00:00',
>>         timestamp_end_residual INT NOT NULL DEFAULT 0,
>>         stamp_inserted timestamp without time zone NOT NULL DEFAULT 
>> CURRENT_TIMESTAMP(0)
>> );
>> GRANT SELECT, INSERT, UPDATE, DELETE ON traffic_%d%m%Y TO pmacct;

>> 1) first "purge" create table traffic_01011970, but current date 13032018. 
>> Why table create on 01011970?
>> 2) next "purge" print this error in log
>> INFO ( storage/pgsql ): *** Purging cache - START (PID: 11285) ***
>> ERROR ( storage/pgsql ): ERROR:  relation "traffic_01011970" already exists
>> INFO ( storage/pgsql ): *** Purging cache - END (PID: 11285, QN: 
>> 137084/137084, ET: 1) ***

>> Why does the table try to be recreated with each purge? what parameter is 
>> responsible for this?

>> -- 
>> ANK32-RIPE


>> _______________________________________________
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists



-- 
ANK32-RIPE
DEBUG ( default/core ): NfV10 template type : flow
DEBUG ( default/core ): NfV10 template ID   : 256
DEBUG ( default/core ): 
-------------------------------------------------------------
DEBUG ( default/core ): |    pen     |         field type         | offset |  
size  |
DEBUG ( default/core ): | 0          | IPv4 src addr      [8    ] |      0 |    
  4 |
DEBUG ( default/core ): | 0          | IPv4 dst addr      [12   ] |      4 |    
  4 |
DEBUG ( default/core ): | 0          | tos                [5    ] |      8 |    
  1 |
DEBUG ( default/core ): | 0          | L4 protocol        [4    ] |      9 |    
  1 |
DEBUG ( default/core ): | 0          | L4 src port        [7    ] |     10 |    
  2 |
DEBUG ( default/core ): | 0          | L4 dst port        [11   ] |     12 |    
  2 |
DEBUG ( default/core ): | 0          | icmp type          [32   ] |     14 |    
  2 |
DEBUG ( default/core ): | 0          | input snmp         [10   ] |     16 |    
  4 |
DEBUG ( default/core ): | 0          | IPv4 src mask      [9    ] |     20 |    
  1 |
DEBUG ( default/core ): | 0          | IPv4 dst mask      [13   ] |     21 |    
  1 |
DEBUG ( default/core ): | 0          | src as             [16   ] |     22 |    
  4 |
DEBUG ( default/core ): | 0          | dst as             [17   ] |     26 |    
  4 |
DEBUG ( default/core ): | 0          | IPv4 next hop      [15   ] |     30 |    
  4 |
DEBUG ( default/core ): | 0          | tcp flags          [6    ] |     34 |    
  1 |
DEBUG ( default/core ): | 0          | output snmp        [14   ] |     35 |    
  4 |
DEBUG ( default/core ): | 0          | in bytes           [1    ] |     39 |    
  8 |
DEBUG ( default/core ): | 0          | in packets         [2    ] |     47 |    
  8 |
DEBUG ( default/core ): | 0          | 152                [152  ] |     55 |    
  8 |
DEBUG ( default/core ): | 0          | 153                [153  ] |     63 |    
  8 |
DEBUG ( default/core ): | 0          | 136                [136  ] |     71 |    
  1 |
DEBUG ( default/core ): 
-------------------------------------------------------------
DEBUG ( default/core ): Netflow V9/IPFIX record size : 72
DEBUG ( default/core ):
DEBUG ( default/core ): Received NetFlow/IPFIX packet from 
[94.125.120.65:54276] version [10] seqno [107972]
DEBUG ( default/core ): NfV10 agent         : 94.125.120.65:0
DEBUG ( default/core ): NfV10 template type : options
DEBUG ( default/core ): NfV10 template ID   : 512
DEBUG ( default/core ): 
-------------------------------------------------------------
DEBUG ( default/core ): |    pen     |         field type         | offset |  
size  |
DEBUG ( default/core ): | 0          | 144                [144  ] |      0 |    
  4 |
DEBUG ( default/core ): | 0          | 160                [160  ] |      4 |    
  8 |
DEBUG ( default/core ): | 0          | 130                [130  ] |     12 |    
  4 |
DEBUG ( default/core ): | 0          | 214                [214  ] |     16 |    
  1 |
DEBUG ( default/core ): | 0          | 215                [215  ] |     17 |    
  1 |
DEBUG ( default/core ): 
-------------------------------------------------------------
DEBUG ( default/core ): Netflow V9/IPFIX record size : 18
DEBUG ( default/core ):
DEBUG ( default/core ): Received NetFlow/IPFIX packet from 
[94.125.120.65:54276] version [10] seqno [107972]
DEBUG ( default/core ): Received NetFlow/IPFIX packet from 
[94.125.120.65:54276] version [10] seqno [107972]
DEBUG ( default/core ): NfV10 agent         : 94.125.120.65:1
DEBUG ( default/core ): NfV10 template type : options
DEBUG ( default/core ): NfV10 template ID   : 512
DEBUG ( default/core ): 
-------------------------------------------------------------
DEBUG ( default/core ): |    pen     |         field type         | offset |  
size  |
DEBUG ( default/core ): | 0          | 144                [144  ] |      0 |    
  4 |
DEBUG ( default/core ): | 0          | 160                [160  ] |      4 |    
  8 |
DEBUG ( default/core ): | 0          | 130                [130  ] |     12 |    
  4 |
DEBUG ( default/core ): | 0          | 214                [214  ] |     16 |    
  1 |
DEBUG ( default/core ): | 0          | 215                [215  ] |     17 |    
  1 |
DEBUG ( default/core ): 
-------------------------------------------------------------
DEBUG ( default/core ): Netflow V9/IPFIX record size : 18
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to