Re: [pmacct-discussion] Wait for BGP peering and massiv ipfix data
Hi Andrey, Inline: On Mon, Mar 05, 2018 at 08:51:34AM +0200, Andrey Koblyuk wrote: > > I would like to create a table once a week with a new name (for > example,acct_%w%Y), which would store non-historical data from > aggregate[storage] : > src_host,dst_host,src_port,dst_port,proto,src_as,dst_as,in_iface,out_iface > sql_optimize_clauses[storage] : true > sql_table[storage] : all_traffic > sql_table_type[storage] : bgp > sql_dont_try_update[storage]: true > sql_use_copy[storage] : true > > Will the directives sql_table_schema/sql_table_with_dynamic_name work without > sql_history_roundoff/sql_history? To aggregate flows in the weekly time bin (which i seem to understand is not what you want, so just mentioning), use the above config including a definition for sql_history and sql_history_roundoff. To not aggretate flows (which i seem to understand is what you want), then please add 'timestamp_start, timestamp_end' to your 'aggregate' line. So, in short, both behaviours are available and can be achieved. Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Wait for BGP peering and massiv ipfix data
Paolo, thanks for your reply! >> sorry typo. of course %w. The problem is that i need to rotate the table, >> but do not aggregate this table. why - to reduce the number of entries in >> the active table (after two days of working with my ipfix flow, the number >> of records in the sql_table reached ~50 millions). So the question is >> whether the table will rotate without using aggregation parameters, such as >> sql_history/sql_history_roundoff? > There is something i struggle in your elaboration: you say you wish to > not aggregate data in the table but at the same time you say it is a > problem to get ~50 millions in the table. Can you make an example to > describe your desired behaviour? I would like to create a table once a week with a new name (for example,acct_%w%Y), which would store non-historical data from aggregate[storage] : src_host,dst_host,src_port,dst_port,proto,src_as,dst_as,in_iface,out_iface sql_optimize_clauses[storage] : true sql_table[storage] : all_traffic sql_table_type[storage] : bgp sql_dont_try_update[storage]: true sql_use_copy[storage] : true Will the directives sql_table_schema/sql_table_with_dynamic_name work without sql_history_roundoff/sql_history? -- ANK32-RIPE ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Wait for BGP peering and massiv ipfix data
Hi Paolo! >> 1) sql_startup_delay does not work for me . I would like to postpone the >> first data processing/cache purging before BGP peering is up. otherwise the >> table contains data without information from BGP daemon. > Correlation happens when flow data is received - not upon purge. So > simply delaying the purge is not going to benefit. I wonder, is it such > a big deal though? How often are you restarting the collector and why? is there a possibility to run a BGP demon standalone earlier than nfacctd? with my current configuration, the following occurs - on begin run nfaccd : NFO ( storage/pgsql ): cache entries=131084 base cache memory=47286240 bytes INFO ( storage/pgsql ): *** Purging cache - START (PID: 12464) *** INFO ( storage/pgsql ): *** Purging cache - END (PID: 12464, QN: 51217/51217, ET: 0) *** INFO ( storage/pgsql ): Finished cache entries (ie. sql_cache_entries). Purging. INFO ( storage/pgsql ): *** Purging cache - START (PID: 12466) *** INFO ( storage/pgsql ): *** Purging cache - END (PID: 12466, QN: 137084/137084, ET: 0) *** INFO ( default/core/BGP ): [MY_RR_IP_ADDR] BGP peers usage: 1/4 INFO ( default/core/BGP ): [MY_RR_IP_ADDR] BGP_OPEN: Local AS: 12593 Remote AS: 12593 HoldTime: 90 can not get into BGP "holdtime event" receive from RR. in "production" - collector always run, does not restart. a problem only at the start of work (for the time of tests several times it was necessary to observe such a problem). >> 2) is it possible to rotate tables, for example, once a week, simply by >> creating a new one with a new name (example all_traffic_%m) ? In this case, >> no aggregation should occur - therefore the parameters are not suitable >> for me if I correctly understood their purpose. > Please elaborate more. 'sql_table: acct_v4_%w' should achieve you the > rotation behaviour you desire. I loose you on the aggregation point you > make. sorry typo. of course %w. The problem is that i need to rotate the table, but do not aggregate this table. why - to reduce the number of entries in the active table (after two days of working with my ipfix flow, the number of records in the sql_table reached ~50 millions). So the question is whether the table will rotate without using aggregation parameters, such as sql_history/sql_history_roundoff? -- ANK32-RIPE ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Wait for BGP peering and massiv ipfix data
Hi Andrey, Inline: On Fri, Mar 02, 2018 at 02:06:30PM +0200, Andrey Koblyuk wrote: > 1) sql_startup_delay does not work for me . I would like to postpone the > first data processing/cache purging before BGP peering is up. otherwise the > table contains data without information from BGP daemon. Correlation happens when flow data is received - not upon purge. So simply delaying the purge is not going to benefit. I wonder, is it such a big deal though? How often are you restarting the collector and why? > 2) is it possible to rotate tables, for example, once a week, simply by > creating a new one with a new name (example all_traffic_%m) ? In this case, > no aggregation should occur - therefore the parameters are not suitable > sql_history: 1h > sql_history_roundoff: h > sql_table: acct_v4_%w > for me if I correctly understood their purpose. Please elaborate more. 'sql_table: acct_v4_%w' should achieve you the rotation behaviour you desire. I loose you on the aggregation point you make. > 3) I also noticed a "strange" delay between "Purging cache - END" and psql > "select * from all_traffic;". about a minute.. O_O this is a question for > nfacctd or postgresql? I'd definitely need more info on this as it's too generic description. If you suspect a bug and can provide more info, please follow-up by unicast email. > 4) Interested in your recommendations - what parameters should be changed > with a large flow of ipfix data, taking into account the fact that I plan to > add plug-ins for aggregating this data to the existing pgsql[storage]. Mainly buffers (ie. plugin_pipe_szie, plugin_buffer_size or as an alternative plugin_pipe_zmq along with plugin_pipe_zmq_profile) and, depending on your aggregation method, the amount of cache entries (ie. sql_cache_entries). On the buffering part you can read the following: https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2153-L2168 https://github.com/pmacct/pmacct/blob/master/CONFIG-KEYS#L286-L331 Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
