Re: [pmacct-discussion] 95 percentile (again)
Hi Klaas, Yes, you can set networks_file_filter to true (by default false): https://github.com/pmacct/pmacct/blob/1.7.5/CONFIG-KEYS#L874=#L880 Paolo On 03/11/2020 12:47, Klaas Tammling wrote: Hi Paolo, thanks. Recording of data seems to work so far. However I want to clean up all this a bit and what I noticed is even I only have one IPv4 prefix inside my networks_file the nfacctd is also dumping IPv6 information into my database. Is there any additional filter I can set to exclude everything which is not defined in my networks_file? Thanks. - Klaas *Von:* Paolo Lucente *Gesendet:* Dienstag, 3. November 2020 00:42 *An:* pmacct-discussion@pmacct.net ; Klaas Tammling *Betreff:* Re: [pmacct-discussion] 95 percentile (again) Hi Klaas, You are right pmacct does not do 95th percentile calculations as these are much better suited to be post-process actions (due to the increased data visibility they require) than done in-line at the collector layer. On your question about bits/s. 95th percentile bases on the assumption you do bucket your data. One min buckets, 5 mins buckets, 1 hour buckets, etc. You make it a discrete exercise where essentially you say for those, say, 5 mins that is the amount of bytes accounted for. This is what pmacct does for you. Then you take the 95th highest measurement of the buckets within a time frame of choice an hour, a day, a week, a month, etc. So then let us say you decide to go for 5 mins buckets, you would just need to do for the winning bucket a "bytes * 8 / 300" operation to convert bytes to bits (* 8) and then divide by the amount of seconds in the bucket (/ 300). Paolo On 02/11/2020 13:07, Klaas Tammling wrote: Hi, this year I'm trying to give pmacct a try for some 95 percentile calculation. I understood that pmacct doesn't do the calculation by itself however it can assist in collecting the needed data. I understood the following: plugins: pgsql[in], pgsql[out] sql_table[in]: acct_in sql_table[out]: acct_out aggregate[in]: dst_host aggregate[out]: src_host sql_history: 1h sql_history_roundoff: h By changing sql_history to 5m I can get the 5 minute aggregate of the received data. By setting a networks_file (networks_file: ...) I would be able to only collect data for networks I'm defining in that list. My question would be now if there is a way to record the bits/s for that flow for the given timestamp. Or am I completely wrong with my assumptions? Thanks very much for any help. - Klaas ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] 95 percentile (again)
Hi Paolo, thanks. Recording of data seems to work so far. However I want to clean up all this a bit and what I noticed is even I only have one IPv4 prefix inside my networks_file the nfacctd is also dumping IPv6 information into my database. Is there any additional filter I can set to exclude everything which is not defined in my networks_file? Thanks. - Klaas Von: Paolo Lucente Gesendet: Dienstag, 3. November 2020 00:42 An: pmacct-discussion@pmacct.net ; Klaas Tammling Betreff: Re: [pmacct-discussion] 95 percentile (again) Hi Klaas, You are right pmacct does not do 95th percentile calculations as these are much better suited to be post-process actions (due to the increased data visibility they require) than done in-line at the collector layer. On your question about bits/s. 95th percentile bases on the assumption you do bucket your data. One min buckets, 5 mins buckets, 1 hour buckets, etc. You make it a discrete exercise where essentially you say for those, say, 5 mins that is the amount of bytes accounted for. This is what pmacct does for you. Then you take the 95th highest measurement of the buckets within a time frame of choice an hour, a day, a week, a month, etc. So then let us say you decide to go for 5 mins buckets, you would just need to do for the winning bucket a "bytes * 8 / 300" operation to convert bytes to bits (* 8) and then divide by the amount of seconds in the bucket (/ 300). Paolo On 02/11/2020 13:07, Klaas Tammling wrote: > Hi, > > this year I'm trying to give pmacct a try for some 95 percentile > calculation. I understood that pmacct doesn't do the calculation by > itself however it can assist in collecting the needed data. > > I understood the following: > > plugins: pgsql[in], pgsql[out] > sql_table[in]: acct_in > sql_table[out]: acct_out > aggregate[in]: dst_host > aggregate[out]: src_host > sql_history: 1h > sql_history_roundoff: h > > By changing sql_history to 5m I can get the 5 minute aggregate of the > received data. > > By setting a networks_file (networks_file: ...) I would be able to only > collect data for networks I'm defining in that list. > > My question would be now if there is a way to record the bits/s for that > flow for the given timestamp. > > Or am I completely wrong with my assumptions? > > Thanks very much for any help. > > - > Klaas > > > > ___ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists > ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] 95 percentile (again)
Hi Klaas, You are right pmacct does not do 95th percentile calculations as these are much better suited to be post-process actions (due to the increased data visibility they require) than done in-line at the collector layer. On your question about bits/s. 95th percentile bases on the assumption you do bucket your data. One min buckets, 5 mins buckets, 1 hour buckets, etc. You make it a discrete exercise where essentially you say for those, say, 5 mins that is the amount of bytes accounted for. This is what pmacct does for you. Then you take the 95th highest measurement of the buckets within a time frame of choice an hour, a day, a week, a month, etc. So then let us say you decide to go for 5 mins buckets, you would just need to do for the winning bucket a "bytes * 8 / 300" operation to convert bytes to bits (* 8) and then divide by the amount of seconds in the bucket (/ 300). Paolo On 02/11/2020 13:07, Klaas Tammling wrote: Hi, this year I'm trying to give pmacct a try for some 95 percentile calculation. I understood that pmacct doesn't do the calculation by itself however it can assist in collecting the needed data. I understood the following: plugins: pgsql[in], pgsql[out] sql_table[in]: acct_in sql_table[out]: acct_out aggregate[in]: dst_host aggregate[out]: src_host sql_history: 1h sql_history_roundoff: h By changing sql_history to 5m I can get the 5 minute aggregate of the received data. By setting a networks_file (networks_file: ...) I would be able to only collect data for networks I'm defining in that list. My question would be now if there is a way to record the bits/s for that flow for the given timestamp. Or am I completely wrong with my assumptions? Thanks very much for any help. - Klaas ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
