Hi Evgeniy, This is great to know. aggregate_filter is quicker for the simple reason it is a giant filter whereas pre_tag_map makes you specify a collection of filters. I would add to the equation: don't under estimate maintainability, which is much better with pre_tag_map: ie. you can reload pre_tag_map content at runtime whereas you have to restart pmacct in order to reload an aggregate_filter.
Cheers, Paolo On Tue, Sep 25, 2012 at 12:27:22AM +0300, Evgeniy Kozhuhovskiy wrote: > I've solved this issue using aggregate_filter[]. > I wrote simple script that generates pcap filter from IX networks > list: http://download.datahata.by/parse.php.txt > > But the only one question: is there any difference between > aggregate_filter and pre_tag_map > inside pmacctd? Which one is faster? Or it doesn't matter in my case? > > On Fri, Sep 21, 2012 at 4:32 PM, Evgeniy Kozhuhovskiy > <[email protected]> wrote: > > I have a list of networks in our IX: http://noc.datahata.by/free.txt > > > > Our customer networks are 31.130.200.0/21 and 178.172.181.0/24 > > > > I wrote a script that generates from IX networks list filter like this: > > > > debian:~# ./parse | head > > id=1 ip=31.130.xxx.xxx filter='dst net 31.24.88.0/21 and not dst net > > 31.130.200.0/21' > > id=1 ip=31.130.xxx.xxx filter='dst net 31.24.88.0/21 and not dst net > > 178.172.181.0/24' > > id=1 ip=31.130.xxx.xxx filter='dst net 31.130.200.0/21 and not dst net > > 31.130.200.0/21' > > id=1 ip=31.130.xxx.xxx filter='dst net 31.130.200.0/21 and not dst net > > 178.172.181.0/24' > > id=1 ip=31.130.xxx.xxx filter='dst net 46.28.96.0/21 and not dst net > > 31.130.200.0/21' > > id=1 ip=31.130.xxx.xxx filter='dst net 46.28.96.0/21 and not dst net > > 178.172.181.0/24' > > id=1 ip=31.130.xxx.xxx filter='dst net 46.53.128.0/17 and not dst net > > 31.130.200.0/21' > > id=1 ip=31.130.xxx.xxx filter='dst net 46.53.128.0/17 and not dst net > > 178.172.181.0/24' > > id=1 ip=31.130.xxx.xxx filter='dst net 46.56.0.0/16 and not dst net > > 31.130.200.0/21' > > id=1 ip=31.130.xxx.xxx filter='dst net 46.56.0.0/16 and not dst net > > 178.172.181.0/24' > > > > Is this correct? > > > > On Wed, Sep 19, 2012 at 11:01 AM, Paolo Lucente <[email protected]> wrote: > >> Hi Evgeniy, > >> > >> It might be better (more maintanable) to use a different approach > >> for your task. Remove the aggregate_filter and go for pre-tagging: > >> > >> pre_tag_map: /path/to/pretag.map > >> pre_tag_filter[bla]: 0 > >> refresh_maps: true > >> pre_tag_map_entries: <say 1000? it's an upper bound anyway> > >> > >> It means filter in traffic tagged as 0 (no tag effectively). Then > >> build /path/to/pretag.map as follows (with the idea what is being > >> tagged with id == 1 within the map is then being discarded): > >> > >> id=1 ip=<NetFlow exporter IP address> filter='dst net <IX net, ie. > >> 192.168.0.0/16> and not dst net <local net, ie. 192.168.100.0/24>' > >> id=1 ip=<NetFlow exporter IP address> filter='src net <IX net, ie. > >> 192.168.0.0/16> and not src net <local net, ie. 192.168.100.0/24>' > >> ... > >> > >> You can indeed merge the two rules above in a single one, i did > >> not do it just for readability. My current understanding of your > >> goal is that you want to bill customer/customer traffic but not > >> customer/IX and vice-versa. > >> > >> Cheers, > >> Paolo > >> > >> On Wed, Sep 19, 2012 at 12:19:45AM +0300, Evgeniy Kozhuhovskiy wrote: > >>> Thanks a lot, it works for now. > >>> > >>> Also i have another question, related to pcap filter. > >>> > >>> I have a list of networks in our IX. It has some big networks that > >>> includes small networks > >>> that are announced from our AS. > >>> For example, we have 192.168.0.0/16 and 172.16.0.0/16 in IX, > >>> but 192.168.100.0/24 and 172.16.100.0/24 are our clients networks. > >>> > >>> Of course, we need to bill traffic from/to our clients > >>> (192.168.100/24, 172.16.100.0/24), but we dont need to bill > >>> traffic to 192.168.0.0/16 and 172.16.0.0/16. > >>> > >>> I wrote such rule: > >>> aggregate_filter[dsmgr]: (net not 192.168.0.0/16 and net not > >>> 172.16.0.0/16) and (net 192.168.100/24 or net 172.16.100.0/24) > >>> > >>> but this filter returns no traffic at all (because it removes our > >>> customers networks from result). Where I'm wrong in filter? > >>> > >>> On Tue, Sep 18, 2012 at 7:05 PM, Paolo Lucente <[email protected]> wrote: > >>> > > >>> > Hi Evgeniy, > >>> > > >>> > You are doing all good except aggregate_filter need to be bound to > >>> > a specific plugin, can't be global. The daemon is surely spitting an > >>> > error out about that. Please rewrite the two following lines and try > >>> > again: > >>> > > >>> > plugins: print[bla] > >>> > aggregate_filter[bla]: net not 178.120.0.0/13 > >>> > > >>> > It should be that the daemon is logging errors somewhere, you might > >>> > want to add a 'logfile' directive for catching errors, etc. > >>> > > >>> > Cheers, > >>> > Paolo > >>> > > >>> > On Tue, Sep 18, 2012 at 04:38:47PM +0300, Evgeniy Kozhuhovskiy wrote: > >>> > > Hello. > >>> > > > >>> > > We're using some proprietary software (dsmgr by ispsystems), that uses > >>> > > nfacctd as it's part to calculate traffic usage (by netflow/sflow). > >>> > > > >>> > > By default dsmgr generate such nfacctd config: > >>> > > daemonize: true > >>> > > plugins: print > >>> > > aggregate: src_host,dst_host > >>> > > nfacctd_port: 9995 > >>> > > print_refresh_time: 900 > >>> > > print_output: csv > >>> > > print_output_file: /var/flowstat/%Y:%m:%d-%H:%M > >>> > > > >>> > > and csv file is processed via proprietary utility runned from cron. > >>> > > > >>> > > We need to exclude some networks (in fact, our local IX) from billing. > >>> > > I've added this line: > >>> > > aggregate_filter: net not 178.120.0.0/13 > >>> > > to config. (178.120.0.0/13 is one of networks that we dont need to > >>> > > bill) > >>> > > > >>> > > But traffic from this network still goes to csv file: > >>> > > > >>> > > debian:/var/flowstat# cat 2012\:09\:18-16\:23 | grep 178.120 | head -4 > >>> > > > >>> > > 0,0,unknown,00:00:00:00:00:00,00:00:00:00:00:00,0,0,0,0,0,0,,0,0,0,0,,,0,0,0:0:0,178.120.88.201,178.172.181.110,0,0,0,0,0,ip,0,1072,0,75723 > >>> > > > >>> > > 0,0,unknown,00:00:00:00:00:00,00:00:00:00:00:00,0,0,0,0,0,0,,0,0,0,0,,,0,0,0:0:0,178.172.181.128,178.120.217.213,0,0,0,0,0,ip,0,22085,0,16089127 > >>> > > > >>> > > 0,0,unknown,00:00:00:00:00:00,00:00:00:00:00:00,0,0,0,0,0,0,,0,0,0,0,,,0,0,0:0:0,178.172.181.95,178.120.60.190,0,0,0,0,0,ip,0,1695,0,159303 > >>> > > > >>> > > 0,0,unknown,00:00:00:00:00:00,00:00:00:00:00:00,0,0,0,0,0,0,,0,0,0,0,,,0,0,0:0:0,178.120.112.234,178.172.181.64,0,0,0,0,0,ip,0,1326,0,120156 > >>> > > > >>> > > Where i'm wrong? > >>> > >>> > >>> -- > >>> With best regards, Evgeniy Kozhuhovskiy > > > > > > > > -- > > With best regards, Evgeniy Kozhuhovskiy > > > > -- > With best regards, Evgeniy Kozhuhovskiy _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
