Re: [pmacct-discussion] protocol classification don't detect http
I see all of those signatures actually working by picking some sites randomly with wget. This is with 0.12.0rc3 but honestly speaking there has not been any major work related to the classification part for the past 3-4 years. I try wget also ;) my version is Promiscuous Mode Accounting Daemon, pmacctd 0.12.0rc2 config DEBUG ( /etc/pmacct/pmacctd.conf ): plugin name/type: 'default'/'core'. DEBUG ( /etc/pmacct/pmacctd.conf ): plugin name/type: 'aggip'/'mysql'. DEBUG ( /etc/pmacct/pmacctd.conf ): plugin name/type: 'class'/'mysql'. DEBUG ( /etc/pmacct/pmacctd.conf ): daemonize:false DEBUG ( /etc/pmacct/pmacctd.conf ): syslog:daemon DEBUG ( /etc/pmacct/pmacctd.conf ): debug:true DEBUG ( /etc/pmacct/pmacctd.conf ): interface:intbi DEBUG ( /etc/pmacct/pmacctd.conf ): classifiers:/var/local/pmacct/classifiers DEBUG ( /etc/pmacct/pmacctd.conf ): snaplen:800 DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate[aggip]:src_host,dst_host DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate_filter[aggip]:dst net 10.1.0.0/18 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_host[aggip]:10.1.10.60 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_db[aggip]:bwstat_int DEBUG ( /etc/pmacct/pmacctd.conf ): sql_user[aggip]:bwstat_int DEBUG ( /etc/pmacct/pmacctd.conf ): sql_passwd[aggip]:avdmagco DEBUG ( /etc/pmacct/pmacctd.conf ): sql_table_version[aggip]:1 DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate[class]:src_host, dst_host, class DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate_filter[class]:dst net 10.1.0.0/18 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_host[class]:10.1.10.60 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_db[class]:pm_class DEBUG ( /etc/pmacct/pmacctd.conf ): sql_user[class]:pm_class DEBUG ( /etc/pmacct/pmacctd.conf ): sql_passwd[class]:Loolhyt7 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_table_version[class]:5 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_refresh_time:120 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_history:1h DEBUG ( /etc/pmacct/pmacctd.conf ): sql_history_roundoff:h DEBUG ( /etc/pmacct/pmacctd.conf ): debug:true wget: --2009-11-17 15:19:01-- http://ya.ru/ Resolving ya.ru... 93.158.134.8, 213.180.204.8, 77.88.21.8 Connecting to ya.ru|93.158.134.8|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4908 (4.8K) [text/html] pmacct saves it to mysql: mysql SELECT class_id,ip_src,ip_dst,ip_proto,packets,bytes,stamp_inserted FROM acct_v5 where ip_src=93.158.134.8; +--+--+---+--+-+---+-+ | class_id | ip_src | ip_dst| ip_proto | packets | bytes | stamp_inserted | +--+--+---+--+-+---+-+ | unknown | 93.158.134.8 | 10.1.4.14 | ip | 15 | 9643 | 2009-11-12 14:00:00 | | unknown | 93.158.134.8 | 10.1.4.14 | ip | 14 | 3 | 2009-11-17 15:00:00 | +--+--+---+--+-+---+-+ I would suggest a couple of checks: * see if HTTP traffic is reaped by some other classifier, but i guess you might have already checked that. if class_id = unknown, i think it's not this case. * see if the HTTP classifier is written correctly. Not referring only to the regexp but to the overall syntax. The implemented format is *veeery* sensible to tabs, spaces, white lines, etc. So try to keep it essential. Strip comments and empty lines out. I delete all comments from file [r...@router ~]# cat /var/local/pmacct/classifiers/http.pat http http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] [\x09-\x0d -~]*(connection:|content-type:|content-length:|date:)|post [\x09-\x0d -~]* http/[01]\.[019] What else may I try to? But sometimes. mysql SELECT class_id,ip_src,ip_dst,ip_proto,packets,bytes,stamp_inserted FROM acct_v5 where class_id=http; ... | http | 85.190.0.3 | 10.1.10.50 | ip | 5 | 315 | 2009-11-12 14:00:00 | | http | 209.85.229.100 | 10.1.4.14 | ip | 12 |1320 | 2009-11-12 14:00:00 | | http | 94.103.92.129 | 10.1.2.17 | ip |1324 | 1926982 | 2009-11-12 14:00:00 | | http | 64.147.188.87 | 10.1.4.12 | ip | 5 | 798 | 2009-11-12 14:00:00 | | http | 74.125.87.100 | 10.1.4.12 | ip | 7 |2121 | 2009-11-17 15:00:00 | | http | 195.2.117.115 | 10.1.4.12 | ip | 27 | 29174 | 2009-11-17 15:00:00 | | http | 74.125.87.113 | 10.1.4.14 | ip | 13 |1217 | 2009-11-17 15:00:00 | | http | 199.7.71.72| 10.1.4.12 | ip | 14 |4710 | 2009-11-17 15:00:00 | | http | 87.250.251.25 | 10.1.4.14 | ip | 71 | 85207 | 2009-11-17 15:00:00 | | http | 77.88.21.25| 10.1.4.14 | ip | 9 |1145 | 2009-11-17 15:00:00 | | http | 213.180.204.25 | 10.1.4.14 | ip | 10 |1185 | 2009-11-17 15:00:00 | | http | 93.158.134.25 | 10.1.4.14 | ip | 11 |1225 | 2009-11-17 15:00:00 |
Re: [pmacct-discussion] create my own mysql table
Hi, On Mon, Nov 16, 2009 at 04:45:57PM -0600, fedora fedora wrote: DEBUG ( default/mysql ): INSERT INTO `test_1` (stamp_updated, stamp_inserted, ip_src, ip_dst, as_src, as_dst, src_port, dst_port, tcp_flags, ip_proto, packets, bytes, flows) VALUES (FROM_UNIXTIME(1258410661), FROM_UNIXTIME(1258410600), 'x.x.x.34', 'x.x.x.2', xx8, xx9, 443, 2608, 24, 'tcp', 1, 1353, 140733193388033) Thanks, for the output. So, only the 'flows' primitive is involved. Hopefully last bit of information i need is: which NetFlow version are your routers exporting to pmacct? If v8, which profile? If v9, you doing anything fancy with it (ie. aggregated NetFlow)? Roughly a week ago i committed to the CVS a minor patch to initialize some variables used at some stage to convert values as counters; can you please see if the version currently in the CVS behaves any better? So pmacct keeps tracking the traffic count and and the end of the given minutes(hours..etc) it calculates the summary and then writes it to the backend database, am I right? Yes pmacct SQL plugins feature a cache to accumulate counters. Then the scanner kicks in at regular intervals (by default 60 secs) and writes to the database. If sql_history matches sql_refresh_time (or is a multiple) then each aggregate is written with a SQL INSERT query; otherwise UPDATE queries are involved. If I am correct, how does pmacct treat netflow data? since all the data it gets already get aggregated by netflow protocol. Will pmacct do something extra? I guess for sflow, it will act differently and do the calculations. Basic thing to consider is pmacct is not a packet/sample/flow logger. This is partly highlighted by Q5 in FAQS. It performs data reduction, ie. temporal aggregation, spatial aggregation, filtering, sampling (or sub-sampling). Timestamps part of the export protocol, ie. NetFlow or sFlow, are used to assign data to a time-bin, when using a SQL plugin (as the in-memory table plugin doesn't have such a concept; you simply grabclean data at regular intervals). Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] protocol classification don't detect http
Hi Mike, On Tue, Nov 17, 2009 at 02:27:06PM +0300, Mike Lykov wrote: I would suggest a couple of checks: * see if HTTP traffic is reaped by some other classifier, but i guess you might have already checked that. if class_id = unknown, i think it's not this case. Yes, correct. But are you getting all the web traffic? I mean, I see you are a) not collecting TCP/UDP ports and b) using an aggregate_filter. Is it web traffic the one left as unknown or something else? Any chance some web traffic is being filtered out, ie. because some mirrored data is VLAN-tagged? You can test this by commenting out the aggregate_filter. * see if the HTTP classifier is written correctly. Not referring only to the regexp but to the overall syntax. The implemented format is *veeery* sensible to tabs, spaces, white lines, etc. So try to keep it essential. Strip comments and empty lines out. I delete all comments from file [r...@router ~]# cat /var/local/pmacct/classifiers/http.pat http http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] [\x09-\x0d -~]*(connection:|content-type:|content-length:|date:)|post [\x09-\x0d -~]* http/[01]\.[019] What else may I try to? Try with a simplified (and polished up) filter. See if the memory table plugin behaves any differently/better compared to the SQL one (this is an always-good troubleshooting step). Increase classification tentatives although with http traffic it should make no difference. After all, if http-marked traffic makes it in the database, as per your previous email, it means the regexp engine itself is working. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
