Re: [pmacct-discussion] protocol classification don't detect http
I see all of those signatures actually working by picking some sites randomly with wget. This is with 0.12.0rc3 but honestly speaking there has not been any major work related to the classification part for the past 3-4 years. I try wget also ;) my version is Promiscuous Mode Accounting Daemon, pmacctd 0.12.0rc2 config DEBUG ( /etc/pmacct/pmacctd.conf ): plugin name/type: 'default'/'core'. DEBUG ( /etc/pmacct/pmacctd.conf ): plugin name/type: 'aggip'/'mysql'. DEBUG ( /etc/pmacct/pmacctd.conf ): plugin name/type: 'class'/'mysql'. DEBUG ( /etc/pmacct/pmacctd.conf ): daemonize:false DEBUG ( /etc/pmacct/pmacctd.conf ): syslog:daemon DEBUG ( /etc/pmacct/pmacctd.conf ): debug:true DEBUG ( /etc/pmacct/pmacctd.conf ): interface:intbi DEBUG ( /etc/pmacct/pmacctd.conf ): classifiers:/var/local/pmacct/classifiers DEBUG ( /etc/pmacct/pmacctd.conf ): snaplen:800 DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate[aggip]:src_host,dst_host DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate_filter[aggip]:dst net 10.1.0.0/18 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_host[aggip]:10.1.10.60 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_db[aggip]:bwstat_int DEBUG ( /etc/pmacct/pmacctd.conf ): sql_user[aggip]:bwstat_int DEBUG ( /etc/pmacct/pmacctd.conf ): sql_passwd[aggip]:avdmagco DEBUG ( /etc/pmacct/pmacctd.conf ): sql_table_version[aggip]:1 DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate[class]:src_host, dst_host, class DEBUG ( /etc/pmacct/pmacctd.conf ): aggregate_filter[class]:dst net 10.1.0.0/18 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_host[class]:10.1.10.60 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_db[class]:pm_class DEBUG ( /etc/pmacct/pmacctd.conf ): sql_user[class]:pm_class DEBUG ( /etc/pmacct/pmacctd.conf ): sql_passwd[class]:Loolhyt7 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_table_version[class]:5 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_refresh_time:120 DEBUG ( /etc/pmacct/pmacctd.conf ): sql_history:1h DEBUG ( /etc/pmacct/pmacctd.conf ): sql_history_roundoff:h DEBUG ( /etc/pmacct/pmacctd.conf ): debug:true wget: --2009-11-17 15:19:01-- http://ya.ru/ Resolving ya.ru... 93.158.134.8, 213.180.204.8, 77.88.21.8 Connecting to ya.ru|93.158.134.8|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4908 (4.8K) [text/html] pmacct saves it to mysql: mysql SELECT class_id,ip_src,ip_dst,ip_proto,packets,bytes,stamp_inserted FROM acct_v5 where ip_src=93.158.134.8; +--+--+---+--+-+---+-+ | class_id | ip_src | ip_dst| ip_proto | packets | bytes | stamp_inserted | +--+--+---+--+-+---+-+ | unknown | 93.158.134.8 | 10.1.4.14 | ip | 15 | 9643 | 2009-11-12 14:00:00 | | unknown | 93.158.134.8 | 10.1.4.14 | ip | 14 | 3 | 2009-11-17 15:00:00 | +--+--+---+--+-+---+-+ I would suggest a couple of checks: * see if HTTP traffic is reaped by some other classifier, but i guess you might have already checked that. if class_id = unknown, i think it's not this case. * see if the HTTP classifier is written correctly. Not referring only to the regexp but to the overall syntax. The implemented format is *veeery* sensible to tabs, spaces, white lines, etc. So try to keep it essential. Strip comments and empty lines out. I delete all comments from file [r...@router ~]# cat /var/local/pmacct/classifiers/http.pat http http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] [\x09-\x0d -~]*(connection:|content-type:|content-length:|date:)|post [\x09-\x0d -~]* http/[01]\.[019] What else may I try to? But sometimes. mysql SELECT class_id,ip_src,ip_dst,ip_proto,packets,bytes,stamp_inserted FROM acct_v5 where class_id=http; ... | http | 85.190.0.3 | 10.1.10.50 | ip | 5 | 315 | 2009-11-12 14:00:00 | | http | 209.85.229.100 | 10.1.4.14 | ip | 12 |1320 | 2009-11-12 14:00:00 | | http | 94.103.92.129 | 10.1.2.17 | ip |1324 | 1926982 | 2009-11-12 14:00:00 | | http | 64.147.188.87 | 10.1.4.12 | ip | 5 | 798 | 2009-11-12 14:00:00 | | http | 74.125.87.100 | 10.1.4.12 | ip | 7 |2121 | 2009-11-17 15:00:00 | | http | 195.2.117.115 | 10.1.4.12 | ip | 27 | 29174 | 2009-11-17 15:00:00 | | http | 74.125.87.113 | 10.1.4.14 | ip | 13 |1217 | 2009-11-17 15:00:00 | | http | 199.7.71.72| 10.1.4.12 | ip | 14 |4710 | 2009-11-17 15:00:00 | | http | 87.250.251.25 | 10.1.4.14 | ip | 71 | 85207 | 2009-11-17 15:00:00 | | http | 77.88.21.25| 10.1.4.14 | ip | 9 |1145 | 2009-11-17 15:00:00 | | http | 213.180.204.25 | 10.1.4.14 | ip | 10 |1185 | 2009-11-17 15:00:00 | | http | 93.158.134.25 | 10.1.4.14 | ip | 11 |1225 | 2009-11-17 15:00:00 |
Re: [pmacct-discussion] protocol classification don't detect http
Hi Mike, On Tue, Nov 17, 2009 at 02:27:06PM +0300, Mike Lykov wrote: I would suggest a couple of checks: * see if HTTP traffic is reaped by some other classifier, but i guess you might have already checked that. if class_id = unknown, i think it's not this case. Yes, correct. But are you getting all the web traffic? I mean, I see you are a) not collecting TCP/UDP ports and b) using an aggregate_filter. Is it web traffic the one left as unknown or something else? Any chance some web traffic is being filtered out, ie. because some mirrored data is VLAN-tagged? You can test this by commenting out the aggregate_filter. * see if the HTTP classifier is written correctly. Not referring only to the regexp but to the overall syntax. The implemented format is *veeery* sensible to tabs, spaces, white lines, etc. So try to keep it essential. Strip comments and empty lines out. I delete all comments from file [r...@router ~]# cat /var/local/pmacct/classifiers/http.pat http http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] [\x09-\x0d -~]*(connection:|content-type:|content-length:|date:)|post [\x09-\x0d -~]* http/[01]\.[019] What else may I try to? Try with a simplified (and polished up) filter. See if the memory table plugin behaves any differently/better compared to the SQL one (this is an always-good troubleshooting step). Increase classification tentatives although with http traffic it should make no difference. After all, if http-marked traffic makes it in the database, as per your previous email, it means the regexp engine itself is working. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] protocol classification don't detect http
Hi all on this list. I am try to install pmacct + protocol classification feature and want to ask some question about it. pmacct + pmacct_v5 base + set of .pat files from l7filter site. See results: successfully detect ftp,nntp,subversion,jabber,ssh,dns,pop3,smtp detect connection to jabber-icq gate as rtp detect ntp as edonkey don't detect http and http-ssl at all don't detect irc (tested on irc.freenode.org + irssi), whois (whois.ripn.net + console whois) False detections rtp/edonkey is a little inconvenience, but not to detect http at all is a big disappointment! I try some variants of regexp: simple HTTP/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] default http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] [\x09-\x0d -~]*(connection:|content-type:|content-length:|date:)|post [\x09-\x0d -~]* http/[01]\.[019] second default http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9]|post [\x09-\x0d -~]* http/[01]\.[019] I dump one session: query (minus binary part) GET / HTTP/1.1 Host: whois.kraft-s.ru User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5pre) Gecko/2008120802 Firefox/3.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: ru,en-us;q=0.7,en;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive responce (minus binary part) HTTP/1.0 200 OK Date: Thu, 12 Nov 2009 10:55:51 GMT Server: Apache/1.3.26 (Unix) mod_perl/1.27 PHP/4.2.3 Content-Type: text/html; charset=ISO-8859-1 X-Cache: MISS from router.local X-Cache-Lookup: MISS from router.local:3128 Via: 1.0 router.local (squid/3.0.STABLE19) Proxy-Connection: close !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.0 Transitional//EN - not detected. Anybody here with http classification working? ;) -- Mike ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] protocol classification don't detect http
Hi Mike, I see all of those signatures actually working by picking some sites randomly with wget. This is with 0.12.0rc3 but honestly speaking there has not been any major work related to the classification part for the past 3-4 years. I would suggest a couple of checks: * see if HTTP traffic is reaped by some other classifier, but i guess you might have already checked that. * see if the HTTP classifier is written correctly. Not referring only to the regexp but to the overall syntax. The implemented format is *veeery* sensible to tabs, spaces, white lines, etc. So try to keep it essential. Strip comments and empty lines out. Let me know. Cheers, Paolo On Mon, Nov 16, 2009 at 01:13:03PM +0300, Mike Lykov wrote: Hi all on this list. I am try to install pmacct + protocol classification feature and want to ask some question about it. pmacct + pmacct_v5 base + set of .pat files from l7filter site. See results: successfully detect ftp,nntp,subversion,jabber,ssh,dns,pop3,smtp detect connection to jabber-icq gate as rtp detect ntp as edonkey don't detect http and http-ssl at all don't detect irc (tested on irc.freenode.org + irssi), whois (whois.ripn.net + console whois) False detections rtp/edonkey is a little inconvenience, but not to detect http at all is a big disappointment! I try some variants of regexp: simple HTTP/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] default http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] [\x09-\x0d -~]*(connection:|content-type:|content-length:|date:)|post [\x09-\x0d -~]* http/[01]\.[019] second default http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9]|post [\x09-\x0d -~]* http/[01]\.[019] [ ... ] Anybody here with http classification working? ;) ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
