[pmacct-discussion] protocol classification don't detect http
Hi all on this list. I am try to install pmacct + protocol classification feature and want to ask some question about it. pmacct + pmacct_v5 base + set of .pat files from l7filter site. See results: successfully detect ftp,nntp,subversion,jabber,ssh,dns,pop3,smtp detect connection to jabber-icq gate as rtp detect ntp as edonkey don't detect http and http-ssl at all don't detect irc (tested on irc.freenode.org + irssi), whois (whois.ripn.net + console whois) False detections rtp/edonkey is a little inconvenience, but not to detect http at all is a big disappointment! I try some variants of regexp: simple HTTP/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] default http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] [\x09-\x0d -~]*(connection:|content-type:|content-length:|date:)|post [\x09-\x0d -~]* http/[01]\.[019] second default http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9]|post [\x09-\x0d -~]* http/[01]\.[019] I dump one session: query (minus binary part) GET / HTTP/1.1 Host: whois.kraft-s.ru User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5pre) Gecko/2008120802 Firefox/3.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: ru,en-us;q=0.7,en;q=0.3 Accept-Encoding: gzip,deflate Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive responce (minus binary part) HTTP/1.0 200 OK Date: Thu, 12 Nov 2009 10:55:51 GMT Server: Apache/1.3.26 (Unix) mod_perl/1.27 PHP/4.2.3 Content-Type: text/html; charset=ISO-8859-1 X-Cache: MISS from router.local X-Cache-Lookup: MISS from router.local:3128 Via: 1.0 router.local (squid/3.0.STABLE19) Proxy-Connection: close !DOCTYPE HTML PUBLIC -//W3C//DTD HTML 4.0 Transitional//EN - not detected. Anybody here with http classification working? ;) -- Mike ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] protocol classification don't detect http
Hi Mike, I see all of those signatures actually working by picking some sites randomly with wget. This is with 0.12.0rc3 but honestly speaking there has not been any major work related to the classification part for the past 3-4 years. I would suggest a couple of checks: * see if HTTP traffic is reaped by some other classifier, but i guess you might have already checked that. * see if the HTTP classifier is written correctly. Not referring only to the regexp but to the overall syntax. The implemented format is *veeery* sensible to tabs, spaces, white lines, etc. So try to keep it essential. Strip comments and empty lines out. Let me know. Cheers, Paolo On Mon, Nov 16, 2009 at 01:13:03PM +0300, Mike Lykov wrote: Hi all on this list. I am try to install pmacct + protocol classification feature and want to ask some question about it. pmacct + pmacct_v5 base + set of .pat files from l7filter site. See results: successfully detect ftp,nntp,subversion,jabber,ssh,dns,pop3,smtp detect connection to jabber-icq gate as rtp detect ntp as edonkey don't detect http and http-ssl at all don't detect irc (tested on irc.freenode.org + irssi), whois (whois.ripn.net + console whois) False detections rtp/edonkey is a little inconvenience, but not to detect http at all is a big disappointment! I try some variants of regexp: simple HTTP/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] default http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] [\x09-\x0d -~]*(connection:|content-type:|content-length:|date:)|post [\x09-\x0d -~]* http/[01]\.[019] second default http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9]|post [\x09-\x0d -~]* http/[01]\.[019] [ ... ] Anybody here with http classification working? ;) ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] create my own mysql table
I still see all flow records having the same number 4294967295 in my mysql table, and debug does not seem to tell me why this happens. And I have another question about the sql_history, what exactly is this? I have read the configuration keys several time but I am still not sure. If I set it to 5m, does it mean pmacct will hold and count the data internally until the end of the 5 minutes then send the final statistic to mysql? One more question, how can i get pmacct to show the flow number in the database? I got all 0 ,and when i change to aggregate to aggregate:proto,src_host, dst_host, src_port, dst_port, src_as, dst_as, tcpflags,flows all the numbers are one same pretty big number MMM, sounds strange. Also, don't manage to reproduce either of the two scenarios by playing your configuration on a test box. The same pretty big number smells like dirty data? Enable debug and you should see on the screen which SQL queries are generated by pmacct and sent to the MySQL database. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] create my own mysql table
Hi, On Mon, Nov 16, 2009 at 11:58:14AM -0600, fedora fedora wrote: I still see all flow records having the same number 4294967295 in my mysql table, and debug does not seem to tell me why this happens. How do you mean? You see that number appearing in the debug? As writing to the database is done through a library (MySQL client library), I'm trying to establish what is generating the issue. Can you also say what version of MySQL you are using, which OS and which architecture you are running pmacct on? And I have another question about the sql_history, what exactly is this? I have read the configuration keys several time but I am still not sure. If I set it to 5m, does it mean pmacct will hold and count the data internally until the end of the 5 minutes then send the final statistic to mysql? It means you have 5 minutes traffic counters (temporal aggregation). The SQL cache scanner sets how often you write to the database. You can have a sql_history set to 5 minutes but have the SQL scanner to kick in every minute (sql_refresh_time: 60). There are more advanced considerations to do at this propo - which are useless to start with and that you can read of in CONFIG-KEYS, sql_dont_try_update feature. Overall, best is to start blank with relevant sections of the sql/README.mysql and EXAMPLES documents in the distribution tarball - and then complicate things bit by bit. Btw, sql_history_roundoff is also a very useful directive to be coupled almost always with sql_history, for example: sql_history: 5m sql_history_roundoff: h It nicely aligns the time bins to the hour boundary. Again, perhaps give it a try to see the effect. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] create my own mysql table
Thanks a ton for the reply! FYI, here is the debug output, DEBUG ( default/mysql ): INSERT INTO `test_1` (stamp_updated, stamp_inserted, ip_src, ip_dst, as_src, as_dst, src_port, dst_port, tcp_flags, ip_proto, packets, bytes, flows) VALUES (FROM_UNIXTIME(1258410661), FROM_UNIXTIME(1258410600), 'x.x.x.34', 'x.x.x.2', xx8, xx9, 443, 2608, 24, 'tcp', 1, 1353, 140733193388033) and here is what shows up in the mysql table for the corresponding record. | xx8 | xx9 | x.x.x.34 | x.x.x.2 | 443 | 2608 |24 | tcp | 0 | 1 | 1353 | 4294967295 | 2009-11-16 16:30:00 | 2009-11-16 16:31:01 | The flow number is 4294967295, and actually all the other records have same number with this one. I am using ubuntu 8.04 (64 bits), with mysql Server version: 5.0.67-0ubuntu6 (Ubuntu) Let me know If you need more info, thanks It means you have 5 minutes traffic counters (temporal aggregation). So pmacct keeps tracking the traffic count and and the end of the given minutes(hours..etc) it calculates the summary and then writes it to the backend database, am I right? If I am correct, how does pmacct treat netflow data? since all the data it gets already get aggregated by netflow protocol. Will pmacct do something extra? I guess for sflow, it will act differently and do the calculations. Here is my current config, debug: true sql_optimize_clauses: true sql_history: 5m sql_history_roundoff: m sql_table: test_%w sql_table_schema: /home/test.schema sql_table_version: 6 aggregate: proto, src_host, dst_host, src_port, dst_port, src_as, dst_as, tcpflags, flows interface: eth1 nfacctd_port: 1 plugins: mysql ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
