Re: [pmacct-discussion] Get TCP SYN totals per IP
Hello Paolo, Thanks for your reply. I actually figured it out a little after I posted this! I used libpcap with pmacctd on a capture box already doing packet capture for FastNetMon among other things. It worked like a charm :) Thanks! On 22/10/2017 3:44 μμ, Paolo Lucente wrote: > Hi Vaggelis, > > Which capturing method are you using, libpcap or NetFlow/IPFIX/sFlow? > And also are you looking for a dedicated solution for this or this is > going to be yet another activity for an existing pmacct deployment? > > Taking the simplest scenario: you using libpcap, so pmacctd, and want > to build something dedicated for this. You can start pmacctd with a > pcap filter like 'tcp[tcpflags] == tcp-syn' (either commandline or via > the pcap_filter config key); this will filter in only TCP SYN packets > then you can simply aggregate things in the most suitable way for you > and count packets out as always. If the solution is to be shared with > existing activities, you can move the filter in a pre_tag_map (using > the 'filter' keyword) so to be able to tag TCP SYN packets; then with a > pre_tag_filter you can intercept such tag and route the specific packets > to a dedicted plugin for this TCP SYN measurement activity. The tag > solution would work similarly for NetFlow/IPFIX/sFlow. > > Paolo > > On Wed, Oct 18, 2017 at 05:08:53PM +0300, Vaggelis Koutroumpas wrote: >> Hello, >> >> Is it possible to get a per IP total of SYN packets? >> >> I am trying to implement some policies to block SYN packets if they >> exceed a certain threshold (to mitigate SYN Floods), but before doing >> that I want to first log all TCP SYN traffic for some time so that I can >> get some useful stats out of it and choose the proper thresholds to >> avoid false positives. >> >> If anyone has some config snippet that would like to share I would be >> grateful :) >> >> Thank you :) >> ___ >> pmacct-discussion mailing list >> http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Get TCP SYN totals per IP
Hello, Is it possible to get a per IP total of SYN packets? I am trying to implement some policies to block SYN packets if they exceed a certain threshold (to mitigate SYN Floods), but before doing that I want to first log all TCP SYN traffic for some time so that I can get some useful stats out of it and choose the proper thresholds to avoid false positives. If anyone has some config snippet that would like to share I would be grateful :) Thank you :) ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] MySQL Timezone handling
Thanks for your suggestion guys, How do you propose to work with UTC? Should the OS' clock run in UTC or only the MySQL server? I just feel that running the server clock in UTC will be a bit uncomfortable having to constantly translate the dates from UTC to UTC+2/3 while working with logs etc. Would running only MySQL in UTC timezone be sufficient? On 31/5/2016 11:40 πμ, raf wrote: > > >> +1 to storing your application data in UTC. >> >> > > +1. Your server and storage backend should be set to UTC. This is the > only way to assure consistency of timestamped data. > Zoning should be make on the frontend side. > > ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] MySQL Timezone handling
Hello, I am using nfacct to process flows from our routers and store them in a mysql database for processing and visualization. I have a 'daily' table for storing the daily traffic for each IP. I've set sql_history to 1d to store one record per day per IP. This worked fine for months. Each record's time started on 00:00:00 of each day. It came to my attention today that since March the 28th all the inserted records have a time of 01:00:00. On 27th we had our clocks changed due to DST (from GMT+2 to GMT+3). The mysql daemon hasn't been restarted since late 2015 so its internal clock does not take into account the recent DST change (as far as I can tell). So after researching about it a little it seems that MySQL has an issue with the datetime type field and timezone changes (without restarting the mysql server or setting the timezone on session start). According to various articles, the 'timestamp' type field handles this better by converting any given date/time during insert from the local timezone to UTC and converting it back from UTC to the local timezone during read (select). One drawback mentioned is the range limitation of timestamp type since it's based on the Unix Epoch Time, so you can't have dates prior to 1970 or after 20385- which obviously is not an issue for storing network flows :) Should I simply change the schema of my database from datetime type to timestamp type? I am no sql expert, so I don't know what are the ramifications of a change like this. I know there's the sql_history_since_epoch config key for nfacct for storing all dates in unix_timestamp format, but this requires major changes on my frontend code and general processing of already stored data, so I would definitely like to avoid it if possible. Does anyone have a similar issue? How do you handle it? Thanks, Vaggelis. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] nfacct total bytes inconsistencies
Hello Mario, Yes they include everything AFAIK, but we don't have any multicast traffic and the the broadcast traffic is very little on our VLANs (mostly standard LAMP servers). Plus the uplink interfaces (which I am monitoring/exporting flows for) do not handle any broadcast traffic (except the standard arp packets of course). On 30/11/2015 12:14 μμ, Jentsch, Mario wrote: > Hi Vaggelis, > > do the SNMP OIDs are you monitoring for these traffic numbers include packets > that are not exported via Netflow (broadcast, multicast etc)? > > Regards, > Mario > ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] nfacct total bytes inconsistencies
ize: 268435456 root@netflow:~# cat /proc/sys/net/core/rmem_max 268435456 root@netflow:~# cat /proc/sys/net/core/rmem_default 268435456 I am not sure what would be proper values for these settings. Are those too small? Too big? I still get those INFO messages sporadically even with the increased buffers. It seems to me that they are already too high (for only 50-150pps of netflow traffic) to account for missing data. I also checked the UDP drop counters while I got those messages and they didn't increase. Thanks again for your help :) On 29/11/2015 6:42 μμ, Paolo Lucente wrote: > Hi Vaggelis, > > In your previous email it seems that for some period of time numbers > were not 'that off'; is it a behaviour that you manage to reproduce if > you stop/start nfacctd? I essentially wonder if it was a coincidence > or there is effectively some degradation. > > Also, i see that you use 'aggregate_filter' in order to split inbound > from outbound traffic; do you have any VLAN-tagged traffic? If yes, > then that traffic would not be captured by your current filters. If > not sure you can check this with a simple pmacct configuration like: > > plugin: memory[test] > aggregate_filter[test]: vlan > aggregate[test]: src_host, dst_host, src_port, dst_port, proto, vlan > > When doing a 'pmacct -s' to query the memory table, you should see no > results (since the NetFlow v9 templates are sent every 60 secs based > on your RouterOS configuration, please wait some 120 secs before > drafting conclusions). > > Should all of this still not bring no anything conclusive, is remote- > access to your collector box a possibility? If yes, we can follow-up > privately: i'd be more than happy to have a look myself. > > Cheers, > Paolo > > On Sun, Nov 29, 2015 at 01:22:34AM +0200, Vaggelis Koutroumpas wrote: >> It seems that the new server shows the same behavior after all :( >> >> >> mysql> SELECT ( >> -> SELECT concat(truncate((sum(bytes)/1024/1024/1024),2), 'GB') >> as bytesFROM hourlyWHERE ip_dst = '0.0.0.0' AND stamp_inserted >> BETWEEN '2015-11-28 20:00:00' AND '2015-11-28 23:59:59' >> -> ) as total_out, ( >> -> SELECT concat(truncate((sum(bytes)/1024/1024/1024),2), 'GB') >> as bytesFROM hourlyWHERE ip_src = '0.0.0.0' AND stamp_inserted >> BETWEEN '2015-11-28 20:00:00' AND '2015-11-28 23:59:59' >> -> ) as total_in; >> +---+--+ >> | total_out | total_in | >> +---+--+ >> | 101.03GB | 15.43GB | >> +---+--+ >> 1 row in set (0.05 sec) >> >> While at the same time-frame observium reports higher 'total out' and >> less 'total in' http://prntscr.com/983ers >> >> I guess the 'total in' discrepancy is acceptable. But the 'total out' is >> over 6Gbytes off! >> >> If I increase the time-frame then the totals are more off. >> >> mysql> SELECT ( >> -> SELECT concat(truncate((sum(bytes)/1024/1024/1024),2), 'GB') >> as bytesFROM hourlyWHERE ip_dst = '0.0.0.0' AND stamp_inserted >> BETWEEN '2015-11-28 19:00:00' AND '2015-11-28 23:59:59' >> -> ) as total_out, ( >> -> SELECT concat(truncate((sum(bytes)/1024/1024/1024),2), 'GB') >> as bytesFROM hourlyWHERE ip_src = '0.0.0.0' AND stamp_inserted >> BETWEEN '2015-11-28 19:00:00' AND '2015-11-28 23:59:59' >> -> ) as total_in; >> +---+--+ >> | total_out | total_in | >> +---+--+ >> | 129.60GB | 19.46GB | >> +---+--+ >> 1 row in set (0.02 sec) >> >> Observium: http://prntscr.com/983nxa >> >> Here the 'total out' is 8GBytes off. >> While 'total in' seems to be a little off but in acceptable range. >> >> >> There are no drops AFAICT. >> >> root@netflow:~# netstat -s | grep Udp\: -A 5 >> Udp: >> 817211 packets received >> 688 packets to unknown port received. >> 122 packet receive errors >> 14971 packets sent >> RcvbufErrors: 122 >> >> Those 122 errors are there for hours (before 20:00:00 of my query). >> >> root@netflow:~# cat /proc/net/udp >> sl local_address rem_address st tx_queue rx_queue tr tm->when >> retrnsmt uid timeout inode ref pointer drops >> 696: :0044 : 07 : 00: >> 00 10611 2 88007b36c780 0 >> 751: :307B : 07 : 00: >> 00 10580 2 88007b36cb00 0 >> >> >> I've also installed munin to monitor the performance of the server. >> MySQL does on average 40 queries/s. >> The server load is steadily 0.1 >> The avg incoming packets are ~40pps >> >> So the server is pretty much idle to lose any data. >> >> Any ideas what else to check? >> What would be an acceptable 'off percentage' of the bytes in comparison >> with SNMP measurements? >> >> >> Thanks. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] nfacct total bytes inconsistencies
Hi Paolo, > Posed I'm no expert of RouterOS; if it has a NetFlow export process, > can you check if it pegs at 100% CPU? Or if anything suspicious emerges > from the router logs? The netflow process runs at 0.1-0.2% CPU (on a 36core router). Unfortunately RouterOS' netflow options and stats are very basic and the logs do not include anything related to it :( But there isn't too much traffic (and the netflow traffic is at 50-150pps) > On the nfacctd side, if logs are clean then it should mean internal > buffering is OK. Still, better to double-check buffering between the > kernel and nfacctd. At this propo, can you please follow notes in > section D of chapter XXI of a recent pmacct QUICKSTART guide ( see > https://github.com/paololucente/pmacct/blob/master/pmacct/QUICKSTART ), > essentially to check if there is any UDP drops? Today I installed a new server just for nfacctd running the latest debian with MySQL 5.7. I've got about 2hours of stats so far and they seem to coincide with Observium's stats. Though I noticed there is a 5 minute skew in what nfacctd and Observium show for the same time frame. I guess it's because nfacctd will insert the flow in the appropriate time-bin based on it's timestamps while the cron script of observium will log the octects from snmp at the time it runs (which maybe 1-2 minutes after the 5 minute mark since it collects data from many ports on each run). The good thing is that regardless of that time skew between the two systems, the amount of data measured stay in par as time goes by. I need to keep an eye on it to make sure that's the case since I've noticed the previous days that during the day when traffic is at its peak the discrepancies increase. By the way, using MySQL 5.7, nfacctd caused errors from time to time complaining that the vlan or tos fields do not have a default value (this was the sql error). I simply altered those fields to have a default value and now those errors stopped. Maybe this is a bug caused by MySQL 5.7 and current pmacct's default db schema (I believe it is not related to my current issue since on the old server I have MySQL 5.5, just mentioning it). Now, checking the udp drop counters on the old server, indeed I see some 25000+ drops. That counter seem to increase during the refresh time of the sql plugin. Not always though. Is there a connection between the drops and the mysql insert/update process? If so, would running the mysql server on a different server eliminate any future possibility of that happening again? I don't see any drops on the new server, so that's a good thing, and that may account for the fact that it seems to count the totals properly (I certainly hope so!) I added the nfacctd_pipe_size and modified the rmem_default & rmem_max as suggested in the FAQ (silly me, I didn't read it all the way to the end!) but I still see the drops counter increase. But if the new server works OK as it is, I don't really care if the old one has drops (for any reason). Also I couldn't find any documenation for this config parameter on the Official Config Keys wiki page. http://wiki.pmacct.net/OfficialConfigKeys > Finally, i see sql_refresh_time and sql_history are set to different > values - meaning SQL UPDATE queries are involved; this is OK as long > as the actual database does not suffer from them; can you check that > SQL writer processes are not piling up? This can be done with a simple > "ps auxw | grep nfacctd". I've set them that low to troubleshoot the problem (check the new data in 1 minute intervals in the database). Watching the insert/update queries fly by the terminal during debug mode, it takes about 10seconds to finish - without any errors, so it doesn't seem to be any issue there. I'll keep an eye on the new setup to see how it goes and I'll keep you posted if the issue persists. Thank you for your help :) Cheers, Vaggelis. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
