> -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf > Of Wim Kerkhoff > Sent: Thursday, 25 November 2004 8:02 PM > To: [email protected] > Subject: Re: [pmacct-discussion] Ideas - Questions > > Michael Ralston wrote: > > >Hi > > > >I've got a couple of ideas I wanted to put forward for pmacct > > > >Recording only top percentage of hosts > >I've been a victim of some denial of service attacks and I'd like to > >use pmacctd to record where they are coming from. It is > fairly obvious > >that src_host summarisation to SQL is going to end up > putting a heavy > >burden on the database, in terms of IO and storage. Would it be > >possible for pmacctd to only insert the highest traffic generating > >hosts to sql? Eg, discard all but the top 5% of hosts every > 5 minutes. > > > > > You might want to look at using something like ntop in > parallel with pmacctd... it's quite handy for identifying > unusual traffic. >
Ntop is more an instantaneous thing though right? Like top is for CPU... I need something that will log the unusual traffic so when I see a weird spike on traffic graphs I can look at the logs and see where it came from. > The only idea I can think of is to let pmacctd write to the > database, then after X hours, prune all the small counters. > The big problem is that there will be thousands and thousands > of records to insert every 5 minutes. Is 5 minutes a good > sample window? > Yeah, pruning the database solves half the problem... The storage... But it doesn't solve the io of writing out the records... I wonder how hard it would be to make pmacctd sort traffic by quantity and only log the biggest traffic. Alternatively I wonder if you could use the command line data fetch thingo and sort the data yourself... Again that is going to be cpu intensive I suppose... Maybe putting a sorted data structure like a heap into pmacctd would help... > > >I guess I could run multiple pmacctd with different update > periods, but > >that would end up using lots of ram I expect Anyone else got > any ideas > >on how to achieve this? > > > > > I'm running 6 instances of pmacctd, with two aggregate > filters in each (IN vs OUT). Works great. The only > differences in the config files are the table table to write > to, and the sql_history parameter. RAM usage is fine - > perhaps 5-15 MB for each pmacctd. This system is counting for > around 8000 IP addresses. > What is the best way to tweak the SQL updates for performance? Ie... sql_refresh_time, sql_history, sql_history_roundoff > I found that doing these sorts of concurrent updates, stored > procedures, etc is handled a lot better by PostgreSQL then > MySQL due to better transactional support. > I'm not running a postgres server... All my other software I mysql based and I don't have the resources to run both :( > >Some other quick questions... > >Does pmacctd only record tcp/udp or also other ip protocols, > icmp etc? > >Is there a way to find out which protocol certain traffic was? > > > Sure... I think there's an aggregate for ip_proto. So if it's > in the sql_table, it'll get populated with 1=icmp, 17=udp, > and so on, as per /etc/protocols. I was doing this for a > while on a dst_host basis, but found it created about 5 times > as many database records without providing a lot of extra > value to us... so I dropped the column and stopped counting on it. > I tried setting ip_proto in the config and it says WARN: ignoring unknown aggregation method: ip_proto. Reason I am curious is because my snmp graphs are showing big traffic when I'm getting a DOS attack, but when I check my pmacctd tables there is no sign of the traffic there. I figured it had to be something not udp/tcp and pmacctd was ignoring it? Michael
