After the discussion earlier I got a few ideas from the suggestions and decided to try some thoughts to improve the script, new script to share with anyone who might find it useful.
Changes - Complete overhaul of list processing the new version manages to perform an atomic full rebuild of the entire list in ~0.03s for the IPv4 list and ~0.3s for the IPv6 list. - Uses the features of ipset restore building an ipset save file which can then be processed in a single transaction by ipset. - Building is now done using an intermediate temp list which is swapped in, there are no changes to your active filter list unless the entire rebuild process goes through correctly. Especially useful if updating using cron where a failure may not be immediately visible. I was really surprised just how well this method works that is until I thought it through some more and remembered something about what I was writing this for then it was a case of why on earth did I fail to think of this earlier, I'm pretty sure that shorewall uses a similar procedure for iptables because now I think about it doesn't shorewall do the same thing with iptables? I am guessing probably for similar reasons, at least for atomicity but I suspect in this case again that perhaps the iptables C binary is probably also faster at doing the job than interpreted shell or perl. Being already pre-compiled into native machine code and able to interface directly with the kernel syscalls without having to deal with a middleman. Oh, and another thought came to me wanted to ask does shorewall have a similar hook for run_ipset as there is for such as run_tc and run_iptables just thinking it would be a better way to do the restore in shorewall-start is that the only way to make successful startup dependant on a start script succeeding, or would it be sufficient if my script exits nonzero on failure? New script is attached for anyone interested to look into the code or make use of it, md5sum bellow: 0a74aab398d14b157ec2371b3a16d1a0 bogon-ipsets-v1.1.tar.gz On 26/02/13 22:22, Matt Joyce wrote: > On 24/02/13 08:09, Matt Joyce wrote: >> On 23/02/13 19:27, Tom Eastep wrote: >>> Using ipsets is the only way that I would try such a thing. >>> >>> -Tom >>> >>> On 2/23/13 10:57 AM, "Cory Oldford" <c...@peaceworks.ca >>> <mailto:c...@peaceworks.ca>> wrote: >>> >>> The overhead associated with matching against the complete bogon >>> list is too much in my humble opinion. >>> >>> >>> Cory Oldford >>> >>> ------------------------------------------------------------------------ >>> *From: *"Dr. Jeffry A. Spain" <spa...@countryday.net >>> <mailto:spa...@countryday.net>> >>> *To: *"Shorewall Users" <shorewall-users@lists.sourceforge.net >>> <mailto:shorewall-users@lists.sourceforge.net>> >>> *Sent: *Saturday, February 23, 2013 12:38:14 PM >>> *Subject: *[Shorewall-users] Full Bogon Filtering >>> >>> What experience have users had using ShoreWall as a bogon filter >>> using the Team Cymru full bogon lists >>> (http://www.team-cymru.org/Services/Bogons/http.html)? The IPv4 >>> full bogon list contains over 4,600 separate networks that need >>> to be denied, and the IPv6 list over 68,300. Having not tried >>> this myself, I would be concerned a priori about ShoreWall server >>> meltdown. >>> >>> Jeffry A. Spain, Network Administrator >>> Cincinnati Country Day School >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Everyone hates slow websites. So do we. >>> Make your web apps faster with AppDynamics >>> Download AppDynamics Lite for free today: >>> http://p.sf.net/sfu/appdyn_d2d_feb >>> _______________________________________________ >>> Shorewall-users mailing list >>> Shorewall-users@lists.sourceforge.net >>> <mailto:Shorewall-users@lists.sourceforge.net> >>> https://lists.sourceforge.net/lists/listinfo/shorewall-users >>> >>> >>> ------------------------------------------------------------------------------ >>> Everyone hates slow websites. So do we. Make your web apps faster >>> with AppDynamics Download AppDynamics Lite for free today: >>> >>> http://p.sf.net/sfu/appdyn_d2d_feb_______________________________________________ >>> Shorewall-users mailing list >>> Shorewall-users@lists.sourceforge.net >>> <mailto:Shorewall-users@lists.sourceforge.net> >>> https://lists.sourceforge.net/lists/listinfo/shorewall-users >>> >>> >>> >>> -Tom >>> You do not need a parachute to skydive. You only need a parachute to >>> skydive twice. >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Everyone hates slow websites. So do we. >>> Make your web apps faster with AppDynamics >>> Download AppDynamics Lite for free today: >>> http://p.sf.net/sfu/appdyn_d2d_feb >>> >>> >>> _______________________________________________ >>> Shorewall-users mailing list >>> Shorewall-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/shorewall-users >> I have tried this myself and I have to second what Tom said above, >> unfortunately I havn't rewritten the shell scripts for it yet after >> losing them in a hard disk crash had other things needed resolving >> first however the actual update of the ipsets will cause a bit of a >> spike especially if your firewall system is a low end machine (An >> intel i7 hexacore was taking a minute or two to chew through it it) >> but I created a shell script to download the lists whenever they were >> modified, compare them and then update the ipsets with changes only. >> I have been intending to implement this again myself and it would >> probably take as long to write a detailed explanation as it would to >> just go ahead and create the scripts again with comments. >> >> I should have some free time this morning so I will try to get that >> done today and get back to you with the shell script and comments. >> >> >> ------------------------------------------------------------------------------ >> Everyone hates slow websites. So do we. >> Make your web apps faster with AppDynamics >> Download AppDynamics Lite for free today: >> http://p.sf.net/sfu/appdyn_d2d_feb >> >> >> _______________________________________________ >> Shorewall-users mailing list >> Shorewall-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/shorewall-users > I have to apologize that things got a bit busy here and I forgot about > this until this evening, I just worked on the script and tested it. It > will provide a short summary after a run which I generally let cron mail > to me so I know if the update stops working for any reason but you could > just as easily redirect it to a file too. > > The script is attached with comments, note that this version works > pretty quickly because it doesn't go through and add them line by line > every load, instead it compares the sorted new list against ipset list > using comm to find added and deleted entries then only processes those, > as a sanity check it will use md5sum if available to confirm the updated > ipset matches checks the md5sum of the two lists match after updating > provided the md5sum binary is installed. > > Few notes I thought I'd mention though the comments in the script > discuss them too: > Note that ipsets are lost on reboot but by default the script uses ipset > save to save the ipset to it's working directory, just need to use ipset > restore before using it for filtering shorewall can handle this for you > if you add something like the following to /etc/shorewall/start > ipset -! restore -f /var/cache/bogon-lists/bogonsip4.ipset > Yes you could just readd each time but ipset restore is a lot faster at > it than a bash script. > > It's your choice but I would suggest it may not hurt to consider my > suggestion in there about using caps support if you have it available, > that way you can avoid having a shell script running as root chewing on > an externally sourced file which to me seemed a bit too much like asking > for trouble. > > Do check the script before using it as there are a few other options you > may wish to change. > > As a side note the script as is will work with any list of CIDR ranges > that one might like to keep up to date and use for filtering in one per > line format including files containing #comments whole or end of line, > doesn't care about whitespace either. It will need a small modification > to work with ftp:// urls, currently it is written to expect a > http_response code which is fine with these lists but might be an issue > if anyone wanted to adapt it. > > Anyway, I'm happy to make freely available to anyone to use or modify as > they see fit hope it's useful to someone. Comments welcome if there is > anything I missed somewhere. > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb > > > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users
bogon-ipsets-v1.1.tar.gz
Description: application/gzip
signature.asc
Description: OpenPGP digital signature
------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
