On 24/02/13 08:09, Matt Joyce wrote: > On 23/02/13 19:27, Tom Eastep wrote: >> Using ipsets is the only way that I would try such a thing. >> >> -Tom >> >> On 2/23/13 10:57 AM, "Cory Oldford" <c...@peaceworks.ca >> <mailto:c...@peaceworks.ca>> wrote: >> >> The overhead associated with matching against the complete bogon >> list is too much in my humble opinion. >> >> >> Cory Oldford >> >> ------------------------------------------------------------------------ >> *From: *"Dr. Jeffry A. Spain" <spa...@countryday.net >> <mailto:spa...@countryday.net>> >> *To: *"Shorewall Users" <shorewall-users@lists.sourceforge.net >> <mailto:shorewall-users@lists.sourceforge.net>> >> *Sent: *Saturday, February 23, 2013 12:38:14 PM >> *Subject: *[Shorewall-users] Full Bogon Filtering >> >> What experience have users had using ShoreWall as a bogon filter >> using the Team Cymru full bogon lists >> (http://www.team-cymru.org/Services/Bogons/http.html)? The IPv4 >> full bogon list contains over 4,600 separate networks that need >> to be denied, and the IPv6 list over 68,300. Having not tried >> this myself, I would be concerned a priori about ShoreWall server >> meltdown. >> >> Jeffry A. Spain, Network Administrator >> Cincinnati Country Day School >> >> >> >> ------------------------------------------------------------------------------ >> Everyone hates slow websites. So do we. >> Make your web apps faster with AppDynamics >> Download AppDynamics Lite for free today: >> http://p.sf.net/sfu/appdyn_d2d_feb >> _______________________________________________ >> Shorewall-users mailing list >> Shorewall-users@lists.sourceforge.net >> <mailto:Shorewall-users@lists.sourceforge.net> >> https://lists.sourceforge.net/lists/listinfo/shorewall-users >> >> >> ------------------------------------------------------------------------------ >> Everyone hates slow websites. So do we. Make your web apps faster >> with AppDynamics Download AppDynamics Lite for free today: >> >> http://p.sf.net/sfu/appdyn_d2d_feb_______________________________________________ >> Shorewall-users mailing list >> Shorewall-users@lists.sourceforge.net >> <mailto:Shorewall-users@lists.sourceforge.net> >> https://lists.sourceforge.net/lists/listinfo/shorewall-users >> >> >> >> -Tom >> You do not need a parachute to skydive. You only need a parachute to >> skydive twice. >> >> >> >> ------------------------------------------------------------------------------ >> Everyone hates slow websites. So do we. >> Make your web apps faster with AppDynamics >> Download AppDynamics Lite for free today: >> http://p.sf.net/sfu/appdyn_d2d_feb >> >> >> _______________________________________________ >> Shorewall-users mailing list >> Shorewall-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/shorewall-users > I have tried this myself and I have to second what Tom said above, > unfortunately I havn't rewritten the shell scripts for it yet after > losing them in a hard disk crash had other things needed resolving > first however the actual update of the ipsets will cause a bit of a > spike especially if your firewall system is a low end machine (An > intel i7 hexacore was taking a minute or two to chew through it it) > but I created a shell script to download the lists whenever they were > modified, compare them and then update the ipsets with changes only. > I have been intending to implement this again myself and it would > probably take as long to write a detailed explanation as it would to > just go ahead and create the scripts again with comments. > > I should have some free time this morning so I will try to get that > done today and get back to you with the shell script and comments. > > > ------------------------------------------------------------------------------ > Everyone hates slow websites. So do we. > Make your web apps faster with AppDynamics > Download AppDynamics Lite for free today: > http://p.sf.net/sfu/appdyn_d2d_feb > > > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users I have to apologize that things got a bit busy here and I forgot about this until this evening, I just worked on the script and tested it. It will provide a short summary after a run which I generally let cron mail to me so I know if the update stops working for any reason but you could just as easily redirect it to a file too.
The script is attached with comments, note that this version works pretty quickly because it doesn't go through and add them line by line every load, instead it compares the sorted new list against ipset list using comm to find added and deleted entries then only processes those, as a sanity check it will use md5sum if available to confirm the updated ipset matches checks the md5sum of the two lists match after updating provided the md5sum binary is installed. Few notes I thought I'd mention though the comments in the script discuss them too: Note that ipsets are lost on reboot but by default the script uses ipset save to save the ipset to it's working directory, just need to use ipset restore before using it for filtering shorewall can handle this for you if you add something like the following to /etc/shorewall/start ipset -! restore -f /var/cache/bogon-lists/bogonsip4.ipset Yes you could just readd each time but ipset restore is a lot faster at it than a bash script. It's your choice but I would suggest it may not hurt to consider my suggestion in there about using caps support if you have it available, that way you can avoid having a shell script running as root chewing on an externally sourced file which to me seemed a bit too much like asking for trouble. Do check the script before using it as there are a few other options you may wish to change. As a side note the script as is will work with any list of CIDR ranges that one might like to keep up to date and use for filtering in one per line format including files containing #comments whole or end of line, doesn't care about whitespace either. It will need a small modification to work with ftp:// urls, currently it is written to expect a http_response code which is fine with these lists but might be an issue if anyone wanted to adapt it. Anyway, I'm happy to make freely available to anyone to use or modify as they see fit hope it's useful to someone. Comments welcome if there is anything I missed somewhere.
#!/bin/bash # This script handles fetching updates to the fullbogons lists hosted by team-cymru.org # processing and preparing these into ipsets which allow effecient filtering by netfilter. # # A NOTE ON SECURITY # Creting of modifying ipsets on linux requires privilages which means that this script is # required to have access to privilages to function but the script is also engaged in the # fetching and processing of a large datafile sourced from the internet, having the script # run as root means having curl, various string manipulation utilities etc running as root # also. # # Personally I would consider that too much of a security risk, there are at least two main # options to get around this situation that occur to me the first would be to run the fetch # with a seperate shell script running with the privilages of a normal user as there is no # need for privilages to conduct those activities but then you still have a full root script # reading and processing external data. # # I have this set up using a second alternative based on group permissions and linux file # capabilities, the ipset binary is installed with execute permission only for owner (root) # and group (ipset) and has the linux capability cap_net_admin set as a file capability, in # this configuration an unprivilaged user who is a member of the ipset group can create and # manage ipsets, this script is executed by cron with the privilages of just such a user. # # If you have linux file capability support and have the setcap utility installed but I would # recommend the following proceedure. Please someone feel free to comment if there is any # pitfalls I am missing with this but it would seem to be significantly safer than full root. # # 1. Add the ipset group to the system with: # $ groupadd ipset # 2. Change the group of the ipset binary and set file permissions, replace /usr/sbin/ipset # with the path to the ipset binary on your system you can find it by typing: # $ which ipset # /usr/sbin/ipset # $ chgrp ipset /usr/sbin/ipset # $ chmod 750 /usr/sbin/ipset # 3. Set the necessary file capability # $ setcap 'cap_net_admin=pe' /usr/sbin/ipset # 4. Create your unprivilaged user and don't forget to also set permissions to allow the # user account to write to the $WORKPATH you set in this script. Note the options set the # account with homedir=$WORKPATH shell=/bin/false group=ipset # $ useradd -r -d <$WORKPATH> -s /bin/false -g ipset bogonmgr # $ chown bogonmgr.root /var/cache/bogon-lists # $ chmod 750 /var/cache/bogon-lists # There is no strict need to deny the others permission on the directory I guess but no # reason not to either so I figure might as well. # SCRIPT CONFIGURATION # BOGON LIST URLs # By default this script is set up to use the fullbogons list from team-cymru.org however # in principle it will work for any blocklist in one per line CIDR format. The list must # contain no other text besides comments starting with a hash character which are ignored # as is any whitespace. Note the script assumes a HTTP retreval and expects a to get a # HTTP status code back from the server, if you change this to a protocol other than HTTP/HTTPS # you will need to modify the return test or it will result in false negatives and no # ipsets being built. IPV4LISTURL="http://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt"; IPV6LISTURL="http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt"; # FETCH OPTIONS # These options are used to control the fetch of the lists by cURL # Maximum retries RETRYMAX=5 # Connect timeout in seconds CONNTIMEOUT=5 # Maximum time be aware this is for the transfer including all retries, retries could take # up to 30 seconds make sure that this number is high enough to allow for the transfer of # the IPv6 list which is approx 1MiB of data and don't forget to consider your value for # the limit rate setting here. MAXTIME=60 # Rate limit in bytes per second, suffixes k, M, G etc are supported. You should adjust # this value depending on your own connection on mine there is usually around 80Mbps of # headroom after accounting for traffic so I have this set to 50% of that 5M (40Mbps). RATELIMIT="5M" # PATH CONFIGURATION # WORKPATH specifies the directory the script will use to download the blocklist WORKPATH="/var/cache/bogon-lists" # IPSET CONFIGURATION # These set the names of the two ipsets which will be created and managed by this script. IPV4IPSET="bogonsip4" IPV6IPSET="bogonsip6" # MISC OPTIONS # IPSET SAVE # This option causes the script to invoke ipset save after making changes to the networks # within the ipset, this is recommended as it is much quicker to ipset restore the saved # file than it is to loop through thousands of entries and add them one by one remember the # ipset will be lost at reboot. To make sure that the ipset exists and is populated when # shorewall starts I added the restore to the shorewall start scripts, the -! in the ipset # command prevents ipset complaining at shorewall (re)start where the ipset already exists. # /etc/shorewall/start # ipset -! restore -f /var/cache/bogon-lists/bogonsip4-save.ipset # /etc/shorewall6/start # ipset -! restore -f /var/cache/bogon-lists/bogonsip6-save.ipset IPSETSAVE=1 # FLUSH # Setting this option to true will cause the ipset to be flushed and rebuilt completely # instead of updating with the changes, this is a much slower process. FLUSH=0 function fetchlist () { # The function used for fetching the list, using cURL means we wont need to explitly handle retries etc. # Use cURL to attempt the fetch and return the HTTP reply code and some stats, the stats we will use later. # Options: s=transfer silently, L=Handle redirect, R=Use remote file time, z=Time condition download only if newer than our file, o=Output filename, w=Write out tells curl what to return # retry limit of 5, Use compression if enabled note team-cmyru.org server doesn't yet but the AcceptEncoding header doesn't hurt and is there if they enable it later STATUS=`curl -s -L -R -z fullbogons-ipv4.txt -o fullbogons-ipv4.txt -w "%{http_code} %{time_total} %{size_download} %{speed_download}" --retry 5 --compressed --connect-timeout $CONNTIMEOUT --max-time $MAXTIME --limit-rate $RATELIMIT --url "$IPV4LISTURL"` # Find the HTTP response code we recieved from cURL and make a note of the other stats for the summary HTTPCODE=`echo $STATUS | cut -d' ' -f1` HTTPSTATS4="$HTTPSTATS`echo $STATUS | cut -d' ' -f2-` " # We check here in case we have an error reply despite our retry efforts if [ $HTTPCODE -ne 304 ] && [ $HTTPCODE -ne 200 ]; then echo "URL fetch for $LISTURL retry limit exceeded."; exit 1; fi # Repeat for IPv6 STATUS=`curl -s -L -R -z fullbogons-ipv6.txt -o fullbogons-ipv6.txt -w "%{http_code} %{time_total} %{size_download} %{speed_download}" --retry 5 --compressed --connect-timeout $CONNTIMEOUT --max-time $MAXTIME --limit-rate $RATELIMIT --url "$IPV6LISTURL"` HTTPCODE=`echo $STATUS | cut -d' ' -f1` HTTPSTATS6="$HTTPSTATS`echo $STATUS | cut -d' ' -f2-` " # We check here in case we have an error reply despite our retry efforts if [ $HTTPCODE -ne 304 ] && [ $HTTPCODE -ne 200 ]; then echo "URL fetch for $LISTURL retry limit exceeded."; exit 1; fi } function listpreprocessor () { # For the incoming list first strip any comments or whitespace, then sort and pass through uniq to make sure each # entry occurs once and only once. cat "fullbogons-ipv4.txt" | sed -re 's/#.*$//g; s/\s+//; /^$/d' | sort | uniq >./ipv4-new ipset list $IPV4IPSET | awk '{ if (listing == 1) print $1; else if ($1 == "Members:") listing=1; }' | sed '/^$/d' | sort >./ipv4-old cat "fullbogons-ipv6.txt" | sed -re 's/#.*$//g; s/\s+//; /^$/d' | sort | uniq >./ipv6-new ipset list $IPV6IPSET | awk '{ if (listing == 1) print $1; else if ($1 == "Members:") listing=1; }' | sed '/^$/d' | sort >./ipv6-old comm -1 -3 ipv4-old ipv4-new | sed 's/\s+//' >./ipv4-add comm -1 -3 ipv6-old ipv6-new | sed 's/\s+//' >./ipv6-add comm -2 -3 ipv4-old ipv4-new | sed 's/\s+//' >./ipv4-del comm -2 -3 ipv6-old ipv6-new | sed 's/\s+//' >./ipv6-del } function updateipsets () { cat ipv4-add | while read CIDR; do ipset add $IPV4IPSET $CIDR; done cat ipv6-add | while read CIDR; do ipset add $IPV6IPSET $CIDR; done cat ipv4-del | while read CIDR; do ipset del $IPV4IPSET $CIDR; done cat ipv6-del | while read CIDR; do ipset del $IPV6IPSET $CIDR; done } function verify () { # Check if we have the md5sum command available, if we do then we can verify the ipset is correct if [ -x "`which md5sum`" ]; then mv ipv4-new ipv4-old mv ipv6-new ipv6-old ipset list $IPV4IPSET | awk '{ if (listing == 1) print $1; else if ($1 == "Members:") listing=1; }' | sed '/^$/d' | sort >./ipv4-new ipset list $IPV6IPSET | awk '{ if (listing == 1) print $1; else if ($1 == "Members:") listing=1; }' | sed '/^$/d' | sort >./ipv6-new CHECK=`md5sum ipv4-new ipv4-old | awk '{ if (NR == 1) sum1=$1; else if (sum1 == $1) print "OK"; else print "FAIL MD5 Mismatch",sum1,$1 }'` if [ $CHECK != "OK" ]; then echo $CHECK; exit 1; fi CHECK=`md5sum ipv6-new ipv6-old | awk '{ if (NR == 1) sum1=$1; else if (sum1 == $1) print "OK"; else print "FAIL MD5 Mismatch",sum1,$1 }'` if [ $CHECK != "OK" ]; then echo $CHECK; exit 1; fi fi } function summary () { echo "Ipset update results" TS4=`date --date="@$(stat fullbogons-ipv4.txt --printf %Y)" '+%d/%m/%Y %H:%M:%S'` TS6=`date --date="@$(stat fullbogons-ipv4.txt --printf %Y)" '+%d/%m/%Y %H:%M:%S'` echo "IPv4 list fetch `echo $HTTPSTATS4 | cut -d' ' -f2` B in `echo $HTTPSTATS4 | cut -d' ' -f1`s rate `echo $HTTPSTATS4 | cut -d' ' -f3` B/s" echo "IPv4 list entries: `cat ipv4-new | wc -l` added: `cat ipv4-add | wc -l` deleted: `cat ipv4-del | wc -l` timestamp: $TS4" echo "IPv6 list fetch `echo $HTTPSTATS6 | cut -d' ' -f2` B in `echo $HTTPSTATS6 | cut -d' ' -f1`s rate `echo $HTTPSTATS6 | cut -d' ' -f3` B/s" echo "IPv6 list entries: `cat ipv6-new | wc -l` added: `cat ipv6-add | wc -l` deleted: `cat ipv6-del | wc -l` timestamp: $TS6" echo "Total processing time `echo "$(date '+%s.%N') - $BEGIN_TIME" | bc | xargs printf "%0.3g"` seconds" } function ipsetsave () { if [ $IPSETSAVE -eq 1 ]; then ipset save $IPV4IPSET -f ${IPV4IPSET}.ipset ipset save $IPV6IPSET -f ${IPV6IPSET}.ipset fi } BEGIN_TIME=`date +%s.%N` cd $WORKPATH || exit 1 # We will create several working lists during execution, this makes sure that we don't leave # them laying around even if we get killed due to an error etc trap "rm ${WORKPATH}/ipv?-???" EXIT # Create ipsets if they don't exist yet # Default settings are adiquite for IPv4 ipset -! create $IPV4IPSET hash:net # For the IPV6 list we override the default hash size and the max entries in the table 65536 is exceeded sometimes # due to the significant unallocated IPv6 space. Also need to explicitly declare family as IPv6 default is IPv4 ipset -! create $IPV6IPSET hash:net family inet6 hashsize 2048 maxelem 131072 # If flush is set this is where the ipsets get cleared if [ $FLUSH -eq 1 ]; then ipset flush $IPV4IPSET ipset flush $IPV6IPSET fi # Fetch the ipset lists from the URLs defined in IPV4LISTURL and IPV6LISTURL fetchlist listpreprocessor updateipsets verify if [ $IPSETSAVE = 1 ]; then ipsetsave; fi summary
signature.asc
Description: OpenPGP digital signature
------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
