> > >> I have been working on a script that downloads over *50 different >> Blocklists* and performs a duplication check to reduce the size of the >> data. It can download .CSV, .TXT, ,GZ, .ZIP files and also scrape from >> certain websites that post only a web copy of their Blocklists. > > ie : ET, Spamhaus, IBlock, dShield, Atlas, Alienvault etc.. I have been researching Blocklists for several Months and have found the current list to be beneficial in Blocking Malicious IPs.
It utilizes a tool called "Grepcidr" to make the de-duplication work. It also looks at the number of IP addresses found in a /24 range and can condense the list and enter a /24 block instead. This is done in three ways 1) Using a "max" variable, if it finds over the Max variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured *Foreign Countries* on an *individual Blocklist Basis*. 2) Using a "dmax" variable if it finds over the dmax variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured *Foreign Countries* at the end of the download process on *all of the Blocklists together*. 3) Using a "pmax" variable, if it finds over the dmax variable it will process a /24 Block *excluding Country Code* *whitelist *at the end of the download process on *all of the Blocklists together*. So I set *max* to 5, *dmax* to 5 and *pmax* to 50 in my setup. Depending on how *aggressive / conservative* an admin wants to configure the processes or disable them completely and just use the de-duplication processes. It is currently outputting to a text file in an "x.x.x.x/zz" format but can be configured for any output format. I have been testing it for several weeks in my pfsense pfBlocker Application which loads the files into Blocklist tables. I also found a way to use the Maxmind database to make a *Country Code Specific Blocklist, *excluding whitelisted countries. If anyone would like to test the script out, I would be more than happy to release it. (send me an email). -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
