Here's an updated patch which supersedes the one I posted last Friday, 13 Mar. I've been using ratelimit.py with my earlier patch and it's made a huge difference in the amount of spam that slips through our filters here. This patch addresses several comments which Gordon made on the patch I posted last Friday.
* "use_blocks" is now "use_groups" to avoid the double-meaning of the word "block" * As per Gordon's suggestion, I've simplified the code that's executed on a per-message basis. I didn't eliminate regexp usage (sorry, Gordon!) but I don't believe the performance hit will be substantial and what's generated with regexp methods is an address group identifier, not a functional address. * The address group identifier (which is simply the full IP address if use_groups=False) is now the 2nd level dictionary index, i.e. instead of the per message counter being _sender[now][SendersMta] it's _sender[now][address_group_id] and the full SendersMta information is only used for logging and for the SMTP dialog with the sending system. This information is useful for researching spam sources. * IPv6 is supported. Whether to rate limit based on a /48 or /64 group is a configuration option. There are potential issues parsing a v6 address containing the shorthand "::" notation if this spans more than one hex 16 bit portion of the address. Someone correct me if I'm wrong, but I don't think this is found in the network portion of real-world assigned v6 addresses. --- ratelimit.py.orig 2015-03-16 11:05:32.000000000 -0500 +++ ratelimit.py 2015-03-16 13:00:25.000000000 -0500 @@ -1,4 +1,5 @@ #!/usr/bin/python +# vim: set expandtab ai ts=4: # ratelimit -- Courier filter which limits the rate of messages from any IP # Copyright (C) 2003-2008 Gordon Messmer <[email protected]> # @@ -21,12 +22,21 @@ import thread import time import courier.control +import re # The rate is measured in messages / interval in minutes maxConnections = 60 interval = 1 +# Limit rate for all addresses in a /24 address group for IPv4 and a +# /64 group for IPv6 +use_groups = False + +# If use_groups = True, use ratelimting for all IPv6 addresses in a +# /48 group insted of a /64 group +v6_groupsize48 = True + # The senders lists will be scrubbed at the interval indicated in # seconds. All records older than the "interval" number of minutes # will be removed from the lists. @@ -41,6 +51,19 @@ global _sendersLock global _senders global _sendersLastPurged + global _igroup4 + global _igroup6 + range4 = 3 + if v6_groupsize48: + range6 = 3 + else: + range6 = 4 + + if use_groups: + _pat4 = re.compile("((:?\d{1,3}\.){%s})" % (range4,)) + _pat6 = re.compile('((:?[0-9a-f]{0,4}:{,1}){%s})' % (range6,)) + _igroup4 = lambda a: _pat4.match(a).group(1) + _igroup6 = lambda a: _pat6.match(a).group(1) _sendersLock = thread.allocate_lock() _senders = {} _sendersLastPurged = 0 @@ -48,6 +71,14 @@ # Record in the system log that this filter was initialized. sys.stderr.write('Initialized the ratelimit python filter\n') +def getSenderGroup(ip): + if use_groups: + try: + return _igroup4(ip) + except AttributeError: + return _igroup6(ip) + else: + return ip def doFilter(bodyFile, controlFileList): """Track the number of connections from each IP and temporarily fail @@ -56,10 +87,13 @@ global _sendersLastPurged try: - sender = courier.control.getSendersMta(controlFileList) + senderMta = courier.control.getSendersMta(controlFileList) + senderIP = courier.control.getSendersIP(controlFileList) except: return '451 Internal failure locating control files' + sender = getSenderGroup(senderIP) + _sendersLock.acquire() try: now = int(time.time() / 60) @@ -78,18 +112,18 @@ if not _senders[now].has_key(sender): _senders[now][sender] = 1 else: - _senders[now][sender] = _senders[now][sender] + 1 + _senders[now][sender] += 1 # Now count the number of connections from this sender connections = 0 for i in range(0, interval): if _senders.has_key(now - i) and _senders[now - i].has_key(sender): - connections = connections + _senders[now - i][sender] + connections += _senders[now - i][sender] # If the connection count is higher than the maxConnections setting, # return a soft failure. if connections > maxConnections: - status = '421 Too many messages from %s, slow down.' % sender + status = '421 Too many messages from %s, slow down.' % (senderMta,) else: status = '' finally: The matching configuration block in pythonfilter-modules.conf is: [ratelimit.py] maxConnections = 60 interval = 1 sendersPurgeInterval = 60 * 60 * 12 # Limit rate for all addresses in a /24 address group for IPv4 and a /64 group for IPv6 use_groups = True # If use_groups = True, ratelimit on a /48 group instead of a /64 group v6_groupsize48 = True I hope others find this patch as useful as I do here, and again, suggestions and comments are welcome. -- Lindsay Haisley | "UNIX is user-friendly, it just FMP Computer Services | chooses its friends." 512-259-1190 | -- Andreas Bogk http://www.fmp.com | ------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/ _______________________________________________ courier-users mailing list [email protected] Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users
