Re: Please evaluate my understanding wrt access files
On 10/31/2009 7:39 PM, Stan Hoeppner wrote: Noel Jones put forth on 10/31/2009 1:12 AM: Each lookup table requires overhead. 30 separate tables requires considerably more overhead than one table. The size of the dataset doesn't change, it's the overhead that gets smaller. The more concurrent smtpd processes running, the more it matters. I guess I was kinda looking for the technical definition of 'overhead' here. If the overhead per file is a straight percentage, then it shouldn't matter if it's one big file or 30 smaller files. overhead is pretty much independent of file size. Each extra lookup table is another open file, buffer space, another library call (that might allocate its own buffer), and an additional lookup. In a tiny environment it's not enough to make a difference -- probably not enough to even notice. But one lookup always beats two lookups, one open file always uses less resources than two open files. Run enough smtpd processes and it starts to make a difference; at some point it can make a critical difference. And it's always prettier to do one thing one time vs. doing the same thing over and over. What is the actual memory footprint overhead that's being added by Posfix for each map file? Measure it on your system. Probably not a lot, but it adds up. And is the amount of overhead the same for all map types, or different/variable depending on map type? Yes, different table types use different system libraries and have different resource requirements. But for each table type it's a fairly constant value unaffected by table size (very small tables might use a little less RAM under some conditions -- but this really doesn't matter). Is it something like 3 bytes of overhead added for every 10 bytes in a map file, for example? No, it's fixed. I can understand that having everything in one map file, or few map files, compared to many, can increase lookup performance due to indexing efficiency. What I don't understand is how having fewer larger map files decreases memory footprint. Fewer files = less overhead. Size doesn't matter here. Also, given that cidr files aren't postmap'd, is it safe to assume they aren't indexed? If they aren't indexed, how does putting the contents in a single file speed up searching? One lookup beats two every time. If they are indexed when Postfix starts or reloads, then why wouldn't Postfix created a single index _after_ reading in all the data from all the cidr files, creating one internal program table? How is postfix supposed to know you want all these tables treated as one? Anyway, postfix would still need to open the files etc. so all you would save is the extra lookups. It's easy enough for you to control by building the tables the way you want them used that postfix shouldn't be guessing. -- Noel Jones
Re: Please evaluate my understanding wrt access files
Stan Hoeppner a écrit :
mouss put forth on 10/31/2009 11:06 AM:
mouss, you rock.
you can use a script if you prefer. the advantage of 'make' is that it
only re-generates files when needed (source change).
The only likely changes would be adding another country.
sometimes, you find a new IP that wasn't there, or an IP is assigned to
another country... rare, but may happen.
In this case,
would I just add the file name to the source section below, and run
the make script?
yes, just add a
COUNTRIES += nowhereland
line and it should work.
[snip]
Neat. I'm using gnu make so this should all work. Still don't
understand how this would affect memory footprint, but at least it would
clean up my main.cf a bit.
it is faster to search a single file, rather than doing 10 searches.
Think of an IP that is not listed in any of your country maps. with 10
files, you multiply the number of lookups by 10 compared to searching in
a single file.
of course latency isn't a problem in your setup. but beauty is enough
reason! see below.
PS. since you use a cidr specific directory, you can get rid of the
.cidr suffix. you could then use (in main.cf):
cidr=cidr:/etc/postfix/cidr_files
smtpd_foo_restrictions =
...
check_client_access ${cidr}/access_foo
...
check_client_access ${cidr}/access_bar
Ahh, also sweet. I'd seen folks using variables but I hadn't really
found a need yet to dig into it. That and I was afraid of confusing
myself down the road. ;) Just implemented this. Works fine, main.cf
much cleaner looking,
yep. it avoids repetition and long lines.
maps_dir = /etc/postfix/maps
cdb = cdb:${maps_dir}/cdb
pcre = pcre:${maps_dir}/pcre
mysql = mysql:${maps_dir}/mysql
...
virtual_mailbox_maps =
${sql}/virtual_mailbox
virtual_alias_maps =
${sql}/virtual_alias
smtpd_foo_restrictions =
# per recipient access rules
check_recipient_access ${mysql}/access_recipient
check_recipient_access ${pcre}/access_recipient
...
# selectively reject mail from generic rDNS and invalid PTR:
check_helo_access ${sql}/access_host
check_helo_access ${pcre}/access_host
check_reverse_client_hostname_access ${sql}/access_host
check_reverse_client_hostname_access ${pcre}/access_host
...
PS. on the other hand, postconf -n doesn't show the custom variables.
but this shouldn't be a problem as long as you don't abuse variables.
_and_ I'm now practicing good form with
check_client_access in front of each CIDR map. ;)
yes. this will allow you to move/copy checks between different
restrictions. In particular, you can move checks to a restriction class
withouth having to wonder which restriction calls the class. (yes, I am
not very clear. but I hope you see what I mean ;-).
Lots of good ancillary info coming out of this thread. Thanks guys.
Still wondering about the map file memory overhead details...
--
Stan
Re: Please evaluate my understanding wrt access files
On 10/31/2009 12:32 AM, Stan Hoeppner wrote: All these cidr maps are a great example of separate tables that should be combined into a single table with a Makefile. Interesting. I may take a look into Makefile some time. I was unaware of it until today. Just out of curiosity, how does putting it all in one file make the resulting single file smaller than the sum of the individual files' contents? Does Makefile compress the contents in some way, such as postmap indexes hash files? I ask as the only overlapping data in my CIDR files is the REJECT + custom error within each CIDR file. Each lookup table requires overhead. 30 separate tables requires considerably more overhead than one table. The size of the dataset doesn't change, it's the overhead that gets smaller. The more concurrent smtpd processes running, the more it matters. It probably doesn't matter in your environment, but posting examples suitable for only a very small environment without labeled as such doesn't help the wider user base. And postmap doesn't compress files, but the index allows postfix to find a particular key quickly -- lookup time and system overhead for hash: and cdb: is essentially independent of the size of the table, so it makes sense to use a single table as big as necessary. In particular, cdb: scales well up to millions of entries, hash: tables easily to hundreds of thousands. -- Noel Jones
Re: Please evaluate my understanding wrt access files
Stan Hoeppner a écrit :
[snip]
Thanks for the hints Noel. I may need them down the road, although not
at the moment. Though I am curious and may play around with Makefile
just to learn something.
you can use a script if you prefer. the advantage of 'make' is that it
only re-generates files when needed (source change).
for your country maps and assuming gnu make, your Makefile would contain
something like this:
#---
#directpry where we store our cidr maps
CIDR_DIR=/etc/postfix/cidr_files
#file to generate
COUNTRIES_DEST=${CIDR_DIR}/countries
#source files, withput directory nor suffix.
COUNTRIES += china
COUNTRIES += korea
COUNTRIES += russia
...
#non portable commands. gnu make assumed.
CIDR_COUNTRIES=$(addsuffix .cidr, ${COUNTRIES})
COUNTRY_MAPS=$(addprefix ${CIDR_DIR}/, ${CIDR_COUNTRIES})
${COUNTRIES_DEST}: ${COUNTRY_MAPS}
cat $^ $@
...
#-
this will catenate all your china.cidr, korea.cidr, ... into countries
which you can then use with
check_client_access cidr:/etc/postfix/cidr_files/countries
PS. since you use a cidr specific directory, you can get rid of the
.cidr suffix. you could then use (in main.cf):
cidr=cidr:/etc/postfix/cidr_files
smtpd_foo_restrictions =
...
check_client_access ${cidr}/access_foo
...
check_client_access ${cidr}/access_bar
Please evaluate my understanding wrt access files
Noel Jones put forth on 10/31/2009 1:12 AM: Each lookup table requires overhead. 30 separate tables requires considerably more overhead than one table. The size of the dataset doesn't change, it's the overhead that gets smaller. The more concurrent smtpd processes running, the more it matters. I guess I was kinda looking for the technical definition of 'overhead' here. If the overhead per file is a straight percentage, then it shouldn't matter if it's one big file or 30 smaller files. What is the actual memory footprint overhead that's being added by Posfix for each map file? And is the amount of overhead the same for all map types, or different/variable depending on map type? Is it something like 3 bytes of overhead added for every 10 bytes in a map file, for example? I can understand that having everything in one map file, or few map files, compared to many, can increase lookup performance due to indexing efficiency. What I don't understand is how having fewer larger map files decreases memory footprint. It probably doesn't matter in your environment, but posting examples suitable for only a very small environment without labeled as such doesn't help the wider user base. Agreed. That's why I described my environment. It's likely much smaller scale than the average Postfix environment. And postmap doesn't compress files, but the index allows postfix to find a particular key quickly -- lookup time and system overhead for hash: and cdb: is essentially independent of the size of the table, so it makes sense to use a single table as big as necessary. In particular, cdb: scales well up to millions of entries, hash: tables easily to hundreds of thousands. Agreed. Indexed files are always much faster to search than flat text files (unless we're talking really tiny flat files). And it's quicker to search fewer indexes than more indexes. This thread began with the premise being stated that many small map files consume more memory in Postfix than fewer larger map files. Indexes are typically proportional in size to the file being indexed, unless the file contains many duplicate entries. I freely admit I'm no indexing or database expert. However, that said, I don't understand how map file consolidation decreases memory footprint. Maybe Wieste can chime in with a brief explanation. Also, given that cidr files aren't postmap'd, is it safe to assume they aren't indexed? If they aren't indexed, how does putting the contents in a single file speed up searching? If they are indexed when Postfix starts or reloads, then why wouldn't Postfix created a single index _after_ reading in all the data from all the cidr files, creating one internal program table? -- Stan
Please evaluate my understanding wrt access files
mouss put forth on 10/31/2009 11:06 AM:
mouss, you rock.
you can use a script if you prefer. the advantage of 'make' is that it
only re-generates files when needed (source change).
The only likely changes would be adding another country. In this case,
would I just add the file name to the source section below, and run
the make script?
for your country maps and assuming gnu make, your Makefile would contain
something like this:
#---
#directpry where we store our cidr maps
CIDR_DIR=/etc/postfix/cidr_files
#file to generate
COUNTRIES_DEST=${CIDR_DIR}/countries
#source files, withput directory nor suffix.
COUNTRIES += china
COUNTRIES += korea
COUNTRIES += russia
...
#non portable commands. gnu make assumed.
CIDR_COUNTRIES=$(addsuffix .cidr, ${COUNTRIES})
COUNTRY_MAPS=$(addprefix ${CIDR_DIR}/, ${CIDR_COUNTRIES})
${COUNTRIES_DEST}: ${COUNTRY_MAPS}
cat $^ $@
...
#-
this will catenate all your china.cidr, korea.cidr, ... into countries
which you can then use with
check_client_access cidr:/etc/postfix/cidr_files/countries
Neat. I'm using gnu make so this should all work. Still don't
understand how this would affect memory footprint, but at least it would
clean up my main.cf a bit.
PS. since you use a cidr specific directory, you can get rid of the
.cidr suffix. you could then use (in main.cf):
cidr=cidr:/etc/postfix/cidr_files
smtpd_foo_restrictions =
...
check_client_access ${cidr}/access_foo
...
check_client_access ${cidr}/access_bar
Ahh, also sweet. I'd seen folks using variables but I hadn't really
found a need yet to dig into it. That and I was afraid of confusing
myself down the road. ;) Just implemented this. Works fine, main.cf
much cleaner looking, _and_ I'm now practicing good form with
check_client_access in front of each CIDR map. ;)
Lots of good ancillary info coming out of this thread. Thanks guys.
Still wondering about the map file memory overhead details...
--
Stan
Please evaluate my understanding wrt access files
I would like to confirm my understanding about access files.
Please let me know if any of this is not correct...
The man (5) access description describes a prototype file, where that
file could be a single file describing any host names, network
addresses, envelope senders or recipient addresses.
The file could also be a set of files all following the same format
rules.
Where such files might be
recipient_checks, helo_checks, sender_checks, client_checks, etc.
The usefulness of the content of an access file is dependent upon the
parameter that selects a routine that reads the file.
If check_client_access causes a read of the file it will only be
looking for IP addresses of a client server that sent the email or a
fully qualified domain name that successfully reverse maps to the IP
address of a client server that sent the email.
If check_sender_access causes a read of the file it will only be
looking for an email SMTP MAIL FROM address or a pattern which could
be a part that email address to the left of the @ sign.
If check_helo_access causes a read of the file it will only be looking
for the HELO or EHLO hostname or any valid parent domain of that
hostname that is in the SMTP HELO.
The routines executed vi the parameters such as check_client_access,
check_sender_access, check_helo_access, etc. return the value the
check to the routine that called for the check where the calling
routine would be instigated by any of these parameters:
smtpd_client_restrictions
smtpd_helo_restrictions
smtpd_sender_restrictions
smtpd_recipient_restrictions
smtpd_data_restrictions
It is possible to have all the lookups done on a single
.../postfix/access.db file but that could mean the file gets confusing
so in practice multiple access files with names like client_access,
helo_access, sender_access, etc.
A single parameter such as check_client_access may be called multiple
times in a situation like this:
smtpd_sender_restrictions =
check_sender_access hash:/etc/postfix/greylist
check_sender_access hash:/etc/postfix/sender_access
permit_mynetworks
However if the above causes a pattern to be found more than once then
only the last pattern match is used. (I think that is what When the
same parameter is defined multiple times, only the last instance is
remembered. means.)
This is how I am putting this in practice on a new virtual server
where I hope to fix some problems on current production servers:
r...@mg0x:/etc/postfix# postconf -d mail_version
mail_version = 2.5.5
I am using 2.5.5 because that is the latest from Ubuntu.
r...@mg0x:/etc/postfix# postconf -n
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_dot_mydomain = yes
biff = no
bounce_size_limit = 1
config_directory = /etc/postfix
default_process_limit = 400
header_checks = regexp:/etc/postfix/header_checks
inet_interfaces = all
mailbox_size_limit = 0
masquerade_domains = $mydomain, cnm.edu, nmvc.org, nmvirtualcollege.org
max_use = 100
message_size_limit = 16777216
mydestination = $myhostname, $mydomain, localhost.localdomain,
cnm.edu, mail.cnm.edu, mg0x.cnm.edu, mg04.cnm.edu, mg05.cnm.edu,
nmvc.org, mail.nmvc.org, mg0x.nmvc.org, mg04.nmvc.org, mg05.nmvc.org,
mg06.nmvc, nmvirtualcollege.org, mail.nmvirtualcollege.org,
mg0x.nmvirtualcollege.org, mg04.nmvirtualcollege.org,
mg05.nmvirtualcollege.org, mg04.nmvirtualcollege.org, nmln.net,
ideal-nm.org, ideal-nm.net, idealnm.org, idealnm.net
myhostname = mg0x.cnm.edu
mynetworks = 198.133.182.0/24, 198.133.181.0/24, 198.133.180.0/24,
172.16.0.0/12, 192.168.0.0/16, 10.0.0.0/8, 127.0.0.0/8
[:::127.0.0.0]/104 [::1]/128
myorigin = /etc/mailname
notify_classes = resource, software
readme_directory = no
recipient_delimiter = +
relay_domains = $mydestination
relayhost =
smtp_host_lookup = dns, native
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_banner = cnm.edu
smtpd_client_restrictions = permit_mynetworks check_client_access
hash:/etc/postfix/accessreject_rbl_client
zen.spamhaus.orgreject_rbl_client bl.spamcop.net
reject_rbl_client
dnsbl.njabl.org permit
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks reject_invalid_hostname
smtpd_recipient_restrictions = check_recipient_access
hash:/etc/postfix/overquota reject_non_fqdn_sender
reject_unknown_sender_domainreject_non_fqdn_recipient
reject_unknown_recipient_domain reject_unlisted_recipient
permit_mynetworks reject_unauth_destination
reject_unauth_pipeliningreject_invalid_helo_hostname
reject_non_fqdn_helo_hostname reject_rbl_client zen.spamhaus.org
smtpd_sender_restrictions = check_sender_access
hash:/etc/postfix/greylist check_sender_access
hash:/etc/postfix/sender_access permit_mynetworks
reject_unknown_sender_domain
smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
Re: Please evaluate my understanding wrt access files
On Fri, Oct 30, 2009 at 1:26 PM, Noel Jones njo...@megan.vbhcs.org wrote: On 10/30/2009 12:55 PM, Robert Lopez wrote: I would like to confirm my understanding about access files. Please let me know if any of this is not correct... The man (5) access description describes a prototype file, where that file could be a single file describing any host names, network addresses, envelope senders or recipient addresses. The file could also be a set of files all following the same format rules. Where such files might be recipient_checks, helo_checks, sender_checks, client_checks, etc. The usefulness of the content of an access file is dependent upon the parameter that selects a routine that reads the file. If check_client_access causes a read of the file it will only be looking for IP addresses of a client server that sent the email or a fully qualified domain name that successfully reverse maps to the IP address of a client server that sent the email. If check_sender_access causes a read of the file it will only be looking for an email SMTP MAIL FROM address or a pattern which could be a part that email address to the left of the @ sign. If check_helo_access causes a read of the file it will only be looking for the HELO or EHLO hostname or any valid parent domain of that hostname that is in the SMTP HELO. The routines executed vi the parameters such as check_client_access, check_sender_access, check_helo_access, etc. return the value the check to the routine that called for the check where the calling routine would be instigated by any of these parameters: smtpd_client_restrictions smtpd_helo_restrictions smtpd_sender_restrictions smtpd_recipient_restrictions smtpd_data_restrictions It is possible to have all the lookups done on a single .../postfix/access.db file but that could mean the file gets confusing so in practice multiple access files with names like client_access, helo_access, sender_access, etc. so far so good... A single parameter such as check_client_access may be called multiple times in a situation like this: smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/greylist check_sender_access hash:/etc/postfix/sender_access permit_mynetworks However if the above causes a pattern to be found more than once then only the last pattern match is used. (I think that is what When the same parameter is defined multiple times, only the last instance is remembered. means.) No, the last parameter is used rule refers to when indexing a single map -- when the map is indexed with postmap, a duplicate key will overwrite the previous identical key. With multiple maps -- or multiple restrictions of any type -- in smtpd_*_restrictions, the first match wins. Ah! Good. That now makes more sense. Postfix places no limit on how many maps you can use, but there is system overhead with each map. Rule of thumb is to combine maps wherever possible -- don't use two check_sender_access statements if you can do it with one. The smart way to do this is use a Makefile to build a single map from multiple similar input files. That is interesting. What is the advantage of that over directly editing a single file? I can see having unique names that pair with the parameters that cause them to be read. It is not clear to me what the benefit of multiple files is beyond this association. We do something similar with the virtualaliases table. There is a table that has all college employees who use an Exchange server, another that has all customers (students) who use Sungard Luminis, and a third that has Mailman lists. So email is delivered to one of those three systems based on that file. We build that single file from three separate files. This is how I am putting this in practice on a new virtual server where I hope to fix some problems on current production servers: r...@mg0x:/etc/postfix# postconf -d mail_version mail_version = 2.5.5 I am using 2.5.5 because that is the latest from Ubuntu. r...@mg0x:/etc/postfix# postconf -n alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases append_dot_mydomain = yes biff = no bounce_size_limit = 1 config_directory = /etc/postfix default_process_limit = 400 header_checks = regexp:/etc/postfix/header_checks inet_interfaces = all mailbox_size_limit = 0 masquerade_domains = $mydomain, cnm.edu, nmvc.org, nmvirtualcollege.org max_use = 100 message_size_limit = 16777216 mydestination = $myhostname, $mydomain, localhost.localdomain, cnm.edu, mail.cnm.edu, mg0x.cnm.edu, mg04.cnm.edu, mg05.cnm.edu, nmvc.org, mail.nmvc.org, mg0x.nmvc.org, mg04.nmvc.org, mg05.nmvc.org, mg06.nmvc, nmvirtualcollege.org, mail.nmvirtualcollege.org, mg0x.nmvirtualcollege.org, mg04.nmvirtualcollege.org, mg05.nmvirtualcollege.org, mg04.nmvirtualcollege.org, nmln.net, ideal-nm.org, ideal-nm.net, idealnm.org, idealnm.net Lots of domains in mydestination... Are you sure
Please evaluate my understanding wrt access files
Robert Lopez put forth on 10/30/2009 6:57 PM: It is not clear to me what the benefit of multiple files is beyond this association. Organization and ease of management for one. For example: smtpd_client_restrictions = check_recipient_access hash:/etc/postfix/access check_client_access hash:/etc/postfix/access pcre:/etc/postfix/check_client_fqdn.pcre hash:/etc/postfix/coolsavings.access hash:/etc/postfix/richk-1.access cidr:/etc/postfix/cidr_files/china.cidr cidr:/etc/postfix/cidr_files/korea.cidr cidr:/etc/postfix/cidr_files/russia.cidr cidr:/etc/postfix/cidr_files/ukraine.cidr cidr:/etc/postfix/cidr_files/malaysia.cidr cidr:/etc/postfix/cidr_files/belarus.cidr cidr:/etc/postfix/cidr_files/indonesia.cidr cidr:/etc/postfix/cidr_files/hongkong.cidr cidr:/etc/postfix/cidr_files/africa.cidr cidr:/etc/postfix/cidr_files/romania.cidr cidr:/etc/postfix/cidr_files/thailand.cidr cidr:/etc/postfix/cidr_files/poland.cidr cidr:/etc/postfix/cidr_files/spammer.cidr cidr:/etc/postfix/cidr_files/hurricane-electric.cidr cidr:/etc/postfix/cidr_files/richk-1.cidr pcre:/etc/postfix/access.pcre My access file contains some whitelist email addresses, some whitelist domains, some blacklist domains, and some whitelist and blacklist IP addresses, so I do have some consolidation in the one file and I use it in multiple restriction classes. However, it's by far my smallest table file. Some of my cidr files are pretty large. Note that I'm using the IPdeny (http://www.ipdeny.com) data and rejecting entire countries' smtp connections. Some of those files have thousands of entries. Note also that I'm using multiple table types, hash, cidr, and pcre. It's better to use multiple files in this kind of setup. This may be another point where I am confused. I am thinking relay is when a postfix server accepts email for u...@cnm.edu and then rewrites that address to what is found in a table for the user where the email is then sent to u...@other.domain. I may have to read that a few times to get it all straight. No, relay strictly means Postfix is not the final destination for a given domain, thus Postfix relays the email to the appropriate server, or rejects the email if Postfix is not configured to relay for a given domain. See: http://www.postfix.org/postconf.5.html#relay_domains Is that cnm.edu ESMTP or default? It's a simple banner message displayed to a remote host when it connects to deliver mail to your MX. You don't need to define it. Postfix does it for you. Just delete that line from main.cf. Humans never see it anyway, unless they telnet to your port TCP 25 for testing or something. Here's an example of the Postfix default banner (mine): 220 greer.hardwarefreak.com ESMTP Postfix Here's the banner on Wietse's server. Wietse is the creator of Postfix. 220 spike.porcupine.org ESMTP Postfix (2.7-20091023) IMHO, there's really no good reason to change the default banner. -- Stan
Re: Please evaluate my understanding wrt access files
On 10/30/2009 9:05 PM, Stan Hoeppner wrote: Robert Lopez put forth on 10/30/2009 6:57 PM: It is not clear to me what the benefit of multiple files is beyond this association. Organization and ease of management for one. For example: smtpd_client_restrictions = check_recipient_access hash:/etc/postfix/access check_client_access hash:/etc/postfix/access pcre:/etc/postfix/check_client_fqdn.pcre hash:/etc/postfix/coolsavings.access hash:/etc/postfix/richk-1.access cidr:/etc/postfix/cidr_files/china.cidr cidr:/etc/postfix/cidr_files/korea.cidr cidr:/etc/postfix/cidr_files/russia.cidr cidr:/etc/postfix/cidr_files/ukraine.cidr cidr:/etc/postfix/cidr_files/malaysia.cidr cidr:/etc/postfix/cidr_files/belarus.cidr cidr:/etc/postfix/cidr_files/indonesia.cidr cidr:/etc/postfix/cidr_files/hongkong.cidr cidr:/etc/postfix/cidr_files/africa.cidr cidr:/etc/postfix/cidr_files/romania.cidr cidr:/etc/postfix/cidr_files/thailand.cidr cidr:/etc/postfix/cidr_files/poland.cidr cidr:/etc/postfix/cidr_files/spammer.cidr cidr:/etc/postfix/cidr_files/hurricane-electric.cidr cidr:/etc/postfix/cidr_files/richk-1.cidr pcre:/etc/postfix/access.pcre It's poor form to omit the check_client_access in front of each map name. All these cidr maps are a great example of separate tables that should be combined into a single table with a Makefile. Maintain the separate input files from different sources for easy management, then combine them into one table for postfix to use. My access file contains some whitelist email addresses, some whitelist domains, some blacklist domains, and some whitelist and blacklist IP addresses, so I do have some consolidation in the one file and I use it in multiple restriction classes.However, it's by far my smallest table file. Some of my cidr files are pretty large. Note that I'm using the IPdeny (http://www.ipdeny.com) data and rejecting entire countries' smtp connections. Some of those files have thousands of entries. Note also that I'm using multiple table types, hash, cidr, and pcre. It's better to use multiple files in this kind of setup. large cidr tables can use huge amounts of memory. If you really feel you need these, 1) consolidate them into one table, and 2) use proxymap to open the table once per postfix instance rather than once per smtpd process. If the data is available in multiple forms, hash: or cdb: tables scale much better than cidr, and a local DNSBL running rbldnsd scales even better. http://www.corpit.ru/mjt/rbldnsd.html -- Noel Jones
Re: Please evaluate my understanding wrt access files
On 10/30/2009 6:57 PM, Robert Lopez wrote: Postfix places no limit on how many maps you can use, but there is system overhead with each map. Rule of thumb is to combine maps wherever possible -- don't use two check_sender_access statements if you can do it with one. The smart way to do this is use a Makefile to build a single map from multiple similar input files. That is interesting. What is the advantage of that over directly editing a single file? I can see having unique names that pair with the parameters that cause them to be read. It is not clear to me what the benefit of multiple files is beyond this association. We do something similar with the virtualaliases table. There is a table that has all college employees who use an Exchange server, another that has all customers (students) who use Sungard Luminis, and a third that has Mailman lists. So email is delivered to one of those three systems based on that file. We build that single file from three separate files. That's a good example of files that can be automated with a Makefile. Maintain the separate files for clear management separation, then just type make to build a single postfix file. General example here: http://www.postfix.org/DATABASE_README.html#safe_db message_size_limit = 16777216 mydestination = $myhostname, $mydomain, localhost.localdomain, cnm.edu, mail.cnm.edu, mg0x.cnm.edu, mg04.cnm.edu, mg05.cnm.edu, nmvc.org, mail.nmvc.org, mg0x.nmvc.org, mg04.nmvc.org, mg05.nmvc.org, mg06.nmvc, nmvirtualcollege.org, mail.nmvirtualcollege.org, mg0x.nmvirtualcollege.org, mg04.nmvirtualcollege.org, mg05.nmvirtualcollege.org, mg04.nmvirtualcollege.org, nmln.net, ideal-nm.org, ideal-nm.net, idealnm.org, idealnm.net Lots of domains in mydestination... Are you sure these don't belong in relay_domains instead? http://www.postfix.org/BASIC_CONFIGURATION_README.html http://www.postfix.org/STANDARD_CONFIGURATION_README.html http://www.postfix.org/ADDRESS_CLASS_README.html No, I am not sure. All email going students and employees are sent to either Sungard Luminis servers or to Microsoft Exchange servers. At the most basic definition, relay_domains are domains that are accepted by postfix and sent to the same address on another box for final delivery. Sounds as if these are all relay_domains. Valid recipients for relay_domains should be listed in relay_recipient_maps, but it sounds as if you list them in virtual_alias_maps. Better to work with the system rather than against it. relay_domains = $mydestination relay_domains should be set explicitly, and generally should not include $mydestination. If there are no relay_domains, it should be set empty. http://www.postfix.org/ADDRESS_CLASS_README.html This may be another point where I am confused. I am thinking relay is when a postfix server accepts email for u...@cnm.edu and then rewrites that address to what is found in a table for the user where the email is then sent to u...@other.domain. Postfix would call those virtual_alias_domains -- domains that are accepted and rewritten to another domain for either local or remote delivery. Your domains are relay_domains. smtpd_banner = cnm.edu Should be cnm.edu ESTMP, or better, just leave it at the default. Is that cnm.edu ESMTP or default? The ESTMP is required to signal other mail servers that your server accepts enhanced command syntax. But there's usually no reason to change this setting from the default $myhostname ESTMP $mail_name, so just remove the parameter from your main.cf. You can't hide the fact you're running postfix, so don't worry too much about the $mail_name in there. smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/greylist check_sender_access hash:/etc/postfix/sender_access permit_mynetworks reject_unknown_sender_domain Seems like permit_mynetworks should come before greylist or other sender access checks. Seems like? :-) This greylist program reads the tail of the mail log and looks for bursts of email in a short period of time. The program adds the bursting account to the map which causes the email to be deferred. OK, you're using that as a outgoing quota control. There are policy servers that do similar and more, particularly policyd. http://www.postfix.org/addon.html#policy -- Noel Jones
Please evaluate my understanding wrt access files
Noel Jones put forth on 10/30/2009 11:50 PM: On 10/30/2009 9:05 PM, Stan Hoeppner wrote: Robert Lopez put forth on 10/30/2009 6:57 PM: It is not clear to me what the benefit of multiple files is beyond this association. Organization and ease of management for one. For example: smtpd_client_restrictions = check_recipient_access hash:/etc/postfix/access check_client_access hash:/etc/postfix/access pcre:/etc/postfix/check_client_fqdn.pcre hash:/etc/postfix/coolsavings.access hash:/etc/postfix/richk-1.access cidr:/etc/postfix/cidr_files/china.cidr cidr:/etc/postfix/cidr_files/korea.cidr cidr:/etc/postfix/cidr_files/russia.cidr cidr:/etc/postfix/cidr_files/ukraine.cidr cidr:/etc/postfix/cidr_files/malaysia.cidr cidr:/etc/postfix/cidr_files/belarus.cidr cidr:/etc/postfix/cidr_files/indonesia.cidr cidr:/etc/postfix/cidr_files/hongkong.cidr cidr:/etc/postfix/cidr_files/africa.cidr cidr:/etc/postfix/cidr_files/romania.cidr cidr:/etc/postfix/cidr_files/thailand.cidr cidr:/etc/postfix/cidr_files/poland.cidr cidr:/etc/postfix/cidr_files/spammer.cidr cidr:/etc/postfix/cidr_files/hurricane-electric.cidr cidr:/etc/postfix/cidr_files/richk-1.cidr pcre:/etc/postfix/access.pcre It's poor form to omit the check_client_access in front of each map name. It may be poor form, but it's legal, gives the same result, and main.cf doesn't look nearly as busy/cluttered by omitting it. If/when it becomes mandatory, I'll add it. I hope it doesn't. All these cidr maps are a great example of separate tables that should be combined into a single table with a Makefile. Interesting. I may take a look into Makefile some time. I was unaware of it until today. Just out of curiosity, how does putting it all in one file make the resulting single file smaller than the sum of the individual files' contents? Does Makefile compress the contents in some way, such as postmap indexes hash files? I ask as the only overlapping data in my CIDR files is the REJECT + custom error within each CIDR file. Maintain the separate input files from different sources for easy management, then combine them into one table for postfix to use. Again, why does a single large map file consume less memory than a bunch of smaller map files, given there is no overlap but some of the custom error message text? My access file contains some whitelist email addresses, some whitelist domains, some blacklist domains, and some whitelist and blacklist IP addresses, so I do have some consolidation in the one file and I use it in multiple restriction classes.However, it's by far my smallest table file. Some of my cidr files are pretty large. Note that I'm using the IPdeny (http://www.ipdeny.com) data and rejecting entire countries' smtp connections. Some of those files have thousands of entries. Note also that I'm using multiple table types, hash, cidr, and pcre. It's better to use multiple files in this kind of setup. large cidr tables can use huge amounts of memory. If you really feel you need these, 1) consolidate them into one table, and 2) use proxymap to open the table once per postfix instance rather than once per smtpd process. If the data is available in multiple forms, hash: or cdb: tables scale much better than cidr, and a local DNSBL running rbldnsd scales even better. http://www.corpit.ru/mjt/rbldnsd.html This is moot in a low volume environment serving around 1000 connections a day, with no concurrency, is it not? My smptd's currently use 8MB VIRT and 4MB RES, and IIRC the max I've ever seen were two smtpd processes running simultaneously. Now, should I start hitting 25 to 50K connections per day, then I should probably look at optimizing my map files a bit, no? Thanks for the hints Noel. I may need them down the road, although not at the moment. Though I am curious and may play around with Makefile just to learn something. -- Stan
