Re: Please evaluate my understanding wrt access files
On 10/31/2009 7:39 PM, Stan Hoeppner wrote: Noel Jones put forth on 10/31/2009 1:12 AM: Each lookup table requires overhead. 30 separate tables requires considerably more overhead than one table. The size of the dataset doesn't change, it's the overhead that gets smaller. The more concurrent smtpd processes running, the more it matters. I guess I was kinda looking for the technical definition of 'overhead' here. If the overhead per file is a straight percentage, then it shouldn't matter if it's one big file or 30 smaller files. "overhead" is pretty much independent of file size. Each extra lookup table is another open file, buffer space, another library call (that might allocate its own buffer), and an additional lookup. In a tiny environment it's not enough to make a difference -- probably not enough to even notice. But one lookup always beats two lookups, one open file always uses less resources than two open files. Run enough smtpd processes and it starts to make a difference; at some point it can make a critical difference. And it's always "prettier" to do one thing one time vs. doing the same thing over and over. What is the actual memory footprint overhead that's being added by Posfix for each map file? Measure it on your system. Probably not a lot, but it adds up. And is the amount of overhead the same for all map types, or different/variable depending on map type? Yes, different table types use different system libraries and have different resource requirements. But for each table type it's a fairly constant value unaffected by table size (very small tables might use a little less RAM under some conditions -- but this really doesn't matter). Is it something like 3 bytes of overhead added for every 10 bytes in a map file, for example? No, it's fixed. I can understand that having everything in one map file, or few map files, compared to many, can increase lookup performance due to indexing efficiency. What I don't understand is how having fewer larger map files decreases memory footprint. Fewer files = less overhead. Size doesn't matter here. Also, given that cidr files aren't postmap'd, is it safe to assume they aren't indexed? If they aren't indexed, how does putting the contents in a single file speed up searching? One lookup beats two every time. If they are indexed when Postfix starts or reloads, then why wouldn't Postfix created a single index _after_ reading in all the data from all the cidr files, creating one internal program table? How is postfix supposed to know you want all these tables treated as one? Anyway, postfix would still need to open the files etc. so all you would save is the extra lookups. It's easy enough for you to control by building the tables the way you want them used that postfix shouldn't be guessing. -- Noel Jones
Re: Please evaluate my understanding wrt access files
Stan Hoeppner a écrit : > mouss put forth on 10/31/2009 11:06 AM: > > mouss, you rock. > >> you can use a script if you prefer. the advantage of 'make' is that it >> only re-generates files when needed (source change). > > The only likely changes would be adding another country. sometimes, you find a new IP that wasn't there, or an IP is assigned to another country... rare, but may happen. > In this case, > would I just add the file name to the "source" section below, and run > the make script? > yes, just add a COUNTRIES += nowhereland line and it should work. >> [snip] > Neat. I'm using gnu make so this should all work. Still don't > understand how this would affect memory footprint, but at least it would > clean up my main.cf a bit. > it is faster to search a single file, rather than doing 10 searches. Think of an IP that is not listed in any of your country maps. with 10 files, you multiply the number of lookups by 10 compared to searching in a single file. of course latency isn't a problem in your setup. but beauty is enough reason! see below. >> PS. since you use a cidr specific directory, you can get rid of the >> ".cidr" suffix. you could then use (in main.cf): >> >> cidr=cidr:/etc/postfix/cidr_files >> smtpd_foo_restrictions = >> ... >> check_client_access ${cidr}/access_foo >> ... >> check_client_access ${cidr}/access_bar > > Ahh, also sweet. I'd seen folks using variables but I hadn't really > found a need yet to dig into it. That and I was afraid of confusing > myself down the road. ;) Just implemented this. Works fine, main.cf > much cleaner looking, yep. it avoids repetition and long lines. maps_dir = /etc/postfix/maps cdb = cdb:${maps_dir}/cdb pcre = pcre:${maps_dir}/pcre mysql = mysql:${maps_dir}/mysql ... virtual_mailbox_maps = ${sql}/virtual_mailbox virtual_alias_maps = ${sql}/virtual_alias smtpd_foo_restrictions = # per recipient access rules check_recipient_access ${mysql}/access_recipient check_recipient_access ${pcre}/access_recipient ... # selectively reject mail from "generic rDNS" and invalid PTR: check_helo_access ${sql}/access_host check_helo_access ${pcre}/access_host check_reverse_client_hostname_access ${sql}/access_host check_reverse_client_hostname_access ${pcre}/access_host ... PS. on the other hand, postconf -n doesn't show the custom variables. but this shouldn't be a problem as long as you don't abuse variables. >_and_ I'm now practicing "good form" with > "check_client_access" in front of each CIDR map. ;) > yes. this will allow you to move/copy checks between different restrictions. In particular, you can move checks to a restriction class withouth having to wonder which restriction calls the class. (yes, I am not very clear. but I hope you see what I mean ;-). > Lots of good ancillary info coming out of this thread. Thanks guys. > Still wondering about the map file memory overhead details... > > -- > Stan
Please evaluate my understanding wrt access files
mouss put forth on 10/31/2009 11:06 AM: mouss, you rock. > you can use a script if you prefer. the advantage of 'make' is that it > only re-generates files when needed (source change). The only likely changes would be adding another country. In this case, would I just add the file name to the "source" section below, and run the make script? > for your country maps and assuming gnu make, your Makefile would contain > something like this: > > #--- > > #directpry where we store our cidr maps > CIDR_DIR=/etc/postfix/cidr_files > > #file to generate > COUNTRIES_DEST=${CIDR_DIR}/countries > > #"source" files, withput directory nor suffix. > COUNTRIES += china > COUNTRIES += korea > COUNTRIES += russia > ... > > #non portable commands. gnu make assumed. > CIDR_COUNTRIES=$(addsuffix .cidr, ${COUNTRIES}) > COUNTRY_MAPS=$(addprefix ${CIDR_DIR}/, ${CIDR_COUNTRIES}) > > ${COUNTRIES_DEST}: ${COUNTRY_MAPS} > cat $^ > $@ > > ... > #- > > this will catenate all your china.cidr, korea.cidr, ... into "countries" > which you can then use with > check_client_access cidr:/etc/postfix/cidr_files/countries Neat. I'm using gnu make so this should all work. Still don't understand how this would affect memory footprint, but at least it would clean up my main.cf a bit. > PS. since you use a cidr specific directory, you can get rid of the > ".cidr" suffix. you could then use (in main.cf): > > cidr=cidr:/etc/postfix/cidr_files > smtpd_foo_restrictions = > ... > check_client_access ${cidr}/access_foo > ... > check_client_access ${cidr}/access_bar Ahh, also sweet. I'd seen folks using variables but I hadn't really found a need yet to dig into it. That and I was afraid of confusing myself down the road. ;) Just implemented this. Works fine, main.cf much cleaner looking, _and_ I'm now practicing "good form" with "check_client_access" in front of each CIDR map. ;) Lots of good ancillary info coming out of this thread. Thanks guys. Still wondering about the map file memory overhead details... -- Stan
Please evaluate my understanding wrt access files
Noel Jones put forth on 10/31/2009 1:12 AM: > Each lookup table requires overhead. 30 separate tables requires > considerably more overhead than one table. The size of the dataset > doesn't change, it's the overhead that gets smaller. The more > concurrent smtpd processes running, the more it matters. I guess I was kinda looking for the technical definition of 'overhead' here. If the overhead per file is a straight percentage, then it shouldn't matter if it's one big file or 30 smaller files. What is the actual memory footprint overhead that's being added by Posfix for each map file? And is the amount of overhead the same for all map types, or different/variable depending on map type? Is it something like 3 bytes of overhead added for every 10 bytes in a map file, for example? I can understand that having everything in one map file, or few map files, compared to many, can increase lookup performance due to indexing efficiency. What I don't understand is how having fewer larger map files decreases memory footprint. > It probably doesn't matter in your environment, but posting examples > suitable for only a very small environment without labeled as such > doesn't help the wider user base. Agreed. That's why I described my environment. It's likely much smaller scale than the average Postfix environment. > And postmap doesn't compress files, but the index allows postfix to find > a particular key quickly -- lookup time and system overhead for hash: > and cdb: is essentially independent of the size of the table, so it > makes sense to use a single table as big as necessary. In particular, > cdb: scales well up to millions of entries, hash: tables easily to > hundreds of thousands. Agreed. Indexed files are always much faster to search than flat text files (unless we're talking really tiny flat files). And it's quicker to search fewer indexes than more indexes. This thread began with the premise being stated that many small map files consume more memory in Postfix than fewer larger map files. Indexes are typically proportional in size to the file being indexed, unless the file contains many duplicate entries. I freely admit I'm no indexing or database expert. However, that said, I don't understand how map file consolidation decreases memory footprint. Maybe Wieste can chime in with a brief explanation. Also, given that cidr files aren't postmap'd, is it safe to assume they aren't indexed? If they aren't indexed, how does putting the contents in a single file speed up searching? If they are indexed when Postfix starts or reloads, then why wouldn't Postfix created a single index _after_ reading in all the data from all the cidr files, creating one internal program table? -- Stan
Re: Please evaluate my understanding wrt access files
Stan Hoeppner a écrit : > [snip] > > Thanks for the hints Noel. I may need them down the road, although not > at the moment. Though I am curious and may play around with Makefile > just to learn something. > you can use a script if you prefer. the advantage of 'make' is that it only re-generates files when needed (source change). for your country maps and assuming gnu make, your Makefile would contain something like this: #--- #directpry where we store our cidr maps CIDR_DIR=/etc/postfix/cidr_files #file to generate COUNTRIES_DEST=${CIDR_DIR}/countries #"source" files, withput directory nor suffix. COUNTRIES += china COUNTRIES += korea COUNTRIES += russia ... #non portable commands. gnu make assumed. CIDR_COUNTRIES=$(addsuffix .cidr, ${COUNTRIES}) COUNTRY_MAPS=$(addprefix ${CIDR_DIR}/, ${CIDR_COUNTRIES}) ${COUNTRIES_DEST}: ${COUNTRY_MAPS} cat $^ > $@ ... #- this will catenate all your china.cidr, korea.cidr, ... into "countries" which you can then use with check_client_access cidr:/etc/postfix/cidr_files/countries PS. since you use a cidr specific directory, you can get rid of the ".cidr" suffix. you could then use (in main.cf): cidr=cidr:/etc/postfix/cidr_files smtpd_foo_restrictions = ... check_client_access ${cidr}/access_foo ... check_client_access ${cidr}/access_bar
Re: Please evaluate my understanding wrt access files
On 10/31/2009 12:32 AM, Stan Hoeppner wrote: All these cidr maps are a great example of separate tables that should be combined into a single table with a Makefile. Interesting. I may take a look into Makefile some time. I was unaware of it until today. Just out of curiosity, how does putting it all in one file make the resulting single file smaller than the sum of the individual files' contents? Does Makefile compress the contents in some way, such as postmap indexes hash files? I ask as the only overlapping data in my CIDR files is the "REJECT" + "custom error" within each CIDR file. Each lookup table requires overhead. 30 separate tables requires considerably more overhead than one table. The size of the dataset doesn't change, it's the overhead that gets smaller. The more concurrent smtpd processes running, the more it matters. It probably doesn't matter in your environment, but posting examples suitable for only a very small environment without labeled as such doesn't help the wider user base. And postmap doesn't compress files, but the index allows postfix to find a particular key quickly -- lookup time and system overhead for hash: and cdb: is essentially independent of the size of the table, so it makes sense to use a single table as big as necessary. In particular, cdb: scales well up to millions of entries, hash: tables easily to hundreds of thousands. -- Noel Jones
Please evaluate my understanding wrt access files
Noel Jones put forth on 10/30/2009 11:50 PM: > On 10/30/2009 9:05 PM, Stan Hoeppner wrote: >> Robert Lopez put forth on 10/30/2009 6:57 PM: >> >>> It is not clear to me what the benefit of multiple files is beyond >>> this association. >> >> Organization and ease of management for one. For example: >> >> smtpd_client_restrictions = >> check_recipient_access hash:/etc/postfix/access >> check_client_access hash:/etc/postfix/access >> pcre:/etc/postfix/check_client_fqdn.pcre >> hash:/etc/postfix/coolsavings.access >> hash:/etc/postfix/richk-1.access >> cidr:/etc/postfix/cidr_files/china.cidr >> cidr:/etc/postfix/cidr_files/korea.cidr >> cidr:/etc/postfix/cidr_files/russia.cidr >> cidr:/etc/postfix/cidr_files/ukraine.cidr >> cidr:/etc/postfix/cidr_files/malaysia.cidr >> cidr:/etc/postfix/cidr_files/belarus.cidr >> cidr:/etc/postfix/cidr_files/indonesia.cidr >> cidr:/etc/postfix/cidr_files/hongkong.cidr >> cidr:/etc/postfix/cidr_files/africa.cidr >> cidr:/etc/postfix/cidr_files/romania.cidr >> cidr:/etc/postfix/cidr_files/thailand.cidr >> cidr:/etc/postfix/cidr_files/poland.cidr >> cidr:/etc/postfix/cidr_files/spammer.cidr >> cidr:/etc/postfix/cidr_files/hurricane-electric.cidr >> cidr:/etc/postfix/cidr_files/richk-1.cidr >> pcre:/etc/postfix/access.pcre > > It's poor form to omit the check_client_access in front of each map name. It may be poor form, but it's legal, gives the same result, and main.cf doesn't look nearly as busy/cluttered by omitting it. If/when it becomes mandatory, I'll add it. I hope it doesn't. > All these cidr maps are a great example of separate tables that should > be combined into a single table with a Makefile. Interesting. I may take a look into Makefile some time. I was unaware of it until today. Just out of curiosity, how does putting it all in one file make the resulting single file smaller than the sum of the individual files' contents? Does Makefile compress the contents in some way, such as postmap indexes hash files? I ask as the only overlapping data in my CIDR files is the "REJECT" + "custom error" within each CIDR file. > Maintain the separate input files from different sources for easy > management, then combine them into one table for postfix to use. Again, why does a single large map file consume less memory than a bunch of smaller map files, given there is no overlap but some of the custom error message text? >> My access file contains some whitelist email addresses, some whitelist >> domains, some blacklist domains, and some whitelist and blacklist IP >> addresses, so I do have some consolidation in the one file and I use it >> in multiple restriction classes.However, it's by far my smallest table >> file. Some of my cidr files are pretty large. Note that I'm using the >> IPdeny (http://www.ipdeny.com) data and rejecting entire countries' smtp >> connections. Some of those files have thousands of entries. Note also >> that I'm using multiple table types, hash, cidr, and pcre. It's better >> to use multiple files in this kind of setup. > > large cidr tables can use huge amounts of memory. If you really feel > you need these, 1) consolidate them into one table, and 2) use proxymap > to open the table once per postfix instance rather than once per smtpd > process. > > If the data is available in multiple forms, hash: or cdb: tables scale > much better than cidr, and a local DNSBL running rbldnsd scales even > better. > http://www.corpit.ru/mjt/rbldnsd.html This is moot in a low volume environment serving around 1000 connections a day, with no concurrency, is it not? My smptd's currently use 8MB VIRT and 4MB RES, and IIRC the max I've ever seen were two smtpd processes running simultaneously. Now, should I start hitting 25 to 50K connections per day, then I should probably look at optimizing my map files a bit, no? Thanks for the hints Noel. I may need them down the road, although not at the moment. Though I am curious and may play around with Makefile just to learn something. -- Stan
Re: Please evaluate my understanding wrt access files
On 10/30/2009 6:57 PM, Robert Lopez wrote: Postfix places no limit on how many maps you can use, but there is system overhead with each map. Rule of thumb is to combine maps wherever possible -- don't use two check_sender_access statements if you can do it with one. The smart way to do this is use a Makefile to build a single map from multiple similar input files. That is interesting. What is the advantage of that over directly editing a single file? I can see having unique names that pair with the parameters that cause them to be read. It is not clear to me what the benefit of multiple files is beyond this association. We do something similar with the virtualaliases table. There is a table that has all college employees who use an Exchange server, another that has all customers (students) who use Sungard Luminis, and a third that has Mailman lists. So email is delivered to one of those three systems based on that file. We build that single file from three separate files. That's a good example of files that can be automated with a Makefile. Maintain the separate files for clear management separation, then just type "make" to build a single postfix file. General example here: http://www.postfix.org/DATABASE_README.html#safe_db message_size_limit = 16777216 mydestination = $myhostname, $mydomain, localhost.localdomain, cnm.edu, mail.cnm.edu, mg0x.cnm.edu, mg04.cnm.edu, mg05.cnm.edu, nmvc.org, mail.nmvc.org, mg0x.nmvc.org, mg04.nmvc.org, mg05.nmvc.org, mg06.nmvc, nmvirtualcollege.org, mail.nmvirtualcollege.org, mg0x.nmvirtualcollege.org, mg04.nmvirtualcollege.org, mg05.nmvirtualcollege.org, mg04.nmvirtualcollege.org, nmln.net, ideal-nm.org, ideal-nm.net, idealnm.org, idealnm.net Lots of domains in mydestination... Are you sure these don't belong in relay_domains instead? http://www.postfix.org/BASIC_CONFIGURATION_README.html http://www.postfix.org/STANDARD_CONFIGURATION_README.html http://www.postfix.org/ADDRESS_CLASS_README.html No, I am not sure. All email going students and employees are sent to either Sungard Luminis servers or to Microsoft Exchange servers. At the most basic definition, relay_domains are domains that are accepted by postfix and sent to the same address on another box for final delivery. Sounds as if these are all relay_domains. Valid recipients for relay_domains should be listed in relay_recipient_maps, but it sounds as if you list them in virtual_alias_maps. Better to work with the system rather than against it. relay_domains = $mydestination relay_domains should be set explicitly, and generally should not include $mydestination. If there are no relay_domains, it should be set empty. http://www.postfix.org/ADDRESS_CLASS_README.html This may be another point where I am confused. I am thinking relay is when a postfix server accepts email for u...@cnm.edu and then rewrites that address to what is found in a table for the user where the email is then sent to u...@other.domain. Postfix would call those virtual_alias_domains -- domains that are accepted and rewritten to another domain for either local or remote delivery. Your domains are relay_domains. smtpd_banner = cnm.edu Should be "cnm.edu ESTMP", or better, just leave it at the default. Is that "cnm.edu ESMTP" or default? The ESTMP is required to signal other mail servers that your server accepts "enhanced" command syntax. But there's usually no reason to change this setting from the default "$myhostname ESTMP $mail_name", so just remove the parameter from your main.cf. You can't hide the fact you're running postfix, so don't worry too much about the $mail_name in there. smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/greylist check_sender_access hash:/etc/postfix/sender_access permit_mynetworks reject_unknown_sender_domain Seems like permit_mynetworks should come before greylist or other sender access checks. Seems like? :-) This greylist program reads the tail of the mail log and looks for bursts of email in a short period of time. The program adds the bursting account to the map which causes the email to be deferred. OK, you're using that as a outgoing quota control. There are policy servers that do similar and more, particularly policyd. http://www.postfix.org/addon.html#policy -- Noel Jones
Re: Please evaluate my understanding wrt access files
On 10/30/2009 9:05 PM, Stan Hoeppner wrote: Robert Lopez put forth on 10/30/2009 6:57 PM: It is not clear to me what the benefit of multiple files is beyond this association. Organization and ease of management for one. For example: smtpd_client_restrictions = check_recipient_access hash:/etc/postfix/access check_client_access hash:/etc/postfix/access pcre:/etc/postfix/check_client_fqdn.pcre hash:/etc/postfix/coolsavings.access hash:/etc/postfix/richk-1.access cidr:/etc/postfix/cidr_files/china.cidr cidr:/etc/postfix/cidr_files/korea.cidr cidr:/etc/postfix/cidr_files/russia.cidr cidr:/etc/postfix/cidr_files/ukraine.cidr cidr:/etc/postfix/cidr_files/malaysia.cidr cidr:/etc/postfix/cidr_files/belarus.cidr cidr:/etc/postfix/cidr_files/indonesia.cidr cidr:/etc/postfix/cidr_files/hongkong.cidr cidr:/etc/postfix/cidr_files/africa.cidr cidr:/etc/postfix/cidr_files/romania.cidr cidr:/etc/postfix/cidr_files/thailand.cidr cidr:/etc/postfix/cidr_files/poland.cidr cidr:/etc/postfix/cidr_files/spammer.cidr cidr:/etc/postfix/cidr_files/hurricane-electric.cidr cidr:/etc/postfix/cidr_files/richk-1.cidr pcre:/etc/postfix/access.pcre It's poor form to omit the check_client_access in front of each map name. All these cidr maps are a great example of separate tables that should be combined into a single table with a Makefile. Maintain the separate input files from different sources for easy management, then combine them into one table for postfix to use. My access file contains some whitelist email addresses, some whitelist domains, some blacklist domains, and some whitelist and blacklist IP addresses, so I do have some consolidation in the one file and I use it in multiple restriction classes.However, it's by far my smallest table file. Some of my cidr files are pretty large. Note that I'm using the IPdeny (http://www.ipdeny.com) data and rejecting entire countries' smtp connections. Some of those files have thousands of entries. Note also that I'm using multiple table types, hash, cidr, and pcre. It's better to use multiple files in this kind of setup. large cidr tables can use huge amounts of memory. If you really feel you need these, 1) consolidate them into one table, and 2) use proxymap to open the table once per postfix instance rather than once per smtpd process. If the data is available in multiple forms, hash: or cdb: tables scale much better than cidr, and a local DNSBL running rbldnsd scales even better. http://www.corpit.ru/mjt/rbldnsd.html -- Noel Jones
Please evaluate my understanding wrt access files
Robert Lopez put forth on 10/30/2009 6:57 PM: > It is not clear to me what the benefit of multiple files is beyond > this association. Organization and ease of management for one. For example: smtpd_client_restrictions = check_recipient_access hash:/etc/postfix/access check_client_access hash:/etc/postfix/access pcre:/etc/postfix/check_client_fqdn.pcre hash:/etc/postfix/coolsavings.access hash:/etc/postfix/richk-1.access cidr:/etc/postfix/cidr_files/china.cidr cidr:/etc/postfix/cidr_files/korea.cidr cidr:/etc/postfix/cidr_files/russia.cidr cidr:/etc/postfix/cidr_files/ukraine.cidr cidr:/etc/postfix/cidr_files/malaysia.cidr cidr:/etc/postfix/cidr_files/belarus.cidr cidr:/etc/postfix/cidr_files/indonesia.cidr cidr:/etc/postfix/cidr_files/hongkong.cidr cidr:/etc/postfix/cidr_files/africa.cidr cidr:/etc/postfix/cidr_files/romania.cidr cidr:/etc/postfix/cidr_files/thailand.cidr cidr:/etc/postfix/cidr_files/poland.cidr cidr:/etc/postfix/cidr_files/spammer.cidr cidr:/etc/postfix/cidr_files/hurricane-electric.cidr cidr:/etc/postfix/cidr_files/richk-1.cidr pcre:/etc/postfix/access.pcre My access file contains some whitelist email addresses, some whitelist domains, some blacklist domains, and some whitelist and blacklist IP addresses, so I do have some consolidation in the one file and I use it in multiple restriction classes. However, it's by far my smallest table file. Some of my cidr files are pretty large. Note that I'm using the IPdeny (http://www.ipdeny.com) data and rejecting entire countries' smtp connections. Some of those files have thousands of entries. Note also that I'm using multiple table types, hash, cidr, and pcre. It's better to use multiple files in this kind of setup. > This may be another point where I am confused. I am thinking relay is when a > postfix server accepts email for u...@cnm.edu and then rewrites that address > to what is found in a table for the user where the email is then sent > to u...@other.domain. > I may have to read that a few times to get it all straight. No, relay strictly means Postfix is not the final destination for a given domain, thus Postfix relays the email to the appropriate server, or rejects the email if Postfix is not configured to relay for a given domain. See: http://www.postfix.org/postconf.5.html#relay_domains > Is that "cnm.edu ESMTP" or default? It's a simple banner message displayed to a remote host when it connects to deliver mail to your MX. You don't need to define it. Postfix does it for you. Just delete that line from main.cf. Humans never see it anyway, unless they telnet to your port TCP 25 for testing or something. Here's an example of the Postfix default banner (mine): 220 greer.hardwarefreak.com ESMTP Postfix Here's the banner on Wietse's server. Wietse is the creator of Postfix. 220 spike.porcupine.org ESMTP Postfix (2.7-20091023) IMHO, there's really no good reason to change the default banner. -- Stan
Re: Please evaluate my understanding wrt access files
On Fri, Oct 30, 2009 at 1:26 PM, Noel Jones wrote: > On 10/30/2009 12:55 PM, Robert Lopez wrote: >> >> I would like to confirm my understanding about access files. >> >> Please let me know if any of this is not correct... >> >> The man (5) access description describes a prototype file, where that >> file could be a single file describing any host names, network >> addresses, envelope senders or recipient addresses. >> >> The file could also be a set of files all following the same format >> rules. >> >> Where such files might be >> recipient_checks, helo_checks, sender_checks, client_checks, etc. >> >> >> The usefulness of the content of an access file is dependent upon the >> parameter that selects a routine that reads the file. >> >> If check_client_access causes a read of the file it will only be >> looking for IP addresses of a client server that sent the email or a >> fully qualified domain name that successfully reverse maps to the IP >> address of a client server that sent the email. >> >> If check_sender_access causes a read of the file it will only be >> looking for an email SMTP "MAIL FROM" address or a pattern which could >> be a part that email address to the left of the "@" sign. >> >> If check_helo_access causes a read of the file it will only be looking >> for the HELO or EHLO hostname or any valid parent domain of that >> hostname that is in the SMTP HELO. >> >> The routines executed vi the parameters such as check_client_access, >> check_sender_access, check_helo_access, etc. return the value the >> check to the routine that called for the check where the calling >> routine would be instigated by any of these parameters: >> >> smtpd_client_restrictions >> smtpd_helo_restrictions >> smtpd_sender_restrictions >> smtpd_recipient_restrictions >> smtpd_data_restrictions >> >> It is possible to have all the lookups done on a single >> .../postfix/access.db file but that could mean the file gets confusing >> so in practice multiple access files with names like client_access, >> helo_access, sender_access, etc. > > so far so good... > > >> >> A single parameter such as check_client_access may be called multiple >> times in a situation like this: >> >> smtpd_sender_restrictions = >> check_sender_access hash:/etc/postfix/greylist >> check_sender_access hash:/etc/postfix/sender_access >> permit_mynetworks >> >> >> However if the above causes a pattern to be found more than once then >> only the last pattern match is used. (I think that is what "When the >> same parameter is defined multiple times, only the last instance is >> remembered." means.) > > No, the "last parameter is used" rule refers to when indexing a single map > -- when the map is indexed with postmap, a duplicate key will overwrite the > previous identical key. > > With multiple maps -- or multiple restrictions of any type -- in > smtpd_*_restrictions, the first match wins. Ah! Good. That now makes more sense. > > Postfix places no limit on how many maps you can use, but there is system > overhead with each map. Rule of thumb is to combine maps wherever possible > -- don't use two check_sender_access statements if you can do it with one. > The smart way to do this is use a Makefile to build a single map from > multiple similar input files. That is interesting. What is the advantage of that over directly editing a single file? I can see having unique names that pair with the parameters that cause them to be read. It is not clear to me what the benefit of multiple files is beyond this association. We do something similar with the virtualaliases table. There is a table that has all college employees who use an Exchange server, another that has all customers (students) who use Sungard Luminis, and a third that has Mailman lists. So email is delivered to one of those three systems based on that file. We build that single file from three separate files. > >> >> This is how I am putting this in practice on a new virtual server >> where I hope to fix some problems on current production servers: >> >> r...@mg0x:/etc/postfix# postconf -d mail_version >> mail_version = 2.5.5 >> >> I am using 2.5.5 because that is the latest from Ubuntu. >> >> >> r...@mg0x:/etc/postfix# postconf -n >> alias_database = hash:/etc/aliases >> alias_maps = hash:/etc/aliases >> append_dot_mydomain = yes >> biff = no >> bounce_size_limit = 1 >> config_directory = /etc/postfix >> default_process_limit = 400 >> header_checks = regexp:/etc/postfix/header_checks >> inet_interfaces = all >> mailbox_size_limit = 0 >> masquerade_domains = $mydomain, cnm.edu, nmvc.org, nmvirtualcollege.org >> max_use = 100 >> message_size_limit = 16777216 >> mydestination = $myhostname, $mydomain, localhost.localdomain, >> cnm.edu, mail.cnm.edu, mg0x.cnm.edu, mg04.cnm.edu, mg05.cnm.edu, >> nmvc.org, mail.nmvc.org, mg0x.nmvc.org, mg04.nmvc.org, mg05.nmvc.org, >> mg06.nmvc, nmvirtualcollege.org, mail.nmvirtualcollege.org, >> mg0x.nmvirtualcollege.org, mg04.nmvirtualco
Re: Please evaluate my understanding wrt access files
On 10/30/2009 12:55 PM, Robert Lopez wrote: I would like to confirm my understanding about access files. Please let me know if any of this is not correct... The man (5) access description describes a prototype file, where that file could be a single file describing any host names, network addresses, envelope senders or recipient addresses. The file could also be a set of files all following the same format rules. Where such files might be recipient_checks, helo_checks, sender_checks, client_checks, etc. The usefulness of the content of an access file is dependent upon the parameter that selects a routine that reads the file. If check_client_access causes a read of the file it will only be looking for IP addresses of a client server that sent the email or a fully qualified domain name that successfully reverse maps to the IP address of a client server that sent the email. If check_sender_access causes a read of the file it will only be looking for an email SMTP "MAIL FROM" address or a pattern which could be a part that email address to the left of the "@" sign. If check_helo_access causes a read of the file it will only be looking for the HELO or EHLO hostname or any valid parent domain of that hostname that is in the SMTP HELO. The routines executed vi the parameters such as check_client_access, check_sender_access, check_helo_access, etc. return the value the check to the routine that called for the check where the calling routine would be instigated by any of these parameters: smtpd_client_restrictions smtpd_helo_restrictions smtpd_sender_restrictions smtpd_recipient_restrictions smtpd_data_restrictions It is possible to have all the lookups done on a single .../postfix/access.db file but that could mean the file gets confusing so in practice multiple access files with names like client_access, helo_access, sender_access, etc. so far so good... A single parameter such as check_client_access may be called multiple times in a situation like this: smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/greylist check_sender_access hash:/etc/postfix/sender_access permit_mynetworks However if the above causes a pattern to be found more than once then only the last pattern match is used. (I think that is what "When the same parameter is defined multiple times, only the last instance is remembered." means.) No, the "last parameter is used" rule refers to when indexing a single map -- when the map is indexed with postmap, a duplicate key will overwrite the previous identical key. With multiple maps -- or multiple restrictions of any type -- in smtpd_*_restrictions, the first match wins. Postfix places no limit on how many maps you can use, but there is system overhead with each map. Rule of thumb is to combine maps wherever possible -- don't use two check_sender_access statements if you can do it with one. The smart way to do this is use a Makefile to build a single map from multiple similar input files. This is how I am putting this in practice on a new virtual server where I hope to fix some problems on current production servers: r...@mg0x:/etc/postfix# postconf -d mail_version mail_version = 2.5.5 I am using 2.5.5 because that is the latest from Ubuntu. r...@mg0x:/etc/postfix# postconf -n alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases append_dot_mydomain = yes biff = no bounce_size_limit = 1 config_directory = /etc/postfix default_process_limit = 400 header_checks = regexp:/etc/postfix/header_checks inet_interfaces = all mailbox_size_limit = 0 masquerade_domains = $mydomain, cnm.edu, nmvc.org, nmvirtualcollege.org max_use = 100 message_size_limit = 16777216 mydestination = $myhostname, $mydomain, localhost.localdomain, cnm.edu, mail.cnm.edu, mg0x.cnm.edu, mg04.cnm.edu, mg05.cnm.edu, nmvc.org, mail.nmvc.org, mg0x.nmvc.org, mg04.nmvc.org, mg05.nmvc.org, mg06.nmvc, nmvirtualcollege.org, mail.nmvirtualcollege.org, mg0x.nmvirtualcollege.org, mg04.nmvirtualcollege.org, mg05.nmvirtualcollege.org, mg04.nmvirtualcollege.org, nmln.net, ideal-nm.org, ideal-nm.net, idealnm.org, idealnm.net Lots of domains in mydestination... Are you sure these don't belong in relay_domains instead? http://www.postfix.org/BASIC_CONFIGURATION_README.html http://www.postfix.org/STANDARD_CONFIGURATION_README.html http://www.postfix.org/ADDRESS_CLASS_README.html myhostname = mg0x.cnm.edu mynetworks = 198.133.182.0/24, 198.133.181.0/24, 198.133.180.0/24, 172.16.0.0/12, 192.168.0.0/16, 10.0.0.0/8, 127.0.0.0/8 [:::127.0.0.0]/104 [::1]/128 myorigin = /etc/mailname notify_classes = resource, software readme_directory = no recipient_delimiter = + relay_domains = $mydestination relay_domains should be set explicitly, and generally should not include $mydestination. If there are no relay_domains, it should be set empty. http://www.postfix.org/ADDRESS_CLASS_README.html relayhost = smtp_host_lookup = dns, native smtp_tls_session_ca
Please evaluate my understanding wrt access files
I would like to confirm my understanding about access files. Please let me know if any of this is not correct... The man (5) access description describes a prototype file, where that file could be a single file describing any host names, network addresses, envelope senders or recipient addresses. The file could also be a set of files all following the same format rules. Where such files might be recipient_checks, helo_checks, sender_checks, client_checks, etc. The usefulness of the content of an access file is dependent upon the parameter that selects a routine that reads the file. If check_client_access causes a read of the file it will only be looking for IP addresses of a client server that sent the email or a fully qualified domain name that successfully reverse maps to the IP address of a client server that sent the email. If check_sender_access causes a read of the file it will only be looking for an email SMTP "MAIL FROM" address or a pattern which could be a part that email address to the left of the "@" sign. If check_helo_access causes a read of the file it will only be looking for the HELO or EHLO hostname or any valid parent domain of that hostname that is in the SMTP HELO. The routines executed vi the parameters such as check_client_access, check_sender_access, check_helo_access, etc. return the value the check to the routine that called for the check where the calling routine would be instigated by any of these parameters: smtpd_client_restrictions smtpd_helo_restrictions smtpd_sender_restrictions smtpd_recipient_restrictions smtpd_data_restrictions It is possible to have all the lookups done on a single .../postfix/access.db file but that could mean the file gets confusing so in practice multiple access files with names like client_access, helo_access, sender_access, etc. A single parameter such as check_client_access may be called multiple times in a situation like this: smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/greylist check_sender_access hash:/etc/postfix/sender_access permit_mynetworks However if the above causes a pattern to be found more than once then only the last pattern match is used. (I think that is what "When the same parameter is defined multiple times, only the last instance is remembered." means.) This is how I am putting this in practice on a new virtual server where I hope to fix some problems on current production servers: r...@mg0x:/etc/postfix# postconf -d mail_version mail_version = 2.5.5 I am using 2.5.5 because that is the latest from Ubuntu. r...@mg0x:/etc/postfix# postconf -n alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases append_dot_mydomain = yes biff = no bounce_size_limit = 1 config_directory = /etc/postfix default_process_limit = 400 header_checks = regexp:/etc/postfix/header_checks inet_interfaces = all mailbox_size_limit = 0 masquerade_domains = $mydomain, cnm.edu, nmvc.org, nmvirtualcollege.org max_use = 100 message_size_limit = 16777216 mydestination = $myhostname, $mydomain, localhost.localdomain, cnm.edu, mail.cnm.edu, mg0x.cnm.edu, mg04.cnm.edu, mg05.cnm.edu, nmvc.org, mail.nmvc.org, mg0x.nmvc.org, mg04.nmvc.org, mg05.nmvc.org, mg06.nmvc, nmvirtualcollege.org, mail.nmvirtualcollege.org, mg0x.nmvirtualcollege.org, mg04.nmvirtualcollege.org, mg05.nmvirtualcollege.org, mg04.nmvirtualcollege.org, nmln.net, ideal-nm.org, ideal-nm.net, idealnm.org, idealnm.net myhostname = mg0x.cnm.edu mynetworks = 198.133.182.0/24, 198.133.181.0/24, 198.133.180.0/24, 172.16.0.0/12, 192.168.0.0/16, 10.0.0.0/8, 127.0.0.0/8 [:::127.0.0.0]/104 [::1]/128 myorigin = /etc/mailname notify_classes = resource, software readme_directory = no recipient_delimiter = + relay_domains = $mydestination relayhost = smtp_host_lookup = dns, native smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache smtpd_banner = cnm.edu smtpd_client_restrictions = permit_mynetworks check_client_access hash:/etc/postfix/accessreject_rbl_client zen.spamhaus.orgreject_rbl_client bl.spamcop.net reject_rbl_client dnsbl.njabl.org permit smtpd_helo_required = yes smtpd_helo_restrictions = permit_mynetworks reject_invalid_hostname smtpd_recipient_restrictions = check_recipient_access hash:/etc/postfix/overquota reject_non_fqdn_sender reject_unknown_sender_domainreject_non_fqdn_recipient reject_unknown_recipient_domain reject_unlisted_recipient permit_mynetworks reject_unauth_destination reject_unauth_pipeliningreject_invalid_helo_hostname reject_non_fqdn_helo_hostname reject_rbl_client zen.spamhaus.org smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/greylist check_sender_access hash:/etc/postfix/sender_access permit_mynetworks reject_unknown_sender_domain smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key smtpd_tls_session_cach