Re: smtpd processes congregating at the pub
Stan Hoeppner: Wietse Venema put forth on 1/29/2010 6:15 AM: Stan Hoeppner: Based on purely visual non-scientific observation (top), it seems my smtpd processes on my MX hang around much longer in (Debian) 2.5.5 than they did in (Debian) 2.3.8. In 2.3.8 Master seemed to build them and tear them down very Perhaps Debian changed this: http://www.postfix.org/postconf.5.html#max_idle The Postfix default is 100s. Yes, I confirmed this on my system. I don't really see why anyone would shorten this - that's a waste of CPU cycles. In particular, stopping Postfix daemons after 10s Allow me to present a tutorial on Postfix and operating system basics. Postfix reuses processes for the same reasons that Apache does; however, Apache always runs a fixed minimum amount of daemons, whereas Postfix will dynamically shrink to zero smtpd processes over time. Therefore, people who believe that Postfix processes should not be running in the absence of client requests, should also terminate their Apache processes until a connection arrives. No-one does that. If people believe that each smtpd process uses 15MB of RAM, and that two smtpd processes use 30MB of RAM, then that would have been correct had Postfix been running on MS-DOS. First, the physical memory footprint of a process (called resident memory size) is smaller than the virtual memory footprint (which comprises all addressable memory including the executable, libraries, data, heap and stack). With FreeBSD 8.0 I see an smtpd VSZ/RSS of 6.9MB/4.8MB; with Fedora Core 11, 4.2MB/1.8MB; and with FreeBSD 4.1 it's 1.8MB/1.4MB. Ten years of system library bloat. Second, when multiple processes execute the same executable file and libraries, those processes will share a single memory copy of the code and constants of that executable file and libraries. Therefore, a large portion of their resident memory sizes will actually map onto the same physical memory pages. 15+15 != 30. Third, some code uses mmap() to allocate memory that is mapped from a file. This adds to the virtual memory footprint of each process, but of course only the pages that are actually accessed will add to the resident memory size. In the case of Postfix, this mechanism is used by Berkeley DB to allocate a 16MB shared-memory read buffer. There are some other tricks that allow for further savings (such as copy-on-write, which allows sharing of a memory page until a process attempts to write to it) but in the case of Postfix, those savings will be modest. Wietse
Re: allowing outside users access to mailman lists
On Thu, Jan 28, 2010 at 4:02 PM, Jeff Weinberger j...@jeffweinberger.com wrote: On Thu, Jan 28, 2010 at 3:39 PM, Noel Jones njo...@megan.vbhcs.org wrote: On 1/28/2010 5:36 PM, Jeff Weinberger wrote: On Thu, Jan 28, 2010 at 3:16 PM, Noel Jonesnjo...@megan.vbhcs.org wrote: On 1/28/2010 4:46 PM, Jeff Weinberger wrote: virtual_alias_domains = mysql:/etc/postfix/mysql_virtual_alias_domains.cf does lists.mylistserver.com match the above lookup? postmap -q lists.mylistserver.com mysql:... No. `postmap -q lists.mylistserver.com mysql:/etc/postfix/mysql_virtual_alias_domains.cf` returns nothing (empty). However, `postmap -q lists.mylistserver.com mysql:/etc/postfix/mysql_relay_domain_maps.cf returns OK (a constant value, as recommended). I was hopeful that you had identified something here...so I did test to be sure. So enable debug output on smtpd (or add a test client to debug_peer_list) and show us the UNALTERED results of a failed transaction. http://www.postfix.org/DEBUG_README.html#debug_peer OK, I see what this will do - thanks for the suggestion. I'll post the complete log here (naturally, anything sensitive masked, but otherwise unaltered). I'm closing this request. I've found some issues with MySQL on my system (no explanation other than user error for the attempts without mysql), that have now been fixed. This has caused the postfix behavior to return to normal and expected, and all is working as I want. My apologies for bringing this to this forum (though Im sure you'll see how I thought this was a postfix issue). Thank you to all who helped find the ways to diagnose this. This has raised other questions that I do not fully understand, but for clarity will post them separately. Thank you.
%d and %s in mysql queries?
I am using mysql (quite successfully in most cases) to do lookups for a variety of reasons in postfix. Recently, I had some issues with a domain lookup and in the testing tried varying my MySQL query between using %d and %s as the lookup key. The documentation is clear on this when the query is for an address. It may be as clear, and I may not be understanding this, but I would like to ask for help understanding the difference when the lookup is for a domain. In this case, I have in main.cf: relay_domains = mysql:/etc/postfix/mysql_relay_domains.cf and the query in /etc/postfix/mysql_relay_domains.cf is: query = SELECT 'OK' FROM domain WHERE domain='%d' AND type='relay' I chose %d because I'm looking up only the domain in this case. I noticed that this worked fine when the domain in question is of the form domain.tld but didn't work (meaning the result returned from postmap -q... was null) when the domain is of the form sub.domain.tld (even though sub.domain.tld is in the domain table with type relay). When I changed the query to use %s, it worked in both cases. So my questions are: 1) Is this expected behavior? 2) in the case where the lookup is for a domain, what is the operative difference between using %d and %s I am hoping to gain a better understanding of these two substitutions. Recommendations on best-practices are appreciated as well, but I'd like to understand the expected behavior. If this is documented and I missed it or didn't understand it well, please point me to the documentation page.
Re: smtpd processes congregating at the pub
Wietse Venema put forth on 1/30/2010 9:03 AM: Allow me to present a tutorial on Postfix and operating system basics. Thank you Wietse. I'm always eager to learn. :) Postfix reuses processes for the same reasons that Apache does; however, Apache always runs a fixed minimum amount of daemons, whereas Postfix will dynamically shrink to zero smtpd processes over time. Possibly not the best reference example, as I switched to Lighty mainly due to the Apache behavior you describe, but also due to Apache resource hogging in general. But I understand your point. It's better to keep one or two processes resident to service the next inbound requests than to constantly tear down and then rebuild processes, which causes significant overhead and performance issues on busy systems. Therefore, people who believe that Postfix processes should not be running in the absence of client requests, should also terminate their Apache processes until a connection arrives. No-one does that. Wouldn't that really depend on the purpose of the server? How about a web admin daemon running on a small network device? I almost do this with Lighty currently. I have a single daemon instance that handles all requests, max processes=1. It's a very lightly loaded server, and a single instance is more than enough. In fact, given the load, I might possibly look into running Lighty from inetd, if possible, as I do Samba. If people believe that each smtpd process uses 15MB of RAM, and that two smtpd processes use 30MB of RAM, then that would have been correct had Postfix been running on MS-DOS. First, the physical memory footprint of a process (called resident memory size) is smaller than the virtual memory footprint (which comprises all addressable memory including the executable, libraries, data, heap and stack). With FreeBSD 8.0 I see an smtpd VSZ/RSS of 6.9MB/4.8MB; with Fedora Core 11, 4.2MB/1.8MB; and with FreeBSD 4.1 it's 1.8MB/1.4MB. Ten years of system library bloat. Debian 5.0.3, kernel 2.6.31 PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 29242 postfix 20 0 22408 18m 2268 S0 4.9 0:00.58 smtpd 29251 postfix 20 0 17264 13m 2208 S0 3.6 0:00.48 smtpd Second, when multiple processes execute the same executable file and libraries, those processes will share a single memory copy of the code and constants of that executable file and libraries. Therefore, a large portion of their resident memory sizes will actually map onto the same physical memory pages. 15+15 != 30. I was of the understanding that top's SHR column described memory shareable with other processes. In the real example above from earlier today, it would seem that my two smtpd processes can only share ~2.2MB of code, data structures, etc. man top: t: SHR -- Shared Mem size (kb) The amount of shared memory used by a task. It simply reflects memory that could be potentially shared with other processes. Am I missing something, or reading my top output incorrectly? Third, some code uses mmap() to allocate memory that is mapped from a file. This adds to the virtual memory footprint of each process, but of course only the pages that are actually accessed will add to the resident memory size. In the case of Postfix, this mechanism is used by Berkeley DB to allocate a 16MB shared-memory read buffer. Is this 16MB buffer also used for hash and/or cidr tables, and is this shareable? AFAIK I don't use Berkeley DB tables, only hash (small,few) and cidr (very large, a handful). There are some other tricks that allow for further savings (such as copy-on-write, which allows sharing of a memory page until a process attempts to write to it) but in the case of Postfix, those savings will be modest. I must be screwing something up somewhere then. According to my top output, I'm only sharing ~2.2MB between smtpd processes, yet I've seen them occupy anywhere from 11-18MB RES. If the top output is correct, there is a huge amount of additional sharing that should be occurring, no? Debian runs Postfix in a chroot by default. I know very little about chroot environments. Could this have something to do with the tiny amount of shared memory between the smtpds? Thanks for taking interest in this Wietse. I'm sure I've probably done something screwy that is easily fixable, and will get that shared memory count up where it should be. -- Stan
Re: %d and %s in mysql queries?
Jeff Weinberger: I am using mysql (quite successfully in most cases) to do lookups for a variety of reasons in postfix. Recently, I had some issues with a domain lookup and in the testing tried varying my MySQL query between using %d and %s as the lookup key. The documentation is clear on this when the query is for an address. The %d is defined when a key or result is an address: %d When the input key is an address of the form u...@domain, %d When a result attribute value is an address of the form u...@domain Therefore, %d is not defined for non-address forms. Wieste
Re: %d and %s in mysql queries?
On Sat, Jan 30, 2010 at 4:41 PM, Wietse Venema wie...@porcupine.org wrote: Jeff Weinberger: I am using mysql (quite successfully in most cases) to do lookups for a variety of reasons in postfix. Recently, I had some issues with a domain lookup and in the testing tried varying my MySQL query between using %d and %s as the lookup key. The documentation is clear on this when the query is for an address. The %d is defined when a key or result is an address: %d When the input key is an address of the form u...@domain, %d When a result attribute value is an address of the form u...@domain Therefore, %d is not defined for non-address forms. Wieste Thank you! So my use of %d in the case of relay_domains lookup was just incorrect. I understand this now - thanks.
Re: smtpd processes congregating at the pub
Stan Hoeppner: AFAIK I don't use Berkeley DB tables, only hash (small,few) and cidr (very large, a handful). hash (and btree) == Berkeley DB. If you have big CIDR tables, you can save lots of memory by using proxy:cidr: instead of cidr: (and running postfix reload). Effectively, this turns all that private memory into something that can be shared via the proxy: protocol. The current CIDR implementation is optimized to make it easy to verify for correctness, and is optimized for speed when used with limited lists of netblocks (mynetworks, unassigned address blocks, reserved address blocks, etc.). If you want to list large portions of Internet address space such as entire countries the current implementation starts burning CPU time (it examines all CIDR patterns in order; with a bit of extra up-front work during initialization, address lookups could skip over a lot of patterns, but the implementation would of course be harder to verify for correctness), and it wastes 24 bytes per CIDR rule when Postfix is compiled with IPv6 support (this roughly doubles the amount memory that is used by CIDR tables). Wietse
Linux users with mixed case names
On my linux system, I have uses with mixed case names. I have one user RosaliE and I want her to get mail but postfix seems to translate this rosalie. How do I change this behaviour. Thanks Chip
Re: Linux users with mixed case names
Quoting Ralph Blach chipp...@nc.rr.com: On my linux system, I have uses with mixed case names. I have one user RosaliE and I want her to get mail but postfix seems to translate this rosalie. How do I change this behaviour. If this is a local user, AFAIK, you can't change the behavior without hacking the code. CASE FOLDING All delivery decisions are made using the bare recipient name (i.e. the address localpart), folded to lower case. http://www.postfix.org/local.8.html Do you actually want Rosalie and rosalie to refer to two distinct users, or are you just considering this as a display problem? Terry
Re: smtpd processes congregating at the pub
Wietse Venema put forth on 1/30/2010 7:14 PM:
Stan Hoeppner:
AFAIK I don't use Berkeley DB tables, only hash (small,few) and cidr
(very large, a handful).
hash (and btree) == Berkeley DB.
Ahh, good to know. I'd thought only btree used Berkeley DB and that hash tables
used something else.
If you have big CIDR tables, you can save lots of memory by using
proxy:cidr: instead of cidr: (and running postfix reload).
Effectively, this turns all that private memory into something that
can be shared via the proxy: protocol.
I implemented proxymap but it doesn't appear to have changed the memory
footprint of smtpd much at all, if any. I reloaded once, and restarted once
just in case.
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
4554 postfix 20 0 20828 17m 2268 S0 4.5 0:00.46 smtpd
4560 postfix 20 0 20036 16m 2268 S0 4.3 0:00.47 smtpd
4555 postfix 20 0 6812 3056 1416 S0 0.8 0:00.10 proxymap
The current CIDR implementation is optimized to make it easy to
verify for correctness, and is optimized for speed when used with
limited lists of netblocks (mynetworks, unassigned address blocks,
reserved address blocks, etc.).
Understood.
If you want to list large portions of Internet address space such
as entire countries the current implementation starts burning CPU
time (it examines all CIDR patterns in order; with a bit of extra
up-front work during initialization, address lookups could skip
over a lot of patterns, but the implementation would of course be
harder to verify for correctness), and it wastes 24 bytes per CIDR
rule when Postfix is compiled with IPv6 support (this roughly
doubles the amount memory that is used by CIDR tables).
I don't really notice much CPU burn on any postfix processes with these largish
CIDRs, never have. I've got 12,212 CIDRs in 3 files, 11,148 of them in just the
countries file alone. After implementing proxymap, I'm not seeing much
reduction in smtpd RES size, maybe 1MB if that. SHR is almost identical to
before. If it's not the big tables bloating smtpd, I wonder what is? Or, have
I not implemented proxymap correctly? Following are my postconf -n and main.cf
relevant parts.
alias_maps = hash:/etc/aliases
append_dot_mydomain = no
biff = no
config_directory = /etc/postfix
disable_vrfy_command = yes
header_checks = pcre:/etc/postfix/header_checks
inet_interfaces = all
message_size_limit = 1024
mime_header_checks = pcre:/etc/postfix/mime_header_checks
mydestination = hardwarefreak.com
myhostname = greer.hardwarefreak.com
mynetworks = 192.168.100.0/24
myorigin = hardwarefreak.com
parent_domain_matches_subdomains = debug_peer_list smtpd_access_maps
proxy_interfaces = 65.41.216.221
proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps
$virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains
$relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps
$recipient_canonical_maps $relocated_maps $transport_maps $mynetworks
$sender_bcc_maps $recipient_bcc_maps $smtp_generic_maps $lmtp_generic_maps
proxy:${cidr}/countries proxy:${cidr}/spammer proxy:${cidr}/misc-spam-srcs
readme_directory = /usr/share/doc/postfix
recipient_bcc_maps = hash:/etc/postfix/recipient_bcc
relay_domains =
smtpd_banner = $myhostname ESMTP Postfix
smtpd_helo_required = yes
smtpd_recipient_restrictions = permit_mynetworks
reject_unauth_destination check_recipient_access
hash:/etc/postfix/whitelist check_sender_access hash:/etc/postfix/whitelist
check_client_access hash:/etc/postfix/whitelist check_client_access
hash:/etc/postfix/blacklist check_client_access
regexp:/etc/postfix/fqrdns.regexp check_client_access
pcre:/etc/postfix/ptr-tld.pcre check_client_access proxy:${cidr}/countries
check_client_access proxy:${cidr}/spammer check_client_access
proxy:${cidr}/misc-spam-srcsreject_unknown_reverse_client_hostname
reject_non_fqdn_sender reject_non_fqdn_helo_hostname
reject_invalid_helo_hostnamereject_unknown_helo_hostname
reject_unlisted_recipient reject_rbl_client zen.spamhaus.org
check_policy_service inet:127.0.0.1:6
strict_rfc821_envelopes = yes
virtual_alias_maps = hash:/etc/postfix/virtual
/etc/postfix/main.cf snippet
cidr=cidr:/etc/postfix/cidr_files
proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps
$virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains
$relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps
$recipient_canonical_maps $relocated_maps $transport_maps $mynetworks
$sender_bcc_maps $recipient_bcc_maps $smtp_generic_maps $lmtp_generic_maps
proxy:${cidr}/countries proxy:${cidr}/spammer proxy:${cidr}/misc-spam-srcs
check_client_access proxy:${cidr}/countries
check_client_access proxy:${cidr}/spammer
check_client_access proxy:${cidr}/misc-spam-srcs
--
Stan
Re: smtpd processes congregating at the pub
Sorry for top posting. Forgot to add something earlier: Proxymap seems to be
exiting on my system immediately after servicing requests. It does not seem to
be obeying $max_use or $max_idle which are both set to 100. It did this even
before I added cidr lists to proxymap a few hours ago. Before that, afaik, it
was only being called for local alias verification, and it exited immediately in
that case as well.
--
Stan
Stan Hoeppner put forth on 1/30/2010 11:13 PM:
Wietse Venema put forth on 1/30/2010 7:14 PM:
Stan Hoeppner:
AFAIK I don't use Berkeley DB tables, only hash (small,few) and cidr
(very large, a handful).
hash (and btree) == Berkeley DB.
Ahh, good to know. I'd thought only btree used Berkeley DB and that hash
tables
used something else.
If you have big CIDR tables, you can save lots of memory by using
proxy:cidr: instead of cidr: (and running postfix reload).
Effectively, this turns all that private memory into something that
can be shared via the proxy: protocol.
I implemented proxymap but it doesn't appear to have changed the memory
footprint of smtpd much at all, if any. I reloaded once, and restarted once
just in case.
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
4554 postfix 20 0 20828 17m 2268 S0 4.5 0:00.46 smtpd
4560 postfix 20 0 20036 16m 2268 S0 4.3 0:00.47 smtpd
4555 postfix 20 0 6812 3056 1416 S0 0.8 0:00.10 proxymap
The current CIDR implementation is optimized to make it easy to
verify for correctness, and is optimized for speed when used with
limited lists of netblocks (mynetworks, unassigned address blocks,
reserved address blocks, etc.).
Understood.
If you want to list large portions of Internet address space such
as entire countries the current implementation starts burning CPU
time (it examines all CIDR patterns in order; with a bit of extra
up-front work during initialization, address lookups could skip
over a lot of patterns, but the implementation would of course be
harder to verify for correctness), and it wastes 24 bytes per CIDR
rule when Postfix is compiled with IPv6 support (this roughly
doubles the amount memory that is used by CIDR tables).
I don't really notice much CPU burn on any postfix processes with these
largish
CIDRs, never have. I've got 12,212 CIDRs in 3 files, 11,148 of them in just
the
countries file alone. After implementing proxymap, I'm not seeing much
reduction in smtpd RES size, maybe 1MB if that. SHR is almost identical to
before. If it's not the big tables bloating smtpd, I wonder what is? Or,
have
I not implemented proxymap correctly? Following are my postconf -n and
main.cf
relevant parts.
alias_maps = hash:/etc/aliases
append_dot_mydomain = no
biff = no
config_directory = /etc/postfix
disable_vrfy_command = yes
header_checks = pcre:/etc/postfix/header_checks
inet_interfaces = all
message_size_limit = 1024
mime_header_checks = pcre:/etc/postfix/mime_header_checks
mydestination = hardwarefreak.com
myhostname = greer.hardwarefreak.com
mynetworks = 192.168.100.0/24
myorigin = hardwarefreak.com
parent_domain_matches_subdomains = debug_peer_list smtpd_access_maps
proxy_interfaces = 65.41.216.221
proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps
$virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains
$relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps
$recipient_canonical_maps $relocated_maps $transport_maps $mynetworks
$sender_bcc_maps $recipient_bcc_maps $smtp_generic_maps $lmtp_generic_maps
proxy:${cidr}/countries proxy:${cidr}/spammer proxy:${cidr}/misc-spam-srcs
readme_directory = /usr/share/doc/postfix
recipient_bcc_maps = hash:/etc/postfix/recipient_bcc
relay_domains =
smtpd_banner = $myhostname ESMTP Postfix
smtpd_helo_required = yes
smtpd_recipient_restrictions = permit_mynetworks
reject_unauth_destination check_recipient_access
hash:/etc/postfix/whitelist check_sender_access hash:/etc/postfix/whitelist
check_client_access hash:/etc/postfix/whitelist check_client_access
hash:/etc/postfix/blacklist check_client_access
regexp:/etc/postfix/fqrdns.regexp check_client_access
pcre:/etc/postfix/ptr-tld.pcre check_client_access proxy:${cidr}/countries
check_client_access proxy:${cidr}/spammer check_client_access
proxy:${cidr}/misc-spam-srcsreject_unknown_reverse_client_hostname
reject_non_fqdn_sender reject_non_fqdn_helo_hostname
reject_invalid_helo_hostnamereject_unknown_helo_hostname
reject_unlisted_recipient reject_rbl_client zen.spamhaus.org
check_policy_service inet:127.0.0.1:6
strict_rfc821_envelopes = yes
virtual_alias_maps = hash:/etc/postfix/virtual
/etc/postfix/main.cf snippet
cidr=cidr:/etc/postfix/cidr_files
proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps
$virtual_alias_domains $virtual_mailbox_maps
create new email address in postfix
Hi GUys I am new to postfix, running unser ubuntu.I want to create a new email address. please help me do that Thanks Dev -- View this message in context: http://old.nabble.com/create-new-email-address-in-postfix-tp27390832p27390832.html Sent from the Postfix mailing list archive at Nabble.com.
RE: create new email address in postfix
-Original Message- From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org] On Behalf Of dd1313 Sent: Sunday, January 31, 2010 1:53 AM To: postfix-users@postfix.org Subject: create new email address in postfix Hi GUys I am new to postfix, running unser ubuntu.I want to create a new email address. please help me do that Thanks Dev -- View this message in context: http://old.nabble.com/create-new-email-address-in-postfix-tp27390832p2739083 2.html Sent from the Postfix mailing list archive at Nabble.com. - I would use postfixadmin and mysql maps if you're not use to using postfix.
