Re: proxymap(8), number of connections, detecting altered tables
* Stefan Foerster cite+postfix-us...@incertum.net: While I agree that it is totally obvious that table are re-read as soon as a new proxymap(8) process is spawned, on a resonably busy system, this won't happen too often. So getting a definitive answer on that one would still be helpful. Has this been answered? It also affects me, so I'd like to know :) -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: proxymap(8), number of connections, detecting altered tables
Ralf Hildebrandt: * Stefan Foerster cite+postfix-us...@incertum.net: While I agree that it is totally obvious that table are re-read as soon as a new proxymap(8) process is spawned, on a resonably busy system, this won't happen too often. So getting a definitive answer on that one would still be helpful. Has this been answered? It also affects me, so I'd like to know :) There is no definitive answer. The strategy of how to detect changes is evolving over time, and some tables (pcre, regexp, cidr) do not implement change detection at this time. If you really must force a change, use postfix reload. Wietse
Re: proxymap(8), number of connections, detecting altered tables
On Thu, Jul 15, 2010 at 02:31:36PM -0400, Wietse Venema wrote: Also, only use proxymap for IPC based tables (ldap, mysql, pgsql, tcp, ...), do not use proxymap for indexed files, cidr tables, pcre/regexp tables, It depends on what the trade-offs are. I know of one user with very large cidr tables - sacrificing performance to avoid running out of physical memory. One might suggest that CIDR is not a good fit for this even if stored just once, an IPC based server that walks trees rather than lists would be far more suitable... -- Viktor.
Re: proxymap(8), number of connections, detecting altered tables
Victor Duchovni: On Thu, Jul 15, 2010 at 02:31:36PM -0400, Wietse Venema wrote: Also, only use proxymap for IPC based tables (ldap, mysql, pgsql, tcp, ...), do not use proxymap for indexed files, cidr tables, pcre/regexp tables, It depends on what the trade-offs are. I know of one user with very large cidr tables - sacrificing performance to avoid running out of physical memory. One might suggest that CIDR is not a good fit for this even if stored just once, an IPC based server that walks trees rather than lists would be far more suitable... I agree that the Postfix CIDR implementation achieves simplicity of implementation (including correctness) by sacrificing space and speed. It seems that speed-wise improvements could be made cheaply by adding IF/ENDIF support, similar to the regexp/pcre tables. Wietse
Re: proxymap(8), number of connections, detecting altered tables
On Thu, Jul 15, 2010 at 03:37:02PM -0400, Wietse Venema wrote: One might suggest that CIDR is not a good fit for this even if stored just once, an IPC based server that walks trees rather than lists would be far more suitable... I agree that the Postfix CIDR implementation achieves simplicity of implementation (including correctness) by sacrificing space and speed. It seems that speed-wise improvements could be made cheaply by adding IF/ENDIF support, similar to the regexp/pcre tables. Yes, this did occur to me: IF 192.0.0.0/4 IF 192.0.0.0/12 IF 192.0.0.0/20 192.0.2.1 REJECT example address ENDIF optional text ENDIF 192.0.0.0/12 ENDIF 192.0.0.0/4 plus a suitable compiler script that constructs a sensibly efficient nested IF structure from a list of CIDR blocks. -- Viktor.
Re: proxymap(8), number of connections, detecting altered tables
Stefan Foerster put forth on 6/20/2010 5:16 AM:
Two questions regarding proxymap:
1. Is a single proxymap(8) process able to handle multiple lookup
tables? I.e., taking the example from the manpage, modifying it to
For read maps, yes. And even better, one process will do multiple map types.
I use read-only hash, regex, and pcre maps with a single proxymap process.
mysql = proxy:mysql:/etc/postfix/
virtual_alias_maps =${mysql}virtual_alias_maps.cf
virtual_alias_domains = ${mysql}virtual_alias_domains.cf
For write maps I don't know.
and furthermore assuming a limit of 40 proxymap(8) processes defined
in master.cf, will this result in 40 or 80 connections to the
database?
I have no idea on this one. The whole point of proxymap is to have one
process reading all maps to cut down on the memory footprint of multiple
concurrent smtpd processes. Why anyone would want 40 proxymap processes is
beyond me. Doing this seems counterproductive.
2. Typically, changes in CIDR, PCRE or RegExp tables are not detected
immediately, which is not a problem for short lived processes like
smtpd(8). If those tables are provided by means of proxymap(8) (to
e.g. save memory on very large CIDR tables), will the tables be
re-read as soon as a newly spawned smtpd(8) makes a connection to
proxymap(8) or is it necessary to reload Postfix?
I believe this is controlled by max_idle and ipc_idle. When the timer
expires, the proxymap process is killed. The next process that needs one of
the tables causes the proxymap process launch. At this point the tables are
read. If you need an update to be read before the process expires, you would
have to do a reload.
--
Stan
Re: proxymap(8), number of connections, detecting altered tables
* Stan Hoeppner s...@hardwarefreak.com: Stefan Foerster put forth on 6/20/2010 5:16 AM: and furthermore assuming a limit of 40 proxymap(8) processes defined in master.cf, will this result in 40 or 80 connections to the database? I have no idea on this one. The whole point of proxymap is to have one process reading all maps to cut down on the memory footprint of multiple concurrent smtpd processes. Why anyone would want 40 proxymap processes is beyond me. Doing this seems counterproductive. The default process limit for proxymap(8) if you didn't change master.cf is $default_process_limit which seems to default to 100. So my question is, indeed, very valid, I guess :-) 2. Typically, changes in CIDR, PCRE or RegExp tables are not detected immediately, which is not a problem for short lived processes like smtpd(8). If those tables are provided by means of proxymap(8) (to e.g. save memory on very large CIDR tables), will the tables be re-read as soon as a newly spawned smtpd(8) makes a connection to proxymap(8) or is it necessary to reload Postfix? I believe this is controlled by max_idle and ipc_idle. When the timer expires, the proxymap process is killed. The next process that needs one of the tables causes the proxymap process launch. At this point the tables are read. If you need an update to be read before the process expires, you would have to do a reload. While I agree that it is totally obvious that table are re-read as soon as a new proxymap(8) process is spawned, on a resonably busy system, this won't happen too often. So getting a definitive answer on that one would still be helpful. Stefan
Re: proxymap(8), number of connections, detecting altered tables
Stefan Foerster:
Two questions regarding proxymap:
1. Is a single proxymap(8) process able to handle multiple lookup
tables? I.e., taking the example from the manpage, modifying it to
mysql = proxy:mysql:/etc/postfix/
virtual_alias_maps =${mysql}virtual_alias_maps.cf
virtual_alias_domains = ${mysql}virtual_alias_domains.cf
and furthermore assuming a limit of 40 proxymap(8) processes defined
in master.cf, will this result in 40 or 80 connections to the
database?
This depends on the query load. When a client needs proxymap service
it is either helped by an existing proxymap process that is idle,
or else it is helped by a new proxymap process. In the first case
it will share the table with other proxymap clients.
2. Typically, changes in CIDR, PCRE or RegExp tables are not detected
immediately, which is not a problem for short lived processes like
smtpd(8). If those tables are provided by means of proxymap(8) (to
e.g. save memory on very large CIDR tables), will the tables be
re-read as soon as a newly spawned smtpd(8) makes a connection to
proxymap(8) or is it necessary to reload Postfix?
CIDR, PCRE or RegExp tables have no change detection mechanism.
That requires keeping the file handle open, which seems wasteful.
Most proxymap daemons will retire after a thousand seconds or so
but it may take longer on busy servers. This may be sped up by
reducing main.cf:ipc_ttl to 100s or so.
Wietse
Re: proxymap(8), number of connections, detecting altered tables
* Wietse Venema wie...@porcupine.org:
Stefan Foerster:
Two questions regarding proxymap:
1. Is a single proxymap(8) process able to handle multiple lookup
tables? I.e., taking the example from the manpage, modifying it to
mysql = proxy:mysql:/etc/postfix/
virtual_alias_maps =${mysql}virtual_alias_maps.cf
virtual_alias_domains = ${mysql}virtual_alias_domains.cf
and furthermore assuming a limit of 40 proxymap(8) processes defined
in master.cf, will this result in 40 or 80 connections to the
database?
This depends on the query load. When a client needs proxymap service
it is either helped by an existing proxymap process that is idle,
or else it is helped by a new proxymap process. In the first case
it will share the table with other proxymap clients.
As a follow-up: If a clients needs to be served by a new proxymap
process but none are available, what happens?
Stefan
Re: proxymap(8), number of connections, detecting altered tables
Stefan Foerster: This depends on the query load. When a client needs proxymap service it is either helped by an existing proxymap process that is idle, or else it is helped by a new proxymap process. In the first case it will share the table with other proxymap clients. As a follow-up: If a clients needs to be served by a new proxymap process but none are available, what happens? The proxymap client waits. Wietse
