> Date: Fri, 30 Apr 2010 01:04:41 -0400
> From: [email protected]
> To: [email protected]
> Subject: Re: smtpd ldap query causes session to hang with postfix 2.6.2
>
> On Thu, Apr 29, 2010 at 01:53:37PM +0000, Charles Account wrote:
>
>> We have a situation where LDAP query is resulting in a LDAP 80 level
>> errorduring a domain lookup. Yes I understand we need to fix this problem.
>
> I've mentioned a number of times on this list that it is unwise to
> make the trivial-rewrite service dependent on LDAP or other services
> that may not always be reliable.
>
> Postfix cannot continue to function when
>
> address -> (transport, nexthop, normalized-address)
>
> lookups fail, there is no sensible recovery path. The lookups must
> loop until the transport table works again.
>
> DO NOT use potentially unreliable LDAP sources for the following tables:
>
> - transport_maps
> - mydestination
> - relay_domains
> - virtual_mailbox_domains
> - virtual_alias_domains
> - relocated_maps
> - sender_dependent_relayhost_maps
> - sender_dependent_default_transport_maps
>
>> However, the side effect we see is the client's SMTP session hangs.
>> Over a period of time all SMTPD sessions are consumed and no mail is
>> processed.The only solution was to stop postfix and restart to free up
>> the processes.
>
> No mail can be processed anyway, as the queue manager is completely
> unable to function when transport resolution is down.
>
>> I have looked for a patch but I have not found one. Does one exist?
>
> No patch exists, as this is not a bug. Transport lookups MUST work.
> If LDAP is not reliable, don't use LDAP.
>
> --
> Viktor.
Viktor,
Thanks for the response. Our LDAP service front ends a distributed cache based
directory service. This means that there are failure scenarios whereby some
small number of directory entries are unavailable while the majority of the
directory is still available.
I noticed in my testing, where an LDAP front end server is brought down on
purpose, the connection
service returns a RESOLVE_FLAG_FAIL and the client MTA returns a 451 error code
(Temporary lookup failure).
We cannot always guarantee a specific directory entry will be available,
therefore, we coded our LDAP interface to
return the appropriate LDAP level error code. Having resolve_clnt_addr() retry
several time before giving up
seems to be an appropriate behavior, thereby allowing the entire complex to
continue to handle mail.
I've included a patch based on 2.6.2 release as a suggested solution. Feed
back is always welcome.
Charles
_________________________________________________________________
Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_1
diff -crB postfix-2.6.2.patched/conf/main.cf.default
postfix-2.6.2.mod/conf/main.cf.default
*** postfix-2.6.2.patched/conf/main.cf.default 2009-09-11 11:20:50.000000000
-0400
--- postfix-2.6.2.mod/conf/main.cf.default 2010-05-03 11:51:35.000000000
-0400
***************
*** 149,154 ****
--- 149,155 ----
ipc_idle = 5s
ipc_timeout = 3600s
ipc_ttl = 1000s
+ remote_query_retry_limit = 0
line_length_limit = 2048
lmtp_assume_final = no
lmtp_bind_address =
diff -crB postfix-2.6.2.patched/src/global/mail_params.c
postfix-2.6.2.mod/src/global/mail_params.c
*** postfix-2.6.2.patched/src/global/mail_params.c 2009-01-22
16:16:49.000000000 -0500
--- postfix-2.6.2.mod/src/global/mail_params.c 2010-05-03 11:52:30.000000000
-0400
***************
*** 36,41 ****
--- 36,42 ----
/* char *var_procname;
/* int var_pid;
/* int var_ipc_timeout;
+ /* int var_remote_query_retry_limit;
/* char *var_pid_dir;
/* int var_dont_remove;
/* char *var_inet_interfaces;
***************
*** 81,86 ****
--- 82,88 ----
/* int var_debug_peer_level;
/* int var_in_flow_delay;
/* int var_fault_inj_code;
+ /* int var_remote_query_retry_limit;
/* char *var_bounce_service;
/* char *var_cleanup_service;
/* char *var_defer_service;
***************
*** 221,226 ****
--- 223,229 ----
char *var_procname;
int var_pid;
int var_ipc_timeout;
+ int var_remote_query_retry_limit;
char *var_pid_dir;
int var_dont_remove;
char *var_inet_interfaces;
***************
*** 591,596 ****
--- 594,600 ----
VAR_MIME_BOUND_LEN, DEF_MIME_BOUND_LEN, &var_mime_bound_len, 1, 0,
VAR_DELAY_MAX_RES, DEF_DELAY_MAX_RES, &var_delay_max_res,
MIN_DELAY_MAX_RES, MAX_DELAY_MAX_RES,
VAR_INET_WINDOW, DEF_INET_WINDOW, &var_inet_windowsize, 0, 0,
+ VAR_REMOTE_QUERY_RETRY_LIMIT, DEF_REMOTE_QUERY_RETRY_LIMIT,
&var_remote_query_retry_limit, 0, 100,
0,
};
static const CONFIG_TIME_TABLE time_defaults[] = {
diff -crB postfix-2.6.2.patched/src/global/mail_params.h
postfix-2.6.2.mod/src/global/mail_params.h
*** postfix-2.6.2.patched/src/global/mail_params.h 2009-06-17
21:18:49.000000000 -0400
--- postfix-2.6.2.mod/src/global/mail_params.h 2010-05-03 11:52:30.000000000
-0400
***************
*** 1841,1846 ****
--- 1841,1855 ----
extern int var_ipc_timeout;
/*
+ * Enable temp fail condition on remote query service after N tries
+ * The default is to disable this feature. The value is the number
+ * of retries we should attempt before giving up.
+ */
+ #define VAR_REMOTE_QUERY_RETRY_LIMIT "remote_query_retry_limit"
+ #define DEF_REMOTE_QUERY_RETRY_LIMIT 0
+ extern int var_remote_query_retry_limit;
+
+ /*
* Time limit on intra-mail triggers.
*/
#define VAR_TRIGGER_TIMEOUT "trigger_timeout"
diff -crB postfix-2.6.2.patched/src/global/resolve_clnt.c
postfix-2.6.2.mod/src/global/resolve_clnt.c
*** postfix-2.6.2.patched/src/global/resolve_clnt.c 2007-05-11
12:19:21.000000000 -0400
--- postfix-2.6.2.mod/src/global/resolve_clnt.c 2010-05-03 11:52:30.000000000
-0400
***************
*** 270,275 ****
--- 270,290 ----
else
break;
}
+ /*
+ * A large LDAP complex may return a temporary error for some queries,
+ * while others may succeed. This prevents an infinite loop and
+ * potentially causing the entire complex to hang waiting for a valid
+ * response. Even if the client MTA has closed the socket, the
process
+ * continues to query, using both system and network bandwith in
+ * addition to no longer accepting new mail.
+ */
+ if (var_remote_query_retry_limit && (count >
var_remote_query_retry_limit))
+ {
+ msg_error("%s: Temporary remote query error result for: <%s>",
+ myname, addr);
+ reply->flags |= RESOLVE_FLAG_FAIL;
+ break;
+ }
sleep(1); /* XXX make configurable */
clnt_stream_recover(rewrite_clnt_stream);
}
diff -crB postfix-2.6.2.patched/src/postconf/int_table.h
postfix-2.6.2.mod/src/postconf/int_table.h
*** postfix-2.6.2.patched/src/postconf/int_table.h 2009-09-11
09:23:33.000000000 -0400
--- postfix-2.6.2.mod/src/postconf/int_table.h 2010-05-03 11:53:24.000000000
-0400
***************
*** 50,55 ****
--- 50,56 ----
VAR_RELAY_RCPT_CODE, DEF_RELAY_RCPT_CODE, &var_relay_rcpt_code, 0, 0,
VAR_CONC_COHORT_LIM, DEF_CONC_COHORT_LIM, &var_conc_cohort_limit, 0, 0,
VAR_DEST_CON_LIMIT, DEF_DEST_CON_LIMIT, &var_dest_con_limit, 0, 0,
+ VAR_REMOTE_QUERY_RETRY_LIMIT, DEF_REMOTE_QUERY_RETRY_LIMIT,
&var_remote_query_retry_limit, 0, 100,
VAR_HOPCOUNT_LIMIT, DEF_HOPCOUNT_LIMIT, &var_hopcount_limit, 1, 0,
VAR_LMTP_TLS_SCERT_VD, DEF_LMTP_TLS_SCERT_VD, &var_smtp_tls_scert_vd, 0, 0,
VAR_LMTP_MXSESS_LIMIT, DEF_LMTP_MXSESS_LIMIT, &var_smtp_mxsess_limit, 0, 0,
diff -crB postfix-2.6.2.patched/src/smtpd/smtpd.c
postfix-2.6.2.mod/src/smtpd/smtpd.c
*** postfix-2.6.2.patched/src/smtpd/smtpd.c 2009-09-11 13:09:59.000000000
-0400
--- postfix-2.6.2.mod/src/smtpd/smtpd.c 2010-05-03 11:53:02.000000000 -0400
***************
*** 1883,1889 ****
return (0);
}
! /* extract_addr - extract address from rubble */
static int extract_addr(SMTPD_STATE *state, SMTPD_TOKEN *arg,
int allow_empty_addr, int strict_rfc821)
--- 1883,1894 ----
return (0);
}
! /* extract_addr - extract address from rubble
! * Return:
! * 0 - Successful query
! * 1 - Error
! * 2 - Temporary error with remote service query
! */
static int extract_addr(SMTPD_STATE *state, SMTPD_TOKEN *arg,
int allow_empty_addr, int strict_rfc821)
***************
*** 1895,1900 ****
--- 1900,1906 ----
int naddr;
int non_addr;
int err = 0;
+ int retcode = 0;
char *junk = 0;
char *text;
char *colon;
***************
*** 1981,1996 ****
* sleep+reject so that attackers can't flood our logfiles. Log the
* original address.
*/
! if (err == 0)
if ((STR(state->addr_buf)[0] == 0 && !allow_empty_addr)
! || (strict_rfc821 && STR(state->addr_buf)[0] == '@')
! || (SMTPD_STAND_ALONE(state) == 0
! && smtpd_check_addr(STR(state->addr_buf)) != 0)) {
msg_warn("Illegal address syntax from %s in %s command: %s",
state->namaddr, state->where,
printable(STR(arg->vstrval), '?'));
! err = 1;
! }
/*
* Cleanup.
--- 1987,2014 ----
* sleep+reject so that attackers can't flood our logfiles. Log the
* original address.
*/
! if (err == 0) {
if ((STR(state->addr_buf)[0] == 0 && !allow_empty_addr)
! || (strict_rfc821 && STR(state->addr_buf)[0] == '@'))
! err = 1;
! else if (SMTPD_STAND_ALONE(state) == 0) {
! retcode = smtpd_check_addr(STR(state->addr_buf));
!
! /* Handle Temporary error from remote query service */
! if (retcode == 2) {
! err = retcode;
! msg_error("Unable to retrieve query result %s in %s command:
%s",
! state->namaddr, state->where,
! printable(STR(arg->vstrval), '?'));
! }
! else if (retcode)
! err = 1;
! }
! if (err == 1)
msg_warn("Illegal address syntax from %s in %s command: %s",
state->namaddr, state->where,
printable(STR(arg->vstrval), '?'));
! }
/*
* Cleanup.
***************
*** 2035,2040 ****
--- 2053,2059 ----
char *verp_delims = 0;
int rate;
int dsn_envid = 0;
+ int retcode = 0;
state->encoding = 0;
state->dsn_ret = 0;
***************
*** 2091,2099 ****
smtpd_chat_reply(state, "501 5.1.7 Bad sender address syntax");
return (-1);
}
! if (extract_addr(state, argv + 2, PERMIT_EMPTY_ADDR,
var_strict_rfc821_env) != 0) {
! state->error_mask |= MAIL_ERROR_PROTOCOL;
! smtpd_chat_reply(state, "501 5.1.7 Bad sender address syntax");
return (-1);
}
for (narg = 3; narg < argc; narg++) {
--- 2110,2125 ----
smtpd_chat_reply(state, "501 5.1.7 Bad sender address syntax");
return (-1);
}
! if ((retcode = extract_addr(state, argv + 2, PERMIT_EMPTY_ADDR,
var_strict_rfc821_env)) != 0) {
! if (retcode == 2) {
! state->error_mask |= MAIL_ERROR_RESOURCE;
! smtpd_chat_reply(state, "451 4.3.0 <%s>: Temporary lookup failure",
! STR(state->addr_buf));
! return (-1);
! } else {
! state->error_mask |= MAIL_ERROR_PROTOCOL;
! smtpd_chat_reply(state, "501 5.1.7 Bad sender address syntax");
! }
return (-1);
}
for (narg = 3; narg < argc; narg++) {
***************
*** 2341,2346 ****
--- 2367,2373 ----
ssize_t dsn_orcpt_addr_len = 0;
const char *dsn_orcpt_type = 0;
int dsn_notify = 0;
+ int retcode = 0;
const char *coded_addr;
const char *milter_err;
***************
*** 2390,2399 ****
smtpd_chat_reply(state, "501 5.1.3 Bad recipient address syntax");
return (-1);
}
! if (extract_addr(state, argv + 2, REJECT_EMPTY_ADDR,
var_strict_rfc821_env) != 0) {
! state->error_mask |= MAIL_ERROR_PROTOCOL;
! smtpd_chat_reply(state, "501 5.1.3 Bad recipient address syntax");
! return (-1);
}
for (narg = 3; narg < argc; narg++) {
arg = argv[narg].strval;
--- 2417,2434 ----
smtpd_chat_reply(state, "501 5.1.3 Bad recipient address syntax");
return (-1);
}
! if ((retcode = extract_addr(state, argv + 2, REJECT_EMPTY_ADDR,
! var_strict_rfc821_env)) != 0) {
! if (retcode == 2) {
! state->error_mask |= MAIL_ERROR_RESOURCE;
! smtpd_chat_reply(state, "451 4.3.0 <%s>: Temporary lookup failure",
! STR(state->addr_buf));
! return (-1);
! } else {
! state->error_mask |= MAIL_ERROR_PROTOCOL;
! smtpd_chat_reply(state, "501 5.1.3 Bad recipient address syntax");
! return (-1);
! }
}
for (narg = 3; narg < argc; narg++) {
arg = argv[narg].strval;
***************
*** 3157,3162 ****
--- 3192,3198 ----
static int vrfy_cmd(SMTPD_STATE *state, int argc, SMTPD_TOKEN *argv)
{
const char *err = 0;
+ int retcode = 0;
/*
* The SMTP standard (RFC 821) disallows unquoted special characters in
***************
*** 3200,3209 ****
}
if (argc > 2)
collapse_args(argc - 1, argv + 1);
! if (extract_addr(state, argv + 1, REJECT_EMPTY_ADDR, SLOPPY) != 0) {
! state->error_mask |= MAIL_ERROR_PROTOCOL;
! smtpd_chat_reply(state, "501 5.1.3 Bad recipient address syntax");
! return (-1);
}
/* Not: state->addr_buf */
if (SMTPD_STAND_ALONE(state) == 0
--- 3236,3252 ----
}
if (argc > 2)
collapse_args(argc - 1, argv + 1);
! if ((retcode = extract_addr(state, argv + 1, REJECT_EMPTY_ADDR, SLOPPY))
!= 0) {
! if (retcode == 2) {
! state->error_mask |= MAIL_ERROR_RESOURCE;
! smtpd_chat_reply(state, "451 4.3.0 <%s>: Temporary lookup failure",
! STR(state->addr_buf));
! return (-1);
! } else {
! state->error_mask |= MAIL_ERROR_PROTOCOL;
! smtpd_chat_reply(state, "501 5.1.3 Bad recipient address syntax");
! return (-1);
! }
}
/* Not: state->addr_buf */
if (SMTPD_STAND_ALONE(state) == 0
diff -crB postfix-2.6.2.patched/src/smtpd/smtpd_check.c
postfix-2.6.2.mod/src/smtpd/smtpd_check.c
*** postfix-2.6.2.patched/src/smtpd/smtpd_check.c 2009-05-28
13:19:04.000000000 -0400
--- postfix-2.6.2.mod/src/smtpd/smtpd_check.c 2010-05-03 11:53:02.000000000
-0400
***************
*** 3948,3954 ****
return (status);
}
! /* smtpd_check_addr - address sanity check */
int smtpd_check_addr(const char *addr)
{
--- 3948,3960 ----
return (status);
}
! /* smtpd_check_addr - address sanity check
! * Return:
! * 0 - Successful query
! * -1 - Error
! * 2 - Temporary error with remote service query
! */
!
int smtpd_check_addr(const char *addr)
{
***************
*** 3966,3971 ****
--- 3972,3984 ----
if (addr == 0 || *addr == 0)
return (0);
resolve_reply = smtpd_resolve_addr(addr);
+ /*
+ * We didn't get a response after N tries
+ * (remote_query_retry_limit), return a temporary error.
+ * This will prevent the entire complex from hanging.
+ */
+ if (resolve_reply->flags & RESOLVE_FLAG_FAIL)
+ return (2);
if (resolve_reply->flags & RESOLVE_FLAG_ERROR)
return (-1);
return (0);
