Re: whitelisting incoming /outgoing mail
Eric Williams:
On Tue, Jan 5, 2010 at 9:12 AM, Stan Hoeppner s...@hardwarefreak.comwrote:
Eric Williams put forth on 1/5/2010 8:02 AM:
I would like to apply the same access list so that users sending mail
through this server can only reach those same domains.
I've tried lots of recipient checking configs but nothing works so far.
I'd rather not do this with the firewall, keeping the whitelist monitored by
postfix only.
So you want a dedicated smtp relay server that will only transfer mail
between a
handful of domains?
You could use a tool such as Fail2Ban to watch the maillog file
and update a Postfix access table.
The steps would be
1) See if the domain is already in the Postfix access table.
2) Add the domain.
3) Rebuild the table.
Example add-domain script:
#!/bin/sh
# usage: add-domain name
case $# in
1) postmap -q $1 the-postfix-access-table /dev/null || {
echo $1 OK the-postfix-access-table
postmap the-postfix-access-table
};;
*) echo Usage: $0 domainame 12; exit 1;;
esac
If you handle lots of mail you will want to read and update the
database files without running postmap commands for each email
logfile record.
Wietse
Re: whitelisting incoming /outgoing mail
Eric Williams:
On Jan 5, 2010, at 9:57 AM, Wietse Venema wrote:
Eric Williams:
On Tue, Jan 5, 2010 at 9:12 AM, Stan Hoeppner
s...@hardwarefreak.comwrote:
Eric Williams put forth on 1/5/2010 8:02 AM:
I would like to apply the same access list so that users sending mail
through this server can only reach those same domains.
I've tried lots of recipient checking configs but nothing works so far.
I'd rather not do this with the firewall, keeping the whitelist monitored
by
postfix only.
So you want a dedicated smtp relay server that will only transfer mail
between a
handful of domains?
You could use a tool such as Fail2Ban to watch the maillog file
and update a Postfix access table.
The steps would be
1) See if the domain is already in the Postfix access table.
2) Add the domain.
3) Rebuild the table.
Example add-domain script:
#!/bin/sh
# usage: add-domain name
case $# in
1) postmap -q $1 the-postfix-access-table /dev/null || {
echo $1 OK the-postfix-access-table
postmap the-postfix-access-table
};;
*) echo Usage: $0 domainame 12; exit 1;;
esac
If you handle lots of mail you will want to read and update the
database files without running postmap commands for each email
logfile record.
Wietse
This is great info. I'll look into applying that is some form.
I think what I'm still missing is the proper restriction in the
smptd_recipient_restrictions section to restrict the outgoing mail.
check_sender_access hash:/etc/postfix/access
works for incoming blocking. I haven't found the right config for the
blocking.
It if is implied in your response I apologize for my in-experience with this.
smtpd_sender_restrictions =
permit_mynetworks
permit_sasl_authenticated
check_sender_access hash:/etc/postfix/sender_access
reject
Wietse
Re: header_checks problem
/dev/rob0: On Mon, Jan 04, 2010 at 09:00:25PM +0100, Ralf Hildebrandt wrote: * Christopher Adams adam...@gmail.com: Summary: I would like to ban an address/domain from posting to my system. I am using header_checks to do that. Why? Wouldn't check_sender_access be more appropriate? In this case, no, because the mail was submitted via sendmail(1) by UID 552. Perhaps authorized_submit_users is the best solution to the real-world problem here. Note that the OP did not describe that, so we're limited to guessing. /^From: testm...@library.state.or.us / REJECT A couple more comments to augment what Ralf said: first, . is a PCRE/regexp metacharacter, as @ is in PCRE. Those should be escaped. Second, I am not sure what happens with REJECT for sendmail submission; it seems to be mostly applicable in smtpd(8) context. The Postfix cleanup daemon creates a bounce message, by using the same built-in mechanism that Postfix delivery agents use. This adds some complexity to the Postfix cleanup daemon. For example Postfix can't simply bounce the unfiltered message to the purported sender, as that would create an opportunity to send unfiltered email. Wietse
Re: XCLIENT patch for postfix
jeff geng: Wietse: Please see my reply marked as blue. :) jeff geng 2010/1/5 Wietse Venema wie...@porcupine.org jeff geng: Wietse: Happy new year :) We use niginx's smtp function to redirect mail to postfix server. But in postfix, XCLIENT command can't support the LOGIN paremeter. Severial months ago, I write a patch for postfix-2.5.3. Now nginx official website also supply a patch for this situation as followed: http://www.citrin.ru/nginx:xclient-login-patch. In additional, my patch also support xforward function. We will be very grateful to you if you can accept these patch (nginx official patch or our patch). Of course, you'd better add this function to your new version. After that We can use new version directly, other than patch the original source manually. Can you explain the following: + int got_login =0; ... + got_login = 1; Why is the got_login variable introduced? It is a write-only variable. We introduce got_login is for the same reason as got_helo and got_proto.It means postfix has got login information from client, and postfix should save the username and password from XCLIENT. Your patch is no code that reads the got_login variable. This this is a dead assignment. #ifdef USE_SASL_AUTH if (var_smtpd_sasl_enable) ! if (got_proto == 0) ! smtpd_sasl_auth_reset(state); #endif Why test for the PROTO attribute here? What happens when the client sends LOGIN=username before PROTO=SMTP? As documented XCLIENT does not require that attributes are sent in a specific order. This is my mistake. It should be if (got_login == 0 ar_smtpd_sasl_enable(state)) . It means if postfix can't get login This explains why you had a variable that was written to but that was never read. + UPDATE_STR(state-sasl_username, attr_value); + printable(state-sasl_username, '?'); + UPDATE_STR(state-sasl_method, xclient); Why not use the real authentication mechanism? Otherwise, if XCLIENT pass LOGIN parameter, state-sasl_username and state-sasl_method will be updated, postfix will deem it as an authenticated client. So , if nginx pass through LOGIN parameter, postfix should identify it as a authenticated client, but postix XCLIENT can't support this parameter. XCLIENT can support both the login name and the authentication method name, and therefore nginx should pass both to Postfix. Wietse PS I don't use a web browser to read mail, so there is no need to color your responses.
Re: XCLIENT patch for postfix
Wietse Venema: + UPDATE_STR(state-sasl_username, attr_value); + printable(state-sasl_username, '?'); + UPDATE_STR(state-sasl_method, xclient); Why not use the real authentication mechanism? Otherwise, if XCLIENT pass LOGIN parameter, state-sasl_username and state-sasl_method will be updated, postfix will deem it as an authenticated client. So , if nginx pass through LOGIN parameter, postfix should identify it as a authenticated client, but postix XCLIENT can't support this parameter. XCLIENT can support both the login name and the authentication method name, and therefore nginx should pass both to Postfix. I noticed that the nginx reverse proxy implements TLS, so it makes sense to plan for future XCLIENT extensions that propagate TLS attributes, besides the extension for SASL that you introduced with this thread. This means using something like SASL_USER and SASL_METH for the proposed SASL attributes, and TLS_XXX for future TLS attributes, so that there will be no conflicts between the names. I keep whining about the SASL authentication method, because that information is used by the Postfix permit_sasl_authenticated access control feature. It would therefore be wrong to set this to a fixed value like your patch does. Now that I understand how your patch is supposed to work, I can put something into Postfix, but it would help if we can agree on the attribute names and on the protocol details. I am sure that there are a few gotchas when you poke Postfix SASL attributes without proper initialization and cleanup of the Postfix SASL layer, but that can be fixed by adding a few functions to that SASL layer that handle support for proxied attributes. Wietse
Re: postdrop?
Port Able: Hi, What is the proper command that a client can use to send an email using Postfix?? Searching through some old posts, I believe the postdrop command is not intended to be used by client software.? Is that correct?? You use the Postfix sendmail command. Wietse
Re: Messages stuck in queue forever
Wendigo Thompson: Hello: I maintain a database of e-mail messages for my law firm client -- I wrote an application that inserts e-mail messages into a database and then used an alias to pipe e-mail to that application, and the clients e-mail server forwards all mail for certain litigants to that e-mail address. Nominally this works great, but I've noticed an odd issue: some messages get queued and are never released. I'd like to force postfix to redeliver these messages (I've tried postsuper and postsuper -r ALL with no luck) -- does anybody have any pointers? It's 2.4.3 on Mac OS X Server 10.5. Beware, I don't have a Mac, and Apple has made some changes to Postfix so I may have to refer you to the vendor if this does not work out on the mailing list. Sample mailq entries (one has been stuck since August, the other since Monday: F423E1976D72 444129 Fri Aug 1 15:23:30 MAILER-DAEMON This message has no recipients, and should have been removed from the queue long ago. Please report results of the following shell command (as root): find /var/spool/postfix -name F423E1976D72 -ls (instead of /var/spool/postfix, specify the name where Apple stores the Postfix queue. That is the output of the command: postconf queue_directory ) 001F224CE9559*3080 Wed Jan 6 14:51:58 MAILER-DAEMON msgrece...@10.188.237.22 This message is in the active queue. Do you have any qmgr processes running? If not, then nothing will happen with this mail. Apple has added an automatic shutdown feature to Postfix that may get in the way of progress. Do you have any mail logfile information for this message? If there is none, then that is also a problem that needs to be solved before we can find out why these messages are not moving. Wietse
Re: XCLIENT patch for postfix
jeff geng: Wietse: If We modify #ifdef USE_SASL_AUTH if (var_smtpd_sasl_enable) if (got_proto == 0) smtpd_sasl_auth_reset(state); #endif to #ifdef USE_SASL_AUTH if (var_smtpd_sasl_enable) if (got_login == 0) smtpd_sasl_auth_reset(state); #endif the got_login will be used.But this is not the key issues. If nginx will support SASL_USER and SASL_METH parameters, it would be best. If you can put something into Postfix for current nginx We will be very grateful to you. Thank you. I'll give it a try. I looked at nginx and I think it does a good job. Wietse jeff geng 2010/1/6 Wietse Venema wie...@porcupine.org Wietse Venema: + UPDATE_STR(state-sasl_username, attr_value); + printable(state-sasl_username, '?'); + UPDATE_STR(state-sasl_method, xclient); Why not use the real authentication mechanism? Otherwise, if XCLIENT pass LOGIN parameter, state-sasl_username and state-sasl_method will be updated, postfix will deem it as an authenticated client. So , if nginx pass through LOGIN parameter, postfix should identify it as a authenticated client, but postix XCLIENT can't support this parameter. XCLIENT can support both the login name and the authentication method name, and therefore nginx should pass both to Postfix. I noticed that the nginx reverse proxy implements TLS, so it makes sense to plan for future XCLIENT extensions that propagate TLS attributes, besides the extension for SASL that you introduced with this thread. This means using something like SASL_USER and SASL_METH for the proposed SASL attributes, and TLS_XXX for future TLS attributes, so that there will be no conflicts between the names. I keep whining about the SASL authentication method, because that information is used by the Postfix permit_sasl_authenticated access control feature. It would therefore be wrong to set this to a fixed value like your patch does. Now that I understand how your patch is supposed to work, I can put something into Postfix, but it would help if we can agree on the attribute names and on the protocol details. I am sure that there are a few gotchas when you poke Postfix SASL attributes without proper initialization and cleanup of the Postfix SASL layer, but that can be fixed by adding a few functions to that SASL layer that handle support for proxied attributes. Wietse
Re: return smtp result based on lookup
Michael: I want to deploy a system whereby it will do an SQL lookup and depending on the result will either continue processing the message as per usual or issue a 'temporary' reject (Ie: Over quota message). This is on a relaying/gateway machine so it does not have direct access to the mail queue to do it's own calculation so would need to work it out based on information available from an SQL table. Pointers: A) Use an SQL-based access table that is queried at SMTP time: http://www.postfix.org/access.5.html http://www.postfix.org/mysql_table.5.html B) Use a policy server that queries the database and does the calculation: http://www.postfix.org/SMTPD_POLICY_README.html Wietse
Re: return smtp result based on lookup
Michael: On Fri, 08 Jan 2010 00:50:49 Wietse Venema wrote: Michael: I want to deploy a system whereby it will do an SQL lookup and depending on the result will either continue processing the message as per usual or issue a 'temporary' reject (Ie: Over quota message). This is on a relaying/gateway machine so it does not have direct access to the mail queue to do it's own calculation so would need to work it out based on information available from an SQL table. Pointers: A) Use an SQL-based access table that is queried at SMTP time: http://www.postfix.org/access.5.html http://www.postfix.org/mysql_table.5.html B) Use a policy server that queries the database and does the calculation: http://www.postfix.org/SMTPD_POLICY_README.html Thanks Wietse. What I was thinking of doing was using check_recipient_access Will a 'NULL' result (ie: no rows returned) be treated as an OK? The format of access map lookup results is documented in the access(5) manual page: http://www.postfix.org/access.5.html This gives you the precise definition of all valid replies. Wietse
Re: Messages stuck in queue forever
Wendigo Thompson: Hi Wietse: Answering your first question, the message in question does show up in find: 26701170 872 -rwx--1 _postfix wheel 444689 Aug 4 2008 /var/spool/postfix/maildrop/F423E1976D72 This is one of two messages that you mentioned. When I look at the queue message, it is definitely representative of what I process by the thousands every day -- a message that came from my clients external mail server and was directed to my mail-drop alias. I have about a hundred of these messages and would really like to get them delivered. Any ideas? Normally, the Postfix pickup daemon will take mail from the maildrop directory and feed it into Postfix. Possibilities: 1) Your Postfix configuration never runs the pickup daemon, therefore mail stays in the maildrop directory. 2) The pickup daemon asks the kernel for the list of file names in the maildrop directory, but the listing does not contain the file named F423E1976D72. This would indicate that your file system is corrupted. You'd need to find a good time to shutdown the system and run a file system check. 3) The pickup daemon finds the file F423E1976D72, but the file content is in error. In this case the pickup daemon would log a warning message of some kind. ** You still need to find out where all the Postfix logging is ** stored on MacOS. Postfix logs all mail delivery attempts, ** whether successful or not. 4) Apple changed something that I don't know about, which makes the pickup daemon work in ways that I don't know about. In this case you need to talk to your vendor. I also looked for the other message I pasted this morning and it looks like it was successfully delivered, so I guess my problem is these messages that are no longer in the active queue, and how to get them delivered. That other message was dated January 6, so it was not stuck in the mail queue. Wietse
Re: Huge active queue and system idle, not delivering
Patrick Chemla: Hi, I am running Postfix 2.5.6 on a Fedora 11 Linux system on a hardware based Intel I5/750 Quad Core, 8 Gb memory, 160Gb SSD hard disk. Incoming messages are entering very fast (500 smtp processes declared) and the active queue is actually of 2 millions messages waiting for delivery. The delivery, for all messages should go through a farm of 30 MX servers from domain localpc2105.com, on load balancing through DNS resolution. DNS server is of course local. All 30 MX servers are running qmail. All of them are more than 90% idle. Before I set up my postfix server, email were sent directly to the qmail servers, and qmail was running at full CPU. So I am sure that qmail can handle much more faster. I have set up the postfix server to load balance the load between all the 30 qmail servers to avoid situation where some were running at full charge and others were not working. http://www.postfix.org/DEBUG_README.html#logging Wietse
Re: Messages stuck in queue forever
Wendigo Thompson: To answer Victor, message_size_limit is so. For Wietse: 1) Pickup seems enabled, from master.cf: pickupfifo n - n 60 1 pickup -o content_filter= Does the mail logfile show records of pickup daemon activity? If there is none, then this is the source of the problem. 2) The files are there via ls, so it seems OK 3) Postfix logs under the mail. syslog facility. What level should I set to see the pickup errors you are looking for? Postfix logs normal activity as mail.info, and warnings at mail.warn and higher levels, depending on the nature of the problem. Is there an option that defines a date after which messages should no longer be picked up? Maybe that's something I'm missing. Nope. As required by Internet mail standards, Postfix returns undeliverable mail to the sender. Wietse
Re: Messages stuck in queue forever
Victor Duchovni: On Thu, Jan 07, 2010 at 01:25:01PM -0600, Wendigo Thompson wrote: Woops, I meant to say: message_size_limit is zero. I went all verbose and looking at pickup logs for the specific queue ID I mentioned earlier: Jan 7 13:16:20 se001 postfix/pickup[57270]: warning: C9EBD24E12F23: message has been queued for 521 days Jan 7 13:16:20 se001 postfix/pickup[57270]: C9EBD24E12F23: uid=27 from= orig_id=F423E1976D72 Jan 7 13:16:20 se001 postfix/cleanup[75242]: C9EBD24E12F23: message-id=11eccdf1ff78c949b5011f37ba264e4619d4d...@10.1.1.1 After that all references to the new queue entry cease, and a minute later the process starts over again with the original queue id appearing to get a new ID, the warning about the message being queued (as little as 2 days and as long as 521, depending on the message). I never see a delivery message -- which I see plenty of for other messages coming in in real time. I recall that this message has no recipient (you moved the message with postsuper -r back to the maildrop queue). In that case the message is in error and it should be skipped and deleted. Since Postfix does accept mail without recipients, this file probably triggers some anomaly somewhere. Wietse
Re: Messages stuck in queue forever
Wietse Venema:
Victor Duchovni:
On Thu, Jan 07, 2010 at 01:25:01PM -0600, Wendigo Thompson wrote:
Woops, I meant to say: message_size_limit is zero.
I went all verbose and looking at pickup logs for the specific queue
ID I mentioned earlier:
Jan 7 13:16:20 se001 postfix/pickup[57270]: warning: C9EBD24E12F23:
message has been queued for 521 days
Jan 7 13:16:20 se001 postfix/pickup[57270]: C9EBD24E12F23: uid=27
from= orig_id=F423E1976D72
Jan 7 13:16:20 se001 postfix/cleanup[75242]: C9EBD24E12F23:
message-id=11eccdf1ff78c949b5011f37ba264e4619d4d...@10.1.1.1
After that all references to the new queue entry cease, and a minute
later the process starts over again with the original queue id
appearing to get a new ID, the warning about the message being queued
(as little as 2 days and as long as 521, depending on the message). I
never see a delivery message -- which I see plenty of for other
messages coming in in real time.
I recall that this message has no recipient (you moved the message
with postsuper -r back to the maildrop queue).
In that case the message is in error and it should be skipped
and deleted. Since Postfix does accept mail without recipients,
this file probably triggers some anomaly somewhere.
This patch should deal with messages that are requeued after all
their recipients are already delivered (or bounced).
Wietse
*** ./pickup.c- Fri Apr 17 16:03:58 2009
--- ./pickup.c Thu Jan 7 15:06:11 2010
***
*** 157,162
--- 157,163
struct stat st; /* queue file status */
char *path; /* name for open/remove */
char *sender; /* sender address */
+ int rcpt_count; /* todo recipients */
} PICKUP_INFO;
/*
***
*** 255,260
--- 256,263
if (type == REC_TYPE_RDR)
/* Discard REDIRECT record after postsuper -r. */
continue;
+ if (type == REC_TYPE_RCPT)
+ info-rcpt_count++;
}
if (*expected == REC_TYPE_EXTRACT[0]) {
if (type == REC_TYPE_RRTO)
***
*** 384,389
--- 387,399
return (status);
/*
+ * If we found no recipients, assume that this was a requeued message
+ * that was not deleted because the queue manager was restarted.
+ */
+ if (info-rcpt_count == 0)
+ return (REMOVE_MESSAGE_FILE);
+
+ /*
* There are no errors. Send the end-of-data marker, and get the cleanup
* service completion status. XXX Since the pickup service is unable to
* bounce, the cleanup service can report only soft errors here.
***
*** 486,491
--- 496,502
info-id = 0;
info-path = 0;
info-sender = 0;
+ info-rcpt_count = 0;
}
/* pickup_free - wipe info structure */
Re: Messages stuck in queue forever
Wendigo Thompson: *** ENVELOPE RECORDS /var/spool/postfix/maildrop/F423E1976D72 *** message_size: 444129 556 1 2 444129 message_arrival_time: Fri Aug 1 15:23:30 2008 create_time: Fri Aug 1 15:23:31 2008 named_attribute: rewrite_context=remote sender: named_attribute: log_client_address=10.10.10.1 named_attribute: log_message_origin=unknown[10.10.10.1] named_attribute: log_helo_name=10.10.10.10 named_attribute: log_protocol_name=ESMTP named_attribute: client_name=unknown named_attribute: reverse_client_name=unknown named_attribute: client_address=10.10.10.10 named_attribute: helo_name=10.10.10.10 named_attribute: client_address_type=2 named_attribute: notify_flags=1 named_attribute: dsn_orig_rcpt=rfc822;my_delivery_mail...@my_system original_recipient: my_delivery_mail...@my_system done_recipient: my_delivery_mail...@my_system *** MESSAGE FILE END /var/spool/postfix/maildrop/F423E1976D72 *** Note the done_recipient record. This message was requeued AFTER all the recipients were delivered (or bounced). The pickup daemon should delete such files, because there os nothing left to deliver. I have posted a patch in my previous response that dels with this special case. Wietse
Re: Messages stuck in queue forever
On Thu, Jan 7, 2010 at 2:14 PM, Wietse Venema wie...@porcupine.org wrote: Wendigo Thompson: *** ENVELOPE RECORDS /var/spool/postfix/maildrop/F423E1976D72 *** message_size: ? ? ? ? ?444129 ? ? ? ? ? ? 556 ? ? ? ? ? ? ? 1 ? ? ?2 ? ? ? ? ?444129 message_arrival_time: Fri Aug ?1 15:23:30 2008 create_time: Fri Aug ?1 15:23:31 2008 named_attribute: rewrite_context=remote sender: named_attribute: log_client_address=10.10.10.1 named_attribute: log_message_origin=unknown[10.10.10.1] named_attribute: log_helo_name=10.10.10.10 named_attribute: log_protocol_name=ESMTP named_attribute: client_name=unknown named_attribute: reverse_client_name=unknown named_attribute: client_address=10.10.10.10 named_attribute: helo_name=10.10.10.10 named_attribute: client_address_type=2 named_attribute: notify_flags=1 named_attribute: dsn_orig_rcpt=rfc822;my_delivery_mail...@my_system original_recipient: my_delivery_mail...@my_system done_recipient: my_delivery_mail...@my_system *** MESSAGE FILE END /var/spool/postfix/maildrop/F423E1976D72 *** Note the done_recipient record. This message was requeued AFTER all the recipients were delivered (or bounced). ?The pickup daemon should delete such files, because there os nothing left to deliver. I have posted a patch in my previous response that dels with this special case. Wendigo Thompson: Oh! You believe the messages were already delivered? That is a relief -- when I found these messages I was alarmed that my application was missing them. That's great news, Wietse -- thanks! All the recipients are marked as done, so there is nothing to deliver. Most likely, the message was requeued before the queue manager had a chance to delete it. Obviously, Postfix should handle this case better, but that requires source code changes as posted a few messages ago. Wietse
Re: Huge active queue and system idle, not delivering
Patrick Chemla: But the CPU of the box is idle more than 80%. It is clear that it is not a matter of CPU, nor memory, nor disk. Something in the number of processes/users/simultaneous tasks is blocking. Indeed, the symptom of blocking is in the third field of the Postfix delays logging. The format of the delays=a/b/c/d logging is as follows: o a = time from message arrival to last active queue entry o b = time from last active queue entry to connection setup o c = time in connection setup, including DNS, EHLO and TLS o d = time in message transmission In your case, it takes a minute or more to set up the connection including DNS lookup and EHLO handshake. That is holding up your mail. - Check if the qmail servers are responsive (telnet hostname 25). - Check if your Postfix needs a /var/spool/postfix/etc/resolv.conf file, and if that file is consistent with /etc/resolv.conf. If Postfix needs /var/spool/postfix/etc/resolv.conf and the file is missong or contains a bogus server that will add time to your deliveries. - If they aren't, increase the concurrency on the qmail side. Wietse
Re: Huge active queue and system idle, not delivering
Patrick Chemla: [ Charset ISO-8859-1 unsupported, converting... ] Le 08/01/2010 00:43, Victor Duchovni a ?crit : On Fri, Jan 08, 2010 at 12:30:34AM +0200, Patrick Chemla wrote: Jan 7 22:02:57 postfix postfix/qmgr[26441]: 5B91F873F6: removed Jan 7 22:02:57 postfix postfix/smtp[27180]: 375DDD5923: to=lexoti...@gmail.com, relay=a139.localpc2105.com[10.0.0.139]:25, conn_use=59, delay=61550, delays=17019/44435/96/0.17, dsn=2.0.0, ^^^ Note that this connection has been reused multiple times (see below for what this means in Postfix). Why does it take 69 seconds to initialize a reused SMTP connection? What happens when you set smtp_connection_cache_on_demand=no in main.cf (and do postfix reload)? If this makes a difference, then a) you have a problem with smtp-scache communication. b) qmail does not like RSET commands c) Your machine is running low on memory and swapping out the scache process. d) something else. Wietse Under high load, smtp(8) processes give their open connections to scache(8). Later, they ask scache(8) for an open connection to a specific destination. Once an smtp(8) client retrieves an open connection, it sends RSET tothe remote server and waits for a 250 reply (i.e. the server is still happy). According to the logfile record this lookup/rset/reply sequence is taking 96 seconds.
Re: Confusing sasl configuration examples
/dev/rob0: On Fri, Jan 08, 2010 at 08:37:16AM -0500, Shaun T. Erickson wrote: Yes, this is what is shown in the SASL Howto and how I have had my server's submission port configured in the past. However, in the 2.6.2 postfix distribution I'm trying to configure now, the default definition of the submission port uses the same restrictions, but it applies them to the smtpd_CLIENT_restrictions parameter, NOT the smtpd_RECIPIENT_restrictions parameter. I'm trying to understand if that is just a typo in master.cf or if the change is legit and, if so, why. Here's the example to which you refer: #submission inet n - n - - smtpd # -o smtpd_tls_security_level=encrypt # -o smtpd_sasl_auth_enable=yes # -o smtpd_client_restrictions=permit_sasl_authenticated,reject If you have already configured your smtpd_recipient_restrictions in main.cf to allow SASL AUTH, this example does indeed work. It's probably not a typo, but I agree, it can be confusing. Why do this with smtpd_client_restrictions, and yet assume that you didn't have smtpd_sasl_auth_enable=yes in main.cf already? The purpose of the submission service is to accept mail only from authenticated clients. The above submission entry implements this particular requirement without depending on main.cf settings. This is done for robustness reasons. Wietse One thing I have learned in my years on this list: Wietse usually has a reason, which might have been beyond my ability to understand. :) -- Offlist mail to this address is discarded unless /dev/rob0 or not-spam is in Subject: header
Re: maildrop transport, why using ${nexthop} instead of ${domain}
Philippe Cerfon:
Hi.
Just for my understanding
http://www.postfix.org/MAILDROP_README.html#direct uses -d
${us...@${nexthop}.
Why is nexthop used instead of domain? And when would these two be different?
Like Rome, Postfix was not built in one day.
Unfortunately these is no way to AUTOMATICALLY find all the pieces
of documentation that need to be updated when a new feature (such
as ${domain}) is introduced.
Wietse
man 8 pipe
PIPE(8)PIPE(8)
NAME
pipe - Postfix delivery to external command
SYNOPSIS
pipe [generic Postfix daemon options] command_attributes...
...
${domain}
This macro expands to the domain portion of the recipient
address. For example, with an address user+...@domain
the domain is domain.
...
This feature is available as of Postfix 2.5.
Re: Huge active queue and system idle, not delivering
Wietse Venema: Patrick Chemla: Le 08/01/2010 00:43, Victor Duchovni a ?crit : On Fri, Jan 08, 2010 at 12:30:34AM +0200, Patrick Chemla wrote: Jan 7 22:02:57 postfix postfix/qmgr[26441]: 5B91F873F6: removed Jan 7 22:02:57 postfix postfix/smtp[27180]: 375DDD5923: to=lexoti...@gmail.com, relay=a139.localpc2105.com[10.0.0.139]:25, conn_use=59, delay=61550, delays=17019/44435/96/0.17, dsn=2.0.0, ^^^ Note that this connection has been reused multiple times (see below for what this means in Postfix). Why does it take 69 seconds to initialize a reused SMTP connection? What happens when you set smtp_connection_cache_on_demand=no in main.cf (and do postfix reload)? If this makes a difference, then Check your qmail configuration for tarpit options. There exist qmail patches that will slow down the qmail SMTP server when the client sends lots of email, or lots of recipients. Wietse a) you have a problem with smtp-scache communication. b) qmail does not like RSET commands c) Your machine is running low on memory and swapping out the scache process. d) something else. Wietse Under high load, smtp(8) processes give their open connections to scache(8). Later, they ask scache(8) for an open connection to a specific destination. Once an smtp(8) client retrieves an open connection, it sends RSET tothe remote server and waits for a 250 reply (i.e. the server is still happy). According to the logfile record this lookup/rset/reply sequence is taking 96 seconds.
Re: question about smtp_connection_cache
Davy Leon: Hi folks I have a Centos 5.3 Box running postfix 2.6.3 . I use an smarthost in a per user authentication basis. I meant every local user has a passwrod to use against the smarthost. My question is, in this conditions is it possible to take advantage of smtp_connection_cache so all the queued messages from the same user can be delivered to the smarthost using the same smtp conection and avoid sasl handshaking for every message who has the same from address? An important note is the delivery is deferred. My server is not online all the time. As documented in the CONNECTION_CACHE_README file: Connection cache limitations Postfix SMTP connection caching conflicts with certain applications: * The Postfix shared connection cache cannot be used with TLS, because saved TLS session information can be used only when a new connection is created (this limitation does not exist in connection caching implementations that reuse a connection only in the process that creates it). For this reason, the Postfix smtp(8) client always closes the connection after completing an attempt to deliver mail over TLS. * Postfix connection caching currently does not support multiple SASL accounts per mail server. Specifically, Postfix connection caching assumes that a SASL credential is valid for all hostnames or domain names that deliver via the same mail server IP address and TCP port, and assume that the SASL credential does not depend on the message originator. The Postfix documentation is a great resource of information. Wietse
Re: Confusing sasl configuration examples
/dev/rob0: On Fri, Jan 08, 2010 at 10:23:38AM -0500, Wietse Venema wrote: /dev/rob0: On Fri, Jan 08, 2010 at 08:37:16AM -0500, Shaun T. Erickson wrote: Yes, this is what is shown in the SASL Howto and how I have had my server's submission port configured in the past. However, in the 2.6.2 postfix distribution I'm trying to configure now, the default definition of the submission port uses the same restrictions, but it applies them to the smtpd_CLIENT_restrictions parameter, NOT the smtpd_RECIPIENT_restrictions parameter. I'm trying to understand if that is just a typo in master.cf or if the change is legit and, if so, why. Here's the example to which you refer: #submission inet n - n - - smtpd # -o smtpd_tls_security_level=encrypt # -o smtpd_sasl_auth_enable=yes # -o smtpd_client_restrictions=permit_sasl_authenticated,reject If you have already configured your smtpd_recipient_restrictions in main.cf to allow SASL AUTH, this example does indeed work. It's probably not a typo, but I agree, it can be confusing. Why do this with smtpd_client_restrictions, and yet assume that you didn't have smtpd_sasl_auth_enable=yes in main.cf already? The purpose of the submission service is to accept mail only from authenticated clients. This, I understand. The above submission entry implements this particular requirement without depending on main.cf settings. This, I do not. The submission service accepts mail only from authenticated clients. To implement this, the master.cf entry enforces these requirements: a) the requirement that the client encrypts the connection TLS. b) the requirement that the client authenticates with SASL. The master.cf entry enforces this without depending on main.cf configuration. In particular, a) It does not depend on TLS being enabled or required in main.cf. b) It does not depend on SASL being enabled or required in main.cf. Wietse
Re: Virtual reject reason
Christopher Hackman: Hello, I'm pretty new to Postfix, and I'm experimenting with using it for our new inbound MTA. Is it possible to customize the following error message? MAIL FROM: u...@remotedomain.com 250 2.1.0 Ok RCPT TO: invalidacco...@virtualdomain.com 550 5.1.1 invalidacco...@virtualdomain.com: Recipient address rejected: virtualdomain.com Depending on the show_user_unknown_table_name parameter setting, Postfix replies with one of the following: 550 5.1.1 u...@example.com: Recipient address rejected: user unknown 550 5.1.1 u...@example.com: Recipient address rejected: user unknown in local recipient table 550 5.1.1 u...@example.com: Recipient address rejected: user unknown in virtual alias table 550 5.1.1 u...@example.com: Recipient address rejected: user unknown in virtual mailbox table 550 5.1.1 u...@example.com: Recipient address rejected: user unknown in relay recipient table In this sanitized example, virtualdomain.com is just that, as is invalidaccount. I see unknown_virtual_mailbox_reject_code will let me change the response code, but not the text. I'd expected to find something like a unknown_virtual_mailbox_reject_reason parameter, but I don't see one. Is this possible? This is not documented, therefore it is not implemented. Wietse
Re: question about smtp_connection_cache
Davy Leon: Thanks Wietse. Very usefull your answer. I was making a huge mistake. Thanks If you have multiple SASL accounts with the same provider, then you may want to set smtp_connection_cache_on_demand = no. At some point someone may add code that labels the connection cache with the SASL login name, but that hasn't happened yet. Wietse
Re: maildrop transport, why using ${nexthop} instead of ${domain}
Philippe Cerfon:
On Fri, Jan 8, 2010 at 4:40 PM, Wietse Venema wie...@porcupine.org wrote:
Like Rome, Postfix was not built in one day.
Unfortunately these is no way to AUTOMATICALLY find all the pieces
of documentation that need to be updated when a new feature (such
as ${domain}) is introduced.
Ah so they're the same?
I will refer you to the pipe(8) manual page.
Another cosmetic thing that could be added here is:
- The null_sender section tells that it's also subject to the q flag,
the description of the q flag however doesn't list it.
The manual page says that the expansion of $sender is subject to
the q flag. This is true regardless of where the expansion result
comes from, so there is no need to repeat this for null_sender.
- user and extension sections they that they're subject to u or h
flags,
These sections say that they are subject to the u flag.
By the way, may I ask two more minor questions, regarding pipe:
1) Why is the default for null_sender=MAILER-DAEMON? I believe you
that it has some good reason, just would like to know which one? Are
there MUAs that misbehave if they get emails with an empty return
path?
As documented in the manual page, a null sender address is easily
mis-parsed by naive programs. If their programmer is aware that
empty strings require special handling, then they can recommend
changing the null_sender setting. Otherwise, don't change it.
2) I do not fully understand the q flag. Does it mean that pipe gets
addresses from postfix which can contain spaces or other non-ascii
character, and with that it's encoded?
As documented in the manual page, the address is quoted per RFC
822 rules. You are welcome to review that document, or any of its
successors, RFC 2822 and RFC 5322.
Wietse
Re: Sender based relay server
Jack Knowlton: Hi all. Our internal postfix server relays all outbound mail thru an external host. How can I set it to use a different relay server when the email comes from a specified domain? Eg. j...@domain1.com - xxx.xxx.xxx.xxx (default), m...@domain2.com - yyy.yyy.yyy.yyy Postfix 2.3 and later: http://www.postfix.org/postconf.5.html#sender_dependent_relayhost_maps And perhaps: http://www.postfix.org/SOHO_README.html Wietse
Re: Huge active queue and system idle, not delivering
Patrick Chemla: Hi all, I got these statistics: Jan 9 19:15:21 postfix postfix/scache[18038]: statistics: start interval Jan 9 19:09:03 Jan 9 19:15:21 postfix postfix/scache[18038]: statistics: domain lookup hits=110 miss=89 success=55% Jan 9 19:15:21 postfix postfix/scache[18038]: statistics: address lookup hits=0 miss=2492 success=0% Jan 9 19:15:21 postfix postfix/scache[18038]: statistics: max simultaneous domains=1 addresses=4 connection=4 Please try the following, as asked half a week ago: postconf -e smtp_connection_cache_on_demand=no postfix reload and report if this makes a difference. Wietse
Re: Huge active queue and system idle, not delivering
Wietse Venema: Patrick Chemla: Hi all, I got these statistics: Jan 9 19:15:21 postfix postfix/scache[18038]: statistics: start interval Jan 9 19:09:03 Jan 9 19:15:21 postfix postfix/scache[18038]: statistics: domain lookup hits=110 miss=89 success=55% Jan 9 19:15:21 postfix postfix/scache[18038]: statistics: address lookup hits=0 miss=2492 success=0% Jan 9 19:15:21 postfix postfix/scache[18038]: statistics: max simultaneous domains=1 addresses=4 connection=4 Please try the following, as asked half a week ago: postconf -e smtp_connection_cache_on_demand=no postfix reload and report if this makes a difference. Oh, and please limit the discussion to people who understand the hard technical internals of Postfix. Other people please stay out of the way. Wietse
Re: how are sysexit.h statues interpreted
Christoph Anton Mitterer: Hi. Is there somewhere some documentation how each of the exit codes from sysexit.h is interpreted by Postfix when used with pipe(8) (returned e.g. by maildrop)? I naively assume that the sysexits.h names speak for themselves. I just now the EX_TEMPFAIL means that mail is defered, and I assume EX_UNAVAILABLE leads to a bounce. What about the others? EX_TEMPFAIL defers mail, as does EX_OSERR (system resource not available). All others are hard coded as non-retryable. Making this configurable is a couple hours of work (design a user interface, implement the code, test the code, preferable with an automated test that exercises all the cases, document the user interface). The current mapping is in global/sys_exits.c. But generally, if the exit status != 0 Postfix first looks at (I assume) the first line of stdout and interprets 4.X.X or 5.X.X as said in the manpage. If found, the EX_* are not interpreted, right? That behavior is documented in the pipe(8) and local(8) manpages, in the paragraphs that discuss RFC 3463 enhanced status codes. If there is anything not correct in that text then I am sure you will post an improvement. Wietse
Re: how are sysexit.h statues interpreted
Christoph Anton Mitterer: On Sat, 2010-01-09 at 19:58 -0500, Wietse Venema wrote: EX_TEMPFAIL defers mail, as does EX_OSERR (system resource not available). All others are hard coded as non-retryable. Thanks. Making this configurable is a couple hours of work (design a user interface, implement the code, test the code, preferable with an automated test that exercises all the cases, document the user interface). The current mapping is in global/sys_exits.c. Ah,.. I see the mappings... If there is anything not correct in that text then I am sure you will post an improvement. No no,.. everything ok. I hope my previous remarks on sections where I thought to have found errors or similar are not disapproved. If so I'll stop trying to make such minor contributions. Constructive feedback is welcome. If a question can be answered by documentation, then I will refer to that text, but won't answer every question. Wietse
Re: Limitations of smtpd_proxy_filter
Stefan Foerster: So, my question is: Are there any plans to further enhance Postfix's functionality as far as SMTP proxy filtering is concerned, and make that kind of load balancing available to pre-queue setups using smtpd_proxy_filter? For TRUE load balancing you need a stateful system that remembers which connections are going where. You could try the nginx reverse proxy; it speaks SMTP among many protocols, and is often used for software-based load balancing. For STATISTICAL load balancing you can get by with multiple IP addresses per proxy filter host name. However this requires new code in Postfix to randomize the address order. You may know that the Postfix smtpd_proxy client does not make its own SMTP connections. Instead it uses the same Postfix library functions for making a connection over TCP, UNIX-domain sockets, System V streams, etc. as all other Postfix programs (except the SMTP client, which does its own name-to-address translation). Changing this library means changing applications that call into this library (Postfix invokes the _connect() functions via function pointers, so a simple #define old(x,y,z) new(x,y,z,0) won't do the job. I have a bunch of deliverables in the second and third quarters of 2010, so I must get Postfix 2.7 out in the first quarter. This means I have few cycles left for Postfix 2.7. I would like to spend those on postscreen. Wietse
Re: Limitations of smtpd_proxy_filter
Stefan Foerster: * Wietse Venema wie...@porcupine.org: For STATISTICAL load balancing you can get by with multiple IP addresses per proxy filter host name. However this requires new The main difference I see here is that delivery to a content_filter will try more than one server, giving us not only (statistical) load balancing, but also some kind of redundancy. That would work only when the smtpd_proxy client is modified to use the SMTP connection cache. Otherwise, a connection still has to time out before the next host is tried. (*) There is no such performance hit with true load balancers, which maintain state about which hosts are busy/dead/other. Wietse (*) It takes major surgery to make the smtpd_proxy client as smart as the Postfix SMTP client (address randomization, connection caching, bug workarounds, and more) Namely, to make the Postfix SMTP client usable as an smtpd_proxy client.
Re: Does Postfix cache resolv.conf?
Dr. Lars Hanke: Apparently postfix missed the switching of nameservers and did not learn of the new DNS until restart. Is this a bug or a feature? Like most programs, Postfix never reads /etc/resolv.conf. Instead, that file is read by the NSSWITCH system library functions, and the DNS system library functions. Postfix does not run on bare metal - instead it runs on an operating system that provides libraries and other runtime support. Wietse
Re: Huge active queue and system idle, not delivering
Patrick Chemla: Wietse, Please try the following, as asked half a week ago: postconf -e smtp_connection_cache_on_demand=no postfix reload and report if this makes a difference. Wietse I have tested this since yesterday night. I got some problems with Linux per user number of processes limit. I fixed it. I also increased some delivery concurrency figures, and now I can see up to 1300 processes delivering emails to the qmail servers. I had a few minutes shot today at a rate of 6300 emails per minute. I ran a full hour at 180,000 emails per hour. The outbound line was saturated. CPU is about 30% loaded, no Wait I/O, no swap, memory is large. I think I will reach about 600,000 emails per hour if I fix some timeout on the qmails (replace by postfix?). Maybe I could reach 1 million? OK, so you can turn back on that connection caching. Note that qmail creates and destroys two processes per SMTP session, so reusing a session is also a win from a CPU resource point of view. 1M/hour, or less than 300/s, should be possible (you mention the queue is on a solid-state disk). Barring brain damage such as synchronous syslogging by default on some Linux boxes, borked DNS, process/file/etc. resource limits, etc. Perhaps this is a good time to mention that SecurityFocus was ezmlm-qmail based, and that they switched to Postfix for outbound deliveries, because qmail simply could not keep up with the volume: inbound mail - qmail - ezmlm - multiple postfix MTAs - internet That was 2001 when I added QMQP support to Postfix, and this is still what they appear to be using now, if I must believe their own Received: message headers. Received: from lists2.securityfocus.com (lists2.securityfocus.com [205.206.231.20]) by outgoing2.securityfocus.com (Postfix) with QMQP id 8AC0814370A; Thu, 7 Jan 2010 14:11:35 -0700 (MST) My very first qmail/Postfix benchmarks showed that qmail was up to three times slower as a transit MTA, simply because qmail creates three queue files where Postfix creates one. Creating/deleting files involves more disk access operations than reading/writing files, and that hurts especially with small email messages. Wietse
Re: Limitations of smtpd_proxy_filter
Stefan Foerster: One of the greatest improvements in the 2.7 tree is the ability to defer transmission of received messages to a SMTP proxy until the message receiption completes (smtpd_proxy_options = speed_adjust). Can you be more specific about the benefits? Wietse
Re: Huge active queue and system idle, not delivering
Patrick Chemla: Wietse: OK, so you can turn back on that connection caching. Note that qmail creates and destroys two processes per SMTP session, so reusing a session is also a win from a CPU resource point of view. Patrick: If I do so, will postfix open more than one connexion to each qmail for parallel deliveries? Of course. Connection caching is a performance IMPROVEMENT feature. However, some qmail implementations are patched and turn on TARPIT delays when the client sends many messages or recipients over the same SMTP connection. Wietse
Re: Limitations of smtpd_proxy_filter
Stefan Foerster: * Wietse Venema wie...@porcupine.org: Stefan Foerster: One of the greatest improvements in the 2.7 tree is the ability to defer transmission of received messages to a SMTP proxy until the message receiption completes (smtpd_proxy_options = speed_adjust). Can you be more specific about the benefits? Number of filters is down by more than 40%. Comparing a typical day before and after we deployed that feature: http://www.incertum.net/~cite/speed_adjust.png As with the first public release release, Postfix avoids the need for a forklist upgrade (that was geek talk for having to install more powerful hardware). Now I can claim it as a green option. Other sites might not notice such a large impact: Our setup includes a large number of clients connected via lines with little bandwidth (2MBit/s) which is typically used for other purposes besides sending mail. If the time a mail transmission takes is negligible, I expect the results to be vastly different. It also depends on how far away the clients are: each round-trip time adds to the duration of a session (from this location near New York I measure some 90ms RTT to europe and 175ms to Japan). Furthermore (I'm not 100% sure about this, would be glad if you could confirm this) it's much harder to DoS a Postfix server with that patch: Previously, all you had to do was opening a number of connections equal to the number of pre-queue content filter processes, go through the initial (E)SMTP steps and then send a mail very slowly. As long as the number of SMTP server slots is finite, there will always be saturation effects. The main trick of speed-adjust is that you can run more SMTP server processes than filters, and thereby reduce the impact of slow or non-cooperative clients. Of course the system still saturates immediately when the PEAK email volume exceeds the filter capacity. With after-queue filters, saturation happens when the AVERAGE volume exceeds filter capacity. With smtpd_proxy_options=speed_adjust, you'd have to DoS the Postfix SMTP server itself, and between postscreen guarding it, high process limits, access control and the stress adaptive server personality, this one is well able to take care of itself. I have been looking for incremental steps to improve Postfix's handling of non-cooperating clients, and speed-adjust is part of those steps. Postscreen greylisting support should also help a lot, but that will probably not be in Postfix 2.7. Of course all these things do is increase the threshold. No system except a wire cutter can deal with unlimited amounts of traffic. Wietse
Re: FILTER nexthop woes
l...@ds.gauner.org: Hi, I'm trying to use header_checks in conjunction with a pcre map to distribute certain mail traffic to certain outgoing transports. I've got a setup like this: --- main.cf snip --- header_checks = pcre:/etc/postfix/header.pcre --- snap --- --- header.pcre snip --- /^X-CUSTOMER-ID: ([0-9])/ FILTER smtpout$1: You MUST specify a nexthop destination. The purpose of FILTER is to send mail for MANY destinations through ONE filter destination. If you don't specify a next-hop destination, then Postfix will choose a default one. To make Postfix routing sender dependent, use sender_dependent_relayhost_maps (Postfix 2.3 and later) or sender_dependent_default_transport_maps (Postfix 2.7 and later). Wietse
Re: Messages are stuck in maildrop
Eugueny Kontsevoy: I had some issues with free disk space so I had to start the server, expand the partition and restart it again. I did the following: postsuper -r ALL It reported that it re-queued a bunch of messages and I can see that they're sitting in maildrop queue. postqueue -p reports that queue is empty (which one???) And those messages just sit there, pickup dameon is ignoring them. Where do I look? 1) Lok in Postfix log for warning messages. 2) Examine a message with postcat. if all recipients are done_recipient then the message was already delivered and it can be deleted. You postsuper-ed the message before the queue manager had a chance to delete it. Wietse
Re: FILTER nexthop woes
Dominik Schulz: Am Montag 11 Januar 2010 15:08:05 schrieb Wietse Venema: l...@ds.gauner.org: Hi, I'm trying to use header_checks in conjunction with a pcre map to distribute certain mail traffic to certain outgoing transports. I've got a setup like this: --- main.cf snip --- header_checks = pcre:/etc/postfix/header.pcre --- snap --- --- header.pcre snip --- /^X-CUSTOMER-ID: ([0-9])/ FILTER smtpout$1: You MUST specify a nexthop destination. The purpose of FILTER is to send mail for MANY destinations through ONE filter destination. If you don't specify a next-hop destination, then Postfix will choose a default one. To make Postfix routing sender dependent, use sender_dependent_relayhost_maps (Postfix 2.3 and later) or sender_dependent_default_transport_maps (Postfix 2.7 and later). Thanks for the reply. Unfortunately I don't seem to have myself clear. I want to controll to nexthop (i.e. the outgoing relay) through some kind of X--Header, e.g. X-CUSTOMER_ID: 34554, and not through the sender/recipient addresses. Is there some way to achieve this behaviour? The short answer: specify a FILTER command that points to a Postfix instance that has its own unique hostname, and that is bound to its own unique IP address. myhostname = unique-name.example.com inet_interfaces = $myhostname Adding Postfix instances is easy with Postfix 2.6. Until someone explains why they must use the header and can't use the envelope sender, I am not inclined to invest a lot of effort on my side to find out if there exists a better solution. Wietse
Re: How to not reject valid MTAs for inconsistent forward/reverse DNS.
Frank Cusack: On January 13, 2010 8:16:36 AM -0600 Stan Hoeppner s...@hardwarefreak.com wrote: Frank Cusack put forth on 1/12/2010 9:46 PM: I think it all ended well though? Except my problem still exists. :\ We know things break when that hosts sends mail to you. What happens when you send mail to that host? Do you see the same disconnect problem or similar? What were the results of tcpdump? Sending mail works fine. Interesting. Contrary to what I said earlier, tcpdump is in fact interesting. I see a 3 way handshake, and that's it. 10 minutes later, a reset. However postfix logs a disconnect immediately. I do notice that their mss is only 1260, so clearly they are going through some kind of packet mangling firewall. But that doesn't excuse postfix for declaring the connection disconnected when in fact no packet came from them. Would you be willing to share this complete evidence so that other people can learn from it, or do you expect that we just take your eyewitness report every claim made here sofar? Perhaps surprisingly, Postfix does not send or receive network packets. Instead, packets are handled by the TCP/IP implementation in the operating system kernel. If anything decides prematurely that the connection is dead, it is your operating system kernel not Postfix. Please do not fall into the trap of accusing the messenger (Postfix) of the bad news that comes from your operatring system kernel (connection is dead). Wietse What I don't understand though, is if smtpd goes away prematurely, then when it exits shouldn't a FIN or RST get sent? I'm going to explore the resolver issue. -frank
Re: How to not reject valid MTAs for inconsistent forward/reverse DNS.
Frank Cusack: Perhaps surprisingly, Postfix does not send or receive network packets. Instead, packets are handled by the TCP/IP implementation in the operating system kernel. If anything decides prematurely that the connection is dead, it is your operating system kernel not Postfix. Unless of course postfix has a bug (heaven forbid). Well, if you can provide unmodified evidence, then people can look into this. - I already asked for the content of the packet dump including packet headers, packet flags, packet contents and so on. Based on an analysis of this, - I may also ask you to collect Postfix verbose logging for that particular IP address. That gives the Postfix view of what's going on. Based on that, - I may also ask you to collect a system call trace. That would give the kernel's view of what's going on. In the meantime, it would help if you could stop posting further speculation and theoretizing until we have had a chance to look at actual concrete unmodified evidence. Wietse
Re: postscreen stress=yes
Noel Jones:
Is postscreen supposed to always run with stress=yes?
Seems to me stress-adaptive behavior would be useful in
postscreen.
The stress=yes setting indicates that a master.cf service is using
up all its process slots. It is applicable only for servers that
accept connections from remote clients.
By design, postscreen runs as a single process. The stress=yes
parameter value therefore carries no useful information. Either
the postscreen service is not running at all, or it uses up all
its process slots because there is only one.
I could make a special case for single-process servers in master.cf
so that they don't have a stress parameter.
Wietse
*** ./master_ent.c- Sat Jan 10 19:02:29 2009
--- ./master_ent.c Wed Jan 13 15:04:47 2010
***
*** 526,532
argv_add(serv-args, -u, (char *) 0);
if (chroot)
argv_add(serv-args, -c, (char *) 0);
! if ((serv-flags MASTER_FLAG_LOCAL_ONLY) == 0) {
argv_add(serv-args, -o, stress= CONFIG_BOOL_YES, (char *) 0);
serv-stress_param_val =
serv-args-argv[serv-args-argc - 1] + sizeof(stress=) - 1;
--- 526,532
argv_add(serv-args, -u, (char *) 0);
if (chroot)
argv_add(serv-args, -c, (char *) 0);
! if ((serv-flags MASTER_FLAG_LOCAL_ONLY) == 0 serv-max_proc 1) {
argv_add(serv-args, -o, stress= CONFIG_BOOL_YES, (char *) 0);
serv-stress_param_val =
serv-args-argv[serv-args-argc - 1] + sizeof(stress=) - 1;
Re: Bounces
Dhiraj Chatpar: Dear All, What string or what configuration to use in postfix in order to not receive any bounces at all. I mean incase there is a bounce it should not be returned back to the sender who initiated the mail. I am sure there is a way to achieve this in postfix See: RFC 3461 (SMTP DSN Extension) The DSN option controls how to request bounces with SMTP mail. See: http://www.postfix.org/sendmail.1.html The -N command-line option controls how to request bounces with the Postfix sendmail command. Wietse
Re: Bounces
Dhiraj Chatpar: Dear All, What string or what configuration to use in postfix in order to not receive any bounces at all. I mean incase there is a bounce it should not be returned back to the sender who initiated the mail. I am sure there is a way to achieve this in postfix Wietse: See: RFC 3461 (SMTP DSN Extension) The DSN option controls how to request bounces with SMTP mail. See: http://www.postfix.org/sendmail.1.html The -N command-line option controls how to request bounces with the Postfix sendmail command. Dhiraj Chatpar: How do i make the changes in postfix so that it stops sending out bounced mail totally? YOU MUST FIX YOUR CONFIGURATION ERROR, and stop accepting mail for non-existent recipients. See http://www.postfix.org/ADDRESS_CLASS_README.html to set up the proper recipient address validation BEFORE accepting mail. Wietse
Re: connections from ipv4 localhost logged as unknown[127.0.0.1]
Tom Hendrikx: Hi, After setting up postfix up on a ipv4/ipv6 dualstack machine I'm seeing the following issue: connections on 127.0.0.1 (where my content_filter re-injects mail) are logged as: 010-01-13T22:51:07+01:00 meredith-vmail postfix/smtpd[4772]: warning: 127.0.0.1: address not listed for hostname ip6-localhost Given the client IP address 127.0.0.1, Postfix gets the name ip6-localhost, but that name does have the address 127.0.0.1. After some time reading google and debugging this is what I found out: - /etc/hosts contains the following stuff regarding localhost (in the specified order): ::1 ip6-localhost ip6-loopback 127.0.0.1 localhost And indeed. Name ip6-localhost does not list address 127.0.0.1. Somewhere, you have a mapping from 127.0.0.1 that returns ip6-localhost, and that mapping is screwing things up, because 127.0.0.1 is not listed as an address for ip6-localhost. Wietse
Re: connections from ipv4 localhost logged as unknown[127.0.0.1]
Tom Hendrikx: Wietse Venema wrote: Tom Hendrikx: Hi, After setting up postfix up on a ipv4/ipv6 dualstack machine I'm seeing the following issue: connections on 127.0.0.1 (where my content_filter re-injects mail) are logged as: 010-01-13T22:51:07+01:00 meredith-vmail postfix/smtpd[4772]: warning: 127.0.0.1: address not listed for hostname ip6-localhost Given the client IP address 127.0.0.1, Postfix gets the name ip6-localhost, but that name does have the address 127.0.0.1. After some time reading google and debugging this is what I found out: - /etc/hosts contains the following stuff regarding localhost (in the specified order): ::1 ip6-localhost ip6-loopback 127.0.0.1 localhost And indeed. Name ip6-localhost does not list address 127.0.0.1. Somewhere, you have a mapping from 127.0.0.1 that returns ip6-localhost, and that mapping is screwing things up, because 127.0.0.1 is not listed as an address for ip6-localhost. I got as far as this conclusion too, which got me checking for the contents of /etc/hosts. Since I can influence the lookup results easily by shuffling the contents of /etc/hosts, can I conclude that this is an issue with my glibc? Obviously, there is no 127.0.0.1 - ip6-localhost mapping in /etc/hosts. Therefore, when Postfix gets ip6-localhost when it asks the name for 127.0.0.1, then that information did not come from /etc/hosts. You can tweak your /etc/hosts until the cows come home, or you can try to find out were that 127.0.0.1-ip6-localhost is coming from. Wietse
Re: Multiple From: in a mail header?
Is that possible for mail headers field to continue multiple /^From: .*/ speaking in terms of maildrop and PCRE? According to RFC 5322: from= From: mailbox-list CRLF mailbox-list= (mailbox *(, mailbox)) / obs-mbox-list Thus, one From: header may contain multiple addresses. Wietse
Re: Use VERP always
ram: I need to configure a postfix server that relays mails from a windows based App. The App is an age old custom application written and has no options to configure except an outgoing IP :-( But I need to manage the bounces and so I was thinking of changing the env from to a VERP format. How do I configure postfix to always change the email to a VERP format I have never had good input on VERP user interface design, so your options are limited: a) Use an SMTP proxy that produces SMTP commands as described in http://www.postfix.org/VERP_README.html b) Edit Postfix source code. Wietse
Re: Use VERP always
Wietse Venema: ram: I need to configure a postfix server that relays mails from a windows based App. The App is an age old custom application written and has no options to configure except an outgoing IP :-( But I need to manage the bounces and so I was thinking of changing the env from to a VERP format. How do I configure postfix to always change the email to a VERP format I have never had good input on VERP user interface design, so your options are limited: a) Use an SMTP proxy that produces SMTP commands as described in http://www.postfix.org/VERP_README.html b) Edit Postfix source code. On the sending side, a possible usability feature would be: - Adding a VERP command to Postfix access maps or header/body_checks, which would turn on VERP-style delivery for a particular message. Turning VERP based on email content would be safe only when you don't receive mail from hostile systems. Wietse
Re: Speeding up Local Delivery
Wendigo Thompson: Victor: I wanted to use Postfix as a delivery mechanism so that I would be able to queue and hold mail during database outages and other maintenance -- a job that it does very well. Is there really no advice available on how to increase the speed of Postfix's local mail delivery? I'm the guy who wrote Postfix. With your numbers, 300k messsages/day and 3ms/delivery, the whole day's volume should be done in about 900s. Since you see deliveries draining into the weekend, your numbers don't seem to add up. In the Postfix logs you can see how long it took to deliver mail. Here is an example of a pipe alias that handles all my own email: Jan 14 00:08:51 spike postfix/local[98331]: 9F53A1F3EA5: to=xxx, relay=local, delay=0.41, delays=0.37/0.02/0/0.03, dsn=2.0.0, status=sent (delivered to command: yyy) Here, it took 0.03 seconds to deliver a small message for xxx to a small program yyy. In addition it took 0.37s+0.02=0.39s to receive the message and schedule it for delivery. This message arrived via a slow connection. More information about this is at http://www.postfix.org/postconf.5.html#delay_logging_resolution_limit Now, If you want to deliver hundreds of thousands each day to a pipe alias, then executing a whole program each time is expensive, but it is possible if your machine has enough horse power. So, we have two options: a) The red option: to waste more resources, increase concurrency, by increasing local_destination_concurrency_limit (default: 2). b) The green option: instead of wasting CPU cycles with a local(8) pipe alias that executes and terminates one program per delivery, use transport_maps to direct mail to a persistent SMTP or LMTP process that keeps the database connection open. This is the option that Victor suggested. Wietse
Re: Speeding up Local Delivery
Wendigo Thompson: Wietse: I know who you are -- big fan of yours ever since tcpwrappers :-) Hah! (As for my claim, of course other people also wrote parts of Postfix, in particular Victor has been a major contributor). The only reason I wrote in is that this queueing of mail issue appeared fairly recently, and other times when I have had to collect large volumes of mail from this client my system has been fine. Below I have pasted some logs that show long delivery times (sitting in the queue for a day) but the last delay (delivery delay?) is in the upper-teens. Just trying to reconcile why this is: if it's something on my end that is fine and I can work it out, but maybe it is an unforeseen issue with Postfix and large volumes of piped mail. The system itself is a Nehalem Xserve, so it seems to have plenty of horsepower to handle the load. (anonymized a bit in hostname and email address) Jan 14 15:45:43 server001 postfix/local[24570]: 26E371265ED: to=receiver_em...@receiver_ip, relay=local, delay=81352, delays=0.59/81343/0/7.7, dsn=2.0.0, status=sent (delivered to command: /usr/bin/db_capture) This looks problematic: it took 7.7s from the time that the Postfix local delivery agent was tasked to deliver the message, to the time that the db_capture command reported completion. Plus, these messages had already been sitting in the queue for almost 24 hours, so the problem will have started much earlier than that. delays=0.5/81344/0/8.2, dsn=2.0.0, ... delays=0.34/81341/0/13, dsn=2.0.0, ... delays=0.41/81344/0/10, dsn=2.0.0, ... delays=0/81339/0/19, dsn=2.0.0, ... delays=0.42/81342/0/14, dsn=2.0.0, ... delays=0.19/81344/0/11, dsn=2.0.0, ... delays=0/81341/0/15, dsn=2.0.0, ... delays=0/81337/0/21, dsn=2.0.0, ... delays=0.57/81343/0/11, dsn=2.0.0, ... delays=0.32/81342/0/16, dsn=2.0.0, ... delays=0.21/81341/0/18, dsn=2.0.0, ... delays=0.35/81348/0/7, dsn=2.0.0, ... delays=0.12/81339/0/21, dsn=2.0.0, ... delays=0.09/81340/0/20, dsn=2.0.0, ... Now the question is why did /usr/bin/db_capture take so long? I am not sure that my red and green options are going to solve your performance problem. If you still have the logs, it will be worthwhile to find out when the delivery delay started climbing through the roof. Wietse
Re: Speeding up Local Delivery
Wendigo Thompson: Wietse: Sadly I generally don't keep logs at this granular of a level (mail.crit is what I usually store forever) but I can turn logging back on after the queue empties over the weekend and see if the delay is persisting. I know the command itself is completing quickly -- it seems like something in between is causing a delay. Do you have any suggestions for how I could profile the handoff from local to the executable? The fourth delays timer, which show 7.7s delays and more, stops as soon as your /usr/bin/db_capture terminates. Perhaps it is doing hostname lookup and the resolv.conf or nsswitch.conf files are out of whack. Perhaps it is taking forever to log in. To investigate, configure Postfix to run it under strace (Linux) truss (Solaris) or other. In your alias: |strace -tt -o /tmp/db_capture. /usr/bin/db_capture (I think you need to specify four to get two, which is what I want here, so that each /tmp/db_capture file is tagged with the process ID). Wietse Thanks again, Wendy On Thu, Jan 14, 2010 at 4:13 PM, Wietse Venema wie...@porcupine.org wrote: Wendigo Thompson: Wietse: ? I know who you are -- big fan of yours ever since tcpwrappers :-) Hah! (As for my claim, of course other people also wrote parts of Postfix, in particular Victor has been a major contributor). ? The only reason I wrote in is that this queueing of mail issue appeared fairly recently, and other times when I have had to collect large volumes of mail from this client my system has been fine. ?Below I have pasted some logs that show long delivery times (sitting in the queue for a day) but the last delay (delivery delay?) is in the upper-teens. ?Just trying to reconcile why this is: if it's something on my end that is fine and I can work it out, but maybe it is an unforeseen issue with Postfix and large volumes of piped mail. ?The system itself is a Nehalem Xserve, so it seems to have plenty of horsepower to handle the load. (anonymized a bit in hostname and email address) Jan 14 15:45:43 server001 postfix/local[24570]: 26E371265ED: to=receiver_em...@receiver_ip, relay=local, delay=81352, delays=0.59/81343/0/7.7, dsn=2.0.0, status=sent (delivered to command: /usr/bin/db_capture) This looks problematic: it took 7.7s from the time that the Postfix local delivery agent was tasked to deliver the message, to the time that the db_capture command reported completion. Plus, these messages had already been sitting in the queue for almost 24 hours, so the problem will have started much earlier than that. delays=0.5/81344/0/8.2, dsn=2.0.0, ... delays=0.34/81341/0/13, dsn=2.0.0, ... delays=0.41/81344/0/10, dsn=2.0.0, ... delays=0/81339/0/19, dsn=2.0.0, ... delays=0.42/81342/0/14, dsn=2.0.0, ... delays=0.19/81344/0/11, dsn=2.0.0, ... delays=0/81341/0/15, dsn=2.0.0, ... delays=0/81337/0/21, dsn=2.0.0, ... delays=0.57/81343/0/11, dsn=2.0.0, ... delays=0.32/81342/0/16, dsn=2.0.0, ... delays=0.21/81341/0/18, dsn=2.0.0, ... delays=0.35/81348/0/7, dsn=2.0.0, ... delays=0.12/81339/0/21, dsn=2.0.0, ... delays=0.09/81340/0/20, dsn=2.0.0, ... Now the question is why did /usr/bin/db_capture take so long? I am not sure that my red and green options are going to solve your performance problem. If you still have the logs, it will be worthwhile to find out when the delivery delay started climbing through the roof. ? ? ? ?Wietse
Re: Speeding up Local Delivery
Wendigo Thompson: Okay, I have some updates. Reducing the concurrency down (and ultimately to 1) increased performance: it seems the delays I was seeing were related to concurrency inside SQL. However, I am still seeing half second delays in local delivery while my insertion application is showing ~.003s processing times. I was able to verify that it's not my insertion application by making a simple script that soft-bounces mail (exit code 75) and does nothing else -- there's still a half a second delay on action. Tracing the individual processes is proving to be a challenge as this is an Apple system and dtruss can only be run as root. Any tips? It is perhaps good to be aware that my support is limited to Postfix. It ends once Postfix hands off the message to another program, be it over a network or on the same machine. You need to determine why /usr/bin/db_capture is taking a long time. Run /usr/bin/db_capture by hand, with the same command-line arguments as when it is invoked by Postfix, and with a prepared email message from file. /usr/bin/db_capture arguments prepared-message-file Where the prepared-message-file has the form that Postfix emits: From u...@example.com Fri Jan 1 05:12:42 2010 From: u...@example.com To: some...@somewhere Subject: blah blah More blah. More blah. More blah. You can use any debug tool that you like. Good luck. Wietse
Re: Postfix, mailman and procmail integration
Taylor, Marc: Good morning, I am new to this list and I am not even sure if this is the place to start, but here goes. I have an Ubuntu 8.0.4 LTS server with postfix 2.5.1-2ubuntu1.2, mailman2.1.9-9ubuntu1 and procmail3.22-16ubuntu3. I have followed the instructions in the Ubuntu community pages regarding configuring postfix, mailman and procmail to work together and I can't seem to get mailman to pay attention to the procmail directives in /etc/procmailrcs/. Is there a trick to this? Anyone else have such a setup working? Hmm... Would you be so kind to turn on your telepathic sensors and guess where I may have made a mistake while I tried to follow instructions. Please see the mailing list welcome message for guidance (repeated below). Wietse TO REPORT A PROBLEM see http://www.postfix.org/DEBUG_README.html#mail TO (UN)SUBSCRIBE see http://www.postfix.org/lists.html Thank you for using Postfix.
Re: LDAP BCC Rules
Joe Postfix: Hi, we're trying to setup our Postfix relays to BCC emails to/from specific users (members of an LDAP group - A/D actually) to a mailbox that logs their correspondence. I wasnt able to find any mention of this in the online documentation anywhere - does anyone know of a way to implement something like this? Adding recipients: http://www.postfix.org/postconf.5.html#always_bcc http://www.postfix.org/postconf.5.html#sender_bcc_maps http://www.postfix.org/postconf.5.html#recipient_bcc_maps http://www.postfix.org/postconf.5.html#virtual_alias_maps LDAP lookups: http://www.postfix.org/ldap_table.html Wietse We have this setup currently so that the backend M$ Exchange does this for us, but would love to offload this functionality from there. Thanks
Re: How to not reject valid MTAs for inconsistent forward/reverse DNS.
Frank Cusack: until a name lookup has been done. But if that name lookup takes a very long time, along with the connect postfix should log how long ago the actual connect was. The SMTP server can find out long the name/address lookup took. It does not juggle TCP packets. The sysadmin should have done those name/address lookups by hand as part of due diligence, and would have observed the exact same delays that Postfix experienced. 2) There is something going on where postfix wasn't logging connects from my test host, which sent FIN instead of the actual problem host which was sending RST. My guess would be that postfix is blocking on getnameinfo() so I don't see how the status of the client connection even comes into play, but it is curious. No is the packet capture, and no Postfix logging. No support. Wietse
Re: Convert _ to + on inbound addresses
Charles Boling: I've been a postfix user for nearly 10 years, but in many ways I'm still quite the newbie. That's the problem with Postfix being such a solid MTA: I don't mess with it much. :-) For a good chunk of that time, I have wrestled on and off with this problem off and on, playing with it for a bit, giving up, then returning to it a year or two later. After all this time, I've finally gotten smart enough to specifically ask for help! I love + addressing, the ability to append ad-hoc suffixes to an email address (like the one used to post to this list). The problem is, a lot of web sites don't. They tend to take the lazy route to preventing SQL injection attacks, and reject any address with a + in it. To address that issue, I would like to be able to use another character (_ or .) that is commonly accepted as part of email addresses, instead. Address transformation mappings are always queried at recipient validation time, so you can't use a wild-card pattern to replace _ by + without becoming a backscatter source. The only way to change an envelope address before SMTP server recipient validation is with the Postfix 2.7 smtpd_command_filter which replaces SMTP commands before they are executed. This is for the desperate. It has no safety mechanisms. http://www.postfix.org/postconf.5.html#smtpd_command_filter Wietse When an email comes into Postfix (v2.2 currently), I would like it to search for, e.g. _ in the recipient address, and replace it with +. This, of course, needs to happen before it does recipient checking. The validated message is eventually passed on to the MDA (Cyrus, in my case) with the + in it. How might be the best way to accomplish this?
Postfix sender reputation support in snapshot 20100117
postfix-2.7-20100117 changes the meaning of content filters of the form transport: (note: no next-hop destination) so that this form can be used to implement sender reputation schemes. Typically, mail is split into classes, and all mail in class X is sent out from an SMTP client IP address that is reserved for class X. According to discussions on the Postfix list there is a legitimate need for such functionality. Until now this requires one Postfix instance per source IP address. The change introduces one minor incompatibility. As always, there is a compatibility option to restore the old behavior. This option is not needed with SMTP-based content filters, because these always have an explicit next-hop destination. With pipe-based content filters that specify no next-hop destination, the compatibility option restores the order of deliveries when a filter is congested (it restores from per-destination round-robin selection back to approximate FIFO selection). More details in the release notes quotes below. Wietse Major changes with snapshot 20100117 The FILTER action in access maps or header/body_checks now supports sender reputation schemes that dynamically choose the SMTP source IP address. This is implemented by specifying FILTER actions with empty next-hop destinations in access maps or header/body_checks, and by configuring in master.cf one Postfix SMTP client for each SMTP source IP address, where each client has its own -o myhostname and -o smtp_bind_address settings. Incompatibility with snapshot 20100117 == The meaning of an empty content filter next-hop destination has changed. Postfix now uses the recipient domain, instead of using $myhostname as in Postfix 2.6 and earlier. To get the old behavior use default_filter_nexthop = $myhostname, or specify a non-empty next-hop content filter destination.
Re: Postfix sender reputation support in snapshot 20100117
Stefan Foerster: * Wietse Venema wie...@porcupine.org: This is implemented by specifying FILTER actions with empty next-hop destinations in access maps or header/body_checks, and by configuring in master.cf one Postfix SMTP client for each SMTP source IP address, where each client has its own -o myhostname and -o smtp_bind_address settings. To be honest, I don't see what exactly has changed here. Manual pages from a stock Postfix 2.6.5: ,[ man 5 access ] | FILTER transport:destination |After the message is queued, send the entire message through the |specified external content filter. The transport:destination |syntax is described in the transport(5) manual page. I'll remove that reference. Wietse ,[ man 5 transport ] | A non-null transport field with a null nexthop field resets the nexthop | information to the recipient domain. ` If my question is answered in other parts of the documentation, I apologize in advance. Stefan
Re: Successful Delivery Notification
Daniel L. Miller: Other than scanning the logfiles, is there a way a service can receive notification of a successful delivery to a remote site? In other words, a trusted client submits mail for a remote site, Postfix connects and receives acknowledgement from the remote site, and then notifies a local process of the result? Postfix 2.3 and later implement RFC 3461 (SMTP DSN Extension). This allows SMTP clients to specify how the sender wants to be informed of successful, delayed or failed deliveries. Wietse
Re: Postfix sender reputation support in snapshot 20100117
Wietse Venema: Stefan Foerster: * Wietse Venema wie...@porcupine.org: This is implemented by specifying FILTER actions with empty next-hop destinations in access maps or header/body_checks, and by configuring in master.cf one Postfix SMTP client for each SMTP source IP address, where each client has its own -o myhostname and -o smtp_bind_address settings. To be honest, I don't see what exactly has changed here. Manual pages from a stock Postfix 2.6.5: ,[ man 5 access ] | FILTER transport:destination |After the message is queued, send the entire message through the |specified external content filter. The transport:destination |syntax is described in the transport(5) manual page. Perhaps you should look at the new manpage! Wietse FILTER transport:destination After the message is queued, send the entire message through the specified external content filter. The transport name specifies the first field of a mail delivery agent definition in mas- ter.cf; the syntax of the next-hop destination is described in the manual page of the corresponding delivery agent. More information about external content filters is in the Postfix FILTER_README file.
Re: The method behind the madness
Victor Duchovni: On Mon, Jan 18, 2010 at 07:01:45PM +0200, Henrik K wrote: I think I prefer a separate daemon that tails postfix log and greps all to=xxx, relay=xxx info and passes it to the policy daemon. That way the policy daemon doesn't need to have a big DNS mess to resolve all the recipient MX ips. MX IPs have nothing to do with it. A sender's sending IP often bears little relation to the IP where mail for the same address is delivered. If you whitelist an outside sender address for a given internal recipient (original sender), no IP or DNS information is appropriate or required. Note, that Victor is talking about sender-receiver PAIRS. It would be a bad idea to accept all mail that pretends to be from a whitelisted sender address. If you don't whitelist sender-receiver PAIRS, then you need to restrict the network blocks that may use a whitelisted sender address, and guessing the /24 (or even /16) is a start. Wietse
Re: The method behind the madness
Daniel L. Miller: Henrik K wrote: On Sun, Jan 17, 2010 at 10:44:48PM -0800, Daniel L. Miller wrote: Daily scanning of logfiles does not accomplish this. Nor would even an hourly scan - and constant logfile scanning strikes me as inelegant. If there is any method currently existing within Postfix to accomplish this goal please point me at it! If not, I'd like to discuss possible means and methods for a future implementation. I've been meaning to make something like this for postfix, your post refreshed me. Amavisd-new penpals is ok (and you can make other custom queries to it's database), but it can't make MTA bypass all the ptr/helo/foo checks. My main problem is all bad SOHO servers etc that get blocked by those. It can be achieved with a simple policy daemon that records all outgoing recipients and/or relays and compares them for incoming messages, possibly adding a header that you can also reuse in SA for scoring. I created one - the problem is execution. I took the greylist.pl and tweaked it - and it now adds recipient addresses to a database very simply. But I only want that to happen with valid senders/clients. Placing this policy daemon, which always returns dunno, in the smtpd_recipient_restrictions very simply generates a list of recipients - but without any validation. And duplicating validation within this daemon strikes me as wasteful. Ideally, I'd like this daemon called immediately upon successful delivery, or at what Postfix determines as successful delivery based on the remote server response. Without tailing the logfile I don't see how else to accomplish it - but I don't LIKE that form of implementation. Might be the only way to do it at this time - but perhaps having such status messages available might be considered sufficiently worthwhile to implement a new feature? I'm not sure you are solving the right problem. What is the likelihood that a local user specifies an invalid remote address, and how problematic is it for your to have a few invalid remote addresses on your whitelist? If you use a btree or hash database, they won't slow down the database searches. Valid addresses don't live forever, either. After some time you will have to expire unused entries from your database anyway, so you might just as well live with a minor degree of pollution from the occasional invalid remote address. Wietse
Whitelisting made easy (was: The method behind the madness)
The following solution solves 99% of the problem: - IF mail is from a local (or authenticated) client - AND the sender has already passed reject_unlisted_sender - THEN store the (sender, recipient) pair in a whitelist. This can be done with trivial modification of an existing greylisting policy daemon. Occasionally, a sender or recipient address will become invalid, or a user mis-types. To clean out junk, maintain a last use time stamp for each (sender, recipient) pair, and periodically remove entries that are too old. Or just rename the database late Saturday night and let it re-populate over time. Wietse
Re: The method behind the madness
Stan Hoeppner: [ Charset ISO-8859-1 unsupported, converting... ] Mark Nernberg (gmail account) put forth on 1/18/2010 4:50 PM: On Jan 18, 2010, at 17:48, Stan Hoeppner s...@hardwarefreak.com wrote: Mark Nernberg (gmail account) put forth on 1/18/2010 4:17 PM: I have achieved this with a slightly hacked TMDA (www.tmda.net). if you want my modifications, contact me off-list. I'm surprised you actually mentioned a solution whose core feature is challenge/response. C/R is one of those cures that far is worse than the disease. -- Stan I got rid of the C/R ... I wanted auto whitelisting some of the other features. hence my modifications. From: http://tmda.sourceforge.net/cgi-bin/moin.cgi/TmdaFaq#head-0b1aee3c2decf32a26ffcc12b397f9d3cec3fdc0 How do I setup an auto-whitelist? I'm sure that it is possible to MODIFY the TMDA source code such that it will handle auto-whitelisting without challenge-response. Conceptually, it's as simple as 1) never sending the challenge and 2) skipping the was this challenge confirmed check. After that change, there is a lot of code that can be deleted because it is no longer used. Given the choice between starting from scratch, and starting with code that already works, the choice should be clear. The only reason to write Postfix in the first place was that I could not live with the alternatives. Wietse
Re: My postfix server sometimes send command less than 4 alphabets
Arora, Sumit: Hi, I'm stuck into a problem. I'm using content filter, which parses email from my postfix server. My postfix server sometimes sends a command which is less than 4 alphabets. I don't know what to do for that command, as I don't know which command is that... Can anybody tell me, is there any command of less than 4 alphabets, postfix sends Please show evidence, and please describe how that evidence was obtained. Wietse
Re: Changes in PCRE handling postfix etch vs lenny?
Harakiri: Hi, after upgrading a machine from etch to lenny i get the following warning postfix/smtpd[23231]: warning: pcre map /etc/postfix/postfix_rbl_check, line 0: ignoring unrecognized request main.cf: check_client_access pcre:/etc/postfix/postfix_rbl_check file: reject_rbl_client zen.spamhaus.org That is not a valid PCRE file entry, and it has never been valid. Postfix promises compatibility only for behavior that is promised by documentation. Postfix behavior for invalid inputs is subject to change without prior warning. postfix/tlsmgr[23233]: warning: request to update table btree:/var/spool/postfix/smtpd_scache in non-postfix directory /var/spool/postfix See the RELEASE_NOTES file. Postfix documentation is created with a great deal of effort. Don't let it go to waste. Wietse
Re: Error: timeout exceeded (in reply to end of DATA command)
Martijn de Munnik: Hi list, I have a problem with delivering mail to a host and get this error: host mx2.amsterdam.nl[145.222.14.10] said: 421 enepmx02.amsterdam.nl Error: timeout exceeded (in reply to end of DATA command) This error only seems to occur with 'large' mails. Currently I have a mail of ~600KB and ~8MB stuck in the queue. I don't think this is a postfix issue on our site but an issue with the mailserver on the other site. What can cause such issues? Record a tcpdump trace. The way the session fails will indicate the kind of problem (MTU, Window scaling, and so on). http://www.postfix.org/DEBUG_README.html Wietse
Re: Custom 550 messages for custom rejections reasons.
Jaroslaw Grzabel: Ralf Hildebrandt wrote: Use an access(5) map on the client: check_client_access hash:/etc/postfix/nice_reject with: unknown 550 5.1.2 Your reverse DNS entries are off Thank you for that Ralf. Do you know if may I operate on any variables ? I need to have a message like 550 Fix your reverse DNS for IP $IP_ADDRESS or use your ISP server. Please see: website to know more. Syntax of access tables is documented: man 5 access. Wietse
Re: Custom 550 messages for custom rejections reasons.
Jaroslaw Grzabel: Wietse Venema wrote: Syntax of access tables is documented: man 5 access. Wietse Yes I know. But as I mention in the post sent a moment ago, the problem is that I'm afraid I will reject unknown hosts with valid PTR. I don't want to do this. See man 5 postconf for the difference between: reject_unknown_reverse_client_hostname reject_unknown_client_hostname Wietse
Re: Custom 550 messages for custom rejections reasons.
Jaroslaw Grzabel: But as I mention in the post sent a moment ago, the problem is that I'm afraid I will reject unknown hosts with valid PTR. I don't want to do this. Wietse Venema wrote: See man 5 postconf for the difference between: reject_unknown_reverse_client_hostname reject_unknown_client_hostname Jaroslaw Grzabel: Sorry Wietse, but you completely lost me. How does it regard to my problem ? You wrote, in the quote above: I'm afraid I will reject unknown hosts with valid PTR. I don't want to do this. And I was responding to that concern. If you are concerned that you reject mail from hosts with valid PTR, then I recommend that you choose between: 1) reject_unknown_reverse_client_hostname. As documented, this will accept any reverse name, without checking whether it resolves to to the client IP address. and 2) reject_unknown_client_hostname. As documented, this requires that the reverse hostname resolves to the client IP address. Wietse
Re: Custom 550 messages for custom rejections reasons.
Jaroslaw Grzabel: I'm afraid I will reject unknown hosts with valid PTR. I don't want to do this. Wietse Venema: If you are concerned that you reject mail from hosts with valid PTR, then I recommend that you choose between: 1) reject_unknown_reverse_client_hostname. As documented, this will accept any reverse name, without checking whether it resolves to to the client IP address. and 2) reject_unknown_client_hostname. As documented, this requires that the reverse hostname resolves to the client IP address. Jaroslaw Grzabel: I understand (or I only think so) what you're saying. But it's not completely related with my problem (or I only think so). I need to reject them with my own custom error message. That's why I'm asking how to do it as I'm afraid that if I will put restriction like: Postfix replies with: 450 4.7.1 Client host rejected: cannot find your hostname, [1.2.3.4] for a temporary error, and 550 5.7.1 Client host rejected: cannot find your hostname, [1.2.3.4] for a permanent error. With the access map you can't choose between 550 and 450. Replying with 550 for a temporary error causes loss of legitimate mail. Wietse
Re: Error: timeout exceeded (in reply to end of DATA command)
Martijn de Munnik: On Wed, 20 Jan 2010 07:20:01 -0500 (EST), wie...@porcupine.org (Wietse Venema) wrote: Martijn de Munnik: Hi list, I have a problem with delivering mail to a host and get this error: host mx2.amsterdam.nl[145.222.14.10] said: 421 enepmx02.amsterdam.nl Error: timeout exceeded (in reply to end of DATA command) This error only seems to occur with 'large' mails. Currently I have a mail of ~600KB and ~8MB stuck in the queue. I don't think this is a postfix issue on our site but an issue with the mailserver on the other site. What can cause such issues? Record a tcpdump trace. The way the session fails will indicate the kind of problem (MTU, Window scaling, and so on). http://www.postfix.org/DEBUG_README.html Wietse Ok, I tried that and I'm not really sure where to look for. I opened the tcpdump file in wireshark and there are a lot of warnings and notes in the file. -- Notes: Duplicate ACK(#1) [145.222.14.10 - 213.207.90.2] Duplicate ACK(#2) [145.222.14.10 - 213.207.90.2] Duplicate ACK(#3) [145.222.14.10 - 213.207.90.2] Duplicate ACK(#4) [145.222.14.10 - 213.207.90.2] . . . Duplicate ACK(#44) [145.222.14.10 - 213.207.90.2] Retransmission (suspected) [213.207.90.2 - 145.222.14.10] Warnings: Fast retransmission (suspected) [213.207.90.2 - 145.222.14.10] Out-Of-Order segment [213.207.90.2 - 145.222.14.10] -- This is abracadabra for me ;) If you can make the tcpdump -nr /file/name output available then people who understand TCP/IP can look at it. Wietse
Re: Exclude recipients from pre-queue spamass-milter checks?
Wolfgang Zeikat: We are experimenting with spamass-milter to check mails and reject them if a configured spamassassin score is reached. That part works, but the milter is (of course) applied to all mails after our smtpd_recipient_restrictions lookups return OK for the recipient, i.e. also postmaster@various domains for whom we want to accept every mail. Is it possible to exclude mails from smtpd_milters = unix:/var/run/spamass.sock? There is no such option. Wietse
Re: Error: timeout exceeded (in reply to end of DATA command)
Here's the TCP initial handshake: 17:30:44.951789 IP 213.207.90.2.48147 145.222.14.10.25: S 50514820:50514820(0) win 49640 mss 1460,nop,wscale 0,nop,nop,sackOK 17:30:44.954496 IP 145.222.14.10.25 213.207.90.2.48147: S 4148480248:4148480248(0) ack 50514821 win 5840 mss 1380,nop,wscale 2 17:30:44.954519 IP 213.207.90.2.48147 145.222.14.10.25: . ack 1 win 49680 Later, as the receiver processes the network packets, it acknowledges the data received, sends its receive window size (how much more it is willing to receive). Above, with wscale 2 the server at 145.222.14.10 announces that its TCP receive window value needs to be multiplied by a factor of 4 (binary number shifted left by 2). But, there is a broken router in the path that does not understand window scaling. Here is an example of what gets f-ed up: 17:30:45.41 IP 213.207.90.2.48147 145.222.14.10.25: . 20853:22233(1380) ack 137 win 49680 17:30:45.412230 IP 213.207.90.2.48147 145.222.14.10.25: . 22233:23613(1380) ack 137 win 49680 17:30:45.412249 IP 213.207.90.2.48147 145.222.14.10.25: P 23613:24993(1380) ack 137 win 49680 17:30:45.412747 IP 145.222.14.10.25 213.207.90.2.48147: P ack 8433 win 5800 17:30:45.412748 IP 145.222.14.10.25 213.207.90.2.48147: P ack 8433 win 5800 17:30:45.412749 IP 145.222.14.10.25 213.207.90.2.48147: P ack 8433 win 5800 The receiver says they can receive bytes 8433-31633, but the broken router does not know that 5800 needs to be multiplied by 4, and it thinks the receiver can receive only bytes 8433-14233. The broken router then throws away the bytes with higher sequence numbers than 14233. Workaround: turn off window scaling support on the sender's kernel. Wietse
Re: Whitelisting made easy
Daniel L. Miller: Wietse Venema wrote: The following solution solves 99% of the problem: - IF mail is from a local (or authenticated) client That's the magic part right there. How do I accomplish this? The client IP address passed along in the policy protocol. - AND the sender has already passed reject_unlisted_sender That's by having reject_unlisted_sender before check_policy_service. Wietse - THEN store the (sender, recipient) pair in a whitelist. This can be done with trivial modification of an existing greylisting policy daemon. Absolutely - already done. Just need that test. -- Daniel
Re: Exclude recipients from pre-queue spamass-milter checks?
Wolfgang Zeikat: Wietse Venema wrote: Is it possible to exclude mails from smtpd_milters = unix:/var/run/spamass.sock? There is no such option. OK. Thank you for the bad news ;) It is not a good idea to simply turn off Milters in the middle of an SMTP session, because that would whitelist all mail that is sent later in that same SMTP session. To fix that one would have to add code to turn Milters back on in the middle of a session. That code is hard to maintain and because almost never executes and therefore it will be buggy, suffer from bitrot and other badness. Would we have that option if we use an smtpd_proxy_filter, i.e. spampd? Postfix does not know where the smtpd_proxy_filter will deliver its output, so it cannot jump over the filter. Again, one can keep piling junk on top of Postfix for rare corner cases, or one can spend the cycles on something that benefits many people. For good reasons, many filters have their own whitelisting features. Wietse
Re: Error: timeout exceeded (in reply to end of DATA command)
Martijn de Munnik: On Jan 20, 2010, at 9:28 PM, Victor Duchovni wrote: On Wed, Jan 20, 2010 at 03:22:56PM -0500, Wietse Venema wrote: The broken router then throws away the bytes with higher sequence numbers than 14233. Workaround: turn off window scaling support on the sender's kernel. This problem is sufficiently common, that on Linux MTAs I always add: net.ipv4.tcp_window_scaling = 0 I'm running Solaris 10 x86 and I did sudo ndd -set /dev/tcp tcp_wscale_always 0 before I did this the value was 1. After I did this I flushed the queue but the messages stay stuck in the queue with the same problem. I'm not sure this is the right kernel parameter for Solaris? You can do ndd /dev/tcp \? to find out what parameters are supported. On my Solaris9 and Solaris10 test boxes it is called tcp_wscale_always. According to Solaris10 documentation: When this parameter is enabled, which is the default setting [since Solaris10], TCP always sends a SYN segment with the window scale option, even if the window scale option value is 0. So, please make another recording and report if setting this to 0 causes Solaris to stop sending wscale 0 options. If the sender does not send a wscale option, then the receiver is not supposed to send a wscale option, either. Your tcpdump recording should confirm that. /etc/system has no specific setting for tcp, so everything is default Solaris 10. Wietse, the broken router you mentioned, could that be a Cisco PIX on the receivers site? I maintain no list of CISCO PIX firewall bugs. It can also be a router at your end. Never exclude that possibility. Jan 20 22:58:43 stevie.youngguns.nl postfix/smtp[18765]: [ID 197553 mail.info] 8A5553BA0C: enabling PIX workarounds: disable_esmtp delay_dotcrlf for mx2.amsterdam.nl[145.222.14.10]:25 That's a PIX bug workaround for the fact that their engone parses only one packet at a time and therefore misses SMTP commands and CRLF.CRLF when these sit across packet boundaries. Wietse
Re: Error: timeout exceeded (in reply to end of DATA command)
Wietse Venema: You can do ndd /dev/tcp \? to find out what parameters are supported. On my Solaris9 and Solaris10 test boxes it is called tcp_wscale_always. According to Solaris10 documentation: When this parameter is enabled, which is the default setting [since Solaris10], TCP always sends a SYN segment with the window scale option, even if the window scale option value is 0. With the default tcp_wscale_always setting, making a connection from a Solaris 10 box to FreeBSD 8.0: 20:13:59.808828 IP 168.100.189.17.32799 168.100.189.10.25: Flags [S], seq 11835, win 49640, options [mss 1460,nop,wscale 0,nop,nop,sackOK], length 0 20:13:59.808892 IP 168.100.189.10.25 168.100.189.17.32799: Flags [S.], seq 538094055, ack 11836, win 65535, options [mss 1460,nop,wscale 3,sackOK,eol], length 0 20:13:59.809327 IP 168.100.189.17.32799 168.100.189.10.25: Flags [.], ack 1, win 49640, length 0 Same system with tcp_wscale_always set to zero: 20:14:52.736959 IP 168.100.189.17.32800 168.100.189.10.25: Flags [S], seq 131413865, win 49640, options [mss 1460,nop,nop,sackOK], length 0 20:14:52.737016 IP 168.100.189.10.25 168.100.189.17.32800: Flags [S.], seq 3072042607, ack 131413866, win 65535, options [mss 1460,sackOK,eol], length 0 20:14:52.737581 IP 168.100.189.17.32800 168.100.189.10.25: Flags [.], ack 1, win 49640, length 0 Thus, Solaris 10 does not send wscale, and neither should the remote server. If this does not make your mail move, then you need to collect another tcpdump recording. In that case mail was not moving because of multiple problems. Wietse
Re: Error: timeout exceeded (in reply to end of DATA command)
It's nice for a change to work with someone who provides actual information, instead of that dork from last week who was just venting his opinions and not giving people a chance to help. In this case it looks like a Solaris TCP bug when sending data over a connection that suffers from heavy packet loss. Apparently, Solaris 10 stops retransmitting, and therefore the other side times out. I'm not wrapping the packets this time, so you will have to widen your window to read this message comfortably. Wietse The three-way TCP handshake show no wscale options, so we don't have to worry about borked firewalls mis-handling this: 09:16:40.786945 IP 213.207.90.2.59301 145.222.14.10.25: S 2466228028:2466228028(0) win 49640 mss 1460,nop,nop,sackOK 09:16:40.789806 IP 145.222.14.10.25 213.207.90.2.59301: S 3886146351:3886146351(0) ack 2466228029 win 5840 mss 1380 09:16:40.789829 IP 213.207.90.2.59301 145.222.14.10.25: . ack 1 win 49680 The handshake shows a round-trip time of 2.9ms. The receiver's MTU is smaller than 1460, which suggests that his packets are encapsulated in some other protocol. It does not matter for the problem at hand. Next is a segment from the middle of transmission. The sender has sent a full window up to byte 22233, but apparently there was major packet loss after byte 5673. 09:16:51.869847 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:51.869861 IP 213.207.90.2.59301 145.222.14.10.25: P 20853:22233(1380) ack 137 win 49680 09:16:51.869874 IP 213.207.90.2.59301 145.222.14.10.25: . 5673:7053(1380) ack 137 win 49680 09:16:51.869976 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:51.869977 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:51.869978 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:51.870110 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:51.870111 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:51.870376 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:56.216513 IP 213.207.90.2.59301 145.222.14.10.25: . 5673:7053(1380) ack 137 win 49680 09:16:56.260383 IP 145.222.14.10.25 213.207.90.2.59301: . ack 7053 win 19320 09:16:56.260414 IP 213.207.90.2.59301 145.222.14.10.25: . 7053:8433(1380) ack 137 win 49680 09:16:56.260421 IP 213.207.90.2.59301 145.222.14.10.25: . 8433:9813(1380) ack 137 win 49680 09:16:56.263577 IP 145.222.14.10.25 213.207.90.2.59301: . ack 8433 win 22080 09:16:56.263588 IP 213.207.90.2.59301 145.222.14.10.25: . 9813:11193(1380) ack 137 win 49680 09:16:56.263610 IP 213.207.90.2.59301 145.222.14.10.25: . 11193:12573(1380) ack 137 win 49680 09:16:56.263844 IP 145.222.14.10.25 213.207.90.2.59301: . ack 9813 win 24840 09:16:56.263855 IP 213.207.90.2.59301 145.222.14.10.25: . 12573:13953(1380) ack 137 win 49680 09:16:56.263865 IP 213.207.90.2.59301 145.222.14.10.25: . 13953:15333(1380) ack 137 win 49680 09:16:56.266641 IP 145.222.14.10.25 213.207.90.2.59301: P ack 9813 win 24840 09:16:56.266776 IP 145.222.14.10.25 213.207.90.2.59301: . ack 11193 win 28980 09:16:56.266777 IP 145.222.14.10.25 213.207.90.2.59301: P ack 11193 win 28980 09:16:56.266800 IP 213.207.90.2.59301 145.222.14.10.25: . 15333:16713(1380) ack 137 win 49680 Sequences like this repeat through the entire session. This network connection is so bad that your machine can send only 107656 bytes in 85 seconds or 1.27 kbyte/s. I notice there are many sequences like this: 09:16:51.869844 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:51.869845 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:51.869846 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:51.869847 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 and this: 09:16:51.869976 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:51.869977 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 09:16:51.869978 IP 145.222.14.10.25 213.207.90.2.59301: P ack 5673 win 16560 and this: 09:17:20.597737 IP 145.222.14.10.25 213.207.90.2.59301: P ack 40173 win 32767 09:17:20.597738 IP 145.222.14.10.25 213.207.90.2.59301: P ack 40173 win 32767 09:17:20.597739 IP 145.222.14.10.25 213.207.90.2.59301: P ack 40173 win 32767 And many more. This is consistent with network congestion. These identical ACKs would have been sent several ms separated from each other (remember, the round-trip time is about 3ms). Apparently the ACKs were queued up behind other data that was clogging some pipe somewere, and so they ended up traveling together. Network congestion is also consistent with increased packet loss. And now for the surprise: after 09:18:16.515592 the sender stops retransmitting. That would be a sender TCP bug. The first thing that happens next is 30s later, at 09:18:46.512567, when the server reports a 421 timeout error and closes the connection. 09:18:16.512400 IP 213.207.90.2.59301
Re: Problem with mail transport unavailable
RaSca: http://www.postfix.org/DEBUG_README.html#logging Look for obvious signs of trouble Hi, I haven't found any message of this kind concerning the message group. I suppose you overlooked this warning then. Jan 21 15:06:06 mail-2 postfix/qmgr[31542]: warning: connect to transport scan: Connection refused Wietse
Re: Problem with mail transport unavailable
RaSca: Il giorno Gio 21 Gen 2010 16:33:23 CET, wie...@porcupine.org (Wietse Venema) ha scritto: RaSca: http://www.postfix.org/DEBUG_README.html#logging Look for obvious signs of trouble Hi, I haven't found any message of this kind concerning the message group. I suppose you overlooked this warning then. Jan 21 15:06:06 mail-2 postfix/qmgr[31542]: warning: connect to transport scan: Connection refused Wietse The message was not related to the ID of the message, so with grep was impossible to see. I will look better in the future. I asked you to egrep '(fatal|panic|error|etc)' but you thought you knew better and therefore lost time. Wietse
Re: Change behavior of return code
Victor Duchovni: On Thu, Jan 21, 2010 at 02:57:17PM +0100, Mickael CANEVET wrote: Hi, I'd like postfix to treat EX_CANTCREAT (73) as temporary failure. I use this command to deliver my mails: mailbox_command = /usr/bin/formail -D 8192 ~/.msgid.cache -s /usr/libexec/dovecot/deliver The problem is that if the filesystem containing home directories is not mounted, formail returns an EX_CANTCREAT error, which is a permanent exception for postfix (by default). Is it possible to tell postfix to not reject permanently this kind of error ? /etc/postfix/main.cf: require_home_directory = yes Wietse
Re: mail for mx2.youngguns.nl loops back to myself
Martijn de Munnik: Jan 21 17:02:30 marcus postfix/qmgr[16421]: 523FD1C11A: from=mart...@youngguns.nl, size=650750, nrcpt=1 (queue active) Jan 21 17:02:30 marcus postfix/smtp[16449]: 523FD1C11A: host mx-cluster1.one.com[91.198.169.10] said: 450 4.7.1 r...@musicscool.nl: Recipient address rejected: Greylisted for 5 minutes (in reply to RCPT TO command) Jan 21 17:02:30 marcus postfix/smtp[16449]: 523FD1C11A: to=r...@musicscool.nl, relay=mx-cluster2.one.com[91.198.169.11]:25, delay=32, delays=32/0.01/0.57/0.13, dsn=4.7.1, status=deferred (host mx-cluster2.one.com[91.198.169.11] said: 450 4.7.1 r...@musicscool.nl: Recipient address rejected: Greylisted for 5 minutes (in reply to RCPT TO command)) Above, musicscool.nl is delivered directly to its MX hosts: musicscool.nl mail is handled by 10 mx-cluster1.one.com. musicscool.nl mail is handled by 10 mx-cluster2.one.com. Jan 21 17:23:02 marcus postfix/qmgr[16900]: 523FD1C11A: from=mart...@youngguns.nl, size=650750, nrcpt=1 (queue active) Jan 21 17:23:02 marcus postfix/smtp[17064]: 523FD1C11A: to=r...@musicscool.nl, relay=none, delay=1264, delays=1264/0.01/0/0, dsn=5.4.6, status=bounced (mail for mx2.youngguns.nl loops back to myself) Jan 21 17:23:02 marcus postfix/bounce[17065]: 523FD1C11A: sender non-delivery notification: B15A81C76E Jan 21 17:23:02 marcus postfix/qmgr[16900]: 523FD1C11A: removed Above, the queue manager was restarted (pid changes from 16421 to 16900), presumably because some Postfix setting was changed. Now, musicscool.nl is NOT delivered directly to its MX hosts. Try undoing the change in Postfix setting. Wietse
Re: Email address with leading whitespace rejected
Doug Robbins: Messages containing leading whitespace in the recipient address are rejected. Only if the recipient does not exist. Example: Jan 22 08:32:41 vps10 postfix/smtpd[5937]: NOQUEUE: reject: RCPT from smtpout.eastlink.ca[24.222.0.30]: 550 5.1.1 soli...@example.com: Recipient address rejected: User unknown in virtual alias table; from=dba...@example2.com to= soli...@example.com proto=ESMTP helo=mta03.eastlink.ca soli...@example.com is a legitimate and functioning mail account. But the client sent soli...@example.com including the quotes. In SMTP, quotes are necessary when a string contains special characters. Postfix does not attempt to typo-correct quoted strings. Is there something I can do to avoid these rejections (other than the obvious -- get dba...@example2.com to fix his address book)? The most practical solution is to educate the user (it might take a long time to get the client software fixed so it better handles stupid data-entry errors like this one). You can use quoted strings in hash/btree/cdb/dbm aliases(5) maps soliver: soliver I don't think that Postfix supports quoted strings in any form of virtual aliases or in canonical maps, not even with *SQL or LDAP. Finally, Postfix 2.7 can compensate for almost all forms of SMTP client-side brain damage with the smtpd_command_filter feature. /etc/postfix/main.cf: smtpd_command_filter = pcre:/etc/postfix/braindead.pcre /etc/postfix/braindead.pcre /^(MAIL FROM:)\s+(.+)/$1$2 There's a similar feature for fixing remote SMTP server replies. Wietse
Re: SOLVED: rbl check being skipped - Postfix logs no error on NXDOMAIN, does on SERVFAIL
Stan Hoeppner: 1. Spamhaus has banned Google Public DNS resolver queries. I didn't know this until today. If Postfix is using Google Public DNS resolvers, rbl queries to zen.spamhaus.org fail but Postfix (Debian Lenny 2.5.5-1.1) logs NOTHING about it. Not the query attempt, not the failure, zilch, nut'n. This explains why I The query returns NXDOMAIN. No-one has asked me to log all the NXDOMAIN results for DNSBL queries. Wietse With query through Google DNS the host is not listed in zen.spamhaus.org: % dig @8.8.8.8 a 105.49.136.89.zen.spamhaus.org ; DiG 9.6.1-P1 @8.8.8.8 a 105.49.136.89.zen.spamhaus.org ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NXDOMAIN, id: 50578 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;105.49.136.89.zen.spamhaus.org.IN A ;; AUTHORITY SECTION: zen.spamhaus.org. 150 IN SOA need.to.know.only. hostmaster.spamhaus.org. 1001221345 3600 600 432000 150 ;; Query time: 169 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri Jan 22 08:48:32 2010 ;; MSG SIZE rcvd: 112 With direct query, the host is listed as you can see for yourself.
Re: Email address with leading whitespace rejected
Victor Duchovni: On Fri, Jan 22, 2010 at 02:13:17PM +0100, Ralf Hildebrandt wrote: Is there something I can do to avoid these rejections (other than the obvious -- get dba...@example2.com to fix his address book)? Hm, you could try and alias soli...@example.com to soli...@example.com But how??? The lookup keys and RHS values for virtual(5) are in rfc822 format. A PCRE table can take care of this: /^ soliver@example\.com$/ soli...@example.com This said, far better to just reject this, and let the sender correct their address list. Virtual alias lookups are done in the unquoted form, while canonical map lookups are in quoted form. So, the above form is good for canonical mapping, but virtual alias mapping would require that the quotes be stripped: /^ soli...@example\.com$/ soli...@example.com (Normally, Postfix transforms addresses to unquoted form because the RFC supports multiple ways to quote the same string. Perhaps to make canonical mappings look more like local aliases, canonical mapping lookups use the external form; but that does not work. Unlike postalias, the postmap command can't blindly apply email quoting rules to its input). Wietse
Re: Email address with leading whitespace rejected
Victor Duchovni:
On Fri, Jan 22, 2010 at 09:16:07AM -0500, Wietse Venema wrote:
The lookup keys and RHS values for virtual(5) are in rfc822 format.
A PCRE table can take care of this:
Virtual alias lookups are done in the unquoted form, while
canonical map lookups are in quoted form.
No, the cleanup(8) server virtual lookup key and value are in rfc822
quoted form, and will correctly process:
biza...@example.com
internal,comma@example.com,
internal whitespace@example.net,
...
So, the above form is good for canonical mapping, but virtual
alias mapping would require that the quotes be stripped:
See src/cleanup/cleanup_map1n.c around line 118:
quote_822_local(state-temp1, argv-argv[arg]);
if ((lookup = mail_addr_map(maps, STR(state-temp1), propagate)) != 0) {
...
}
I stand corrected.
Of course the address may not get that far, because the SMTP server
uses the internal form of the address when doing recipient validation:
static int check_rcpt_maps(SMTPD_STATE *state, const char *recipient,
const char *reply_class)
{
...
if (MATCH(rcpt_canon_maps, CONST_STR(reply-recipient))
|| MATCH(canonical_maps, CONST_STR(reply-recipient))
|| MATCH(virt_alias_maps, CONST_STR(reply-recipient)))
return (0);
One could argue that the SMTP server should use the external form of the
recipient for these lookup, to match downstream behaviour in cleanup(8)...
Indeed. There was no address validation in the initial design and
implementation, so address validation support does not fit as nicely
as one would like. It suffers from code duplication which leads to
inconsistency. But I would rather fight the duplication than the
inconsistency that results from it.
Just like address verification probes reuse the Postfix delivery
mechanisms, address validation should reuse those mechanisms, too.
Otherwise we would just be moving code duplication around to other
places in Postfix, and that is no progress.
Wietse
Re: Postfix Majordomo problem
On Fri, Jan 22, 2010 at 01:27:06PM -0500, Wietse Venema wrote: If I recall correctly, the wrapper program needs to be installed set-uid, and it needs to be configured at compile time with the right uid/gid information. Ruben Safir: I made it SIUD and the wrapper config-test seems to believe everything is working. But it is still failing. Of course it is failing, because you are fixing it by hand. YOU have a majordomo configuration error. THIS is the Postfix mailing list. Wietse
Re: Postfix Majordomo problem
Ruben Safir: [ Charset ISO-8859-1 unsupported, converting... ] On 01/22/2010 05:22 PM, Wietse Venema wrote: On Fri, Jan 22, 2010 at 01:27:06PM -0500, Wietse Venema wrote: If I recall correctly, the wrapper program needs to be installed set-uid, and it needs to be configured at compile time with the right uid/gid information. Ruben Safir: I made it SIUD and the wrapper config-test seems to believe everything is working. But it is still failing. Of course it is failing, because you are fixing it by hand. YOU have a majordomo configuration error. THIS is the Postfix mailing list. Thanks. It was initially installed by openSUSE11.2 ...I'm trying to fix what was working by hand. But my first problem is trying to understand what postfix is telling me or how I can get postfix to tell me more. You should to install the program from package, instead of trying to fix things by hand. However, if you insist on fixing things by hand, then you don't need Postfix to debug the program. Simply run the /some/where/wrapper majordomo... command by hand, with standard input connected to a file that contains an email message in UNIX mbox format. Majordomo is a Perl script, so you can debug it with all the standard Perl debugging features. This discussion is no longer appropriate for the Postfix mailing list, so this is my last post. Wietse
Re: smtp auth over ssl for smartrelay configuration
Jelle de Jong: Hello everybody, I got a hole set 20 of Debian systems connected to mobile broadband internet. They are behind a NAT of with dynamic ip's. I want these systems to be able to sent emails to my server for all kind of reasons like monitoring, security updates etcetera. I want to use postfix to authorise to my secured SMTP server to be able to deliver mail. The authorisation should be like the one's used on my MTA's like Mozilla Thunderbird with SMTP authorisation. Configuration option I made up: authuser=usern...@powercraft.nl authpass=password authmethod=plain mailhub=secure.powercraft.nl:465 usessl=true Can somebody show me an example how to setup up a simple outgoing only email configuration that uses SMTP AUTH over SSL? Postfix SASL: http://www.postfix.org/SASL_README.html Postfix TLS: http://www.postfix.org/TLS_README.html These are organized in client and server sections, with examples. There is no need to repeat this information on the mailing list. Wietse
Re: Best way to put spam on hold queue?
Jozsef Kadlecsik: Hello, We plan to add the possibility for our users to choose that messages categorized as spam are put on the hold queue instead of the default reject. Thus it'll be possible to release the false positives, which can make life easier for them. Currently I can see two ways to accomplish it, both have got pros and cons. a. Single postfix instance with large enough queue partition. Easier, simpler, however if the queue partition becomes full, the normal traffic is blocked as well. b. Two postfix instances, the first one sends the messages to be held to the second one via a selected transport. (Two policy daemons are required as well.) The first instance handles all the good traffic and diverts the bad one to the second instance, which has the single job to put messages onto its hold queue and release from there. That way the hold queue can be separated and moved to a second partition. Still, if the second instance stops working for whatever reason, the messages to be held will stuck at the first instance. How could one achieve that the held messages are separated from the normal traffic (i.e. hold queue on another partition), but if the messages cannot be held, then those gets rejected instead of queued? Given Postfix's architecture, a loosely-coupled pipeline without global feedback, there is no obvious way to build global feedback into Postfix itself. I suggest that you run an email monitoring system. When the secondary system stops accepting mail for more than some minimum amount of time (allowing enough time for the system to reboot), update the configuration on the primary system. Wietse
Re: Best way to put spam on hold queue?
Jozsef Kadlecsik: On Sat, 23 Jan 2010, Wietse Venema wrote: Jozsef Kadlecsik: How could one achieve that the held messages are separated from the normal traffic (i.e. hold queue on another partition), but if the messages cannot be held, then those gets rejected instead of queued? Given Postfix's architecture, a loosely-coupled pipeline without global feedback, there is no obvious way to build global feedback into Postfix itself. I suggest that you run an email monitoring system. When the secondary system stops accepting mail for more than some minimum amount of time (allowing enough time for the system to reboot), update the configuration on the primary system. Yes, monitoring is inevitable. And that imply you would prefer the two instances setup :-). Just as a theoretical question, would it fit somehow into Postfix to support the hold queue *alone* in a separated partition? The architecture requires Postfix can move messages between queues without having to make copies. Wietse
Re: Timeout of SMTP servers
Martijn de Munnik: On Jan 23, 2010, at 4:24 PM, Sahil Tandon wrote: On Fri, 22 Jan 2010, Martijn de Munnik wrote: RFC2821 section 4.5.3.2 Timeouts reads An SMTP server SHOULD have a timeout of at least 5 minutes while it is awaiting the next command from the sender. The key word is SHOULD, as opposed to MUST. SHOULD equals MUST unless you have a really good reason. I'm trying to figure out if somebody on the list knows a really good reason. Ask THEIR postmaster. Wietse
Re: Putting $data_directory on a RAM filesystem
Stefan Foerster: In case of severe server overload, with postscreen(8) complaining about lookup and update times around 400ms almost every mail, is it (reasonably) safe as a last desperate measure to put $data_directory, or at least the file referenced by $postscreen_cache_map, on a ramdisk (e.g. tmpfs with Linux)? Sure, but why do you expect that it will be FASTER? Are you perhaps running a busy mail server off a single disk drive? In that case, splitting the load over multiple drives would make a big difference. Is perhaps your system slow because of swapping? In that case, using tmpfs will make the system swap even more. Instead, even reducing the process count by 20% could result in a big over-all improvement. I haven't seen any attempt at a quantitative analysis of the problem, so I won't speculate further. Wietse
Re: 451 4.3.0 Error: queue file write error
Shawn Fee: Is there a fix for the 451 4.3.0 Error: queue file write error yet? I heard to increase the smtp_proxy_timeout = 600s in the main.cf file, but then I heard that can run down your server. You have a configuration error, and you need to look in the Postfix mail logfile for details. Postfix will not reveal these details in its reponses to SMTP clients. Wietse
Re: 451 4.3.0 Error: queue file write error
Shawn Fee: Sorry I meant Plesk 9.3.0. And the problem was suppose to be fixed in this realease. What is the Postfix logfile warning message? Wietse
