Re: Fwd: spf: lookup failed
2015-09-07, Czarek wrote: For incomming messages I found in the logs: spf: lookup failed: addr is not a string at /usr/share/perl5/vendor_perl/IO/Socket/IP.pm line 662 A bug in getnameinfo() in perl 5.16 and older. Fixed with perl 5.18 and later, which deal with pPOK vs. POK flags somewhat differently. See: http://marc.info/?l=spamassassin-users=141461245312708 one possible workaround: http://marc.info/?l=spamassassin-users=141467352930918 See the whole thread there for details. Mark
Re: Fwd: spf: lookup failed
A bug in getnameinfo() in perl 5.16 and older. Fixed with perl 5.18 and later, which deal with pPOK vs. POK flags somewhat differently. See: http://marc.info/?l=spamassassin-users=141461245312708 one possible workaround: http://marc.info/?l=spamassassin-users=141467352930918 or a workaround with Unbound: http://marc.info/?l=spamassassin-users=141962452928681 A backward compatible fix (instead of upgrading perl) is also provided by using a version of the perl module Socket 2.018 or later, as indicated in its changelog: 2.018 2015/02/12 13:42:41 * Fix for "addr is not a string" test to use SvPOKp() before 5.18 Mark
Re: AntiSpam & AntiVirus Integration with Postfix: lots of tools, but which one's AREN'T 'dead'?
joh...@fastmail.com wrote: I'm now at the phase of looking into Anti-Virus and Anti-Spam. Looks like ClamAV and Spamassassin are the main options here. Both of those projects seem to be pretty alive and kicking too. So I'm left with how to integrate them into and with Postfix. I've poked around in all the options listed on the Postfix wiki & the common option seems to be Amavisd. Another one on the A/S side is spampd. But both of those seem to be pretty inactive. Amavisd's not had any releases or updates afaict since last October, and I was told that the author isn't really participating in it anymore (I gotta check myself on that). Admittedly I have neglected Amavis lately, even though I was about to do a 2.10.2 release before this Summer holiday - but days were slipping between my fingers and day-job projects had priority. Luckily there are no glaring bugs in the last 2.10.1 release and it is quite stable. I spent most of my free Perl cycles lately working on SpamAssassin, helping to get its long-overdue version 3.4.1 out in Spring, and more recently bringing some of my Amavis international mail support experience to SpamAssassin - which is taking more effort than anticipated. Still the Amavis is my main open-source project and is not dead, although it may be understandable that as the SpamAssassin library is the main anti-spam tool of Amavis, the success of Amavis is coupled with well-being of SpamAssassin. Spampd seems a lot simpler, but is years old afaict. ... and can be very slow on processing large mail My question is - if I want to Postfix-integrate A/V & A/S *AND* stick to projects that have active development and communities like Postfix, what would you recommend? I know this isn't strictly a Postfix question, but I'm hoping that all the Postfix-Gods in here might share an opinion or advice. Mark
Re: SMTPUTF8 usage
Michael Ströder wrote: Does anybody here have experience with current usage of SMTPUTF8? I have a discussion whether that's already used in the wild or not. Google does support SMTPUTF8 : $ host -t mx gmail.com gmail.com mail is handled by 20 alt2.gmail-smtp-in.l.google.com. gmail.com mail is handled by 5 gmail-smtp-in.l.google.com. gmail.com mail is handled by 10 alt1.gmail-smtp-in.l.google.com. gmail.com mail is handled by 40 alt4.gmail-smtp-in.l.google.com. gmail.com mail is handled by 30 alt3.gmail-smtp-in.l.google.com. $ telnet gmail-smtp-in.l.google.com 25 Trying 2a00:1450:400c:c04::1a... Connected to gmail-smtp-in.l.google.com. Escape character is '^]'. 220 mx.google.com ESMTP gk19si9434995wjc.187 - gsmtp ehlo test 250-mx.google.com at your service, [...] 250-SIZE 35882577 250-8BITMIME 250-STARTTLS 250-ENHANCEDSTATUSCODES 250-PIPELINING 250-CHUNKING 250 SMTPUTF8 quit Apparently also a commercial mailer Momentum supports it. Mark
Re: spampd + amavis? [pre-accept filtering and amvis]
There is no difference for the remote SMTP client whether you use spampd in pre-accept mode, or amavisd-new in pre-accept mode. Both approaches have the same problem: when it takes too much time to inspect a message, the remote SMTP client will time out. Right. Amavisd tries to get all processing done in the allocated time ($child_timeout) and does pass the information on remaining available time to SpamAssassin too, so that the less essential processing may be skipped if a deadline is near. I'm not sure if spampd is capable of doing it too. Amavis release notes say: Provided that required minimal versions of Postfix and SpamAssassin are available, on can try amavisd in a Postfix proxy setup. The $child_timeout setting needs to be radically reduced in this setup, matching the longest time most SMTP clients are willing to wait, and must be less than Postfix is willing to wait (smtpd_proxy_timeout), which by default is 100 s. A sensible value is somewhat less then a minute (e.g. 45 seconds). Even though RFC 5321 (section 4.5.3.2.6) recommends that clients SHOULD be willing to wait for 10 minutes at data-end stage, it is not uncommon that this recommendation is not adhered to. So the FAQ for Amavis: --- The Postfix Before-Queue Content Filter setup, also known as smtpd_proxy setup, is not a supported or recommended setup with amavisd-new, which is not a transparent SMTP proxy by design. See caveats in README_FILES/SMTPD_PROXY_README. This setup might work amavisd-new for low-traffic sites which do not use authentication, but is not recommended. --- This is a cautionary speech - amavisd is not 100% transparent proxy and in this sense it does not meet Wietse's requirement for a support/warranty. Nevertheless, it is close enough to the 'fully transparent' that it plays well and reliably with current versions of Postfix and no surprises are expected for the future. So, as you said - *if* spampd was an acceptable solution for someone, there's no additional risk to using Amavis as described above. Do, I have that right? Yes. There is likely and advantage in amavis regarding meeting the configured deadlines. Mark
Re: Why does SPF fail sometimes?
Peter wrote: On 12/16/2014 05:25 AM, Darren Pilgrim wrote: It's extra fun when they do so to an email with a DKIM signature covering the From: header. MLMs should strip the DKIM header anyways and add their own if appropriate. There is (and must not be) any semantic or practical difference between a non-valid DKIM signature and absence of a DKIM signature. MLMs should *not* strip existing DKIM header fields even when they know it will get invalidated. Doing so strips away potentially valuable troubleshooting or informational data with no other benefit. MLMs are however free to add their own signature. Mark
anvil statistics log entry syntax
Just came across the following logged message which failed to be parsed by our log parser: postfix/anvil[29988]: statistics: max message rate 4/60s for ([2001:1470:ff80::25]:10088:2001:1470:ff80:88::80:c) at Dec 8 19:26:44 Btw, 10088 is a port number, not part of an IP address. Perhaps an IP address should be enclosed in brackets to avoid ambiguity. No big deal, just commenting... Mark
Re: google bouncing emails - ipv6 ptr problem?
Robert Moskowitz wrote: Perhaps this should go to the bind list, but all of my checking shows my ipv6 ptr record is working. This started, I think, last week. I was running an old mailserver and sent many an email to the cubieboard list. Just today I finally upgraded my mailserver, but still get the bounce. My current system is running Redsleeve 6 (Centos 6 for ARM), and postfix-2.6.6-6 The bounce from google says: The mail system cubiebo...@googlegroups.com: host gmr-smtp-in.l.google.com[2607:f8b0:4001:c08::e] said: 550-5.7.1 [2607:f4b8:3:3:67:15ff:fe00:180] Our system has detected that this 550-5.7.1 message does not meet IPv6 sending guidelines regarding PTR records 550-5.7.1 and authentication. Please review 550-5.7.1 https://support.google.com/mail/?p=ipv6_authentication_error for more 550 5.7.1 information. f9si145249igt.0 - gsmtp (in reply to end of DATA command) Note the '...and authentication' in that message! Read that web document again, note the 'Additional guidelines for IPv6' and the second bullet there: . The sending domain should pass either SPF check or DKIM check. Otherwise, mail might be marked as spam. I checked via my Verizon wireless MiFi connection and that IPv6 address ptr does point to z9m9z.htt-consult.com I have gone to the listed web page, and believe my ipv6 dns is correct. Looks alright, although DNS checkers have some complaints regarding your domain: http://www.webwiz.co.uk/domain-tools/ http://www.intodns.com/ http://www.dnsqueries.com/en/ I have not implemented spf or dkim. There you go! Consider also publishing a DMARC record, not necessarily restrictive. Mark
RFE: Using a link-local (scoped) IP address
After an instant success of switching our Redis server to listen only on a link-local (scoped) IPv6 address (RFC 4007), along with switching its clients (Amavis, SpamAssassin, logfeeder), I got greedy and tried to do the same with postfix, which didn't like my idea: master.cf: [fe80::1%lo0]:10025 inet n - n - - smtpd results in: postfix/master[83621]: fatal: /usr/local/etc/postfix/master.cf: line 245: valid hostname or network address required in [fe80::1%lo0]:10025 similarly: smtpd_proxy_filter=inet:[fe80::1%lo0]:10012 results in: postfix/smtpd[23918]: fatal: [fe80::1%lo0]:10012: valid hostname or network address required Admittedly it's not a particularly asked-for feature, although it may come handy with limiting access to some service to hosts on the same L2 link for free (without a need for a host-firewall), and it'd be nice for consistency with other services (telnet, ping6, ...). Considering that getaddrinfo() already handles scope syntax avoiding a need to deal with if_nametoindex(3) manually, it's probably just a matter of updating the config file syntax parsing to recognize scoped IP address. Or am I wrong? A quick reference: RFC 4007: IPv6 Scoped Address Architecture, sect 11: textual representation RFC 6874 A zone_id SHOULD contain only ASCII characters classified as unreserved for use in URIs [RFC 3986] RFC 3986: unreserved = ALPHA / DIGIT / - / . / _ / ~ http://tools.ietf.org/html/rfc4291 http://tools.ietf.org/html/rfc3493 http://tools.ietf.org/html/rfc4007 Mark
Internationalized Email now supported by amavisd (SMTPUTF8, EAI, IDN)
To go hand-in-hand with the Postfix support for Internationalized Email, the new version 2.10.0 of amavisd mail content filter was released today. So now that we have it covered at an MTA and at a content filter stages, it's perhaps time to step up the heat on developers of mail clients and IMAP servers, reminding them that RFC 6530 is now two and a half years old, and that only 5.43 % of the world population are native speakers of English (2007) according to Wikipedia/Nationalencyklopedin: https://en.wikipedia.org/wiki/List_of_languages_by_number_of_native_speakers Key new features: - added support for Internationalized Email: * RFC 6530 - Overview and Framework for Internationalized Email * RFC 6531 - SMTP Extension for Internationalized Email (SMTPUTF8) * RFC 6532 - Internationalized Email Headers * RFC 6533 - Internationalized Delivery Status Notifications This supports UTF-8 (EAI) in SMTP/LMTP sender addresses, recipient addresses, and message header section. Feature parity with Postfix version 2.12 (support introduced in development snapshot 20140715). The SMTPUTF8 extension is supported by Gmail since 2014-08-05: http://googleblog.blogspot.com/2014/08/a-first-step-toward-more-global-email.html - added support for Internationalized Domain Names (IDN) according to IDNA (RFC 5890, RFC 5891; RFC 3490); Release notes are at: http://www.ijs.si/software/amavisd/release-notes.txt Download at: http://www.ijs.si/software/amavisd/amavisd-new-2.10.0.tar.xz Mark
Re: Discuss: safety net for other compatibility breaks
Wietse wrote: What else needs to be considered? There are more settings whose defaults can be confusing to people who aren't familiar with 10+ years of Postfix history. - relay_domains (default: $mydestination). This should be empty. - mynetworks-style (default: subnet). This should be host. I like the above two changes. The defaults often are confusing for people new to postfix. - mynetworks-style (default: subnet). This should be host. Yes, definitely. Some more archaisms that can be changed to: biff = no swap_bangpath = no allow_percent_hack = no Mark
Re: Internationalized Domain Names (?)
Ronald F. Guilmette wrote: These days, whenever one builds any kind of tool that does anything with e-mail, it is necessary to think about this new-fangled phenomenon of Internationalized Domain Names, so... In what (if any) mail headers generated by Postfix might one reasonably expect to find either (a) punycoded domain names or else (b) Unicode characters. I don't think that currently Postfix does any conversions between ASCII-compatible encoding (ACE) and UTF-8 forms of a domain name, so both forms could be seen in a mail header with SMTPUTF8 enabled, and ACE forms would be seen in a traditional setup (but possibly also UTF-8 forms and arbitrary non-ASCII junk, despite not being legitimate). Docs says (SMTPUTF8_README): The initial Postfix SMTPUTF8 implementation performs no automatic conversions on UTF8 strings beyond what is needed to perform DNS lookups. Header fields carrying a domain name (that come to mind) are: Return-Path, Received, (Resent-)Sender, (Resent-)From, (Resent-)To, (Resent-)Cc, Reply-To, (Resent-)Message-ID, In-Reply-To, References, DKIM-Signature, DomainKey-Signature, List-*, potentially any X-*, and I'm sure there are others (Content-Location ?, ...). And of course, I have the same two questions with respect to the requests that are sent from Postfix to any installed and activated policy server. Within that stuff, where might one expect to see either (a) punycode or else (b) Unicode? Any of these, which can appear in a mail message. No conversions by MTA for the time being. Btw, amavisd since 2.10.0 converts ACE domain names to UTF-8 for presentation purposes (logging, JSON structured report, DNS and admin notfications), and encodes non-ASCII UTF-8 domains in sender and recipient addresses into ACE if the next hop MTA (e.g. back-end postfix) does *not* announce support for SMTPUTF8. P.S. Now that I have skimmed, briefly, the RFCs relating to this new SMTPUTF8 extension, I do have a question about that. But it seems like it would be inappropriate and off-topic for me to post that here, since it is not in any way Postfix-specific. Where are RFCs (such as RFC 6531) normally discussed? Try these: ietf-smtp -- Discussion of issues related to Simple Mail Transfer Protocol (SMTP) https://www.ietf.org/mailman/listinfo/ietf-smtp IMA -- EAI (Email Address Internationalization) https://www.ietf.org/mailman/listinfo/ima Mark
Re: Internationalized Domain Names (?)
Wietse wrote: Mark Martinec: Btw, amavisd since 2.10.0 converts ACE domain names to UTF-8 for presentation purposes (logging, JSON structured report, DNS and admin notfications), and encodes non-ASCII UTF-8 domains in sender and recipient addresses into ACE if the next hop MTA (e.g. back-end postfix) does *not* announce support for SMTPUTF8. What about non-ASCII text in Subject: and other headers, including the user name in ``User Name u...@example.com'', are you converting these according to RFC 2047, or is the above specifically for email addresses? Yes, these are converted to UTF-8 as well. More specifically: - logging (syslog or file) is nominally UTF-8, meaning that best effort is made to decode mail addresses and mail header content into UTF-8; there is no guarantee however that other non-ASCII junk (other than C0 controls) won't make it into log, i.e. garbage-in / garbage-out principle applies for the log in the interest of preserving as much of original data as sensible and safe. It is expected that syslogd will not clobber UTF-8 (i.e. a command line option -8 is needed for FreeBSD's syslog, otherwise C1 controls end up as ugly protected controls in a syslog file. - structured log in JSON is strictly UTF-8, as demanded by JSON specs; non-ASCII non-UTF-8 junk from mail is interpreted as Latin-1; - data from mail header or envelope that ends up in DSN is strictly UTF-8, potential non-ASCII non-UTF-8 garbage is interpreted as Latin-1; - RFC 2047 MIME encoding in Subject and From header fields is converted to UTF-8 for presentation purposes (top-level logging, JSON structured log); - ACE domain names in Message-ID and From header fields are decoded into UTF-8 for logging and JSON. (Other Display names from similar fields are currently not logged); - ACE domain names in sender and recipient envelope addresses is decoded into UTF-8 for logging and JSON; - UTF-8 domain names in sender and recipient envelope addresses is converted to ACE when passing mail to an MTA which does not announce SMTPUTF8. There are no changes to passed mail other than this one. Mark
SMTPUTF8: XFORWARD PROTO and WITH protocol types / UTF8SMTPS?A?
The XFORWARD_README / XFORWARD Command syntax currently tersely states: - The PROTO attribute specifies the mail protocol for receiving mail from the up-stream host. This may be an SMTP or non-SMTP protocol name of up to 64 characters, or [UNAVAILABLE] when the information is unavailable. This is somewhat vague, as it does not state whether the protocol name is just something like SMTP, ESMTP, QMQP, or whether this also includes the RFC 3848 Transmission Types (ESMTPA, ESMTPS, ESMTPSA), or whether it carries the WITH protocol types as registered at IANA: http://www.iana.org/assignments/mail-parameters/mail-parameters.xhtml like: SMTP, ESMTP, ESMTPA, ESMTPS, ESMTPSA, (LMTP, LMTPA, LMTPS, LMTPSA), UTF8SMTP, UTF8SMTPA, UTF8SMTPS, UTF8SMTPSA, UTF8LMTP, UTF8LMTPA, UTF8LMTPS, UTF8LMTPSA, ... In particular, it would be handy (for logging purposes) if the UTF8SMTPS?A? set would be included in the XFORWARD PROTO field. Mark
Re: SMTPUTF8: XFORWARD PROTO and WITH protocol types / UTF8SMTPS?A?
me said: The XFORWARD_README / XFORWARD Command syntax currently tersely states: - The PROTO attribute specifies the mail protocol for receiving mail from the up-stream host. This may be an SMTP or non-SMTP protocol name of up to 64 characters, or [UNAVAILABLE] when the information is unavailable. This is somewhat vague, as it does not state whether the protocol name is just something like SMTP, ESMTP, QMQP, [...] In particular, it would be handy (for logging purposes) if the UTF8SMTPS?A? set would be included in the XFORWARD PROTO field. 2014-09-30 Wietse wrote: This is intentional. It is some string, encoded if required. At the time I did not attempt to prescribe what names might exist or come into existence. [...] All those names would satisfy the XFORWARD spec, but requiring such names would be a different matter. As a final data point, Postfix currently does not log the extended with protocol name. This information does show up in the Received: header. The coming postfix-2.12 may be a good opportunity to enhance the information value of this attribute by switching to WITH protocol type IANA values, especially since such string is already assembled elsewhere in postfix. Thanks to the current documentation the change would not risk breaking backward compatibility :) Mark
smtp-sink: fatal: sockaddr_to_hostaddr: Non-recoverable failure in name resolution
Was investigating why I can't connect to my smtp-sink: $ smtp-sink -v [::1]:10055 10 smtp-sink: name_mask: all smtp-sink: trying... [::1]:10055 then in another window: $ smtp-source [::1]:10055 and the smtp-sink aborts with: smtp-sink: fatal: sockaddr_to_hostaddr: Non-recoverable failure in name resolution Turns out that the problem is a structure declared too short by two bytes to receive a sockaddr_in6 from accept(), and the two bytes of a received IP address are then clobbered. In smtp-sink.c/connect_event() the sa is declared as struct sockaddr instead of struct sockaddr_storage (RFC 3493). Seems like elsewhere this is handled correctly ( like in inet_listen.c/inet_accept() ). Mark
Re: Berkeley DB6 and Postfix
Robert Sander wrote: I remember that Mark Martinec mentioned a license change in Berkeley DB version 6 to the Affero GPL that forces Amavis to switch to LMDB. The additional provision requires that the complete source code be made available to any network user of the AGPL-licensed work https://en.wikipedia.org/wiki/Berkeley_DB#Licensing Actually it was Wietse who said that, referring to postfix. Regarding Amavis I said that the main reason for switching BerkeleyDB to ZMQ message passing was lock contention and a slowdown of a database after long use, requiring maintenance. The license change was just an additional incentive for the change. I may look at LMDB in the future, but the current needs of Amavis are well served by ZMQ, and Redis (or SQL). The LMDB would probably be more suitable for use by SpamAssassin. Mark
Re: disable ipv6 when sending to gmail ?
Dominik George wrote: if i would be you i would *not* use v=spf1 mx ~all here you go for ipv6 http://www.openspf.org/SPF_Record_Syntax#ip6 Jeez, I don't believe it. The problem is that the mx mechanism simply only enumerates A records of MXs. That's broken ... Wietse wrote: That's retarded. I wonder how many sites have been bitten by that bug. Joni wrote: The only place I've seen this problem with the lookup of IPv6 addresses via the 'mx' construct in SPF records was Gmail, which was resolved, and recently some small local operator who kept insisting that the problem was on our side until the evidence was so overwhelmingly pointing to his own setup that he could no longer ignore it. He made the same claim, however, but never backed it up. How are you reaching your conclusion? Because this only mentions A records and IPv4 prefixes? http://www.openspf.org/SPF_Record_Syntax#mx That http://www.openspf.org/SPF_Record_Syntax wiki page is wrong, or misleading in the least. The SPF specification in RFC 4408 does not fall into this trap, it talks about a (generic) ip address. Some excerpts from RFC 4408: When any mechanism fetches host addresses to compare with ip, when ip is an IPv4 address, A records are fetched, when ip is an IPv6 address, records are fetched. 5.3. a This mechanism matches if ip is one of the target-name's IP addresses. A= a [ : domain-spec ] [ dual-cidr-length ] An address lookup is done on the target-name. The ip is compared to the returned address(es). If any address matches, the mechanism matches. 5.4. mx This mechanism matches if ip is one of the MX hosts for a domain name. MX = mx [ : domain-spec ] [ dual-cidr-length ] check_host() first performs an MX lookup on the target-name. Then it performs an address lookup on each MX name returned. The ip is compared to each returned IP address. [...] dual-cidr-length = [ ip4-cidr-length ] [ / ip6-cidr-length ] Mark
Re: disable ipv6 when sending to gmail ?
HQJaTu writes: Google chose to change the wording in their 550 error. 550-5.7.1 [2001:-my-IPv6-address-here-16] Our system has detected 550-5.7.1 that this message does not meet IPv6 sending guidelines regarding 550-5.7.1 PTR records and authentication. Please review 550-5.7.1 https://support.google.com/mail/?p=ipv6_authentication_error 550 5.7.1 for more information. dj7si12191118bkc.191 - gsmtp Thanks for this information! That page now clearly states: Additional guidelines for IPv6 The sending IP must have a PTR record (i.e., a reverse DNS of the sending IP) and it should match the IP obtained via the forward DNS resolution of the hostname specified in the PTR record. Otherwise, mail will be marked as spam or possibly rejected. The sending domain should pass either SPF check or DKIM check. Otherwise, mail might be marked as spam. IMO, instead of working on workarounds, people's efforts would be better spent on setting up their DKIM and/or SPF, reverse DNS mapping, and making sure that postfix only binds to an intentionally configured IPv6 address (not on SLAAC or 'privacy extensions' random address). Mark
Re: Auto-whitelist recipients
Eddy, I'd like to continously update whitelist for spamassassin of recipients that my sasl users have sent mail to (i.e. when the recipients reply they will surely not be considered as spam). I am not using per-user spamassassin configurations (only a global configuration). I've found several approaches but none seems to fit... Where is the best place to insert a script that grabs recipient of mails being sent by sasl-authenticated users? Amavis offers this functionality through its PenPals feature. It keeps info on past messages in evidence (in SQL) and correlates sender/recipient pairs of incoming mail with their previous correspondence, but can also match previous correspondence based on Message-ID of a past message with a References or In-Reply-To header fields of new messages, which can deal also with mailing lists traffic or certain auto-responses. If an inbound message is found to match a past outbound correspondence, then negative score points are contributed to SpamAssassin score. The contributed score points are computed by an exponential decay formula, taken into account time from the last matching correspondence, a half-life interval, and a max allowed contribution score. By pushing these limits a hard whitelisting can be achieved, although keeping more moderate settings normally suffices. Search for pen pals soft-whitelisting in amavisd release notes for details. Mark
Re: warning: /usr/libexec/postfix/smtpd: bad command startup -- throttling
Kshitij, Feb 1 10:21:43 D1OKH680RL postfix/master[11324]: warning: process /usr/libexec/postfix/smtpd pid 11339 killed by signal 11 The smtpd service crashed with segmentation violation (SEGV). There is something wrong with your installation of postfix or libraries (like database access). Try recompiling/reinstalling. Feb 1 10:21:15 D1OKH680RL postfix/master[11324]: daemon started -- version 2.7.2-RC2, configuration /etc/postfix 2.7.2-RC2 ? Preferably install some officially released version like 2.7.7, or 2.8.8 or 2.9.0. Mark
Re: post-install, IPv6-only: could not find any active network interfaces (again)
Sahil Tandon wrote: I do not believe Mark should have to jump through extra hoops, or that you should revert the change. This is a FreeBSD port-specific problem created by me that I will address as soon as I can. Wietse Venema wrote: Considering the short time left before the next stable release I am considering the following schedule: - Revert to Postfix 2.8 behavior, and complete the 2.9 release cycle. - In the 2.10 development cycle, make Postfix build on hosts that have no network interfaces. That would eliminate problems like Mark's hosts without IPv4, FreeBSD port builds on hosts with dysfunctional IPv6, and other weird environments. - In the 2.10 development cycle, (re)start the first phase of the IPv6-on-by-default transition, and do this early enough that there is time to make sure that all maintainers are on board. That would be sad news, considering how long it takes for distributions to jump on each new major version. As long as main.cf gets adjusted if necessary during install to maintain backward compatibility, the builtin default does not matter, as long as the package can be build and installed. Mark
post-install, IPv6-only: could not find any active network interfaces (again)
Reviving an old thread from 2011-09: Mark Martinec: Trying to install postfix on an IPv6-only host FreeBSD 9.0B1, http://wiki.freebsd.org/IPv6Only ports: mail/postfix-current, but the installation chokes in the post-install phase. Running that failing command manually (in the ports work directory) gives: # bin/postfix -v post-install postfix: name_mask: ipv4 postfix: name_mask: host postfix: inet_addr_local: configured 0 IPv4 addresses postfix: fatal: could not find any active network interfaces Wietse Venema wrote: The built-in default enables IPv4 only, to avoid headaches with systems that have IPv6 support but no external IPv6 connectivity. Lookup up addresses would be wasteful, and trying to connect to them would be pointless. A workaround is to do the same thing as when local_recipient_maps was introduced: the inet_protocols built-in default different from the installed configuration. In this case, the built-in default would enable IPv6, while the post-install installed configuration is IPv4 only (but post-install would not change an explicit inet_protocols main.cf setting). I was hoping this approach would make its way into current eventually ... It will. I have added the note to the wishlist. I thought this was implemented in current by: | HISTORY | 20110914 | Incompatibility: the default inet_protocols value is now | all instead of ipv4, meaning use both IPv4 and IPv6. | As a compatibility workaround for sites without global IPv6 | connectivity, the commands make upgrade and postfix | upgrade-configuration append inet_protocols = ipv4 to | main.cf when no explicit setting is present. This compatibility | workaround will be phased out in a future release. Files: | util/sys_defs.h, conf/post-install, proto/postconf.proto. | | Incompatibility: the default smtp_address_preference value | is now any instead of ipv6, meaning choose randomly | between IPv6 and IPv4. With this the Postfix SMTP client | will have more success delivering mail to sites that have | problematic IPv6 configurations. Files: global/mail_params.h, | proto/postconf.proto. ... but apparently it was not in its entirety. Trying to install postfix-2.9-20111219 from ports on FreeBSD 9.0 (with an inet6-only kernel), the installation fails as before: postfix-current$ make install ... Skipping /usr/local/share/doc/postfix/postfix-power.png... Skipping /usr/local/share/doc/postfix/scache.8.html... Skipping /usr/local/share/doc/postfix/tlsmgr.8.html... postfix: fatal: could not find any active network interfaces *** Error code 1 Stop in /usr/ports/mail/postfix-current. or manually in the /usr/ports/mail/postfix-current/work/postfix-2.9-20111219 : # bin/postfix -v post-install postfix: name_mask: ipv4 postfix: name_mask: host postfix: inet_addr_local: configured 0 IPv4 addresses postfix: fatal: could not find any active network interfaces It will. I have added the note to the wishlist. So ... Dear Santa ... :) Mark
Re: post-install, IPv6-only: could not find any active network interfaces (again)
postfix: fatal: could not find any active network interfaces *** Error code 1 How do you want to proceed: wait until I have time to reproduce your IPv6-only setup for which I have no specification, or spend all of next year doing blind testing? No offense and not intending to rush things. I just thought that the mentioned entry in HISTORY may indicate that you considered it done. If you want to test it by yourself, either fetch a pre-built 9.0-RC3 inet6-only installation ISO from: http://wiki.freebsd.org/IPv6Only or re-build a 9.0 kernel by giving it the following KERNCONF file: include GENERIC ident GENERIC-IPV6ONLY makeoptions MKMODULESENV+=WITHOUT_INET_SUPPORT= nooptions INET nodevice gre It is all documented in: http://www.freebsd.org/ipv6/ipv6only.html Or I can make available a small VirtualBox disk image with the necessary ports already installed, or provide a ssh root access to such. Mark
Re: post-install, IPv6-only: could not find any active network interfaces (again)
I'm assuming that you have inet_protocols=ipv6 in main.cf, instead of the backwards-compatibility inet_protocols=ipv4 workaround, because that would not work on your machine. No, that was a fresh install attempt, no directory /etc/postfix or /usr/local/etc/postfix, no previous main.cf or master.cf files. The problem is in the hardwired inet_protocols=ipv4 in the code, if I remember correctly. Mark
Re: Possibility to store all incoming mail (pre-content_filter)
Michael, Yeah, unlikely but possible. In fact the mail passes through 2 filters before being returned to postfix: postfix:25 - amavis:10024 - apache-james:10025 - postfix:10026 - smarthost All i can tell is that some mails (like 1 out of 2) get corrupted in the process and end up being unusable. I cannot disable amavis completely as spam hell would break lose. I cannot disable apache-james because it contains some custom filters. The most likely culprit here is apache-james because it contains some custom code. But if i disable it i cannot tell which mails would have triggered the bug and which ones didn't. That's why i want to store mails at postfix:25 before they get altered. You may enable archive quarantine in your pre-queue amavis, e.g.: $archive_quarantine_method = 'local:archive-%m'; $archive_quarantine_to = 'archive-quarantine'; # default to be able to compare a corrupted message to what was seen by amavisd. This would not help if a problem lies in stages prior to or in amavisd, but at least it can help troubleshooting later stages (SMTP output from amavisd and apache-james). Mark
Re: Per-Recipient Data Responses (was: ... per-recipient treatment of messages in a milter environment)
Wietse wrote: To make per-recipient end-of-data replies useful with Postfix, PRDR would need to be supported by at least one third-party content inspection mechanism (such as Amavisd-new or Milter), because I see no obvious user interface for PRDR with Postfix header/body_checks. - SMTP-based filters such as Amavisd-new would use the PRDR protocol to send per-recipient end-of-data replies. If postfix implements PRDR, then I can add it to amavisd too. Alternatively, if smtpd_proxy_filter would automagically speak LMTP to a filter, than I needn't change anything, as amavisd already speaks either SMTP or LMTP, both on input and output. Mark
unused parameter: smtpd_client_connection_limit_exceptions
postfix 2.9.2019 warns me:
unused parameter: smtpd_client_connection_limit_exceptions
Yet if I remove this option from master.cf, I soon reach the connection
limit at the pre-queue content filter's re-entry smtpd service:
421-4.7.0 mail.ijs.si Error: too many connections from ::1
451 4.5.0 Contact postmas...@ijs.si for assistance.
Also I don't see the smtpd_client_connection_limit_exceptions
in the postconf(5) man page, yet it shows in the source (mail_params.h):
#define DEF_SMTPD_HOGGERS
${smtpd_client_connection_limit_exceptions:$ VAR_MYNETWORKS }
What am I missing?
10025 inet n - n - - smtpd
-o smtpd_client_restrictions=permit_mynetworks,reject
-o mynetworks=127.0.0.0/8,[::1]
-o smtpd_client_connection_limit_exceptions=0.0.0.0/0
[...]
Mark
Re: unused parameter: smtpd_client_connection_limit_exceptions
-o smtpd_client_connection_limit_exceptions=0.0.0.0/0 Jeroen Geilman wrote: This is probably old code, since postconf(5) says: http://www.postfix.org/postconf.5.html#smtpd_client_event_limit_exceptions *smtpd_client_event_limit_exceptions( default: $mynetworks postconf.5.html#mynetworks)* Clients that are excluded from smtpd_client_*_count/rate_limit restrictions. You've apparently found the old setting; the above is the new (current) version. Configuration parameters get renamed from time to time when a better description for added or changed behaviour applies. The old ones tend to hang around for several versions to avoid breaking existing configurations. Actually I did replace the smtpd_client_connection_limit_exceptions with smtpd_client_event_limit_exceptions (not just delete it, as may have been understood from my posting), but that did not achieve the goal of turning off connection limiting. Wietse wrote: The problem is that postconf no longer parses parameter values that have their default value. You can work around this with an explicit [...] I'll roll out a correction. Thanks! Mark
smtp-sink pipelining slow: TCP Nagle delayed ACK stalls
While benchmarking a SMTP content filter, using smtp-source as a traffic generator and smtp-sink as sink, the message transfer rates were much worse than expected (about 100 seconds, instead of just a few seconds for 1000 messages). It turned out the problem is in a TCP session over a loopback interface between a content filter and smtp-sink. When pipelining is used and all the MAIL FROM, RCPT TO, and DATA arrive in one packet, the smtp-sink responds in two packets: the first is a 250 2.1.0 Ok response to a MAIL FROM command, and the second packet carries a response to the rest: 250 2.1.5 Ok\r\n354 End data with...\r\n. The trouble is that there is a 0.1 second delay between the two response packets. The second packet is only sent by smtp-sink after receiving an ACK to the first, and that only happens after 0.1 seconds due to a delayed ACK setting of a system (FreeBSD 9.0, net.inet.tcp.delayed_ack=1). The workarounds are: - disable net.inet.tcp.delayed_ack globally or: - disable pipelining announcement in smtp-sink (option -p) or: - ignore pipelining announcement in a content filter The true solution seems to be to either disable Nagle in smtp-sink (TCP_NODELAY), or to send all the SMTP responses in one go. Seems the postfix itself does not suffer from this problem, only the smtp-sink. A tcpdump packet capture is available at: http://www.ijs.si/~mark/tmp/0.log.gz Mark
Re: MIME Parser Error - Can't Send Email
Just for the archive: (host 127.0.0.1[127.0.0.1] said: 451 4.5.0 Error in processing, id=10796-01, mime_decode-1 FAILED: MIME::Parser: can't open tmpfile: Invalid argument As Patrick and Gary said, looks like a trouble with a /tmp directory (protection?) or its file system (full or out of inodes). Older versions of MIME-tools (5.419 and older) use IO::File-new_tmpfile to create a temporary file (typically in a /tmp directory), and return a silly diagnostics Invalid argument on encountering any problem. Versions of MIME-tools starting with 5.500 use File::Temp-new instead. On a failure this returns a much more sensible diagnostics. Mark
smtp-sink shows one more empty EHLO option
Seems like the smtp-sink appends one empty EHLO option at the end of its reply to an ehlo command. Should this be fixed? - my content filter is currently logging a warning, I wonder if I should remove the warning :) Using postfix-current-2.9.20111012 from FreeBSD ports. $ smtp-sink 127.0.0.1:20025 30 $ telnet 127.0.0.1 20025 # connection to smtp-sink 220 smtp-sink ESMTP ehlo test 250-smtp-sink 250-PIPELINING 250-8BITMIME 250-AUTH PLAIN LOGIN 250-XCLIENT NAME HELO 250-XFORWARD NAME ADDR PROTO HELO 250-ENHANCEDSTATUSCODES 250 quit 221 Bye $ telnet 127.0.0.1 25 # connection to postfix, normal 220 mail.ijs.si ESMTP Postfix ehlo test 250-mail.ijs.si 250-PIPELINING 250-SIZE 26214400 250-VRFY 250-ETRN 250-STARTTLS 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN quit 221 2.0.0 Bye Mark
Re: smtp-sink shows one more empty EHLO option
Nope, RFC 2821 and RFC 5321 still has the same text. It even goes on to say ... RFC 5321 does not allow empty ehlo-keyword: section 4.1.1.1: ehlo-ok-rsp= ( 250 SP Domain [ SP ehlo-greet ] CRLF ) / ( 250- Domain [ SP ehlo-greet ] CRLF *( 250- ehlo-line CRLF ) 250 SP ehlo-line CRLF ) ehlo-line = ehlo-keyword *( SP ehlo-param ) ehlo-keyword = (ALPHA / DIGIT) *(ALPHA / DIGIT / -) ; additional syntax of ehlo-params depends on ; ehlo-keyword and neither does RFC 2821. Mark
Re: PIX timed out while sending end of data -- message may be sent more than once
John, Oct 5 00:10:22 myhost postfix/smtp[28713]: 125BC2400A7: to=fred.blo...@abc.tld, relay=mail.abc.tld[123.456.789.123]:25, delay=187500, delays=186888/0.01/0.16/612, dsn=4.4.2, status=deferred (conversation with mail.abc.tld[123.456.789.123] timed out while sending end of data -- message may be sent more than once) My understanding is that the first message is the original, and the second and subsequent are postfix retrying the delivery. my postconf -n output follows below, Your message probably hit one of several Cisco PIX/ASA bugs related to mail header section parsing: http://www.arschkrebs.de/postfix/postfix_cisco_pix_bugs.shtml 1) Am I misconfigured? Probably not. 2) How can I handle this sort of problem in future? If it is your PIX or ASA or you can establish a contact with their hostmaster, the box should be upgraded, or the smtp protocol fixup (mis)feature turned off - best to do both. If it is entirely out of control, removing a DKIM signature header field for mail to such site will probably help. Ralf posted some workaround here some time ago. Mark
Re: post-install, IPv6-only: could not find any active network interfaces
On Aug 23, 12:30 pm, Mark Martinec wrote: Trying to install postfix on an IPv6-only host FreeBSD 9.0B1,http://wiki.freebsd.org/IPv6Only ports: mail/postfix-current, but the installation chokes in the post-install phase. Running that failing command manually (in the ports work directory) gives: # bin/postfix -v post-install postfix: name_mask: ipv4 postfix: name_mask: host postfix: inet_addr_local: configured 0 IPv4 addresses postfix: fatal: could not find any active network interfaces Nick wrote: Hi Mark, have you found a solution to this? I have an IPv6 only jail in which I'm trying to install Postfix from Ports but it fails with this error. Any pointers would be welcome. Well, I followed Wietse's suggestion: | A workaround is to do the same thing as when local_recipient_maps | was introduced: the inet_protocols built-in default different from | the installed configuration. | | In this case, the built-in default would enable IPv6, while the | post-install installed configuration is IPv4 only (but post-install | would not change an explicit inet_protocols main.cf setting). ...so I hacked the source to change a built-in default value of inet_protocols to all, and then a main.cf can replace this value with whatever it chooses. I was hoping this approach would make its way into current eventually ... Mark
post-install, IPv6-only: could not find any active network interfaces
Trying to install postfix on an IPv6-only host FreeBSD 9.0B1, http://wiki.freebsd.org/IPv6Only ports: mail/postfix-current, but the installation chokes in the post-install phase. Running that failing command manually (in the ports work directory) gives: # bin/postfix -v post-install postfix: name_mask: ipv4 postfix: name_mask: host postfix: inet_addr_local: configured 0 IPv4 addresses postfix: fatal: could not find any active network interfaces # ifconfig em0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 mtu 1500 options=9bRXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM ether 08:00:27:99:9a:19 inet6 fe80::a00:27ff:fe99:9a19%em0 prefixlen 64 scopeid 0x1 inet6 2001:1470:ff80:88:a00:27ff:fe99:9a19 prefixlen 64 autoconf inet6 2001:1470:ff80:0:a00:27ff:fe99:9a19 prefixlen 64 autoconf nd6 options=23PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL media: Ethernet autoselect (1000baseT full-duplex) status: active lo0: flags=8049UP,LOOPBACK,RUNNING,MULTICAST metric 0 mtu 16384 options=3RXCSUM,TXCSUM inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 nd6 options=21PERFORMNUD,AUTO_LINKLOCAL While it may not currently be of much use to have a MX without an IPv4 address, it still does make sense to have such mailer on the intranet. Suggestions for a solution? Mark
Re: conversation with ... timed out while sending end of data -- message may be sent more than once
On Wednesday June 15 2011 05:42:36 Noel Jones wrote: At this time I'm inclined to set this aside. The DKIM bug doesn't seem to be widespread; there is no compelling case to add a new workaround right now. Indeed the situation has much improved in the past year or two. Many sites have turned off smtp fixups or upgraded their ASA firmware or both. It also helps to send mail to postmasters of affected sites with a pointer to Ralf's web page and the Heise article, and suggest turning off the (mis)feature. Perhaps the incentive was when they started missing some of the mail from gmail.com and the like. Mark
Re: conversation with ... timed out while sending end of data -- message may be sent more than once
Ralf wrote: Today I found that some sites behind a PIX/ASA firewall with smtp protocol fixup would not accept DKIM signed mails. But you already knew that! :) ASA bug CSCsy28792 and a couple of related header-parsing bugs, triggered by encountering a content-type or content-transfer-encoding in a header field body of some unrelated header field, such as an 'h' tag of a DKIM signature: http://www.arschkrebs.de/postfix/postfix_cisco_pix_bugs.shtml Mark
Re: conversation with ... timed out while sending end of data -- message may be sent more than once
How does an SMTP client recognize an ASA box before it breaks email? Only from the /^[02 *]+$/ banner. # telnet mx.interfree.it 25 220 ** I think the newer versions of ASA can be configured to let ESMTP pass through without censoring the greeting, while still exhibiting one of the header parsing bugs - which can lead to dropping the TCP session without a RST (but with a message in the log ... which noone reads). Mark
Re: Anyone run Postfix in FreeBSD jails environement ?
Does anyone is running postfix in FreeBSD jails environement with success on a production server ? I'm thinking of it and would be interrested by any successful experience. FreeBSD older than 7.2 did not support multiple IP addresses in jail (e.g. an IPv6 address, or a separate mail submission IP address). More recent version should be fine. Mark
Re: Timed out while sending message body
Tomasz K. Jarzynka: Finally, I ran a tcpdump on our origin mail server, our firewall and the destinantion mail server (thanks to the help of its administrator) but the output is inconclusive to me. On our side, It looks like transmission stalls after a couple hundred bytes + subsequent retransmissions eventually leading to a timeout. On the destination side, it looks like the connection is ended immediately after the start of transmission. Google for 1 - IP path MTU discovery. 2 - TCP window scaling. This doesn't look like a PMTU discovery problem, more like a NAT screwup. A duplicate ACK packet #78 in mnisw-fw.dump seems to get translated into a packet #81 in mnisw-mailhub.dump with a wrong sequence number and with no NAT translation. This causes your mailer to RST the seemingly new session, this RST is then seen by the remote side as an original session reset. Mark
Re: Google 7720 Error
I have no proxies and have turned off the firewall although the fact it works for some gmail and mindspring and not other is puzzling Any Cisco firewall (ASA or PIX) on your side? Mark
Re: postfix performance
I installed both pdns-recursor and unbound (running without any zone data) on a test box and got very similar performance results from both. We happened to go with unbound, but based on your recommendation, maybe I'll give pdns-recursor another look (it's still running on our test box). We switched from bind to unbound for our mailer's and content filter's DNS server a couple of months ago, I think it was as good choice: lean, mean and designed from grounds up with standards, security, speed, low footprint and DNSSEC in mind. (I even run unbound on a laptop natively under Win7 as its only resolver/server :) Mark
Re: warning: truncate before-queue filter speed-adjust log: Permission denied
Wietse Venema wrote: Please file a ZFS bug reportug. As per POSIX, when the O_CREAT is specified to open(), The third argument does not affect whether the file is open for reading, writing or for both. In other words, read/write access is controlled with the O_RDWR flags, not the read/write permissions argument. When the above error happens, Postfix discards the file. Consequently, Postfix performance will be reduced, because it creates one extra file per MAIL transaction, instead of one extra file over the process lifetime. Thanks for the insight. Reported: http://www.freebsd.org/cgi/query-pr.cgi?pr=154873 we'll see what comes out of it. Mark ZFS fix has been committed to HEAD! | Re: standards/154873: | ZFS violates POSIX on open/O_CREAT - ftruncate | | State-Changed-From-To: open-patched | State-Changed-By: pjd | State-Changed-When: czw 24 mar 2011 20:28:17 UTC | State-Changed-Why: | Fix committed to HEAD. Thanks! | http://www.freebsd.org/cgi/query-pr.cgi?pr=154873 Mark
Re: Long queue ID support gotcha
The idea is to prepend the 30 least significant bits of the time in seconds to the queue ID. Btw, 6 more hours to the next 'pretty' decimal unix timestamp: 13 Mark
warning: truncate before-queue filter speed-adjust log: Permission denied
A freshly installed postfix 2.8.0 from FreeBSD ports on FreeBSD 8.2-RC3, with a file system on ZFS (zpool v15, zfs v4) shows an interesting warning when smtpd_proxy_options=speed_adjust is enabled on a smtpd service which uses a proxy filter: Feb 18 20:25:39 xxx postfix/smtpd[3620]: warning: truncate before-queue filter speed-adjust log: Permission denied Other than that, everything looks normal. There are no leftover files in the spool area, and mail is delivered normally. I took a liberty to let smtpd_proxy.c log a path of the log stream, added a printout of the path just after a file creation, and commented out the unlink, so that the temporary file remains in place for observation. Seems to me like the problem might be a temporary file created with *no* permission bits, and perhaps ftruncate chokes on that even though the file has been unlinked by now: # ll -d /var/spool/postfix/incoming/243E8171 -- 1 postfix wheel 596 Feb 18 20:46 /var/spool/postfix/incoming/243E8171 # ll -d /var/spool/postfix/incoming drwx-- 2 postfix wheel 4 Feb 18 20:46 /var/spool/postfix/incoming # ll -d /var/spool/postfix drwxr-xr-x 16 root wheel 16 Feb 10 12:18 /var/spool/postfix # ll -d /var/spool drwxr-xr-x 9 root wheel 9 Feb 10 12:18 /var/spool # ll -d /var drwxr-xr-x 28 root wheel 28 Feb 11 03:37 /var # ll -d / drwxr-xr-x 47 root wheel 55 Feb 13 18:41 / Feb 18 20:46:16 xxx postfix/smtpd[12704]: connect from localhost[127.0.0.1] Feb 18 20:46:16 xxx postfix/smtpd[12704]: warning: CREATED before-queue filter speed-adjust log: incoming/243E8171 = Feb 18 20:46:16 xxx postfix/smtpd[12704]: NOQUEUE: client=localhost[127.0.0.1] Feb 18 20:46:23 xxx postfix/smtpd[12709]: connect from localhost[127.0.0.1] Feb 18 20:46:23 xxx postfix/smtpd[12709]: D7A31172: client=localhost[127.0.0.1], orig_client=localhost[127.0.0.1] Feb 18 20:46:23 xxx postfix/cleanup[12711]: D7A31172: message-id=201102181946.p1ijkgaw012...@xxx.ijs.si Feb 18 20:46:23 xxx postfix/qmgr[12698]: D7A31172: from=t...@xxx.ijs.si, size=968, nrcpt=1 (queue active) Feb 18 20:46:23 xxx postfix/smtpd[12704]: proxy-accept: END-OF-MESSAGE: 250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as D7A31172; from=t...@xxx.ijs.si to=t...@ijs.si proto=ESMTP helo=xxx.ijs.si Feb 18 20:46:23 xxx postfix/smtpd[12704]: warning: truncate before-queue filter speed-adjust log incoming/243E8171: Permission denied = Feb 18 20:46:23 xxx sendmail[12703]: p1IJkGaW012703: to=t...@ijs.si, ctladdr=test (401/401), delay=00:00:07, xdelay=00:00:07, mailer=relay, pri=30049, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as D7A31172) Feb 18 20:46:23 xxx postfix/smtpd[12704]: disconnect from localhost[127.0.0.1] Feb 18 20:46:25 xxx postfix/smtp[12712]: D7A31172: to=t...@ijs.si, relay=mail.ijs.si[193.2.4.66]:25, delay=1.4, delays=0.02/0/0.03/1.3, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[::0001]:10011): 250 2.0.0 Ok: queued as 0A4611D1E4E) Feb 18 20:46:25 xxx postfix/qmgr[12698]: D7A31172: removed Feb 18 20:47:17 xxx postfix/smtpd[12709]: disconnect from localhost[127.0.0.1] Mark
Re: warning: truncate before-queue filter speed-adjust log: Permission denied
Wietse Venema wrote: Please file a ZFS bug reportug. As per POSIX, when the O_CREAT is specified to open(), The third argument does not affect whether the file is open for reading, writing or for both. In other words, read/write access is controlled with the O_RDWR flags, not the read/write permissions argument. When the above error happens, Postfix discards the file. Consequently, Postfix performance will be reduced, because it creates one extra file per MAIL transaction, instead of one extra file over the process lifetime. Thanks for the insight. Reported: http://www.freebsd.org/cgi/query-pr.cgi?pr=154873 we'll see what comes out of it. Mark
Re: postfix/trivial-rewrite: warning: mysql query failed: Illegal mix of collations
Claudio Prono wrote: Uhm, i have another information about that case: the mail are sended to postfix from an antispam appliance (Symantec). Can be a problem of config of that antispam results illegal characters are sended to postfix? Can i add something to solve that problem? That is possible, but more likley illegal characters appear in the original received mail and the appliance just lets them through. Until such novelty as UTF8SMTP gets some traction (not anytime soon it seems), I still think the cleanest and safest way is to treat envelope addresses as strings of bytes with no associated character set. That means declaring data type of such SQL fields as VARBINARY or BYTEA, and letting a client use a data type SQL_VARBINARY (or PG_BYTEA for PostgreSQL) for passing such data to SQL. This avoids all the fancy magic in SQL, like verifying compliance to a given character set, and using collations associated with a character set (including making compares case-insensitive). Mark
Re: postfix/trivial-rewrite: warning: mysql query failed: Illegal mix of collations
How does MySQL know that the query parameter(s) should be UTF-8 and not ISO LATIN mumble or something else? By a client executing a command: SET NAMES 'utf8' as far as I can tell. SET NAMES indicates what character set the client will use to send SQL statements to the server. http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html Mark
Re: postfix/trivial-rewrite: warning: mysql query failed: Illegal mix of collations
What MySQL makes of such data is up to the MySQL client and server libraries, but Postfix does not promise that the input will be well-formed UTF-8, or ISO Latin or anything of the sort. Just an array of bytes. Right, as it should be. Envelope addresses are not associated with any character set according to RFC 5321, they are just strings of octets. For this reason an appropriate SQL data type for such fields is VARBINARY (or BYTEA in PostgreSQL). A data type CHAR or VARCHAR is inappropriate, as it associates a character set with data. SQL may perform validation of data according to the specified character set. MySQL tend to be quite premissive to such violations, but there is no guarantee. Also, comparing CHAR or VARCHAR strings with relational operators is case-insensitive and may even apply special (like Unicode) rules for character equivalency. Mark
Re: postfix/trivial-rewrite: warning: mysql query failed: Illegal mix of collations
Jeroen Geilman wrote: Urgh. Which RFC are you reading ? I quote: Systems MUST NOT define mailboxes in such a way as to require the use in SMTP of non-ASCII characters True (tell it to generators of malicious mail or just incompetent sending sw). This does not prevent illegal data to appear on the wire on the receiving MTA. Receiving such data must not cause MTA or SQL to malfunction. There is also an initiative to allow UTF-8 characters to appear in SMTP (RFC 5336 and related documents). A malformed UTF-8 could easily appear there, despite being prohibited. If an SQL database would declare an e-mail address field of an UTF-8 data type, a lookup could abort when given such invalid data. Mark
Re: postscreen access list
postscreen_dnsbl_sites = zen.dnsbl*2 ??? You mean zen.spamhaus.org Mark
Re: postfix-2.8.0-RC3 and postfix-2.9-20110118
I have uploaded new tarballs to ftp.porcupine.org. Let's hope that things stabilize this week. Below are the changes since RC2. Last-minute incompatible syntax change: Postfix now uses ; instead of , to separate DNSBL/DNSWL address filter fields inside []. RELEASE_NOTES (in RC3) needs to be updated to the new syntax. Also in RELEASE_NOTES: suppport - support Mark
Re: postfix-2.8.0-RC3 and postfix-2.9-20110118
Anything else? Does it work? So far so good, it works. Perhaps it's time (in the next RC, if any) to remove the safety net need for postscreen_whitelist_networks = postscreen_blacklist_networks = Mark
postscreen_dnsbl_sites filter syntax?
I must be doing something silly, but I can't see my mistake. $ postconf postscreen_dnsbl_sites postscreen_dnsbl_sites = zen.spamhaus.org=127.0.0.[2,3,4..8,10..11] postfix/postscreen[26161]: fatal: bad DNSBL filter syntax: need , or ] at 127.0.0.[2 Or to simplify the matter: $ postconf postscreen_dnsbl_sites postscreen_dnsbl_sites = zen.spamhaus.org=127.0.0.[3,4] postfix/postscreen[25207]: fatal: bad DNSBL filter syntax: need , or ] at 127.0.0.[3 Without a filter or with just a plain dotted-quad it works normally. This is postfix-2.8.0-RC2, FreeBSD, installed from ports mail/postfix-current after adjusting the version of a filename and a checksum. Mark
Re: PREPEND problems
mouss wrote: anyway, reading your prepend info tells us that you're trying to do something regarding spamhaus based on the From header. This is most probably wrong. if you tell us what you're trying to do, we will tell you why you are wrong ;-p If we are talking about VBR-Info based on a DKIM signature (not SPF), then making it depend on a domain in From header field is likely the right thing to do, because the signing domain (the 'd' tag) is also likelyto be derived from a domain in a From header header field (making ADSP happy). Even if occasionally this isn't so (e.g. when VBR-Info is inserted but signature is not), it isn't too bad. Recipients take the VBR-Info just as a hint. If there is no valid DKIM signature in a message, the information in VBR-Info will be ignored - DNS whitelists like DWL should only be consulted if signature is valid (or spf passes). Mark
Re: Spamhaus DWL in postfix
OT, sorry, just to finish up this thread: myself: I'm working on a SpamAssassin plugin to implement Spamhaus DWL (and other 'SA tag'- based DNS lookups). Done. Available in the SpamAssassin SVN trunk (on its way to become 3.4.0): https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6518 Also the IPv6 DNSWL/DNSBL queries according to RFC 5782 are there now: https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6519 Mark
Re: Spamhaus DWL in postfix
Christian Roessner: I am interested in including the DWL feature from SpamHaus into postfix. Wietse: DWL requires content external content inspection. For example, a Milter, or a before-or-after-queue SMTP-based filter. Either approach can be used to verify the DKIM signature and the VBR header. I'm working on a SpamAssassin plugin to implement Spamhaus DWL (and other 'SA tag'- based DNS lookups). It is a bit more tricky than it seems at first glance. Coupled with amavisd-new 2.7.0 in a pre-queue proxy mode this could accomplish the job (i.e. meeting Spamhaus DWL requirements, not a general VBR implementation). Mark
Re: cidr table performance
Jeroen Geilman wrote:
for (entry = list; entry; entry = entry-next) {
Each map is a linked list of CIDR patterns, so consolidate as much as
possible - 10 single IPs will cause noticable delays when the last
entry matches!
Funny coincidence: just yesterday I added a Patricia (radix) trie search
to SpamAssassin, which also needs to check if an IP address matches
a list of included/excluded CIDR ranges of IPv6 or IPv4 addresses.
For large lists the speedup can be substantial. Now a lookup takes
about 0.2 ms (in Perl), regardless of the size of a table - which is
a nice property of a radix trie (commonly used with IP routing tables).
Bug 6508: Speeding up lookups on {trusted,internal,msa}_networks
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6508
Patricia trie on Wikipedia:
http://en.wikipedia.org/wiki/Patricia_trie
Net::Patricia perl module on CPAN:
http://search.cpan.org/dist/Net-Patricia/
Mark
Re: postscreen vs. (all?|some?) address verification milter(s) in sendmail
Here is a similar incident with a milter not understanding multiline responses, as well as shooting out the query without waiting for a greeting. Below is my side of the correspondence with its author and with the postmaster of the site where it was first observed. From: Mark Martinec mark.marti...@ijs.si To: Eugene Kurmanin ... Subject: smf-sav shoots HELO without waiting for full greeting, violating SMTP protocol Date: Tue, 27 Oct 2009 15:41:40 +0100 I know the smf-sav hasn't been maintained for a while now, but my guess is that some sites are still using it. I wonder if you are aware of the SMTP protocol violation caused by smf-sav, which can cause the receiving MTA to drop the connection right away, so subsequently the SAV fails on the sending site. The key problem here is that a SMTP protocol allows continuation of responses (a '-' instead of a ' ' after the status code), while the smf-sav fails to read all response lines first, but goes straight to sending a HELO command. A more picky MTAs will disconnect the session at that point. Below is my message to postmasters of Max Planck Institute for chemical physics of solids, which possibly is using your software, according to tcpdump. Please consider fixing the issue in the next release. ( my message to mpg.de : ) While investigating mail delivery problems for messages sent from our institute (e.g. from x...@ijs.si) to xxx.mpg.de, I discovered that the reason for rejections like: 550 5.7.1 x...@xxx.mpg.de... Sender address verification failed is that your sender address verification implementation violates the SMTP protocol by shooting out a command: HELO mailgate.xxx.mpg.de without first waiting for the reception of the complete greeting message. I have now listed the xxx.xxx.xxx.xxx and xxx.xxx.xxx.xxx as broken mailers, so as to ignore the violation in the future, but it would be nice to fix your SAV, as it may give you trouble with other mail as well. I wonder what software are you using for the SAV component. Could it be that you are using an old smf-sav milter, which is known to suffer from this defect? [...] Eugene writes: Sure, you are right. This bug was fixed in commercial software more than 2 years ago. This is a fixed code (replaced code of buggy function): Hmmm. I don't think this will work reliably. I see two problems there. One is that it assumes the SMTP response lines will start at a TCP segment boundaries - which happens to be true in most cases, but may not be, perhaps due to anti-spam firewalls stalling the SMTP response and feeding it back by a slow trickle in several packets, or perhaps due to packet fragmentation. The SMTP response code can only start at a line boundary (first line, or following a CRLF). The other problem, which affects us here directly is that the while loop exits as soon as it encounters a LF at the end of a TCP segment. It should continue reading the response continuation lines for as long as the continuation character indicates 'more' (i.e. '2xx-...'). Let me illustrate what happens in case of a Postfix MTA with postscreen enabled. It sends it greeting in two lines, the first line indicates (with a '-') that continuation follows. There is also a few seconds pause between the first and the second response line: 220-mail.ijs.si ESMTP Postfix 220 mail.ijs.si ESMTP Postfix Only after a line with no 'more' mark is received, is the client allowed to speak. So I don't see a shortcut way out here, the client code needs to recognize the concept of 'lines', and it must understand the concept of multiline SMTP responses to be truly SMTP-compliant. I've seen other mailers responding with multi-line greetings too. I think some OpenBSD anti-spam sw is even more challenging to the client, breaking the initial response into multiple TCP segments, with pauses inbetween. The idea is to allow SMTP-compliant clients to send mail, while stopping some quick-and-dirty malware mail-sending software which tries to cut corners. Regards Mark But it works, as far as I know :) It was tested with OpenBSD spamd anti-spam daemon smth like that techniques. Here is a SMTP SAV session as captured by a tcpdump between my [...] computer running sendmail + smf-sav with your patch applied, and our institute's main mailer (as a SAV response to my message sent from office to home): 220-mail.ijs.si ESMTP Postfix HELO xxx.ijs.si 521 5.5.1 Protocol error RSET As you can see, the smf-sav does not wait for the complete greeting message, it only waits until the end of the first line. The greeting is a multiline response with a short pause between the first line and its continuation line: 220-mail.ijs.si ESMTP Postfix (pause) 220 mail.ijs.si ESMTP Postfix and the client must not send a SMTP command until it receives the complete SMTP response (as dictated by RFC 2821). The session should have looked like: 220-mail.ijs.si ESMTP Postfix 220 mail.ijs.si ESMTP Postfix HELO xxx.ijs.si 250 mail.ijs.si
Re: Seeking recommendation for before-queue content filter capable of removing headers
Ralph,
On 12.09.10 10:46, mouss wrote:
Received headers should not be included in the DKIM signature. so
removing them won't invalidate DKIM.
If you have a look at my message which you quoted, you'll see
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=seichter.de; h=
content-transfer-encoding:content-type:content-type:in-reply-to
:references:subject:subject:mime-version:user-agent:from:from
:date:date:message-id:received:received; s=sep2010; t=
1284280386; x=1286872386; bh= [...]
My interpretation of this is that amavisd-new will DKIM-sign Received:
headers if they are present.
Yes, by default Received header fileds are signed by amavisd-new
(it does not violate RFC 4871, it just does not match its suggestion),
but this is configurable.
amavisd-new-2.6.0 release notes
An associative array %signed_header_fields controls which header fields
are to be signed. By default it contains a standard (RFC 4871) set of
header field names, augmented by some additional header field names
considered appropriate at the time of a release (RFC 4021, RFC 3834).
In addition a 'Sender' header field is excluded because it is frequently
replaced by a mailing list, and as the RFC 2821 mandates there can only
be one such header field the original one is dropped, invalidating a
signature. Also the 'To' and 'Cc' are excluded from a default set because
sendmail mailers are known to gratuitously reformat the list, invalidating
a signature. A value in key/value pairs of %signed_header_fields is
currently interpreted as a boolean, but stick to values 0 and 1 for now
to allow for future enhancements.
The default set of header fields to be signed can be controlled
by setting %signed_header_fields elements to true (to sign) or
to false (not to sign). Keys must be in lowercase, e.g.:
$signed_header_fields{'received'} = 0; # turn off signing of Received
$signed_header_fields{'sender'} = 1;# turn on signing of Sender
$signed_header_fields{'to'} = 1;# turn on signing of To
$signed_header_fields{'cc'} = 1;# turn on signing of Cc
Maybe there's something amiss anyway, because
subject:subject or date:date look kind of strange?
This is intentional, but is configurable:
amavisd-new-2.6.4 release notes
- support inclusion of null header field names in an 'h' tag of a DKIM
signature generated by amavisd for specified header field names, thus
preventing third parties from prepending additional occurrences of
these header fields without breaking a signature; useful for example
for protecting a recipient's mail reader or a filter from being tricked
by supplying a duplicate From or Subject header fields; the protection is
requested by specifying a value larger than 1 in %signed_header_fields,
e.g.:
$signed_header_fields{'from'} = 2;
$signed_header_fields{'subject'} = 2;
$signed_header_fields{'message-id'} = 2;
$signed_header_fields{'content-type'} = 2;
Please restrict values used in %signed_header_fields to 0, 1, or 2,
consider other values reserved for future use.
By default the following header fields are protected from duplicates by
a DKIM signature generated by amavisd: From, Date, Subject, Content-Type.
To revert to a classical behaviour, set their value in %signed_header_fields
to 1, e.g.:
$signed_header_fields{lc($_)} = 1 for qw(From Date Subject Content-Type);
Mark
Re: Seeking recommendation for before-queue content filter capable of removing headers
Ralph,
A1: Have Postfix accept mail on port 587 from SASL-authenticated
clients only.
A2: Remove Received: headers to hide internal routing.
A3: Inject mail into amavisd-new for virus checks and DKIM signing.
With the help of custom hooks this can be achieved by amavisd itself:
amavisd.conf:
include_config_files('/etc/amavisd-custom.conf');
/etc/amavisd-custom.conf:
package Amavis::Custom;
use strict;
BEGIN {
import Amavis::Conf qw(c cr ca);
import Amavis::Util qw(do_log);
}
sub new {
my($class,$conn,$msginfo) = @_;
my($self) = bless {}, $class;
$self;
}
sub before_send {
my($self,$conn,$msginfo) = @_;
if (c('originating')) {
my($hdr_edits) = $msginfo-header_edits;
$hdr_edits-delete_header('Received');
}
}
1; # insure a defined return
If you need to be more selective and delete only selective
Received header fields, this is a bit more tricky, but doable.
Instead of $hdr_edits-delete_header(...) use the following:
$hdr_edits-edit_header('Received',
sub { my($hn,$b)=...@_; $b=~/from xxx \[10\./si ? (undef,0) : ($b,1) });
matching the regexp to your needs. The $b is a header field body,
the result is a replacement body, or undef to delete it.
$signed_header_fields{lc('Received')} = 0;
@Mark Martinec (in case you're reading this): Do you think
this would make a reasonable default setting for amavisd-new?
It is certainly reasonable, but I most likely won't be changing
the default. The reason the DKIM document suggests not to sign
Received header fields is for fear that MTAs in the chain may
modify them and thus break a signature. In my experience this
practically never happens. It is much more likely that other
header fields get mangled, such as To, Cc, Sender, Message-ID.
The Received header field if one of the few which survives
practically intact even the more obscure mailers.
Mark
Re: timeout trouble with postfix and amavisd in BQCF
Patrick, Versions before amavisd-new 2.7.0 and SA older than 3.3.0 are not particularly suitable for a pre-queue filtering setup. The combined new features of 2.7.0, SA 3.3.* and the postfix 'speed_adjust' made such a setup much better behaved. Please read the introductory sections of 2.7.0 (pre)release notes: http://www.ijs.si/software/amavisd/release-notes.txt Sep 2 13:00:47 ru amavis[87682]: (87682-15) TIMING [total 257879 ms] - SMTP greeting: 25055 (10%)10, 25 s to get a greeting probably means there were more incoming sessions than the number of available amavisd processes. The smtpd_proxy_options=speed_adjust can help here. SMTP DATA: 24052 (9%)19, The 2.7.0-pre7 brings a 4-fold speedup in receiving of large mail messages over SMTP. On our mailer the receiving transfer rate (postfix-amavisd) is now about 32 MiB/s. check_init: 25053 (10%)29, Hard to say what is going on here. Either a very busy machine, or maybe access contention on a Berkeley DB. gen_mail_id: 21050 (8%)37, A slow/overloaded SQL server? mime_decode: 21063 (8%)45, A very busy machine, or huge mail - probably both. AV-scan-1: 21058 (8%)53, Very busy machine or a very slow virus scanner. decide_mail_destiny: 25156 (10%)63, Explicable only by a very much overloaded host, or memory shortage and swapping. fwd-connect: 64265 (25%)88, postfix too busy to accept another connection??? Or perhaps you have too low a limit on the number of back-end smtpd processes. nop, it's mostly shared memory here, With 80 amavisd process, I've still 2GB RAM free, and absolutely no swap. With 120 amavisd process and a max of 120 smtpd, I still have more than 1.5 MB of free RAM, and no swap. If you are not running SpamAssassin, the 80 processes on a 4 Gb host is probably still reasonable. So it must be the CPU overload, not a memory exhaustion. Also, to RUN those 80 amavis threads, I would start with at least 4 cores. I do agree about cores. The VM what tuned for 30-40 amavisd, I should double the CPU, or remove amavisd and smtpd processes. I think I'm going to remote amavisd/smtpd processes. I've found out few hours after sending my question here that updating few perl modules could be the solution to my problem. I'm writing could be, because I'm still not sure it's THE solution. After updating few perl modules (including NET::Socket) and restarted amavisd, it immediately started to work great. It does not guaranty it will not break under load again. Maybe, although I'm not aware of any performance-related problems with underlying perl modules. More processors would definitely help. Mark
Re: timeout trouble with postfix and amavisd in BQCF
Sep 2 13:00:47 ru amavis[87682]: (87682-15) TIMING [total 257879 ms] - SMTP greeting: 25055 (10%)10, SMTP EHLO: 0 (0%)10, SMTP pre-MAIL: 0 (0%)10, SMTP pre-DATA-flush: 7 (0%)10, SMTP DATA: 24052 (9%)19, check_init: 25053 (10%)29, digest_hdr: 1 (0%)29, digest_body: 0 (0%)29, gen_mail_id: 21050 (8%)37, mime_decode: 21063 (8%)45, get-file-type1: 21 (0%)45, decompose_part: 1 (0%)45, parts_decode: 0 (0%)45, check_header: 2 (0%)45, AV-scan-1: 21058 (8%)53, spam-wb-list: 5 (0%)53, update_cache: 2 (0%)53, decide_mail_destiny: 25156 (10%)63, fwd-connect: 64265 (25%)88, fwd-xforward: 1 (0%)88, fwd-mail-pip: 5 (0%)88, fwd-rcpt-pip: 1 (0%)88, fwd-data-chkpnt: 0 (0%)88, write-header: 2 (0%)88, fwd-data-contents: 0 (0%)88, fwd-end-chkpnt: 4 (0%)88, prepare-dsn: 1 (0%)88, main_log_entry: 12 (0%)88, update_snmp: 2 (0%)88, SMTP pre-response: 31057 (12%)100, SMTP response: 0 (0%)100, unlink-2-files: 1 (0%)100, rundown: 2 (0%)100 Machine does not look busy at all during those problems. Load is under 0.5 and CPU is 90% idle. Even small emails are affected. If the host is not busy, again, my primary suspect is a berkeley db. These multiples of 20..25 second delays, some at inexplicable sections, seem to coincide with updating a child process status in the nanny database. Try disabling it altogether: $enable_db=0; If that helps, consider upgrading libdb to a more recent version (along with the BerkeleyDB perl module). Mark
Better logging for a unix socket connection failure in a proxy filtering setup
I'd like to report a rather minor/cosmetic problem - namely a lack of useful logging when an smtpd service tries to connect to a proxy content filter over a Unix socket which is too heavily protected - but which took me far longer to understand than necessary (the strong protection was on a parent directory, not on a socket itself, which obscured the obvious). All I got was a '451 4.3.0 Error: queue file write error' for the client, and just a disconnect and a double bounce in the log. Turning on verbosity on smtpd did not help to explain the issue. (version: postfix-current-2.8.20100728 from ports @ FreeBSD 8.1) In the end I tried a post-queue setup, and the failing lmtp service was far more helpful, clearly reporting a 'Permission denied' and giving the socket name. Fixing the socket permission problem resolved the original proxy setup case too. My test setup (proxy over a Unix socket): 10111inet n- n - - smtpd -o smtpd_proxy_filter=unix:/var/amavis/amavisd.sock -o mynetworks=127.0.0.0/8,[::1] -o smtpd_client_restrictions=permit_mynetworks,reject # telnet 127.0.0.1 10111 220 xxx.ijs.si ESMTP Postfix ehlo bla 250-[...] mail from:mark.marti...@ijs.si 250 2.1.0 Ok rcpt to:mark.marti...@ijs.si 451 4.3.0 Error: queue file write error quit 221 2.0.0 Bye postfix/smtpd[55739]: connect from localhost.ijs.si[127.0.0.1] postfix/cleanup[55754]: 7E4A510E: message-id=20100902231515.7e4a5...@xxx.ijs.si postfix/smtpd[55739]: disconnect from localhost.ijs.si[127.0.0.1] postfix/qmgr[55737]: 7E4A510E: from=double-bou...@xxx.ijs.si, size=841, nrcpt=1 (queue active) postfix/cleanup[55754]: ACD2010F: message-id=20100902231515.7e4a5...@xxx.ijs.si postfix/local[55755]: 7E4A510E: to=postmas...@xxx.ijs.si, orig_to=postmaster, relay=local, delay=0.21, delays=0.19/0.01/0/0.02, dsn=2.0.0, status=sent (forwarded as ACD2010F) postfix/qmgr[55737]: ACD2010F: from=double-bou...@xxx.ijs.si, size=969, nrcpt=1 (queue active) postfix/qmgr[55737]: 7E4A510E: removed postfix/smtp[55756]: ACD2010F: to= [...] 250 2.0.0 Ok: queued as 2359B1D1C5B postfix/qmgr[55737]: ACD2010F: removed With a verbosity turned on (smtpd -v) the log is just a little bit more specific, but still fails to report the permission problem connecting to the unix socket: [...] postfix/smtpd[55877]: generic_checks: name=permit_mynetworks postfix/smtpd[55877]: permit_mynetworks: localhost.ijs.si 127.0.0.1 postfix/smtpd[55877]: match_hostname: localhost.ijs.si ~? 127.0.0.0/8 postfix/smtpd[55877]: match_hostaddr: 127.0.0.1 ~? 127.0.0.0/8 postfix/smtpd[55877]: generic_checks: name=permit_mynetworks status=1 postfix/smtpd[55877]: smtpd_check_rewrite: trying: permit_inet_interfaces postfix/smtpd[55877]: permit_inet_interfaces: localhost.ijs.si 127.0.0.1 postfix/smtpd[55877]: smtp_get: EOF postfix/smtpd[55877]: warning: lost connection with proxy unix:/var/amavis/amavisd.sock postfix/smtpd[55877]: localhost.ijs.si[127.0.0.1]: 451 4.3.0 Error: queue file write error postfix/smtpd[55877]: localhost.ijs.si[127.0.0.1]: quit postfix/smtpd[55877]: localhost.ijs.si[127.0.0.1]: 221 2.0.0 Bye Now here below is the post-queue filtering setup, letting an lmtp service talk to the same content filter's Unix socket as above. The logging is clear and to the point in this case: 10111inet n- n - - smtpd -o content_filter=testfeed:unix:/var/amavis/amavisd.sock -o mynetworks=127.0.0.0/8,[::1] -o smtpd_client_restrictions=permit_mynetworks,reject testfeed unix - - n - 50 lmtp -o lmtp_send_xforward_command=yes postfix/lmtp[55986]: 61C8A117: to=mark.marti...@ijs.si, relay=none, delay=24, delays=24/0/0/0, dsn=4.4.1, status=deferred (connect to xxx.ijs.si[/var/amavis/amavisd.sock]: Permission denied) Perhaps an improved error logging could save somebody the effort of troubleshooting. Btw, for the curios: the result of my experiment is that feeding a message to a proxy content filter over a Unix socket is sometimes as fast (Mb/s) as over an inet socket, and sometimes about 27% slower. The lines with two distinct slopes clearly emerge on a diagram of elapsed time vs. message size for a DATA phase. No idea why, but I won't bother to investigate, as the inet is clearly a winner at times, or an exact even otherwise. Funny. Mark
Re: Better logging for a unix socket connection failure in a proxy filtering setup
All I got was a '451 4.3.0 Error: queue file write error' for the client, and just a disconnect and a double bounce in the log. Turning on verbosity on smtpd did not help to explain the issue. (version: postfix-current-2.8.20100728 from ports @ FreeBSD 8.1) By design, Postfix does not report error details to SMTP clients. I'm complaining that it does not LOG the permission problem. postfix/lmtp[55986]: 61C8A117: to=mark.marti...@ijs.si, relay=none, delay=24, delays=24/0/0/0, dsn=4.4.1, status=deferred (connect to xxx.ijs.si[/var/amavis/amavisd.sock]: Permission denied) Perhaps an improved error logging could save somebody the effort of troubleshooting. What is unclear about connect to site/[pathname]: Permission denied? This one (lmtp/post-queue) is perfect. I'm complaining about the other one, the proxy setup, which does not LOG the reason for a failure. Mark
Re: Delayed-ACK holdups to a proxy content filter on lo0 for mid-size messages
On Friday August 27 2010 19:06:02 Victor Duchovni wrote: Just so everyone else is clear on the context, this is not a post-queue content_filter issue (post-queue content filters use the SMTP/LMTP delivery agent which already does the right thing). This applies only to the pre-queue proxy filters. True. Post-queue content filtering setup was not affected. You could try the following patch: That was fast!!! Looks good, both a test case and later our main mailer now behave as they should, no more 100ms entries in the logs. I'll grep the logs on Monday to re-gather statistics just in case, but it seems the patch does the right thing. Thank you! Mark
db50 (DB11gR2) - Unsupported Berkeley DB version
postfix-2.8-20100323, FreeBSD ports: mail/postfix-current, databases/db50 /etc/make.conf: WITH_BDB_VER=50 --- src/util/dict_db.c~ 2010-01-02 22:28:08.0 +0100 +++ src/util/dict_db.c 2010-06-11 15:50:48.0 +0200 @@ -676,5 +676,5 @@ if (type == DB_HASH db-set_h_nelem(db, DICT_DB_NELM) != 0) msg_fatal(set DB hash element count %d: %m, DICT_DB_NELM); -#if (DB_VERSION_MAJOR == 4 DB_VERSION_MINOR 0) +#if (DB_VERSION_MAJOR == 4 DB_VERSION_MINOR 0 || DB_VERSION_MAJOR == 5) if ((errno = db-open(db, 0, db_path, 0, type, db_flags, 0644)) != 0) msg_fatal(open database %s: %m, db_path); Seems to work fine. There wasn't any API change in db-open between 4.8 and 5.0 as far as I can tell. Mark
Re: All email forward a copy to testing server
Patric, I looked in to it a little more and it looks like Maia re-writes the new.sub.domain.com to sub.domain.com. I get: /usr/sbin/amavisd-new[22834]: (22834-04) Checking: [62.127.194.20] patric.falin...@omg.nu - patric.falin...@sub.domain.com,patric.falin...@sub.domain.com When I guess it should be: /usr/sbin/amavisd-new[22834]: (22834-04) Checking: [62.127.194.20] patric.falin...@omg.nu - patric.falin...@sub.domain.com,patric.falin...@new.sub.domain.com Maybe this is more of a Maia problem so I will ask there if no one here knows whats wrong. I very much doubt it is the Maia doing a rewrite. More likely your smtp_generic mapping or masquerading. Keep in mind that a post-queue content filtered message goes through Postfix twice. Mark
Re: amavis Delivery status notification(DSN) failing
Ashish, Attached is the full level 5 log for your reference. Thank you! Apr 9 07:17:31 ip-10-194-99-63 amavis[18885]: (18885-05) (about to connect to [127.0.0.1]:10030) FWD via SMTP: ashiish.sha...@gmail.com - ida6786ombo...@dev1.cpgtest.ostinet.net Apr 9 07:17:31 ip-10-194-99-63 amavis[18885]: (18885-05) smtp session: setting up a new session Apr 9 07:17:31 ip-10-194-99-63 amavis[18885]: (18885-05) establish_or_refresh, state: down Apr 9 07:17:31 ip-10-194-99-63 amavis[18885]: (18885-05) smtp creating socket by IO::Socket::INET to [127.0.0.1]:10030 Apr 9 07:17:31 ip-10-194-99-63 amavis[18885]: (18885-05) rw_loop: needline=1, flush=0, wr=0, timeout=35 Apr 9 07:17:31 ip-10-194-99-63 amavis[18885]: (18885-05) rw_loop: receiving Apr 9 07:17:31 ip-10-194-99-63 amavis[18885]: (18885-05) rw_loop read 10 chars 220 Hello\n Apr 9 07:17:31 ip-10-194-99-63 amavis[18885]: (18885-05) rw_loop: needline=1, flush=0, wr=0, timeout=35 Apr 9 07:17:56 ip-10-194-99-63 postfix/smtpd[19431]: disconnect from mail-pw0-f48.google.com[209.85.160.48] Apr 9 07:18:06 ip-10-194-99-63 amavis[18885]: (18885-05) (!)rw_loop: leaving rw loop, no progress, last event (select) 34.994 s ago Your java filter sent a greeting: 220 Hello\n instead of: 220 Hello\r\n. Amavisd waited 30 seconds but end of line (CR LF) never arrived, so the session was aborted. RFC 5321 (and RFC 2821 and RFC 821) requires that SMTP commands and replies are terminated by a CRLF, not by a single LF character. Mark
Re: amavis Delivery status notification(DSN) failing
Ashish, Your java filter sent a greeting: 220 Hello\n instead of: 220 Hello\r\n. Amavisd waited 30 seconds but end of line (CR LF) never arrived, so the session was aborted. RFC 5321 (and RFC 2821 and RFC 821) requires that SMTP commands and replies are terminated by a CRLF, not by a single LF character. Accepted your point, but I am actually doing what you have suggested already. Maybe the proof lies in the fact that my custom content filter is working fine when amavisd is not used and postfix is directly communicating with my component. I am not receiving such problem in that case. Is it because postfix is little lax in enforcing smtp for content filter? Yes, it is. (in general, not specific to content filter interfacing) Mark
Re: amavis Delivery status notification(DSN) failing
Ashish, I have a postfix mail server over which I have deployed a custom content filter written in java. Now I introduced amavisd (containing clamav and spamassassin) as content filter such that the mail is passing in following manner: ===mail from outside === Postfix amavisd === custom content filter I referred the link(http://wiki.centos.org/HowTos/Amavisd) to setup amavisd(containing clamav and spamassassin) Here my setup is running ok when amavisd is bypassed and mail is directly being sent to custom content filter but I receive following error logs as soon as I start passing my mails via amavisd: Apr 7 11:36:54 ip-10-194-99-63 amavis[3264]: (03264-02) (!)rw_loop: leaving rw loop, no progress, last event (select) 34.987 s ago Apparently your custom content filter is not responding with a SMTP greeting when amavisd tries to send a delivery notification to it. At log level 5 you can see what traffic is being passed on to the back-end server/scanner specified by $notify_method . Or do a packet capture on a loopback interface if using inet socket. Make sure your backend filter can handle multiple transactions within a single SMTP session. Btw, do you really want to send bounces too to the java filter? These could be sent directly to Postfix ($notify_method), while filtered messages could go to your filter ($forward_method). Mark
Re: log message
Jon L Miller: postfix/postsuper[4932]: warning: bogus file name: hold/razor-agent.log Some NON-POSTFIX software is leaving its NON-POSTFIX garbage in the Postfix queue. Sounds like a MailScanner issue. Mark
Re: My postfix server sometimes send command less than 4 alphabets
I'm using content filter, which parses email from my postfix server. My postfix server sometimes sends a command which is less than 4 alphabets. I don't know what to do for that command, as I don't know which command is that... Can anybody tell me, is there any command of less than 4 alphabets, postfix sends 4 letters. Hmm. EXPN, VRFY, HELO, EHLO, STARTTLS, ETRN,... Hm, I'd think the commands are 4 letters or more. Maybe a firewall bug: http://fanf.livejournal.com/102206.html Mark
Re: Multiple From: in a mail header?
On Friday January 15 2010 09:11:27 Kārlis Repsons wrote: But have you seriously seen a mail client, which would allow sending such mail? I would think, this is an extreme rarity, but is it? It is very rare alright. Multiple author addresses in a single From header field are legitimate, but some mail processing software breaks on them. Multiple From header fields are prohibited by rfc, but that does not stop malicious or broken senders from doing it if they feel like it. If one or the other turns out to be profitable for malware, it will be used, no doubt about it, so better be ready. Btw, of the header fields that may occur only once, it is currently more usual to see multiple Message-ID, or Subject, or To or Cc, or MIME-Version, or Content-Type. Very rare are duplicate Reply-To or Date. The least common is to see multiple From. Mark
Re: Multiple From: in a mail header?
On Thursday January 14 2010 20:14:48 Victor Duchovni wrote: It may be prudent to also treat: From: authorA From: authorB as synonymous with: From: authorA, authorB the implied meaning is that the people with those email addresses, co-authored the email. ...or treated with utmost suspicion, as that could be an attempt to fool a MUA + a content filter combination to let mail through but still show to a reader what was desired by malware. Note that a message with a DKIM signature with only one occurrence of a 'from' in its 'h' tag is covering the bottommost instance of a 'From' header field, yet a MUA might show the topmost. When amavisd-new (since version 2.6.4) is DKIM-signing a message, it inserts a :from:from: into the 'h' tag, which makes a later appending of another From header field invalidate a signature. Mark
Re: does order of postscreen_* params matter?
Wietse Venema wrote: The postscreen manpage lists the tests in the order of execution. Thus, the blacklist is done tested first. If the client is not blacklisted, then the whitelist test is done. And so on. I could swap the order of black/white tests if there is agreement that the current order is not optimal, but something has to go first. That reminds me of a mistake in early tcp_wrappers, which kept allow and deny lists separate - later fixed by introducing hosts_options(5) which allowed mixing of allow and deny entries. The same mistake was repeated by Apache httpd config, which works around the problem of having two lists instead of one merged list by a horrible kluge in a form of an 'Order' directive. There is no 'one size fits all' right order of allow/deny or black/white lists, if these are limited to one list of each type. Conceivably one could need a: white 10.1.1.1, black 10.1.1.0/24, white 10.0.0.0/8 Some solution like a postfix access(5) or cidr_table(5) table would be best, with entries each claiming an OK/REJECT or whatever is documented. Mark
Re: PATCH: smtpd_proxy logging
Wietse Venema wie...@porcupine.org: Like this? Dec 5 20:15:25 server postfix/smtpd[16712]: proxy-accept: END-OF-MESSAGE: 250 2.0.0 Ok: queued as 91BE3547AFE; from=sen...@example.com to=recipi...@example.com proto=ESMTP helo=client.example.com (with the same form for proxy-reject at END-OF-MESSAGE; the format of the reject message would be consistent with other Postfix reject messages.) Works perfectly, thank you! On Sunday 06 December 2009 13:11:16 Stefan Foerster wrote: 2. The number of SMTP proxy processes busy talking to Postfix went down from 16.44 to 8.73 (5s resolution) - that's a completely unexpected result, possibly flawed by the small data sample. I'll verify this again once I got a full month's data. A drop in the number of necessary content filtering processes is to be expected, and is in fact the main reason for the smtpd_proxy_options=speed_adjust feature. Very useful / indispensable in the pre-queue filtering setup. Mark
Re: Postfix DKIM
On Tuesday 24 November 2009 20:38:51 Michael Saldivar wrote: On Wed, Sep 9, 2009 at 8:08 PM, KLaM Postmaster postmas...@klam.ca wrote: I found the easiest way by far, was to use the DKIM feature of amavisd-new http://www.ijs.si/software/amavisd/amavisd-new-docs.html simple to setup and work like a charm. I couldn't tell from the docs: http://www.ijs.si/software/amavisd/amavisd-new-docs.html#dkim Does amavisd-new insert DomainKeys signatures also? or just DKIM? Just DKIM. I'm impressed at how easy it looks to implement and configure for multiple domains, but we need both DK and DKIM. The DomainKeys is historical. Most checkers now understand both, so there is not much need for new installations to also provide a DomainKeys signature, when they can generate a DKIM signature. Mark
Experience with the new speed_adjust feature
For the last couple of days I'm now experimenting with the 2.7-20091105-nonprod with the new speed_adjust experimental feature turned on at the MX port, along with the postscreen. Seems to work as advertised: timing reports by a pre-queue proxy content filter confirm that the content filter is invoked only after data has been received. Current mail load on our server is not high, so I haven't yet seen what happens under high mail rate conditions. Here is the relevant part of my master.cf: dnsblog unix - - n - 0 dnsblog smtp inet n - n - 1 postscreen smtpdpass - - n - 150 smtpd -o smtpd_proxy_filter=127.0.0.1:10010 -o smtpd_proxy_options=speed_adjust -o smtpd_client_connection_count_limit=30 -o smtpd_proxy_timeout=600 Two things are on my mind: 1. I'm dearly missing in the postfix log the SMTP response from a pre-queue content filter (or better, a response that was sent back to a SMTP client), at least in case of a 5xx or 4xx response. With a post-queue filter setup the response is clearly logged, unlike in the pre-queue setup where it is not. It makes it hard to correlate a spartanic postfix log of a rejected transaction (due to a proxy filter rejection) with a log from a content filter. Even if the correlation were somehow available, seeing a confirmation of what was really sent back to the client in a postfix log would contribute to the confidence in the correctness of operation. 2. A couple of times per day I see that postfix terminates its smtpd service: +pid 9297 (smtpd), uid 125: exited on signal 6 +pid 12680 (smtpd), uid 125: exited on signal 6 +pid 9306 (smtpd), uid 125: exited on signal 6 and a corresponding log entry from a smtpd process: panic: smtpd_proxy_replay_setup: non-empty before-queue filter speed-adjust log The log (some other PIDs) can look like (a grep by PID, showing previous two transaction of the same process): Nov 13 13:07:07 dorothy postfix/smtpd[82601]: connect from unknown[74.63.109.233] Nov 13 13:07:08 dorothy postfix/smtpd[82601]: NOQUEUE: client=unknown[74.63.109.233] Nov 13 13:07:12 dorothy postfix/smtpd[82601]: disconnect from unknown[74.63.109.233] Nov 13 13:07:29 dorothy postfix/smtpd[82601]: connect from 201-92-220-64.dsl.telesp.net.br[201.92.220.64] Nov 13 13:07:30 dorothy postfix/smtpd[82601]: NOQUEUE: client=201-92-220-64.dsl.telesp.net.br[201.92.220.64] Nov 13 13:07:36 dorothy postfix/smtpd[82601]: disconnect from 201-92-220-64.dsl.telesp.net.br[201.92.220.64] Nov 13 13:07:43 dorothy postfix/smtpd[82601]: warning: 122.168.228.121: hostname ABTS-mpdynamic-121.228.168.122.airtelbroadband.in verification failed: hostname nor servname provided, or not known Nov 13 13:07:43 dorothy postfix/smtpd[82601]: connect from unknown[122.168.228.121] Nov 13 13:08:26 dorothy postfix/smtpd[82601]: NOQUEUE: client=unknown[122.168.228.121] Nov 13 13:08:26 dorothy postfix/smtpd[82601]: lost connection after RCPT from unknown[122.168.228.121] Nov 13 13:08:26 dorothy postfix/smtpd[82601]: disconnect from unknown[122.168.228.121] Nov 13 13:08:32 dorothy postfix/smtpd[82601]: connect from unknown[121.139.242.179] Nov 13 13:08:34 dorothy postfix/smtpd[82601]: panic: smtpd_proxy_replay_setup: non-empty before-queue filter speed-adjust log Here is another case: Nov 13 13:52:43 dorothy postfix/smtpd[4186]: NOQUEUE: client=200-199-119-208.user.neoline.com.br[200.199.119.208] Nov 13 13:52:53 dorothy postfix/smtpd[4186]: disconnect from 200-199-119-208.user.neoline.com.br[200.199.119.208] Nov 13 13:52:58 dorothy postfix/smtpd[4186]: connect from unknown[77.29.4.99] Nov 13 13:53:01 dorothy postfix/smtpd[4186]: NOQUEUE: client=unknown[77.29.4.99] Nov 13 13:54:23 dorothy postfix/smtpd[4186]: lost connection after DATA (1589 bytes) from unknown[77.29.4.99] Nov 13 13:54:23 dorothy postfix/smtpd[4186]: disconnect from unknown[77.29.4.99] Nov 13 13:54:28 dorothy postfix/smtpd[4186]: connect from unknown[170.51.195.142] Nov 13 13:54:31 dorothy postfix/smtpd[4186]: panic: smtpd_proxy_replay_setup: non-empty before-queue filter speed-adjust log So what does this mean? Mark
Re: Experience with the new speed_adjust feature
On Friday 13 November 2009 14:48:27 Wietse Venema wrote: 20091105-nonprod has a known problem when the temp file can't be written for some reason (fixed in 20091109). As for the second problem, it would help if you could add a missing sanity check here: Thanks. Done both: upgraded to 20091109 and applied the patch. Assume that the maximal number of filter processes is tuned such that the machine does not collapse under that load. For the time being it is all very comfortable, less then 10% of available content filtering processes are actually busy. It will help to get closer to the root cause. So far no smtpd_proxy_create: handle still exists were logged, despite several panics. I have simplified the setup, no more postscreen, and TLS is no longer offered, but the problem remains. Here is another fresh sample, this one covers the full lifetime of one smtpd process (full grep by its PID): Nov 13 17:42:44 dorothy postfix/smtpd[28345]: warning: 190.205.148.176: hostname 190-205-148-176.dyn.dsl.cantv.net verification failed: hostname nor servname provided, or not known Nov 13 17:42:44 dorothy postfix/smtpd[28345]: connect from unknown[190.205.148.176] Nov 13 17:42:46 dorothy postfix/smtpd[28345]: NOQUEUE: reject: RCPT from unknown[190.205.148.176]: 550 5.1.8 cb...@colossus.cantv.net: Sender address rejected: Domain not found; from=cb...@colossus.cantv.net to=alf...@anzlovar.com proto=ESMTP helo=colossus.cantv.net Nov 13 17:42:48 dorothy postfix/smtpd[28345]: NOQUEUE: reject: RCPT from unknown[190.205.148.176]: 550 5.1.8 te...@colossus.cantv.net: Sender address rejected: Domain not found; from=te...@colossus.cantv.net to=alf...@anzlovar.com proto=ESMTP helo=colossus.cantv.net Nov 13 17:42:50 dorothy postfix/smtpd[28345]: NOQUEUE: reject: RCPT from unknown[190.205.148.176]: 550 5.1.8 tenghuitr...@colossus.cantv.net: Sender address rejected: Domain not found; from=tenghuitr...@colossus.cantv.net to=x...@xxx.com proto=ESMTP helo=colossus.cantv.net Nov 13 17:42:51 dorothy postfix/smtpd[28345]: disconnect from unknown[190.205.148.176] Nov 13 17:42:56 dorothy postfix/smtpd[28345]: connect from gtwout2.ill.fr[193.49.43.102] Nov 13 17:42:56 dorothy postfix/smtpd[28345]: NOQUEUE: client=gtwout2.ill.fr[193.49.43.102] Nov 13 17:42:59 dorothy postfix/smtpd[28345]: disconnect from gtwout2.ill.fr[193.49.43.102] Nov 13 17:42:59 dorothy postfix/smtpd[28345]: warning: 189.20.244.170: hostname 189-20-244-170.customer.tdatabrasil.net.br verification failed: hostname nor servname provided, or not known Nov 13 17:42:59 dorothy postfix/smtpd[28345]: connect from unknown[189.20.244.170] Nov 13 17:43:01 dorothy postfix/smtpd[28345]: NOQUEUE: client=unknown[189.20.244.170] Nov 13 17:43:03 dorothy postfix/smtpd[28345]: disconnect from unknown[189.20.244.170] Nov 13 17:43:08 dorothy postfix/smtpd[28345]: connect from 71-231-58-66.gci.net[66.58.231.71] Nov 13 17:43:09 dorothy postfix/smtpd[28345]: NOQUEUE: client=71-231-58-66.gci.net[66.58.231.71] Nov 13 17:43:13 dorothy postfix/smtpd[28345]: disconnect from 71-231-58-66.gci.net[66.58.231.71] Nov 13 17:43:15 dorothy postfix/smtpd[28345]: connect from cernmx13.cern.ch[137.138.142.200] Nov 13 17:43:15 dorothy postfix/smtpd[28345]: NOQUEUE: client=cernmx13.cern.ch[137.138.142.200] Nov 13 17:43:18 dorothy postfix/smtpd[28345]: NOQUEUE: client=cernmx13.cern.ch[137.138.142.200] Nov 13 17:43:25 dorothy postfix/smtpd[28345]: disconnect from cernmx13.cern.ch[137.138.142.200] Nov 13 17:43:28 dorothy postfix/smtpd[28345]: connect from bpost.kek.jp[130.87.105.111] Nov 13 17:43:29 dorothy postfix/smtpd[28345]: NOQUEUE: client=bpost.kek.jp[130.87.105.111] Nov 13 17:43:33 dorothy postfix/smtpd[28345]: disconnect from bpost.kek.jp[130.87.105.111] Nov 13 17:43:37 dorothy postfix/smtpd[28345]: connect from host-93-124-65-201.dsl.sura.ru[93.124.65.201] Nov 13 17:43:37 dorothy postfix/smtpd[28345]: NOQUEUE: client=host-93-124-65-201.dsl.sura.ru[93.124.65.201] Nov 13 17:43:41 dorothy postfix/smtpd[28345]: disconnect from host-93-124-65-201.dsl.sura.ru[93.124.65.201] Nov 13 17:43:44 dorothy postfix/smtpd[28345]: connect from backup002.host1.eu[62.75.216.27] Nov 13 17:43:46 dorothy postfix/smtpd[28345]: NOQUEUE: client=backup002.host1.eu[62.75.216.27] Nov 13 17:43:51 dorothy postfix/smtpd[28345]: disconnect from backup002.host1.eu[62.75.216.27] Nov 13 17:43:56 dorothy postfix/smtpd[28345]: connect from unknown[41.130.65.71] Nov 13 17:43:57 dorothy postfix/smtpd[28345]: NOQUEUE: client=unknown[41.130.65.71] Nov 13 17:44:02 dorothy postfix/smtpd[28345]: disconnect from unknown[41.130.65.71] Nov 13 17:44:05 dorothy postfix/smtpd[28345]: connect from unknown[195.94.133.112] Nov 13 17:44:09 dorothy postfix/smtpd[28345]: NOQUEUE: reject: RCPT from unknown[195.94.133.112]: 550 5.1.7 x...@xxx.com: Sender address rejected: undeliverable address: ... Nov 13 17:44:11 dorothy postfix/smtpd[28345]: disconnect from unknown[195.94.133.112] Nov 13 17:44:11 dorothy
Re: Experience with the new speed_adjust feature
On Friday 13 November 2009 18:52:03 Wietse Venema wrote: Thanks for the logging. If you have time, can you change the code to print information about the non-zero size? This could be a filesystem feature where ftruncate() does not reset st_size until the file is rewritten or closed (in which case my attempt to force easly release of disk blocks are in vain). This is running on FreeBSD 7.2-RELEASE-p2. The usual UFS file system. *** ./smtpd_proxy.c.orig Mon Nov 9 19:41:50 2009 --- ./smtpd_proxy.c Fri Nov 13 12:44:46 2009 Thanks, applied the patch. $ tail -500 -f /var/log/mail.log | egrep -i 'panic|smtpd_proxy|speed-adjust' Nov 13 19:04:14 dorothy postfix/smtpd[65777]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 20081 Nov 13 19:04:38 dorothy postfix/smtpd[65645]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 114 Nov 13 19:04:40 dorothy postfix/smtpd[65624]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 201 Nov 13 19:05:04 dorothy postfix/smtpd[67884]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 49 Nov 13 19:05:12 dorothy postfix/smtpd[68082]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 37 Nov 13 19:05:21 dorothy postfix/smtpd[68339]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 37 Nov 13 19:05:45 dorothy postfix/smtpd[67901]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 43 Nov 13 19:05:45 dorothy postfix/smtpd[67884]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 31 Nov 13 19:06:01 dorothy postfix/smtpd[68614]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 26 Nov 13 19:06:12 dorothy postfix/smtpd[67997]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 25 Nov 13 19:06:13 dorothy postfix/smtpd[68397]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 32 Nov 13 19:06:19 dorothy postfix/smtpd[68557]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 31 Nov 13 19:06:22 dorothy postfix/smtpd[68064]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 26 Nov 13 19:06:23 dorothy postfix/smtpd[68155]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 35 Nov 13 19:06:23 dorothy postfix/smtpd[67953]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 218 Nov 13 19:06:26 dorothy postfix/smtpd[67999]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 37 Nov 13 19:06:27 dorothy postfix/smtpd[68670]: warning: smtpd_proxy_replay_setup: non-zero speed-adjust logfile size: 38 but no more panics. Mark
Re: Experience with the new speed_adjust feature
On Friday 13 November 2009 19:17:07 Wietse Venema wrote: Victor found it (missing fflush before ftruncate). If you can back out the changes and apply the patch below. *** ./smtpd_proxy.c.orig Mon Nov 9 19:41:50 2009 --- ./smtpd_proxy.c Fri Nov 13 13:15:25 2009 Thanks, done. So far so good, running smoothly. Mark
postscreen lookalike, but in FreeBSD kernel (presentation)
Just came across this one, might be interesting. It sounds similar to postscreen's functionality: EuroBSDCon 2009: FreeBSD kernel protection measures against SMTP DDoS attacks, by Martin Blapp http://people.freebsd.org/~mbr/ http://www.ukuug.org/events/eurobsdcon2009/papers/BSDCON09-SMTP-DDoS-Final.pdf http://www.disruptiveproactivity.com/other/eurobsdcon2009/smtp-ddos-protection.mp3 Mark
before-queue proxy filter and SMTP dot stuffing sanitation
So far I lived under impression that smtpd service does some basic sanitation, de-pipelining etc to a SMTP session, before passing data to a smtpd_proxy_filter content filter. Apparently dot-stuffing sanitation is not performed, as (invalid) lines with a single leading dot can still reach a proxy content filter (which yields a warning in the log in case of amavisd). Is this expected behaviour and I should turn off the warnings? Mark
Re: rbl checks, best place + ipv6?
On Sunday August 23 2009 04:10:06 Dave Täht wrote: What I found after fighting with an exchange server that what seems to work best is assigning my first mx host to be ipv6 only, and my fallback to be a mx ipv6 and ipv4 host. My choice is to have the first MX have both the IPv6 and IPv4 addresses, but have a lower priority MX be IPv4-only. This way it should provide a fallback connectivity even if some mailer which thinks it has an IPv6 connectivity but doesn't, then fails to walk through multiple records of a multihomed host name. (even though RFC 5321 requires to try at least two records). Mark
OT: dkim-milter forked to an OpenDKIM project
For those who missed it, the dkim-milter project forked. Its principal developer is now with the OpenDKIM project. The OpenDKIM v1.0.0 brings a couple of bug fixes over the dkim-milter, and uses a new build mechanism. Mark Here is the announcement posted on 2009-08-14: == The OpenDKIM project announces availability of OpenDKIM v1.0.0, its first release. OpenDKIM is derived from dkim-milter v2.8.3, produced and maintained by Sendmail, Inc. The major change in this release is the build process which uses autoconf. The configuration files used by dkim-milter are fully compatible with OpenDKIM. The release notes list the bugs that have been fixed in this release. For more information, or to download the package, please visit http://www.opendkim.org. For general questions or discussion, please subscribe to the opendkim-users list at the above URL. To report problems, use that list or the bug trackers, also at the above URL.
Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once
On Thursday 23 April 2009 10:02:29 Jørn Odberg wrote: I can now see that the recieving side has an ESTABLISHED connection from the sender, even after the sender tell me it has lost the connection with the reciever. So it seems like something in the middle is forcing the connection to a close... I have now captured some more tcpdumping from both sides. http://postfix.jorno.net/2009_04_23-BamBib-NotBib/ The root of evil are some misguided firewall configurations which block ICMP type 3 packets. As a misguided attempt to work around the first problem, some firewalls or routers intentionally ignore the DF flag (don't fragment), and fragment a long packet anyway, instead of sending an ICMP notification and dropping a packet. And some receiving firewalls drop fragments of a packet which has a DF flag set. A workaround is sometimes to force a smaller MTU at your mailer. Or to turn off MTU discovery (= not to set a DF flag). A fix is to disable blocking of ICMP type 3 packets in firewalls (your outgoing, or recipient's incoming), and turn off the second mentioned misfeature. Mark
Re: Another SMTP protocol breakage by ASA
Ralf, here is another one for your list of Cisco PIX and ASA problems with inspection of a SMTP protocol (actually, parsing of a mail header section): http://www.arschkrebs.de/postfix/postfix_cisco_pix_bugs.shtml CSCsy28792 SMTP session disconnects due to improper parsing of a DKIM header field by ASA ...to be fixed in releases 8.1.2(22) and 8.1.3 Mark
Re: Conversation with DOMAIN timed out while sending end of data -- message may be sent more than once
Jørn, As I said in the first email, I control both ends (both the sender- and the receiver-server). But I do not control neither network-connectivity or Internet-connectivity at either sites. I did try turning of Window Scaling at both ends, but it did not help at all. It still won't deliver. I know there is a Cisco PIX at the senders side. They have already turned of the ESMTP Fixup (or fuckup if you'd like) feature by an earlier request from me, and that solved some problems. But it seems like the damn PIX has some other features which fucks up some more... Just don't know what. It is strange that the sending side transmitted the large data packet (with mail contents) with a DF (don't fragment) bit turned on, yet it is captured as two fragments on the transmitting side. Of these two fragments only the second (smaller) reaches the receiver. Looks like something is forcefully breaking packets despite a DF, and I don't find it unusual that a receiving side reluctantly discards a fragment. Mark
Re: Plus Addressing
Jeff, One more thing I noticed today also. All messages which have the + in the e-mail are sent to Dovecot's Deliver twice. So, I receive the message twice in the folder. All other messages are only sent to Deliver once. Any idea what I have configured wrong for the message to be sent twice? It probably has to do with where you implemented recipient_bcc_maps, along with a post-queue content filter which sends a checked message back to Postfix for delivery - so recipient_bcc_maps could be invoked twice. Aw, that makes sense. I have Amavis configured. I must have done something wrong with that. Here is a copy of my master.cf file. Would you be able to tell me what I have set wrong? 127.0.0.1:10025 inet n- n - - smtpd -o content_filter= [...] -o receive_override_options=no_header_body_checks, no_unknown_recipient_checks,no_milters You can add 'no_address_mappings' to the list in receive_override_options. It turns off automatic BCC, among other things. See: http://www.postfix.org/postconf.5.html#receive_override_options It this option is too strong (e.g. you don't want recipient_bcc_maps on port 10025, yet you do want canonical or virtual alias map expansion after a content filter), then you need to use a more complicated solution with two cleanup services (or two Postfix instances). See: http://www.ijs.si/software/amavisd/README.postfix.html - Multiple cleanup service architecture Mark
Another SMTP protocol breakage by ASA
Ralf, here is another one for your list of Cisco PIX and ASA problems with inspection of a SMTP protocol (actually, parsing of a mail header section): http://www.arschkrebs.de/postfix/postfix_cisco_pix_bugs.shtml CSCsy28792 SMTP session disconnects due to improper parsing of a DKIM header field by ASA Problem description: SMTP session is disconnected during DATA phase of a SMTP transaction for mail messages with a DKIM signature, where the start of a string content-type or content-transfer-encoding in a tag's value of an h tag of a DKIM signature happens to fall on a packet boundary at a start of a packet. The session is dropped with the next packet containing a Content-Type or Content-Transfer-Encoding header field. Platform: ASA5580-40 Cisco Adaptive Security Appliance Software Version 8.1(2)
Re: Plus Addressing
Jeff, One more thing I noticed today also. All messages which have the + in the e-mail are sent to Dovecot's Deliver twice. So, I receive the message twice in the folder. All other messages are only sent to Deliver once. Any idea what I have configured wrong for the message to be sent twice? It probably has to do with where you implemented recipient_bcc_maps, along with a post-queue content filter which sends a checked message back to Postfix for delivery - so recipient_bcc_maps could be invoked twice. Mark
rw_loop: leaving rw loop, no progress
-- Forwarded Message -- Subject: Re: [AMaViS-user] rw_loop: leaving rw loop, no progress Date: Friday 20 March 2009 From: Mark Martinec mark.martinec+ama...@ijs.si To: amavis-u...@lists.sourceforge.net Ivan, This is log in attached files Thanks, interesting and strange. I'll CC this to the Postfix mailing list, in case someone can provide an explanation. I have problem with amavisd-new. My system: FreeBSD 5.4, postfix-2.3.14, amavisd-new-2.6.2_2, p5-Mail-SpamAssassin-3.2.5 For each message to my server I see this error in amavis log: Mar 5 18:17:59 mydomain.ua /usr/local/sbin/amavisd[32001]: (32001-01) (!)rw_loop: leaving rw loop, no progress After 30-60 minutes message will have delivered. | It happens during fwd-connect phase, i.e. during server greeting and | EHLO exchange. It seems the Postfix smtpd service takes a long time | to respond for some reason, and amavisd times out meanwhile. | Does it happen often? When it happens to one message, is it likely | that it happens to other messages following it too? When this | happens, are you able to telnet to 127.0.0.1 port 10025 and | receive a greeting and some response to an 'EHLO foo' command? | Is there any problem with (slow?) DNS resolving of '127.0.0.1' | or 'localhost'? It's happens for 80% of all message. When this happens, I able to telnet to 127.0.0.1 port 10025 and receive a greeting and some response to an 'EHLO foo' command smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated, check_sender_access hash:/usr/local/etc/postfix/check_sender, reject_unknown_sender_domain, reject_non_fqdn_sender, reject_unverified_sender smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination content_filter=smtp-amavis:[127.0.0.1]:10024 smtp inet n - n - 5 smtpd smtp-amavis unix - - n - 5 smtp -o smtp_data_done_timeout=1200 -o smtp_send_xforward_command=yes -o disable_dns_lookups=yes -o max_use=20 127.0.0.1:10025 inet n - n - 5 smtpd -o content_filter= -o local_recipient_maps= -o relay_recipient_maps= -o smtpd_restriction_classes= -o smtpd_delay_reject=no -o smtpd_client_restrictions=permit_mynetworks,reject -o smtpd_helo_restrictions= -o smtpd_sender_restrictions= -o smtpd_recipient_restrictions=permit_mynetworks,reject -o mynetworks_style=host -o mynetworks=127.0.0.0/8 -o strict_rfc821_envelopes=yes -o smtpd_error_sleep_time=0 -o smtpd_soft_error_limit=1001 -o smtpd_hard_error_limit=1000 -o smtpd_client_connection_count_limit=0 -o smtpd_client_connection_rate_limit=0 -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks I'll summarize the events from your two logs and a tcpdump capture: 15:50:35, amavisd requests a connect to postfix at 127.0.0.1 port 10025: Mar 18 15:50:35 inspe.com.ua /usr/local/sbin/amavisd[43524]: (43524-01) smtp creating socket by IO::Socket::INET to [127.0.0.1]:10025 Mar 18 15:50:35 inspe.com.ua /usr/local/sbin/amavisd[43524]: (43524-01) rw_loop: needline=1, flush=0, wr=0, timeout=30 tcpdump confirms the triple TCP handshake at that time: 1 15:50:35.184300 localhost.ijs.si localhost.ijs.si TCP 49355 10025 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=1 TSV=18493818 TSER=0 2 15:50:35.184354 localhost.ijs.si localhost.ijs.si TCP 10025 49355 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=16344 WS=1 TSV=18493818 TSER=18493818 3 15:50:35.184390 localhost.ijs.si localhost.ijs.si TCP 49355 10025 [ACK] Seq=1 Ack=1 Win=71680 Len=0 TSV=18493818 TSER=18493818 nothing in the Postfix log at 15:50:35 or close to it! Then, precisely 30 seconds later after a 30 second timer in amavisd expires, amavisd reports a timeout: Mar 18 15:51:05 inspe.com.ua /usr/local/sbin/amavisd[43524]: (43524-01) (!)rw_loop: leaving rw loop, no progress the exception handler sends a QUIT just in case, then drops the session. Now a surprise, tcpdump shows that Postfix send its greeting at just that moment (i.e. 30 seconds after TCP session was established), then a QUIT is captured, a response to it, and a normal TCP session rundown: 4 15:51:05.235130 localhost.ijs.si localhost.ijs.si TCP 10025 49355 [PSH, ACK] Seq=1 Ack=1 Win=71680 Len=32 TSV=18496823 TSER=18493818 220 inspe.com.ua ESMTP Postfix\r\n Postfix only now logs a connect: Mar 18 15:51:05 inspe postfix/smtpd[43289]: connect from localhost[127.0.0.1] 5 15:51:05.281348 localhost.ijs.si localhost.ijs.si TCP 49355 10025 [PSH, ACK] Seq=1 Ack=33 Win=71680 Len=6 TSV=18496827 TSER=18496823 QUIT\r\n 6 15:51:05.281489 localhost.ijs.si localhost.ijs.si TCP 10025 49355 [PSH, ACK] Seq=33 Ack=7 Win=71680 Len=15 TSV=18496827 TSER=18496827 221 2.0.0 Bye\r\n followed by a normal TCP rundown: 7 15:51:05.284290 localhost.ijs.si
Re: Reject Non-Ascii characters
On Tuesday 25 November 2008 12:26:17 bijayant kumar wrote: Some days ago at my original amavis server logs I observed some strange lines like (16188-21) WARN: address modified (recip): [EMAIL PROTECTED] - \240singh.richa09@gmail.com (16188-21) (!) lookup_sql: sql exec: err=7, 22021, DBD::Pg::st execute failed: ERROR: invalid byte sequence for encoding UTF8: 0xa0 See amavisd-new-2.6.0 release notes, search for invalid byte sequence for encoding (either set $sql_allow_8bit_address to false, or ALTER sql tables as described there; use a recent version of amavisd-new: 2.6.1 or 2.6.2-rc1) Mark
Re: timeout after END-OF-MESSAGE (still)
Justin, Even after using: http://www.mail-archive.com/[EMAIL PROTECTED]/msg11500.html I still get these occasionally, both from localhost and other (real) hosts: 13:28:27 p34 postfix/smtpd[21926]: timeout after END-OF-MESSAGE from localhost.localdomain[127.0.0.1] Is there another parameter that I need to tweak to avoid timeouts after the end-of-message? Per: http://marc.info/?l=postfix-usersm=121857914614836w=2 $ grep smtp_connection_cache_enable -r * conf.d/50-user: conf.d/50-user:$smtp_connection_cache_enable = 0; After having set the paramter to that - at the end of that thread, it seems to happen less often, but it still occurs nevertheless... Turn both the initial state off, as well as adaptive behaviour off: amavisd.conf: $smtp_connection_cache_on_demand = 0; $smtp_connection_cache_enable = 0; Mark
Re: XFORWARD Vs PDP
WTF PDP? Policy delegation protocol, I suppose. Mark
Re: Postfix - altermime - amavis - Too many hops
Peter,
disclaimer unix- n n - - pipe
flags=Rq user=altermime argv=/etc/postfix/filter/disclaimer -f
${sender} -- ${recipient}
It leads to error: Too many hops , in the log there is loop.
so your filter is passing mail back to an smtpd that passes it to
the same filter, and so on.
So the question is where can I put the content_filter=disclaimer:
work only for outgoing emails , into which smtpd process?
Amavisd can call altermime directly for adding disclaimers,
it already knows how to distinguish inbound vs. outbound mail
(by matching recipients to @local_domains_maps, and mail source
to @mynetworks or 'originating' attribute). As you are already
using it, it would probably simplify setup to call altermime
from there.
Search RELEASE_NOTES for:
- provided interface code to allow mangling/defanging/sanitation
to be performed by an external utility, either by [...]
Mark
Re: [OFF]: DKIM broken by certain email clients
James, I'm sending this reply using Thunderbird rather than Mail.app to see how the headers differ. I've tried sending without going through the ASSP anti-spam proxy to no avail. Likewise using amavisd-new. Yes, this one is a PASS! It still has two MIME-Version header fields, but unlike your previous mail, both are now MIME-Version: 1.0, while previously one of them was modified after signing. Mark
Re: [OFF]: DKIM broken by certain email clients
James, I'll have to work out how to turn off going through amavisd-new next. amavisd-new is DKIM-clean, it will not break a signature. Something else is modifying your Mime-Version header field. Mark
