Re: RESOLVED: RE: wrong From: and Return Path: address

[Viktor Dukhovni]

On Wed, Sep 27, 2017 at 07:00:39PM -0700, Michael Fox wrote:

> > This is an obsolete option in Sendmail that violates decades old SMTP
> > standards, CNAME canonicalization of recipient domains went away in
> > RFC 2821 and there are many email domains that are aliases where the
> > mailbox address is expected to remain unmangled by CNAME expansion of
> > the domain part.  The operators of the sending system are stuck in the
> > distant past.
> 
> Thanks Viktor.  Yes, I see section 3.6 now explicitly allows "CNAME RRs
> whose targets can be resolved, in turn, to MX or A RRs", which was the case
> in my situation.
> 
> Good to know that Postfix does it better!

Even Sendmail does it better in default and recommended configurations,
the users in question are deliberately choosing to continue no longer
sound practices.

-- 
Viktor.

RE: RESOLVED: RE: wrong From: and Return Path: address

[Michael Fox]

> > On Sep 26, 2017, at 11:23 PM, Michael Fox  wrote:
> >
> > BTW, the mail provider found that the default sendmail config and their
> own
> > customized config both rewrote the From: header when the From: address
> was
> > for a domain that had a CNAME record.  They said this is a config option
> in
> > sendmail and they prefer to operate this way.  Interesting that my other
> > Postfix machines don't do this, nor do many other large email providers.
> 
> This is an obsolete option in Sendmail that violates decades old SMTP
> standards, CNAME canonicalization of recipient domains went away in
> RFC 2821 and there are many email domains that are aliases where the
> mailbox address is expected to remain unmangled by CNAME expansion of
> the domain part.  The operators of the sending system are stuck in the
> distant past.

Thanks Viktor.  Yes, I see section 3.6 now explicitly allows "CNAME RRs
whose targets can be resolved, in turn, to MX or A RRs", which was the case
in my situation.

Good to know that Postfix does it better!

Michael

Re: RESOLVED: RE: wrong From: and Return Path: address

[Viktor Dukhovni]

> On Sep 26, 2017, at 11:23 PM, Michael Fox  wrote:
> 
> BTW, the mail provider found that the default sendmail config and their own
> customized config both rewrote the From: header when the From: address was
> for a domain that had a CNAME record.  They said this is a config option in
> sendmail and they prefer to operate this way.  Interesting that my other
> Postfix machines don't do this, nor do many other large email providers.

This is an obsolete option in Sendmail that violates decades old SMTP
standards, CNAME canonicalization of recipient domains went away in
RFC 2821 and there are many email domains that are aliases where the
mailbox address is expected to remain unmangled by CNAME expansion of
the domain part.  The operators of the sending system are stuck in the
distant past.

-- 
Viktor.

RESOLVED: RE: wrong From: and Return Path: address

[Michael Fox]

> > Michael Fox skrev den 2017-09-21 19:52:
> > > I have a problem that seems to have started when I upgraded from
> Ubuntu
> > > 14.04/Postfix 2.11.0 to Ubuntu 16.04/Postfix 3.1.0.  It involves the
> > > From:
> > > and Return Path: addresses seen by recipients of mail sent from a
> > > virtual
> > > domain on that machine.

This problem turned out to be a DNS issue.  The Postfix 3.1.0 machine's
virtual domain in the From: address was entered in DNS with a CNAME record
pointing to the gateway machine (because they are, indeed, the same
address).  When it was changed to an A record, the problem went away.
Evidently, the user that was certain that the problem only started after the
upgrade was, uhm, mistaken.  The problem also didn't occur on a separate
Postfix 2.11 machine because it's DNS entry was correct.  

BTW, the mail provider found that the default sendmail config and their own
customized config both rewrote the From: header when the From: address was
for a domain that had a CNAME record.  They said this is a config option in
sendmail and they prefer to operate this way.  Interesting that my other
Postfix machines don't do this, nor do many other large email providers.

Michael

RE: wrong From: and Return Path: address

[Michael Fox]

> sorry for late reply on this here,

No problem Benny.  Thanks for taking the time to review ...


> i noted from logs that you use
> mimedefang and amavisd for same mails, why ?

amavisd runs spamassassin and clamav.  No difference in setup between
Postfix 2.11 and 3.1.  

I just added mimedefang to perform some additional message mangling to help
out really old clients (like removing redundant html).  But I'm confident
that's irrelevant to the From: domain problem since I can take out
mimedefang and the problem persists.
 

> and as well that postfix send auth users mails to amavisd inbound so its
> classified as incomming mails, clean that up :=)
>
> -o content-filter override in master.cf on postfix solves this very
> nice, dont use content-filer in postfix main.cf, little hint here

We want all emails to go through amavis (spamassassin and clamav), whether
they are from one of our relay domains, virtual domains, a local user, or
the outside world.  We follow /usr/share/doc/amavisd-new/README.postfix.html
section 3.1 - Filtering E-mail globally.

Are you suggesting something different?  Can you be more explicit?

Regardless, I don't think this is related to the From: domain problem since
the config is the same for Postfix 2.11 and 3.1(unless there's a mistake
that I can't find).

 
> make sure amavisd have same mynetworks settings as postfix have, both
> should know all border ips aswell as rfc1918, and ipv6 dito, basicly all
> in ifconfig as a minimal

If you're referring to the following line:

amavis[2735]: (02735-07) Open relay? Nonlocal recips but not
originating: @

then, yeah, I've struggled with that.  (And based on Internet searches many
others do, too!)  I've verified it's not an open relay.  We're not using
IPv6 and the IPv4 nets are the same in amavis mynetworks and postfix
mynetworks.  Yet it continues to complain.  

I'm not sure what you mean by "... both should know all border IPs ...".
Can you be more explicit?

Regardless, the configuration is the same in postfix 2.11 and 3.1  So I
don't see how that could be causing the difference in behavior either
(unless there's a mistake that I can't find).

 
> and i think you have a problem on how sasl is configured on dovecot, is
> it only local system users auth that can relay mails ?, that way the
> auth only check local part of the mails to allow senders, that explains
> possible to change domain part and still authed for the local part of
> email, check that and ask for help with that on dovecot maillist

Dovecot performs SASL authentication of virtual domain users on the
submission port.  There are only a couple of local accounts for sysadmins,
and they don't use submission or SASL.  Since the problem with the Postfix
3.1 machine DOES involve the virtual domain being changed to the mail
gateway's hostname and this doesn't happen for relay domains, you may be
onto something.  But SASL does check both local part and domain part.  And
the SASL config hasn't changed between 2.11 and 3.1 (unless there's a
mistake that I can't find).

I don't understand what problem you see.  Can you be more specific?

 
> basicly random realm domain
> 
> to big logs make it harder for me to nerrow it more dowm

Well, thanks for taking a look.  I've been over and over the configs using
diff tools and I don't find anything significantly different between the
2.11 and the 3.1 configs.  Yet the 3.1 system results in the recipient
seeing the wrong From: domain at one (so far) email hosting provider.

The email hosting provider took a pcap trace and doesn't see anything wrong
yet with the SMTP session, but will continue to research on Monday.  

Michael

Re: wrong From: and Return Path: address

[Benny Pedersen]

Michael Fox skrev den 2017-09-22 04:07:

I'm not very skilled at interpreting the logs, but I've looked at them 
line
by line and I don't see where the destination server would ever get 
"From:
n6...@w6xsc-gw.scc-ares-races.org".  I'm hoping that someone here with 
more

knowledge than me can see where I went wrong.  I'm stumped.


sorry for late reply on this here, i noted from logs that you use 
mimedefang and amavisd for same mails, why ?


and aswell that postfix send auth users mails to amavisd inbound so its 
classified as incomming mails, clean that up :=)


-o content-filter override in master.cf on postfix solves this very 
nice, dont use content-filer in postfix main.cf, little hint here


make sure amavisd have same mynetworks settings as postfix have, both 
should know all border ips aswell as rfc1918, and ipv6 dito, basicly all 
in ifconfig as a minimal


and i think you have a problem on how sasl is configured on dovecot, is 
it only local system users auth that can relay mails ?, that way the 
auth only check local part of the mails to allow senders, that explains 
possible to change domain part and still authed for the local part of 
email, check that and ask for help with that on dovecot maillist


basicly random realm domain

to big logs make it harder for me to nerrow it more dowm

RE: wrong From: and Return Path: address

[Michael Fox]

> Michael Fox skrev den 2017-09-21 19:52:
> > I have a problem that seems to have started when I upgraded from Ubuntu
> > 14.04/Postfix 2.11.0 to Ubuntu 16.04/Postfix 3.1.0.  It involves the
> > From:
> > and Return Path: addresses seen by recipients of mail sent from a
> > virtual
> > domain on that machine.
> 
> you should not care of return-path at all, and if you try to make them
> equal with from you have a hard time with that job, no logs no problem,
> but thanks for postconf -n and postconf -Mf anyway

Thanks Benny.

I don't really care about Return-path and I'm not trying to make them equal.
Again, what I reported is that recipients on Gmail, Yahoo, Rackspace and
others see the correct value (user@virtualdomain) in both headers (Return
Path: and From: ), whether I send from the new Postfix 3.1.0 machine or the
older Postfix 2.11.0 machine.  However, recipients at this one (large) email
provider see the wrong value (user@gatewayhostname) in both headers when
sending from Postfix 3.1.0 and the right value (user@virtualdomain) when
sending from Postfix 2.11.0. 

> need more help show logs of a real problem

Here are two sets of logs.  The first is from the Postfix 3.1.0 machine
which results in the recipient seeing the wrong From: address.  The second
is from the Postfix 2.11.0 machine which results in the recipient seeing the
correct address.  In both cases, I included submission of the message
through delivery to the destination.

I'm not very skilled at interpreting the logs, but I've looked at them line
by line and I don't see where the destination server would ever get "From:
n6...@w6xsc-gw.scc-ares-races.org".  I'm hoping that someone here with more
knowledge than me can see where I went wrong.  I'm stumped.

Thanks,
Michael

>From Postfix 3.1.0 - recipient sees From: n6...@w6xsc-gw.scc-ares-races.org,
should be From: n6...@email6.scc-ares-races.org

Sep 21 18:45:41 w6xsc-gw postfix/submission/smtpd[26419]: connect from
n6mef-gw.n6mef.org[173.167.109.217]
Sep 21 18:45:41 w6xsc-gw postfix/submission/smtpd[26419]: Anonymous TLS
connection established from n6mef-gw.n6mef.org[173.167.109.217]: TLSv1.2
with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
Sep 21 18:45:41 w6xsc-gw dovecot: auth:
passwd-file(n6...@email6.scc-ares-races.org,173.167.109.217): unknown user
Sep 21 18:45:41 w6xsc-gw postfix/submission/smtpd[26419]: 6A9E31F70E:
client=n6mef-gw.n6mef.org[173.167.109.217], sasl_method=CRAM-MD5,
sasl_username=n6...@email6.scc-ares-races.org
Sep 21 18:45:41 w6xsc-gw postfix/pre-cleanup/cleanup[26426]: 6A9E31F70E:
message-id=<4c1afef8-9ab0-738c-a20a-212e7141a...@email6.scc-ares-races.org>
Sep 21 18:45:41 w6xsc-gw opendkim[1408]: 6A9E31F70E: DKIM-Signature field
added (s=mail61709, d=email6.scc-ares-races.org)
Sep 21 18:45:41 w6xsc-gw postfix/qmgr[26352]: 6A9E31F70E:
from=, size=673, nrcpt=1 (queue active)
Sep 21 18:45:41 w6xsc-gw amavis[2735]: (02735-07) LMTP [127.0.0.1]:10024
/var/lib/amavis/tmp/amavis-20170921T061543-02735-H33h2gd8:
 ->  SIZE=673
BODY=8BITMIME Received: from w6xsc-gw.scc-ares-races.org ([127.0.0.1]) by
localhost (w6xsc-gw.scc-ares-races.org [127.0.0.1]) (amavisd-new, port
10024) with LMTP for ; Thu, 21 Sep 2017 18:45:41 -0700
(PDT)
Sep 21 18:45:41 w6xsc-gw postfix/submission/smtpd[26419]: disconnect from
n6mef-gw.n6mef.org[173.167.109.217] ehlo=2 starttls=1 auth=1 mail=1 rcpt=1
data=1 quit=1 commands=8
Sep 21 18:45:41 w6xsc-gw amavis[2735]: (02735-07) dkim: VALID
Author+Sender+MailFrom signature by d=email6.scc-ares-races.org, From:
, a=rsa-sha256, c=simple/simple,
s=mail61709, i=@email6.scc-ares-races.org
Sep 21 18:45:41 w6xsc-gw amavis[2735]: (02735-07) Checking: EmxshYSM9dtH
[173.167.109.217]  -> 
Sep 21 18:45:41 w6xsc-gw amavis[2735]: (02735-07) Open relay? Nonlocal
recips but not originating: n6...@prismatic.com
Sep 21 18:45:41 w6xsc-gw amavis[2735]: (02735-07) p001 1 Content-Type:
text/plain, size: 10 B, name:
Sep 21 18:45:42 w6xsc-gw postfix/amavisreturn/smtpd[26431]: connect from
localhost.localdomain[127.0.0.1]
Sep 21 18:45:42 w6xsc-gw postfix/amavisreturn/smtpd[26431]: 380AA1F824:
client=localhost.localdomain[127.0.0.1]
Sep 21 18:45:42 w6xsc-gw postfix/cleanup[26432]: 380AA1F824:
message-id=<4c1afef8-9ab0-738c-a20a-212e7141a...@email6.scc-ares-races.org>
Sep 21 18:45:42 w6xsc-gw postfix/amavisreturn/smtpd[26431]: disconnect from
localhost.localdomain[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1
commands=5
Sep 21 18:45:42 w6xsc-gw postfix/qmgr[26352]: 380AA1F824:
from=, size=1552, nrcpt=1 (queue active)
Sep 21 18:45:42 w6xsc-gw amavis[2735]: (02735-07) EmxshYSM9dtH FWD from
 -> , BODY=7BIT 250
2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 380AA1F824
Sep 21 18:45:42 

Re: wrong From: and Return Path: address

[Benny Pedersen]

Michael Fox skrev den 2017-09-21 19:52:

I have a problem that seems to have started when I upgraded from Ubuntu
14.04/Postfix 2.11.0 to Ubuntu 16.04/Postfix 3.1.0.  It involves the 
From:
and Return Path: addresses seen by recipients of mail sent from a 
virtual

domain on that machine.


you should not care of return-path at all, and if you try to make them 
equal with from you have a hard time with that job, no logs no problem, 
but thanks for postconf -n and postconf -Mf anyway


need more help show logs of a real problem

wrong From: and Return Path: address

[Michael Fox]

I have a problem that seems to have started when I upgraded from Ubuntu
14.04/Postfix 2.11.0 to Ubuntu 16.04/Postfix 3.1.0.  It involves the From:
and Return Path: addresses seen by recipients of mail sent from a virtual
domain on that machine.

Clients of Google, Yahoo, Rackspace, . see the From: and Return Path:
address as @, which is correct.
Clients of one (rather large) email service provider see the From: and
Return Path:  address as @, which is wrong.

The one email provider might have something wrong on their end.  BUT:  The
problem doesn't happen with mail received at that provider from a similarly
configured gateway/virtual domain, which is still running Ubuntu
14.04/Postfix 2.11.0.  And the problem didn't start happening on the machine
in question until the machine was upgraded to Ubuntu 16.04/Postfix 3.1.0.
So my money is on a mistake on my end.  I just can't find it.

I've done file comparisons between the postfix 2.11.0 and 3.1.0 machines,
and between the old and new configs of the 3.1.0 machine, and I just can't
find any significant differences (i.e. other than hostname changes, etc.).

Below is postconf info for the current main.cf and master.cf.  

Thanks in advance for any help.
Michael

$ postconf -pnf
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
append_at_myorigin = yes
append_dot_mydomain = yes
biff = no
body_checks = pcre:${config_directory}/body_checks.pcre
bounce_queue_lifetime = 12h
bounce_template_file = ${config_directory}/bounce.cf
broken_sasl_auth_clients = yes
canonical_maps = pcre:${config_directory}/canonical.pcre
compatibility_level = 2
content_filter = amavisfeed:[127.0.0.1]:10024
delay_warning_time = 2h
fast_flush_domains = $relay_domains
header_checks = pcre:${config_directory}/header_checks.pcre
html_directory = /usr/share/doc/postfix/html
inet_interfaces = $xsc_inet_interfaces
mailbox_size_limit = 5120
maximal_queue_lifetime = 12h
message_size_limit = 1024
milter_default_action = accept
milter_protocol = 6
mime_header_checks = pcre:${config_directory}/mime_header_checks.pcre
mua_client_connection_count_limit = 5
mua_client_connection_rate_limit = 10
mua_client_message_rate_limit = 10
mua_client_recipient_rate_limit = 50
mua_client_restrictions = check_sasl_access
hash:${config_directory}/sasl_access
permit_sasl_authenticated reject
mua_discard_ehlo_keyword_address_maps =
cidr:${config_directory}/ehlo_keyword.cidr
mua_helo_restrictions =
mua_recipient_limit = 25
mua_recipient_overshoot_limit = 25
mua_recipient_restrictions = reject_non_fqdn_recipient
reject_unknown_recipient_domain check_sasl_access
hash:${config_directory}/sasl_access check_recipient_access
hash:${config_directory}/roleaccount_exceptions check_recipient_access
pcre:${config_directory}/recipient_access.pcre check_recipient_access
pcre:${config_directory}/relay_recipient_access.pcre
check_recipient_access
pcre:${config_directory}/virtual_recipient_access.pcre permit
mua_relay_restrictions = permit_sasl_authenticated reject
mua_sender_restrictions = $mua_tls_client_restrictions
reject_non_fqdn_sender
reject_sender_login_mismatch permit_sasl_authenticated
reject_unknown_sender_domain reject_unlisted_sender permit
mua_tls_client_restrictions = check_client_access
cidr:${config_directory}/tls_clients.cidr
mydestination = $xsc_mydestination
mydomain = $xsc_mydomain
myhostname = $xsc_myhostname
mynetworks = $xsc_mynetworks
myorigin = $xsc_myorigin
non_smtpd_milters = inet:localhost:8891
postscreen_access_list = permit_mynetworks
cidr:${config_directory}/postscreen_access.cidr
postscreen_blacklist_action = drop
postscreen_dnsbl_action = enforce
postscreen_dnsbl_reply_map =
pcre:${config_directory}/postscreen_dnsbl_reply_map.pcre
postscreen_dnsbl_sites = zen.spamhaus.org*3 bl.spameatingmonkey.net*2
psbl.surriel.com*2 bl.spamcop.net
hostkarma.junkemailfilter.com=127.0.0.2
dnsbl.sorbs.net bl.mailspike.net swl.spamhaus.org*-4
list.dnswl.org=127.0.[0..255].0*-1 list.dnswl.org=127.0.[0..255].1*-2
list.dnswl.org=127.0.[0..255].2*-3 list.dnswl.org=127.0.[0..255].3*-4
postscreen_dnsbl_threshold = 3
postscreen_dnsbl_ttl = 5m
postscreen_greet_action = enforce
proxy_interfaces = $xsc_proxy_interfaces
readme_directory = /usr/share/doc/postfix
recipient_delimiter = +
relay_domains = $xsc_relay_domains
relay_recipient_maps = pcre:${config_directory}/relay_recipients.pcre
relay_restrictions = check_sender_access
pcre:${config_directory}/relay_sender_access.pcre
remote_header_rewrite_domain = invalid.domain
smtp_host_lookup = native
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
smtpd_client_connection_count_limit = 10
smtpd_client_connection_rate_limit = 20
smtpd_client_message_rate_limit = 20
smtpd_client_recipient_rate_limit = 200
smtpd_client_restrictions = permit_mynetworks check_client_access
pcre:${config_directory}/client_access.pcre