Reputation databases and content filtering algorithms do not appear out of
thin air. They arise because somebody works hard to extract useful data
from live message streams. Content filtering blocks messages for two
classes or objections:
- "This particular message looks like unwanted noise." The message is
blocked with no impact on sender reputation
- "This particular message looks like an attack." When these types of
alarms are raised by content filtering, the response should be more than
blocking one message. If the attack is verified, the sender should be
blocked.
For sender authentication, every unauthenticated message is possibly a
malicious impersonation. If verified, the attacking sender needs to be
blocked. There is no situation where "just block one message" is an
intelligent response. And there is no situation where automated blocks
provide a correct response.
Keep in mind that I have a working environment based on successful
authentication of essentially everything, without harm to wanted messages.
I spent the weekend writing up everything I have learned, intending to post
it as a standalone draft. But I am pretty sure that my XML2RFC template
is obsolete and the current ones are temporarily unavailable for download.
So I am attaching it below. I suggest that this is the first really
clear argument for why evaluators should reverse strategy on mailing list
blocks. The goal that everyone wanted finally has a theoretical
foundation. So I hope you will read it.
Doug Foster
Sender Authentication Best Practices
This document describes a suggested use of sender authentication as part of
a program to maximize sender authentication while minimizing harm to wanted
messages.
All human communication is interpreted in the context of the speaker or
author. For email, the purported author is indicated by the RFC5322.From
address. Malicious actors seek to impersonate trusted authors to deceive
both email filtering software and email recipients. To avoid deception,
and to ensure correct message interpretation, the ideal outcome is for
every accepted email message to have an authenticated author.
DMARC is an established tool for authenticating authors, based on the
RFC5322.From address being part of the same organization as a verified DKIM
signature domain or a SPF-verified RFC5321.MailFrom domain. The key
problem at implementation is that the set of correctly identified authors
is significantly greater than the set of verifiable authors. An effective
implementation seeks to close that gap.
Correct authors can fail to authenticate for at least these reasons:
- The domain owner did not publish the necessary information in DNS to
support DMARC.
- The domain owner did not configure DKIM signing on some sending servers.
- The DNS policies, DKIM public keys, or DKIM signing algorithms have
errors.
- The message has an indirect mail flow, causing the final state of the
message to differ from the original state of the message, rendering the
authentication test inaccurate.
- The message is sent by a legitimate source, on behalf of a domain
account, but outside of domain owner control. The message is an
impersonation, but it is safe and likely wanted.
Evaluators need to adapt to all of these challenges. This means that
messages should not be blocked simply because of authentication failure.
Instead, they should be investigated, so that the root cause is determined.
Sender authentication helps to identify the messages that warrant
investigation while limiting the volume of messages that require this
effort. The investigation process allows unwanted message sources to be
confirmed and blocked, while wanted message sources are given alternate
authentication to avoid authentication problems in the future.
For safe and wanted messages, the corrective measures correspond to the
root causes:
- If a domain lacks a DMARC policy, a default policy should be applied
using an alignment rule selected by the evaluator, typically relaxed
alignment. Some operational experience indicates that this will
approximately double the percentage of messages that produce DMARC Pass.
- If a domain has a missing, invalid, or incorrect SPF Policy, the
evaluator provides a substitute for it in local policy. An effective way
to define an SPF rule in local policy is by using three attributes: When
server domain is <value1>,
and server name is verified by forward-confirmed DNS,
and RFC5321.MailFrom domain is <value2>,
then the message is equivalent to SPF PASS.
A rule based on source IP and RFC5321.MailFrom domain is also possible, but
may be insufficient for messages coming from large server farms with
variable source IP addresses.
- Email Service Providers (ESPs) can be considered for delegated
authentication. ESPs send messages on behalf of their clients, usually for
a fee, using the ESP domain in the RFC5321.MailFrom address and the client
domain in the RFC5322.From address. The normal DMARC test will produce
Fail for any message that is not signed, and ESPs are not able to obtain
DKIM private keys for every client account. To ensure correct billing,
the ESP needs to authenticate the client during both account setup and
message initiation. If the evaluator trusts a specific ESP to have these
characteristics, messages that are SPF-verified as coming from the ESP can
be accepted as equivalent to DMARC PASS, even when a message is not signed
by the client domain. Determination of which ESPs should be granted this
level of trust is a local policy decision.
Because the volume of SPF corrections can be onerous, it is desirable to
consider deviations from the RFC7208 standard, to minimize unnecessary
investigation:
- Relax the lookup count: Some SPF policies are so complex that the policy
violates the 10-lookup rule, triggering a PERMERROR result. Operational
experience indicates that a lookup limit of 20 will change most PERMERROR
results to PASS, without creating a denial of service problem for the
evaluators or the DNS.
- Relax handling of invalid include clauses: If an include reference is
invalid, the recommended result is PERMERROR, because some authorized
servers may be undetected as a result of the missing information. By
continuing to process the SPF policy despite a missing include term, the
evaluator may be able to produce a usable PASS result, and the missing
include term becomes irrelevant.
- Relax handling of duplicate policies: If an SPF lookup returns multiple
policies, the recommended result is PERMERROR, because the policies may
contain incompatible instructions. Operational experience shows that most
duplicated policies are the result of minor edits done incorrectly, with a
second policy being created instead of the first policy being updated.
Because of the nature of this mistake, most aspects of both polices are
identical. An evaluator could minimize the impact of this mistake by
evaluating both policies, and return a final PASS result when both policies
produce PASS independently.
- Use hosting service proxy signatures. Some hosting services add
signatures using a parent domain owned by the hosting service and a
subdomain that is client-specific. If the evaluator determines that the
signature can be reliably interpreted as referring to the RFC5321.MailFrom
domain, and the signature can be reliably interpreted as representing
authentication of an account in the RFC5321.MailFrom domain, then the
hosting service signature is an alternative to SPF PASS. It is actually
preferable to SPF PASS, since the signature is not subject to cross-tenant
impersonation. For many such messages, the RFC5321.MailFrom domain and the
RFC5322.From domain will be identical or aligned, so the hosting service
proxy signature also provides an equivalent to DMARC PASS.
Even after all of these adjustments to increase PASS rates, and after
handling messages that are unacceptable to content filtering, a portion of
the mail stream will be unauthenticated and ambiguous. The safest way to
resolve the remaining ambiguity is to quarantine the message until an
investigation can be completed, but this still overwhelm the available
quarantine sources. Evaluators may choose to release some unauthenticated
messages, without quarantine, due to this overload, while depending on the
evidence that the message did not raise content alarms. Sender
authentication is a process of moving as many messages as possible from a
content-only defense to a sender-plus-content defense. That process
cannot be completed overnight. What is certain is that each investigation
leads to a decision that resolves ambiguity for an entire mail stream,
either by providing alternate authentication for wanted messages, or
blockage of unwanted messages. The lasting effect of each decision
insures that all effort applied to investigation adds value, and that the
process will converge.
As all valuable and recurring message streams become authenticated, and a
long list of attack sources become blocked, the volume of malicious
messages and the volume of unauthenticated messages will drop steadily.
Unauthenticated messages will be from new senders who need to be
categorized.
Indeterminate Message Sources
SPF and DMARC evaluate message attributes at reception, in an attempt to
infer authentication state at message origination. When the flow from
originator to recipient is complex, the message state can change in ways
that make the sender authentication results incorrect or irrelevant. These
situations will become evident when authentication failures occur and are
investigated.
- Automatic forwarding without SPF rewrite: All messages will produce SPF
FAIL. Unsigned messages will also produce DMARC FAIL, while signed
messages will produce DMARC PASS. The common attribute will be the
forwarding organization. The forwarding account may be detectable by
parsing other header fields.
- Automatic forwarding with SPF rewrite: All messages will produce SPF
PASS, but without alignment. Unsigned messages will produce DMARC FAIL,
while signed messages will produce DMARC PASS. The common attribute will
be the rewritten RFC5321.MailFrom account. The forwarding account may be
detectable by parsing other header fields.
- Mailing list forwards: This is equivalent to automatic forwarding with
SPF rewrite, but the message subject or body are also modified with
list-specific text, causing DKIM signatures to be invalidated. All
messages will fail both SPF and DKIM, causing DMARC FAIL. The common
attribute will be the RFC5321.MailFrom address.
When these configurations cause authentication failure, most of the
unauthenticated messages are expected to be legitimate, but some malicious
impersonation may be sprinkled into the mix. The standard authentication
tests cannot distinguish between the two, so the failure results should be
considered INDETERMINATE rather than FAIL.
Authorizing Indeterminate Message Sources
Any defense against forwarded message flows begins with a consideration of
whether the flow should be allowed at all. Some list platforms allow
attackers to create fake mailing lists, where subscribers are configured
into the list without the subscriber’s knowledge or consent. The list
then becomes a form of open relay with a limited set of attack targets.
If the evaluator can determine that the recipient is not an intentional
subscriber to the list, then the list should be blocked.
Similar issues apply to automatic forwarding. Both forwarding and
receiving servers have reasons to ensure that forwarding configurations are
pre-approved.
For the forwarding server:
- An inappropriate forward may be configured to exfiltrate data as part of
an account compromise or an insider attack.
- An inappropriate forward may be configured by a reckless user, and permit
release of data that is prohibited from disclosure by regulatory
requirements or organizational policy.
- An incorrectly configured forwarding destination may cause the recipient
to report the forwarding server as a spam source. This complaint could
cause delivery problems for all users on the forwarding environment.
For the recipient server:
- To prevent targeted attacks, the recipient server should confirm that the
forwarding flow is approved by the intended recipient and is otherwise
compatible with organizational policy.
- For many purposes, automatic forwarding is not necessary because all
major email clients allow configuration of multiple mailboxes. Since the
forwarding configuration makes message filtering more difficult, incoming
forwards should only be allowed when absolutely necessary.
- The recipient server should assess the spam filtering characteristics of
the forwarding server. If a forwarding environment is determined to have
poor spam filtering, the forwarding configuration should be rejected.
Handling Indeterminate Results
When a forwarding configuration is allowed, the easiest solution to the
indeterminate message flow is to configure a local policy rule to allow the
entire mail stream to proceed to content filtering. The evaluator then
depends on content filtering to detect any malicious traffic. This is a
minimal incremental risk, since plenty of malicious traffic arrives with
authentication, so the evaluator is already dependent on content filtering
as the primary defense against many attackers.
A more difficult approach is to parse the header fields to infer additional
information, in the hope that at least some of the indeterminate messages
can be reliably authenticated or reliably classified as an impersonation.
These information sources are known to be available.
- The evaluator can check for prior authentication results stored in the
message header as ARC-Authentication-Results, Authentication-Results, or
Received-SPF fields.
- The evaluator could check for pre-forwarding SMTP recipient addresses
stored in Received fields using a "FOR user@domain" term. A
pre-forwarding recipient address may indicate the subsequent forwarding
source.
- The evaluator could check for prior RFC5321.MailFrom addresses stored in
Received fields using an (envelope-from <user@domain>) comment.
If usable data is found, the evaluator could rely on the provided
authentication results, or perform its own SPF test using the
pre-forwarding RFC5321.MailFrom account and source IP address. However.
parsing Received fields and interpreting them to produce useful
authentication results is difficult and error prone. Knowing which
authentication results to trust and which to ignore is also difficult.
Even the most complex analysis is likely to leave some ambiguous results,
simply because the message header wll not contain enough information to
solve the problem. Even when the analysis produces a usable result but the
result is not Pass, the message may not be a malicious impersonation, for
all the reasons already discussed in this document. Given all of these
complexities, most evaluators should choose to allow unauthenticated
messages from indeterminate message sources to proceed, and depend on
content filtering to find any problematic messages hidden within the
indeterminate flow.
On Sun, Dec 8, 2024 at 8:21 AM Alessandro Vesely <[email protected]> wrote:
> On Fri 06/Dec/2024 13:19:29 +0100 Douglas Foster wrote:
> > Email Filtering begins with content filtering, which is and always will
> be an
> > essential part of the defenses.
>
>
> *I really hope not*. Content filtering is mumbo jumbo. One ought to
> fully
> understand the content of a message in order to judge its wantedness.
> Recipients themselves are at times unable to do so.
>
> SPF, DKIM, DMARC and ARC, AFAIUI, are part of an effort which aims at a
> mail
> system that can run automatically, without herds of content analyzers
> pedaling
> in the background.
>
>
> Best
> Ale
> --
>
>
>
>
>
> _______________________________________________
> dmarc mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
>
_______________________________________________
dmarc mailing list -- [email protected]
To unsubscribe send an email to [email protected]
