This week I was asked to help out colleagues who were trying to
make Postfix work with some email archive system. I drafted up a
few solutions, one of which was adopted.
During this exercise I realized that this would be less painful if
Postfix had built-in support to create "archive-quality" copies of
email messages. Below is a straw-man design.
Wietse
Draft design: Building email archival support into Postfix.
Over-all design
===============
- Create copies of email envelope addresses and content for archive
purposes, and send them as email messages to a configurable archive
recipient address. The archive recipient address may be specified
in main.cf (archive_recipient = recipient@recipient-domain), or it
may be specified with an access map or header/body_checks action
(ARCHIVE recipient@recipient-domain).
- When a queue file is created for a new email message, the file
may contain a record with an archive recipient address.
- When the queue manager opens a queue file that contains an archive
recipient address record, it first contacts the archive daemon
before attempting to deliver the message.
- When the archive daemon creates an archive message, it preserves
in the From: and To: headers the envelope sender and recipient
addresses of the original message, together with a copy of the Date:
and Subject: headers of the original message, making the email
archive easy to examine with standard email tools. The original
message contents (header and body) are preserved as a message/rfc822
attachment.
- The archive message queue file contains a flag saying that Postfix
must never bounce the message.
- After the archive daemon has queued an archive message, the archive
daemon clears the archive recipient record in the queue file and
reports successful completion to the queue manager. Then, the queue
manager processes the queue file for normal delivery.
New Postfix code and data
=========================
- A new queue file record (archive recipient) with the archive
recipient email address. This can be set with a new main.cf
archive_recipient parameter, and can be overruled with an ARCHIVE
action in an access map or header/body_checks action. Only the
"last" archive recipient address in a queue file will be used. This
similar to how Postfix handles a queue file with multiple FILTER
records.
- A new queue file record (soft bounce) saying that Postfix must
never bounce the message containing that record. This is a safety
mechanism for when delivery to the archive server fails.
- A new archive daemon that receives a queue manager request to
submit an archive message into the mail queue. This is similar to
how the queue manager asks the bounce daemon to submit a delivery
status notification message. After successful submission the archive
daemon clears all archive recipient records in the original queue
file and reports successful completion.
- A new archive client library module that talks to the archive
daemon. This will use event-driven communication, similar to the
abounce(3) and adefer(3) client library modules, and will use the
same communication infrastructure to handle timeouts and I/O errors.
- A new queue manager state. When a queue file contains an archive
recipient record, the queue manager first asks the archive daemon
to create an archive message. Once the archive daemon reports
successful completion, the queue manager delivers the queue file
as usual.
- A new receive_override_options option (no_archive) that suppresses
the generation of archive copies before or behind a post-queue
content filter.
- Gory details: need to decide whether an archive message should
be subject to any form of header/body_checks, Milter or external
content filter. It probably should not, to avoid archive loops.
Example 1: original SMTP transaction
====================================
In this transaction, Postfix receives an email message that will
later be archived (S=Postfix mailhub; C=remote mailhost):
S: 220 postfix.example.com ESMTP Postfix
C: EHLO mailhost.sender-domain
S: 250-postfix.example.com
...other ESMTP feature announcements...
C: MAIL FROM:<original-sender@sender-domain>
S: 250 2.1.0 Ok
C: RCPT TO:<original-recipient1@recipient-domain1>
S: 250 2.1.5 Ok
C: RCPT TO:<original-recipient2@recipient-domain2>
S: 250 2.1.5 Ok
C: DATA:
S: 354 End data with <CR><LF>.<CR><LF>
C: Received: from hostname (hostname [ipaddr]) by mailhost.sender-domain....
Received: by hostname...
From: ...some sender address...
To: ...some recipient address...
Date: ...date...
Subject: ...subject...
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
eiusmod tempor incididunt ut labore et dolore magna aliqua.
.
S: 250 2.0.0 Ok: queued as 3Zbm2D5lQZznjbn
C: QUIT
S: 221 2.0.0 Bye
Example 2: archive SMTP transaction
===================================
Here, Postfix delivers an archive copy of the message to the archive
server (S=archive server; C=Postfix hub), preserving the original
envelope sender and recipients in the archival copy's From: and To:
headers together with a copy of the Date: and Subject: headers, and
preserving the original message (headers and body) as a message/rfc822
attachment.
S: 220 mailhost.archive-domain ESMTP
C: EHLO splitter.example.com
S: 250-mailhost.archive-domain
...other ESMTP feature announcements...
C: MAIL FROM:<original-sender@sender-domain>
S: 250 Ok
C: RCPT TO:<archive-recipient@archive-domain>
S: 250 Ok
C: DATA:
S: 354 End data with <CR><LF>.<CR><LF>
C: From: <original-sender@sender-domain>
To: <original-recipient1@recipient-domain1>,
<recipient2@recipient-domain2>
Date: ...same date as original message...
Subject: ...same subject as original message...
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: Multipart/Mixed;
boundary="unique-boundary-string"
--unique-boundary-string
Content-Type: message/rfc822
Received: from mailhost.sender-domain (hostname [ipaddr]) by
postfix.example.com...
Received: from hostname (hostname [ipaddr]) by mailhost.sender-domain....
Received: by hostname...
From: ...some sender address...
To: ...some recipient address...
Date: ...date...
Subject: ...subject...
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
eiusmod tempor incididunt ut labore et dolore magna aliqua.
--unique-boundary-string--
.
S: 250 Ok: queued as 97C84AC2947
C: QUIT
S: 221 2.0.0 Bye
