Hi there,
here's the promised document. It's still in early stages, but should
give you an idea about what it'll look like, and where I'm heading with
it.
Another thing I should mention is the fact that I still haven't gotten
around to putting more energy into the coverage of the programs that
together create the email toolchain. Just a few snippets.
And one last note: I'm by no means an expert in either of the things I
cover in the howto, so please _do_ send me corrections if you find any
false statements or other nonsense.
TIA
--
FreeBSD 4.4-STABLE
1:49PM up 51 days, 32 mins, 13 users, load averages: 0.17, 0.13, 0.04
********** THE MAIL HOWTO **********
* Using email on Unix the hard way *
************************************
###### TODO: ######
### add "screenshots". super-mega-hyper cool, and useful. a single picture
saves a thousand words.
### add links to websites, and names of the authors of the individual
components used in this doc.
(links added)
### mailcap + mime.types
### "minitutorials" on individual components.
###### :TODO #####
1) SYNOPSIS
Email is perhaps the most widely used form of electronic communication.
Without all those servers running various flavors of Unix, there would be no
Internet, nor email. However, as you might have already found out, most
documents on setting up Unix hosts for email is aimed at the server side of
things, and seems to expect the reader already knows much about the topic.
Frankly, finding information on setting up your Unix workstation to handle
email aimed at users new to Unix is one of the harder parts of switching to the
new operating system.
Note: Unix is (R) of OpenGroup.
2) TARGET AUDIENCE
This HowTo is not for you if:
a) you prefer mouse over keyboard
b) you like html email
c) you can't be bothered to learn how to operate new software
But honestly, if you answered yes to any of the above questions, you should
probably not think about using Unix at all.
This HowTo is for you if:
a) you handle large amounts of email (twice so if substantive amount comes
from various mailing lists)
b) you want to handle your email effectively
c) you are just plain 'techy' in general
Before reading this document you should:
* have a Unix-like operating system installed on a computer connected to
Internet (I'm running FreeBSD, which means that you should also be able
to adjust the FreeBSD-specific details to your system if you're running
Linux for example)
* be capable of basic shell operations (creating directories, changing
permissions, etc.)
* be able to edit files in a text editor
* know how to install additional third-party software. This means you must
be able to type (on FreeBSD, that is)
# cd /usr/ports/foo/bar
# make install
or
# pkg_add foo-1.0.tgz
or (on RedHat Linux)
# rpm -i foo-1.0-i686.rpm
The following skills are not absolutely required, but would help a lot:
* basic knowledge of regular expressions
Note however that some of the packages described in this document are
configured with the use of regular expressions, while others employ downright
cryptographic format for their configuration files! As a rule of thumb, the
approach excercised in this howto is not for the faint of heart or lazy.
After reading this document you will:
* know what software components are involved in manipulating email
* have a functional email subsystem on your personal computer running
a Unix-like operating system.
3) THE PROBLEM
To put it straight: configuring your Unix workstation to receive and send mail
can prove to be hell. While Windows, MacOS, and alike were designed to be
usable even by (almost) completely clueless people, abstracting the fact that
the user is in fact operating a computer, and the way it works away from the
user, unices expose the gory details to the user.
After all, operating systems like FreeBSD, Solaris or Linux, do not strive to
be the system of choice for my granny. They don't attempt to amuse their users
with bells and wistles, they strive to be technically superior, even at the
expense of point'n'click eyecandy.
Of course. There's Netscape Communicator available for Unix (the Linux binary
runs--of course--on FreeBSD), there's Mozilla... And numerous other, far less
resource-hungry X11-based mail clients. All you need to do to continue to use
your mail (and web) if you switch from windows or mac to a Unix flavor is to
install mozilla or netscape, click through the wizard, and you're set up.
Except, well, that netscape 4.x won't allow you to use more than one pop3
server, and mozilla is a resource hog. Plus, why the heck do I have to take my
hand off the keyboard, and click the button when I could just as well just
press a key, and be done with it in a hundredth of the time? These all-in-one
packages are really all in one, all or nothing, bringing the windows
philosophy to Unix.
4) THE GOAL
The goal this document is to help you understand how to set up your Unix
workstation to receive, read, process, send email, and do everything else you
are used to doing with email -- with an amount of flexibility (and, hopefully
security) you could never achieve with common MS Windows-based email clients.
But power, of course, comes at a price. The freedom you get with the
mind-boggling level of configurability of the software this howto presents
means you sometimes have to spend mind-boggling amounts of energy to get it to
do what you want. Also the documentation isn't always as hi-fi as the software
it accompanies.
5) TERMS
MUA, MDA, MTA...
When I started thinking about leaving the MS world, one of the things I was
concerned about first was email. I receive from 200 to 400 messages a day, and
while I could afford some glitches in comprehending the new and different
philosophy of the system (and GUI), I certainly didn't want to run into any
bumps with my mail. When I started doing some research, I was confused. There
were too many alien terms: mbox, maildir, MH, MUA, MTA, MDA... What the heck!?
I just want to read my mail, I don't want to run a mailing list! Where's my
Eudora? Help!
I since then realized that it's not all that alien once you get your hands
dirty. Really. It might be just the way my brain works, but what seemed to be
a great gap turned out to be pretty obvious--once I saw the various tools in
action, the strange and foreign terms suddenly got their meanings. To ease
your pain, I'll try to translate what you might have read (or will read) about
handling mail on Unix into terms you should be familiar with.
An AVBM (Average Windows-Based Mailer) can do sooo many things. Just think
about it: it can fetch your mail from the servers to your computer, filter it
into various folders, you can use it to search for that message with your
friend's telephone number, you use it to write emails, and it will, in the
end, send them out. And, of course, it can display the messages you received.
Plus, it has an address book. That's, lemme count, six different things, which
means that, according to the Unix philosophy of "do just one thing, but do it
right", it serves the purpose of at least six different programs. This may
sound absurd at first. Why would someone want to learn to use six different
pieces of software just to read their email? Here, of course, the answer is:
flexibility.
When your AWBM fetches your mail from the server to your home PC, and sorts it
into different folders, it acts as an MDA: Mail Delivery Agent. When you view
those messages in it, it plays the role of an MUA: Mail User Agent. And when
you send out your replies, it's an MTA: Mail Transport Agent. And when you type
those replies, it fills the purpose of a text editor. Plus there's the
beforementioned address book function, and other small details.
What if, for instance, you weren't happy with your AWBM's builtin text editor?
Your choices are limited: you can either conform to the situation, start using
another AVBM (but hey, I don't want the whole new mailer, I just don't want to
suffer whenever I'm typing a message!), or type your emails in your favorite
text editor outside the AVBM, and copy&paste them back into it. But who would
want to do this?
But let's leave text editors for a while, and get back to email.
MTA - Mail Transfer Agent
These days, "MTA" equals "SMTP server". SMTP is not the only way computers can
exchange email, but it's the most used.
An MTA accepts messages locally from the command line, and send them to the
recipients' MTAs.
They also accept connections from other hosts, and store the messages locally
for their recipients to pick them up, or send them out if the recipients'
addresses indicate that their email is to be sent to other hosts.
The former is called "delivery". How does an MTA recognize which messages
should go on elsewhere, and which are to be considered "delivered"? First, it
consults the domain part of the recipient's address with its own hostname.
Then it looks whether it knows the recipient: it is either a local user, or
is otherwise "registered". If the message meets both requirements, it is
written in the recipient's "spool".
The latter operation is called "relay", and usually requires the hosts that
want to relay through another MTA to meet certain criteria. For example, it's
a common practice that ISPs' SMTP servers reject email from IP addresses
outside thse ISPs' networks.
To prevent spammers from abusing other people's MTAs,
tbw
Another common requirement is that the name relaying host identifies itself
with must resolve. IOW, it must be in DNS. That means that if you don't meet
this criteria, you will have to use your ISP's SMTP server.
The fact that you can send email from Windows 3.11 indicates that even this
poor system sports an MTA. This is not exactly the case. Instead, every email
client contains a SMTP nullclient -- a scaled-down MTA that can only relay
through a single SMTP server (or a fixed set).
The MTA accepts a few parameters from the command line (for example, the
recipients), along with the headers and the body of the message from the
standard input, adds some headers on its own, and stores the message in the
queue. If you send a single message to more than one recipient, it is split
into separate copies in this moment. Then it looks at the recipients'
addresses, and issues its DNS server for MX (Mail eXchange) records for the
domain parts of the addresses. Now it tries to contact the remote MTAs, and
send them the queued messages. Messages transfered succesfully are removed
from the queue. Messages destined at hosts that could not be immediatelly
delivered stay in the queue scheduled for future delivery. Messages rejected
by the remote MTAs are "returned" to their senders. These are the "Returned
mail: see transcript for details" emails everybody receives from time to time.
add a few links to other flavors
http://www.sendmail.org/
http://www.postfix.org/
POP3 - Post Office Protokol v. 3
tbw
Protocol allowing users without a shell acount on hosts where their email
is delivered fetch it, and process it elsewhere. In a "normal" world, you
would log into the host that stores your email, start a MUA, read the
email, and maybe send some, using the host's MTA. This would of course
require a connection between the two computers during that time. Today,
most computers are not online 24x7, and ISPs don't grant their users shell
accounts. POP3 is designed to facilitate a single operation, handing off
email that has been requested by its recipients.
IMAP - Internet Message Access Protocol
tbw
MDA - Mail Delivery Agent
MDAs are tools specialized at filtering messages, and writing them to
folders. These programs usually have a complex, regexp-based language used
to identify messages, and process them according to their configuration.
For example, you can configure an MDA to save all messages from
[EMAIL PROTECTED] in ~/Mail/friends/john, and discard all messages that
have any combination of words "make", "money" and "fast" in their
subjects.
The existence of MDAs as separate programs whose only purpose is sorting
mail is a shining example of the distance between the worlds of Windows
and Unices.
tbw
add a few links to other flavors
http://www.flounder.net/~mrsam/maildrop/
http://www.procmail.org/
.forward
tbw
MUA - Mail User Agent
Mail User Agent is basically a glue that binds together a text editor, a
pager, and a command line interface to an MTA. You could probably write a
shell script implementing a simple MUA. And this is for the most part what
mutt does: you open a folder (see below), view a message, which is
displayed in pager [mutt has a builtin one, but can use less(1), for
example], and upon replying to it, mutt will fire your $EDITOR so you can
type it.
add a few links to other flavors
http://www.mutt.org/
folder
This is the same meaning as used in various AWBMs. While vast majority (if
not all) those use proprietary formats, requiring you to either find a
converter, often unreliable, or lose mail when you switched mailers, this
is generally not the case in the Unix world. Most of Unix mailers use one
of the few standard formats, so you can use more than one MUA to read your
mail without a hassle.
mbox
This is the traditional folder format. It is a simple plain text file
with the messages in it. Each message has this structure:
From [EMAIL PROTECTED] date
Headers
<empty line>
Body
<empty line>
Since MUAs recognize the beginning of a message in an mbox by a line
starting with "From " (this is commonly called "the From_ line"), any
lines in message bodies starting with this string will get rewritten as
">From " by your MDA.
It is not a good idea to use this folder format if you have your mail on a
NFS share. Many NFS implementations have buggy locking, and you can easily
have your folders stomped.
See mbox(5) for more info (different man pages isntalled at least with
mutt and qmail).
maildir
This is a newer format, introduced by the qmail MTA. Each folder is a
directory containing three subdirectories: tmp/, new/, and cur/. When your
MDA delivers a message to a maildir folder, it writes it to the tmp/
directory, and when the message is succesfully written, it moves it to
the new/ directory. Since this is atomical operation, the folder is safe
from curruption.
It is quite safe to use over NFS.
Also, it can be easier when you want to process your mail using regular
tools provided in the base system: you can use a small shell script to
move old mail away from your "active" folders to an "archive" using
find(1), xargs(1), and sed(1).
This format can get _very_ slow with large folders on filesystems that do
not handle directoris with many files in them. This should include the
Linux ext2fs.
FreeBSD post-4.4 FFS with softupdates and dirhash should shine with this
format.
See maildir(5) (installed with qmail) for more info.
6) SCENARIO
In the rest of this document, I will assume the following example
configuration:
* two POP3 accounts (one only accessible with SSL), and an IMAP one,
* FreeBSD machine connected to the interned via dialup.
Next, I will use these example values:
POP3 account nr. 1: [EMAIL PROTECTED]
(login johny, password mypass123, server pop3.mymail.net)
POP3 account nr. 2: [EMAIL PROTECTED]
(login jdoe234, password foobar, server mailhub.corporate.com)
this one is only accessible through SSL
You will probably want to have your mail stored in an organized manner
somewhere in your $HOME. This is usually ~/Mail/. For example, mine looks
something like this:
/home/roman/Mail/
IN.mail.cz
IN.mobil.cz
IN.spool@ -> /var/mail/roman
lists/
freebsd-questions
mutt-users
php-dev
...
postponed
sent
var/
costra
var
zak
trash
work/
cvs
mobil.cz
The structure of this directory is of course dependant on its contents, but
the point here is that it pays off to think it through. Grouping of 'folders'
will make
So, what will we use to get this to work?
getmail - to fetch the messages
maildrop - to sort them
* MDA (or two) to fetch the messages off the servers, filter them, and store
on the computer;
* MUA to display the messages, and let us to compose replies;
* MTA that will accept the outgoing messages from the MUA, and take care of
their departure;
* a couple of other utilities.
+----------+ +----------+
| server A | _____*| server B |
+----------+ \ / +----------+
* *_____ /
| |_____| |
| stunnel \___\ __
\ / * _____
\ | |_____| getmail
\ | |
\ ______ ______/ __*__
|______| |_____| procmail
* |
\ __*__
\ |_____| maildrop
\ /
\ /
\_____ *
____ | |
|____|*-----*| | mutt
gnupg ___*|_____|*___________
/ * * \
_______/ / \ \
/ / \ \
__*_ __*_ _*___ __\__
| | | | | | | |
|____| |____| |____| |____|
vim abook links sigrot
7) SOFTWARE
XXX: this needs to go to the TERMS section
Note: I don't know a Unix-like operating system that doesn't come with
Sendmail preinstalled. What's Sendmail? Sendmail is an MTA: it can send out,
and receive, email messages. But it isn't exactly designed to be used on
personal computers that just need to hand the outgoing mail over to their
ISP's SMTP servers: it _is_ a full-blown SMTP server, and one that's
notoriously known for its cryptic configuration. We'll ditch it.
Python
http://www.python.org/
Stunnel -- will explain later
http://www.stunnel.org/
Getmail -- an MDA
http://www.qcc.sk.ca/~charlesc/software/getmail-2.0/
Maildrop -- another MDA
http://www.flounder.net/~mrsam/maildrop/
Procmail -- yet another MDA
http://www.procmail.org/
Mutt -- the famous MUA
http://www.mutt.org/
Vim -- the best text editor out there
http://vim.sf.net/
Abook -- yes, an address book
http://abook.sf.net/
PGP/GnuPG -- you want privacy, right?
http://www.gnupg.org/
Links -- let's integrate mail and web
http://artax.karlin.mff.cuni.cz/~mikulas/vyplody/links/
Nullmailer -- a simple MTA
http://untroubled.org/nullmailer/
sigrot -- let's have some fun with signatures (or something else for signatures)
-- your local SunSite
8) INSTALLATION
If you use FreeBSD like I do, this step is perhaps the easiest of the whole
process. If you don't know what all those various programs do, just read on.
Note, however, that this document doesn't try to replace the wealth of manuals,
manpages, INSTALL and README files that comes with these packages.
---------->8---------->8----------
#! /bin/sh
SEC = security/stunnel security/gnupg security/pgp6
MAIL = mail/getmail mail/maildrop mail/procmail mail/mutt mail/abook mail/nullmailer
LANG = lang/python
APPS = $LANG $SEC $MAIL
cd /usr/ports && \
for app in $APPS; do \
(cd $app && make install clean) \
done
----------8<----------8<----------
XXX Of course, if you feel like adding any Linux- (or whatever-) specific
XXX equivalents, I won't complain. :) To be honest, I'm starting to think
XXX whether it's worth the effort to try to keep this cross-platform...
XXX
XXX There are two mutually-exclusive goals I would like to achieve:
XXX 1) make this howto as helpful as possible
XXX 2) keep it useful for the "target audience"... I mean... just a look at
XXX the list of the required packages would probably distract anyone not
XXX really interested in going this path, and honestly, I don't think that
XXX someone who can't type "make install" is worth the trouble...
XXX when I started this thing, I was thinking about people who are like you
XXX or me, people who can read man pages, who can read the output of
XXX ./configure --help, and understand (simple) shell scripts.
XXX
XXX I definitely do NOT want disappointed outlook users send me hate mails
XXX about mutt being stupid or something... as an example, I _don't_ want to
XXX burn energy on making this understandable to people like the author of
XXX this email:
XXX
XXX > Date: Thu, 22 Nov 2001 12:11:08 -0800
XXX > To: <[EMAIL PROTECTED]>
XXX > Subject: ?
XXX >
XXX > Hi: I did install the 4.4 version of Linux X86, and after spending 6 hours
XXX > trying to get thru LOGIN I get a dollar sign, and nothing else.
XXX > What do you use this thing for, it does not seem to be able to do any thing.
XXX > Please help me understand, I am lost.
XXX > Thank you.
XXX
XXX hmmm... i'm afraid it's not that easy to find the right balance. for
XXX example, when I say "create a directory foo", should I suppose the user
XXX is dumb enough to not be able do it by himself? if so, I can include
XXX $ mkdir foo
XXX but if I assume the reader is so clueless, I must also assume he doesn't
XXX know how to edit files... and this is going to hell. I don't want to wind
XXX up writing "Unix for complete dummies from the ground up" when this should
XXX be helping people set up a handful of console-based programs.
XXX anyway... someone _that_ green (or clueless) wouldn't want to use mutt
XXX
XXX as for the cross-platform issue... I think I don't want this howto to be
XXX aimed at people who don't know their operating systems enough to gather
XXX that when I say "install foo-1.0", they will have to do whatever it takes
XXX on their platform to install a piece of software. again, this is a _mail_
XXX howto.
9) CONFIGURATION
This is where the fun begins.
** getmail:
First, you need to get mail from the servers to your PC. That's where getmail
comes in handy. To configure it, follow these steps:
1) create a directory called .getmail/ in your home directory:
$ mkdir ~/.getmail
2) create the configuration file for getmail, which has syntax you might be
familiar with from the various windows ini files, and is really simple:
$ vim ~/.getmail/getmailrc
---------->8---------->8----------
[default]
readall = 1
delete = 1
verbose = 0
[private]
server = pop3.mymail.net
port = 110
username = johny
password = mypass123
postmaster = "|/usr/bin/env maildrop ~/.maildrop/mymail.net.rc"
[work]
server = localhost
port = 10110
username = jdoe234
password = foobar
postmaster = "|/usr/bin/env maildrop ~/.maildrop/corporate.com.rc"
----------8<----------8<----------
** stunnel:
edit /usr/local/etc/rc.d/stunnel.sh.sample:
---------->8---------->8----------
case "$1" in
start)
${STUNNEL} -c -r mailhub.corporate.com:995 -d 10110
;;
----------8<----------8<----------
** maildrop:
.maildrop/maildroprc
---------->8---------->8----------
MAILDIR = "$HOME/Mail"
LISTDIR = "$MAILDIR/lists"
PHPDIR = "$LISTDIR/php"
INBOX = "$MAILDIR/IN.mymail.net"
include ~/.maildrop/spamfilterrc
if (/^list-post: <mailto:[EMAIL PROTECTED]>/)
{
to "$PHPDIR/php-dev"
}
if (/^list-id: <freebsd-questions.FreeBSD.ORG>/)
{
to "$LISTDIR/freebsd-questions"
}
to "$INBOX"
----------8<----------8<----------
~/.maildrop/spamfilterrc:
---------->8---------->8----------
----------8<----------8<----------
Note: the maildrop package includes a program called reformail. This program
is very useful if you have a folder (in the mbox format -- see above) of
unsorted mail that you want to pull through maildrop. This may well happen
(this exact thing happened to me) if you have been using your computer for
some time now -- you will have mail in /var/mail/$USER, aka spool. If you fed
this file directly to maildrop, it would treat it as a single large message,
which isn't what you want. Here, reformail comes handy, as it can split the
mbox into individual messages:
$ reformail -s| maildrop ~/.maildrop/maildroprc
Now, you can cafely concatenate the file.
** mutt:
this only runs @startup (folders created while mutt is running don't show up
without :source -ing .muttrc
messages = `find ~/Mail -type f|perl -ne 'chomp;print "$_ " unless
m/.*(sent|postponed|trash)$/'`
figure out (semi-)automatic detection of mailing lists [where possible]
problem being it's not standardized :[
---------->8---------->8----------
set signature = "~/bin/signature|"
roman@roman ~ > cat ~/bin/signature
#!/bin/sh
#
# results in something like:
# FreeBSD 4.4-STABLE
# 2:56PM up 3 days, 39 mins, 13 users, load averages: 0.09, 0.11, 0.14
echo `uname -sr` && echo `uptime`
----------8<----------8<----------
roman@roman ~ > ls -l ~/bin/
total 1
-rwxr--r-- 1 roman roman 150 Oct 26 15:07 signature
vim:tw=78:ts=4:fo+=cnoqrt:com=n\:XXX
