Re: Overview of rcpt checkers

[Juerd Waalboer]

Guy Hulbert skribis 2007-12-17  9:33 (-0500):
> One more.
> I think you should post something about this to the qmail list.  They
> will probably put the link below onto the qmail.org page.

I'm not on that list, and I don't feel comfortable plugging my own work
as a first post. It'd be rude.

If anyone who has already contributed to the qmail list wants to notify
the folks over there, please be my guest!
-- 
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  
  Convolution: ICT solutions and consultancy <[EMAIL PROTECTED]>

Re: Overview of rcpt checkers

[Guy Hulbert]

On Mon, 2007-12-17 at 13:39 +0100, Juerd Waalboer wrote:
> Hi list,
> 
> Early this morning [CET] I released Qmail::Deliverable version 1.03 to
> CPAN. This release includes Qmail::Deliverable::Comparison. Thanks again
> for all the suggestions.

One more.

I think you should post something about this to the qmail list.  They
will probably put the link below onto the qmail.org page.

The authors of netqmail have released a 1.06 (on nov 30, i think), which
integrates their patches and claim to be working on a 1.07, which will
integrate TLS and AUTH.  Possibly the 1.07 is already released but I
haven't checked for a few days.

> 
> With this release, I will no longer announce changes to ::Comparison on
> the mailinglist. The latest version can be found at
> http://tnx.nl/Qmail::Deliverable::Comparison - which is a redirect to
> the document on search.cpan.org.
-- 
--gh

Re: Overview of rcpt checkers

[Juerd Waalboer]

Hi list,

Early this morning [CET] I released Qmail::Deliverable version 1.03 to
CPAN. This release includes Qmail::Deliverable::Comparison. Thanks again
for all the suggestions.

With this release, I will no longer announce changes to ::Comparison on
the mailinglist. The latest version can be found at
http://tnx.nl/Qmail::Deliverable::Comparison - which is a redirect to
the document on search.cpan.org.

I'll probably shuffle some solutions between level 0 and level 1, but I
don't expect any huge changes anymore. This categorization by qmail
support level (or "outoftheboxiness") works well for my purpose; much
better than the feature matrix that I originally used.

Here's the changelog for Q::D:

| Revision history for Perl extension Qmail::Deliverable.
| 
| 1.03
| - qmail-deliverabled now takes a pidfile on the command line, and can
|   stop itself using that.
| - Added Qmail::Deliverable::Comparison to compare with other Qmail
|   deliverability checkers.
| - Now correctly loads "me" if "locals" does not exist.
| - Provided an example init.d script.
| 
| 1.02
| - Support for bouncesaying, although without using the configured
|   error message. Plesk puts |bouncesaying in .qmail-default.
| 
| 1.01
| - qpsmtpd plugin check_qmail_deliverable installs as a binary, so
|   that it has a manpage. If you execute it, you get installation
|   instructions.
| - $Qmail::Deliverable::Client::SERVER can be a callback now
| - Plugin now uses the callback option for cleaner code
| - Plugin now allows hostnames instead of IP adresses only
| - Exclusions now enabled for smtproutes
| - ::Client::qmail_local no longer returns undef on connection error,
|   because undef already meant something else
| - qmail-deliverabled has basic statistics in $0
| - minor documentation updates
| 
| 1.00  Sun Dec  2 17:36 2007
| - first CPAN release
-- 
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  
  Convolution: ICT solutions and consultancy <[EMAIL PROTECTED]>

Re: Overview of rcpt checkers

[Juerd Waalboer]

Peter J. Holzer skribis 2007-12-10 23:01 (+0100):
> The aliases plugin supports catch-all domains and catch-all extensions,

Support for catch-all domains isn't obvious from the documentation, but
indeed the source clearly supports a [EMAIL PROTECTED] wildcard, with a literal 
*
in the alias file.
-- 
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  
  Convolution: ICT solutions and consultancy <[EMAIL PROTECTED]>

Re: Overview of rcpt checkers

[Juerd Waalboer]

Peter J. Holzer skribis 2007-12-10 23:01 (+0100):
> > Qmail's modular design made replacing the SMTP daemon very simple, and 
> > because
> > development on qmail stopped with version 1.03 in 1993, several replacement
>^
>  1998.

Heh, I've got this patch 4 times now. I'm lucky to have such careful
reviewers here! Thanks.

> The aliases plugin supports catch-all domains and catch-all extensions,
> although the separator character is hard-coded to be "+" (like in
> sendmail) instead of "-" (as in qmail). That should be fairly easy to
> change. User extensions are not supported, but I'm not sure how they are
> different from catch-all extensions at the the MTA level - IMHO that's a
> matter for the MDA.

I'll take this into account in the next version. However, I'm not going
for "easy to change" unless it's actually a configuration option.

Qmail has, in addition to catch all user extensions, specific user
extensions. E.G., on a bare qmail installation, I could have both
[EMAIL PROTECTED] and [EMAIL PROTECTED], without having
[EMAIL PROTECTED], if I wanted.

> (But I am thinking about "user-registered" extensions at the MX: So if I
> have a username of "hjp", and I have registered and extension of "foo",
> mails to  are accepted but mails to
>  are not ("/" here is an arbitrary separator
> character - it could be "-", but "-" does occur in personal names, so
> I'm reluctant to use that - it should be configurable anyway).)

That's what qmail allows by default. The aliases plugin currently
*strips* the extension before checking (and adds it back after
expanding), so indeed that makes it awkward to use with a "-" separator.

Note that qmail has no simple way of preserving the extension in a
forward instruction.
-- 
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  
  Convolution: ICT solutions and consultancy <[EMAIL PROTECTED]>

Re: Overview of rcpt checkers

[Peter J. Holzer]

On 2007-12-09 23:42:51 +0100, Juerd Waalboer wrote:
> Juerd Waalboer skribis 2007-12-07  1:53 (+0100):
> > I'm going to email all the authors, as far as I can find their
> > addresses, so they have a chance to respond to my critique.
> 
> Instead of complaining about lack of qmail support, I've decided to
> simply recategorize things into less offending "support levels". After
> all, many weren't even written with qmail in mind. I'm including them
> anyway, because they might still be viable alternatives for some
> systems.

Great.

> Qmail's modular design made replacing the SMTP daemon very simple, and because
> development on qmail stopped with version 1.03 in 1993, several replacement
   ^
   1998.

> =head3 Level 0
> 
> Level 0 solutions are limited to checking specific [EMAIL PROTECTED], without 
> support
> for catch-all addresses.
> 
> =over 4
> 
[...]
> =item aliases plugin for qpsmtpd
> 
> Written in Perl, checks against a text file with aliases and expands them.
> 
> http://www.hjp.at/projekte/qpsmtpd/aliases/aliases_check.html
> 
[...]
> =back
> 
> =head3 Level 1
> 
> Level 1 solutions are limited to checking [EMAIL PROTECTED] and also provide 
> support
> for catch-all domains ([EMAIL PROTECTED]) and sometimes catch-all extensions
> ([EMAIL PROTECTED]), but not specific user extensions
> ([EMAIL PROTECTED]).


The aliases plugin supports catch-all domains and catch-all extensions,
although the separator character is hard-coded to be "+" (like in
sendmail) instead of "-" (as in qmail). That should be fairly easy to
change. User extensions are not supported, but I'm not sure how they are
different from catch-all extensions at the the MTA level - IMHO that's a
matter for the MDA.

(But I am thinking about "user-registered" extensions at the MX: So if I
have a username of "hjp", and I have registered and extension of "foo",
mails to  are accepted but mails to
 are not ("/" here is an arbitrary separator
character - it could be "-", but "-" does occur in personal names, so
I'm reluctant to use that - it should be configurable anyway).)

hp

-- 
   _  | Peter J. Holzer| It took a genius to create [TeX],
|_|_) | Sysadmin WSR   | and it takes a genius to maintain it.
| |   | [EMAIL PROTECTED] | That's not engineering, that's art.
__/   | http://www.hjp.at/ |-- David Kastrup in comp.text.tex


signature.asc
Description: Digital signature

Re: Overview of rcpt checkers

[Juerd Waalboer]

Juerd Waalboer skribis 2007-12-09 23:42 (+0100):
> Qmail::Deliverable module, as Qmail::Deliverable::CPAN. Thanks to the

s/CPAN/Comparison/ :)
-- 
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  
  Convolution: ICT solutions and consultancy <[EMAIL PROTECTED]>

Re: Overview of rcpt checkers

[Juerd Waalboer]

Juerd Waalboer skribis 2007-12-07  1:53 (+0100):
> I'm going to email all the authors, as far as I can find their
> addresses, so they have a chance to respond to my critique.

Instead of complaining about lack of qmail support, I've decided to
simply recategorize things into less offending "support levels". After
all, many weren't even written with qmail in mind. I'm including them
anyway, because they might still be viable alternatives for some
systems.

Attached is the draft POD of what I want to release with my
Qmail::Deliverable module, as Qmail::Deliverable::CPAN. Thanks to the
people who've replied to my feature matrix, it's more complete than
that.

Any comments?
-- 
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  
  Convolution: ICT solutions and consultancy <[EMAIL PROTECTED]>
=head1 NAME

Qmail::Deliverable::Comparison - Overview of recipient checkers for qmail

=head1 SYNOPSIS

There is more than one way to do it
-- Larry Wall

=head1 DESCRIPTION

Qmail::Deliverable is not the only software that checks if a local qmail user
exists. Other solutions have existed for a long time, but after evaluating
them, I felt more comfortable rolling my own.

While exploring the existing options, I learned their strengths and weaknesses.
I revisited them all again after writing Q::D, and now share my findings.

It is most amazing, though, that no other solution provides a re-usable
library. One comes very close, by providing a daemon that you can query via a
very simple protocol. Q::D has both a library and a daemon. Also, good support
for dot-qmail files like .qmail-ext-default, is rare, but absolutely essential
for interoperability with, for example, ezmlm.

The main problem with checking address validity during the SMTP transaction, is
one of permissions and separation. Root access is required to peek into home
directory of system users. Privilege separation is one of the classic strengths
of qmail, and it would be unwise to break that. Additional mechanisms are needed
to keep things separate, but as you can see, this wheel has been invented plenty
of times.

Qmail's modular design made replacing the SMTP daemon very simple, and because
development on qmail stopped with version 1.03 in 1993, several replacement
daemons have been released. There is a patch for qmail-smtpd, to make it
extensible with plugins, called qmail-spp. Then, there is a daemon called
magic-smtpd. My favourite, however, is the very extensible Perl based qpsmtpd.

http://cr.yp.to/qmail.html

http://qmail-spp.sourceforge.net/

http://www.linuxmagic.com/opensource/magicmail/magic-smtpd

http://smtpd.develooper.com/

=head2 Using qmail-local logic

Any solution that is to integrate well with existing qmail systems, has to copy
the delivery logic that qmail itself uses.

They need to use the qmail control files L and C, the
L text file or CDB, and some mechanism to detect local users
(getpw).

Vpopmail support can be added by checking dot-qmail files for the presence of
C and C. In a similar fashion, fastforward and
bouncesaying can be supported at SMTP level. Bouncesaying is used by Plesk.

=head2 List of patches and plugins for qmail SMTP daemons

I've divided the solutions into support levels. Level 0 supports only lists
of specific email addresses, level 3 solutions are (almost) drop-in compatible
with existing setups.

Before installing any solution, carefully read the documentation (if there is
any) and/or source code to determine if everything you need is sufficiently
supported.

As you know, programmers start counting at 0 :)

=head3 Level 0

Level 0 solutions are limited to checking specific [EMAIL PROTECTED], without 
support
for catch-all addresses.

=over 4

=item db/user plugin for qpsmtpd

Written in Perl, checks against an SQL database

http://dienstleistung-kultur.de/qpsmtpd/

=item magic-smtpd, support built in

Written in C, checks against a Berkely DB.

http://www.linuxmagic.com/opensource/magicmail/magic-smtpd

=item aliases plugin for qpsmtpd

Written in Perl, checks against a text file with aliases and expands them.

http://www.hjp.at/projekte/qpsmtpd/aliases/aliases_check.html

=item check_user_cdb plugin for qpsmtpd

Written in Perl, checks against a CDB

http://ankh-morp.org/~vetinari/projects/qpsmtpd/

=item check_vuser plugin for qpsmtpd

Written in Perl, checks against a directory tree.

http://ankh-morp.org/~vetinari/projects/qpsmtpd/

=back

=head3 Level 1

Level 1 solutions are limited to checking [EMAIL PROTECTED] and also provide 
support
for catch-all domains ([EMAIL PROTECTED]) and sometimes catch-all extensions
([EMAIL PROTECTED]), but not specific user extensions
([EMAIL PROTECTED]).

=over 4

=item goodrcptto patch for qmail-smtpd

Written in C, checks against a text file or CDB.

http://web.netdevice.com:9080/qmail/patch/

=item validrcppto patch for qmail-smtpd

Written in C, checks against a text file.

htt

Re: Overview of rcpt checkers

[Juerd Waalboer]

Ernesto skribis 2007-12-07 13:39 (+0100):
> Please include
> * db/suite -> db/user

I'll consider these too, thanks.
-- 
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  
  Convolution: ICT solutions and consultancy <[EMAIL PROTECTED]>

Re: Overview of rcpt checkers

[Ernesto]

Please include

* db/suite -> db/user

in your overview.

See http://dienstleistung-kultur.de/qpsmtpd/

Ernesto

Re: Overview of rcpt checkers

[Juerd Waalboer]

Hanno Hecker skribis 2007-12-07  9:06 (+0100):
> ...and rcpt_ldap, check_dns_user, aliases, rcpt_regexp (all listed on
> http://wiki.qpsmtpd.org/plugins)

(And check_user_cdb, check_vuser. Any more?)

I excluded ldap because I don't speak LDAP and couldn't verify the
functionality easily.

The others you mention were excluded because they're not *qmail* rcpt
checkers. But since they can obviously be used for that with the right
database generator, perhaps I need different categories. I have now used
pregenerated/dynamic, but already some are dynamic in nature but that
doesn't work: if something rereads its list of valid recipients for
every message, it should be considered dynamic too.

So perhaps I should divide the software into programs that generate a
database of valid recipients, and programs that check against a
database, where the kind of database is another column. A special
"qmail" database type would mean: uses qmail-local logic.

I need to think about how I'm going to put this in a matrix.
-- 
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  
  Convolution: ICT solutions and consultancy <[EMAIL PROTECTED]>

Re: Overview of rcpt checkers

[Hanno Hecker]

On Fri, 7 Dec 2007 01:47:41 +0100
Juerd Waalboer <[EMAIL PROTECTED]> wrote:

> Robin Bowes skribis 2007-12-07  0:24 (+):
> > You forgot my plugin:
> > http://robinbowes.com/projects/check_validrcptto_cdb
> 
> Sorry. Will add to the next version.
...and rcpt_ldap, check_dns_user, aliases, rcpt_regexp (all listed on
http://wiki.qpsmtpd.org/plugins)

Re: Overview of rcpt checkers

[Juerd Waalboer]

James Turnbull skribis 2007-12-07 11:30 (+1100):
> This would make a great addition to the Wiki - http://wiki.qpsmtpd.org

Agreed in principble, but I'm planning to release this as a document on
CPAN, and don't want to maintain two versions. A link to the page on
CPAN, when it's there, is of course very much possible.

I'm going to email all the authors, as far as I can find their
addresses, so they have a chance to respond to my critique. I'm not
going to wait longer than a few days though, because I want to go ahead
and write an article about replacing qmail-smtpd with qpsmtpd, and want
the document ready by then.

On IRC I've already gotten way too many "a solution already exists, stop
wasting our time with your announcement" responses, so I extended my
notes to a readable document to show whoever wants to read it, that a
sufficiently compliant solution did not pre-exist.
-- 
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  
  Convolution: ICT solutions and consultancy <[EMAIL PROTECTED]>

Re: Overview of rcpt checkers

[Juerd Waalboer]

Robin Bowes skribis 2007-12-07  0:24 (+):
> You forgot my plugin:
> http://robinbowes.com/projects/check_validrcptto_cdb

Sorry. Will add to the next version.
-- 
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  
  Convolution: ICT solutions and consultancy <[EMAIL PROTECTED]>

Re: Overview of rcpt checkers

[James Turnbull]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Juerd Waalboer wrote:
> For your convenience,
> 

This would make a great addition to the Wiki - http://wiki.qpsmtpd.org

regards

James Turnbull

- --
Author of:
- - Pulling Strings with Puppet
(http://www.amazon.com/gp/product/1590599780/)
- - Pro Nagios 2.0
(http://www.amazon.com/gp/product/1590596099/)
- - Hardening Linux
(http://www.amazon.com/gp/product/159059/)

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHWJQ89hTGvAxC30ARAoMSAKCx/mcS9AkFdr0mRxBkwiVs4kZc5wCglhjM
xBhZo0/Cea9Qb2qXIQ+OBVk=
=Yv6o
-END PGP SIGNATURE-

Re: Overview of rcpt checkers

[Robin Bowes]

You forgot my plugin:

http://robinbowes.com/projects/check_validrcptto_cdb

R.

Overview of rcpt checkers

[Juerd Waalboer]

For your convenience,

   +--includes build script
Pregenerated list  | +--script uses qmail-local logic
   | | +--supports users/assign
   | | | +--detects local users/getpw
   | | | | +--.qmail-default/catch-all
   | | | | | +--.qmail-ext-default
   | | | | | | +--vpopmail
   | | | | | | | +--ezmlm
   | | | | | | | | +--bouncesaying
   | | | | | | | | | +-- db/cdb files
  for lang V V V V V V V V V V
Solution:
goodrcpttoq-s CY n n Y Y n x n x Y
check_goodrcptto  qp  perl n x x x Y n x x x n
validrcptto   q-s Cn x x x Y n x x x n
validrcptto.cdb   q-s CY Y Y Y Y n Y n n Y
magic-smtpd builtin   -   Cn x x x Y n x x x Y

   +--uses qmail-local logic
Dynamic (real time) checking   | +--queries daemon
   | | +--supports users/assign
   | | | +--supports local users(getpw)
   | | | | +--.qmail-default
   | | | | | +--.qmail-ext-default
   | | | | | | +--vpopmail
   | | | | | | | +--ezmlm
   | | | | | | | | +--bouncesaying
   | | | | | | | | | +-- cdb files
  for lang V V V V V V V V V V
Solution:
qmail-realrcptto  q-s CY n Y P Y Y n I n Y
check_deliveryqp  perl Y n Y Y Y P Y ? n Y
vpopmail_check_recipient  spp sh   n n n n P P Y Y n x
local_check_recipient spp sh   n n n P Y P n n n N
check_recipient   spp perl n n P P Y P n I n N
vpopmail-check-user   m-s sh   n n n n n n Y n n x
qmail-verify  q-s CY Y Y Y Y Y n I n Y
check_qmail_deliverable   qp  perl Y Y Y Y Y Y Y I P N

   q-s is qmail-smtpd, qp is qpsmtpd, spp is qmail-spp, m-s is magic-smtpd

   Y = yes, n = no, I = implied by .qmail-ext-default, P = partial, x =
   not applicable

I will release a more verbose document with a future version of
Qmail::Deliverable, and this comparison document will be called
Qmail::Deliverable::Comparison
-- 
Met vriendelijke groet,  Kind regards,  Korajn salutojn,

  Juerd Waalboer:  Perl hacker  <[EMAIL PROTECTED]>  
  Convolution: ICT solutions and consultancy <[EMAIL PROTECTED]>