Hello,
This is a copy of a mail I sent to OpenBSD hackers a few days ago so you are
aware of work
being done on OpenSMTPD by Omar Polo.
~~~
TL;DR: proposal to change table backends wire protocol to one that's closer to
filters, it
has proven to work for years now, comes with many benefits and it is a
very trivial
change that we can pull in a handful of hours:
https://tmp.omarpolo.com/smtpd-tables.7.html
Longer mail with historical context.
A long time ago, OpenSMTPD shipped with two table backends for its table API:
file and db.
Then we started seeing a need for ldap and SQLite, so we wrote backends for
them and since
we had aldap.[ch] and SQLite in base this wasn't really a problem.
Then we started seeing a need for mySQL and postgreSQL which became problematic
because we
didn't want OpenSMTPD to ship with backends that couldn't be built in base out
of the box,
but also because if we had to fix a bug in table_mysql, it meant we had to do
an OpenSMTPD
release to provide the fix for portable users.
To solve this, we introduced the "procexec" API, a mean for OpenSMTPD to spawn
tables in a
separate process with socketpairs in place to perform lookups and obtain
results via imsg.
This made it possible to release table backends separately as standalone
programs, without
limitations to their dependencies and without requiring releases to be
synchronised. Table
backends were releases as part of OpenSMTPD-extras, and we could do several
releases of it
in the timespan of a single OpenSMTPD release.
Later, we did the same for filters, it turned out to be much more complex and
we failed so
hard our first attempts that we reverted and waited two years to make a new
proposal. That
new proposal replaced imsg with an stdio line-based protocol which was much
simpler for us
to implement in the daemon and it had the benefit that users could easily
interact with it
to handle use-cases we didn't care about using the languages they wanted. I
have seen some
filters in awk, rust, C, golang, Perl, Python, ... users are happy with the
simple API and
we didn't have to handle exotic use-cases in the daemon, which is good.
Recently there's been a few contributions to table-ldap in OpenSMTPD-extras,
the first one
in years, and I think that the fact table backends still use "procexec" is a
reason why we
can't rely on the community to contribute: you can't just read a line, parse it
and reply,
instead you need to know how imsg works, you need to understand non-blocking
async I/O and
you can't start a table backend of your own in a preferred language... so
unless one of us
does the work, it stales (I wrote table_ldap ... I don't even use or care for
ldap).
A while back I told eric@ we could switch tables to a filter-like protocol, he
agreed, but
we never got to do it. Recently, as op@ began contributing more and more, I
told him about
that idea again and my 3-steps plan to switch to it:
1- implement a table_stdio backend which translates imsg protocol to stdio
protocol
2- convert all table backends to the new protocol
3- move the table_stdio code in smtpd so we no longer need table_stdio
These are all trivial changes and op@ actually already implemented 1- and 2-,
we'd like to
start the work on 3- considering that all of this project is easy: we're
changing the wire
protocol but the table API remains 100% the same from OpenSMTPD and filters
perspective.
The new table protocol is identical to the smtpd-filters(7) protocol (draft by
op@), it is
readable in HTML at the following link and I've included the man page at end of
mail:
https://tmp.omarpolo.com/smtpd-tables.7.html
The operations are exactly the same as with imsg, and since the imsg protocol
already made
results available as serialised strings, we already parse them from string to
structs, and
we'll re-use the same code. The change is mainly in not encapsulating them in
imsg packets.
~~~
Note that we're past this as of today:
1- table_stdio was implemented as a PoC:
- https://github.com/OpenSMTPD/table-stdio
2- all table backends from OpenSMTPD-extras were converted to the new interface:
- https://github.com/OpenSMTPD/table-socketmap
- https://github.com/OpenSMTPD/table-passwd
- https://github.com/OpenSMTPD/table-ldap
- https://github.com/OpenSMTPD/table-sqlite
- https://github.com/OpenSMTPD/table-redis
- https://github.com/OpenSMTPD/table-postgres
- https://github.com/OpenSMTPD/table-mysql
3- Omar has sent a mail to tech@ with the diff to integrate protocol in smtpd:
- https://marc.info/?l=openbsd-tech&m=171439783225288&w=2
There you go, now you know !
