Re: smtpd.conf new grammar

[Leighton Sheppard]

On Thu, May 24, 2018 at 02:18:56PM +0200, Gilles Chehade wrote:
> Hi,
> 
> I have just committed a major change in smtpd that'll require smtpd.conf
> to be rewritten before your update to the new code.
> 
> The new grammar is not TOO different from the former one, a lot of stuff
> remains exactly identical, but the ruleset is now split into two parts:
> 
> - a named action
> - a matching pattern which is associated to a named action
> 
> In effect, instead of having:
> 
> accept from any for local deliver to mbox
> 
> 
> You will have:
> 
> action "my_action" mbox
> 
> match from any for local action "my_action"
> 
> 
> There are a few keywords that have been shortened too but all in all the
> switch to new grammar is easy, the smtpd.conf man page has been updated,
> and it continues being improved thanks to ingo and jmc.
> 
> The man page by itself should be enough to do the switch.
> 
> Since this is quite a major change, I also wrote a post that describes a
> conversion of my own complex smtpd.conf to new grammar:
> 
> https://poolp.org/posts/2018-05-21/switching-to-opensmtpd-new-config/
> 
> 
> I have also compiled a list of directives recognized by the parser which
> I intend to use for regress tests:
> 
> https://poolp.org/~gilles/smtpd.conf
> 
> 
> As for the reasons behind the change they are numerous, I explained some
> at EuroBSDCon 2017, I explained some on my blog, the bottom line is that
> while one-line rules were apparently an awesome idea, they were actually
> a design error that had consequences on pretty much the entire daemon.
> 
> We didn't realize it until a few months ago, we tried hard to maintain a
> one-line rule grammar but it became more and more obvious that this just
> isn't doable without creating issues and unnecessary complexity.
> 
> The new grammar is cleaner, it helped remove ~700 lines of complex code,
> made the handling of .forward files as well much safer, removed a lot of
> very unpleasant side-effects most people didn't even realize existed ...
> until they hit that one case for which we had no way to work around.
> 
> 
> Anyways,
> looking forward for you to test and report how it works for you :-)
> 
> 
> -- 
> Gilles Chehade
> 
> https://www.poolp.org  @poolpOrg
> 

Hi,

I upgraded my laptop and VM both to the latest snapshot this morning, and 
migrated the configuration for smtpd.conf to the new grammar. No issues 
to report and I like the new format!

Regards,
Leighton

Re: smtpd.conf new grammar

[Gilles Chehade]

On Thu, May 24, 2018 at 04:38:17PM -0400, Rupert Gallagher wrote:
> On Thu, May 24, 2018 at 14:18, Gilles Chehade  wrote:
> 
> > In effect, instead of having:
> > accept from any for local deliver to mbox
> >
> > You will have:
> > action "my_action" mbox
> > match from any for local action "my_action"
> 
> It may solve some obscure technical problem, but is a horrible thing to read 
> and write. How about keeping the best of both worlds? Leave the old beautiful 
> PF-like syntax to humans, and translate it into the newEgyptian(tm) on the 
> fly?

It doesn't solve "obscure" technical problems, it solves _many_ issues a
lot of users have reported ranging from syntax ambiguity to envelopes on
disk not reflecting changes in configuration. One-line rules have lot of
consequences which go far beyond the configuration phase: the structures
are impacted, the ruleset evaluation is impacted, the aliases expansions
are impacted, the queue is impacted, format of envelopes are impacted, I
could go on and on since this impacts almost the entire daemon actually.


The impact is not just cosmethic stuff. I removed _hundreds_ of lines of
very unnecessary and complex code that was ONLY there to make the design
error work. Some of these removals led to concrete security improvements
like .forward files, written by untrusted users, be processed with their
privileges rather than _smtpd. Not doable with one-line rules.


I could write pages about the shortcomings from the previous config, and
pages about why it can't be made to work and why the new config fixes it
in the proper way. We tried hard to find ways to retain a one-line rules
configuration but after months we exhausted the ideas and we have a much
clearer understanding that it's NOT doable. Either we accept that, or we
are just going to grow more complex and slowly stop writing code because
every time you touch an area there's a risk you run into the complexity.


You don't have to trust my word:

> How about keeping the best of both worlds? Leave the old beautiful PF-like 
> syntax to humans,
> and translate it into the newEgyptian(tm) on the fly?

If this was possible, then you could easily write a translator that will
convert old grammar to new one without removing all the benefits and not
reintroducing the complex code.

By all means, show me, I'd be impressed for real.

-- 
Gilles Chehade

https://www.poolp.org  @poolpOrg

Re: smtpd.conf new grammar

[Todd C. Miller]

On Thu, 24 May 2018 16:38:17 -0400, Rupert Gallagher wrote:

> It may solve some obscure technical problem, but is a horrible thing
> to read and write. How about keeping the best of both worlds? Leave
> the old beautiful PF-like syntax to humans, and translate it into
> the newEgyptian(tm) on the fly?

It's not bad as long as you use symbolic names for the actions.
When you have multiple match rules using the same action it may
actually be an improvement.

Here are the action and match rules from my smtpd.conf to give you
an idea of what I mean:

# Actions
action deliver-mbox mbox alias 
action dkim-relay relay host smtp://127.0.0.1:10027 helo mydomain.com
action just-relay relay

# Accept mail between local users
match for local action deliver-mbox
match for domain  action deliver-mbox

# Relay or deliver DKIM-signed messages from dkimproxy_out
match tag DKIM for any action just-relay

# Allow local net to send mail to the outside + relay bounces
# All messages are signed by dkimproxy_out on port 10027
match from local for any action dkim-relay
match from src  for any action dkim-relay

# Accept from anyone for any of our local domains and deliver to mbox
match from any for domain  action deliver-mbox

Re: smtpd.conf new grammar

[Rupert Gallagher]

On Thu, May 24, 2018 at 14:18, Gilles Chehade  wrote:

> In effect, instead of having:
> accept from any for local deliver to mbox
>
> You will have:
> action "my_action" mbox
> match from any for local action "my_action"

It may solve some obscure technical problem, but is a horrible thing to read 
and write. How about keeping the best of both worlds? Leave the old beautiful 
PF-like syntax to humans, and translate it into the newEgyptian(tm) on the fly?

smtpd.conf new grammar

[Gilles Chehade]

Hi,

I have just committed a major change in smtpd that'll require smtpd.conf
to be rewritten before your update to the new code.

The new grammar is not TOO different from the former one, a lot of stuff
remains exactly identical, but the ruleset is now split into two parts:

- a named action
- a matching pattern which is associated to a named action

In effect, instead of having:

accept from any for local deliver to mbox


You will have:

action "my_action" mbox

match from any for local action "my_action"


There are a few keywords that have been shortened too but all in all the
switch to new grammar is easy, the smtpd.conf man page has been updated,
and it continues being improved thanks to ingo and jmc.

The man page by itself should be enough to do the switch.

Since this is quite a major change, I also wrote a post that describes a
conversion of my own complex smtpd.conf to new grammar:

https://poolp.org/posts/2018-05-21/switching-to-opensmtpd-new-config/


I have also compiled a list of directives recognized by the parser which
I intend to use for regress tests:

https://poolp.org/~gilles/smtpd.conf


As for the reasons behind the change they are numerous, I explained some
at EuroBSDCon 2017, I explained some on my blog, the bottom line is that
while one-line rules were apparently an awesome idea, they were actually
a design error that had consequences on pretty much the entire daemon.

We didn't realize it until a few months ago, we tried hard to maintain a
one-line rule grammar but it became more and more obvious that this just
isn't doable without creating issues and unnecessary complexity.

The new grammar is cleaner, it helped remove ~700 lines of complex code,
made the handling of .forward files as well much safer, removed a lot of
very unpleasant side-effects most people didn't even realize existed ...
until they hit that one case for which we had no way to work around.


Anyways,
looking forward for you to test and report how it works for you :-)


-- 
Gilles Chehade

https://www.poolp.org  @poolpOrg